Image processing device, endoscope system, and method of operating the image processing device

Abstract

Provided are an image-processing device, an endoscopic system, and an image-processing method, for stably displaying a result of determination related to a disease by using an endoscopic image.　An image-processing device 16 comprising a processor, wherein the processor acquires, by using an endoscope 12, a plurality of endoscopic images by imaging an object to be observed, calculates a raw score related to determination of the severity or stage of a disease to be observed, on the basis of each of the endoscopic images, determines a final score on the basis of the raw scores, and performs control to display, in real time on a display 18, the final score and / or the temporal change of the final score.

Description

[Technical field]

[0001] The present invention relates to an image processing device, an endoscope system, and an image processing method for performing diagnostic support using an endoscopic image captured by an endoscope. Operation of the device It concerns the method. [Background technology]

[0002] In view of the above, the present invention provides an image processing device, an endoscope system, and an image processing method for stably displaying a diagnosis result regarding a disease using an endoscopic image. Operation of the device The present invention aims to provide a method.

[0003] Techniques for making it easier to use CAD have been disclosed. For example, an image analysis device capable of automatically distinguishing between super-magnified and non-magnified images for CAD using super-magnified images in endoscopic images is known (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-111040 A Summary of the Invention [Problem to be solved by the invention]

[0005] In real-time CAD using endoscopic images, the position of the observation target in the endoscopic image is easily changed, so stable judgment results may not be obtained. For example, when judging the stage of ulcerative colitis by image processing of endoscopic images, the judgment result may change slightly if the observation position changes slightly, so the result of stage judgment may not be stable. Therefore, in order to make CAD easier to use, it is desired to obtain stable judgment results.

[0006] In view of the above-mentioned circumstances, an object of the present invention is to provide an image processing device, an endoscope system, and an image processing method that can stably display a diagnosis result regarding a disease using an endoscopic image. [Means for solving the problem]

[0007] The present invention provides an image processing device, comprising a processor, which acquires a plurality of endoscopic images obtained by capturing images of an observation target at different times using an endoscope, calculates a raw score related to the assessment of the severity or stage of a disease of the observation target based on each of the endoscopic images, For each endoscopic image, a determination is made as to whether the endoscopic image is suitable or unsuitable for calculating a raw score, and a raw score is not calculated for an endoscopic image determined to be unsuitable for calculating a raw score. A raw score not-calculation ratio is obtained based on the determination results of a plurality of endoscopic images acquired during a specified period prior to the time of determination of the final score based on the raw scores, and if the raw score not-calculation ratio is equal to or greater than a specified ratio, the final score is not calculated, and if the raw score not-calculation ratio is less than the specified ratio, Calculated based on each endoscopic image Other than non-calculated A final score is determined based on multiple raw scores. and the time of final score determination It controls the display to be displayed in real time. The present invention also provides an image processing device comprising a processor. The processor acquires a plurality of endoscopic images obtained by capturing images of an observation target at different times using an endoscope, and calculates a raw score related to the assessment of the severity or stage of a disease of the observation target based on each of the endoscopic images. For each endoscopic image, the processor determines whether the endoscopic image is appropriate or inappropriate for the calculation of a raw score, and does not calculate a raw score for an endoscopic image determined to be inappropriate for the calculation of a raw score. most Final Score of Acquired during a specified period prior to the time of the decision 、 Multiple endoscopic images Based on the discrimination result, a raw score non-calculation ratio is obtained, and if the raw score non-calculation ratio is a predetermined ratio or more, the final score is not calculated. If the raw score non-calculation ratio is less than the predetermined ratio, Calculated based on Other than non-calculated From multiple raw scores Final score The final score is determined, and the final score is displayed on the display in real time. If the final score is smaller than a preset threshold value, the final score is not displayed. and the time of final score determination may be displayed on the display in real time.

[0008] The processor preferably calculates two or more different raw scores.

[0009] It is preferable that the processor calculates a raw score based on a first feature amount obtained by analyzing the endoscopic image.

[0010] The first feature amount is preferably an amount related to a superficial vascularized area, an intramucosal bleeding area, or an extramucosal bleeding area contained in the endoscopic image.

[0011] It is preferable that the processor executes a trained first machine learning model generated by inputting past endoscopic images associated with raw scores into the machine learning model, and calculates the raw score based on the endoscopic images.

[0012] The processor determines the final score of It is preferable to determine the final score from raw scores calculated based on a plurality of endoscopic images acquired during a predetermined period prior to the time of determination.

[0013] The processor preferably determines the final score by running average or FIR or IIR filtering a number of raw scores.

[0016] It is preferable that the processor determines whether the endoscopic image is suitable for calculating the raw score based on a second feature amount possessed by the endoscopic image.

[0017] It is preferable that the second feature amount is a quantity related to at least one selected from the group consisting of a halation distribution, a spatial frequency distribution, a brightness value distribution, a shadow distribution, a magnification index, and a reflected light distribution of illumination light irradiated to the object of observation of the endoscopic image.

[0018] It is preferable that the processor executes a trained second machine learning model generated by inputting past endoscopic images, which are associated with whether they are appropriate or inappropriate for calculating the raw score, into the machine learning model, and determines whether the endoscopic image is appropriate or inappropriate for calculating the raw score.

[0020] The processor, when the number of raw scores other than the uncalculated raw scores based on the plurality of endoscopic images acquired during a predetermined period is equal to or less than a predetermined number, or When the number of uncalculated raw scores is equal to or greater than a predetermined number, it is preferable to not calculate the final score.

[0021] Processor At least one graph showing the relationship between the final score and the time when the final score was determined of It is preferable to display it.

[0022] The processor performs image analysis on the endoscopic image to determine a region of an observation target included in the endoscopic image; most At least one graph showing the relationship between the final score and the time and location at which the final score was determined of It is preferable to display it.

[0023] The processor issues an instruction to acquire a still image, and when the instruction is received, the final score and the time of final score determination It is preferable to perform control to display the above.

[0024] Preferably, the disease is ulcerative colitis.

[0025] The present invention also provides an endoscope system comprising an endoscope for capturing an image of an observation target and an image processing device including a processor, the processor acquiring a plurality of endoscopic images obtained by capturing an image of the observation target at different times, and calculating a raw score related to the assessment of the severity or stage of a disease of the observation target based on each of the endoscopic images; For each endoscopic image, a determination is made as to whether the endoscopic image is suitable or unsuitable for calculating a raw score, and a raw score is not calculated for an endoscopic image determined to be unsuitable for calculating a raw score. A raw score not-calculation ratio is obtained based on the determination results of a plurality of endoscopic images acquired during a specified period prior to the time of determination of the final score based on the raw scores, and if the raw score not-calculation ratio is equal to or greater than a specified ratio, the final score is not calculated, and if the raw score not-calculation ratio is less than the specified ratio, Calculated based on each endoscopic image Other than non-calculated A final score is determined based on multiple raw scores. and the time of final score determination It controls the display to be displayed in real time. The present invention also provides an endoscope system comprising an endoscope for capturing an image of an observation target and an image processing device having a processor, the processor acquiring a plurality of endoscopic images obtained by capturing images of the observation target at different times, calculating a raw score related to the assessment of the severity or stage of a disease of the observation target based on each of the endoscopic images, determining whether each endoscopic image is appropriate or inappropriate for the calculation of the raw score, and not calculating the raw score for an endoscopic image determined to be inappropriate for the calculation of the raw score, most Final Score of Acquired during a specified period prior to the time of the decision 、 Multiple endoscopic images Based on the discrimination result, a raw score non-calculation ratio is obtained, and if the raw score non-calculation ratio is a predetermined ratio or more, the final score is not calculated. If the raw score non-calculation ratio is less than the predetermined ratio, Calculated based on Other than non-calculated From multiple raw scores Final score The final score is displayed in real time. If the final score is less than a preset threshold, the final score is not displayed. Take control.

[0026] The present invention also provides a method for operating an image processing device, comprising: an image acquiring step in which an image acquiring unit acquires a plurality of endoscopic images obtained by imaging an observation target at mutually different times; and a raw score calculating step in which a raw score calculating unit calculates a raw score related to a judgment of a severity or a stage of a disease of the observation target based on each of the endoscopic images. an inappropriate image discrimination step in which an inappropriate image discrimination unit discriminates, for each endoscopic image, whether the endoscopic image is appropriate or inappropriate for calculation of a raw score; The final scorer will: A raw score is not calculated for an endoscopic image that is determined to be unsuitable for calculation of a raw score, and a raw score non-calculation ratio is obtained based on the discrimination results of a plurality of endoscopic images acquired during a predetermined period prior to the time of determination of the final score based on the raw scores. If the raw score non-calculation ratio is equal to or greater than a predetermined ratio, the final score is not calculated. If the raw score non-calculation ratio is less than the predetermined ratio, Calculated based on each endoscopic image Other than non-calculated A final score determination step of determining a final score based on the plurality of raw scores; and the time of final score determination and a display control step of controlling the display of the image on a display in real time. The present invention also provides a method for operating an image processing device, comprising: an image acquiring step in which an image acquiring unit acquires a plurality of endoscopic images obtained by imaging an object of observation at different times; a raw score calculating step in which a raw score calculating unit calculates a raw score related to the determination of the severity or stage of a disease of the object of observation based on each of the endoscopic images; an inappropriate image discriminating step in which an inappropriate image discriminating unit discriminates, for each endoscopic image, whether the endoscopic image is appropriate or inappropriate for calculation of a raw score; and a final score determining unit not calculating a raw score for an endoscopic image that has been discriminated to be inappropriate for calculation of a raw score, most Final Score Decision Acquired during a specified period prior to a specified time 、 Multiple endoscopic images Based on the discrimination result, a raw score non-calculation ratio is obtained, and if the raw score non-calculation ratio is a predetermined ratio or more, the final score is not calculated. If the raw score non-calculation ratio is less than the predetermined ratio, Calculated based on Other than non-calculated From multiple raw scores Final score and a display control step of displaying the final score on a display in real time and, if the final score is smaller than a preset threshold, performing control not to display the final score. Effect of the Invention

[0027] According to the present invention, it is possible to stably display a diagnosis result regarding a disease using an endoscopic image. [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 is an external view of an endoscope system. [Diagram 2] FIG. 2 is an external view of an operation section of the endoscope. [Diagram 3] 2 is a block diagram showing the functions of the endoscope system. FIG. [Figure 4] FIG. 2 is an explanatory diagram illustrating four color LEDs included in the light source unit. [Diagram 5] 1 is a graph showing the spectra of violet light V, blue light B, green light G, and red light R. [Figure 6] 1 is a graph showing a spectrum of special light. [Figure 7] 13 is a graph showing the spectrum of special light containing only purple light V. [Figure 8] FIG. 2 is a block diagram showing the functions of a score processing unit. [Figure 9] FIG. 1 is an explanatory diagram showing the pattern of vascular structure that changes depending on the severity of ulcerative colitis. [Figure 10] FIG. 1 is a schematic diagram showing a cross section of the large intestine. [Figure 11] FIG. 13 is an explanatory diagram illustrating that a superficial vascularized area, an intramucosal bleeding area, and an extramucosal bleeding area are classified based on brightness value and spatial frequency. [Figure 12] 11 is an explanatory diagram illustrating the relationship between an endoscopic image and raw score calculation in a chronological order. FIG. [Figure 13] FIG. 13 is an explanatory diagram illustrating display control of a final score based on a moving average of raw scores. [Figure 14] FIG. 13 is an explanatory diagram illustrating display control of the final score showing the name of the part. [Figure 15] FIG. 13 is an explanatory diagram illustrating display control of a final score based on a raw score obtained immediately before. [Figure 16] FIG. 13 is an explanatory diagram illustrating display control of a final score based on a raw score obtained immediately after the first score. [Figure 17] 11 is an explanatory diagram illustrating the relationship between an endoscopic image and calculation of a raw score that is not calculated in a time series manner. FIG. [Figure 18] 13 is an explanatory diagram illustrating display control of a final score when an uncalculated raw score is included. FIG. [Figure 19] FIG. 13 is an explanatory diagram illustrating display control of a final score indicating a discrimination result. [Figure 20] 13 is a flowchart showing a sequence of steps in a score display mode. [Figure 21] FIG. 1 is a block diagram showing a diagnosis support device. [Figure 22] FIG. 2 is a block diagram showing a medical service support device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] In Fig. 1, an endoscope system 10 includes an endoscope 12, a light source device 14, a processor device 16, a display 18, and a console 19. The endoscope 12 is optically connected to the light source device 14 and electrically connected to the processor device 16. The endoscope 12 includes an insertion section 12a to be inserted into the body of an observation subject, an operation section 12b provided at a base end portion of the insertion section 12a, and a bending section 12c and a tip section 12d provided at the tip side of the insertion section 12a. The bending section 12c is bent by operating an angle knob 12e (see Fig. 2) of the operation section 12b. The tip section 12d is directed in a desired direction by the bending operation of the bending section 12c.

[0030] 2, the operation unit 12b has, in addition to the angle knob 12e, a mode changeover switch 12g used for switching the observation mode, a zoom operation unit 12h for changing the imaging magnification, and a still image acquisition instruction unit 12f for issuing an instruction to acquire a still image. Note that the observation mode switching operation, zoom operation, or still image acquisition instruction may be performed by operation or instruction using the console 19, a foot switch (not shown), or the like, in addition to the mode changeover switch 12g or the scope switch of the still image acquisition instruction unit 12f.

[0031] The endoscope system 10 has three modes: a normal observation mode, a special observation mode, and a score display mode. In the normal observation mode, a normal image with natural coloring is displayed on the display 18 by illuminating the observation target with normal light and capturing an image. In the special observation mode, a special image in which a specific structure is emphasized is displayed on the display 18 by illuminating the observation target with special light having a different wavelength band from the normal light and capturing an image. In the score display mode, a score is determined for judging the severity or stage of a disease of the observation target based on an endoscopic image consisting of a normal image or a special image, and the determined score, changes in the score over time, and / or the judgment result, etc. are displayed. 18 Display in.

[0032] The severity of a disease refers to the degree of prognosis obtained by treatment. For example, when the disease is ulcerative colitis, it is classified into three categories: mild, moderate, and severe. When the disease is ulcerative colitis, the stage of the disease (disease stage) is classified into two categories: active and remission. Therefore, the result of the severity determination is either mild, moderate, or severe, and the result of the stage determination is either active or remission, or remission or non-remission. The score is a display that allows the degree of severity or stage of the disease to be observed to be recognized, and is a numerical value, a text, or the like. In this embodiment, a case where an image processing device determines remission or non-remission of ulcerative colitis will be described.

[0033] The processor device 16 to which the endoscope 12 is connected is an image processing device that executes the score display mode. The image processing device includes a processor. The image processing device has programs related to the image signal acquisition unit 51, the DSP 52, the noise reduction unit 53, the signal processing unit 55, the video signal generation unit 56, and the like built into the memory. The programs are executed by a control unit (not shown) configured by a processor, thereby realizing the functions of the image signal acquisition unit 51, the DSP 52, the noise reduction unit 53, the signal processing unit 55, the video signal generation unit 56, and the like. The score display mode may be executed by other configurations, for example, an image processing device function may be provided in an external image processing system other than the endoscope system 10, an endoscopic image may be input to the external image processing device to execute the score display mode, and the execution result may be displayed on an external display connected to the external image processing system.

[0034] The processor device 16 is electrically connected to a display 18 and a console 19. The display 18 outputs and displays the image of the observation target, the score, the change in the score over time, the judgment result, and / or information accompanying the image of the observation target. The console 19 functions as a user interface that accepts input operations such as function settings. An external recording unit (not shown) that records images and image information may be connected to the processor device 16.

[0035] In FIG. 3, the light source device 14 emits illumination light to irradiate an object to be observed. The light source device 14 includes a light source section 20 and a light source control section 21 that controls the light source section 20. The light source section 20 is configured, for example, of a semiconductor light source such as a multi-color LED (Light Emitting Diode), a combination of a laser diode and a phosphor, or a halogen light source such as a xenon lamp. The light source section 20 includes an LED or the like. Illumination The light source control unit 21 includes an optical filter for adjusting the wavelength band of the illuminated light. The light source control unit 21 controls the amount of illumination light by turning on / off each LED, etc., and adjusting the drive current and drive voltage of each LED, etc. The light source control unit 21 also controls the wavelength band of the illumination light by changing the optical filter, etc.

[0036] As shown in FIG. 4, in this embodiment, the light source unit 20 has four color LEDs: a V-LED (Violet Light Emitting Diode) 20a, a B-LED (Blue Light Emitting Diode) 20b, a G-LED (Green Light Emitting Diode) 20c, and an R-LED (Red Light Emitting Diode) 20d.

[0037] As shown in FIG. 5, the V-LED 20a generates purple light V with a central wavelength of 405±10 nm and a wavelength range of 380 to 420 nm. The B-LED 20b generates blue light B with a central wavelength of 460±10 nm and a wavelength range of 420 to 500 nm. The G-LED 20c generates green light G with a wavelength range of 480 to 600 nm. The R-LED 20d generates red light R with a central wavelength of 620 to 630 nm and a wavelength range of 600 to 650 nm. The purple light V is a short-wavelength light used to detect a superficial vascularized area, a mucosal bleeding area, or an extramucosal bleeding area used in the score display mode, and it is preferable that the central wavelength or peak wavelength includes 410 nm. Therefore, it is preferable that the endoscopic image used in the score display mode is an image of an observation target illuminated with the purple light V.

[0038] The light source control unit 21 controls the V-LED 20a, B-LED 20b, G-LED 20c, and R-LED 20d. In the normal observation mode, the light source control unit 21 controls each of the LEDs 20a to 20d to emit normal light such that the light intensity ratio between the purple light V, the blue light B, the green light G, and the red light R is Vc:Bc:Gc:Rc.

[0039] In the special observation mode or the score display mode, the light source control unit 21 controls each of the LEDs 20a to 20d to emit special light in which the light intensity ratio between the purple light V as a short wavelength light and the blue light B, the green light G, and the red light R is Vs:Bs:Gs:Rs. It is preferable that the special light emphasizes superficial blood vessels and the like. Therefore, it is preferable that the light intensity ratio Vs:Bs:Gs:Rs of the special light is set so that the light intensity of the purple light V is greater than the light intensity of the blue light B. For example, as shown in FIG. 6, the ratio of the light intensity Vs of the purple light V to the light intensity Bs of the blue light B is set to "4:1". Also, as shown in FIG. 7, the light intensity ratio between the purple light V, the blue light B, the green light G, and the red light R may be set to 1:0:0:0 for the special light, so that only the purple light V as a short wavelength light is emitted.

[0040] In this specification, the light intensity ratio includes the case where the ratio of at least one semiconductor light source is 0 (zero). Therefore, it includes the case where one or more of the semiconductor light sources are not lit. For example, the light intensity ratio is also considered to exist even when only one of the semiconductor light sources is lit and the other three are not lit, such as when the light intensity ratio between the purple light V, the blue light B, the green light G, and the red light R is 1:0:0:0.

[0041] Each LED20a~ 20d The light emitted by the endoscope 12 is incident on the light guide 41 via an optical path coupling section (not shown) constituted by a mirror, a lens, etc. The light guide 41 is built in the endoscope 12 and a universal cord (a cord that connects the endoscope 12 with the light source device 14 and the processor device 16). The light guide 41 propagates the light from the optical path coupling section to the tip 12d of the endoscope 12.

[0042] An illumination optical system 30a and an imaging optical system 30b are provided at the tip 12d of the endoscope 12. The illumination optical system 30a has an illumination lens 42, and illumination light propagated by a light guide 41 is irradiated onto an observation object via the illumination lens 42. The imaging optical system 30b has an objective lens 43, a zoom lens 44, and an imaging sensor 45. Various types of light, such as reflected light, scattered light, and fluorescent light, from the observation object are incident on the imaging sensor 45 via the objective lens 43 and the zoom lens 44. As a result, an image of the observation object is formed on the imaging sensor 45. The zoom lens 44 is freely movable between the telephoto end and the wide-angle end by operating the zoom operation unit 12h, and enlarges or reduces the observation object imaged on the imaging sensor 45.

[0043] The imaging sensor 45 is a color imaging sensor in which a color filter of R (red), G (green), or B (blue) is provided for each pixel, and captures an object to be observed and outputs image signals of each color of RGB. A charge coupled device (CCD) imaging sensor or a complementary metal oxide semiconductor (CMOS) imaging sensor can be used as the imaging sensor 45. A complementary imaging sensor equipped with complementary color filters of C (cyan), M (magenta), Y (yellow), and G (green) may be used instead of the imaging sensor 45 equipped with primary color filters. When a complementary imaging sensor is used, image signals of four colors of CMYG are output. Therefore, an RGB image signal similar to that of the imaging sensor 45 can be obtained by converting the four-color image signals of CMYG into three-color image signals of RGB by complementary-primary color conversion. A monochrome sensor without color filters may be used instead of the imaging sensor 45.

[0044] The imaging sensor 45 is driven and controlled by an imaging control unit (not shown). The control by the imaging control unit differs depending on the mode. In the normal observation mode or the score display mode, the imaging control unit controls the imaging sensor 45 to capture an image of an observation target illuminated with normal light. As a result, a Bc image signal is output from the B pixel of the imaging sensor 45, a Gc image signal is output from the G pixel, and an Rc image signal is output from the R pixel. In the special observation mode or the score display mode, the imaging control unit teeth, The imaging sensor 45 is controlled so as to capture an image of an object illuminated with special light, whereby Bs image signals are output from the B pixels of the imaging sensor 45, Gs image signals are output from the G pixels, and Rs image signals are output from the R pixels.

[0045] A CDS / AGC (Correlated Double Sampling / Automatic Gain Control) circuit 46 performs correlated double sampling (CDS) and automatic gain control (AGC) on the analog image signal obtained from the image sensor 45. The image signal passed through the CDS / AGC circuit 46 is converted into a digital image signal by an A / D (Analog / Digital) converter 48. The digital image signal after A / D conversion is input to the processor device 16.

[0046] The processor device 16 includes an image signal acquisition unit 51, a DSP (Digital Signal Processor) 52, a noise reduction unit 53, a memory 54, a signal processing unit 55, and a video signal generation unit 56. The signal processing unit 55 includes a normal image generation unit 61, a special image generation unit 62, and a score processing unit 63.

[0047] The image signal acquisition unit 51 acquires a digital image signal of an endoscopic image input from the endoscope 12. The acquired image signal is transmitted to the DSP 52. The DSP 52 performs various signal processing such as defect correction processing, offset processing, gain correction processing, linear matrix processing, gamma conversion processing, demosaic processing, and YC conversion processing on the received image signal. In the defect correction processing, signals of defective pixels of the imaging sensor 45 are corrected. In the offset processing, dark current components are removed from the image signal that has been subjected to the defect correction processing, and an accurate zero level is set. In the gain correction processing, the signal level of each image signal is adjusted by multiplying the image signal of each color after the offset processing by a specific gain.

[0048] The image signals of each color after the gain correction process are subjected to linear matrix processing to improve color reproducibility. Then, the brightness and saturation of each image signal are adjusted by gamma conversion. The image signals after the gamma conversion process are subjected to demosaic processing (also called isotropic processing or synchronization processing), and signals of the missing colors of each pixel are generated by interpolation. The demosaic processing ensures that all pixels have signals of each of the RGB colors. The DSP 52 performs YC conversion processing on each image signal after the demosaic processing, and outputs a luminance signal Y and color difference signals Cb and chrominance signals Cr to the noise reduction unit 53.

[0049] The noise reduction unit 53 performs noise reduction processing, for example, using a moving average method or a median filter method, on the image signal that has been subjected to demosaic processing and the like by the DSP 52. The image signal with the noise reduced is stored in a memory .

[0050] The signal processing unit 55 acquires the noise-reduced image signal from the memory 54. Then, the acquired image signal is subjected to signal processing such as color conversion processing, color enhancement processing, and structure enhancement processing as necessary to generate a color endoscopic image showing the observation target. The color conversion processing is a processing for converting the color of the image signal by 3×3 matrix processing, gradation conversion processing, and 3D LUT (look-up table) processing. The color enhancement processing is performed on the image signal that has already been color converted. The structure enhancement processing is a processing for enhancing a specific tissue or structure contained in the observation target, such as a blood vessel or a pit pattern, and is performed on the image signal after the color enhancement processing.

[0051] The signal processing unit 55 includes a normal image generating unit 61, a special image generating unit 62, and a score processing unit 63. Depending on the mode set, the signal processing unit 55 sets the destination of the image signal from the noise reduction unit 53 to either the normal image generating unit 61, the special image generating unit 62, or the score processing unit 63. Specifically, for example, when the normal observation mode is set, the image signal is input to the normal image generating unit 61. When the special observation mode is set, the image signal is input to the special image generating unit 62. When the score display mode is set, the image signal is input to the score processing unit 63.

[0052] The normal image generating unit 61 performs normal image processing on the input Rc image signal, Gc image signal, and Bc image signal for one frame. The normal image processing includes color conversion processing such as 3×3 matrix processing, tone conversion processing, and 3D LUT (Look Up Table) processing, color enhancement processing, and structure enhancement processing such as spatial frequency enhancement. The Rc image signal, Gc image signal, and Bc image signal that have been subjected to the normal image processing are input to the video signal generating unit 56 as a normal image.

[0053] The special image generating unit 62 performs special image image processing on the input Rs image signal, Gs image signal, and Bs image signal for one frame. The special image processing includes 3×3 matrix processing, tone conversion processing, color conversion processing such as 3D LUT processing, color enhancement processing, and structure enhancement processing such as spatial frequency enhancement. The Rs image signal, Gs image signal, and Bs image signal that have been subjected to the special image image processing are input to the video signal generating unit 56 as a special image.

[0054] The endoscopic image generated by the signal processing unit 55 is a normal observation image when the observation mode is the normal observation mode, and a special observation image when the observation mode is the special observation mode, so that the contents of the color conversion process, color enhancement process, and structure enhancement process differ depending on the observation mode. In the normal observation mode, the signal processing unit 55 generates a normal observation image by performing the above-mentioned various signal processes that make the observation target have a natural color. In the special observation mode, the signal processing unit 55 generates a special observation image by performing the above-mentioned various signal processes that at least emphasize the blood vessels of the observation target. In the special observation image generated by the signal processing unit 55, blood vessels (so-called superficial blood vessels) or blood that are relatively shallow in the observation target based on the surface of the mucosa are colored in magenta (e.g., brown), and blood vessels (so-called deep blood vessels) that are relatively deep in the observation target based on the surface of the mucosa are colored in cyan (e.g., green). Therefore, the blood vessels or bleeding (blood) of the observation target are emphasized by the difference in color compared to the mucosa that is displayed in pink.

[0055] The video signal generating unit 56 converts the normal image, special image, or final score determined by the score processing unit 63, etc., output from the signal processing unit 55, into a video signal that enables full-color display on the display 18. The converted video signal is input to the display 18. As a result, the normal image, special image, final score, etc. are displayed on the display 18. In addition, when a still image acquisition instruction (freeze instruction or release instruction) is input by operating the still image acquisition instruction unit 12f, the signal processing unit 55 stores the generated endoscopic image in the image storage unit 75 (see FIG. 8) or storage (not shown). The storage is connected to the processor device 16 via a LAN (Local Area Network) or the like. Via The connected external storage device is, for example, a file server of a system for filing endoscopic images such as a PACS (Picture Archiving and Communication System, see FIG. 21), or a NAS (Network Attached Storage).

[0056] The score processing unit 63 determines a final score and performs control to display the final score and / or a change in the final score over time in real time on the display 18. As shown in Fig. 8, the score processing unit 63 includes an image acquisition unit 71, a raw score calculation unit 72, a final score determination unit 73, and a display control unit 76. It may further include an inappropriate image discrimination unit 74, an image storage unit 75, and a part determination unit 77.

[0057] Score display In the mode, the image acquisition unit 71 automatically acquires a plurality of endoscopic images obtained by capturing images of an observation target at different times using the endoscope 12. There are normal observation images and special observation images as endoscopic images, and in this embodiment, the image acquisition unit 71 acquires special observation images in which blood vessels and the like are emphasized. The image acquisition unit 71 may acquire endoscopic images from storage in some cases. The endoscopic images acquired by the image acquisition unit 71 are sent to the raw score calculation unit 72, the inappropriate image discrimination unit 74, or the image storage unit 75.

[0058] The raw score calculation unit 72 calculates a raw score related to the determination of the severity or stage of the disease of the observation target based on the multiple endoscopic images acquired by the image acquisition unit 71. The raw score related to the determination of the severity or stage of the disease of the observation target is a numerical value that enables recognition of the degree of severity or stage of the disease of the observation target contained in the endoscopic image. The raw score calculation unit 72 can include one or both of a first calculation unit 81 (see FIG. 8) and a second calculation unit 82 (see FIG. 8).

[0059] It is preferable that the raw score calculation unit 72 calculates the raw score based on a feature amount (first feature amount) obtained by analyzing an endoscopic image in the first calculation unit 81. Examples of the feature amount include feature amounts related to blood vessels such as the number, thickness, length, number of branches, branching angle, distance between branching points, number of intersections, inclination, density, color, blood concentration, oxygen saturation, presence or absence of bleeding, area or flow rate of bleeding, or feature amounts related to the color of mucous membranes.

[0060] For example, as a feature quantity of a raw score related to the severity or stage of ulcerative colitis, quantities related to a superficial vascularized area, a mucosal bleeding area, and an extramucosal bleeding area included in an endoscopic image are preferably mentioned. analysis By this, the blood vessels to be observed are extracted from the endoscopic image using a frequency filter or the like, and the number of pixels in the portion where the blood vessels are densely concentrated in the endoscopic image is counted to obtain the raw score. Similarly, for intramucosal or extramucosal bleeding, the intramucosal or extramucosal bleeding to be observed is extracted from the endoscopic image using a G value relative to an R value proportional to the amount of hemoglobin, and the number of pixels in the intramucosal or extramucosal bleeding portion in the endoscopic image is counted to obtain the raw score.

[0061] The following method can also be used to count the number of pixels in the dense superficial blood vessels, intramucosal bleeding, and extramucosal bleeding included in the endoscopic image. First, the present inventor has found that the pattern of the vascular structure of ulcerative colitis, which is a disease judged in this embodiment, changes every time the severity worsens, as shown in Figs. 9(A) to (E). When ulcerative colitis is in remission or does not occur, the pattern of the superficial blood vessels 85 is regular (Fig. 9(A)), or the regularity of the pattern of the superficial blood vessels 85 is only slightly disturbed (Fig. 9(B)). On the other hand, when ulcerative colitis is not in remission and the severity is mild, the density of the superficial blood vessels 85 is sparse (Fig. 9(C)). When ulcerative colitis is not in remission and the severity is moderate, intramucosal bleeding 86 occurs (Fig. 9(D)). Furthermore, when ulcerative colitis is not in remission and the severity is moderate to severe, extramucosal bleeding 87 occurs (FIG. 9(E)). The raw score calculation unit 72 can calculate the raw score by utilizing the above-mentioned pattern changes of the blood vessel structure.

[0062] Here, the term "area densely packed with superficial blood vessels" refers to a state in which superficial blood vessels meander and gather, and in terms of appearance on an image, refers to an area in which many superficial blood vessels surround an intestinal crypt (see Figure 10). Intramucosal bleeding refers to bleeding within the mucosa (see Figure 10) that must be differentiated from bleeding into the lumen. Intramucosal bleeding refers to bleeding that is within the mucosa and not within the lumen (lumen, fold holes) in terms of appearance on an image. Extramucosal bleeding refers to a small amount of blood into the lumen, blood that is visible oozing out of the lumen or mucosa in front of the endoscope even after the lumen has been washed, or blood in the lumen accompanied by oozing on the bleeding mucosa.

[0063] The raw score calculation unit 72 classifies the superficial vascular dense area, the mucosal bleeding area, or the extramucosal bleeding area according to the frequency characteristics and brightness values ​​obtained from the special observation image. Specifically, the superficial vascular dense area, the mucosal bleeding area, and the extramucosal bleeding area are classified as shown in FIG. 11. The superficial vascular dense area is represented by a low brightness value and a high frequency characteristic. The intramucosal bleeding area is represented by a medium brightness value and a medium frequency frequency characteristic. The extramucosal bleeding area is represented by a low brightness value and a low frequency frequency characteristic. When the various structures of the special observation image are represented by the brightness value and the frequency characteristic, in addition to the above three superficial vascular dense area, the mucosal bleeding area, and the extramucosal bleeding area, the blurred dark area of ​​the special observation image or the endoscopic shadow (the shadow that appears in the center of the endoscopic image when the tip portion 12d of the endoscope is moved along the lumen) are also included. Using the above classification, superficial vascular congestion, intramucosal bleeding, and extramucosal bleeding, which are necessary for determining whether ulcerative colitis is in remission or not, are extracted from the superficial vascular congestion areas, intramucosal bleeding areas, and extramucosal bleeding areas.

[0064] For spatial frequency, a Laplacian filter is applied to the special observation image to calculate the spatial frequency component distribution. Based on the spatial frequency component distribution, for example, if the standard deviation of the frequencies of 9 pixels in the vicinity of the specific pixel is equal to or less than a certain value, the specific pixel is determined to belong to the low frequency region. The high frequency region is extracted by Hessian analysis of the spatial frequency component distribution. The mid-frequency region is determined to be the portion of the special observation image excluding the low frequency region and the high frequency region. In this way, by classifying the pixels of the special observation image according to the spatial frequency and the brightness value, the number of pixels in the superficial vascular dense area, the intramucosal bleeding area, or the extramucosal bleeding area can be calculated. As described above, the above configuration allows the severity or stage of the disease to be well determined, so that the disease is preferably ulcerative colitis.

[0065] The raw scores calculated based on different first feature amounts are regarded as different types of raw scores. The raw score calculation unit 72 may calculate one type of raw score or two or more types of raw scores.

[0066] The raw score calculation unit 72 calculates the raw score based on the endoscopic image in the second calculation unit 82. The second calculation unit 82 includes a trained first machine learning model. The trained first machine learning model is generated by inputting past endoscopic images associated with raw scores into the machine learning model. That is, the machine learning model is generated by learning so that the associated raw scores are correctly output by inputting past endoscopic images into the machine learning model. Therefore, the term "trained" includes adjustment of various parameters in addition to inputting past endoscopic images associated with scores into the machine learning model. The raw scores associated in the past endoscopic images may be, for example, feature amounts or may be numerical values ​​of the severity or stage of the disease to be observed. For example, the first machine learning model may include two or more machine learning models corresponding to the feature amounts, and may calculate two or more raw scores.

[0067] The final score determination unit 73 determines the final score based on the raw score. The raw score calculated by the raw score calculation unit 72 is a quantity related to the determination of the severity or stage of a disease, and is an index of the degree of symptoms of the disease or the degree of progression of the disease. Therefore, the method of determining the final score may be adjusted according to the purpose of the determination. For example, when determining the level of severity or the progression of the stage, the final score is set in a direction in which the raw score value is large, the severity is high, and the disease stage is advanced, and the severity or stage is determined for the part where the disease is most deteriorated. In this case, the method of determining the final score may include, for example, a method of setting a threshold value in advance for each of the three types of raw scores, the number of pixels in the superficial vascular dense area, the number of pixels in the intramucosal bleeding area, and the number of pixels in the extramucosal bleeding area, and determining the raw score that exceeds the threshold value as the final score, or a method of determining the raw score that exceeds the preset threshold value the widest in the higher direction among the values ​​of each of the three types of raw scores as the final score. On the other hand, for example, in the case of screening, the threshold value is set low.

[0068] The display control unit 76 controls the display of the final score and / or the change in the final score over time on the display 18 in real time. The final score may be displayed, for example, as a numerical value of the final score on the display 18 in real time, as a graph in which the vertical axis represents the final score value and the horizontal axis represents the time course of the final score, or as a message notifying the result of the assessment of the severity or stage of the disease based on the final score value and a preset threshold value. The display control unit 76 also controls not to display the final score, etc. For example, if the final score is smaller than a preset threshold value, the final score is not displayed. Note that displaying in real time means displaying immediately, not displaying at the exact same time.

[0069] When the position of the photographing subject changes and the observation subject changes, the obtained endoscopic image also changes, and the raw score calculated based on the endoscopic image also changes. However, since the score processing unit 63 configured as described above uses the raw score to determine the final score and then displays it on the display 18, the change in the raw score is not displayed as it is on the display 18, and by using the raw score as the final score, a more stable score is displayed on the display 18 even in real time. In addition, depending on the value of the final score, the final score is controlled not to be displayed, so that when the score is displayed, it is displayed as a more stable score. In addition, by displaying the change in the final score over time in real time, when the photographing subject position changes and the observation subject changes in one observation, the change in the severity or stage due to the photographing subject position, or the height of the final score at the position with the highest severity or the most advanced stage, can be grasped at a glance. Therefore, the user can grasp the location where the severity or stage of the disease was the worst during the observation process while concentrating on the observation. As described above, the image processing device calculates a more stable and robust score and displays the final score, etc., without the final score fluctuating even if the observation subject changes minutely. Therefore, the image processing device contributes to preventing the user performing the observation from overlooking a lesion or to making the observation easier.

[0070] The score processing unit 63 may include an inappropriate image discrimination unit 74. The inappropriate image discrimination unit 74 discriminates whether an endoscopic image is appropriate or inappropriate for calculating a raw score. The endoscopic image is determined to be either appropriate or inappropriate by the discrimination. An endoscopic image may be inappropriate as an image for calculating a raw score. For example, an endoscopic image may be blurred because it was captured while the endoscope tip 12d was moving, may be out of focus because of water droplets, may have many blurred parts because the observation target was oblique, or may only be captured in the distance and hardly includes the observation target, and an extreme value may be calculated when the raw score is calculated. Such an endoscopic image that may result in an inappropriate score is determined to be inappropriate for calculating a raw score. The score processing unit 63 may include one or both of a first discrimination unit 83 (see FIG. 8) and a second discrimination unit 84 (see FIG. 8).

[0071] The first discrimination unit 83 discriminates whether the endoscopic image is suitable for calculating the raw score based on a feature amount (second feature amount) obtained by analyzing the endoscopic image. In this case, the feature amount is preferably an amount related to at least one selected from the halation distribution, spatial frequency distribution, luminance value distribution, shadow distribution, magnification index, and reflected light distribution of the illumination light irradiated to the observation target of the endoscopic image.

[0072] In an endoscopic image, an area where halation occurs or an area where a shadow is cast by an endoscope hood or the like is an extremely bright or dark area, and therefore an endoscopic image with many of these areas is unsuitable for raw score calculation. An endoscopic image with many extremely bright or dark areas due to a luminance value distribution or a shadow distribution is also unsuitable for raw score calculation. An endoscopic image with many blurred areas due to a spatial frequency distribution is an image that is blurred or out of focus, and therefore is unsuitable for raw score calculation. In addition, when the magnification rate index is referenced and the magnification rate is large, for example, when a superficial vascular dense portion is calculated, the way blood vessels are captured changes compared to when there is no magnification, and the blood vessel density per unit area also changes. Therefore, it is good to calculate the raw score taking the magnification rate into consideration, but if not, it is unsuitable for raw score calculation. In addition, when the observation target changes to a different type due to a change in the observation site, the reflected light distribution of the illumination light irradiated to the observation target may change. An endoscopic image with many extremely bright or dark areas due to a change in the reflected light distribution is also unsuitable for raw score calculation.

[0073] Moreover, values ​​calculated based on different second feature amounts are different types of second feature amounts. The first discrimination unit 83 may calculate one type of second feature amount or two or more types of second feature amounts. The first discrimination unit 83 uses one type or two or more types of second feature amounts to discriminate whether the endoscopic image is appropriate or inappropriate for calculating the raw score.

[0074] The inappropriate image discrimination unit 74 discriminates whether an endoscopic image is appropriate or inappropriate for calculation of a raw score in the second discrimination unit 84 based on the endoscopic image. The second discrimination unit 84 includes a trained second machine learning model. The trained second machine learning model is generated by inputting past endoscopic images, which are associated with appropriateness or inappropriateness for calculation of a raw score, into the machine learning model. That is, the past endoscopic images are machine By inputting the data into the learning model, the system was trained to correctly output whether the data was appropriate or inappropriate for calculating the raw score. machineTherefore, the term "trained" includes the input of past endoscopic images, which are associated with whether they are appropriate or inappropriate for the calculation of the raw score, into the machine learning model, as well as the adjustment of various parameters.

[0075] An endoscopic image that is determined to be appropriate for raw score calculation by the inappropriate image discrimination unit 74 is sent to the raw score calculation unit 72, where a raw score is calculated. An endoscopic image that is determined to be inappropriate for raw score calculation is given a raw score of "not calculated." In other words, a label of "not calculated" is attached as the raw score, and the image is distinguished from an endoscopic image for which a raw score has not yet been calculated.

[0076] It is preferable that the final score determination unit 73 determines the final score from a raw score calculated based on a plurality of endoscopic images acquired during a predetermined period prior to the time of final score determination. The determination of the final score in this case will be specifically described with reference to FIG. 12. In FIG. 12, the flow from the start of endoscopic observation is shown in the upper part, and the state of the obtained endoscopic image is shown in the lower part. Endoscopic observation is started, and endoscopic image acquisition 101 starts at the observation start time S. The image signal acquisition unit 51 automatically performs endoscopic image acquisition 101 at a predetermined image acquisition interval a. As soon as endoscopic image acquisition 101 is performed, the raw score of the endoscopic image is calculated. Note that inappropriate image discrimination may be performed prior to the calculation. In FIG. 12, endoscopic image acquisition 101 is shown as a filled circle, but only some of them are marked with a symbol to avoid the figure becoming too complicated. Also, Image Acquisition Interval Regarding a, to avoid complicating the drawing, only some of the a are labeled. The final score is determined based on the endoscopic images acquired during a period Δt prior to the final score determination time t, which is the time when the final score is determined. The period Δt is a predetermined period and is set in advance.

[0077] The first endoscopic image acquired in the period Δt is an endoscopic image 121 acquired at time t-Δt. The endoscopic image 121 included a mucosal bleeding area 126 and a dense blood vessel area 127. In FIG. 12, the mucosal bleeding area 126 and the dense blood vessel area 127 are shown as hatched and shaded areas, respectively, but only some of them are labeled to avoid complicating the drawing. The inappropriate image discrimination unit 74 discriminates whether the endoscopic image 121 obtained by the endoscopic image acquisition 101 at time t-Δt is appropriate or inappropriate for calculating the raw score using period B, and obtains a discrimination result at the time of inappropriate image discrimination 102. The discrimination result was "appropriate."

[0078] After the inappropriate image discrimination 102, the raw score calculation unit 72 calculated the raw score for the endoscopic image 121 using the period C, and obtained the calculation result at the time of raw score calculation 103. The raw score calculation unit calculated raw scores for two types of pixels, the number of intramucosal and extramucosal bleeding areas (hereinafter referred to as bleeding area pixel number) and the number of pixels in dense vascular areas (hereinafter referred to as dense area pixel number), using the first calculation unit 81. The calculation results were "bleeding area pixel number: 100" and "dense area pixel number: 70".

[0079] Next, among the endoscopic images acquired during the period Δt, the second image for which the raw score is calculated is the endoscopic image 122 acquired at time t-Δt+4a. The endoscopic image 122 included an intramucosal bleeding area 126, a densely vascular area 127, and an extramucosal bleeding area 128. In FIG. 12, the extramucosal bleeding area 128 is shown as a filled circle, but only a part of it is labeled to avoid complicating the drawing. The inappropriate image discrimination unit 74 discriminates whether the endoscopic image 122 obtained by the endoscopic image acquisition 101 at the time t-Δt+4a is appropriate or inappropriate for calculating the raw score using the period B, and obtains a discrimination result at the time of the inappropriate image discrimination 104. The discrimination result was "appropriate."

[0080] After the inappropriate image discrimination 104, the raw score calculation unit 72 calculated the raw score for the endoscopic image 122 using the period C, and obtained the calculation result at the time of raw score calculation 105. The raw score calculation unit calculated raw scores for two types of the bleeding portion pixel count and the dense portion pixel count by the first calculation unit 81. The calculation results were "bleeding portion pixel count: 120" and "dense portion pixel count: 90".

[0081] Next, of the endoscopic images acquired during period Δt, the third image for which a raw score is calculated is endoscopic image 123 acquired at time t-Δt+8a. Since endoscopic image 123 was acquired while the endoscope was moving, the observation target was a blurred image 129, and the observation target could not be identified. For endoscopic image 123 acquired by endoscopic image acquisition 101 at time t-Δt+8a, inappropriate image discrimination unit 74 discriminates whether the image is appropriate or inappropriate for calculating a raw score using period B, and obtains a discrimination result at the time of inappropriate image discrimination 106. The discrimination result is "inappropriate."

[0082] After the inappropriate image discrimination 106, the raw score calculation unit 72 does not calculate a raw score because the result of the inappropriate image discrimination is "inappropriate." That is, the calculation result of the raw score is "not calculated."

[0083] Next, of the endoscopic images acquired during the period Δt, the fourth image for which a raw score is calculated is endoscopic image 124 acquired at time t-Δt+12a. Endoscopic image 124 included intramucosal bleeding area 126, dense blood vessel area 127, and extramucosal bleeding area 128. For endoscopic image 124 acquired by endoscopic image acquisition 101 at time t-Δt+12a, inappropriate image discrimination unit 74 discriminates whether it is appropriate or inappropriate for calculating a raw score using period B, and obtains a discrimination result at the time of inappropriate image discrimination 108. The discrimination result was "appropriate."

[0084] After the inappropriate image discrimination 108, the raw score calculation unit 72 calculated the raw score for the endoscopic image 124 using the period C, and obtained the calculation result at the time of raw score calculation 109. The raw score calculation unit calculated raw scores for two types of numbers of pixels of bleeding area and dense area by the first calculation unit 81. The calculation results were "number of pixels of bleeding area: 140" and "number of pixels of dense area: 140".

[0085] In this way, the endoscopic images acquired during the period Δt are four, endoscopic images 121, 122, 123, and 124, in relation to the image acquisition interval a, the period for determining inappropriate images by the inappropriate image determining unit 74, and the period for calculating raw scores by the raw score calculating unit 72. It is preferable that the final score determining unit 73 determines the final score based on the raw scores other than "not calculated" among these endoscopic images. Therefore, the final score determining unit 73 determines the final score from the raw scores calculated based on the three endoscopic images 121, 122, and 124.

[0086] The final score determination unit 73 preferably determines the final score by performing a moving average, FIR (finite impulse response) filter processing, or IIR (infinite impulse response) filter processing on the multiple raw scores. The moving average is preferably any one of a simple moving average, a weighted moving average, an exponential moving average, and a triangular moving average. When weighting is used in the moving average, a value determined to obtain a preferable result depending on the observed site or the difference in the conditions for acquiring an endoscopic image can be set in advance and used. By using the moving average, FIR filter processing, or IIR filter processing, the final score can be determined by reducing the influence of noise, such as raw scores with extremely different values, in the multiple raw scores. Therefore, the final score is determined stably.

[0087] At the final score determination time t, for example, when the final score is calculated by a simple moving average of the raw scores, if the raw score is the number of bleeding pixels, the number of bleeding pixels in the endoscopic images 121, 122, and 124 is averaged. That is, the first type of final score is calculated by the following formula (1): Dense area pixel count: 70 And, endoscopic image 122 Dense area pixel count 90 The number of pixels of the bleeding area in endoscopic image 121 (100), the number of pixels of the bleeding area in endoscopic image 122 (120), and the number of pixels of the bleeding area in endoscopic image 124 (140) are averaged to arrive at 120. Similarly, the second type of final score is calculated by averaging the number of pixels of the bleeding area in endoscopic image 121 (100), the number of pixels of the bleeding area in endoscopic image 122 (120), and the number of pixels of the bleeding area in endoscopic image 124 (140) according to the following formula (2), to arrive at 100.

[0088] Final score (pixel count of bleeding area) = (100 + 120 + 140) / 3 = 120 (1) Final score (number of pixels in dense areas) = ​​(70 + 90 + 140) / 3 = 100 (2)

[0089] The interval for determining the final score may be automatic, or the final score may be determined by an instruction. For example, the instruction may be a time when the user acquires a still image by the still image acquisition instruction unit 12f (freeze button), which is set as the final score determination time t. Therefore, in this case, the display control unit 76 performs control to display the final score and / or the change over time of the final score when an instruction to acquire a still image is given by the still image acquisition instruction unit 12f. The observation target for which the user saves a still image often includes a notable area. It is preferable to determine the final score for such an area and display it on the display 18, since this contributes to a more appropriate diagnosis for the user.

[0090] Next, the display control unit 76 performs control to display the final score and / or the change in the final score over time on the display 18 in real time. A method of display control that can stably display the final score determination result is preferable. Therefore, the display control unit 76 may, for example, display the largest final score from the start of observation, and update the display of the final score when a final score larger than the previous number is obtained, or display the change in the final score over time.

[0091] When displaying the change in the final score over time, it is preferable to display it as a graph. The graph is preferably displayed as at least one graph showing the relationship between the final score and the time when the final score was determined. When displaying as a graph, it is preferable to set a threshold value in advance for the final score value, and display the final score on the graph only when the final score is equal to or greater than the threshold value.

[0092] As shown in FIG. 13, the display control unit 76 controls whether or not to display the final score based on the threshold value at the final score determination time t described above. When there are two types of final scores, two types of graphs are displayed on the display 18. In the final score, the threshold value T1 of the bleeding portion pixel count is set to 800, and the threshold value T2 of the dense portion pixel count is set to 40. The final score of the bleeding portion pixel count at the final score determination time t was 120, which is smaller than the threshold value T1 of 800. Therefore, the bleeding portion pixel count at the final score determination time t is not plotted in the graph 131 of the bleeding portion pixel count, which is the final score. In addition, the final score of the dense portion pixel count at the final score determination time t was 100, which is larger than the threshold value T2 of 40. Therefore, the dense portion pixel count at the final score determination time t is plotted with an auxiliary line 133 added so that the numerical value can be grasped at a glance in the graph 132 of the dense portion pixel count, which is the final score. In graphs 131 and 132 in FIG. 13, threshold value T1 or threshold value T2 is indicated by diagonal lines.

[0093] The display control unit 76 controls the display of the final score and / or the change in the final score over time on the display 18 in real time, so that the user can see at a glance which item is thought to be the most deteriorating in the disease even while continuing to observe the subject. In addition, by showing the change over time in a graph, it is possible to know approximately where the most deteriorating part is located. In addition, by controlling the display of the final score using a threshold value, only the item where the disease is thought to be the most deteriorating is displayed, so that the display of the final score can be made more stable.

[0094] In addition, as for controlling the display of the final score based on the threshold value, when there are two or more types of final scores, the number of graphs displayed on the display 18 may be limited to one, etc. Specifically, for example, when one of two types of final scores is equal to or greater than the threshold value and the other is less than the threshold value, only the graph of the final score equal to or greater than the threshold value is displayed. By carefully selecting and displaying the types of parameters such as the final score, the reliability or stability of the displayed final score is improved. In addition, the user can grasp the information for diagnosing the severity or stage at a glance.

[0095] In addition, a site determination unit 77 (see FIG. 8) may be provided that performs image analysis on the endoscopic image to determine the site of the observation target contained in the endoscopic image, and the change over time of the final score may be displayed as a graph showing the relationship between the final score, the final score determination time, and the site. In this way, the endoscopic image is automatically image analyzed, and the site name is displayed in response to the display of the change over time of the final score. As shown in FIG. 14, for example, in observing the large intestine, when screening is performed while pulling the endoscope from the cecum located deep inside the large intestine, site names 134 such as "cecum, ascending colon, transverse colon, descending colon, sigmoid colon, rectum" are displayed. By displaying the site names 134 in this way, it is possible to grasp all the sites where the disease is worsening during observation, which is preferable.

[0096] The final score determination unit 73 may determine the final score based on a raw score calculated immediately before or immediately after the time point at which the final score is determined. The determination of the final score in these cases will be specifically described with reference to Figs. 15 and 16. As shown in Fig. 15, in a method of determining the final score based on a raw score calculated immediately before the time point at which the final score is determined, the final score is determined based on the raw score calculated immediately before the time point at which the final score is determined, instead of using a plurality of raw scores. In other words, the raw score calculated immediately before the time point at which the final score is determined is determined as the final score. The endoscopic image acquisition and raw score acquisition are similar to those described above.

[0097] The raw score calculated immediately before the final score determination time t is "bleeding area pixel count: 140" and "dense area pixel count: 140" calculated at the time of raw score calculation 109 based on the endoscopic image acquired by endoscopic image acquisition at time t-Δt+12a, so the final score is "bleeding area pixel count: 140, dense area pixel count: 140." The display of the final score using the threshold value by the display control unit 76 is the same as described above. Therefore, as shown in FIG. 15, a graph 132 of the dense area pixel count in which the dense area pixel count is plotted at 140 as the final score is displayed on the display 18.

[0098] Also, as shown in Fig. 16, in a method of determining a final score based on a raw score calculated immediately after the time of final score determination, instead of using multiple raw scores, the final score is determined based on the raw score calculated immediately after the final score determination time t. That is, the raw score calculated immediately after the final score determination time t is determined as the final score. This is the raw score calculated for the endoscopic image acquired at the final score determination time t. The endoscopic image acquisition and raw score acquisition are similar to those described above.

[0099] The raw score calculated immediately after the final score determination time t is "bleeding area pixel count: 160" and "dense area pixel count: 70" calculated at the time of raw score calculation 111 based on the endoscopic image acquired by endoscopic image acquisition at time t, so the final score is "bleeding area pixel count: 160, dense area pixel count: 70." The display of the final score using the threshold value by the display control unit 76 is the same as described above. Therefore, as shown in FIG. 16, a graph 132 of the dense area pixel count in which the dense area pixel count is plotted at 70 is displayed on the display 18 as the final score.

[0100] The method of determining the final score based on the raw score calculated immediately before or after the time of determining the final score is suitable for cases where one wishes to know the final score of the currently observed object, and is also suitable because it allows the most recent final score to be stably displayed.

[0101] In addition, when the final score is determined at the time when the user acquires a still image by the still image acquisition instruction unit 12f, by combining with a method of determining the final score based on the raw score calculated immediately before or after the time of determining the final score, the final score based on the observation site where the user instructed to acquire a still image, which is the area that the user wants to focus on, can be displayed quickly and stably. Therefore, the final score is displayed at the time when the user wants to obtain information about the diagnosis, which is preferable because it meets the needs of the user.

[0102] The final score determination unit 73 may determine whether to calculate or not calculate the final score based on the number of raw scores that were not calculated by the inappropriate image discrimination unit 74 and the number of calculated raw scores among the raw scores based on multiple endoscopic images acquired during a predetermined period. Specifically, the final score is set to "not calculated" when the number of raw scores other than not calculated among the raw scores based on multiple endoscopic images acquired during a predetermined period is a predetermined number or less, when the number of not calculated raw scores is a predetermined number or more, or when the ratio of the number of not calculated raw scores to the number of raw scores based on multiple endoscopic images acquired during a predetermined period is a predetermined value or more, etc.

[0103] The determination of the final score in this case will be specifically described with reference to Figs. 17 and 18. As shown in Fig. 17, during a period Δt, which is a predetermined period, endoscopic image acquisition 101 is automatically performed at an image acquisition interval a, and endoscopic images are acquired at time t-Δt, time t-Δt+4a, time t-Δt+8a, and time t-Δt+12a. As soon as the endoscopic images are acquired, the inappropriate image discrimination unit 74 discriminates whether the endoscopic images acquired at each time after the period B are appropriate or inappropriate for the calculation of the raw score. Since the endoscopic images 122 and 123 acquired at the times t-Δt+4a and t-Δt+8a among the multiple endoscopic images were blurred images 129, the raw score calculation unit 72 determined the raw scores of the endoscopic images 122 and 123 to be "not calculated", respectively.

[0104] In this embodiment, when the number of raw scores other than the uncalculated ones among the raw scores based on the multiple endoscopic images acquired in a predetermined period is two or less, the final score is set to "uncalculated". Therefore, in this case, since the number of multiple endoscopic images is four, the number of uncalculated raw scores is two, and the number of raw scores other than the uncalculated ones is two, this corresponds to "the case where the number of raw scores other than the uncalculated ones is two or less". Therefore, the final score at the final score determination time t is set to "uncalculated". The display control unit 76 performs control so as not to display the uncalculated final scores on the display. Therefore, as shown in FIG. 18, the final scores are not plotted on either the graph 131 or the graph 132 on the display 18.

[0105] 17, when a method is adopted in which the final score is set to "not calculated" when the number of uncalculated raw scores is equal to or greater than a predetermined number, for example, if the predetermined number is set to 2, the number of uncalculated raw scores is 2, and the final score is set to "not calculated." Similarly, when a method is adopted in which the final score is set to "not calculated" when the ratio of the number of uncalculated raw scores to the number of raw scores based on a plurality of endoscopic images acquired in a predetermined period is equal to or greater than a predetermined value, for example, if the predetermined value is set to 0.5, the number of endoscopic images and raw scores acquired in the period Δt is 4, of which the number of "not calculated" raw scores is 2, and the above ratio is 2 / 4 (0.5), and the final score is set to "not calculated."

[0106] As described above, by determining whether to calculate or not calculate the final score based on the number of raw scores that were not calculated by the inappropriate image discrimination unit 74 and the number of raw scores that were calculated, the number or proportion of undesirable raw scores is reduced, the final score is appropriately determined, and the score display is stable, which is preferable.

[0107] The final score may be displayed by a message notifying the result of the assessment of the severity or stage of the disease being observed. The severity or stage is assessed by the final score. For example, the display control unit 76 uses a threshold value for controlling whether or not to display the final score. That is, in the final score, the severity or stage is assessed by a threshold value T1 of the number of pixels in the bleeding area and a threshold value T2 of the number of pixels in the dense area. Regarding the severity, when the number of pixels in the bleeding area is equal to or greater than the threshold value T1, it is assessed as severe; when the number of pixels in the bleeding area is less than the threshold value T1 and the number of pixels in the dense area is equal to or greater than the threshold value T2, it is assessed as moderate; or when the number of pixels in the bleeding area is less than the threshold value T1 and the number of pixels in the dense area is equal to or greater than the threshold value T2, it is assessed as moderate. T1 For the stage, the number of pixels in the hemorrhage area is less than the threshold T1 and the number of pixels in the dense area is less than the threshold T2, which is considered mild. For the stage, the number of pixels in the hemorrhage area is less than the threshold T1 and the number of pixels in the dense area is more than the threshold T2, which is considered moderate. T1 When the mean tumor volume is less than the threshold T2 and the number of pixels in the dense area is less than the threshold T2, the condition is considered to be mild.

[0108] As shown in Fig. 19, the message notifying the result of the severity or stage assessment is, for example, displayed as a message 135 on a part of the display 18. In addition to "non-remission", the message 135 may be any of "severe", "moderate", or "mild" for severity, and "remission" or "non-remission" for stage. Displaying the final score as a message notifying the result of the severity or stage assessment of the disease is preferable because it allows the user to obtain information to support the diagnosis of severity or stage at a glance without interfering with observation.

[0109] Next, a series of steps in the score display mode will be described with reference to the flowchart shown in FIG. 20. When the score display mode is switched to, the observation target is irradiated with special light. The endoscope 12 captures an image of the observation target illuminated by the special light (step ST110) to obtain an endoscopic image, which is a special image, at a certain point in time. In this flowchart, the final score is set to be automatically obtained at a predetermined interval. The image acquisition unit 71 acquires the special image from the endoscope 12 (step ST120).

[0110] The special image is sent to the inappropriate image discrimination unit 74, where it is judged whether it is appropriate or inappropriate for calculating the raw score. If the special image is judged to be appropriate (YES in step ST130), the raw score is calculated based on the special image (step ST140). If the special image is judged to be inappropriate (NO in step ST130), the raw score is set to "not calculated" (step ST150).

[0111] A final score is determined from the calculation result of the raw score (step ST160). Once the final score is determined, the display control unit 76 controls the display of the final score (step ST170). The display displays the final score through the controlled display (step ST180). If the observation is to be ended (YES in step ST190), the observation is ended. If the observation is not to be ended (NO in step ST190), the process returns to endoscopic image acquisition.

[0112] In the above embodiment, the present invention is applied to an endoscope system that processes endoscopic images, but the present invention can also be applied to a medical image processing system that processes medical images other than endoscopic images. In this case, the medical image processing system includes the image processing device of the present invention. The present invention can also be applied to a diagnosis support device that uses medical images to provide diagnostic support to a user. The present invention can also be applied to a medical work support device that uses medical images to support medical work such as preparing diagnostic reports.

[0113] For example, as shown in Fig. 21, a diagnosis support device 201 is used in combination with modalities such as a medical image processing system 202 and a PACS (Picture Archiving and Communication Systems) 203. Also, as shown in Fig. 22, a medical work support device 210 is connected to various inspection devices such as a first medical image processing system 211, a second medical image processing system 212, ..., an Nth medical image processing system 213 via an arbitrary network 214. The medical work support device 210 receives medical images from the first to Nth medical image processing systems 211, 212, ..., 213, and supports medical work based on the received medical images.

[0114] In the above embodiment, the hardware structure of the processing units that execute various processes, such as the image signal acquisition unit 51, the DSP 52, the noise reduction unit 53, the signal processing unit 55, and the video signal generation unit 56 included in the processor device 16, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing various processes.

[0115] One processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with one processor. As an example of configuring multiple processing units with one processor, first, as represented by a computer such as a client or server, one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form using a processor that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0116] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit in the form of a combination of circuit elements such as semiconductor elements.

[0117] The present invention can also be carried out in the following alternative forms. In a processor device, Acquiring a plurality of endoscopic images by capturing images of an object to be observed at different times using an endoscopic device; Calculating a raw score based on each endoscopic image that is related to the severity or stage of the disease being observed; Determine a final score based on the raw scores; A processor device that controls the display of the final score and / or the change in the final score over time on a display in real time. [Explanation of symbols]

[0118] 10 Endoscope System 12 Endoscopy 12a Insertion part 12b Operation section 12c Curved section 12d Tip 12e Angle knob 12f Still image acquisition instruction section 12g mode switch 12h Zoom operation section 14 Light source device 16 Processor unit (image processing unit) 18 Display 19 Console 20 Light source section 20a V-LED 20b B-LED 20c G-LED 20d R-LED 21 Light source control unit 30a illumination optical system 30b Imaging optical system 41 Light Guide 42 Lighting Lens 43 Objective Lens 44 Zoom Lens 45 Image sensor 46 CDS / AGC circuit 47 A / D Converter 51 Image signal acquisition unit 52 DSP 53 Noise reduction section 54 Memory 55 Signal Processing Section 56 Video signal generator 61 Normal image generation unit 62 Special Image Generation Unit 63 Score Processing Unit 71 Image acquisition unit 72 Raw score calculation section 73 Final Score Decision Section 74 Inappropriate Image Discrimination Unit 75 Image Storage Section 76 Display control section 77 Part determination section 81 First Calculation Section 82 Second Calculation Section 83 1st discrimination part 84 Second discrimination part 85 Superficial blood vessels 86 Intramucosal bleeding 87 Extramucosal bleeding 101 Endoscopic image acquisition 102, 104, 106, 108, 110, 111 Inappropriate image discrimination 103, 105, 107, 109 Raw score calculation 121~125 Endoscopic images 126 Intramucosal bleeding 127 Dense blood vessels 128 Extramucosal bleeding 129 Blurred Image 131, 132 Graph 133 Auxiliary Line 134 Part name 135 Messages 201 Diagnostic support device 202 Medical Image Processing System 203 PACS 210 Medical business support equipment 211 First Medical Image Processing System 212 2nd Medical Image Processing System 213 Nth Medical Image Processing System 214 Network t Final score decision time ST110~ST190 Step

Claims

1. A processor is provided. The processor, Acquiring a plurality of endoscopic images obtained by capturing images of an object to be observed at different times using an endoscope; Calculating a raw score related to the assessment of the severity or stage of the disease of the observation subject based on each of the endoscopic images; determining whether the endoscopic image is appropriate or inappropriate for calculating the raw score for each of the endoscopic images; The raw score for the endoscopic image determined to be inappropriate for calculation of the raw score is not calculated, A raw score non-calculation ratio is obtained based on the discrimination results of the plurality of endoscopic images acquired during a predetermined period prior to the determination of the final score; If the raw score non-calculation ratio is equal to or greater than a predetermined ratio, the final score is not calculated; When the raw score non-calculation ratio is less than the predetermined ratio, the final score is determined based on a plurality of the raw scores other than the non-calculation ratio calculated based on each of the endoscopic images; An image processing device that performs control to display on a display in real time the relationship between the final score and the time when the final score was determined.

2. A processor is provided. The processor, Acquiring a plurality of endoscopic images obtained by capturing images of an object to be observed at different times using an endoscope; Calculating a raw score related to the assessment of the severity or stage of the disease of the observation subject based on each of the endoscopic images; determining whether the endoscopic image is appropriate or inappropriate for calculating the raw score for each of the endoscopic images; The raw score for the endoscopic image determined to be inappropriate for calculation of the raw score is not calculated, A raw score non-calculation ratio is obtained based on the discrimination results of the plurality of endoscopic images acquired during a predetermined period prior to the determination of the final score; If the raw score non-calculation ratio is equal to or greater than a predetermined ratio, the final score is not calculated; When the raw score non-calculation ratio is less than the predetermined ratio, the final score is determined from the raw scores other than the non-calculation ratio calculated based on each of the endoscopic images; An image processing device that displays the final score on a display in real time, and performs control not to display the final score if the final score is smaller than a preset threshold value.

3. The image processing device according to claim 1 , wherein the processor calculates two or more different raw scores.

4. The image processing device according to claim 1 , wherein the processor calculates the raw score based on a first feature amount obtained by analyzing the endoscopic image.

5. The image processing apparatus according to claim 4 , wherein the first feature amount is an amount related to a superficial vascularized area, an intramucosal bleeding area, or an extramucosal bleeding area contained in the endoscopic image.

6. 4. The image processing device according to claim 1, wherein the processor executes a trained first machine learning model generated by inputting the past endoscopic images to which the raw scores are associated into a machine learning model, and calculates the raw scores based on the endoscopic images.

7. The image processing device according to claim 1 , wherein the processor determines the final score from the raw score calculated based on a plurality of the endoscopic images acquired during a predetermined period prior to the time point at which the final score is determined.

8. The image processing device according to claim 2 or 7, wherein the processor determines the final score by performing a moving average, FIR filter processing, or IIR filter processing on a plurality of the raw scores.

9. The image processing device according to claim 1 , wherein the processor determines whether the endoscopic image is appropriate or inappropriate for calculating the raw score based on a second feature amount of the endoscopic image.

10. 10. The image processing device according to claim 9, wherein the second feature amount is a quantity related to at least one selected from the group consisting of a halation distribution, a spatial frequency distribution, a luminance value distribution, a shadow distribution, a magnification ratio index, and a reflected light distribution of illumination light irradiated to the object of observation of the endoscopic image.

11. 3. The image processing device according to claim 1, wherein the processor executes a trained second machine learning model generated by inputting past endoscopic images, each of which is associated with whether it is appropriate or inappropriate for calculating the raw score, into a machine learning model, and determines whether the endoscopic image is appropriate or inappropriate for calculating the raw score.

12. The image processing device of claim 1 , wherein the processor displays at least one graph showing a relationship between the final score and a time when the final score was determined.

13. The processor determines a region of the observation target included in the endoscopic image by performing image analysis on the endoscopic image; The image processing device according to claim 1 , wherein at least one graph showing a relationship between the final score, the determination time of the final score, and the part of the body is displayed.

14. The processor issues an instruction to acquire a still image; The image processing device according to claim 1 , wherein, when the instruction is given, control is performed to display a relationship between the final score and the determination time of the final score.

15. The processor issues an instruction to acquire a still image; The image processing device according to claim 2 , wherein when the instruction is given, control is performed to display the final score.

16. The image processing device according to claim 1 , wherein the disease is ulcerative colitis.

17. The image processing device according to claim 2 , wherein the processor controls to display, on a display, in real time, the relationship between the final score and the determination time of the final score.

18. The image processing device according to claim 1 , wherein the processor performs control such that the final score is not displayed when the final score is smaller than a preset threshold value.

19. An endoscope for capturing an image of an object to be observed and an image processing device having a processor are provided, The processor acquires a plurality of endoscopic images obtained by capturing images of the observation target at different times, and calculates a raw score related to the assessment of the severity or stage of a disease of the observation target based on each of the endoscopic images; determining whether the endoscopic image is appropriate or inappropriate for calculating the raw score for each of the endoscopic images; The raw score for the endoscopic image determined to be inappropriate for calculation of the raw score is not calculated, A raw score non-calculation ratio is obtained based on the discrimination results of the plurality of endoscopic images acquired during a predetermined period prior to the time of determining the final score based on the raw score; If the raw score non-calculation ratio is equal to or greater than a predetermined ratio, the final score is not calculated; When the raw score non-calculation ratio is less than the predetermined ratio, the final score is determined based on a plurality of the raw scores other than the non-calculation ratio calculated based on each of the endoscopic images; An endoscope system that performs control to display on a display in real time the relationship between the final score and the time when the final score was determined.

20. An endoscope for capturing an image of an object to be observed and an image processing device having a processor are provided, The processor acquires a plurality of endoscopic images obtained by capturing images of the observation target at different times, calculates a raw score related to the assessment of the severity or stage of a disease of the observation target based on each of the endoscopic images, and determines whether each of the endoscopic images is appropriate or inappropriate for the calculation of the raw score; The raw score for the endoscopic image determined to be inappropriate for calculation of the raw score is not calculated, A raw score non-calculation ratio is obtained based on the discrimination results of the plurality of endoscopic images acquired during a predetermined period prior to the determination of the final score; If the raw score non-calculation ratio is equal to or greater than a predetermined ratio, the final score is not calculated; When the raw score non-calculation ratio is less than the predetermined ratio, the final score is determined from the raw scores other than the non-calculation ratio calculated based on each of the endoscopic images; An endoscope system that displays the final score on a display in real time, and performs control so as not to display the final score if the final score is smaller than a preset threshold value.

21. 1. A method of operating an image processing device, comprising: an image acquiring step in which an image acquiring unit acquires a plurality of endoscopic images by capturing images of an observation target at different times; a raw score calculation step in which a raw score calculation unit calculates a raw score related to the determination of the severity or stage of the disease of the observation target based on each of the endoscopic images; an inappropriate image discrimination step in which an inappropriate image discrimination unit discriminates, for each of the endoscopic images, whether the endoscopic image is appropriate or inappropriate for calculation of the raw score; a final score determination step in which the final score determination unit does not calculate the raw score for the endoscopic image determined to be unsuitable for calculation of the raw score, obtains a raw score non-calculation ratio based on the discrimination results of a plurality of endoscopic images acquired during a predetermined period prior to the time of determination of the final score based on the raw score, and does not calculate the final score if the raw score non-calculation ratio is equal to or greater than a predetermined ratio, and determines the final score based on a plurality of the raw scores other than the non-calculation ratio calculated based on each of the endoscopic images if the raw score non-calculation ratio is less than the predetermined ratio; a display control step in which a display control unit controls a display of a relationship between the final score and a determination time of the final score on a display in real time; A method of operating an image processing apparatus comprising:

22. 1. A method of operating an image processing device, comprising: an image acquiring step in which an image acquiring unit acquires a plurality of endoscopic images by capturing images of an observation target at different times; a raw score calculation step in which a raw score calculation unit calculates a raw score related to the determination of the severity or stage of the disease of the observation target based on each of the endoscopic images; an inappropriate image discrimination step in which an inappropriate image discrimination unit discriminates, for each of the endoscopic images, whether the endoscopic image is appropriate or inappropriate for calculation of the raw score; a final score determination step in which a final score determination unit does not calculate the raw score for the endoscopic image determined to be unsuitable for calculation of the raw score, obtains a raw score non-calculation ratio based on the discrimination results of a plurality of endoscopic images acquired during a predetermined period prior to the time of determination of the final score, and does not calculate the final score if the raw score non-calculation ratio is equal to or greater than a predetermined ratio, and determines the final score from a plurality of raw scores other than the non-calculation ratio calculated based on each of the endoscopic images if the raw score non-calculation ratio is less than the predetermined ratio; a display control step of displaying the final score on a display in real time, and controlling not to display the final score when the final score is smaller than a preset threshold value; A method of operating an image processing apparatus comprising:

Images