Medical image processing device and endoscope system

Abstract

The present invention provides a medical image processing device (17) and an endoscope system (10) that are capable of yielding diagnosis information and diagnosis reference information with excellent precision on the basis of a plurality of types of medical images.　The medical image processing device (17) acquires a plurality of types of medical images in which a subject has been imaged under imaging conditions that are different from each other. In a case in which a first medical image (72), which is one type of a medical image from a plurality of types, is input, diagnosis information (91A) relating to diagnosis of a subject in the first medical image (72) is generated. Diagnosis reference information (92Y) is generated using imparted reference information (94A) that is included in a medical image and that is reference information imparted to a second medical image (71) of a different type from the first medical image (72).

Description

[Technical Field]

[0001] The present invention relates to a medical image processing apparatus and an endoscopy system that provides diagnostic information and reference information. [Background technology]

[0002] CAD (Computer-Aided Diagnosis) technology has been developed that applies appropriate image processing to endoscopic images to determine the stage of disease, etc. Image-processing-based CAD uses images (hereinafter referred to as endoscopic images) taken of the subject being observed with an endoscope, with a computer processor calculating the external appearance (endoscopic features) on behalf of a doctor, and then identifies the internal contents (pathology) of the subject through diagnostics. In this way, image-processing-based CAD estimates the severity of disease, etc., based on feature values ​​that quantify external characteristics such as the shape of blood vessels, which doctors and other humans can understand, allowing doctors to understand the basis for the CAD estimates.

[0003] On the other hand, with CAD based on image processing, if there are variations in the endoscopic image, such as the distance and angle between the scope and the subject, the presence or absence of halation, or the presence or absence of blisters, the calculated feature values ​​change, making it difficult to achieve accurate CAD estimation results, and in some cases high accuracy cannot be expected for images taken under conditions other than those specified.

[0004] Recently, CAD technology based on AI (Artificial Intelligence) using machine learning and other methods has been developed. AI-based CAD does not rely on diagnostics, but instead identifies external features that correlate highly with the contents, and calculates and outputs the contents as an inferred result accordingly. In this way, AI-based CAD ignores diagnostics, so humans cannot interpret what features the CAD used to arrive at its inferred result.

[0005] In response to this, for example, an information processing device is known that includes a model that, when an endoscopic image is input, outputs a diagnosis result regarding a disease diagnosis, as well as regions that contributed to the diagnosis or a diagnostic criterion prediction (Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2020 / 116115 Summary of the Invention [Problem to be solved by the invention]

[0007] While AI-based CAD has the so-called black box problem of being unable to interpret the characteristics on which the CAD based its judgment, it also has the advantage that even if there are variations in endoscopic images, such as the distance and angle between the scope and the object of observation, the presence or absence of halation, or the presence or absence of blisters, as long as it has learned a variety of such scenes, the accuracy of the estimation results can be fully guaranteed.

[0008] In order to improve the accuracy of inference results in AI-based CAD, it is necessary to use features that humans cannot grasp or manage and to perform calculations by combining these in more complex ways, but this is thought to make it more difficult to explain to humans what features were used to make a judgment. In other words, in AI-based CAD, there is a trade-off between "explainability of judgment" and "high accuracy."

[0009] Therefore, there is a need to develop CAD that combines the advantage of image processing-based CAD, which is "explainability of decisions," and the advantage of AI-based CAD, which is "high accuracy."

[0010] An object of the present invention is to provide a medical image processing apparatus and an endoscope system that can obtain highly accurate diagnostic information and reference information related to diagnostic standards using a plurality of types of medical images. [Means for solving the problem]

[0011] A medical image processing device of the present invention includes a processor that acquires multiple types of medical images of a subject captured under different imaging conditions, and when a first medical image that is one of the multiple types of medical images is input, generates diagnostic information related to a diagnosis of the subject appearing in the first medical image, and when a medical image of the same subject as the subject appearing in the first medical image is input, generates reference information related to a diagnostic standard and outputs the diagnostic information in association with the reference information, the reference information being generated using reference information that is included in the medical image and that is reference information that has been assigned to a second medical image that is different in type from the first medical image.

[0012] Preferably, the processor generates the reference information using the attached reference information and a medical image showing the same subject as the subject shown in the second medical image to which the attached reference information is attached.

[0013] When a medical image depicting the same subject as that depicted in the first medical image is input, it is preferable that the processor acquires features of the input medical image and generates reference information by converting the features into reference information.

[0014] When a medical image depicting the same subject as that depicted in the first medical image is input, it is preferable that the processor acquires multiple features of the input medical image and generates reference information by converting the multiple features into at least one piece of reference information.

[0015] The processor preferably includes a first model for generating diagnostic information, the first model having a layered structure including a first output layer for outputting the diagnostic information and at least one first hidden layer.

[0016] The processor preferably includes a second model that generates features, the second model having a layered structure including a second output layer that outputs the features and at least one second intermediate layer, and the second intermediate layer preferably shares the first intermediate layer.

[0017] The assigned reference information is preferably reference information assigned to the second medical image by a doctor by visually inspecting the second medical image.

[0018] When the first medical image is input, the processor preferably acquires a first feature amount of the first medical image and generates diagnostic information based on the first feature amount.

[0019] It is preferable that the processor generates the reference information by converting the first feature into the reference information using the reference information and a first medical image depicting the same subject as that depicted in the second medical image to which the reference information is attached.

[0020] When a second medical image depicting the same subject as that depicted in the first medical image is input, the processor preferably acquires a second feature of the second medical image and generates reference information by converting the second feature into reference information using the assigned reference information and the second medical image.

[0021] Preferably, the processor acquires the second feature of a second medical image taken within a predetermined period of time before and / or after the time the first medical image was taken.

[0022] The processor preferably controls the display of the first medical image and / or the second medical image on the display.

[0023] The processor preferably controls the display of the associated diagnostic information and reference information on a display.

[0024] The photographing condition is preferably the spectrum of the illumination light.

[0025] The endoscopic system of the present invention also includes a processor device having a plurality of light sources that emit light in different wavelength bands and a light source processor that controls the emission of each of a plurality of types of illumination light having different combinations of light intensity ratios from the plurality of light sources, an endoscope that photographs a subject illuminated by the illumination light, and a medical image processing device. [Effects of the Invention]

[0026] According to the present invention, it is possible to obtain highly accurate diagnostic information and reference information relating to diagnostic standards using a plurality of types of medical images. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is an external view of an endoscope system. [Figure 2] FIG. 2 is a block diagram showing the functions of the endoscope system. [Figure 3] FIG. 2 is an explanatory diagram illustrating four color LEDs included in the light source unit. [Figure 4] 1 is a graph showing the spectra of violet light V, blue light B, green light G, and red light R. [Figure 5] 10 is a graph showing the spectrum of the first illumination light. [Figure 6] FIG. 2 is an explanatory diagram illustrating the types and order of endoscopic images captured by the endoscopic system. [Figure 7] FIG. 2 is a block diagram showing the functions of the medical image processing apparatus. [Figure 8] FIG. 3 is an explanatory diagram illustrating the function of a first diagnostic model. [Figure 9] FIG. 10 is an explanatory diagram illustrating finding information. [Figure 10] 10 is a block diagram showing functions of a medical image processing apparatus in which a reference information generating unit includes a first feature amount model. FIG. [Figure 11] FIG. 3 is an explanatory diagram illustrating the function of a first feature quantity model. [Figure 12] FIG. 10 is an explanatory diagram illustrating converter information. [Figure 13]FIG. 10 is an explanatory diagram illustrating a method for generating a first reference converter. [Figure 14] FIG. 10 is an explanatory diagram illustrating a method for generating diagnostic information and reference information. [Figure 15] FIG. 10 is an image diagram showing diagnostic information and reference information displayed on a display. [Figure 16] 10 is a flowchart showing a series of steps for displaying diagnostic information and reference information in a medical image processing apparatus. [Figure 17] 10 is a block diagram showing functions of a medical image processing apparatus in which a reference information generating unit includes a first feature amount acquiring unit. FIG. [Figure 18] FIG. 10 is an explanatory diagram illustrating the flow of a first feature amount. [Figure 19] FIG. 10 is an image diagram showing diagnostic information and reference information on a display together with the type of image used. [Figure 20] FIG. 10 is a block diagram showing functions of a medical image processing apparatus in which a reference information generating unit includes a second feature amount model. [Figure 21] FIG. 10 is an explanatory diagram illustrating the function of a second feature amount model. [Figure 22] FIG. 10 is an image diagram showing diagnostic information and reference information on a display together with the type of image used. [Figure 23] FIG. 10 is an explanatory diagram illustrating a case where a medical image processing apparatus is included in a diagnosis support apparatus. [Figure 24] FIG. 10 is an explanatory diagram illustrating a case where a medical image processing apparatus is included in a medical service support apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0028] 1, the endoscope system 10 includes an endoscope 12, a light source device 13, a processor device 14, a display 15, a keyboard 16, and a medical image processing device 17. The endoscope 12 is optically connected to the light source device 13 and electrically connected to the processor device 14. The processor device 14 is connected to the medical image processing device 17. The medical image processing device 17 acquires an endoscopic image, which is a medical image, from the processor device 14 and performs various processes to acquire various information, etc.

[0029] In this embodiment, the medical image is an endoscopic image. Furthermore, in this embodiment, the medical image processing device 17 and the processor device 14 are separate devices, but a device that performs the functions of the medical image processing device 17 may be disposed within the processor device 14, and the processor device 14 may perform the functions of the medical image processing device 17. Furthermore, various connections may be wired or wireless, or may be made via a network. Therefore, the functions of the medical image processing device 17 may be performed by an external device connected via a network.

[0030] The endoscope 12 has an insertion section 12a that is inserted into the body of a subject having an observation target, an operation section 12b provided at the base end of the insertion section 12a, and a bending section 12c and a tip section 12d provided at the tip side of the insertion section 12a. The bending section 12c is bent by operating an angle knob 12e (see FIG. 2) of the operation section 12b. The tip section 12d is directed in a desired direction by the bending operation of the bending section 12c.

[0031] The operation unit 12b has an angle knob 12e, a zoom operation unit 12f for changing the imaging magnification, and a mode changeover switch 12g used for switching the observation mode. Note that the switching operation of the observation mode or the zoom operation may be performed or instructed using the keyboard 16, a foot switch (not shown), or the like, in addition to the mode changeover switch 12g or the zoom operation unit 12f.

[0032] The endoscope system 10 has three observation modes: a normal observation mode, a special observation mode, and a diagnosis assistance observation mode. The normal observation mode is a mode in which a normal image, which is an image with natural coloring obtained by capturing an image of an observation target using white light as illumination light, is displayed on the display 15. The special observation modes include a first special observation mode. The first special observation mode is a mode in which a first medical image (hereinafter referred to as the first image) in which surface information such as superficial blood vessels is emphasized is displayed on the display 15.

[0033] The diagnosis assistance observation mode is a mode in which the normal image and / or the first image, diagnostic information related to the diagnosis of the observation object and reference information related to the criteria for this diagnosis, which are generated and output by the medical image processing device 17, are displayed on the display 15. The diagnostic information is information generated and output by the medical image processing device 17 related to the diagnosis of the observation object based on the endoscopic image. The reference information is information generated and output by the medical image processing device 17 related to the criteria for the diagnosis of the observation object based on the endoscopic image. The diagnostic information and reference information are information related to the diagnosis of the observation object, which is the subject captured in the endoscopic image, and the criteria related to this diagnosis, and are information that supports the doctor's diagnosis. By displaying the diagnostic information and reference information on the display 15, the displayed diagnostic information and reference information support the doctor's diagnosis when the doctor makes a diagnosis by looking at the normal image and / or the first image displayed on the display 15.

[0034] The processor device 14 is electrically connected to a display 15 and a keyboard 16. The display 15 displays the normal image, the first image, the diagnostic information, the reference information, and / or information associated therewith. The keyboard 16 functions as a user interface that accepts input operations such as function settings. An external storage (not shown) for saving images, image information, etc. may be connected to the processor device 14.

[0035] As shown in FIG. 2, the light source device 13 emits illumination light to illuminate an object to be observed and includes a light source unit 20 and a light source processor 21 that controls the light source unit 20. The light source unit 20 is configured, for example, with a semiconductor light source such as a multicolored light-emitting diode (LED), a combination of a laser diode and a phosphor, or a xenon lamp or halogen light source. The light source unit 20 also includes an optical filter or the like for adjusting the wavelength band of the light emitted by the LED or the like. The light source processor 21 controls the amount of illumination light by turning on / off each LED or adjusting the drive current or drive voltage of each LED or the like. The light source processor 21 also controls the wavelength band of the illumination light by changing the optical filter or the like.

[0036] As shown in FIG. 3, in this embodiment, the light source unit 20 has four color LEDs: a V-LED (Violet Light Emitting Diode) 20a, a B-LED (Blue Light Emitting Diode) 20b, a G-LED (Green Light Emitting Diode) 20c, and an R-LED (Red Light Emitting Diode) 20d.

[0037] As shown in FIG. 4, the V-LED 20a emits violet light V with a central wavelength of 410±10 nm and a wavelength range of 380 to 420 nm. The B-LED 20b emits blue light B with a central wavelength of 450±10 nm and a wavelength range of 420 to 500 nm. The G-LED 20c emits green light G with a wavelength range of 480 to 600 nm. The R-LED 20d emits red light R with a central wavelength of 620 to 630 nm and a wavelength range of 600 to 650 nm.

[0038] The light source processor 21 controls the V-LED 20a, B-LED 20b, G-LED 20c, and R-LED 20d. In the normal observation mode, the light source processor 21 controls each of the LEDs 20a to 20d so that the LEDs 20a to 20d emit normal light in which the light intensity ratio between the purple light V, the blue light B, the green light G, and the red light R is Vc:Bc:Gc:Rc.

[0039] When the first special observation mode is set, the light source processor 21 controls the LEDs 20a to 20d to emit first illumination light in which the light intensity ratio between the purple light V, the blue light B, the green light G, and the red light R is Vs1:Bs1:Gs1:Rs1. The first illumination light preferably emphasizes superficial blood vessels. Therefore, it is preferable that the light intensity of the purple light V is greater than the light intensity of the blue light B. For example, as shown in FIG. 5, the ratio between the light intensity Vs1 of the purple light V and the light intensity Bs1 of the blue light B is set to "4:1."

[0040] In this specification, a combination of light intensity ratios includes a case where the ratio of at least one semiconductor light source is 0 (zero). Therefore, it also includes a case where one or more of the semiconductor light sources are not lit. For example, when only one of the semiconductor light sources is lit and the other three are not lit, such as when the light intensity ratio combination among purple light V, blue light B, green light G, and red light R is 1:0:0:0, this also has a light intensity ratio and is one of the combinations of light intensity ratios.

[0041] As described above, the combinations of the light intensity ratios of the violet light V, blue light B, green light G, and red light R, i.e., the types of illumination light, emitted in the normal observation mode and the first special observation mode are different from each other. In the diagnosis assistance observation mode, multiple different types of illumination light are emitted by automatically switching between them. Note that an observation mode using a different type of illumination light having a combination of light intensity ratios different from the illumination light used in these observation modes may also be used.

[0042] When the diagnostic assistance observation mode is set, the light source processor 21 switches between emitting specific types of illumination light. Specifically, it alternates between a normal light period in which normal light is continuously emitted and a first illumination light period in which first illumination light is continuously emitted. Regarding the period, the normal light period in which normal light is emitted is performed for a predetermined number of frames, followed by a first illumination light period in which first illumination light is emitted for a predetermined number of frames. After that, the normal light period begins again, and a set of the normal light period and the first illumination light period is repeated.

[0043] Note that the term "frame" refers to a unit for controlling the image sensor 45 (see FIG. 2) that captures an image of an object to be observed, and for example, "one frame" refers to a period that includes at least an exposure period in which the image sensor 45 is exposed to light from the object to be observed and a readout period in which an image signal is read out. In this embodiment, various periods such as a normal light period or a first illumination light period are defined corresponding to the "frame" that is the unit of photography.

[0044] As shown in FIG. 6, in the diagnostic assistance observation mode, a normal light period in which normal light, indicated by "Normal" in the illumination light column, is performed for a period of three frames, followed by a switch in illumination light and a first illumination light period in which first illumination light, indicated by "First" in the illumination light column, is performed for a period of one frame. The normal light period then resumes, and a set of normal light and first illumination light periods is repeated for four frames. Therefore, three normal images 71 are captured consecutively during the three-frame normal light period, and then a first image 72 is captured once during the first illumination light period. The normal light period then resumes, and this pattern is repeated. In the figure, the first image 72 is shown shaded because it has a different color from the normal image 71.

[0045] Each LED20a~20 d The light emitted from the endoscope 12 is incident on the light guide 41 via an optical path coupling section (not shown) composed of a mirror, a lens, etc. The light guide 41 is built into the endoscope 12 and a universal cord (a cord that connects the endoscope 12 with the light source device 13 and the processor device 14). The light guide 41 propagates the light from the optical path coupling section to the tip 12d of the endoscope 12.

[0046] An illumination optical system 30a and an imaging optical system 30b are provided at the distal end 12d of the endoscope 12. The illumination optical system 30a has an illumination lens 42, and illumination light propagated by a light guide 41 is irradiated onto the observation object via the illumination lens 42. The imaging optical system 30b has an objective lens 43, a zoom lens 44, and an imaging sensor 45. Various types of light, such as reflected light, scattered light, and fluorescence from the observation object, are incident on the imaging sensor 45 via the objective lens 43 and the zoom lens 44. As a result, an image of the observation object is formed on the imaging sensor 45. The zoom lens 44 is freely movable between the telephoto end and the wide-angle end by operating the zoom operation unit 12f, thereby enlarging or reducing the image of the observation object formed on the imaging sensor 45.

[0047] The image sensor 45 is a color image sensor in which each pixel is provided with an R (red), G (green), or B (blue) color filter. It captures an image of the object to be observed and outputs image signals of each of the RGB colors. The image sensor 45 may be a charge-coupled device (CCD) image sensor or a complementary metal-oxide semiconductor (CMOS) image sensor. Instead of the image sensor 45 equipped with primary color filters, a complementary image sensor equipped with complementary color filters of C (cyan), M (magenta), Y (yellow), and G (green) may be used. When a complementary image sensor is used, four CMYG image signals are output. Therefore, by converting the four CMYG image signals into three RGB image signals using complementary-to-primary color conversion, an RGB image signal similar to that of the image sensor 45 can be obtained. Alternatively, a monochrome sensor without color filters may be used instead of the image sensor 45.

[0048] The imaging sensor 45 is driven and controlled by an imaging control unit (not shown). The central control unit 58 (see FIG. 2) controls the light emission of the light source unit 20 through the light source processor 21 in synchronization with the imaging control unit, thereby controlling the imaging sensor 45 to capture an image of an observation target illuminated with normal light in the normal observation mode. As a result, Bc image signals are output from the B pixels of the imaging sensor 45, Gc image signals are output from the G pixels, and Rc image signals are output from the R pixels. In the first special observation mode, the central control unit 58 controls the light emission of the light source unit 20 to control the imaging sensor 45 to capture an image of an observation target illuminated with the first illumination light. As a result, in the first special observation mode, Bs1 image signals are output from the B pixels of the imaging sensor 45, Gs1 image signals are output from the G pixels, and Rs1 image signals are output from the R pixels.

[0049] Furthermore, in the diagnosis assistance observation mode, the central controller 58 (see FIG. 2) controls the light emission of the light source unit 20 and controls the imaging sensor 45 to capture images of the observation target illuminated with normal light and first illumination light for each preset period. As a result, in the diagnosis assistance observation mode, during the normal light period, Bc image signals are output from the B pixels of the imaging sensor 45, Gc image signals are output from the G pixels, and Rc image signals are output from the R pixels. Then, during the first illumination light period, Bs1 image signals are output from the B pixels of the imaging sensor 45, Gs1 image signals are output from the G pixels, and Rs1 image signals are output from the R pixels.

[0050] A CDS / AGC (Correlated Double Sampling / Automatic Gain Control) circuit 46 performs correlated double sampling (CDS) and automatic gain control (AGC) on the analog image signal obtained from the image sensor 45. The image signal passed through the CDS / AGC circuit 46 is converted into a digital image signal by an A / D (Analog / Digital) converter 47. The digital image signal after A / D conversion is input to the processor device 14.

[0051] The processor device 14 stores programs related to image processing and other processes in a program memory (not shown). In the processor device 14, a central control unit 58, which is composed of an image processor serving as a first processor, etc., runs the programs in the program memory, thereby realizing the functions of an image acquisition unit 51, a DSP (Digital Signal Processor) 52, a noise reduction unit 53, a memory 54, an image processing unit 55, a display control unit 56, a video signal generation unit 57, and the central control unit 58. The central control unit 58 also receives information from the endoscope 12 and the light source device 13, and controls the various units of the processor device 14 as well as the endoscope 12 or the light source device 13 based on the received information. The central control unit 58 also receives information such as instructions from the keyboard 16.

[0052] The image acquisition unit 51 acquires digital image signals of endoscopic images input from the endoscope 12. The image acquisition unit 51 acquires, for each frame, image signals of an object to be observed illuminated by each illumination light. The type of illumination light, i.e., the spectrum of the illumination light, is one of the imaging conditions. In this embodiment, the spectrum of the illumination light is used as the imaging condition, and the image acquisition unit 51 acquires multiple types of endoscopic images with different imaging conditions, such as the spectrum of the illumination light.

[0053] The imaging conditions include the illumination light spectrum, i.e., the light intensity ratio of each LED 20a to 20d, as well as the imaging time, the observation distance to the object of observation, and the zoom magnification of the endoscope 12. The light intensity ratio is acquired from the central control unit 58. The imaging time may be acquired from header information or the like contained in the endoscopic image and then acquired from the central control unit 58. The observation distance may be, for example, a non-magnified observation distance where the observation distance is a long distance, or a magnified observation distance where the observation distance is a short distance, and is acquired by the exposure amount obtained from the endoscopic image or a length measurement laser beam, etc. Note that the observation distance may also be acquired by frequency analysis of the image. The zoom magnification of the endoscope 12 may be, for example, a non-magnified observation for non-magnified observation, or a low to high magnification that allows magnified observation, and can be acquired by operating the zoom operation unit 12f.

[0054] The acquired image signal is transmitted to the DSP 52. The DSP 52 performs digital signal processing such as color correction on the received image signal. The noise reduction unit 53 performs noise reduction processing, such as a moving average method or a median filter method, on the image signal that has been subjected to color correction and other processing by the DSP 52. The noise-reduced image signal is stored in the memory 54.

[0055] The image processing unit 55 acquires the noise-reduced image signal from the memory 54. Then, the acquired image signal is subjected to signal processing such as color conversion processing, color enhancement processing, and structure enhancement processing as necessary to generate a color endoscopic image showing the object of observation. The image processing unit 55 includes a normal image processing unit 61 and a special image processing unit 62.

[0056] In the image processing unit 55, the normal image processing unit 61 performs image processing for the normal observation mode, such as color conversion processing, color enhancement processing, and structure enhancement processing, on the input image signal for a normal image after noise reduction for one frame in the normal observation mode or the diagnostic assistance observation mode. The image signal that has been subjected to this image processing for the normal observation mode is input as a normal image 71 to the medical image processing device 17 and / or the display control unit 56.

[0057] In the special observation mode or the diagnostic assistance observation mode, the special image processing unit 62 performs image processing for the first special observation mode, such as color conversion processing, color enhancement processing, and structure enhancement processing, on the image signal of the first image after noise reduction for one frame input in the first special observation mode. The image signal that has been subjected to the image processing for the first special observation mode is input to the medical image processing device 17 and / or the display control unit 56 as a first image 72. Note that the image processing unit 55 may adjust the frame rate when inputting the endoscopic image to the medical image processing device 17 and / or the display control unit 56.

[0058] The endoscopic image generated by the image processing unit 55 is a normal image 71 when the observation mode is the normal observation mode, and a first image 72 when the observation mode is the first special observation mode. The contents of the color conversion processing, color enhancement processing, and structure enhancement processing differ depending on the observation mode. In the normal observation mode, the image processing unit 55 generates the normal image 71 by performing the various signal processing described above to give the observation target a natural color. In the special observation mode, the image processing unit 55 generates the first image 72 by performing the various signal processing described above to, for example, enhance the blood vessels of the observation target.

[0059] The semiconductor light source has a wavelength band with a center wavelength of 410±10 nm and a wavelength range of 38 0~ 42 V-LED20a emitting purple light V (first narrowband light) with a wavelength band of 450±10 nm and a wavelength range of 450±10 nm 42 0~ 50 and a B-LED 20b that emits blue light B (second narrowband light) having a wavelength of 100 nm. Therefore, in the first image 72 generated by the image processing unit 55, blood vessels (so-called superficial blood vessels) or blood that are located relatively shallow within the object of observation relative to the surface of the mucosa appear in a magenta color (e.g., brown). Therefore, in the first image 72, the blood vessels or bleeding (blood) of the object of observation are emphasized by the difference in color compared to the mucosa, which is displayed in a pink color.

[0060] The display control unit 56 receives the endoscopic image generated by the image processing unit 55 and controls the display on the display 15. The endoscopic image controlled for display by the display control unit 56 is generated into a video signal for display on the display 15 by the video signal generation unit 57 and sent to the display 15. The display 15 displays the endoscopic image sent from the video signal generation unit 57 under the control of the display control unit 56.

[0061] The medical image processing device 17 acquires the endoscopic image generated by the image processing unit 55 and generates and outputs diagnostic information and reference information based on the endoscopic image. The medical image processing device 17 is a general-purpose PC equipped with a processor, and various functions are realized by installing software. Like the processor device 14, the medical image processing device 17 also has programs related to processes such as image analysis stored in a program memory (not shown). In the medical image processing device 17, a central control unit (not shown) composed of an image processor, which is a second processor, executes the programs stored in the program memory to realize the functions of a medical image acquisition unit 81, a diagnostic information generation unit 82, a reference information generation unit 83, an information output unit 84, and a display control unit 85 (see FIG. 7). The central control unit also receives information from the processor device 14 and other devices and controls each unit of the medical image processing device 17 based on the received information. The central control unit is also connected to a user interface, such as a keyboard (not shown), and receives information such as instructions from the user interface.

[0062] The medical image processing device 17 is connected to the display 15 and controls the display of various information generated by the medical image processing device 17. Various devices may be connected to the medical image processing device 17. Examples of the various devices include a user interface such as a keyboard for issuing instructions, and storage for saving data such as images and information. The medical image processing device 17 also has a network connection function for connecting to various devices. The network connection function allows the medical image processing device 17 to be connected to, for example, a medical service support device 630 (see FIG. 24 ).

[0063] As shown in FIG. 7 , the medical image processing device 17 includes a medical image acquisition unit 81, a diagnostic information generation unit 82, a reference information generation unit 83, an information output unit 84, and a display control unit 85. The medical image acquisition unit 81 acquires multiple types of endoscopic images sent from the processor device 14. The acquired endoscopic images are sent to the diagnostic information generation unit 82 and the reference information generation unit 83. The diagnostic information generation unit 82 includes a first diagnostic model (first model) 91, and the reference information generation unit 83 includes a reference converter 92 and a finding information storage unit 93. The information output unit 84 receives the diagnostic information generated by the diagnostic information generation unit 82 and the diagnostic information reference information generated by the reference information generation unit 83, and stores or outputs this information for notification to a user such as a doctor. The display control unit 85 receives the diagnostic information and reference information from the information output unit 84 and controls display on the display 15.

[0064] In this embodiment, the medical image acquisition unit 81 transmits a first image 72 from among multiple types of captured endoscopic images to the diagnostic information generation unit 82. When the diagnostic information generation unit 82 receives the first image 72 sent from the medical image acquisition unit 81, it generates diagnostic information related to the diagnosis of the observation target shown in the endoscopic image. The diagnostic information generation unit 82 performs AI-based CAD as a method of acquiring diagnostic information.

[0065] Diagnostic information indicates the predicted severity and progression of various diseases. In diagnoses using endoscopic images, various information about the surface structure of the object being observed or the biological information of the mucosal surface can be obtained from endoscopic image findings obtained by image-enhanced observation using image-enhanced endoscopy (IEE). There are various methods for IEE, such as digital image processing of endoscopic images obtained by capturing an image of the object, or capturing an image of the object being observed by illuminating it with a specific illumination light.

[0066] Endoscopic images taken with IEE may reveal image features that differ from those of endoscopic images taken with normal white light, or may reveal features similar to those of normal images due to the higher resolution. Therefore, by predicting and diagnosing the severity and progression based on endoscopic images taken with IEE, it may be possible to predict the severity and progression with high accuracy.

[0067] For example, IEE diagnostic technology, which uses IEE to predict pathology for various cancers, is being developed at the initiative of physicians. IEE diagnostic technology statistically finds the relationship between the appearance obtained from endoscopic images, i.e., the structure of the surface blood vessels and mucosa of the object being observed, and the pathology of the object being observed, i.e., the depth of cancer infiltration, and when the blood vessels and mucosa are in a certain state, a classification of the pathology, i.e., the depth of cancer infiltration, is specified. The classification of the depth of cancer infiltration of the object being observed is diagnostic information. In this case, the classification of the structure of the surface blood vessels and mucosa of the object being observed is reference information.

[0068] Note that the severity and progression include pathological severity and pathological progression determined by observing biopsy tissues by a pathologist, etc., as well as endoscopic severity and endoscopic progression determined by visual evaluation of endoscopic images by an endoscopist, etc. In this specification, the accuracy of predicting pathological severity and pathological progression, etc., means a high rate at which the results of pathological severity and pathological progression predicted based on endoscopic images match the actual pathological severity and pathological progression of the subject, and the accuracy of predicting endoscopic severity and endoscopic progression, etc., means a high rate at which the results of endoscopic severity and endoscopic progression predicted based on endoscopic images match the actual endoscopic severity and endoscopic progression of the subject, etc.

[0069] Specific examples of IEE diagnostic techniques include VS classification (vessel plus surface classification) for diagnosing gastric cancer, the Japan Esophageal Society classification (IPCL classification) for diagnosing esophageal cancer, and the JNET classification or NICE classification for diagnosing colorectal cancer.

[0070] For example, in VS classification, a diagnosis is made by combining the criteria of microvascular (MV) architecture (V) (Regular, Irregular, Absent) and the criteria of microsurface (MS) structure (S) (Regular, Irregular, Absent) in endoscopic findings. For example, if both V and S are "Regular," the diagnosis is a hyperplastic polyp, not cancer.

[0071] In addition, in the JNET classification, endoscopic findings are divided into four categories, Type 1, 2A, 2B, and 3, in the categories of vessel pattern and surface pattern, respectively. Type 1 is diagnosed as a hyperplastic polyp, Type 2A as an adenoma or low-grade cancer, Type 2B as a high-grade cancer, and Type 3 as a high-grade cancer, based on pathological findings.

[0072] In this embodiment, the observation target is the large intestine, and diagnostic information and reference information related to ulcerative colitis are acquired. The diagnostic information generation unit 82 generates diagnostic information by performing AI-based CAD using a first image 72 acquired by IEE, which is a method of illuminating the observation target with a first illumination light, which is a specific illumination light, and capturing the image. The first image 72 is an endoscopic image acquired by IEE, in which superficial blood vessels and the like are emphasized. By using the first image 72, which is acquired by emphasizing the superficial blood vessels and the like of the mucosa of the large intestine, for CAD, the accuracy of predicting the endoscopic severity of ulcerative colitis is often increased. Therefore, by using the first image 72, the diagnostic information generation unit 82 can obtain diagnostic information with high accuracy.

[0073] The diagnostic information can be generated in accordance with various diagnostic techniques, regardless of the classification in the IEE diagnostic techniques. In this embodiment, the diagnostic information and reference information corresponding to the Mayo score, which is a classification based on endoscopic findings of ulcerative colitis and indicates endoscopic severity, are output. Therefore, the diagnostic information generated corresponds to the Mayo score. The Mayo score classifies endoscopic severity into 0, 1, 2, or 3 based on the criteria of endoscopic findings in endoscopic findings of normal images, and is widely used to evaluate the endoscopic severity of ulcerative colitis.

[0074] The Mayo score is used to diagnose the severity of ulcerative colitis. Endoscopic findings such as redness, visible blood vessels, erosion, or ulcers are used as criteria. If the endoscopic findings of the observed subject do not meet the criteria for "redness, visible blood vessels, or erosion," the severity is classified as normal or inactive, i.e., Mayo 0; if the criteria include "redness, decreased visible blood vessels, and mild easy bleeding," the severity is classified as mild, i.e., Mayo 1; if the criteria include "marked redness, loss of visible blood vessels, easy bleeding, and erosion," the severity is classified as moderate, i.e., Mayo 2; and if the criteria include "spontaneous bleeding and ulcers," the severity is classified as severe, i.e., Mayo 3. Therefore, the diagnostic information generator 82 uses the first image 72 to generate diagnostic information that is one of Mayo 0, Mayo 1, Mayo 2, or Mayo 3.

[0075] The diagnostic information generation unit 82 outputs diagnostic information related to the diagnosis of the subject shown in one of multiple types of endoscopic images. While any type of endoscopic image may be used, in this embodiment, the diagnostic information generation unit 82 uses the first image 72 obtained by IEE to generate diagnostic information related to the endoscopic severity of ulcerative colitis based on the Mayo score. For example, information indicating that the endoscopic severity based on the Mayo score is "Mayo 2" is diagnostic information. Therefore, in diagnosis, highly accurate diagnostic information can be generated using the first image 72 obtained by IEE. This diagnostic information is based on the endoscopic severity classification of the Mayo score, which is familiar to physicians, and therefore can facilitate physicians' understanding of the endoscopic severity.

[0076] The diagnostic information generation unit 82 performs AI-based CAD using a first diagnostic model 91. Therefore, the first diagnostic model 91 is a learning model in machine learning. As shown in FIG. 8 , the first diagnostic model 91 is trained and adjusted so that it receives a first image 72, which is an endoscopic image, and outputs diagnostic information 91A. In this embodiment, the first image 72, which is an image of an observation target in the large intestine, is input, and a diagnosis of the endoscopic severity of ulcerative colitis in the Mayo score, for example, "Mayo2," is output as diagnostic information 91A. Therefore, the first image 72, to which a diagnosis result of ulcerative colitis has been previously attached, can be used as training data prior to diagnosis.

[0077] The first diagnostic model 91 is preferably a multi-layer neural network model because it may provide more accurate diagnostic information 91A. Since it is a learning model that inputs an endoscopic image and outputs diagnostic information 91A, it may be a convolutional neural network model or a deep learning model. Furthermore, the first diagnostic model 91 preferably has a layered structure including a first output layer that outputs diagnostic information and at least one first intermediate layer. When the first diagnostic model 91 receives the first image 72, which is an endoscopic image, as input and outputs diagnostic information 91A, it may employ various machine learning techniques to output highly accurate diagnostic information 91A.

[0078] When a medical image depicting the same subject as that depicted in the first image 72 is input, the reference information generation unit 83 generates reference information related to the diagnostic criteria of the diagnostic information 91A generated by the diagnostic information generation unit 82. Medical images depicting the same subject do not necessarily require the subjects depicted in the medical images to be exactly the same, but rather mean that at least a portion of the subject depicted in one medical image is depicted in the other medical image. In other words, when multiple medical images contain a common portion of a subject, these medical images are medical images depicting the same subject. The reference information is information related to the criteria for the diagnosis made by the diagnostic information generation unit 82. The diagnostic information generation unit 82 generates the diagnostic information 91A using CAD, but does not generate information related to the diagnostic criteria. Therefore, the reference information related to the diagnosis made by the diagnostic information generation unit 82 is generated by the reference information generation unit 83.

[0079] The reference information is the standard used when the diagnostic information generating unit 82 acquires, based on the first image 72, diagnostic information 91A related to the diagnosis of the subject appearing in the first image 72. When the diagnostic information generating unit 82 generates and acquires the endoscopic severity according to the Mayo score as the diagnostic information 91A, the reference information is a finding for determining the Mayo score, specifically, information related to redness, loss of a vascular transparency, erosion, ulcer, or the like. The Mayo score determines the severity based on the level of these findings. In this embodiment, in order for the diagnostic information generating unit 82 to perform an endoscopic diagnosis of the severity of ulcerative colitis, the reference information generating unit 83 generates reference information related to the criteria for redness, loss of a vascular transparency, and ulcer in the Mayo score.

[0080] The reference information generating unit 83 generates reference information relating to the diagnostic standard of the diagnostic information 91A using a medical image depicting the same subject as that depicted in the first image 72 and a reference converter 92. The reference converter 92 is created before the diagnosis and is created using added reference information, which is reference information added to the normal image 71 (second medical image). The endoscopic image to which the added reference information is added is an endoscopic image acquired by the medical image acquiring unit 81 and is an endoscopic image of a different type from the first image 72. In this embodiment, the types of endoscopic images are distinguished by the spectrum of the illumination light, so the first image 72 acquired with the first illumination light and the normal image 71 acquired with normal light are endoscopic images of different types.

[0081] In order to create the reference converter 92 before diagnosis, the reference information to be added is acquired in advance before diagnosis. The normal image 71 acquired in advance before diagnosis is added with the reference information to form the finding information, which is then stored in the finding information storage unit 93. The reference converter 92 is created using the finding information stored in the finding information storage unit 93.

[0082] In this embodiment, the reference information is information on redness, loss of vascular visibility, and ulcers, which are findings for determining the Mayo score. Therefore, the normal image 71 to which these reference information have been added is used as the assigned reference information. The assigned reference information can be the doctor's findings on the normal image 71. In other words, the doctor visually inspects the normal image 71 and assigns the degree of the above findings as a subscore.

[0083] As shown in FIG. 9 , the finding information 94 is information in which a doctor evaluates redness, loss of vascular visibility, and ulcers based on the normal image 71 and assigns evaluation values ​​ranging from 0 to 4, with a minimum of 0 and a maximum of 4. The finding information 94 includes the results of the doctor's diagnosis of endoscopic severity based on the normal image 71. For example, for image number "W000001," the "redness" column is entered as "1," the "vascular visibility" column is entered as "0," the "ulcer" column is entered as "0," and the "endoscopic severity" column is entered as "Mayo 1." This records that the doctor's findings for the subject captured in the normal image 71 with image number W000001 are redness level 1, loss of vascular visibility level 0, ulcer level 0, and endoscopic severity level Mayo 1. Note that the type of endoscopic image whose image number begins with W is the normal image 71.

[0084] Next, an endoscopic image of the same subject as that shown in the normal image 71 having the reference information is prepared. From this endoscopic image, features for outputting reference information are acquired using machine learning or the like. Then, a reference converter 92 is created that converts the features into reference information using the features and the reference information. This reference converter 92 can convert the features acquired from the endoscopic image of the same subject as that shown in the normal image 71 having the reference information into items and values ​​similar to the subscores of the Mayo score, such as the finding information 94, and output them. Note that one or more features are acquired. Since the features are converted to reference information, it is preferable that when there are multiple pieces of reference information, there are also multiple features.

[0085] An endoscopic image depicting the same subject as that depicted in a normal image 71 having the attribution reference information may be, for example, a medical image captured at a time close to the time of capture of the normal image 71. It is preferable that the endoscopic images depicting the same subject as that depicted in a normal image 71 having the attribution reference information are captured at times close enough to result in multiple endoscopic images depicting the same subject as the normal image 71. For example, when the frame rate is 60 fps (frames per second), endoscopic images acquired in consecutive frames are considered to almost certainly depict the same subject.

[0086] 10 , when features for obtaining reference information are obtained using an endoscopic image depicting the same subject as that depicted in the normal image 71 used in the finding information 94, the reference information generating unit 83 may include a first reference converter 92X and a first feature model 95. Note that, assuming that the first diagnostic model is the first model, the first feature model and a second feature model described later are second models different from the first model.

[0087] The first feature quantity model 95 is preferably a learning model in machine learning that receives an endoscopic image as input and outputs a feature quantity.

[0088] As shown in FIG. 11 , the first feature quantity model 95 is trained and adjusted so as to input a first image 72, which is an endoscopic image, and output feature quantities such as feature quantity A as numerical values ​​such as a. The first feature quantity model 95 is preferably a multi-layer neural network model. Since it is a learning model that inputs an endoscopic image and outputs feature quantities, it may be a convolutional neural network model or a deep learning model. Furthermore, the first feature quantity model 95 preferably has a layered structure including a second output layer that outputs feature quantities and at least one second intermediate layer.

[0089] In this embodiment, the first feature model 95 preferably inputs a first image 72 depicting the same subject as that appearing in a normal image 71 having the assignment criteria information, and outputs it as a feature relating to a sub-score such as redness, which is a criterion in the Mayo score for ulcerative colitis. Therefore, the first image 72, to which a sub-score such as redness, which is a criterion for ulcerative colitis, has been assigned in advance before diagnosis, can be used as training data.

[0090] It is preferable that the feature quantities output by the first feature quantity model 95 can acquire feature quantities highly correlated with the reference information without any restrictions. When many feature quantities exist, it is possible to effectively select feature quantities, so it is preferable that the first feature quantity model 95 acquires feature quantities from an intermediate layer as an autoencoder or acquires feature quantities by clustering. In this way, when acquiring feature quantities by inputting the first image 72, which is an endoscopic image, the first feature quantity model 95 can employ various machine learning techniques to acquire feature quantities that are highly correlated with the reference information and easy to select.

[0091] As shown in FIG. 12 , the feature values ​​output by the first feature value model 95 can be recorded as information for converter 96 together with findings information 94. In the information for converter 96, the image number beginning with B is the first image 72. The first image 72 with image number B000001 is an endoscopic image of the same subject as the normal image 71 with image number W000001, etc., which has the same image number. Since three types of feature values ​​were acquired, feature value A, feature value B, and feature value C, the information for converter 96 shows feature value A in the "A" column, feature value B in the "B" column, and feature value C in the "C" column. Note that each feature value was adjusted so that its maximum value was 100 and its minimum value was 0.

[0092] For example, for image number "B000001," the converter information 96 has "11" written in the feature "A" column, "5" written in the "B" column, and "0" written in the "C" column, indicating that the features output by the first feature model 95 based on the first image 72 with image number B000001 are feature A: 11, feature B: 5, and feature C: 0. Note that, judging from the image number digits, the endoscopic image with image number B000001 and the like was acquired in the frame immediately before W000001 of normal image 71, which is a different type of endoscopic image, and contains the same subject.

[0093] A first reference converter 92X is created that converts the feature amount into reference information using the subject appearing in the normal image 71 having the reference information and the feature amount and reference information acquired based on an endoscopic image of the same subject. The first reference converter 92X can be created by performing regression analysis on the feature amount and the reference information to associate them with each other.

[0094] For each of the pieces of reference information and the feature quantities, a regression analysis may be performed by matching one piece of reference information with one of the feature quantities most correlated therewith, but to achieve a better correlation, it is preferable to acquire multiple feature quantities and perform regression analysis by matching one piece of reference information with the multiple feature quantities. The regression analysis may be performed using any method that can associate the two with a good correlation, and may use a known mathematical method or a machine learning technique such as a support vector machine.

[0095] As shown in FIG. 13 , a normal image 71 and a first image 72 of the same subject are used. From the first image 72, a first feature model 95 is used to obtain three different feature quantities, A to C, as first feature quantities 95A, including "feature quantity A: a," "feature quantity B: b," and "feature quantity C: c." Here, a, b, and c are numbers or the like indicating quantities. Meanwhile, using the normal image 71, a doctor visually assigns scores to the criteria for "redness," "loss of vascular visibility," and "ulcer," which are subscores of the Mayo score, such as "redness: 2," "loss of vascular visibility: 0," and "ulcer," respectively, and these scores are used as assignment reference information 94A. A regression analysis is performed on the three values ​​of "feature quantity A: a," "feature quantity B: b," and "feature quantity C: c," and, for example, the reference information value "redness: 2." A function is obtained by regression analysis to convert the three values ​​of feature A, feature B, and feature C into a "redness" value of the reference information. This function is stored in the reference converter 92. The "redness" value of the reference information calculated using this function can be used as a subscore of the Mayo score.

[0096] A plurality of first reference converters 92X may be generated for each piece of reference information. In this embodiment, the system includes a first reference converter A92A that converts the three values ​​of feature amount A, feature amount B, and feature amount C into a value of "redness" in the reference information, a first reference converter B92B that converts the three values ​​of feature amount A, feature amount B, and feature amount C into a value of "vascular visibility" in the reference information, and a first reference converter C92C that converts the three values ​​of feature amount A, feature amount B, and feature amount C into a value of "ulcer" in the reference information.

[0097] As described above, the diagnostic information generating unit 82 and the reference information generating unit 83 are generated in advance before diagnosis. Thereafter, during diagnosis, the diagnostic assistance observation mode is activated, and the normal image 71 and the first image 72 are captured during the diagnosis. When the first image 72 is input to the diagnostic information generating unit 82 and the reference information generating unit 83, the diagnostic information generating unit 82 outputs diagnostic information 91A, and the reference information generating unit 83 outputs reference information.

[0098] In the diagnosis assistance observation mode, the information output unit 84 receives diagnostic information 91A from the diagnostic information generation unit 82 and reference information from the reference information generation unit 83, and outputs the diagnostic information 91A in association with the reference information. As shown in FIG. 14 , the diagnostic information 91A is obtained by sending the first image 72 to the diagnostic information generation unit 82 and inputting the first image 72 into the first diagnostic model 91.

[0099] The diagnostic information may be, for example, "Mayo2," which is the diagnosis result of the Mayo score. The reference information 92Y is generated by sending the same first image 72 sent to the diagnostic information generating unit 82 to the reference information generating unit 83, inputting the first image 72 into the first feature model 95 to acquire features. Three types of features, feature D, feature E, and feature F, are acquired, and these features are input into the first reference converter A 92A, the first reference converter B 92B, and the first reference converter C 92C, which convert them into reference information 92Y, such as criteria for "redness," "loss of vascular visibility," and "ulcer," respectively. The reference information 92Y is output as scores such as "redness: 50," "vascular visibility: 0," and "ulcer: 30." The information output unit 84 outputs the diagnostic information 91A and the reference information 92Y in association with each other. The output destination may be the display 15, a recording device (not shown), or the like.

[0100] In the diagnostic assistance observation mode, the information output unit 84 sends diagnostic information 91A and reference information 92Y to the display control unit 85. The display control unit 85 creates a screen displaying the diagnostic information 91A and reference information 92Y and controls the display of the diagnostic information 91A and reference information 92Y on the display 15, for example, by arranging them at predetermined positions. As shown in FIG. 15 , the display 15 displays a normal image 71 of the frame following the frame capturing the first image 72, which is an endoscopic image used by the medical image processing device 17, along with diagnostic information and reference information 97, such as "Diagnostic Information: Mayo 2" and "Reference Information: Redness: 50, Vascular Visibility: 0, Ulcer: 30." When the normal image 71 is displayed on the display 15, the medical image processing device 17 and CAD (Computer Aided Design) may be used in combination to indicate the lesion, and the lesion may be indicated by a lesion area indicator 98 generated using the CAD (Computer Aided Design). As a result, by simply glancing at the display 15, the doctor can proceed with the endoscopic examination by referring to the normal image 71, which is displayed in natural colors that are easy for humans to see, and the Mayo score diagnosis results and diagnostic criteria values ​​calculated by the medical image processing device 17 for the subject shown in the normal image 71.

[0101] The flow of endoscopic image processing of this embodiment by the medical image processing device 17 will be described with reference to the flowchart shown in FIG. 16. First, a first reference converter 92X is generated. To this end, a first image 72 and a normal image 71, which are successively captured, are prepared, and the doctor assigns reference information 92Y to the normal image 71 (step ST110). The first image 72 and the normal image 71 are endoscopic images of the same subject. The reference information 92Y is three criteria, namely, "redness," "loss of vascular visibility," and "ulcer," which are subscores of the Mayo score. The assigned reference information 92Y, such as assigned reference information 94A, is stored in the findings information storage unit 93.

[0102] In the reference information generating unit 83, the first image 72 is input to the first feature model 95 to obtain three types of feature, feature A, feature B, and feature C (step ST120). The three types of feature are stored in the finding information storage unit 93, and a regression analysis is performed to associate the three types of feature with one of the reference information 92Y (step ST130). As a result, a first reference converter 92X is generated that converts the three types of feature into one of the reference information 92Y. Since the first reference converter 92X is generated for each piece of reference information 92Y, three first reference converters 92 are generated: a first reference converter A 92A that converts the three types of feature into a reference for "redness," a first reference converter B 92B that converts the three types of feature into a reference for "vascular visibility," and a first reference converter C 92C that converts the three types of feature into a reference for "ulcer" (step ST140).

[0103] Next, the endoscopic examination is started in the diagnosis assistance observation mode, and the first image 72 is acquired (step ST150). The medical image acquisition unit 81 receives the first image 72 and transmits it to the diagnostic information generation unit 82 and the reference information generation unit 83 (step ST160). In the diagnostic information generation unit 82, the first diagnostic model 91 receives the first image 72 and outputs diagnostic information 91A (step ST170). In the reference information generation unit 83, the first feature amount model 95 receives the first image 72 and obtains three types of feature amounts, feature amount D, feature amount E, and feature amount F (step ST180). By inputting the three types of features, feature D, feature E, and feature F, into the first reference converter A92A, first reference converter B92B, and first reference converter C92C, respectively, the value of "redness," which is a subscore of the Mayo score, is output from the first reference converter A92A, the value of "vascular visibility" is similarly output from the first reference converter B92B, and the value of "ulcer" is similarly output from the first reference converter C92C as reference information 92Y (step ST190).

[0104] The output diagnostic information and reference information 97 are received by the information output unit 84 (step ST200). The information output unit 84 associates the diagnostic information and reference information 97 and outputs them to the display control unit 85 (step ST210). The display control unit 85 performs control to display the diagnostic information and reference information 97 on the display 15. The display 15 displays the normal image 71 acquired by endoscopic examination and the diagnostic information and reference information 97 output by the medical image processing device 17 for the subject appearing in this normal image 71 (step ST220).

[0105] As described above, the medical image processing device 17 obtains diagnostic information 91A by using an endoscopic image obtained by IEE to obtain highly accurate diagnostic information that cannot be obtained with a normal image 71, while obtaining reference information 92Y by using an endoscopic image obtained by IEE but obtaining reference information linked to the normal image 71 obtained under normal light, which is commonly seen by doctors, thereby achieving both high diagnostic accuracy and interpretability of the diagnostic results. Furthermore, if the first reference converter 92X is configured to convert one piece of reference information 92Y using multiple feature amounts, reference information 92Y with higher accuracy can be obtained.

[0106] The diagnostic information generating unit 82 may acquire the first feature amount 95A of the first image 72, and acquire the diagnostic information 91A based on the first feature amount 95A. Since the diagnostic information generating unit 82 generates the diagnostic information 91A of the first image 72 using the first diagnostic model 91, the first feature amount 95A may be generated in the first diagnostic model 91.

[0107] The first diagnostic model 91 that generates the first feature 95A may be any model capable of acquiring the first feature 95A, such as a model capable of acquiring the first feature 95A from an intermediate layer of the first diagnostic model 91. Since the first feature 95A can be appropriately acquired, a model similar to the first feature model 95 can be employed. Furthermore, since the first diagnostic model 91 can effectively select features when many feature values ​​exist, it is preferable that the first diagnostic model 91 acquires the feature values ​​from an intermediate layer as an autoencoder or acquires the feature values ​​by clustering. When acquiring diagnostic information 91A by inputting the first image 72, which is an endoscopic image, the first diagnostic model 91 may employ various machine learning techniques to output highly accurate diagnostic information 91A and acquire features that are easy to select.

[0108] Furthermore, when the first diagnostic model 91 generates the first feature 95A, the reference converter 92 generates it using the assigned reference information 94A and the first image 72 depicting the same subject as the subject appearing in the normal image 71 to which the assigned reference information 94A is attached, and the reference information generation unit 83 may acquire the first feature 95A and convert the first feature 95A using the reference converter 92 to acquire the reference information 92Y.

[0109] 17, in this case, the reference information generation unit 83 includes a first feature acquisition unit 100, which acquires the feature generated by the first diagnostic model 91. The first diagnostic model 91 may generate multiple feature amounts. The reference converter 92 converts one or more feature amounts acquired from the first diagnostic model 91 into reference information 92Y.

[0110] 18, the first diagnostic model 91 outputs diagnostic information 91A based on the first image 72, and in doing so generates a first feature 95A. For example, the first diagnostic model 91 has a first output layer that outputs the diagnostic information 91A and a first hidden layer, where the first hidden layer is arranged before the first output layer and calculates the first feature 95A. The first output layer outputs the diagnostic information 91A based on the first feature 95A. In this embodiment, the first diagnostic model 91 generates three types of first feature 95A: a feature G, a feature H, and a feature I.

[0111] Note that when the first feature quantity model 95 has a layered structure including a second output layer that outputs features and at least one second hidden layer, and the first diagnostic model 91 has a layered structure including a first output layer that outputs features and at least one first hidden layer, the second hidden layer may share the first hidden layer. Even with this method, the first feature quantity model 95 can generate three types of feature quantities: feature quantity G, feature quantity H, and feature quantity I.

[0112] The reference information generating unit 83 inputs the feature G, feature H, and feature I acquired by the first feature acquiring unit 100 into a first reference converter A92A, a first reference converter B92B, and a first reference converter C92C, respectively, and converts them into three types of reference information 92Y. As described above, the first reference converter A92A generates, as reference information 92Y, a value for "redness" as a subscore in the Mayo score, the first reference converter B92B similarly generates, as reference information 92Y, a value for "vascular visibility," and the first reference converter C92C similarly generates, as reference information 92Y, a value for "ulcer."

[0113] Similarly to the above, the diagnostic information and reference information 97 are sent to the information output unit 84 and displayed on the display 15 by the display control unit 85. As shown in FIG. 19 , the basis image indicator 99 may indicate which type of endoscopic image was input into CAD to obtain the diagnostic information 91A or the reference information 92Y. For example, when the basis image indicator 99 displays "B," this indicates that the value was obtained by CAD using the first image 72. In this embodiment, both the diagnostic information and the reference information 97 are obtained by CAD using the first image 72, and therefore "B" is displayed. Note that when the value is obtained by CAD using the input normal image 71, the basis image indicator 99 displays "W."

[0114] As described above, in the diagnostic information generating unit 82, the first diagnostic model 91 acquires the first feature amount 95A based on the first image 72, and the reference information generating unit 83 can acquire the reference information 92Y using the first feature amount 95A generated by the first diagnostic model 91, and the first feature amount 95A can be calculated only once, which is particularly preferable in terms of saving computational resources. Furthermore, since the reference information generating unit 83 acquires the reference information 92Y using the first feature amount 95A acquired by the first diagnostic model 91 based on the first image 72, the diagnostic information 91A and the reference information 92Y are linked, and the reference information 92Y is used as the diagnostic information 91A. A This can be considered as a diagnostic basis.

[0115] In the diagnosis assistance observation mode, the reference information generating unit 83 may acquire features (hereinafter referred to as second features) of the normal image 71 (second medical image) and generate the reference information 92Y by converting the second features using a second reference converter 120 that converts the second features into reference information 92Y. In this case, the second reference converter 120 generates the reference information 92Y using the added reference information 94A and the normal image 71. The reference information generating unit 83 then generates second features of the normal image 71, which depicts the same subject as that depicted in the first image 72, and generates the reference information 92Y by converting the second features into reference information 92Y using the second reference converter 120. The normal image 71 from which the second features are acquired and the first image 72 from which the diagnostic information 91A are acquired are endoscopic images depicting the same subject.

[0116] 20 , in this case, the reference information generation unit 83 includes a second reference converter 120 and a second feature model 110. In the diagnosis assistance observation mode, the medical image acquisition unit 81 sends a first image 72 to the diagnostic information generation unit 82, and sends a normal image 71 captured immediately after the first image 72 to the reference information generation unit 83.

[0117] 21, in the diagnosis assistance observation mode, the diagnostic information generation unit 82 acquires diagnostic information 91A based on the first image 72. Details are as described above. The second feature model 110 is preferably a machine learning learning model that outputs feature amounts by inputting an endoscopic image.

[0118] The second feature model 110 is trained and adjusted to input a normal image 71 and output a feature. The second feature model 110 is preferably a multi-layer neural network model. Since it is a learning model that inputs an endoscopic image and outputs a feature, it may be a convolutional neural network model or a deep learning model.

[0119] The second feature model 110 preferably inputs a normal image 71 depicting the same subject as that depicted in the normal image 71 having the assignment reference information 94A, and outputs features relating to sub-scores such as redness, which are criteria in the Mayo score for ulcerative colitis. Therefore, the normal image 71 to which sub-scores such as redness, which are criteria for ulcerative colitis, have been assigned in advance before diagnosis, can be used as training data.

[0120] It is preferable that the feature quantities output by the second feature quantity model 110 can acquire feature quantities highly correlated with the reference information 92Y without any restrictions. When many feature quantities exist, it is possible to effectively select feature quantities. Therefore, it is preferable that the second feature quantity model 110 acquires feature quantities from an intermediate layer as an autoencoder or acquires feature quantities by clustering. In this way, when inputting the normal image 71 to acquire feature quantities, the second feature quantity model 110 can employ various machine learning techniques to acquire feature quantities that are highly correlated with the reference information 92Y and are easy to select.

[0121] The feature amount obtained from the second feature amount model 110 may be one or more. In this embodiment, the second feature amount model outputs three types of feature amounts: a feature amount J, a feature amount K, and a feature amount L. Each feature amount is converted into reference information 92Y by a second reference converter A 120A, a second reference converter B 120B, and a second reference converter C 120C, which are respectively provided.

[0122] The second reference converter A120A, the second reference converter B120B, and the second reference converter C120C are generated from the attached reference information 94A and the feature amounts of the normal image 71. Unlike the first reference converter 92X, which is generated using the feature amounts of the first image 72, the second reference converter 120 is generated using the feature amounts of the normal image 71. In other respects, the second reference converter 120 can be similar to the first reference converter 92X.

[0123] In this case, it is preferable that the reference information generating unit 83, in the diagnosis assistance observation mode, acquires the second feature amount of the normal image 71 (second medical image) captured within a predetermined period before and / or after the time when the first image 72 was acquired. By setting the above-mentioned period, it is possible to ensure that the normal image 71 and the second medical image almost certainly depict the same subject.

[0124] The preset period can be set, for example, as follows: In this embodiment, when normal light and the first illumination light are automatically switched between in one cycle of 60 fps in the diagnosis assistance observation mode, for example, a normal image 71, which is the second medical image, is acquired in the first 40 frames, and a first image 72, which is the first medical image, is acquired in the latter 20 frames. During observation in the diagnosis assistance observation mode, endoscopic images are captured in the first cycle (1 second has elapsed), the second cycle (2 seconds has elapsed), the third cycle (3 seconds has elapsed), ..., the Xth cycle (X seconds has elapsed), where X is a positive integer.

[0125] If the time at which the first image 72 was acquired is the time at which one frame of the last 20 frames of the Xth cycle was acquired, the value n of the Xnth cycle is set to be within a predetermined period before the time at which the first image 72 was acquired, and any one normal image 71 frame of the first 40 frames of the Xnth cycle is adopted as the second medical image. Similarly, the value m of the X+mth cycle is set to be within a predetermined period after the time at which the first image 72 was acquired, and any one normal image 71 frame of the first 40 frames of the X+mth cycle is adopted as the second medical image. Note that n and m are positive integers. n or m can be set in advance by the doctor before starting the examination.

[0126] The normal image 71 of any one of the first 40 frames may be selected as follows. For example, the normal image 71 of the first 40 frames may be the frame with the best image quality, a randomly selected frame, or a frame selected according to a preset rule. The frame with the best image quality may be detected as a frame free of blur, blisters, halation, or the like using machine learning or image processing. Another preset rule may be, for example, to assign the normal image 71 the same frame number as the frame number at which the first image 72 was acquired. Specifically, if the first image 72 is the fifth frame captured among the first images 72 of the last 20 frames in an X cycle, the normal image 71 may also be the fifth frame captured among the normal images 71 of the first 40 frames in the same X cycle.

[0127] The reference information generating unit 83 may also acquire the second feature amount of a second medical image captured consecutively with the normal image 71. This is because the images are captured consecutively and therefore almost certainly depict the same subject.

[0128] As described above, the diagnostic information and reference information 97 are sent to the information output unit 84 and displayed on the display 15 by the display control unit 85. As shown in Fig. 22, a basis image indicator 99 may be used to indicate which type of endoscopic image was input into CAD to obtain the diagnostic information 91A or the reference information 92Y. In this embodiment, the diagnostic information 91A is obtained by CAD using the first image 72, and the reference information 92Y is obtained by CAD using the normal image 71, so "B" is displayed for the diagnostic information 91A, and "W" is displayed for the reference information 92Y.

[0129] As described above, for the purpose of obtaining diagnostic information 91A, by using endoscopic images from IEE, highly accurate diagnostic information that cannot be obtained with normal images 71 can be obtained, while for the purpose of obtaining reference information 92Y, by obtaining reference information 92Y linked to normal images 71 that doctors are accustomed to seeing, both high diagnostic accuracy and interpretability of the diagnostic results can be obtained.

[0130] In the above embodiment, the present invention is applied to the processing of endoscopic images, but the present invention can also be applied to medical image processing devices that process medical images other than endoscopic images, as well as endoscopic systems, etc.

[0131] In addition, Inside A part or all of the image processing unit 55 and / or the central control unit 58 of the endoscope system 10 may be controlled by the endoscope 12, for example, directly from the endoscope system 10 or indirectly from a PACS (Picture Archiving and Communication Systems) 22. The medical image processing device 17 of the endoscope system 10 may be provided in a diagnosis support device 610 that acquires the captured images. Similarly, a part or all of the medical image processing device 17 of the endoscope system 10 may be provided in the diagnosis support device 610 that acquires the images captured by the endoscope 12, for example, directly from the endoscope system 10 or indirectly from a PACS (Picture Archiving and Communication Systems) 22.

[0132] Also, Inside A medical work support device 630 connected via a network 626 to various inspection devices including the endoscopic system 10, such as a first inspection device 621, a second inspection device 622, ..., an Nth inspection device 623, can be provided with part or all of the image processing unit 55 and / or central control unit 58 of the endoscopic system 10, or part or all of the medical image processing device 17.

[0133] In the above embodiment, the hardware structures of the processing units that perform various processes, such as the central control unit 58, image acquisition unit 51, DSP 52, noise reduction unit 53, memory 54, image processing unit 55, display control unit 56, and video signal generation unit 57 included in the processor device 14 including the light source processor and the first processor, and the medical image acquisition unit 81, diagnostic information generation unit 82, reference information generation unit 83, information output unit 84, and display control unit 85 included in the medical image processing device 17 including the second processor, are various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, which is a processor having a circuit configuration specifically designed to perform various processes.

[0134] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which one processor is configured with a combination of one or more CPUs and software, as typified by client or server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a System on Chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0135] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit in the form of a combination of circuit elements such as semiconductor elements. [Explanation of symbols]

[0136] 10 Endoscopy System 12 Endoscopy 12a Insertion part 12b Operation section 12c curved section 12d Tip 12e Angle Knob 12f Zoom control 12g mode switch 13 Light source device 14 Processor unit 15 Display 16 keyboards 17 Medical image processing equipment 20 Light source section 20a V-LED 20b B-LED 20c G-LED 20d R-LED 21 Light Source Processor 22 PACS 30a illumination optical system 30b Imaging optical system 41 Light Guide 42 Lighting lens 43 Objective Lens 44 Zoom Lens 45 imaging sensor 46 CDS / AGC circuit 47 A / D converter 51 Image acquisition unit 52 DSP 53 Noise reduction section 54 memory 55 Image processing section 56, 85 Display control unit 57 Video signal generator 58 Central Control Unit 61 Normal image processing section 62 Special image processing section 71 Normal Images 72 Image 1 81 Medical Image Acquisition Department 82 Diagnostic Information Generation Unit 83 Standard information generation section 84 Information output section 91 First diagnostic model 91A Diagnostic Information 92 Reference Converter 92X 1st Reference Converter 92A First reference converter A 92B 1st reference converter B 92C 1st reference converter C 92Y standard information 93 Finding information storage unit 94 Findings 94A Grant Criteria Information 95 Model for the first feature 95A First feature 96 Transducer Information 97 Diagnostic and Reference Information 98 Lesion Area Indicator 99 Evidence Image Indicator 100 First feature acquisition unit 110 Second feature model 110A Second feature 120 Second reference converter 120A Second reference converter A 120B Second reference converter B 120C 2nd reference converter C 610 Diagnostic support device 621 First Inspection Device 622 Second Inspection Device 623 Nth Inspection Device 626 Network 630 Medical Business Support Devices ST110~ST220 Step

Claims

1. a processor; The processor: acquiring a plurality of types of medical images of a subject taken under different imaging conditions; When a first medical image, which is one of the plurality of types of medical images, is input, diagnostic information regarding a diagnosis of the subject appearing in the first medical image is generated; generating reference information relating to the diagnostic criteria when the medical image showing the same subject as the subject shown in the first medical image is input; outputting the diagnostic information and the reference information in association with each other; the reference information is generated by a reference converter created using assigned reference information that is included in the medical image and is assigned to a second medical image of a different type from the first medical image; a medical image processing device that outputs a basis image indicator that indicates from which type of medical image the reference information was obtained;

2. The medical image processing device described in claim 1, wherein the processor generates the reference information using a reference converter created using the assignment reference information and the medical image depicting the same subject as the subject depicted in the second medical image to which the assignment reference information is attached.

3. when the medical image showing the same subject as the subject shown in the first medical image is input, the processor acquires a feature amount of the input medical image; The medical image processing apparatus according to claim 1 or 2, wherein the reference information is generated by converting the feature amount into the reference information.

4. When the medical image showing the same subject as the subject shown in the first medical image is input, the processor acquires a plurality of feature amounts of the input medical image; The medical image processing apparatus according to claim 1 or 2, wherein the reference information is generated by converting a plurality of the feature amounts into at least one piece of the reference information.

5. the processor includes a first model for generating the diagnostic information; The medical image processing apparatus according to claim 1 , wherein the first model has a layered structure including a first output layer that outputs the diagnostic information and at least one first intermediate layer.

6. When the medical image showing the same subject as the subject shown in the first medical image is input, the processor acquires the feature amount of the input medical image, the processor includes a second model that generates the feature quantity; 6. The medical image processing device according to claim 5, wherein the second model has a layered structure including a second output layer that outputs the feature amount and at least one second intermediate layer, and the second intermediate layer shares the first intermediate layer.

7. 7. The medical image processing apparatus according to claim 1, wherein the assigned reference information is the reference information assigned to the second medical image by a doctor by visually inspecting the second medical image.

8. The medical image processing device according to any one of claims 1 to 7, wherein the processor, when the first medical image is input, acquires a first feature of the first medical image and generates the diagnostic information based on the first feature.

9. The medical image processing device described in claim 8, wherein the processor generates the reference information by converting the first feature into the reference information using a first reference converter created using the assignment reference information and the first medical image that depicts the same subject as the subject that appears in the second medical image to which the assignment reference information is attached, as the medical image that depicts the same subject as the subject that appears in the first medical image.

10. when the second medical image, which shows the same subject as the subject shown in the first medical image, is input as the medical image showing the same subject as the subject shown in the first medical image, the processor acquires a second feature amount of the second medical image; A medical image processing device according to any one of claims 1 to 8, wherein the reference information is generated by converting the second feature into the reference information using a second reference converter created using the assigned reference information and the second medical image.

11. The medical image processing device according to claim 10 , wherein the processor acquires the second feature of the second medical image taken within a predetermined period before and / or after the time when the first medical image was taken.

12. 12. The medical image processing apparatus according to claim 1, wherein the processor controls displaying the first medical image and / or the second medical image on a display.

13. The medical image processing apparatus according to claim 12 , wherein the processor controls displaying the associated diagnostic information and reference information on the display.

14. 14. The medical image processing apparatus according to claim 1, wherein the imaging condition is a spectrum of illumination light.

15. a plurality of light sources that emit light in different wavelength bands; a processor device including a light source processor that controls the emission of each of a plurality of types of illumination light having different combinations of light intensity ratios of the plurality of light sources; an endoscope that captures an image of the subject illuminated by the illumination light; An endoscope system comprising the medical image processing device according to any one of claims 1 to 14.

Images