Medical image processing system

Abstract

The present invention provides a medical image processing system that makes it possible to grasp information related to diseases under geometric or spatial conditions.　A three-dimensional structure information acquisition unit (50) acquires three-dimensional structure information indicative of a three-dimensional structure of a hollow organ. A disease area information acquisition unit (51) acquires disease area information indicative of a disease area in the three-dimensional structure of the hollow organ from a plurality of medical images. A disease evaluation information acquisition unit (52) acquires disease evaluation information indicative of an evaluation of a disease in the disease area. A disease-related information output unit (53) outputs the three-dimensional structure information, the disease area information, and the disease evaluation information in association with each other.

Description

Medical Image Processing System

[0001] The present invention relates to a medical image processing system for obtaining disease-related information from medical images.

[0002] In endoscopic diagnoses of various digestive system diseases, written reports are prepared, but there is a problem in that it is difficult to express geometric information such as the location of a tumor or an area of ​​inflammation in written form. Regarding the display of geometric information, Patent Document 1 discloses that when the endoscope moves inside a lumen (for example, a bronchi) during endoscopic diagnosis, the movement or path of the endoscope is displayed on a schematic diagram of the lumen, and the position of the lesion corresponding to the target site is also displayed.

[0003] JP 2015-107268 A

[0004] In recent years, in the field of endoscopy, for example, in the case of tumor diseases, the distance from the anus to the tumor is sometimes recorded in the report. However, even when referring to the previous report at the time of the next diagnosis, it is sometimes difficult to identify the tumor location and find it. Furthermore, in the case of inflammatory diseases, it is difficult to grasp spatial information such as the location or area of ​​inflammation as well as the degree or severity of inflammation from the report. Therefore, there has been a demand for a system that allows disease information such as tumor location, degree or severity of inflammation to be grasped in a geometric or spatial context.

[0005] An object of the present invention is to provide a medical image processing system that can grasp information about a disease in a geometric or spatial context.

[0006] The medical image processing system of the present invention includes a processor that acquires multiple medical images of a luminal organ taken from inside the lumen, acquires three-dimensional structural information showing the three-dimensional structure of the luminal organ, acquires disease area information showing the disease area in the three-dimensional structure of the luminal organ, and disease evaluation information showing the disease evaluation of the disease area from the multiple medical images, and outputs the three-dimensional structural information, disease area information, and disease evaluation information in association with each other.

[0007] Preferably, the processor acquires a first illumination light image and a second illumination light image as the plurality of medical images, acquires three-dimensional structure information from the first illumination light image, acquires disease region information from the first illumination light image or the second illumination light image, and acquires disease evaluation information from the second illumination light image. Preferably, the first illumination light image and the second illumination light image are acquired by successive capture, and the wavelength band of the first illumination light used to generate the first illumination light image is wider than the wavelength band of the second illumination light used to generate the second illumination light image. Preferably, the processor acquires the disease region information using the successive first illumination light image and the second illumination light image.

[0008] It is preferable that the processor generates a base image and a texture image from one frame of medical image, obtains three-dimensional structural information from the base image, obtains disease area information from the base image or the texture image, and obtains disease evaluation information from the texture image.

[0009] The processor preferably acquires the three-dimensional structural information using at least one of a plurality of medical images and an extraluminal image obtained by photographing the hollow organ from outside the lumen, and preferably acquires the three-dimensional structural information by interpolating the extraluminal image for a portion of the hollow organ for which no medical image exists.

[0010] The processor preferably displays on the display a three-dimensional structure in which the position of the diseased area is indicated by a disease display marker. The processor preferably displays, within the three-dimensional structure displayed on the display, areas of the body part not photographed by the multiple medical images, distinguishing them from areas of the body part for which medical images are present. In addition to displaying the three-dimensional structure, the display preferably displays a moving image based on the medical images. The three-dimensional structure information is preferably a predetermined schematic image.

[0011] When the processor stores the three-dimensional structural information, the disease evaluation information, and the three-dimensional structural information set in which the disease evaluation information is associated with each other in the disease-related information memory, it is preferable that the processor stores the three-dimensional structural information set from the first examination in association with the three-dimensional structural information set from the second examination that is performed after the first examination in the disease-related information memory. It is preferable that the processor display the three-dimensional structural information set from the first examination and the three-dimensional structural information set from the second examination on the same screen of the display.

[0012] It is preferable that the processor obtains difference information representing the difference between the disease assessment in the disease area at the time of the first examination and the disease assessment in the disease area at the time of the second examination from the three-dimensional structural information set at the time of the first examination and the three-dimensional structural information set at the time of the second examination, and displays the difference information on the three-dimensional structural information.

[0013] When the three-dimensional structure information is displayed on the display, it is preferable that when a user designates a specific region of the three-dimensional structure, disease region information and disease evaluation information corresponding to the specific region are displayed on the display. The disease is preferably an inflammatory disease or a neoplastic disease.

[0014] According to the present invention, information about a disease can be grasped in a geometric or spatial context.

[0015] 1 is a schematic diagram of an endoscope system. FIG. 1 is a block diagram showing the functions of the endoscope system. FIG. 2 is an explanatory diagram of a first light emission mode. FIG. 3 is an explanatory diagram of a third light emission mode. FIG. 4 is a block diagram showing the functions of an image processing unit. FIG. 4 is an explanatory diagram showing acquisition of three-dimensional structural information from a first illumination light image. FIG. 5 is an explanatory diagram showing interpolation using an extraluminal image. FIG. 6 is an explanatory diagram showing acquisition of three-dimensional structural information and disease region information from first illumination light images obtained by continuous imaging. FIG. 7 is an explanatory diagram showing a disease display marker representing a disease region in a three-dimensional structure. FIG. 8 is an explanatory diagram showing an ellipse representing a disease region in a three-dimensional structure. FIG. 9 is an explanatory diagram showing three-dimensional structural information in which areas where a medical image is present and areas where it is not present are displayed separately. FIG. 10 is an image diagram of a display displaying three-dimensional structural information during diagnosis. FIG. 11 is an image diagram of a display displaying three-dimensional structural information after completion of diagnosis. FIG. 12 is an image diagram of a display showing three-dimensional structural information at the time of the first examination and the second examination. FIG. 13 is an image diagram of a display displaying difference information. FIG. 14 is an image diagram showing an information display area related to a specific region SPR. FIG. 15 is an image diagram showing an information display area related to a specific region SPR. FIG. 16 is an explanatory diagram showing acquisition of three-dimensional structural information, disease region information, and disease evaluation information from one frame of a medical image.

[0016] 1, an endoscopic system 10 includes an endoscope 12, a light source device 13, a processor device 14, a display 15, and a user interface 16. The endoscope 12 is optically or electrically connected to the light source device 13, and is also electrically connected to the processor device 14. The endoscopic system 10 corresponds to the medical image processing system of the present invention, which processes images obtained by the endoscope 12 as medical images.

[0017] The endoscope 12 has an insertion section 12a, an operating section 12b, a bending section 12c, and a tip section 12d. The insertion section 12a is inserted into the body of the subject. The operating section 12b is provided at the base end of the insertion section 12a. The bending section 12c and the tip section 12d are provided on the tip side of the insertion section 12a. The bending section 12c is bent by operating the angle knob 12e of the operating section 12b. The tip section 12d is directed in a desired direction by the bending of the bending section 12c. A forceps channel (not shown) is provided from the insertion section 12a to the tip section 12d for inserting a treatment tool or the like. The treatment tool is inserted into the forceps channel through the forceps port 12j.

[0018] The endoscope 12 is provided with an optical system for forming an image of a subject and an optical system for irradiating the subject with illumination light. The operation unit 12b is provided with an angle knob 12e, a mode selector switch 12f, a still image acquisition instruction switch 12h, and a zoom operation unit 12i. The mode selector switch 12f is used to switch the observation mode. The still image acquisition instruction switch 12h is used to instruct acquisition of a still image of the subject. The zoom operation unit 12i is used to enlarge or reduce the observation target.

[0019] The light source device 13 generates illumination light. The processor device 14 performs system control of the endoscope system 10 and generates medical images by performing image processing on image signals transmitted from the endoscope 12. The display 15 displays the medical images transmitted from the processor device 14. The user interface 16 has a keyboard, mouse, microphone, tablet, touch pen, etc., and accepts input operations such as function settings.

[0020] 2, the light source device 13 includes a light source unit 20 and an optical path coupling unit 22. The light source unit 20 has multiple semiconductor light sources, each of which is turned on or off. When the multiple semiconductor light sources are turned on, the light emission amount of each semiconductor light source is controlled to emit illumination light that illuminates the subject. The light source unit 20 has four color LEDs: a V-LED (Violet Light Emitting Diode) 20a, a B-LED (Blue Light Emitting Diode) 20b, a G-LED (Green Light Emitting Diode) 20c, and an R-LED (Red Light Emitting Diode) 20d. The light source unit 20 may be built into the endoscope 12.

[0021] The V-LED 20a emits violet light V with a central wavelength of 405±10 nm and a wavelength range of 380 to 420 nm. The B-LED 20b emits blue light B with a central wavelength of 450±10 nm and a wavelength range of 420 to 500 nm. The G-LED 20c emits green light G with a wavelength range of 480 to 600 nm. The R-LED 20d emits red light R with a central wavelength of 620 to 630 nm and a wavelength range of 600 to 650 nm.

[0022] The light emitted by each of the LEDs 20a to 20d is incident on a light guide 23 via an optical path coupling unit 22 composed of a mirror, a lens, etc. The light guide 23 propagates the light from the optical path coupling unit 22 to the tip 12d of the endoscope 12.

[0023] An illumination optical system 30 and an imaging optical system 32 are provided at the tip 12d of the endoscope 12. The illumination optical system 30 has an illumination lens 31, and illumination light propagated by the light guide 23 is irradiated onto the subject via the illumination lens 31. On the other hand, when the light source unit 20 is built into the tip 12d of the endoscope 12, light is emitted toward the subject via the illumination lens of the illumination optical system without passing through a light guide.

[0024] The imaging optical system 32 has an objective lens 35, a zoom lens 36, and an imaging sensor 37. Light from the subject irradiated with illumination light is incident on the imaging sensor 37 via the objective lens 35 and the zoom lens 36. As a result, an image of the subject is formed on the imaging sensor 37. The zoom lens 36 is a lens for enlarging the subject, and is moved between the telephoto end and the wide-angle end by operating the zoom operation unit 12i.

[0025] The image sensor 37 is a primary color sensor and has three types of pixels: B pixels (blue pixels) with blue color filters, G pixels (green pixels) with green color filters, and R pixels (red pixels) with red color filters.

[0026] The imaging sensor 37 is preferably a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). In this embodiment, a CMOS type imaging sensor is used as the imaging sensor 37, but a CCD type imaging sensor may also be used. The imaging processor 38 controls the imaging sensor 37. Specifically, the imaging processor 38 reads out the signal from the imaging sensor 37, causing the imaging sensor 37 to output an image signal. The output image signal is sent to the processor unit 14.

[0027] The processor device 14 has a medical image acquisition unit 40, an image processing unit 41, a display control unit 42, and a central control unit 43. In the processor device 14, the central control unit 43, which is made up of a processor, runs a program in a program memory (not shown), thereby realizing the functions of the medical image acquisition unit 40, the image processing unit 41, and the display control unit 42. In addition, the functions of the three-dimensional structure information acquisition unit 50, the disease region information acquisition unit 51, the disease evaluation information acquisition unit 52, the disease-related information output unit 53, the difference information acquisition unit 57, and the specific region information output unit 59 (see FIG. 6 ), which are included in the image processing unit 41, are realized.

[0028] The medical image acquisition unit 40 acquires image signals from the endoscope 12 as image signals for medical images. The image processing unit 41 performs various signal processing on the image signals acquired by the medical image acquisition unit 40, such as defect correction, offset processing, demosaic processing, matrix processing, white balance adjustment, gamma conversion, and YC conversion. Next, a color medical image is generated by performing image processing including color conversion processing such as 3x3 matrix processing, tone conversion processing, and 3D LUT (Look Up Table) processing, and structure enhancement processing such as color enhancement and spatial frequency enhancement. In addition to generating color medical images, the image processing unit 41 also acquires three-dimensional structural information, disease region information, and disease evaluation information. The acquisition of this three-dimensional structural information will be described later. The display control unit 42 displays various information, including the medical images generated by the image processing unit 41, on the display 15.

[0029] Next, details of the light emission control will be described. The endoscope system 10 has a mono-light emission mode and a multi-light emission mode as light emission modes for emitting illumination light. By operating the mode selector switch 12f, the light emission mode can be switched via the central control unit 43. The mono-light emission mode is a mode in which illumination light of the same spectrum is continuously emitted to illuminate the subject to be observed. The multi-light emission mode is a mode in which multiple illumination lights of different spectrums are emitted while being switched according to a specific pattern to illuminate the subject.

[0030] The illumination light includes a first illumination light L1 and a second illumination light L2 having a spectrum different from that of the first illumination light. The first illumination light L1 is preferably used to provide brightness to the entire subject for screening observation. The second illumination light L2 is preferably used to highlight specific structures of the subject, such as glandular ducts and blood vessels in the mucosa. In the mono-emission mode, either the first illumination light or the second illumination light is emitted. In the multi-emission mode, the first illumination light and the second illumination light are emitted while being switched according to a specific pattern.

[0031] The first illumination light L1 is preferably a broadband light such as white light. The second illumination light L2 preferably includes, for example, second illumination light L2SP for emphasizing superficial blood vessels, second illumination light L2SQ for emphasizing very superficial blood vessels that are shallower than superficial blood vessels, second illumination light L2SR for generating an oxygen saturation image utilizing the difference in the absorption coefficients of oxyhemoglobin and deoxyhemoglobin, and second illumination light L2SS for generating a color-difference-enhanced image in which the color difference between multiple subject ranges is enhanced. These four types of second illumination light L2SP, L2SQ, L2SR, and L2SS each have a different spectrum.

[0032] The light source device 13 independently controls the light intensity of the four colors of violet light V, blue light B, green light G, and red light R, and emits the first illumination light L1 or the second illumination light L2 (e.g., second illumination light L2SP, second illumination light L2SQ, second illumination light L2SR, and second illumination light L2SS) by changing the light intensity. The light emission control of the light source device 13 in the mono emission mode and the multi emission mode is performed by a light source processor (not shown).

[0033] In the mono illumination mode, illumination light of the same spectrum is continuously emitted for each frame. For example, a first illumination light image is displayed on the display 15 by illuminating a subject with a first illumination light for each frame and capturing an image. Also, a second illumination light image is displayed on the display 15 by illuminating a subject with a second illumination light for each frame and capturing an image. Note that a frame refers to a unit of time that includes at least the period from the time of light emission to the time when the image sensor 37 completes reading of the image signal.

[0034] In the multi-light emission mode, the LEDs 20a to 20d are controlled to automatically switch between the first illumination light and the second illumination light according to a specific light emission pattern. Specifically, the light intensities of the purple light V, blue light B, green light G, and red light R are controlled to change for each specific frame F according to the specific light emission pattern.

[0035] Examples of light emission patterns are given below. For example, in a first light emission pattern, as shown in Fig. 3, two frames of first illumination light L1 are emitted during a first light emission period Pe1 in which the subject is illuminated with the first illumination light L1, and one frame of second illumination light L2 is emitted during a second light emission period Pe2 in which the subject is illuminated with the second illumination light L2. In the first light emission pattern, the same second illumination light L2SP is emitted during each second light emission period Pe2.

[0036] In the figure, the arrow indicates the direction in which time progresses. In the first light emission pattern, two frames of a first illumination light image P1 are obtained in the first light emission period Pe1, and one frame of a second illumination light image P2SP is obtained in the second light emission period Pe2. In the first light emission pattern, first illumination light L1 with different spectra may be emitted in each first light emission period Pe1 (the same applies to the second light emission pattern and third light emission pattern described below).

[0037] In the second light emission pattern, as shown in FIG. 4 , the first illumination light L1 is emitted for two frames during the first light emission period Pe1, and the second illumination light L2 is emitted for one frame during the second light emission period Pe2. This pattern is repeated. In each second light emission period Pe2, the second illumination light L2 having a different spectrum is emitted. Specifically, the second illumination light L2SP and the second illumination light L2SQ are emitted alternately during the second light emission period Pe2. In the second light emission pattern, two frames of the first illumination light image P1 are obtained during the first light emission period Pe1, and the second illumination light image P2SP and the second illumination light image P2SQ are obtained during each second light emission period Pe2.

[0038] In the third light-emitting pattern, as shown in FIG. 5 , the first illumination light L1 is emitted for one frame during the first light-emitting period Pe1, and the second illumination light L2 is emitted for four frames during the second light-emitting period Pe2. In this case, during the second light-emitting period Pe2, the second illumination light L2 is automatically switched between second illumination light L2SP, second illumination light L2SQ, second illumination light L2SR, and second illumination light L2SS, each of which has a different spectrum, for each frame. In the third light-emitting pattern, the first illumination light image P1 is obtained during the first light-emitting period Pe1, and second illumination light images P2SP, P2SQ, P2SR, and P2SS are obtained during the second light-emitting period Pe2. In the third light-emitting pattern, four types of second illumination light L2 with different spectra are switched between, but multiple types of second illumination light other than four may be switched between.

[0039] Next, details of the image processing unit 41 will be described. As shown in Fig. 6 , the image processing unit 41 includes a three-dimensional structure information acquisition unit 50, a disease region information acquisition unit 51, a disease evaluation information acquisition unit 52, a disease-related information output unit 53, a disease-related information memory 55, a difference information acquisition unit 57, and a specific region information output unit 59.

[0040] The three-dimensional structural information acquisition unit 50 acquires three-dimensional structural information indicating the three-dimensional structure of a hollow organ. Examples of hollow organs include the esophagus, stomach, and large intestine. In this embodiment, the large intestine is described as an example of a hollow organ. Specifically, the three-dimensional structural information acquisition unit 50 acquires three-dimensional structural information from a first illumination light image P1, which is one of multiple medical images of a hollow organ captured from inside the lumen. Each time a first illumination light image is acquired, three-dimensional structural information is generated from the two-dimensional first illumination light image using a three-dimensional shape reconstruction technique (SfM (Structure from Motion)). Methods for generating three-dimensional shapes include the following: https: / / webbigdata.jp / ai / post-7118 https: / / ai.googleblog.com / 2020 / 08 / using-machine-learning-to-detect.html https: / / shiropen.com / 2020 / 07 / 22 / 53981 /

[0041] As shown in FIG. 7 , the three-dimensional shape restoration technology acquires a depth map image PD from a first illumination light image P1 through a depth estimation process. Then, three-dimensional structure information TSI is acquired from the depth map image. The depth map image PD displays a depth distribution indicating the distance between each position of the observation target and the distal end 12d of the endoscope. Of depth regions PD1, PD2, PD3, and PD4, depth region PD1 is the farthest region with the greatest distance, and depth region PD4 is the nearest region with the smallest distance. The three-dimensional structure information acquisition unit 50 may generate a learning model through machine learning or deep learning based on the first illumination light image and the three-dimensional structure information obtained from the first illumination light image, input the first illumination light image to the learning model, and output the three-dimensional structure information.

[0042] The three-dimensional structural information acquisition unit 50 can acquire three-dimensional structural information in a variety of ways, including acquiring three-dimensional structural information only from a plurality of medical images that are intraluminal images, acquiring three-dimensional structural information only from extraluminal images obtained by photographing a luminal organ from outside the lumen, and acquiring three-dimensional structural information from both a plurality of medical images and extraluminal images. Extraluminal images include computed tomography (CT) images, magnetic resonance imaging (MRI) images, and ultrasound images, and three-dimensional structural information can also be acquired from extraluminal images such as CT images.

[0043] Furthermore, when the three-dimensional structural information acquisition unit 50 acquires three-dimensional structural information using both multiple medical images and extraluminal images, the presence of both images of the inside and outside of the lumen at each position in the three-dimensional structure allows for more accurate display of the three-dimensional structure. Furthermore, when the three-dimensional structural information acquisition unit 50 acquires three-dimensional structural information using both multiple medical images and extraluminal images, three-dimensional structural information may be acquired by interpolating extraluminal images for portions of a luminal organ for which no medical images exist. For example, as shown in FIG. 8 , when there are no medical images of the rectum among luminal organs and the rectum portion (shown by the dotted line) is insufficient in the three-dimensional structural information TSI, three-dimensional structural information of the entire large intestine, including the rectum, can be acquired by interpolating the extraluminal image Pout of the rectum. Note that, in addition to extraluminal images, previous medical images acquired during a previous examination may also be used as images for interpolation.

[0044] The disease region information acquisition unit 51 acquires disease region information indicating a disease region in the three-dimensional structure of a luminal organ from a first illumination light image P1 or a second illumination light image P2 obtained by continuous imaging as multiple medical images. Continuous imaging preferably refers to, for example, medical images captured at a frame rate per second that include both the first illumination light image and the second illumination light image. In this embodiment, the disease is preferably an inflammatory disease such as ulcerative colitis or a neoplastic disease such as a tumor. Preferably, a learning model obtained by machine learning or deep learning based on the first illumination light image P1 or the second illumination light image P2 and the disease region information is used, and the disease region information is output by inputting the first illumination light image P1 or the second illumination light image P2 into the learning model. The disease region information may also be input by a user operating the user interface 16.

[0045] Specifically, disease region information is acquired using first illumination light images P11 and P12 and a second illumination light image P21 obtained by successive imaging. Here, as shown in FIG. 9 , when a three-dimensional structure is created from the first illumination light image P11 by the three-dimensional structure information acquisition unit 50, it is preferable to identify the anatomical position in the three-dimensional structure from the first illumination light image P12 (in the first emission mode) obtained consecutively with the first illumination light image P11, and use the identified result as disease region information. For example, if the identification result for the first illumination light image P12 is "rectum," "rectum" is used as disease region information. As disease region information, both acquisition of three-dimensional structure information and anatomical identification may be performed from one frame of the first illumination light image P11. Note that disease region information may be acquired by aligning the first illumination light image P1 and the second illumination light image P2 when the coordinates of the first illumination light image P1 and the second illumination light image P2 are the same.

[0046] As described above, when disease region information is acquired by identifying an anatomical position, the wavelength band of the first illumination light used to generate the first illumination light image is preferably wider than the wavelength band of the second illumination light used to generate the second illumination light image. For example, it is preferable that the first illumination light is white light and the second illumination light is a specific narrowband light. This is because illuminating the observation target with broadband light is likely to increase the accuracy of identifying the anatomical position. Furthermore, the anatomical position in the three-dimensional structure may be identified from the second illumination light image P2 instead of the first illumination light image P1. Furthermore, the disease region information acquisition unit 51 may simply use position information in the three-dimensional structure as disease region information without identifying the anatomical position.

[0047] The disease evaluation information acquisition unit 52 acquires disease evaluation information indicating a disease evaluation of the disease region from the second illumination light image P2. The disease evaluation information is preferably a pixel-by-pixel evaluation value of the second illumination light image P2 or a statistic based on the pixel-by-pixel evaluation value of the second illumination light image P2. Examples of the disease evaluation information include Mayo, UCEIS, UCEIS-vascular visibility (a subscore of UCEIS), UCEIS bleeding level, UCEIS-ulcer, whether or not endoscopic remission is achieved, Geboes, indicating the degree of pathological inflammation, and whether or not pathological remission is achieved. The disease evaluation information acquisition unit 52 preferably associates the disease evaluation information with disease region information obtained based on the second illumination light image P2 and the first illumination light image captured consecutively, and outputs or stores the information to the disease-related information output unit 53. For example, the disease evaluation information acquisition unit 52 associates "rectum" in the disease region information with "Mayo 2" in the disease evaluation information. It is preferable to use a learning model obtained by machine learning or deep learning based on the second illumination light image P2 and disease area information, and to output disease evaluation information by inputting the second illumination light image P2 into the learning model.

[0048] The disease-related information output unit 53 outputs the three-dimensional structure information, disease area information, and disease evaluation information in association with each other. The output destination is the display control unit 42 when displaying on the display 15, and the disease-related information memory 55 when storing. When displaying the three-dimensional structure information, etc. on the display 15, the display control unit 42 displays the output three-dimensional structure information, disease area information, and disease evaluation information on the display 15.

[0049] Specifically, as shown in Fig. 10 , the display control unit 42 preferably displays on the display 15 a three-dimensional structure in which the positions of disease regions are indicated by disease indicator markers DM. The display mode of the disease indicator marker DM is preferably changed according to the content of the disease evaluation information. For example, when the disease evaluation information is Mayo, a red disease indicator marker DMr is preferably used for Mayo 3, a green disease indicator marker DMg is used for Mayo 2, a blue disease indicator marker DMb is used for Mayo 1, and a light blue disease indicator marker DMw is used for Mayo 0 (all of which are represented by hatching in Fig. 10 ). On the other hand, a purple disease indicator marker DMv is preferably used for disease regions for which disease evaluation information has not been calculated.

[0050] Furthermore, as shown in Fig. 11 , the three-dimensional structural information may display disease region information in addition to disease evaluation information. Here, each disease region information is grouped and displayed by an ellipse. Specifically, the ellipse EP1 indicates the rectum, the ellipse EP2 indicates the sigmoid colon, and the ellipse EP3 indicates the descending colon. In Fig. 11 , disease evaluation information is acquired for each region, and a disease indicator marker DM is displayed according to the disease evaluation information for each region. In the three-dimensional structural information TSI, a red disease indicator marker DMr is displayed for the rectum, a purple disease indicator marker DMv is displayed for the sigmoid colon, and a blue disease indicator marker DMb is displayed for the descending colon.

[0051] Furthermore, as a display mode for the three-dimensional structural information, etc., it is preferable to distinguish a region Rgx of a region not captured by the multiple medical images from a region Rgy of a region for which a medical image is present, among the three-dimensional structural information displayed on the display 15. Specifically, as shown in Fig. 12, if a region not captured by the first illumination light image P1 or the second illumination light image P2 is the rectum, the rectum portion is displayed with a dotted line indicating the region Rgx, and other regions captured by the first illumination light image P1 or the second illumination light image P2 (other than the rectum) are displayed with a solid line indicating the region Rgy. When the rectum portion is interpolated using an extraluminal image, it is preferable to switch from dotted line display to solid line display.

[0052] Furthermore, when displaying three-dimensional structural information on the display 15, it is preferable to also display a real-time moving image based on a medical image. As shown in Figures 13 and 14, it is preferable to display the three-dimensional structural information on the right half of the display 15, and to display a real-time moving image MV based on a medical image on the left half. For example, in the case of Figure 13, a diagnosis is currently underway, so the three-dimensional structural information is being constructed sequentially, and therefore only a portion of the three-dimensional structural information is displayed on the display 15. On the other hand, in the case of Figure 14, the diagnosis has been completed, so all of the three-dimensional structural information is displayed on the display 15.

[0053] The three-dimensional structural information may be obtained by other methods than using medical images captured by the endoscope 12. For example, a schematic image (schematic image) showing the average three-dimensional structure of a hollow organ such as the large intestine may be determined in advance, and this predetermined schematic image may be obtained as the three-dimensional structural information. Furthermore, since there may be individual differences in hollow organs, such as the length of the large intestine, it is preferable to be able to correct the three-dimensional structural information based on individual differences.

[0054] When saving three-dimensional structural information, etc., the disease-related information output unit 53 saves a three-dimensional structural information set in which the three-dimensional structural information, disease region information, and disease evaluation information are associated with each other in the disease-related information memory 55. For example, when three-dimensional structural information of region X is acquired from the first illumination light image P11, if "Mayo 1" is obtained as disease evaluation information from the second illumination light image P2 obtained consecutively to the first illumination light P11, by saving the three-dimensional structural information and the disease evaluation information in association with each other, it is possible to determine that the disease evaluation information for region X of the three-dimensional structural information is "Mayo 1." Furthermore, when the disease region information obtained from the first illumination light image P12 is "rectum," if "Mayo 2" is obtained as disease evaluation information from the second illumination light image P2 obtained consecutively to the first illumination light image P12, it is possible to determine that the disease evaluation information of "rectum" is "Mayo 2" by saving the disease region information and the disease evaluation information in association with each other.

[0055] It is also preferable to store examination time information such as the date and time of the examination when the diagnosis was performed in association with the information. After the diagnosis is completed, it is also preferable to store the entire three-dimensional structural information set as the three-dimensional structural information set at the time of the first examination in the disease-related information memory 55 in association with the three-dimensional structural information set at the time of the second examination after the first examination, such as a past diagnosis.

[0056] As described above, by storing the three-dimensional structural information sets from the first examination and the second examination in the disease-related information memory 55, as shown in Fig. 15, the display control unit 42 can read the three-dimensional structural information sets from the first examination and the second examination from the disease-related information memory 55 and display the three-dimensional structural information set TSI1 from the first examination and the structural information set TSI2 from the second examination on the same screen of the display 15. Note that for each of the three-dimensional structural information sets TSI1 and TSI2 currently displayed on the display 15, it is preferable to display the examination date and time ("March 8, 2010" for TSI1 and "June 1, 2021" for TSI2), and also to display the type of disease evaluation information currently displayed ("Mayo" for TSI1 and "UCEIS" for TSI2).

[0057] The difference information acquisition unit 57 reads the three-dimensional structural information sets from the first and second examinations from the disease-related information memory 55, and acquires difference information representing the difference between the disease evaluation in the disease region at the first examination and the disease evaluation in the disease region at the second examination from the disease region information and disease evaluation information at the first examination and the disease region information and disease evaluation information at the second examination. The difference information is preferably displayed in the three-dimensional structural information. Specifically, as shown in FIG. 16 , the difference information DF is preferably the difference between the disease evaluation at the first examination and the disease evaluation at the second examination in a common region CM common to the disease region at the first examination and the disease region at the second examination.

[0058] If the disease evaluation is Mayo, the difference in the Mayo numerical level (1 to 4) between the first and second examinations is the difference in the disease evaluation. The difference in the disease evaluation is preferably expressed in shading, with the greater the difference value being the darker the shade (in FIG. 16 , the shading is represented by the spacing between the hatched lines). For example, among the common regions CM1 to CM6, the common region CM1 at the rectum has a larger difference value (the spacing between the hatched lines is smaller) compared to the common regions CM2 to CM6 at the other sites, indicating that the disease state has worsened or improved. In FIG. 16 , the type of disease evaluation information displayed in the difference information DF is indicated as "Mayo."

[0059] When displaying three-dimensional structural information on the display 15, the specific region information output unit 59, upon receiving a user's designation of a specific region of the three-dimensional structure, displays disease region information and disease evaluation information corresponding to the specific region on the display 15. The user designates the specific region by operating the user interface 16. As shown in FIG. 17 , a specific region SPR is designated by moving the pointer PT to the disease region for which the disease region information and disease evaluation information are to be displayed and then performing a confirmation operation. The outer frame of the designated specific region SPR is displayed in bold. In accordance with the designation of the specific region SPR, the specific region information output unit 59 reads out disease region information and disease evaluation information corresponding to the specific region from the disease-related information memory 55 and displays them in the information display area AR.

[0060] In Fig. 17, when a disease region near the sigmoid colon is designated as the specific region SPR, the disease evaluation information corresponding to the specific region SPR is displayed in the information display area AR as follows: "Mayo" is "1," "UCEIS" is "2," "UCEIS-Vascular Visibility" is "1," "UCEIS-Bleeding" is "0," "UCEIS-Ulcer" is "0," "Gebose" is "2A." 1, "Endoscopic Remission" is "Remission," and "Pathological Remission" is "Remission." Also, as shown in Fig. 18, when a disease region CS (displayed with a solid line) whose disease evaluation information is "tumor" is displayed and disease regions other than "tumor" (displayed with a dotted line) are hidden, when a specific region SRP is designated within the disease region of "tumor," information about the tumor, such as "size: 3 mm, type A, 16 cm from the anus," is displayed in the information display area AR as the disease region information corresponding to the specific region SPR.

[0061] In the above embodiment, the three-dimensional structure information, disease region information, and disease evaluation information are obtained using at least two frames of the first illumination light image P1 and the second illumination light image P2 obtained at different times, but the three-dimensional structure information, disease region information, and disease evaluation information may also be obtained from one frame of a medical image obtained at the same time. In this case, it is preferable to use the first illumination light image P1, such as a white light image, as the medical image.

[0062] Specifically, as shown in Figure 19, an image segmentation process is performed to segment one frame of a medical image into two types of spatial frequency components, low frequency and high frequency, to generate a base image from which the low frequency components are extracted and a texture image from which the high frequency components are extracted. The base image is an image that serves as a brightness reference, and the texture image is an image from which the mucosal structure, vascular patterns, etc. are extracted. It is preferable to obtain three-dimensional structural information from the base image. It is preferable to obtain disease region information from the base image or the texture image. It is preferable to obtain disease evaluation information from the texture image. Before obtaining three-dimensional structural information, etc., it is preferable to perform brightness correction processing on the base image, and it is preferable to perform texture enhancement processing, such as frequency component extraction processing or contrast enhancement processing, on the texture image.

[0063] In the above embodiment, the hardware structure of the processing units that perform various processes, such as the medical image acquisition unit 40, the display control unit 42, the three-dimensional structure information acquisition unit 50, the disease region information acquisition unit 51, the disease evaluation information acquisition unit 52, the disease-related information output unit 53, the difference information acquisition unit 57, and the specific region information output unit 59, is the following various processors: The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a GPU (Graphical Processing Unit), a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, which is a processor having a circuit configuration specifically designed for performing various processes.

[0064] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Multiple processing units may also be configured with a single processor. Examples of multiple processing units configured with a single processor include: a first configuration, as typified by client or server computers, in which a single processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units; and a second configuration, as typified by system-on-chip (SoC), in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip. In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0065] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit formed by combining circuit elements such as semiconductor elements, and the hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD).

[0066] 10 Endoscope system 12 Endoscope 12a Insertion section 12b Operation section 12c Bending section 12d Tip section 12e Angle knob 12f Mode changeover switch 12h Still image acquisition instruction switch 12i Zoom operation section 12j Forceps port 13 Light source device 14 Processor device 15 Display 16 User interface 20 Light source section 20a V-LED 20b B-LED 20c G-LED 20d R-LED 22 Optical path coupling section 23 Light guide 30 Illumination optical system 31 Illumination lens 32 Imaging optical system 35 Objective lens 36 Zoom lens 37 Imaging sensor 38 Imaging processor 40 Medical image acquisition section 41 Image processing section 42 Display control section 43 Central control section 50 Three-dimensional structure information acquisition section 51 Disease area information acquisition section 52 Disease evaluation information acquisition unit 53 Disease-related information acquisition unit 55 Disease-related information memory 57 Difference information acquisition unit 59 Specific region information output unit CM1 to CM6 Common region DF Difference information DM, DMv, DMb, DMg, DMr, DMw Disease display marker EP1 to EP3 Ellipse L1 First illumination light L2, L2SP, L2SQ, L2SR, L2SS Second illumination light MV Video display P1, P11, P12 First illumination light image P2SP, P2SQ, P2SR, P2SS Second illumination light image PD Depth map PD1 to PD4 Depth region Pe1 First emission period Pe2 Second emission period Pout Extraluminal image PT Pointer Rgx, Rgy Region SPR Specific region TSI Three-dimensional structure information TSI1: 3D structural information from the first inspection TSI2: 3D structural information from the second inspection

Claims

1. a processor; The processor: Acquire multiple medical images of a hollow organ taken from inside the lumen, acquiring three-dimensional structural information indicating a three-dimensional structure of the hollow organ; acquiring disease area information indicating a disease area in the three-dimensional structure of the hollow organ and disease evaluation information indicating an evaluation of a disease in the disease area from the plurality of medical images; outputting the three-dimensional structure information, the disease region information, and the disease evaluation information in association with each other; A medical image processing system, wherein when the three-dimensional structure in which the position occupied by the disease region is displayed by a disease display marker is displayed on a display, the disease display marker is changed in accordance with the content of the disease evaluation information.

2. a processor; The processor: Acquire multiple medical images of a hollow organ taken from inside the lumen, acquiring three-dimensional structural information indicating a three-dimensional structure of the hollow organ; acquiring disease area information indicating a disease area in the three-dimensional structure of the hollow organ and disease evaluation information indicating an evaluation of a disease in the disease area from the plurality of medical images; outputting the three-dimensional structure information, the disease region information, and the disease evaluation information in association with each other; When the three-dimensional structure information is displayed on a display, and when a user specifies a specific area of ​​the three-dimensional structure, the medical image processing system displays the outer frame of the specified specific area in bold, and also displays the disease area information and disease evaluation information corresponding to the specific area on the display.

3. The processor: acquiring a first illumination light image and a second illumination light image as the plurality of medical images; obtaining the three-dimensional structure information from the first illumination light image; obtaining the disease region information from the first illumination light image or the second illumination light image; The medical image processing system according to claim 1 , wherein the disease evaluation information is acquired from the second illumination light image.

4. the first illumination light image and the second illumination light image are obtained by successive photographing; 4. The medical image processing system according to claim 3, wherein a wavelength band of the first illumination light used to generate the first illumination light image is wider than a wavelength band of the second illumination light used to generate the second illumination light image.

5. The medical image processing system according to claim 4 , wherein the processor acquires the disease region information using successive images of the first illumination light and the second illumination light.

6. The processor: generating a base image and a texture image from one frame of the medical image; obtaining the three-dimensional structural information from the base image; obtaining the disease region information from the base image or the texture image; The medical image processing system according to claim 1 , wherein the disease evaluation information is obtained from the texture image.

7. The processor:

2. The medical image processing system according to claim 1, wherein the three-dimensional structural information is obtained using at least one of the plurality of medical images and an extraluminal image obtained by photographing the hollow organ from outside the lumen.

8. The medical image processing system according to claim 7 , wherein the processor obtains the three-dimensional structural information by interpolating the extraluminal image for a portion of the hollow organ for which no medical image exists.

9. 2. The medical image processing system of claim 1, wherein the processor displays areas of the three-dimensional structure displayed on the display that are not captured by the plurality of medical images, distinguishing them from areas of the structure for which the medical images are present.

10. 3. The medical image processing system according to claim 1, wherein said display displays a moving image based on said medical image in addition to displaying said three-dimensional structure.

11. 2. The medical image processing system according to claim 1, wherein the three-dimensional structural information is a predetermined schematic image.

12. The processor:

3. A medical image processing system as described in claim 1 or 2, wherein when the three-dimensional structural information, the disease evaluation information, and a three-dimensional structural information set in which the disease evaluation information is associated with each other are stored in a disease-related information memory, the three-dimensional structural information set from a first examination and the three-dimensional structural information set from a second examination that is later than the first examination are associated and stored in the disease-related information memory.

13. The processor:

13. A medical image processing system according to claim 12, wherein the set of three-dimensional structural information at the time of the first examination and the set of three-dimensional structural information at the time of the second examination are displayed on the same screen of a display.

14. The processor: obtaining difference information representing a difference between a disease assessment in the disease region at the time of the first examination and a disease assessment in the disease region at the time of the second examination from the three-dimensional structural information set at the time of the first examination and the three-dimensional structural information set at the time of the second examination; 13. The medical image processing system according to claim 12, wherein the difference information is displayed on the three-dimensional structural information.

15. 3. A medical image processing system according to claim 1, wherein the disease is an inflammatory disease or a tumor disease.