Endoscope system, medical image processing device, and operation method thereof

Abstract

To provide an endoscope system, a medical image processing device and an operation method therefor which allow a user to recognize a correct boundary line in real time.SOLUTION: A medical image processing device acquires a reference image that is a medical image with which boundary line information related to a boundary line that is a boundary between an abnormal region and a normal region and landmark information related to a landmark and a captured image that is the medical image captured in real time, detects the landmark from the captured image, calculates a ratio of match between the landmark included in the reference image and the landmark included in the captured image, estimates a correspondence relationship between the reference image and the captured image on the basis of the ratio of match and information regarding the landmarks included in the reference image and the captured image, and generates a superimposition image in which the boundary line associated with the reference image is superimposed on the captured image on the basis of the correspondence relationship.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an endoscope system for assisting surgery such as endoscopic submucosal dissection, a medical image processing device, and an operating method thereof. [Background technology]

[0002] Endoscopic submucosal dissection (ESD) has made it possible to remove tumors larger than those that can be treated with endoscopic mucosal resection (EMR), eliminating the need for more invasive surgery. Because ESD is performed under an endoscope, it has the advantage of being minimally invasive. However, the gastrointestinal tract is extremely thin, measuring only 5-7 mm for the stomach and 3-5 mm for the large intestine, requiring highly advanced techniques from physicians performing ESD.

[0003] There are known techniques for displaying lesion areas to assist surgeons during surgery. For example, a medical image processing device is known that detects a region of interest that includes a lesion, and superimposes on the region of interest the coordinates of the boundary between the region of interest and a region of non-interest, the coordinates of a position inside the region of interest along the boundary, or the coordinates of the center of gravity of the region of interest (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020 / 090731 Summary of the Invention [Problem to be solved by the invention]

[0005] Advances in image processing have made it possible to display the area and border of a differentiated lesion on an image. However, when determining the extent of a lesion on a frame-by-frame basis, the displayed area of ​​the lesion changes over time. Because real-time images change constantly, it is difficult to accurately distinguish the border between abnormal and normal regions in real time. Depending on the imaging conditions, the displayed border may be inaccurate. Furthermore, when performing ESD, border determination is difficult using low-magnification observation (distant view) that provides a bird's-eye view of the entire lesion; instead, it requires high-magnification observation (close-up view) that magnifies a portion of the lesion. Given these circumstances, there is a need for technology that can accurately recognize borders determined in real time using images captured under optimal conditions. Furthermore, there is a need for technology that can confirm how previously diagnosed borders have changed on current real-time images.

[0006] An object of the present invention is to provide an endoscope system, a medical image processing device, and an operating method thereof that are capable of recognizing accurate boundary lines in real time. [Means for solving the problem]

[0007] The medical image processing device of the present invention is a medical image processing device equipped with a processor, which acquires a medical image of a subject photographed with an endoscope, acquires a reference image which is a medical image that associates boundary information associated with the boundary line that is the boundary between an abnormal area and a normal area with landmark information associated with landmarks that are characteristic structures of the subject, acquires a photographed image which is a medical image photographed in real time, detects landmarks from the photographed image, calculates the degree of correspondence between the landmarks contained in the reference image and the landmarks contained in the photographed image, estimates the correspondence between the reference image and the photographed image based on the degree of correspondence and the landmark information contained in the reference image and the photographed image, and generates a superimposed image in which the boundary line associated with the reference image is superimposed on the photographed image based on the correspondence.

[0008] It is preferable that the photographed image and the reference image are medical images photographed at the same magnification. It is preferable that the photographed image is a medical image photographed from a distance, and the reference image is a medical image photographed from a close distance.

[0009] The captured image is a medical image captured to include a portion of the abnormal area, and the reference image is a medical image that includes the entire abnormal area, and it is preferable to generate a superimposed image in which a portion of the boundary line associated with the reference image is superimposed on the captured image.

[0010] The reference image is preferably generated by joining together a plurality of enlarged medical images, each of which is a medical image in which a portion of an abnormal region is photographed in close-up.

[0011] When the reference image consists of a first enlarged medical image and a second enlarged medical image taken at a different position from the first enlarged medical image, it is preferable that the processor generates the reference image by connecting the first enlarged medical image and the second enlarged medical image based on the common relationship between the landmark information and boundary information associated with the first enlarged medical image and the landmark information and boundary information associated with the second enlarged medical image.

[0012] The processor preferably distinguishes between abnormal and normal regions and sets boundaries, which are preferably set by user interaction.

[0013] Preferably, the reference image is a medical image captured by illuminating the subject with special light, and the captured image is a medical image captured by illuminating the subject with normal light.

[0014] It is preferable that the photographed image from which the landmark is detected and the photographed image on which the boundary line is superimposed are photographed at the same time.

[0015] It is preferable that the captured image for detecting the landmark and the captured image on which the boundary line is superimposed are captured at different times. It is preferable that the calculation of the degree of coincidence continues until an instruction to end is given.

[0016] It is preferable that the processor generates a display image, displays the superimposed image in a first display section of the display image, and controls the display of the reference image in a second display section different from the first display section of the display image.

[0017] It is preferable that the processor acquires an enlarged image as the captured image, identifies abnormal and normal areas contained in the enlarged image to set boundary lines, detects landmarks from the enlarged image, associates boundary line information associated with the boundary lines and landmark information associated with the landmarks with the enlarged image, and uses the resulting image as a reference image.

[0018] The processor generates a superimposed image, and when an update instruction is received, identifies abnormal and normal areas contained in the captured image related to the superimposed image on which the boundary line is superimposed, and when an update boundary line is set as the boundary between the abnormal and normal areas and when an update determination instruction is received, it is preferable to update the boundary line superimposed on the captured image using the update boundary line as the determined update boundary line.

[0019] The operating method of the medical image processing device of the present invention comprises the steps of acquiring a medical image of a subject photographed with an endoscope, acquiring a reference image which is a medical image that associates boundary information associated with the boundary line that is the boundary between an abnormal area and a normal area with landmark information associated with landmarks that are characteristic structures of the subject, acquiring a photographed image which is a medical image photographed in real time, detecting landmarks from the photographed image, calculating the degree of correspondence between the landmarks included in the reference image and the landmarks included in the photographed image, estimating the correspondence between the reference image and the photographed image based on the degree of correspondence and the landmark information included in the reference image and the photographed image, and generating a superimposed image in which the boundary line associated with the reference image is superimposed on the photographed image based on the correspondence.

[0020] An endoscope system of the present invention includes the above-described medical image processing device and an endoscope. [Effects of the Invention]

[0021] According to the present invention, accurate boundary lines can be recognized in real time. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is an explanatory diagram of a configuration of an endoscope system. [Figure 2] FIG. 2 is a block diagram showing the functions of the endoscope system. [Figure 3] 1 is a graph showing the spectra of violet light V, blue light B, green light G, and red light R in normal light. [Figure 4] 1 is a graph showing the spectra of purple light V, blue light B, green light G, and red light R in special light. [Figure 5] FIG. 10 is an explanatory diagram showing a first light emission pattern in an image analysis mode. [Figure 6] FIG. 10 is an explanatory diagram showing a second light emission pattern in an image analysis mode. [Figure 7] FIG. 10 is an image diagram showing an example of a captured image in which landmarks are detected. [Figure 8] FIG. 10 is an image diagram showing an example of a captured image in which landmarks are connected by link lines. [Figure 9] FIG. 10 is an image diagram showing an example of a reference image in which a boundary line is indicated at the boundary between an abnormal region and a normal region. [Figure 10] 10A and 10B are explanatory diagrams showing an example of calculating the degree of coincidence between a reference image and a captured image; [Figure 11] FIG. 10 is an explanatory diagram showing generation of a superimposed image. [Figure 12] 10A and 10B are explanatory diagrams showing generation of a superimposed image when a photographed image and a reference image are photographed at the same magnification; [Figure 13] 10A and 10B are explanatory diagrams showing the generation of a superimposed image when a photographed image is observed from a distance and a reference image is photographed from a close distance; [Figure 14] 10A and 10B are explanatory diagrams showing generation of a superimposed image when a captured image is captured so as to include a part of an abnormal region and a reference image includes the entire abnormal region. [Figure 15]FIG. 10 is an explanatory diagram showing that a reference image is generated by joining enlarged medical images together. [Figure 16] FIG. 10 is a block diagram showing the functions of a reference image generating unit in a boundary line display mode. [Figure 17] FIG. 10 is an explanatory diagram showing generation of a reference image by joining enlarged medical images using common relationships. [Figure 18] 10A and 10B are explanatory diagrams showing the identification of an abnormal region in an enlarged medical image, the setting of a boundary line, and the detection of a landmark when the boundary line is automatically set. [Figure 19] FIG. 10 is a block diagram showing the functions of a reference image generating unit when a boundary line is set by a user operation. [Figure 20] FIG. 10 is an explanatory diagram illustrating how a boundary line is set by a user operation. [Figure 21] 10A and 10B are explanatory diagrams showing generation of a superimposed image when a reference image is captured using special light and a captured image is captured using normal light; [Figure 22] 10 is an explanatory diagram showing generation of a superimposed image when a photographed image from which a landmark is to be detected and a photographed image on which a boundary line is to be superimposed are photographed at the same time. FIG. [Figure 23] 10 is an explanatory diagram showing generation of a superimposed image when a photographed image from which a landmark is to be detected and a photographed image on which a boundary line is to be superimposed are photographed at different times. FIG. [Figure 24] 10 is a flowchart showing a series of steps in a boundary line display mode. [Figure 25] FIG. 10 is an image diagram showing a display image. [Figure 26] FIG. 10 is a block diagram showing functions of a captured image input unit and a reference image generation unit in a reference image generation mode. [Figure 27] 10A and 10B are explanatory diagrams showing generation of a reference image in a reference image generation mode. [Figure 28] FIG. 10 is a block diagram showing functions of a captured image input unit and a reference image generation unit in a boundary line update mode. [Figure 29]10A and 10B are explanatory diagrams showing generation of a superimposed image in a boundary line update mode. [Figure 30] 10A and 10B are explanatory diagrams showing generation of an updated boundary line superimposed image; [Figure 31] 10A and 10B are explanatory diagrams showing generation of a determined and updated boundary line superimposed image; [Figure 32] FIG. 1 is an explanatory diagram of the configuration of an endoscope system when a rigid endoscope is used. [Figure 33] FIG. 2 is an explanatory diagram showing the configuration of a light receiving unit. DETAILED DESCRIPTION OF THE INVENTION

[0023] 1, the endoscope system 10 includes an endoscope 12, a light source device 14, a processor device 15, a medical image processing device 11, a display 17, and a user interface 19. The medical image processing device 11 is connected to the endoscope system 10 via the processor device 15. The endoscope 12 is optically connected to the light source device 14 and electrically connected to the processor device 15.

[0024] The endoscope 12 has an insertion section 12a that is inserted into the body of an object to be observed, an operation section 12b provided at the base end of the insertion section 12a, and a bending section 12c and a distal end section 12d provided at the distal end of the insertion section 12a. The bending section 12c is bent by operating an angle knob 12e of the operation section 12b. The distal end section 12d is oriented in a desired direction by bending the bending section 12c. A forceps channel (not shown) is provided from the insertion section 12a to the distal end section 12d for inserting a treatment tool or the like. The treatment tool is inserted into the forceps channel through a forceps port 12j.

[0025] The endoscope 12 is provided inside with an optical system for forming an image of a subject and an optical system for irradiating the subject with illumination light. The operation unit 12b is provided with an angle knob 12e, a mode selector switch 12f, a still image acquisition instruction switch 12h, and a zoom operation unit 12i. The mode selector switch 12f is used to switch the observation mode. The still image acquisition instruction switch 12h is used to instruct acquisition of a still image. The zoom operation unit 12i is used to operate the zoom lens 42.

[0026] The light source device 14 generates illumination light. The display 17 displays medical images. The medical images include captured images, which are medical images captured in real time by the endoscope 12; reference images, which are medical images that associate information on the boundary between abnormal and normal regions with information on landmarks, which are characteristic structures of the subject, as described below; and superimposed images, in which the boundary is superimposed on the captured images. Note that "real time" does not refer to a specific time or a strict coincidence, but rather refers to a time period that includes the most recent period in a single endoscopic examination. An abnormal region refers to a region of the observation object where an abnormality is found, such as a region where a tumor is present or a region where inflammation is observed. A normal region refers to a region other than the abnormal region where no abnormality is found. Furthermore, an abnormal region may be defined as a "tumor region" and a normal region as a "non-tumor region," with only regions where a tumor is specifically identified as abnormal regions, and areas where inflammation is found around the tumor may also be included in the normal region.

[0027] The user interface 19 has a keyboard, a mouse, a touchpad, a microphone, a tablet 241, a touch pen 242, etc., and has a function of accepting input operations such as function settings. The processor device 15 controls the endoscope system 10 and performs image processing on image signals transmitted from the endoscope 12.

[0028] 2, the light source device 14 includes a light source unit 20 and a light source processor 21 that controls the light source unit 20. The light source unit 20 has, for example, multiple semiconductor light sources that are turned on or off and, when turned on, control the light emission amount of each semiconductor light source to emit illumination light that illuminates the observation target. The light source unit 20 has four color LEDs: a V-LED (Violet Light Emitting Diode) 20a, a B-LED (Blue Light Emitting Diode) 20b, a G-LED (Green Light Emitting Diode) 20c, and an R-LED (Red Light Emitting Diode) 20d. The light source unit 20 may be built into the endoscope 12, and the light source control unit may be built into the endoscope 12 or the processor device 15.

[0029] The endoscope system 10 has a mono-emission mode, a multi-emission mode, a boundary display mode, a reference image generation mode, and a boundary update mode, which can be switched by the mode switch 12f. The mono-emission mode is a mode in which illumination light of the same spectrum is continuously irradiated onto the observation object to illuminate it. The multi-emission mode is a mode in which illumination light of different spectrums is irradiated onto the observation object while being switched according to a specific pattern. The illumination light includes normal light (broadband light such as white light) used to provide brightness to the entire observation object to observe the entire observation object, and special light used to highlight specific regions of the observation object. In the mono-emission mode, illumination light of different spectrums may be switched by operating the mode switch 12f. For example, normal light may be used as the first illumination light and special light may be used as the second illumination light, or special light may be used as the first illumination light and normal light may be used as the second illumination light.

[0030] As shown in FIG. 3, when emitting normal light, the V-LED 20a emits purple light V with a central wavelength of 405±10 nm and a wavelength range of 380 to 420 nm. The B-LED 20b emits blue light B with a central wavelength of 450±10 nm and a wavelength range of 420 to 500 nm. The G-LED 20c emits green light G with a wavelength range of 480 to 600 nm. The R-LED 20d emits red light R with a central wavelength of 620 to 630 nm and a wavelength range of 600 to 650 nm. As shown in FIG. 4, a special light may be used in which the amount of purple light V is greater than the amounts of blue light B, green light G, and red light R.

[0031] The boundary line display mode is a mode in which, when a tumor is found in the subject, a superimposed image is generated by superimposing a boundary line from a reference image, which has a predefined accurate boundary line between the abnormal area and the normal area, on a captured image taken in real time, and the generated image is displayed on the display 17 to show to the user, thereby assisting in the incision of the mucosa during ESD. The reference image generation mode is a mode in which a reference image is generated using a captured image. The boundary line update mode is a mode in which the boundary line is updated while a superimposed image in which the boundary line from the reference image is superimposed on the captured image is displayed.

[0032] The light source processor 21 independently controls the light intensity of the four colors of violet light V, blue light B, green light G, and red light R. In the mono emission mode, illumination light of the same spectrum is continuously emitted for each frame. For example, a first illumination light image with natural coloring may be displayed on the display 17 by illuminating the object of observation with normal light (first illumination light) and capturing an image. Alternatively, in the mono emission mode, the first illumination light and second illumination light may be switched, and special light (second illumination light) may be illuminated on the object of observation and captured, and a second illumination light image with a specific structure emphasized may be displayed on the display 17. The first illumination light image and the second illumination light image are types of medical images.

[0033] The light used when performing ESD is usually the first illumination light. The second illumination light may be used when it is necessary to check the extent of infiltration into the lesion before performing ESD. In the boundary display mode, it is possible to select whether to obtain a medical image using the illumination pattern of either the mono-emission mode or the multi-emission mode, and also to select whether to obtain a medical image using the first illumination light image or the second illumination light image using the illumination pattern of the mono-emission mode.

[0034] On the other hand, in the multi-light-emission mode, the light intensities of the purple light V, blue light B, green light G, and red light R are controlled according to a specific pattern. For example, as shown in FIG. 5, the light-emission pattern is a first light-emission pattern in which the number of frames in each first illumination period during which the subject is illuminated with the first illumination light is the same for each first illumination period. As shown in FIG. 6, the light-emission pattern is a second light-emission pattern in which the number of frames in each first illumination period is different for each first illumination period. In FIGS. 5 and 6, arrows indicate the direction of time. A frame refers to the time from when an imaging sensor (not shown) provided at the distal end 12d of the endoscope starts receiving return light from the subject to when it completes outputting the charge signal accumulated based on the received light. The second illumination period is the period during which the subject is illuminated with the second illumination light.

[0035] The light emitted from each of the LEDs 20a to 20d (see FIG. 2) is incident on the light guide 23 via an optical path combining unit 22 configured with a mirror, a lens, etc. The light guide 23 propagates the light from the optical path combining unit 22 to the tip 12d of the endoscope 12.

[0036] An illumination optical system 30a and an imaging optical system 30b are provided at the distal end 12d of the endoscope 12. The illumination optical system 30a has an illumination lens 31, and illumination light propagated by the light guide 23 is irradiated onto the object of observation via the illumination lens 31. When the light source unit 20 is built into the distal end 12d of the endoscope 12, light is emitted toward the object via the illumination lens of the illumination optical system without passing through a light guide. The imaging optical system 30b has an objective lens 41 and an imaging sensor 43. Light from the object of observation irradiated with illumination light is incident on the imaging sensor 43 via the objective lens 41 and a zoom lens 42. As a result, an image of the object of observation is formed on the imaging sensor 43. The zoom lens 42 is a lens for enlarging the object of observation, and is moved between the telephoto end and the wide-angle end by operating the zoom operation unit 12i.

[0037] The imaging sensor 43 is a primary color sensor and includes three types of pixels: B pixels (blue pixels) with blue color filters, G pixels (green pixels) with green color filters, and R pixels (red pixels) with red color filters.

[0038] Furthermore, the imaging sensor 43 is preferably a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging processor 44 controls the imaging sensor 43. Specifically, the imaging processor 44 reads out a signal from the imaging sensor 43, thereby outputting an image signal from the imaging sensor 43. The output image signal is sent to the captured image acquisition unit 60 of the processor device 15.

[0039] The captured image acquisition unit 60 performs various signal processing on the received image signal, such as defect correction, offset processing, demosaic processing, matrix processing, white balance adjustment, gamma conversion processing, YC conversion processing, etc. Next, the captured image signal is subjected to image processing including color conversion processing such as 3x3 matrix processing, gradation conversion processing, and 3D LUT (Look Up Table) processing, color enhancement processing, and structure enhancement processing such as spatial frequency enhancement.

[0040] If the user wishes to capture the captured image as a still image, the still image capture instruction switch 12h is operated, and a signal regarding the still image capture instruction is sent to the endoscope 12, the light source device 14, and the processor device 15, and a still image is captured and stored in a still image storage unit (not shown).

[0041] In the processor device 15, the functions of the photographed image acquisition section 60 are realized by the first central control section 55, which is made up of an image control processor, running a program in a program memory.

[0042] The captured image generated by the captured image acquisition unit 60 is transmitted to the medical image processing device 11. The medical image processing device 11 includes a captured image input unit 100, a reference image recording unit 110, a first landmark detection unit 120, a coincidence calculation unit 130, a boundary line position estimation unit 140, a display control unit 150, a reference image generation unit 200, and a second central control unit 101 (see FIG. 2).

[0043] In the medical image processing device 11, the second central control unit 101, which is composed of an image analysis processor, operates programs in the program memory, thereby realizing the functions of the captured image input unit 100, reference image recording unit 110, first landmark detection unit 120, matching degree calculation unit 130, boundary line position estimation unit 140, display control unit 150, and reference image generation unit 200.

[0044] In the boundary display mode, the captured image is transmitted to the captured image input unit 100 of the medical image processing device 11 (see FIG. 2). In the boundary display mode, the captured image input unit 100 transmits the captured image to the first landmark detection unit 120. As shown in FIG. 7, the first landmark detection unit 120 detects landmarks, which are characteristic structures of the subject, from the captured image 121 captured in real time (landmarks are indicated by circles 122 in FIG. 7). The landmarks include digestive tract folds 123, blood vessels 124, and components of glandular structures, as well as lesions 125 (shown by diagonal lines in FIG. 7) and structures highlighted with a fluorescent agent. The fluorescent agent is, for example, ICG (Indocyanine Green). Hereafter, to avoid cluttering the diagrams, one of the circles 122 indicating the landmarks will be marked with a leader line and a symbol unless otherwise noted.

[0045] After detecting landmarks, the first landmark detection unit 120 further acquires position information of the landmarks. As shown in FIG. 7, when multiple landmarks are detected from the captured image 121 by the detection process, it is preferable to be able to distinguish between the landmarks. For example, a distinguishing number can be assigned to each landmark for distinguishing them. Furthermore, as shown in FIG. 8, it is preferable to connect the landmarks (indicated by circles 122) with link lines 126, associate the position information of each landmark with each other, and record the positional relationship of the landmarks in the captured image 121.

[0046] The captured image in which the landmarks have been detected is transmitted to the coincidence calculation unit 130. In addition to the captured image in which the landmarks have been detected, the reference image recording unit 110 transmits to the coincidence calculation unit 130 a reference image 111 in which boundary information associated with a boundary line 114 (shown by a dashed line in FIG. 9 ) between an abnormal region 112 and a normal region 113 and landmark information associated with the landmarks are associated, as shown in FIG. 9 . The reference image is a medical image recorded in advance in the reference image recording unit 110 by a user, in which an accurate boundary line 114 has been set in advance at the boundary between the abnormal region 112 and the normal region 113. The boundary information includes position information of the boundary line associated with the boundary line. The landmark information includes position information of landmarks associated with the landmarks and associated with the reference image, and the positional relationship between the landmarks connected by link lines 126 (not shown in FIG. 9 ). The boundary information and the landmark information are preferably associated by connecting the boundary line and each landmark with the link line 126.

[0047] As shown in FIG. 10 , the matching degree calculation unit 130 compares the reference image 111 with the captured image 121 and calculates the degree of matching. The degree of matching is a value indicating the degree to which landmarks associated with the reference image 111 are included in the landmarks detected from the captured image 121. In the example of FIG. 10 , the degree of matching is calculated between the landmarks included in the captured image 121 (locations indicated by circles 122a, 122b, 122c, 122d, 122e, and 122f) and the landmarks included in the reference image 111 (locations indicated by circles 122a, 122b, 122c, 122d, 122f, and 122g). In this case, all but one landmark (circle 122g) included in the reference image 111 is included in the captured image 121, so five out of six landmarks included in the reference image are included in the captured image 121. Therefore, in the example of FIG. 10 , for example, the degree of matching is 83% (rounded to the nearest whole number).

[0048] As shown in FIG. 11 , if the degree of match is equal to or greater than a threshold, the boundary line position estimation unit 140 estimates the correspondence between the reference image and the captured image from the position information of the landmarks detected from the captured image, the positional relationship of each landmark, the position information of the landmarks associated with the reference image, the positional relationship of each landmark, and the positional information of the boundary line. The correspondence is transmitted to the display control unit 150. The threshold of the degree of match can be set arbitrarily. The correspondence between the reference image and the captured image refers to a relationship in which each landmark on the reference image matches with which landmark on the captured image, based on the positional information of the landmarks in the reference image and the captured image, and an estimation of which part of the captured image the boundary line corresponds to, based on the positional relationship between the landmarks connected by the link line 126 in the reference image and the positional relationship between the landmarks connected by the link line 126 in the captured image.

[0049] 11 based on the correspondence between the reference image 111 and the photographed image 121. The display control unit 150 generates a superimposed image 151 in which a boundary line 114 associated with the reference image 111 is superimposed on the photographed image 121. The superimposed image 151 is a medical image in which the boundary line 114 associated with the reference image 111 is superimposed on the photographed image 121 photographed in real time.

[0050] With the above configuration, the accurate boundary line associated with the reference image is superimposed on the captured image captured in real time using the positional relationship of the landmarks, making it possible to confirm the most likely boundary line in real time. In addition, since it becomes possible to visually confirm the extent to which the boundary line of the reference image differs from that of the real-time image, by comparing the reference image, which is a past medical image, with the captured image, which is a current medical image, it is possible to confirm the expansion or contraction of the tumor infiltration range that has changed between the past and the present.

[0051] The superimposed image is preferably created when the photographed image 121 and the reference image 111 are photographed at the same magnification, as shown in Fig. 12. When the photographed image 121 and the reference image 111 are acquired at the same magnification, the boundary line 114 can be accurately superimposed. Furthermore, even when the scene of the photographed image changes due to ESD treatment, such as marking the mucosa or local injection of a liquid, the positional relationship of the landmarks can be used to display an accurate boundary line.

[0052] Furthermore, as shown in FIG. 13 , it is preferable to create superimposed image 151 when photographed image 121 is a medical image photographed from a distance, while reference image 111 is a medical image photographed from a close distance. In this case, photographed image 121 is photographed at a lower magnification than reference image 111. By estimating the position at which to superimpose the boundary line based on the positional relationship of landmarks (indicated by circles 122) included in reference image 111 that match the landmarks in photographed image 121, boundary line 114 can be superimposed on photographed image 121 photographed from a distance. When determining the boundary between an abnormal region and a normal region, it may be difficult to set boundary line 114 without closely observing the mucosal structure and microvessels 127 using a medical image photographed from a close distance. When close-up photography is required to set the boundary line, it becomes even more difficult to accurately determine boundary line 114 using distant-view photography that provides a bird's-eye view of the entire lesion. In such a case, if there is a landmark that can be identified in both a medical image taken at close range and a medical image taken at a distance, even if the image is acquired in real time using a long-distance shot at a distance where the boundary line cannot be distinguished, it is possible to estimate the boundary line 114 using the correspondence between the boundary line and the landmark, and display the boundary line in the long-distance shot image 121.

[0053] Furthermore, as shown in FIG. 14 , when the photographed image 121 is a medical image captured so as to include a portion of the abnormal region 112 and structures such as microvessels 127, and the reference image 111 is a medical image including the entire abnormal region 112, it is preferable to create a superimposed image 151 by superimposing a portion of the boundary line associated with the reference image 111 on the photographed image 121. In this case, the photographed image 121 is captured at a higher magnification than the reference image 111. In close-up photography, even a slight blur significantly affects the photographed image 121. By generating the superimposed image 151 when close-up photography, which is significantly affected by blur, is performed in real time, an accurate boundary line can be presented to a user such as a surgeon. In addition, it is possible to easily check the extent of the tumor that has changed between the acquisition of the reference image 111 and the acquisition of the photographed image 121.

[0054] As shown in FIG. 15, the reference image is preferably generated as a single reference image 111 by stitching together magnified medical images 202 and 203, each of which is a medical image in which a portion of an abnormal region is captured at a close distance. The magnified medical images are associated by an association unit 260 with boundary line 114, information associated with boundary line 114, and landmarks (represented by circles 122h, 122i, 122j, 122k, 122l, 122m, and 122n) and information associated with the landmarks, as described below. The magnified medical images are still images captured at a close distance at a magnification that allows the boundary between the abnormal region and the normal region to be distinguished, and are captured under optimal conditions without blurring or halation. By stitching together magnified medical images to generate a reference image, a previously obtained high-resolution and accurate boundary line can be superimposed on the captured image.

[0055] An example of generating a reference image by joining enlarged medical images will be described. Joining of enlarged medical images 202, 203 is performed by inputting each of the enlarged medical images 202, 203 to an enlarged medical image input unit 201 of a reference image generation unit 200 shown in FIG. 16 and then transmitting the images to an enlarged medical image combination unit 210. The enlarged medical images may be enlarged photographed images, which will be described later, or medical images input from outside the endoscope system 10. The reference image generation unit 200 includes an enlarged medical image input unit 201, an enlarged medical image combination unit 210, an abnormality identification unit 220, which will be described later, a boundary line setting unit 230, a second landmark detection unit 250, and an association unit 260.

[0056] 17, an example will be shown in which the reference image 111 is made up of a first enlarged medical image 204 and a second enlarged medical image 205 that was captured at a position different from that of the first enlarged medical image 204. In this case, the enlarged medical image input unit 201 first transmits the first enlarged medical image 204 and the second enlarged medical image 205 to the enlarged medical image combination unit 210. The enlarged medical image combination unit 210 recognizes common relationships from landmark information (landmarks are shown by circles 122h, 122i, 122j, 122k and 122l) and boundary information (boundary line 114 is shown by a dotted line) associated with the first enlarged medical image 204, and landmark information (landmarks are shown by circles 122j, 122j, 122k, 122m and 122n) and boundary information (boundary line 114 is shown by a dotted line) associated with the second enlarged medical image, and connects the first enlarged medical image 204 and the second enlarged medical image 205 based on the common relationships to generate a reference image.

[0057] The common relationship refers to a relationship determined by determining whether the landmark information (landmarks) and boundary information (boundary lines) associated with the first enlarged medical image 204 and the second enlarged medical image 205, respectively, are common (match) with each other. The landmark information includes landmark position information and the positional relationship of the landmarks. The boundary line information includes boundary line position information.

[0058] 17, the reference image 111 is generated based on the common relationship between the landmark information relating to the two landmarks indicated by the circle 112i and the circle 112j and the boundary line information. In order to combine enlarged medical images, it is preferable that at least two or more landmarks are common to each other among the enlarged medical images.

[0059] While the above specific example illustrates a case where the reference image consists of a first enlarged medical image 204 and a second enlarged medical image 205, the present invention can also be applied to cases where there are three or more enlarged medical images, i.e., when stitching together the first to Nth enlarged medical images, by recognizing the common relationships between each enlarged medical image. The above configuration allows a reference image to be generated by stitching together multiple enlarged medical images associated with landmark information and boundary information. This allows for improved accuracy of boundary lines superimposed on a captured image based on a reference image with precisely defined boundary lines. Furthermore, because the number of landmarks associated with a single reference image increases, the correspondence between the reference image and the captured image can be estimated with higher accuracy from the positional relationship of the landmarks.

[0060] Preferably, the boundary line associated with the reference image is set by automatically identifying abnormal and normal regions. In this case, the enlarged medical image is transmitted from the enlarged medical image input unit 201 to the abnormality identification unit 220 (see FIG. 16). As shown in FIG. 18, the abnormality identification unit 220 identifies abnormal regions 112 and normal regions 113 in the enlarged medical image 206, and the boundary line setting unit 230 (see FIG. 16) sets a boundary line 114 and acquires position information of the boundary line. Next, the second landmark detection unit 250 (see FIG. 16) detects landmarks (indicated by circles 122) from the enlarged medical image 206 and acquires position information of the landmarks and their positional relationships. The associating unit 260 (see FIG. 16) receives the boundary line information (position information of the boundary line) from the boundary line setting unit 230 and also receives the landmark information (position information of the landmarks and their positional relationships) from the second landmark detection unit 250. The associating unit 260 associates the enlarged medical image, boundary line information, and landmark information, and uses the enlarged medical image as the reference image. The reference image is transmitted from the reference image generating unit 200 to the reference image recording unit 110 and recorded therein. The enlarged medical image associated with the boundary information and landmark information may be transmitted to the enlarged medical image combining unit 210 and used to combine with other enlarged medical images. With the above configuration, the automatically set boundary information can be used to superimpose on the captured image.

[0061] The anomaly identification unit 220 is preferably a trained model for distinguishing between abnormal and normal regions, which is trained using training medical image data in which abnormal and normal regions have been previously identified using machine learning. The information on abnormal and normal regions in the training medical image data may be added by an experienced physician or automatically by a device other than the medical image processing device 11. Deep learning is preferably used for the machine learning used to generate the trained model, and for example, a multilayer convolutional neural network is preferably used. In addition to deep learning, machine learning includes decision trees, support vector machines, random forests, regression analysis, supervised learning, semi-unsupervised learning, unsupervised learning, reinforcement learning, deep reinforcement learning, learning using neural networks, generative adversarial networks, and the like.

[0062] Furthermore, it is preferable that the boundary line associated with the reference image be set by user operation. In this case, as shown in FIG. 19, the abnormality identification unit 220 and boundary line setting unit 230 of the reference image generation unit 200 may be configured as a boundary line input unit 240. When the boundary line is set by user operation, a user such as a doctor determines the boundary between the abnormal region 112 and the normal region 113 and sets the boundary line. For example, as shown in FIG. 20, the boundary line may be set by drawing a boundary line 114 on an enlarged medical image 206 displayed on a tablet 241 with a touch pen 242 and issuing a boundary line input instruction using a boundary line input button (not shown) or via voice. When a boundary line input instruction is issued, position information of the boundary line is automatically acquired. With the above configuration, the boundary line identified by the doctor can be used to superimpose on the captured image. In this case too, it is preferable that the second landmark detection unit 250 detects landmarks in the enlarged medical image and obtains landmark information, and further, the association unit 260 receives boundary information from the boundary line input unit 240 and landmark information from the second landmark detection unit 250, associates the enlarged medical image, boundary line information, and landmark information, and uses the enlarged medical image as a reference image.

[0063] As shown in FIG. 21 , it is preferable that the reference image 111 is a medical image captured by illuminating the subject with special light, and the captured image 121 is a medical image captured by illuminating the subject with normal light. Special light is an illumination light that makes blood vessels and ductal structures more visible, making it easier to recognize the characteristic vascular arrangements and pit patterns of tumors. Therefore, medical images captured using special light make it easier to distinguish the boundary between abnormal and normal regions. For the above reasons, it is preferable to use a medical image captured using special light as the reference image. On the other hand, normal light has a natural color, and normal light is used to illuminate the subject during ESD treatment. Therefore, it is preferable that the captured image be a medical image captured using normal light. With the above configuration, it is possible to superimpose the boundary line of the reference image, in which the boundary between the abnormal and normal regions is accurately distinguished using special light, onto the captured image captured using normal light.

[0064] When superimposing a boundary line associated with a reference image onto a captured image, the captured image in which the landmark is detected and the captured image in which the boundary line is superimposed may be the same or different, depending on the processing speed at which the first landmark detection unit 120 detects landmarks from the captured image. It is preferable that a processor with a high processing speed for detecting landmarks be used, and that the captured image in which the landmark is detected and the captured image in which the boundary line is superimposed are captured at the same time. A specific explanation is given below. As illustrated in FIG. 22, if the group of captured images 160 acquired in chronological order are acquired in the order of time t-2, time t-1, and time t, the captured image acquired at time t is designated as captured image A161. If the processing speed for detecting landmarks is high, both the captured image 121 in which the landmark is detected and the captured image in which the boundary line associated with the reference image 111 is superimposed are designated as captured image A161. In this case, a superimposed image in which an accurate boundary line is superimposed can be obtained with high accuracy in real time.

[0065] On the other hand, a processor with a low processing speed for detecting landmarks may be used, and the captured image for detecting landmarks and the captured image for superimposing a boundary line may be captured at different times. Specifically, as illustrated in FIG. 23 , if a group of captured images 160 acquired in chronological order is acquired in the order of time t-2, time t-1, time t, and time t+1, the captured image acquired at time t is captured image A161, and the captured image acquired at time t+1 is captured image B162. If the processing speed for detecting landmarks is low, the captured image 121 in which the landmark is detected is captured image A161, and the captured image for superimposing a boundary line associated with the reference image 111 is captured image B162. In this case, even if a low-speed processor is used, a superimposed image with an accurate boundary line superimposed can be obtained in almost real time while maintaining a certain degree of accuracy.

[0066] 21 and 22, one frame of photographed image is acquired at time t-2, time t-1, time t, and time t+1, but multiple frames of photographed image may be acquired between each time. For example, there may be a difference of several frames between time t and time t+1.

[0067] In the boundary display mode, calculation of the degree of match between landmarks in the captured image and the reference image continues until an end command is received. In the boundary display mode, landmarks are detected sequentially for captured images acquired in real time, and the degree of match with the reference image is calculated. If the degree of match is equal to or greater than a threshold, a boundary is superimposed on the captured image. This series of operations for displaying the boundary line ends, for example, when marking or incising the mucosa during ESD is completed, or when an end command is received via the user interface 19. In addition, a change in mode using the mode selector switch 12f is also considered an end command, and calculation of the degree of match is terminated.

[0068] The flow of the boundary line display mode is shown in the flowchart of FIG. 24. First, the captured image acquired by the captured image acquisition unit 60 is input to the captured image input unit 100 (ST101). Next, the first landmark detection unit 120 detects landmarks in the captured image (ST102), and the matching calculation unit 130 calculates the matching degree, which is a value indicating the extent to which landmarks associated with the reference image are included in the landmarks detected from the captured image (ST103). If the matching degree is less than a threshold, the process proceeds to a selection of whether to issue an end instruction. On the other hand, if the matching degree is equal to or greater than the threshold (ST104), a correspondence relationship is estimated, which estimates which position in the captured image corresponds to the boundary line associated with the reference image (ST105), and a superimposed image is generated in which the boundary line is superimposed on the captured image (ST106). To end the generation of the superimposed image in which the boundary line is superimposed, an end instruction is issued (ST07), and the boundary line display mode ends.

[0069] 25, the display control unit 150 preferably generates a display image 170 and displays the superimposed image 151 in a first display section 171 of the display image 170 and the reference image 111 in a second display section 172. Note that the display method of the first display section 171 and the second display section 172 is not limited to this. For example, they may be arranged vertically instead of horizontally. Furthermore, the endoscope system 10 may include a first display (not shown) and a second display (not shown) different from the first display, and the superimposed image 151 and the reference image 111 may be displayed on different displays.

[0070] The following describes the reference image generation mode for generating a reference image from a captured image. The method for generating a reference image from a magnified medical image has been described above (see FIG. 18). In the reference image generation mode, a reference image is generated from a magnified medical image, in particular, by taking a captured image acquired in real time as a close-up magnified image so that the boundary between abnormal and normal areas can be distinguished.

[0071] In the reference image generation mode, as shown in FIG. 26, the photographed image input unit 100 transmits the input medical image (photographed image) to the magnified medical image input unit 201 of the reference image generation unit 200. The magnified photographed image is preferably a still image acquired using the still image acquisition instruction switch 12h. The subsequent steps are the same as the method for generating a reference image from a magnified medical image (see FIG. 18). This will be explained below. The magnified medical image input unit 201 transmits the magnified photographed image to the abnormality identification unit 220. As shown in FIG. 27, the abnormality identification unit 220 identifies an abnormal region 112 and a normal region 113 in the magnified photographed image 208, and the boundary setting unit 230 sets a boundary line 114 and acquires position information of the boundary line. Next, the second landmark detection unit 250 detects landmarks (indicated by circles 122) from the magnified photographed image 208 and acquires position information of the landmarks and their positional relationships. The associating unit 260 receives boundary line information (position information of boundary lines) from the boundary line setting unit 230, and further receives landmark information (position information of landmarks and the positional relationship of the landmarks) from the second landmark detection unit 250. The associating unit 260 associates the enlarged photographed image, boundary line information, and landmark information, and sets the enlarged photographed image 208 as a reference image. The reference image generated from the enlarged photographed image 208 is transmitted from the reference image generating unit 200 to the reference image recording unit 110 and recorded therein.

[0072] Furthermore, the enlarged photographed image 208 associated with the boundary information and landmark information may be sent to the enlarged medical image combining unit 210 and used as a new reference image to combine with another enlarged photographed image associated with the boundary information and landmark information. With the above configuration, a new reference image can be generated from the photographed image, and the boundary line of the newly generated reference image can be used to superimpose on the photographed image.

[0073] Hereinafter, a boundary update mode will be described in which a boundary line is set from a captured image and updated while a superimposed image in which a boundary line associated with a reference image is superimposed on a captured image is displayed. The boundary update mode is a mode in which the boundary line can be updated at the user's instruction when the boundary line displayed in the superimposed image differs from the boundary line perceived by the user observing the superimposed image. This mode is effective, for example, when an endoscopic examination is performed again the day after an endoscopic examination in which a reference image was obtained, and a captured image is obtained from the same region as the reference image, and the tumor infiltration range has expanded, and a new updated boundary line is desired.

[0074] In the boundary line update mode, first, a superimposed image is generated by superimposing a boundary line associated with a reference image on a photographed image, following the same procedure as in the boundary line display mode (see the flowchart in FIG. 23). At this time, an update instruction is sent to the medical image processing device 11 via the user interface 19 at any timing when it is desired to update the boundary line. FIG. 28 is a block diagram showing the functions within the medical image processing device 11, and FIGS. 29 and 30 are explanatory diagrams of the boundary line update mode using an image diagram. When an update instruction is received, as shown in FIG. 28, the photographed image input unit 100 sends the photographed image 121 used to generate the superimposed image 151 shown in FIG. 29 on which the boundary line 114 is superimposed (in FIG. 29, the photographed image on which the landmark is detected and the photographed image on which the boundary line is superimposed are the same photographed image) as a magnified photographed image 209 to the magnified medical image input unit 201 of the reference image generation unit 200. 30 , the abnormality identification unit 220 identifies the abnormal region 112 and the normal region 113 in the enlarged captured image 209, and the boundary setting unit 230 sets an updated boundary line 115 (shown by a two-dot chain line) and acquires position information of the updated boundary line. Next, the display control unit 150 receives the position information of the updated boundary line from the boundary setting unit 230, and superimposes the updated boundary line 115 on the captured image in addition to the boundary line 114 to generate an updated boundary line superimposed image 270, and displays the updated boundary line superimposed image 270 on the display 17.

[0075] When the user views the updated boundary line superimposed image 270 and determines that the updated boundary line may be determined as a new boundary line to be associated with the photographed image, the user transmits an update determination instruction to the medical image processing device 11 via the user interface 19. When the update determination instruction is given, the display control unit 150 updates the boundary line 114 superimposed on the photographed image 121 by setting the updated boundary line 115 (shown by a two-dot chain line) as a determined updated boundary line 116 (shown by a dotted line), as shown in FIG. 31 , and generates a determined updated boundary line superimposed image 271 to display on the display 17.

[0076] Furthermore, when an update determination instruction is given, the enlarged photographed image in which the updated boundary line has been set may be used as the reference image. When an update determination instruction is given, the second landmark detection unit 250 may detect landmark codes from the enlarged photographed image code and acquire position information of the landmarks and the positional relationship of the landmarks. The associating unit 260 receives position information of the updated boundary line from the boundary line setting unit 230, and further receives landmark information from the second landmark detection unit 250. The associating unit 260 associates the enlarged photographed image, information on the updated boundary line, and information on the landmarks, and sets the enlarged photographed image as the reference image. The reference image generated from the enlarged photographed image is transmitted from the reference image generating unit 200 to the reference image recording unit 110 and recorded.

[0077] In addition, the enlarged photographed image associated with the boundary information and landmark information may be sent to the enlarged medical image combining unit 210 and used to combine with other enlarged photographed images. With the above configuration, a reference image can be newly generated from the photographed image, and the updated boundary line of the newly generated reference image can be used to superimpose on the photographed image as a boundary line.

[0078] In the above embodiment, a flexible endoscope is used as the endoscope 12, but the present invention is also suitable when a rigid endoscope (laparoscopy, rigid endoscope) used in surgery, etc. is used. When a flexible endoscope is used, photographed images of the surface mucosa of the observation object viewed from the lumen side of the hollow organ are obtained. When a rigid endoscope is used, photographed images of the observation object viewed from the serous membrane side are obtained.

[0079] 32, when a rigid endoscope is used as the endoscope, an endoscope system 300 includes a medical image processing device 311, an endoscope 312, a light receiving unit 330, a light source device 314, a processor device 315, a display 317, and a user interface 319. Illumination light from the light source device 314 enters the endoscope 312 via a light guide 323 and illuminates the object to be observed inside the abdominal cavity.

[0080] 33, the light receiving unit 330 includes a spectroscopic element 331 such as a dichroic mirror that disperses light from the endoscope 312, and an imaging element 332 such as a CMOS image sensor that senses the dispersed light and captures an image, and outputs an image signal based on the light reflected from the object of observation. There may be one or more spectroscopic elements 331, or there may be no spectroscopic elements 331. At least one imaging element 332 is provided, and there may also be multiple imaging elements 332.

[0081] In surgical procedures using a rigid endoscope, ICG fluorescence imaging is sometimes used to identify sentinel lymph nodes, evaluate blood flow, and determine the extent of resection. In ICG fluorescence imaging, the biological half-life of ICG is approximately three minutes, so the period during which ICG can be visualized and observed after intravenous administration is limited, necessitating repeated ICG administration. On the other hand, when the present invention is used in surgical procedures using a rigid endoscope (12), ICG is administered once, and the boundary setting unit 230 sets the boundary of the observation target during the period during which the observation target can be evaluated. The second landmark detection unit 250 acquires landmark information and generates a reference image. This makes it possible to subsequently reproduce the boundary in images captured in real time without administering ICG.

[0082] In the present embodiment, the medical image processing device 11 is connected to the endoscopic system 10. However, the present invention is not limited to this example, and other medical devices, such as an ultrasound imaging device or a radiographic imaging device, may also be used. Furthermore, part or all of the captured image acquisition unit 60 and / or the first central control unit 55 of the endoscopic system 10 may be provided in an image processing device that communicates with the processor device 15 and cooperates with the endoscopic system 10. For example, they may be provided in a diagnosis support device that acquires images captured by the endoscope 12 directly from the endoscopic system 10 or indirectly from a PACS. Furthermore, part or all of the captured image acquisition unit 60 and / or the first central control unit 55 of the endoscopic system 10 may be provided in a medical service support device that is connected via a network to various examination devices, such as a first examination device, a second examination device, ..., an Nth examination device, including the endoscopic system 10.

[0083] In this embodiment, the hardware configuration of processing units that perform various processes, such as the captured image acquisition unit 60, the captured image input unit 100, the reference image recording unit 110, the first landmark detection unit 120, the matching degree calculation unit 130, the boundary line position estimation unit 140, the display control unit 150, and the reference image generation unit 200, is made up of various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to function as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for performing various processes.

[0084] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which a single processor is configured with a combination of one or more CPUs and software, as typified by client or server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a System on Chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0085] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit formed by combining circuit elements such as semiconductor elements, and the hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). [Explanation of symbols]

[0086] 10,300 Endoscopy System 11, 311 Medical imaging equipment 12 Endoscope (flexible endoscope) 12a Insertion part 12b Operation section 12c curved section 12d Tip 12e Angle Knob 12th floor Mo Mode switch 12h Still image acquisition command switch 12i Zoom control 12j forceps mouth 14, 314 Light source device 15,315 processor unit 17,317 Display 19, 319 User Interface 20 Light source section 20a V-LED 20b B-LED 20c G-LED 20d R-LED 21 Light Source Processor 22 Optical path coupling section 23 Light Guide 30a illumination optical system 30b Imaging optical system 31 Lighting lens 41 Objective Lens 42 Zoom Lens 43 Image sensor 44 Imaging processor 55 First Central Control Department 60 Image acquisition unit 100 Photographed image input unit 101 Second Central Control Department 110 Reference image recording unit 111 Reference Image 112 Abnormal area 113 Normal area 114 Borderline 115 Update border 116 Update Decision Boundary 120 First landmark detection unit 121 images 122, 122a, 122b, 122c, 122d, 122e, 122f, 122g, 122h, 122i, 122j, 122k, 122l, 122m, 122n yen 123 Digestive tract folds 124, 127 blood vessels 125 Lesions 126 Link Line 130 Matching degree calculation unit 140 Boundary line position estimation part 150 Display control unit 151 Superimposed Images 160 images 161 Photo A 162 Photographed Image B 170 display images 171 First display area 172 Second display area 200 Reference image generation unit 201 Enlarged Medical Image Input Unit 202, 203, 206 Magnified medical images 204 First Enlarged Medical Image 205 Second Enlarged Medical Image 208, 209 Enlarged images 210 Enlarged Medical Image Combination Unit 220 Abnormality Identification Unit 230 Boundary Setting Section 240 Boundary Input Section 241 tablets 242 Touch Pen 250 Second landmark detection unit 260 Association section 270 Updated boundary line superimposed image 271 Decision Update Boundary Overlay Image 312 Endoscopes (rigid endoscopes) 330 Light receiving section 331 Spectroscopic element 332 Image sensor

Claims

1. 1. A medical imaging device comprising a processor, The processor: Obtaining medical images of the subject using an endoscope, acquiring a reference image, which is the medical image in which boundary information associated with a boundary line that is a boundary between an abnormal region and a normal region and landmark information associated with a landmark that is a characteristic structure of the subject are associated; Acquire a captured image, which is the medical image captured in real time; Detecting the landmark from the captured image; calculating a degree of coincidence between the landmark included in the reference image and the landmark included in the captured image; If the degree of match is equal to or greater than a threshold, a correspondence relationship between the reference image and the captured image is estimated based on the landmark information included in the reference image and the captured image; generating a superimposed image in which the boundary line associated with the reference image is superimposed on the captured image based on the correspondence relationship; A medical image processing device, wherein the degree of match is a ratio of the number of landmarks included in the captured image to the number of landmarks included in the reference image.

2. 2. The medical image processing apparatus according to claim 1, wherein the photographed image and the reference image are medical images photographed at the same magnification.

3. A medical image processing device comprising a processor, The processor: Obtaining medical images of the subject using an endoscope, acquiring a reference image, which is the medical image in which boundary information associated with a boundary line that is a boundary between an abnormal region and a normal region and landmark information associated with a landmark that is a characteristic structure of the subject are associated; Acquire a captured image, which is the medical image captured in real time; Detecting the landmark from the captured image; calculating a degree of coincidence between the landmark included in the reference image and the landmark included in the captured image; estimating a correspondence between the reference image and the captured image based on the degree of match and the landmark information included in the reference image and the captured image; generating a superimposed image in which the boundary line associated with the reference image is superimposed on the captured image based on the correspondence relationship; A medical image processing apparatus, wherein the photographed image is the medical image photographed from a distance, and the reference image is the medical image photographed from a close distance.

4. A medical image processing device comprising a processor, The processor: Obtaining medical images of the subject using an endoscope, acquiring a reference image, which is the medical image in which boundary information associated with a boundary line that is a boundary between an abnormal region and a normal region and landmark information associated with a landmark that is a characteristic structure of the subject are associated; Acquire a captured image, which is the medical image captured in real time; Detecting the landmark from the captured image; calculating a degree of coincidence between the landmark included in the reference image and the landmark included in the captured image; estimating a correspondence between the reference image and the captured image based on the degree of match and the landmark information included in the reference image and the captured image; generating a superimposed image in which the boundary line associated with the reference image is superimposed on the captured image based on the correspondence relationship; the photographed image is a medical image photographed so as to include a part of the abnormal region, The reference image is the medical image including the entire abnormal region, a medical image processing device that generates the superimposed image by superimposing a portion of the boundary line associated with the reference image on the captured image;

5. A medical image processing device comprising a processor, The processor: Obtaining medical images of the subject using an endoscope, acquiring a reference image, which is the medical image in which boundary information associated with a boundary line that is a boundary between an abnormal region and a normal region and landmark information associated with a landmark that is a characteristic structure of the subject are associated; Acquire a captured image, which is the medical image captured in real time; Detecting the landmark from the captured image; calculating a degree of coincidence between the landmark included in the reference image and the landmark included in the captured image; estimating a correspondence between the reference image and the captured image based on the degree of match and the landmark information included in the reference image and the captured image; generating a superimposed image in which the boundary line associated with the reference image is superimposed on the captured image based on the correspondence relationship; A medical image processing device in which the reference image is generated by joining together enlarged medical images, which are medical images in which a portion of the abnormal area is photographed in close-up.

6. The processor: When the reference image is composed of a first enlarged medical image and a second enlarged medical image captured at a position different from that of the first enlarged medical image, A medical image processing device as described in claim 5, which generates the reference image by connecting the first enlarged medical image and the second enlarged medical image based on a common relationship between the landmark information and the boundary information associated with the first enlarged medical image and the landmark information and the boundary information associated with the second enlarged medical image.

7. The processor: distinguishing between the abnormal region and the normal region; The medical image processing apparatus according to claim 1 , wherein the boundary line is set.

8. The medical image processing apparatus according to claim 1 , wherein the boundary line is set by a user operation.

9. the reference image is the medical image captured by illuminating the subject with special light, 9. The medical image processing apparatus according to claim 1, wherein the photographed image is a medical image photographed by illuminating the subject with normal light.

10. The medical image processing apparatus according to claim 1 , wherein the captured image from which the landmark is detected and the captured image on which the boundary line is superimposed are captured at the same time.

11. A medical imaging device comprising a processor, The processor: Obtaining medical images of the subject using an endoscope, acquiring a reference image, which is the medical image in which boundary information associated with a boundary line that is a boundary between an abnormal region and a normal region and landmark information associated with a landmark that is a characteristic structure of the subject are associated; Acquire a captured image, which is the medical image captured in real time; Detecting the landmark from the captured image; calculating a degree of coincidence between the landmark included in the reference image and the landmark included in the captured image; estimating a correspondence between the reference image and the captured image based on the degree of match and the landmark information included in the reference image and the captured image; generating a superimposed image in which the boundary line associated with the reference image is superimposed on the captured image based on the correspondence relationship; A medical image processing apparatus, wherein the captured image for detecting the landmark and the captured image on which the boundary line is superimposed are captured at different times.

12. A medical imaging device comprising a processor, The processor: Obtaining medical images of the subject using an endoscope, acquiring a reference image, which is the medical image in which boundary information associated with a boundary line that is a boundary between an abnormal region and a normal region and landmark information associated with a landmark that is a characteristic structure of the subject are associated; Acquire a captured image, which is the medical image captured in real time; Detecting the landmark from the captured image; calculating a degree of coincidence between the landmark included in the reference image and the landmark included in the captured image; estimating a correspondence between the reference image and the captured image based on the degree of match and the landmark information included in the reference image and the captured image; generating a superimposed image in which the boundary line associated with the reference image is superimposed on the captured image based on the correspondence relationship; The medical image processing apparatus continues calculating the degree of coincidence until an instruction to end the calculation is given.

13. the processor generates an image for display; Displaying the superimposed image in a first display section of the display image; The medical image processing apparatus according to claim 1 , wherein the control is performed to display the reference image in a second display section different from the first display section of the display image.

14. A medical imaging device comprising a processor, The processor: Obtaining medical images of the subject using an endoscope, acquiring a reference image, which is the medical image in which boundary information associated with a boundary line that is a boundary between an abnormal region and a normal region and landmark information associated with a landmark that is a characteristic structure of the subject are associated; Acquire a captured image, which is the medical image captured in real time; Detecting the landmark from the captured image; calculating a degree of coincidence between the landmark included in the reference image and the landmark included in the captured image; estimating a correspondence between the reference image and the captured image based on the degree of match and the landmark information included in the reference image and the captured image; generating a superimposed image in which the boundary line associated with the reference image is superimposed on the captured image based on the correspondence relationship; The processor: an enlarged photographed image is acquired as the photographed image; distinguishing the abnormal area and the normal area included in the enlarged photographed image and setting the boundary line; Detecting the landmark from the enlarged photographed image; The medical image processing apparatus associates the boundary line information associated with the boundary line and the landmark information associated with the landmark with the enlarged photographed image, and uses the resulting image as the reference image.

15. The processor: generating the superimposed image; When an update instruction is given, the abnormal region and the normal region included in the captured image related to the superimposed image on which the boundary line is superimposed are identified, and an updated boundary line is set using the boundary between the abnormal region and the normal region as the boundary line, If there is an update decision instruction, The medical image processing apparatus according to claim 1 , wherein the updated boundary line is used as a determined updated boundary line, and the boundary line superimposed on the captured image is updated.

16. acquiring a medical image of a subject by an endoscope; acquiring a reference image, which is the medical image in which boundary information associated with a boundary line that is a boundary between an abnormal region and a normal region and landmark information associated with a landmark that is a characteristic structure of the subject are associated; acquiring a captured image, the medical image being captured in real time; detecting the landmark from the captured image; calculating a degree of coincidence between the landmark included in the reference image and the landmark included in the captured image; If the degree of match is equal to or greater than a threshold, estimating a correspondence between the reference image and the captured image based on the landmark information included in the reference image and the captured image; generating a superimposed image in which the boundary line associated with the reference image is superimposed on the captured image based on the correspondence relationship; A method for operating a medical image processing device, wherein the degree of match is the ratio of the number of landmarks included in the captured image to the number of landmarks included in the reference image.

17. An endoscope system comprising: the medical image processing device according to claim 1; and an endoscope.

Images