Endoscope system, medical image processing device, and operation method thereof

Abstract

To provide an endoscope system capable of discriminating and outputting a site of a subject that can be incised safely, a medical image processing device, and an operation method thereof.SOLUTION: A medical image processing device includes a processor. The processor executes control to acquire an examination image capturing a subject with an endoscope, discriminate a site suitable for incision in the subject included in the examination image on the basis of the examination image, and output information on the site suitable for incision related to the site suitable for incision. The discrimination of the information on the site suitable for incision is executed using an image for learning associated with the position of a muscle layer in the subject.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an endoscope system for assisting surgery such as endoscopic submucosal dissection, a medical image processing device, and an operating method thereof. [Background technology]

[0002] Endoscopic submucosal dissection (ESD) has made it possible to remove tumors that are too large for endoscopic mucosal resection (EMR), eliminating the need for highly invasive surgical procedures. ESD has the advantage of being minimally invasive because it is performed under an endoscope. However, it also has the disadvantage of the risk of accidentally perforating an organ.

[0003] To prevent perforation, a technology is known in which, based on endoscopic images acquired during an examination, situations in which there is a possibility of perforation that would cut the muscle layer are judged as "dangerous" and situations in which there is no possibility of perforation being caused as "safe," and in the case of "danger," a signal is sent to an electric scalpel device to stop the output of the electric scalpel device (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-3453 Summary of the Invention [Problem to be solved by the invention]

[0005] ESD involves processes such as marking the area around the tumor, local injection into the submucosal layer, incision of the mucosa, dissection of the submucosal layer, and hemostasis. Because ESD requires advanced techniques to properly remove the lesion without causing perforation, there is a need for technology to support surgeons who are inexperienced in the procedure.

[0006] An object of the present invention is to provide an endoscope system, a medical image processing device, and an operating method thereof that can identify and output a region in a subject that can be safely incised. [Means for solving the problem]

[0007] The medical image processing device of the present invention is a medical image processing device equipped with a processor, which acquires an examination image of a subject taken with an endoscope, identifies incision-suitable areas in the subject included in the examination image based on the examination image, and controls the output of incision-suitable area information regarding the incision-suitable areas, and the identification of the incision-suitable area information is performed using a learning image associated with the position of the muscle layer in the subject.

[0008] The processor identifies areas unsuitable for incision in the subject included in the examination image and controls the output of incision unsuitable area information regarding the areas unsuitable for incision, and it is preferable that the identification of the incision unsuitable area information is performed using a learning image associated with the location of fibrosis in the subject.

[0009] The processor preferably distinguishes between suitable incision sites and unsuitable incision sites based on the degree of suitability, and outputs suitable incision site information as the degree of suitability. A learning model is preferably generated using the learning images.

[0010] The training image preferably includes a subject to which a local injection liquid has been injected. The local injection liquid preferably includes a staining liquid. The staining liquid is preferably indigo carmine, and the training image preferably has a concentration of indigo carmine associated therewith.

[0011] The training images are preferably associated with the presence or absence of cauterization scars and / or blood coagulation in the subject.

[0012] It is preferable that the learning image be associated with the position of a hood attached to the tip of the endoscope in the subject.

[0013] It is preferable that the learning image is associated with the distance from the submucosa to the muscle layer and / or the distance from the submucosa to the lesion in the subject.

[0014] The processor preferably generates a first display image showing the suitable incision site based on the suitable incision site information, and controls the display to display the first display image superimposed on the examination image.

[0015] The first display image preferably indicates the incision suitable site with a color, a symbol, or a graphic. The graphic is preferably a line.

[0016] The processor preferably performs control to display the distance from the submucosal layer to the muscular layer on the first display image.

[0017] The processor preferably generates a second display image showing the incision inappropriate site based on the incision inappropriate site information, and controls the display to superimpose the second display image on the examination image.

[0018] The processor preferably generates a perforation risk image showing the degree of conformance as an image, and performs control to superimpose the perforation risk image on the inspection image and display it on the display.

[0019] Preferably, the processor controls to notify by sound or visual notification when the inspection image includes the incision inappropriate site.

[0020] The processor preferably generates incision support information corresponding to the incision suitable region information based on the incision suitable region information, and controls the display to superimpose the incision support information on the examination image.

[0021] The operating method of the medical image processing device of the present invention comprises the steps of acquiring an examination image of a subject photographed with an endoscope, and performing a control step of identifying incision-suitable sites in the subject included in the examination image and outputting incision-suitable site information regarding the incision-suitable sites, where the identification of the incision-suitable site information is performed using a learning image associated with the position of the muscle layer in the subject.

[0022] An endoscope system of the present invention includes the above-described medical image processing device and an endoscope. [Effects of the Invention]

[0023] According to the present invention, it is possible to provide an endoscope system, a medical image processing device, and an operating method thereof that can identify and output a region in a subject that can be safely incised. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is an explanatory diagram of a configuration of an endoscope system. [Figure 2] FIG. 2 is a block diagram showing the functions of the endoscope system. [Figure 3] 1 is a graph showing the spectra of violet light V, blue light B, green light G, and red light R. [Figure 4] FIG. 10 is an explanatory diagram showing a first light emission pattern in an image analysis mode. [Figure 5] FIG. 10 is an explanatory diagram showing a second light emission pattern in an image analysis mode. [Figure 6] 1 is a graph showing the spectral transmittance of each color filter of the image sensor. [Figure 7] FIG. 10 is a block diagram showing the function of the classifier to output incision suitable site information. [Figure 8] FIG. 10 is an image diagram showing an example of a display image. [Figure 9] FIG. 10 is an image diagram showing an example of a training image including an area of nearly exposed muscle layer. [Figure 10] FIG. 10 is an image diagram showing an example of a learning image including an area where the muscle layer is covered by the submucosal layer. [Figure 11] FIG. 10 is a block diagram showing the function of the classifier to output incision unsuitable site information. [Figure 12] FIG. 10 is an image diagram showing an example of a learning image including fibrosis. [Figure 13] FIG. 10 is a block diagram showing the function of the classifier to output the fitness. [Figure 14] FIG. 10 is an image diagram showing an example of a learning image in which no local injection liquid is injected into the submucosal layer. [Figure 15] FIG. 10 is an image diagram showing an example of a learning image in which a local injection liquid is locally injected into the submucosal layer. [Figure 16] FIG. 10 is an image diagram showing an example of a learning image in which a local injection solution containing a low concentration of indigo carmine is locally injected into the submucosal layer. [Figure 17] FIG. 10 is an image diagram showing an example of a learning image in which a local injection solution containing a high concentration of indigo carmine is locally injected into the submucosal layer. [Figure 18] FIG. 10 is an image diagram showing an example of a learning image including a cauterization scar and blood clots. [Figure 19] FIG. 10 is an image diagram showing an example of a training image in which the edge of a hood is included in the field of view. [Figure 20] FIG. 2 is a block diagram showing functions of a display image generating unit. [Figure 21] FIG. 2 is an image diagram showing an example of a first display image. [Figure 22] FIG. 10 is an image diagram showing an example of a second display image. [Figure 23] FIG. 10 is an image diagram showing an example of a perforation risk image. [Figure 24] FIG. 10 is an image diagram showing an example of an image displaying the distance between tissues. [Figure 25] FIG. 3 is a block diagram showing the function of a notification control unit. [Figure 26] FIG. 10 is an image diagram showing an example of a warning display screen in which a frame is provided around the inspection image. [Figure 27] FIG. 10 is an image diagram showing an example of a warning display screen using a warning mark. [Figure 28] FIG. 2 is a block diagram showing functions of an incision support information generating unit. [Figure 29]FIG. 10 is an image diagram showing an example of an incision support information screen. DETAILED DESCRIPTION OF THE INVENTION

[0025] As shown in FIG. 1, a medical image processing device 11 is connected to an endoscope system 10 via a processor device 15. The endoscope system 10 includes an endoscope 12, a light source device 14, a processor device 15, the medical image processing device 11, a display 17, and a user interface 19. The endoscope 12 is optically connected to the light source device 14 and electrically connected to the processor device 15. The endoscope 12 has an insertion section 12a that is inserted into the body of an observation subject, an operation section 12b provided at the base end of the insertion section 12a, and a bending section 12c and a distal end section 12d provided at the distal end of the insertion section 12a. The bending section 12c is bent by operating an angle knob 12e of the operation section 12b. The distal end section 12d is directed in a desired direction by bending the bending section 12c. A forceps channel (not shown) for inserting a treatment tool or the like is provided from the insertion section 12a to the distal end section 12d. The treatment tool is inserted into the forceps channel through a forceps port 12j.

[0026] The endoscope 12 is provided inside with an optical system for forming an image of a subject and an optical system for irradiating the subject with illumination light. The operation unit 12b is provided with an angle knob 12e, an observation mode selector switch 12f, an image analysis mode selector switch 12g, a still image acquisition instruction switch 12h, and a zoom operation unit 12i. The observation mode selector switch 12f is used to switch the observation mode. The still image acquisition instruction switch 12h is used to instruct acquisition of a still image of the observation target. The zoom operation unit 12i is used to operate the zoom lens 42.

[0027] The light source device 14 generates illumination light. The display 17 outputs and displays an examination image and an image in which incision suitable area information and / or incision unsuitable area information described below is superimposed on the examination image. The user interface 19 has a keyboard, mouse, touchpad, microphone, etc., and has the function of accepting input operations such as function settings. The processor device 15 controls the endoscope system 10 and performs image processing on image signals transmitted from the endoscope 12.

[0028] 2, the light source device 14 includes a light source unit 20 and a light source processor 21 that controls the light source unit 20. The light source unit 20 has, for example, multiple semiconductor light sources that are turned on or off and, when turned on, control the light emission amount of each semiconductor light source to emit illumination light that illuminates the observation target. The light source unit 20 has four color LEDs: a V-LED (Violet Light Emitting Diode) 20a, a B-LED (Blue Light Emitting Diode) 20b, a G-LED (Green Light Emitting Diode) 20c, and an R-LED (Red Light Emitting Diode) 20d. The light source unit 20 may be built into the endoscope 12, and the light source control unit may be built into the endoscope 12 or the processor device 15.

[0029] As shown in FIG. 3, the V-LED 20a emits violet light V with a central wavelength of 405±10 nm and a wavelength range of 380 to 420 nm. The B-LED 20b emits blue light B with a central wavelength of 450±10 nm and a wavelength range of 420 to 500 nm. The G-LED 20c emits green light G with a wavelength range of 480 to 600 nm. The R-LED 20d emits red light R with a central wavelength of 620 to 630 nm and a wavelength range of 600 to 650 nm.

[0030] The endoscope system 10 has three observation modes: a first illumination observation mode, a second illumination observation mode, and an image analysis mode. When the observation mode selector switch 12f is pressed, the mode is switched via the image processing selector 54 (see FIG. 2).

[0031] In the first illumination observation mode, the object of observation is illuminated with normal light (first illumination light), such as white light, and an image is taken, thereby displaying a first illumination light image with natural coloring on the display 17. In the second illumination observation mode, the object of observation is illuminated with special light (second illumination light), which has a wavelength band different from that of the normal light, and an image is taken, thereby displaying a second illumination light image, which emphasizes a specific structure, on the display 17. The first illumination light image and the second illumination light image are types of inspection images.

[0032] The light used when performing ESD is usually the first illumination light. The second illumination light may be used when it is desired to confirm the extent of infiltration of the lesion before performing ESD. The training image used for training the classifier 110 (see FIG. 2), which will be described later, is preferably a first illumination light image. Also, a second illumination light image, in which the lesion is particularly emphasized, may be associated with various information and used as a training image. The test image and the training image are types of medical images.

[0033] The light source processor 21 controls the V-LED 20a, B-LED 20b, G-LED 20c, and R-LED 20d. By independently controlling each of the LEDs 20a to 20d, the light source processor 21 can emit purple light V, blue light B, green light G, or red light R with different light intensities. In addition, in the first illumination observation mode, the light source processor 21 controls each of the LEDs 20a to 20d to emit white light in which the light intensity ratio between the purple light V, blue light B, green light G, and red light R is Vc:Bc:Gc:Rc, where Vc, Bc, Gc, and Rc>0.

[0034] Furthermore, in the second illumination observation mode, the light source processor 21 controls the LEDs 20a to 20d to emit special light that is short-wavelength narrowband light, consisting of violet light V, blue light B, green light G, and red light R, in a light intensity ratio of Vs:Bs:Gs:Rs. The light intensity ratio Vs:Bs:Gs:Rs differs from the light intensity ratio Vc:Bc:Gc:Rc used in the first illumination observation mode and is determined appropriately depending on the observation purpose.

[0035] In addition, in the image analysis mode, the light source processor 21 switches between first illumination light and second illumination light, which have different emission spectra. Specifically, the light source processor 21 switches between the first illumination light and the second illumination light in two emission patterns: a first illumination pattern in which the number of frames in each first illumination period is the same as that in each first illumination period, as shown in Fig. 4, and a second illumination pattern in which the number of frames in each first illumination period is different as shown in Fig. 5. In the figures, "time" indicates the direction of time passage.

[0036] When performing ESD, a first illumination light image and a second illumination light image may be obtained by automatically switching between the first illumination light and the second illumination light, and the first illumination light image, in which the surgical field is visually recognized in natural colors, and the second illumination light image, in which the lesion is emphasized, may be aligned and associated with each other to be used as learning images for training the classifier 110. By using learning images in which the positions of the lesion, muscle layer, submucosal layer, etc. are associated between the first illumination light image and the second illumination light image, a learning model that can more precisely identify the distance from the submucosal layer to the lesion can be generated.

[0037] The light emitted from each of the LEDs 20a to 20d (see FIG. 2) is incident on the light guide 23 via an optical path combining unit 22 configured with a mirror, a lens, etc. The light guide 23 propagates the light from the optical path combining unit 22 to the tip 12d of the endoscope 12.

[0038] An illumination optical system 30a and an imaging optical system 30b are provided at the distal end 12d of the endoscope 12. The illumination optical system 30a has an illumination lens 31, and illumination light propagated by the light guide 23 is irradiated onto the object of observation via the illumination lens 31. When the light source unit 20 is built into the distal end 12d of the endoscope 12, light is emitted toward the object via the illumination lens of the illumination optical system without passing through a light guide. The imaging optical system 30b has an objective lens 41 and an imaging sensor 43. Light from the object of observation irradiated with illumination light is incident on the imaging sensor 43 via the objective lens 41 and a zoom lens 42. As a result, an image of the object of observation is formed on the imaging sensor 43. The zoom lens 42 is a lens for enlarging the object of observation, and is moved between the telephoto end and the wide-angle end by operating the zoom operation unit 12i.

[0039] The imaging sensor 43 is a primary color sensor and has three types of pixels: B pixels (blue pixels) with blue color filters, G pixels (green pixels) with green color filters, and R pixels (red pixels) with red color filters. As shown in FIG. 6, the blue color filter BF transmits mainly light in the blue band, specifically light in the wavelength band of 380 to 560 nm. The transmittance of the blue color filter BF reaches its peak in the vicinity of wavelengths of 460 to 470 nm. The green color filter G The red color filter RF transmits mainly light in the red wavelength range, specifically light in the 580 to 760 nm wavelength range.

[0040] Furthermore, the imaging sensor 43 is preferably a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging processor 44 controls the imaging sensor 43. Specifically, the imaging processor 44 reads out a signal from the imaging sensor 43, causing the imaging sensor 43 to output an image signal.

[0041] A CDS / AGC (Correlated Double Sampling / Automatic Gain Control) circuit 45 (see FIG. 2) performs correlated double sampling (CDS) and automatic gain control (AGC) on the analog image signal obtained from the image sensor 43. The image signal passed through the CDS / AGC circuit 45 is converted into a digital image signal by an A / D (Analog / Digital) converter 46. The digital image signal after A / D conversion is input to the processor device 15.

[0042] In the processor device 15, the first central control unit 55, which is composed of an image control processor, operates the programs in the program memory, thereby realizing the functions of the image acquisition unit 50, DSP (Digital Signal Processor) 52, noise reduction unit 53, image processing switching unit 54, and inspection image acquisition unit 60.

[0043] The image acquisition unit 50 acquires a color image input from the endoscope 12. The color image includes a blue signal (B image signal), a green signal (G image signal), and a red signal (R image signal) output from the B pixels, G pixels, and R pixels of the imaging sensor 43. The acquired color image is transmitted to the DSP 52. The DSP 52 performs various signal processing on the received color image, such as defect correction, offset processing, demosaic processing, matrix processing, white balance adjustment, gamma conversion processing, and YC conversion processing.

[0044] The noise reduction unit 53 performs noise reduction processing, for example, by a moving average method or a median filter method, on the color image that has been subjected to YC conversion processing and the like by the DSP 52. The color image with reduced noise is input to the image processing switching unit .

[0045] The image processing switching unit 54 switches the destination of the image signal from the noise reduction unit 53 depending on the set mode. Specifically, when the first illumination observation mode is set, the image signal from the noise reduction unit 53 is input to a first illumination light image generation unit 70 of the inspection image acquisition unit 60. When the second illumination observation mode is set, the image signal from the noise reduction unit 53 is input to a second illumination light image generation unit 80. When the image analysis mode is set, the image signal from the noise reduction unit 53 is input to the first illumination light image generation unit 70 and the second illumination light image generation unit 80.

[0046] In the first illumination observation mode, the first illumination light image generation unit 70 performs image processing. In the second illumination observation mode, the second illumination light image generation unit 80 performs image processing. In the image analysis mode, the first illumination light image generation unit 70 performs image processing on the image signal obtained using the first illumination light, and the second illumination light image generation unit 80 performs image processing on the image signal obtained using the second illumination light. Image processing includes color conversion processing such as 3x3 matrix processing, tone conversion processing, and 3D LUT (Look Up Table) processing, color enhancement processing, and structure enhancement processing such as spatial frequency enhancement. The image signal that has undergone image processing is transmitted to the medical image processing device 11 as an examination image.

[0047] The examination image generated by the examination image acquisition unit 60 of the processor device 15 is transmitted to the medical image processing device 11. The medical image processing device 11 includes an image input unit 100, a classifier 110, a display image generation unit 150, a notification control unit 210, a display control unit 200, and a second central control unit 101 (see FIG. 2).

[0048] In the medical image processing device 11, the second central control unit 101, which is composed of an image analysis processor, operates the programs in the program memory, thereby realizing the functions of the image input unit 100, classifier 110, display image generation unit 150, notification control unit 210, and display control unit 200.

[0049] The examination image is transmitted to the image input unit 100 of the medical image processing device 11 (see FIG. 2). The image input unit 100 inputs the examination image to the classifier 110. The classifier 110 identifies incision-suitable regions, which are regions suitable for incision in ESD, from among the subjects included in the input examination image. Identification means distinguishing between incision-suitable regions and unsuitable regions and selecting the incision-suitable regions from the examination image. As shown in FIG. 7, the classifier 110 identifies the incision-suitable regions based on the examination image and outputs incision-suitable region information regarding the incision-suitable regions. The incision-suitable region information is used to generate a display image 113 that displays the incision-suitable regions 112 on the display 17, as shown in FIG. 8.

[0050] The classifier 110 is trained using training images. The classifier 110 is a learning model that performs learning using training images through machine learning. Training images are images in which various information is associated with medical images. Association means adding information to training images.

[0051] The incision site is the site suitable for dissecting the submucosal layer during ESD. ESD involves the following steps: (1) marking the area around the tumor, (2) local injection into the submucosal layer, (3) incision of the mucosa, (4) dissection of the submucosal layer, and (5) hemostasis. Identifying the appropriate incision site is important for (4) dissection of the submucosal layer. Local injection of a solution into the submucosal layer deeper than the marked lesion site raises the lesion from the surrounding mucosa. Then, by incising the mucosal epithelium outside the marking with a surgical instrument such as an electric scalpel, the submucosal layer beneath the mucosal epithelium becomes visible, allowing dissection of the lesion. If the incision or dissection is performed too deeply into the mucosa, the muscle layer, which lies deeper beneath the submucosa, may be reached. Dissecting deeper than this may rupture the muscle layer and the thin serous membrane surrounding the muscle layer, resulting in perforation of the digestive tract. Therefore, by identifying the muscle layer in advance, it is possible to prevent the muscle layer from being incised by mistake and to support ESD. By having the classifier 110 learn the positions of muscle layers that are not suitable for incision, it becomes possible to identify areas suitable for incision, and the results of the identification can be output as information on areas suitable for incision.

[0052] The incision suitable site information is information output by the classifier 110 that receives the test image as input. The training image used for training the classifier 110 is associated with information regarding the position of the muscle layer in the subject. The position of the muscle layer includes two-dimensional information indicating the extent of the muscle layer area in the training image, as well as three-dimensional information indicating the thickness of the submucosal layer covering the surface of the muscle layer (thickness from the submucosal layer to the muscle layer). The thicker the submucosal layer on the surface of the muscle layer, the lower the risk of incising the muscle layer, making it more suitable as a site for incision. Such a site with a thick submucosal layer can be a suitable site for incision. Conversely, the thinner the submucosal layer, the higher the risk of perforation, making it unsuitable for incision. A thin submucosal layer on the surface and a nearly exposed muscle layer poses a high risk and is therefore unsuitable for incision.

[0053] As shown in FIG. 9 , in the case of a training image 120 including a region 121 of the nearly exposed muscle layer, the region of the muscle layer is associated with the position of the muscle layer in two dimensions. Also, as shown in FIG. 10 , in the case of a training image 123 including a region 124 where the muscle layer is covered by the submucosa, the thickness of the submucosa to the muscle layer is associated with the training image as the position of the muscle layer in three dimensions. The thickness of the submucosa to the muscle layer may be semi-quantitative, such as "thin," "medium," or "thick," or may be quantitative, such as "10 μm" or "100 μm." The position of the muscle layer in the subject may also include the absence of the muscle layer, i.e., the subject does not include the muscle layer. It is preferable that a lesion 122 be included in the training image. This is because, as will be explained in detail later, both the distance from the dissection position to the muscle layer and the distance to the lesion are important in ESD when dissecting the submucosa.

[0054] The position of the muscle layer may be associated with the learning image by a skilled doctor, or may be automatically performed by a device other than the medical image processing device 11. Furthermore, information output by the classifier 110 or another classifier may be associated with the examination image and used as a learning image for training the classifier 110.

[0055] ESD can remove even relatively large lesions under an endoscope, but the procedure is difficult and there is a risk of the surgeon accidentally incising the muscle layer and resulting in perforation. It is particularly difficult for surgeons with few ESD cases to determine the location of the muscle layer on the examination image. Therefore, the classifier 110, trained using training images associated with the location of the muscle layer, automatically identifies the incision-suitable area based on the examination image and outputs the incision-suitable area information. The medical image processing device 11 controls this output, allowing for the identification of areas that can be safely incised. This configuration prevents accidental incision of the muscle layer and assists ESD.

[0056] 11, it is preferable that the classifier 110 identifies incision-unsuitable sites, which are sites unsuitable for incision in ESD, from among the subjects included in the input examination image, and output incision-unsuitable site information regarding the incision-unsuitable sites. Sites unsuitable for incision include fibrosis that cannot be distinguished from the muscle layer, muscle layer not covered by the submucosa, and submucosa that is separated from the muscle layer but is close to the lesion and may not be able to completely resect the lesion.

[0057] The training images used to train the classifier 110, which outputs information about incision-unsuitable areas, are preferably associated with the location of fibrosis. For example, as shown in FIG. 12, a training image 125 including fibrosis is associated with the location of muscle layer 126 and the location of fibrosis 127. The training image 125 preferably includes a lesion 122. Fibrosis is an area where extracellular matrix containing collagenous fibers has increased. While fibrosis itself can be incised without causing any problems, advanced fibrosis can be difficult to distinguish from muscle layer. If such fibrosis that is difficult to distinguish from muscle layer is present in the test image together with the muscle layer, there is a risk of incising the muscle layer by mistake, so it is preferable to identify it as an incision-unsuitable area. Alternatively, fibrosis that is completely distinguishable from muscle layer and has little risk of being mistaken for muscle layer may be associated as training images and trained by the classifier 110, and the classifier 110 may then identify such fibrosis as an incision-suitable area.

[0058] As shown in Figure 13, the classifier 110 preferably distinguishes between incision-suitable and incision-unsuitable regions based on the degree of suitability and outputs incision-suitable region information as the degree of suitability. The degree of suitability is higher for regions that are identified as being more safely incisable, such as regions with a thick submucosal layer on the surface of the muscle layer, and lower for regions that are identified as being less safely incisable, such as regions where the muscle layer is exposed and there is a risk of perforation. Specifically, the degree of suitability varies depending on the position of the muscle layer, the position of fibrosis, the degree of similarity of the fibrosis with the muscle layer, the distance from the submucosal layer to the muscle layer, and the distance from the submucosal layer to the lesion. By distinguishing the degree of suitability based on the evaluation of incision-suitable and incision-unsuitable regions, the degree of incision safety can be identified.

[0059] It is preferable that the learning image is associated with the position of the submucosal layer where the local injection liquid has been locally injected in the subject. Fig. 14 shows a learning image 131 where the local injection liquid has not been locally injected into the submucosal layer, and Fig. 15 shows a learning image 133 where the local injection liquid containing no staining liquid has been locally injected into the submucosal layer. In Fig. 14, the submucosal layer 132 where no local injection has been made is visually recognized as a white net. On the other hand, in Fig. 15, the learning image 133 where the local injection liquid containing no staining liquid has been locally injected, the local injection liquid containing no staining liquid is visually recognized as a white net. difference The submucosal layer 134 that has been locally injected can be seen transparently, making it easier to identify the area suitable for incision. Hereinafter, the submucosal layer that is particularly suitable for incision will be referred to as the peeled layer. The peeled layer is an area suitable for incision with a particularly high degree of suitability. An image in which the local injection liquid has been locally injected, such as that in Figure 15, makes it easier to identify the presence of the peeled layer and the thickness to the muscle layer, thereby improving the analysis accuracy of the classifier 110. Note that the training images may also include a training image 131 in which the subject has not had local injection liquid injected into the submucosal layer, such as that in Figure 14.

[0060] The training image is preferably associated with the position of the submucosal layer in the subject where a local injection solution containing a staining solution has been locally injected. The staining solution is preferably indigo carmine. When the local injection solution contains indigo carmine, the concentration of indigo carmine is preferably associated with the training image. FIG. 16 shows a training image 135 in which a local injection solution containing a low concentration of indigo carmine has been locally injected into the submucosal layer. In the training image 135 shown in FIG. 16, the submucosal layer 136 into which the local injection solution containing indigo carmine has been locally injected appears bluish and transparent, and the peeling layer 137 in which the local injection solution has accumulated and which can be dissected safely appears a darker blue than the surrounding area. The training image 135 is preferably associated with the submucosal layer 136 into which the local injection solution containing indigo carmine has been locally injected and the peeling layer 137 which can be dissected safely.

[0061] FIG. 17 shows a training image 138 in which a local injection solution containing a high concentration of indigo carmine has been locally injected. In the training image 138 shown in FIG. 17, the submucosal layer 136 into which the local injection solution containing indigo carmine has been locally injected appears bluish and transparent, as in FIG. 16, but the peeling layer 139, which can be dissected safely, appears a deeper blue than in the case of a low concentration. A local injection solution containing a high concentration of indigo carmine is, for example, a local injection solution obtained by adding 0.5 ml of indigo carmine to 20 ml of Mucoup® concentrate. This high-concentration indigo carmine local injection solution is a local injection solution in which the indigo carmine concentration has been adjusted to make the peeling layer more visible than in a regular local injection solution. The training image 138 is preferably associated with the submucosal layer 136 into which the local injection solution containing indigo carmine has been locally injected and the peeling layer 139, which can be dissected safely. In the learning images shown in Figures 16 and 17, in which a local injection solution containing indigo carmine is locally injected into the submucosal layer, the peeled layer that can be safely dissected is easily visible, thereby improving the analysis accuracy of the classifier 110.

[0062] The training images preferably include training images associated with the presence or absence of ablation scars and / or blood clots in the subject. Figure 18 shows training images 140 including ablation scars 141 and blood clots 142. The ablation scars 141 and blood clots 142 reduce the visibility of the entire examination image, including the muscle layer, fibrosis, the submucosal layer including the dissected layer, and the like. Therefore, by training the classifier 110 in advance using training images 140 including ablation scars 141 and blood clots 142, the classifier 110 can identify safe incision sites even in situations where ESD is difficult due to the presence of ablation scars 141 and blood clots 142.

[0063] The training images preferably include training images associated with the position of a hood attached to the tip of the endoscope relative to the subject. FIG. 19 shows a training image 143 in which a hood edge 144 is included in the field of view. Because a hood is often used when performing ESD, it is preferable to use only training images 143 in which the hood edge 144 is included in the field of view as training images when training the classifier 110. Furthermore, when a hood is attached, it is determined that the distance between the tip 12d of the endoscope 12 and the subject is 3 to 5 mm. Because this distance allows the subject to be clearly visible after ESD, it is preferable to use inspection images in which the hood is included in the field of view (the distance between the tip 12d of the endoscope 12 and the subject is 3 to 5 mm) as training images.

[0064] Furthermore, when a hood is attached to the distal end 12d of the endoscope 12 and an edge 144 of the hood is captured in an image, the visibility of the hood edge 144 and its outer portion is reduced. The position of the hood included in a training image 143 as shown in Fig. 19 is associated with the training image 143 and used for training the classifier 110, and the hood edge 144 and its outer portion, which have reduced visibility, are excluded from the analysis, thereby improving the analysis accuracy of the classifier 110.

[0065] Furthermore, it is preferable that the learning image be associated with the distance from the submucosa to the muscular layer and the distance from the submucosa to the lesion. A muscular layer with a thin submucosa on the surface increases the risk of perforation. Even if the submucosa is far from the muscular layer, if it is close to the lesion (within 500 μm), there is a risk that the lesion will not be completely resected. Therefore, it is preferable to identify areas where the distance from the submucosa to the muscular layer and areas where the distance from the submucosa to the lesion is close as incision-unsuitable areas and output this as incision-unsuitable area information. On the other hand, it is preferable to identify areas of the submucosa that are an appropriate distance from the muscular layer and the lesion as incision-suitable areas and output this as incision-suitable area information. It is also preferable to output the distance from the submucosa to the muscular layer as incision-suitable area information. The distance may be semi-quantitative, such as "close," "medium," or "far," or quantitative, such as "10 μm" or "100 μm."

[0066] Deep learning is preferably used for machine learning to generate a learning model, for example, a multilayer convolutional neural network. In addition to deep learning, machine learning also includes decision trees, support vector machines, random forests, regression analysis, supervised learning, semi-unsupervised learning, unsupervised learning, reinforcement learning, deep reinforcement learning, learning using neural networks, generative adversarial networks, etc.

[0067] 20 shows signals transmitted and received by the display image generating unit 150. The incision suitable area information, incision unsuitable area information, and suitability output by the classifier 110 are transmitted to the display image generating unit 150 (see FIG. 2). The display image generating unit 150 includes a first display image generating unit 160, a second display image generating unit 170, and a perforation risk image generating unit 190. The display image generating unit 150 transmits image signals to the display control unit 200.

[0068] The first display image generating unit 160 generates a first display image 161 that shows the suitable incision site in an image based on the suitable incision site information. FIG. 21 shows the first display image. In the first display image 161, it is preferable that the suitable incision site is shown in color, symbol, or figure. In particular, it is preferable that the suitable incision site is shown by a line. In FIG. 21, the peeling layer 162 is shown by a line. The first display image 161 is transmitted to the display control unit 200 and displayed on the display 17 superimposed on the examination image. With the above configuration, it is possible to proactively present to the surgeon the site that can be safely incised during ESD.

[0069] The second display image generation unit 170 generates a second display image 171 that shows the incision-unsuitable area in an image based on the incision-unsuitable area information. FIG. 22 shows the second display image 171. In FIG. 22, a muscle layer 172 and fibrosis 173 that have been identified as the incision-unsuitable area are displayed. The second display image 171 is transmitted to the display control unit 200 and displayed on the display 17 superimposed on the examination image. With the above configuration, it is possible to proactively present to the surgeon areas that are dangerous to incise during ESD.

[0070] The perforation risk image generating unit 190 generates a perforation risk image 191 that indicates the degree of suitability based on the incision suitable area information, the incision unsuitable area information, and the suitability. FIG. 23 shows the perforation risk image 191. In FIG. 23, the perforation risk image 191 is displayed as a heat map in which areas are color-coded (indicated by the density of diagonal lines) according to the suitability. For example, the muscle layer 192 and fibrosis 193, which are unsuitable for incision and have low suitability, are displayed in red, the submucosal layer 194 other than the dissection layer, which has medium suitability, is displayed in green, and the dissection layer 195, which has high suitability, is displayed in blue. The color coding is not limited to this. The probability of perforation may be displayed using the color coding as a legend. The probability of perforation may be displayed only as a numerical value. The perforation risk image 191 is transmitted to the display control unit 200 and displayed on the display 17 superimposed on the examination image. With the above configuration, during ESD, the surgeon can be informed of areas that are dangerous to dissect in addition to areas that can be safely dissected.

[0071] When the distance from the submucosal layer to the muscle layer is included in the incision suitable site information, the display control unit 200 preferably superimposes the distance 182 from the dissection layer to the muscle layer on the examination image and displays it as a tissue distance display image 181, as shown in Fig. 24. With the above configuration, it is possible to present to the surgeon the safe incision distance during ESD.

[0072] The first display image 161, the second display image 171, the tissue distance display image 181, and the perforation risk image 191 may be superimposed in combination on the examination image.

[0073] FIG. 25 shows signals transmitted and received by the notification control unit 210. The incision suitable area information, incision unsuitable area information, and suitability output by the classifier 110 are transmitted to the notification control unit 210 (see FIG. 2). The notification control unit 210 includes an audio notification control unit 220 and an alarm display control unit 230. When incision unsuitable area information is received indicating that an incision unsuitable area is included in the examination image, the audio notification control unit 220 transmits an audio notification signal and issues an alarm sound from a speaker (not shown). It is also possible to set a threshold value for the suitability for notification, and to issue an alarm sound when there is an incision unsuitable area whose suitability is below the threshold. The threshold value can be set arbitrarily. With the above configuration, the surgeon can more easily visually recognize the presence of an area that is unsuitable for incision.

[0074] 26 and 27 show a warning display screen 231 generated by the notification display control unit 230. When incision inappropriate site information is received indicating that an incision inappropriate site is included in the examination image, the notification display control unit 230 transmits a notification display signal to the display control unit 200. For example, as shown in FIG. 26, the notification display control unit 230 provides a frame 232 around the examination image to notify that a treatment tool such as an electric scalpel is near the incision inappropriate site. The form of the frame is not limited to one that surrounds the entire periphery of the examination image. Furthermore, as shown in FIG. 27, a warning mark 233 may be used for display. The form of the warning mark 233 is not limited to this. A threshold value for notification may be set for the degree of compatibility, and a notification may be displayed when there is an incision inappropriate site whose degree of compatibility is below the threshold. The threshold value can be set arbitrarily.

[0075] FIG. 28 shows signals transmitted and received by the incision support information generating unit 240. The incision support information, incision unsuitable area information, and / or suitability output by the classifier 110 are transmitted to the incision support information generating unit 240 (see also FIG. 2). The incision support information generating unit 240 generates incision support information based on the incision support information, incision unsuitable area information, and suitability. The incision support information includes, for example, information on a treatment tool suitable for the examination image, information on the angle and recommended angle of a treatment tool equipped with an acceleration sensor, and information on the recommended output power of the treatment tool. The incision support information is transmitted to the display control unit 200 and displayed on the display 17 together with the examination image as an incision support information screen 241. FIG. 29 is an example of the incision support information screen 241, which indicates that the recommended output power 242 of the treatment tool is 100V. It is preferable that the incision support information is associated with the examination image as a learning image, and the classifier 110 is trained. With the above configuration, information on treatment tools for safe incision can be presented to the surgeon.

[0076] In the present embodiment, the medical image processing device 11 is connected to the endoscope system 10. However, the present invention is not limited to this example, and other medical devices may be used. The endoscope 12 may be a rigid or flexible endoscope. The examination image acquisition unit 60 and / or the first central control unit 55 of the endoscope system 10 may be partially or entirely provided in an image processing device that communicates with the processor device 15 and cooperates with the endoscope system 10. For example, they may be provided in a diagnosis support device that acquires images captured by the endoscope 12 directly from the endoscope system 10 or indirectly from a PACS. The examination image acquisition unit 60 and / or the first central control unit 55 of the endoscope system 10 may be partially or entirely provided in a medical service support device that is connected via a network to various examination devices, such as a first examination device, a second examination device, ..., an Nth examination device, including the endoscope system 10.

[0077] In this embodiment, the hardware structure of processing units that perform various processes, such as the image acquisition unit 50, DSP 52, noise reduction unit 53, image processing switching unit 54, inspection image acquisition unit 60, image input unit 100, classifier 110, display image generation unit 150, display control unit 200, and notification control unit 210, is made up of various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, which is a processor having a circuit configuration specifically designed to perform various processes.

[0078] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which a single processor is configured with a combination of one or more CPUs and software, as typified by client or server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a System on Chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0079] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit formed by combining circuit elements such as semiconductor elements, and the hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). [Explanation of symbols]

[0080] 10 Endoscopy System 11 Medical image processing equipment 12 Endoscopy 12a Insertion part 12b Operation section 12c curved section 12d Tip 12e Angle Knob 12f Observation mode switch 12g Image analysis mode switch 12h Still image acquisition command switch 12i Zoom control 12j forceps mouth 14 Light source device 15 Processor unit 17. Display 19 User Interface 20 Light source section 20a V-LED 20b B-LED 20c G-LED 20d R-LED 21 Light Source Processor 22 Optical path coupling section 23 Light Guide 30a illumination optical system 30b Imaging optical system 31 Lighting lens 41 Objective Lens 42 Zoom Lens 43 Image sensor 44 Imaging processor 45 CDS / AGC circuit 46 A / D converter 50 Image acquisition unit 52 DSP 53 Noise reduction section 54 Image processing switching unit 55 First Central Control Department 60 Inspection image acquisition unit 70 First illumination light image generation unit 80 Second illumination light image generation unit 100 Image input unit 101 Second Central Control Unit 110 Classifier 112 Incision Suitable Site 113 Display Images 120, 123, 125, 131, 133, 135, 138, 140, 143 Training images 121 Muscle Layer Region 122 Lesion 124 Area where the submucosa covers the muscularis 126, 172, 192 muscularis 127, 173, 193 fibrosis 132 Submucosal layer without local injection 134 Locally injected submucosa 136 Submucosal layer where a solution containing indigo carmine was locally injected 137, 139, 162, 195 peeling layer 141 Cauterization scars 142 Blood coagulation 144 Hood edge 150 Display image generation section 160 First display image generation unit 161 1st display image 170 Second display image generation unit 171 2nd display image 181 Intertissue Distance Display Image 182 Distance from the submucosa to the muscularis 190 Perforation risk image generation unit 191 Perforation Risk Image 194 Submucosa 200 Display control unit 210 Notification control unit 220 Sound notification control unit 230 Notification display control unit 231 Warning display screen 232 slots 233 Warning Mark 240 Incision support information generation section 241 Incision support information screen 242 Recommended output of treatment tool

Claims

1. 1. A medical imaging device comprising a processor, The processor: Obtaining an inspection image of the subject using an endoscope, Based on the inspection image, an incision inappropriate site is identified in the subject included in the inspection image, and incision inappropriate site information relating to the incision inappropriate site is output; The medical image processing device identifies the incision unsuitable area when both the muscle layer and the fibrosis are present, using a classifier trained using training images in which the position of the muscle layer in the subject and the position of fibrosis in the subject are associated.

2. 1. A medical imaging device comprising a processor, The processor: Obtaining an inspection image of the subject using an endoscope, Based on the inspection image, an incision suitable site is identified in the subject included in the inspection image, and incision suitable site information relating to the incision suitable site is output. Alternatively, an incision unsuitable site is identified in the subject included in the inspection image, and incision unsuitable site information relating to the incision unsuitable site is output. The medical image processing device identifies the incision suitable area information and the incision unsuitable area based on the distance from the submucosa to the muscle layer and / or the distance from the submucosa to the lesion using a classifier trained using training images in which the distance from the submucosa to the muscle layer and / or the distance from the submucosa to the lesion in the subject is associated.

3. The medical image processing apparatus according to claim 2 , wherein the processor performs control to distinguish the suitable incision site and the unsuitable incision site based on suitability, and to output the suitable incision site information as the suitability.

4. The medical image processing device according to claim 1 , wherein a learning model is generated using the learning images.

5. The medical image processing apparatus according to claim 1 , wherein the learning image includes the subject into which a local injection of a liquid has been administered.

6. The medical image processing apparatus according to claim 5 , wherein the locally injected liquid includes a staining liquid.

7. The staining solution is indigo carmine, The medical image processing device according to claim 6 , wherein the training images are associated with the concentrations of the indigo carmine.

8. The medical image processing apparatus according to claim 1 , wherein the learning image is associated with the presence or absence of a cauterization scar and / or blood coagulation in the subject.

9. The medical image processing apparatus according to claim 1 , wherein the learning image is associated with the position of a hood attached to the tip of the endoscope in the subject.

10. The processor, Based on the test image, an incision suitable site is identified in the subject included in the test image, and incision suitable site information regarding the incision suitable site is output; 10. The medical image processing device according to claim 1, wherein the processor generates a first display image showing the suitable incision site based on the suitable incision site information, and controls the display to superimpose the first display image on the examination image.

11. The medical image processing apparatus according to claim 10 , wherein the first display image indicates the suitable incision site with a color, a symbol, or a graphic.

12. The medical image processing apparatus according to claim 11 , wherein the graphic is a line.

13. The medical image processing apparatus according to claim 10 , wherein the processor performs control to display the distance from the submucosal layer to the muscle layer in the subject in the first display image.

14. The medical image processing device according to claim 2, wherein the processor generates a second display image showing the incision unsuitable area based on the incision unsuitable area information, and controls the second display image to be superimposed on the examination image and displayed on a display.

15. The medical image processing apparatus according to claim 3 , wherein the processor generates a perforation risk image showing the degree of conformance as an image, and controls the display to superimpose the perforation risk image on the examination image.

16. The medical image processing apparatus according to claim 2 , wherein the processor controls to notify the patient by sound or visual notification when the examination image includes the incision inappropriate region.

17. The processor, Based on the test image, an incision suitable site is identified in the subject included in the test image, and incision suitable site information regarding the incision suitable site is output; 17. A medical image processing device according to claim 1, wherein the processor generates incision support information corresponding to the incision suitable area information based on the incision suitable area information, and controls the display to superimpose the incision support information on the examination image.

18. acquiring an inspection image of a subject by an endoscope; and performing a control step of identifying an incision inappropriate site in the subject included in the examination image and outputting incision inappropriate site information regarding the incision inappropriate site, The method for operating a medical image processing device identifies the incision unsuitable area by using a classifier trained using training images in which the position of the muscle layer in the subject and the position of fibrosis in the subject are associated, and identifies the incision unsuitable area when both the muscle layer and fibrosis are present.

19. acquiring an inspection image of a subject by an endoscope; identifying incision suitable sites in the subject included in the inspection image and outputting incision suitable site information relating to the incision suitable sites, or identifying incision unsuitable sites in the subject included in the inspection image and outputting incision unsuitable site information relating to the incision unsuitable sites, A method for operating a medical image processing device in which the incision suitable area information and the incision unsuitable area information are identified by a classifier trained using training images in which the distance from the submucosal layer to the muscle layer and / or the distance from the submucosal layer to the lesion in the subject is associated, and the incision suitable area information and the incision unsuitable area information are identified based on the distance from the submucosal layer to the muscle layer and / or the distance from the submucosal layer to the lesion.

20. An endoscope system comprising: the medical image processing device according to claim 1; and the endoscope.

Images