Medical image processing device, its operating method, and endoscope system

Abstract

To provide a medical image processing device capable of displaying a gravity direction without installing a special device to an endoscope, an operation method thereof, and an endoscope system.SOLUTION: A medical image processing device 17 acquires an endoscope image obtained by imaging a subject with an endoscope 12, detects a gravity direction 101 in the endoscope image on the basis of the endoscope image, generates a gravity direction display 121 for indicating the gravity direction 101, and displays the gravity direction display 121 on a display 15. An endoscope system 10 includes the medical image processing device 17 and the endoscope 12.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] The present invention relates to a medical image processing apparatus and an operating method thereof, and an endoscope system. [Background technology]

[0002] In recent years, diagnosis or treatment using endoscopes has become popular in the medical field. For example, endoscopic submucosal dissection (ESD) has made it possible to remove tumors that are too large for endoscopic mucosal resection (EMR), eliminating the need for highly invasive surgery. ESD has the advantage of being minimally invasive, since it is performed under an endoscope. On the other hand, the doctor who performs ESD must perform it using limited information based on images obtained by photographing the subject of the endoscope (hereinafter referred to as endoscopic images), and so skill is required.

[0003] A system and method are known that displays an indicator that indicates the vertical orientation of an endoscopic image using a rotation sensor or the like attached to a housing or the like so that a physician can know which is up and which is down in an endoscopic image (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-275257 A Summary of the Invention [Problem to be solved by the invention]

[0005] When detecting the direction of gravity using a physical method such as a rotation sensor, it is difficult to do so with a regular endoscope, and it may be necessary to install a special device in the endoscope. Furthermore, installing a special device leads to an increase in the size of the endoscope.

[0006] An object of the present invention is to provide a medical image processing device capable of displaying the direction of gravity without installing a special device on an endoscope, a method for operating the same, and an endoscope system. [Means for solving the problem]

[0007] The medical image processing device of the present invention is equipped with a processor, which acquires an endoscopic image obtained by photographing a subject with an endoscope, detects the direction of gravity in the endoscopic image based on the endoscopic image, generates a gravity direction display indicating the direction of gravity, and controls the display of the gravity direction display on a display.

[0008] The subject preferably includes a pool of liquid.

[0009] The subject preferably includes a mucous membrane, at least a portion of which has been incised.

[0010] The subject preferably includes the expelled liquid.

[0011] It is preferable that the processor detects the direction of gravity in the endoscopic image using an endoscopic image and a learning model that outputs the direction of gravity in the endoscopic medical image when the endoscopic image is input.

[0012] The processor preferably performs control to generate a superimposed image by superimposing a gravity direction indication on a medical image, and to display the superimposed image on the display.

[0013] The processor preferably controls the display of the gravity direction indication by at least one of a color, a symbol, and a graphic.

[0014] The processor preferably controls the display of the gravity direction by the color of the periphery of the endoscopic image.

[0015] It is preferable that the processor controls whether or not to display the gravity direction display based on an instruction from the user.

[0016] It is preferable that the processor controls whether or not to display the gravity direction display based on the endoscopic image.

[0017] It is preferable that the processor determines a display manner of the gravity direction display based on the endoscopic image.

[0018] In addition, the operating method of the medical image processing device of the present invention includes the steps of obtaining an endoscopic image by photographing a subject with an endoscope, detecting the direction of gravity in the endoscopic image based on the endoscopic image, generating a gravity direction display indicating the direction of gravity, and controlling the display of the gravity direction display on a display.

[0019] An endoscope system according to the present invention includes a medical image processing device and an endoscope. Effect of the Invention

[0020] According to the present invention, the direction of gravity can be displayed without installing a special device on the endoscope. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a schematic diagram of an endoscope system. [Diagram 2] 2 is a block diagram showing the functions of the endoscope system. FIG. [Diagram 3] FIG. 2 is an explanatory diagram illustrating four color LEDs included in the light source unit. [Figure 4] 1 is a graph showing the spectra of violet light V, blue light B, green light G, and red light R. [Diagram 5] FIG. 2 is a block diagram showing the functions of the medical image processing apparatus. [Figure 6] FIG. 4 is an explanatory diagram illustrating a function of a direction detection unit. [Figure 7] FIG. 11 is an image diagram showing an example of a normal image used as a learning image. [Figure 8] FIG. 11 is an explanatory diagram illustrating an example of a method for creating learning images. [Figure 9] FIG. 11 is an explanatory diagram illustrating an example of a method for creating learning images. [Figure 10] FIG. 11 is an explanatory diagram illustrating an example of a method for creating learning images. [Figure 11] 11 is an explanatory diagram illustrating an example of the content output by a direction detection unit; FIG. [Figure 12] FIG. 13 is an image diagram showing an example of a gravity direction display. [Figure 13] FIG. 13 is an image diagram showing an example of a gravity direction display. [Figure 14] FIG. 11 is an image diagram showing an example of a superimposed image. [Figure 15] FIG. 11 is an image diagram showing an example of a superimposed image. [Figure 16] FIG. 11 is an image diagram of a normal image including a liquid puddle in the subject. [Figure 17] FIG. 2 is an image diagram of a normal image including an incised mucosa in a subject. [Figure 18] FIG. 1 is an image diagram of a normal image including liquid ejected onto a subject. [Figure 19] FIG. 2 is a block diagram showing functions of a medical image processing apparatus including an image analysis unit. [Figure 20] FIG. 13 is an explanatory diagram illustrating that the gravity direction display is not automatically displayed. [Figure 21] FIG. 13 is an explanatory diagram illustrating how the gravity direction display is automatically displayed. [Figure 22] FIG. 1 is an explanatory diagram illustrating a case where a medical image processing apparatus is included in a diagnosis support apparatus. [Figure 23] FIG. 13 is an explanatory diagram illustrating a case where a medical image processing apparatus is included in a medical service support apparatus. [Figure 24] FIG. 1 is an explanatory diagram illustrating a case where a medical image processing apparatus is included in a diagnosis support apparatus. [Diagram 25] FIG. 13 is an explanatory diagram illustrating a case where a medical image processing apparatus is included in a medical service support apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] 1, an endoscope system 10 includes an endoscope 12, a light source device 13, a processor device 14, a display 15, a keyboard 16, and a medical image processing device 17. The endoscope 12 is optically connected to the light source device 13 and electrically connected to the processor device 14. The processor device 14 is connected to the medical image processing device 17. The medical image processing device 17 acquires an endoscopic image, which is a medical image, from the processor device 14 and performs various processes to acquire various information, etc.

[0023] In this embodiment, the medical image is an endoscopic image. In this embodiment, the medical image processing device 17 and the processor device 14 are separate devices, but a device that performs the functions of the medical image processing device 17 may be disposed within the processor device 14, and the processor device 14 may perform the functions of the medical image processing device 17. In addition, various connections may be wireless, not limited to wired, or may be made via a network. Therefore, the functions of the medical image processing device 17 may be performed by an external device connected via a network.

[0024] The endoscope 12 has an insertion section 12a to be inserted into the body of a subject having an observation target, an operation section 12b provided at the base end of the insertion section 12a, and a bending section 12c and a tip section 12d provided at the tip side of the insertion section 12a. The bending section 12c is bent by operating an angle knob 12e (see FIG. 2) of the operation section 12b. The tip section 12d is directed in a desired direction by the bending operation of the bending section 12c. A forceps channel (not shown) for inserting a treatment tool or the like is provided from the insertion section 12a to the tip section 12d. The treatment tool is inserted into the forceps channel from a forceps port 12h. Air supply, water supply, or suction is also performed from the forceps port 12h.

[0025] The operation unit 12b has, in addition to the angle knob 12e, a zoom operation unit 12f for changing the imaging magnification, and a mode changeover switch 12g used for switching the observation mode. Note that the switching operation of the observation mode or the zoom operation may be performed or instructed using the keyboard 16, a foot switch (not shown), or the like, in addition to the mode changeover switch 12g or the zoom operation unit 12f.

[0026] The endoscope system 10 has a normal observation mode that displays on the display 15 a normal image, which is an endoscopic image with natural coloring obtained by capturing an image of an observation target using white light as illumination light. In addition to the normal observation mode, the endoscope system 10 can also have a special observation mode that displays on the display 15 a special image, which is an endoscopic image captured of an observation target by emitting illumination light having a specific spectrum different from white light, and a multi-observation mode that automatically switches between the normal observation mode and the special observation mode. In this embodiment, an endoscopic image is acquired in the normal observation mode. Since the special image can be an endoscopic image that highlights specific structures such as blood vessels or glands and is displayed on the display 15, a mode other than the normal observation mode may be used in some cases.

[0027] The processor device 14 is electrically connected to a medical image processing device 17, a display 15, and a keyboard 16. The display 15 displays normal images, special images, and / or various information. The keyboard 16 functions as a user interface that accepts input operations such as function settings. The processor device 14 may be connected to an external storage (not shown) for storing images, image information, and the like.

[0028] As shown in FIG. 2, the light source device 13 emits illumination light to be irradiated onto an observation target, and includes a light source unit 20 and a light source processor 21 that controls the light source unit 20. The light source unit 20 is, for example, a semiconductor light source such as a multi-colored LED (Light Emitting Diode), a combination of a laser diode and a phosphor, or a xenon lamp or a halogen light source. The light source unit 20 also includes an optical filter for adjusting the wavelength band of the light emitted by the LED or the like. The light source processor 21 controls the amount of illumination light by turning on / off each LED or adjusting the drive current or drive voltage of each LED or the like. The light source processor 21 also controls the wavelength band of the illumination light by changing the optical filter or the like.

[0029] As shown in FIG. 3, in this embodiment, the light source unit 20 has four color LEDs: a V-LED (Violet Light Emitting Diode) 20a, a B-LED (Blue Light Emitting Diode) 20b, a G-LED (Green Light Emitting Diode) 20c, and an R-LED (Red Light Emitting Diode) 20d.

[0030] As shown in FIG. 4, the V-LED 20a emits violet light V with a central wavelength of 410±10 nm and a wavelength range of 380-420 nm. The B-LED 20b emits blue light B with a central wavelength of 450±10 nm and a wavelength range of 420-500 nm. The G-LED 20c emits green light G with a wavelength range of 480-600 nm. The R-LED 20d emits red light R with a central wavelength of 620-630 nm and a wavelength range of 600-650 nm.

[0031] The light source processor 21 controls the V-LED 20a, the B-LED 20b, the G-LED 20c, and the R-LED 20d. In the normal observation mode, the light source processor 21 controls each of the LEDs 20a to 20d to emit normal light in which the combination of the light intensity ratios among the purple light V, the blue light B, the green light G, and the red light R is Vc:Bc:Gc:Rc.

[0032] When the light source processor 21 is set to a special observation mode, it emits illumination light of a specific spectrum, for example, by changing the combination of light intensity ratios between purple light V, blue light B, green light G, and red light R.

[0033] The light emitted by each of the LEDs 20a to 20e is incident on the light guide 41 via an optical path coupling section (not shown) formed of a mirror, a lens, etc. The light guide 41 is built in the endoscope 12 and a universal cord (a cord that connects the endoscope 12 with the light source device 13 and the processor device 14). The light guide 41 propagates the light from the optical path coupling section to the tip 12d of the endoscope 12.

[0034] An illumination optical system 30a and an imaging optical system 30b are provided at the tip 12d of the endoscope 12. The illumination optical system 30a has an illumination lens 42, and illumination light propagated by a light guide 41 is irradiated onto an observation object via the illumination lens 42. The imaging optical system 30b has an objective lens 43, a zoom lens 44, and an imaging sensor 45. Various types of light, such as reflected light, scattered light, and fluorescent light, from the observation object are incident on the imaging sensor 45 via the objective lens 43 and the zoom lens 44. As a result, an image of the observation object is formed on the imaging sensor 45. The zoom lens 44 is freely movable between the telephoto end and the wide-angle end by operating the zoom operation unit 12f, and enlarges or reduces the observation object imaged on the imaging sensor 45.

[0035] The imaging sensor 45 is a color imaging sensor in which a color filter of R (red), G (green), or B (blue) is provided for each pixel, and captures an object to be observed and outputs image signals of each color of RGB. A charge-coupled device (CCD) imaging sensor or a complementary metal-oxide semiconductor (CMOS) imaging sensor can be used as the imaging sensor 45. A complementary imaging sensor equipped with complementary color filters of C (cyan), M (magenta), Y (yellow), and G (green) may be used instead of the imaging sensor 45 equipped with primary color filters. When a complementary imaging sensor is used, image signals of four colors of CMYG are output. Therefore, an RGB image signal similar to that of the imaging sensor 45 can be obtained by converting the four-color image signals of CMYG into three-color image signals of RGB by complementary-primary color conversion. A monochrome sensor not equipped with color filters may be used instead of the imaging sensor 45.

[0036] The imaging sensor 45 is driven and controlled by an imaging control unit (not shown). The central control unit 58 (see FIG. 2) controls the light emission of the light source unit 20 through the light source processor 21 in synchronization with the imaging control unit, thereby controlling the imaging of an observation target illuminated with normal light in normal observation mode. As a result, Bc image signals are output from the B pixels of the imaging sensor 45, Gc image signals are output from the G pixels, and Rc image signals are output from the R pixels.

[0037] A CDS / AGC (Correlated Double Sampling / Automatic Gain Control) circuit 46 performs correlated double sampling (CDS) and automatic gain control (AGC) on the analog image signal obtained from the image sensor 45. The image signal passed through the CDS / AGC circuit 46 is converted into a digital image signal by an A / D (Analog / Digital) converter 47. The digital image signal after A / D conversion is input to the processor device 14.

[0038] In the processor device 14, a program for processing such as image processing is stored in a program memory (not shown). In the processor device 14, a central control unit 58 constituted by an image processor as a first processor and the like operates a program in the program memory, thereby realizing the functions of an image acquisition unit 51, a DSP (Digital Signal Processor) 52, a noise reduction unit 53, a memory 54, an image processing unit 55, a display control unit 56, a video signal generation unit 57, and the central control unit 58. The central control unit 58 also receives information from the endoscope 12 and the light source device 13, and controls each unit of the processor device 14 as well as the endoscope 12 or the light source device 13 based on the received information. It also receives information such as instructions from the keyboard 16.

[0039] The image acquisition unit 51 acquires a digital image signal of an endoscopic image input from the endoscope 12. The image acquisition unit 51 acquires an image signal for each frame that captures an observation target illuminated by each illumination light. Note that the image acquisition unit 51 may acquire an endoscopic image that is obtained by capturing an observation target illuminated by illumination lights of predetermined mutually different spectra.

[0040] The acquired image signal is transmitted to the DSP 52. The DSP 52 performs digital signal processing such as color correction on the received image signal. The noise reduction unit 53 performs noise reduction processing, for example, using a moving average method or a median filter method, on the image signal that has been subjected to color correction and other processing by the DSP 52. The image signal with noise reduction is stored in the memory 54.

[0041] The image processing unit 55 acquires the noise-reduced image signal from the memory 54. Then, the acquired image signal is subjected to signal processing such as color conversion processing, color enhancement processing, and structure enhancement processing as necessary to generate a color endoscopic image showing the observation target. The image processing unit 55 includes a normal image processing unit 61 and a special image processing unit 62.

[0042] In the image processing unit 55, the normal image processing unit 61 performs image processing for normal images, such as color conversion processing, color enhancement processing, and structure enhancement processing, on the input image signal for a normal image after noise reduction for one frame in the normal observation mode or multi-observation mode. The image signal that has been subjected to the image processing for a normal image is input to the medical image processing device 17 and / or the display control unit 56.

[0043] In the special observation mode or the multi-observation mode, the special image processing unit 62 performs image processing for the first image, such as color conversion processing, color enhancement processing, and structure enhancement processing, on the input image signal of the first image after noise reduction for one frame. The image signal that has been subjected to the image processing for the first image is input as the first image to the medical image processing device 17 and / or the display control unit 56. Note that the image processing unit 55 may adjust the frame rate when inputting the endoscopic image to the medical image processing device 17 and / or the display control unit 56.

[0044] The endoscopic image generated by the image processing unit 55 is a normal image when the observation mode is the normal observation mode, and a special image when the observation mode is the special observation mode, and the contents of the color conversion processing, color enhancement processing, and structure enhancement processing differ depending on the observation mode. In the normal observation mode, the image processing unit 55 generates a normal image by performing the various signal processing described above that gives the observation target a natural color. In the special observation mode, the image processing unit 55 generates a special image by performing the various signal processing described above that, for example, emphasizes the blood vessels of the observation target.

[0045] The display control unit 56 receives the endoscopic image generated by the image processing unit 55 and performs control for displaying the image on the display 15. The endoscopic image controlled for display by the display control unit 56 is generated by the video signal generating unit 57 into a video signal for displaying on the display 15 and sent to the display 15. The display 15 displays the endoscopic image sent from the video signal generating unit 57 under the control of the display control unit 56.

[0046] The medical image processing device 17 acquires the endoscopic image generated by the image processing unit 55, detects the direction of gravity in the endoscopic image based on the endoscopic image, generates a gravity direction display showing the direction of gravity, and controls the display of the gravity direction display on the display.

[0047] The medical image processing device 17 is a general-purpose PC equipped with a processor, and various functions are realized by installing software. Like the processor device 14, the medical image processing device 17 also has programs related to processing such as image analysis stored in a program memory (not shown). In the medical image processing device 17, a central control unit 71 (see FIG. 5) constituted by an image processor, which is a second processor, operates the programs in the program memory to realize the functions of a medical image acquisition unit 72, a direction detection unit 73, and a direction display control unit 74 (see FIG. 5).

[0048] The medical image processing device 17 is connected to a display 15, and the display 15 displays various information generated and output by the medical image processing device 17. Various devices may be connected to the medical image processing device 17. Examples of the various devices include a user interface such as a keyboard for issuing instructions, and a storage for storing data such as images and information. The medical image processing device 17 also has a network connection function in order to connect to various devices. The medical image processing device 17 can be connected to, for example, a medical service support device 630 (see FIG. 26) through the network connection function.

[0049] 5, the medical image processing device 17 includes a central control unit 71, a medical image acquisition unit 72, a direction detection unit 73, and a direction display control unit 74. The central control unit 71 controls each unit of the medical image processing device 17, receives information from the processor device 14, etc., and controls each unit of the medical image processing device 17 based on the received information. The central control unit 71 is also connected to a user interface such as a keyboard (not shown), and receives information such as instructions from the user interface.

[0050] The medical image acquisition unit 72 acquires multiple types of endoscopic images sent from the processor device 14. The acquired images are sent to the direction detection unit 73. The direction detection unit 73 detects the direction of gravity in the endoscopic image based on the endoscopic image acquired by the medical image acquisition unit 72. The gravity direction is the direction of gravity, with the direction in which gravity heads being downward and the opposite direction being upward. The direction display control unit 74 generates a gravity direction display that notifies the user of the gravity direction detected by the direction detection unit 73 and controls the display on the display 15.

[0051] In this embodiment, the medical image acquisition unit 72 acquires a normal image obtained by photographing an observation target illuminated with normal light using an endoscope from the processor device 14, and sends the image to the direction detection unit 73. The direction detection unit 73 detects the direction of gravity in the normal image sent from the medical image acquisition unit 72. The medical image acquisition unit 72 may acquire an image other than the normal image, such as a special image, depending on the circumstances, and the direction detection unit 73 may detect the direction of gravity in the image other than the normal image, such as a special image acquired by the medical image acquisition unit 72. The special image may include, for example, an image in which a specific structure, such as a superficial blood vessel, is emphasized, and may contain information that cannot be grasped from a normal image. Therefore, depending on the circumstances, it may be possible to detect the direction of gravity more accurately and precisely by detecting the direction of gravity in the special image.

[0052] Generally, when a user such as a doctor wants to know the direction of gravity in an endoscopic image, the user does so based on the patient's body position and landmarks that appear in the endoscopic image. A landmark is a structure that is characteristic of a specific part of an organ, and the imaging position in the lumen, including the direction of gravity, can be known based on the landmark. However, for example, when the endoscope is close to an observation target for detailed observation of the observation target, the landmark may not be visible on the endoscope. In addition, when the landmark frequently changes as the endoscopic examination progresses, the user must track the changing direction of gravity and know the direction of gravity at that time, which is troublesome. Therefore, by showing the direction of gravity in such a case, the user can concentrate on tasks other than knowing the direction of gravity.

[0053] The direction detection unit 73 detects the direction of gravity in the normal image based on the normal image. The detection method can use image processing or machine learning technology. It is preferable to use a learning model in machine learning because there is a possibility that the direction of gravity can be detected more accurately and reliably. The learning model used is one that has been generated by learning and adjusting in advance so that the correct direction of gravity is output when a normal image is input.

[0054] As shown in FIG. 6, the direction detection unit 73 is a learning model, specifically a program, that detects and outputs the direction of gravity in a normal image 81 when the normal image 81 is input. The learning model can employ various techniques in machine learning to detect the position of a specific blood vessel with higher accuracy, and can also perform various adjustments. Since there is a possibility that the position of a specific blood vessel can be obtained with higher accuracy, the direction detection unit 73 is preferably a multi-layer neural network model. Since the direction detection unit 73 is a learning model that inputs an image such as the normal image 81 and detects the direction of gravity, it may be a convolutional neural network model or a deep learning model.

[0055] The direction detection unit 73 is generated by learning in advance using learning images. The learning images are endoscopic images associated with the direction of gravity in the normal image 81. The learning images are preferably learning images including a scene from which gravity is desired to be detected, and the learning model is preferably trained using learning images including a scene from which gravity is desired to be detected. For example, specifically, when the direction detection unit 73 of this embodiment detects and displays the direction of gravity in an endoscopic examination of the upper digestive tract, it is preferable to use a learning model trained using learning images in which the direction of gravity is associated with the normal image 81 of the upper digestive tract.

[0056] As shown in FIG. 7, specifically, a normal image 81 in which the stomach is observed as the upper digestive tract can be used as the learning image. In this embodiment, the normal image 81 displayed on the display 15 is a normal image 81 including a pylorus 91 of the stomach as an observation target. In addition to the pylorus 91, the lesser curvature 92, the greater curvature 93, the anterior wall 94, the posterior wall 95, and the vestibule 96 are included in the observation target, so these parts can be used as landmarks for obtaining the direction of gravity. With these parts as landmarks, the direction of gravity in the normal image 81 is a lower right direction on a plane consisting of the xy axes, as shown by the direction of gravity 101. Note that if the direction of gravity in the normal image 81 is directly above or directly below and cannot be indicated by an arrow or the like, the direction of gravity can be indicated by displaying characters such as "directly above" or "directly below."

[0057] In reality, the direction of gravity 101 is a three-dimensional vector in a three-dimensional space including the object of observation. However, the direction of gravity 101 is the direction of gravity in a plane consisting of xy axes perpendicular to the optical axis of the imaging optical system 30b of the endoscope. In the normal image 81, the object of observation and the endoscope tip 12d are approximately parallel to each other, and the direction indicated by the direction of gravity 101 can be said to be, for example, that when a staining solution is applied to the object of observation, the staining solution flows in the direction of the direction of gravity 101. Since the flow direction of the liquid can be roughly recognized, the associated direction of gravity 101 may be a two-dimensional direction on a plane perpendicular to the optical axis of the imaging optical system 30b of the endoscope as shown in FIG. 7. More precisely, the direction of gravity 101 may be a three-dimensional direction in a three-dimensional space including the object of observation shown in the normal image 81.

[0058] Moreover, to associate means to associate the normal image 81 with the gravity direction 101 in the normal image 81. It is sufficient that the normal image 81 and the gravity direction 101 are associated with each other, and it is sufficient that the gravity direction 101 in the normal image 81 can be recognized when the normal image 81 is selected. There is no restriction on the method of association. It is also possible to perform so-called tagging on the normal image 81. Furthermore, information on the gravity direction 101 may be included in the image data of the normal image 81 as a header or the like, or information on the normal image 81 and the gravity direction 101 may be stored in a format such as a table. In FIG. 7, the normal image 81 and the gravity direction 101 are associated by including a vector of the gravity direction 101 in the data of the normal image 81.

[0059] The gravity direction 101 associated with the normal image 81 is preferably as accurate as possible, but does not need to be highly accurate, and may be obtained by a skilled doctor visually inspecting the normal image 81, or may be obtained by image analysis of the normal image 81, or by machine learning based on the normal image 81. If it is sufficient to know the direction or position where the liquid accumulates, or the direction in which the incised mucosa hangs down, the gravity direction 101 associated with the normal image 81 may have an error of about ±45 degrees or less from the actual gravity direction, or about 90 degrees or less in total. This is because the direction in which the liquid accumulates or the direction in which the incised mucosa hangs down can be roughly grasped.

[0060] A method for a doctor to obtain the gravity direction 101 of the normal image 81 and associate it with the normal image 81 can be, for example, by displaying the normal image 81 on the display 15, visually checking it, and specifying the gravity direction 101. As a method for specifying the gravity direction 101, for example, as a method for specifying the gravity direction in two dimensions, as shown in Fig. 8, a method can be mentioned in which a downward direction is simply written as an arrow 112 on the normal image 81 with a mouse or a finger 111 on the display 15.

[0061] As an example of specifying the gravity direction 101 in three dimensions, as shown in Fig. 9, there is a method of specifying using a gravity direction specification screen 113. The gravity direction specification screen 113 includes a normal image 81 and a gravity direction specification screen 113, and includes a three-dimensional schema 114 of each part, etc. A rotatable display 113a indicates that the three-dimensional schema 114 is rotatable. In this embodiment, the three-dimensional schema 114 of the stomach is used. In addition, a camera position 116 may be included.

[0062] Specifically, the gravity direction 101 is specified by displaying a normal image 81 and a gravity direction specification screen 113 on the display 15 and pasting the normal image 81 at an imaging position 115 of a three-dimensional schema 114 on the display 15. The three-dimensional schema 114 and the camera position 116 of each part can be moved three-dimensionally. A doctor or the like looks at the normal image 81 and drags the position of the normal image 81 on the display 15 to the position of the normal image 81 in the three-dimensional schema 114. When the position of the normal image 81 on the three-dimensional schema 114 is determined, the imaging position 115 is determined, and for example, the display color of the imaging position 115 on the display 15 changes. The icon of the camera position 116 can also be moved.

[0063] Then, information on the posture of the patient having the observation target from which the normal image 81 was taken is acquired. The information on the posture of the patient is stored in the form of a three-dimensional schematic diagram 11. 5 This is used to determine the position to be displayed on the display. By doing so, the direction of gravity 101 can be obtained by specifying the shooting position 115. Also, the three-dimensional schema FIG. 5 The patient's position and stomach 3D diagram are recorded. 5 The correct gravity direction 101 of the normal image 81 may be obtained by calculating the gravity direction 101 from the normal image 81 attached to the normal image 81. In addition to these methods, various methods can be used as long as they can associate the gravity direction 101 in the normal image 81.

[0064] When obtaining the direction of gravity of a normal image by image analysis or machine learning, etc. and associating this with the normal image, for example, the direction of gravity 101 can be obtained as a vector in three-dimensional coordinates and associated with the normal image.

[0065] The method of writing a two-dimensional direction in the normal image 81 is preferable because a doctor or the like can easily specify the direction of gravity 101 in the normal image 81, and for example, when a cleaning fluid is discharged through an endoscope or when bleeding occurs, it is possible to detect in which direction the liquid will flow. Also, the method of using a three-dimensional model or a three-dimensional vector is preferable because it is possible to show the direction of gravity 101 even if the normal image 81 is taken in the directly upward or directly downward direction.

[0066] The direction detection unit 73 includes a learning model. The direction detection unit 73 detects the direction of gravity 101 based on the normal image 81 by using the learning model. The learning model is learned by the learning image as described above. After that, a test is performed using a test image associated with the direction of gravity 101, and parameters and the like are adjusted to generate the direction of gravity 101 associated with the test image so as to be correctly output. The generated learning model is a learning model that outputs the direction of gravity 101 when the normal image 81 is input, so that when the normal image 81 for which the direction of gravity 101 is to be known is input, the direction of gravity 101 in the input normal image 81 is detected and output. The output direction of gravity 101 is sent to the direction display control unit 74.

[0067] As shown in FIG. 10, a normal image 81 to be detected is Direction detector 73 11, a learning model that detects the direction of gravity 101 in two dimensions detects the direction of gravity 101 in the normal image 81 and outputs it as "downward and rightward." Also, as shown in FIG 11, a learning model that detects the direction of gravity 101 in three dimensions detects the direction of gravity in the normal image 81 and outputs it as a three-dimensional vector, such as "(1, -3, -2)."

[0068] The detected direction of gravity 101 is sent to the direction display control unit 74. The direction display control unit 74 displays a gravity direction display in a manner that allows the user to recognize the gravity direction, in order to notify the user of the gravity direction 101. The gravity direction display can be an image indicating the gravity direction 101 in the normal image 81.

[0069] The details of the gravity direction display can be set in advance by the user. For example, the frequency of the gravity direction display can be set so that the display is not updated if the change in the gravity direction 101 is within a predetermined angle range in order to prevent the gravity direction display from changing frequently. The predetermined angle can be set so that the display is not updated if the difference in angle on a plane before and after the change is ±45 degrees or less, or a total of 90 degrees or less, and the display is updated if it is 90 degrees or more.

[0070] Also, for example, the gravity direction display may be displayed superimposed on the normal image 81. In this case, the user can set in advance the position of the gravity direction display on the normal image 81. For example, the gravity direction display may be displayed at a position that does not interfere with the user's observation, or the gravity direction display may be displayed in an inconspicuous display mode such as transparent in the center of the normal image 81.

[0071] The direction display control unit 74 preferably performs control to display the gravity direction display by at least one of a color, a symbol, and a figure. As shown in Fig. 12, for example, the gravity direction display 121 is in the shape of an arrow, and a superimposed image is generated by superimposing this on the normal image 81, and the superimposed image is displayed on the display 15. The gravity direction display 121 in the shape of an arrow is preferably displayed so as to reliably notify the user of the gravity direction 101 and not to interfere with the user's observation. Therefore, the arrow may be displayed in the shape of a transparent figure.

[0072] In some cases, the direction display control unit 74 preferably performs control to display the gravity direction display 121 in the color of the periphery of the area displaying the normal image 81. As shown in Fig. 13, for example, the gravity direction display 121 is a frame of a specific color around the area displaying the normal image 81, a superimposed image is generated by superimposing this on the normal image 81, and the superimposed image is displayed on the display 15. It is preferable that the gravity direction display 121, which is a frame of a specific color, is displayed so as to reliably notify the user of the gravity direction and not to interfere with the user's observation. Therefore, the specific color frame around the normal image 81 may be displayed in a noticeable color, or by blinking, or the like.

[0073] When displaying gravity direction display 121 in the form of an image on display 15 to notify the detected gravity direction, direction display control unit 74 performs control such as aligning the image with normal image 81 and then superimposing the image on normal image 81 to generate superimposed image 122. Superimposed image 122 generated by direction display control unit 74 is displayed on display 15.

[0074] As shown in Fig. 14, the superimposed image 122 generated by the direction display control unit 74 is an image in which a gravity direction display 121 shown in the shape of a transparent arrow is superimposed on a normal image 81. Also, as shown in Fig. 15, the superimposed image 122 is an image in which a gravity direction display 121 is superimposed on a normal image 81 as a frame of a specific color around an area in which the normal image 81 is displayed. The superimposed image 122 is preferably generated every time a normal image 81 is captured. This makes it possible to display the superimposed image 122 in real time while the observation target is being captured by the endoscope.

[0075] The display of the superimposed image 122, i.e., the display of the gravity direction display 121, can be turned on and off as appropriate. Therefore, if it is considered that the gravity direction display 121 may be an impediment to the progress of the endoscopic examination, or that the gravity direction display 121 changes frequently and is bothersome, it is possible to turn off the superimposition of the gravity direction display 121 and display the normal image 81.

[0076] As described above, the medical image processing device 17 detects the gravity direction 101 in the normal image 81 based on the normal image 81, generates a gravity direction display 121 indicating the detected gravity direction 101, and displays it on the display 15. Since the gravity direction 101 is detected based on the normal image 81, the gravity direction 101 can be displayed without installing a special device for detecting the gravity direction in the endoscope, and can be applied regardless of the type of endoscope. Furthermore, when the gravity direction 101 is detected by machine learning, it is possible to detect the gravity direction 101 with high accuracy using a three-dimensional vector or the like. Furthermore, the display frequency or display form of the gravity direction display 121 can be set in advance by the user, and therefore it can be displayed as desired by the user.

[0077] In addition, when detecting the direction of gravity based on the normal image 81, it is preferable that the normal image 81 is obtained by photographing a subject including a liquid puddle. The liquid is cleaning liquid, blood, or other liquid. When detecting the direction of gravity 101 by the direction detection unit 73 for the normal image 81 obtained by photographing a subject including a liquid puddle, the liquid puddle can be detected by image processing technology. Liquid gathers and accumulates in the direction of gravity 101. Therefore, the direction in which the liquid puddle is detected in the normal image 81 can be the direction of gravity 101. Furthermore, by using machine learning technology, the direction of gravity 101 can be detected more reliably and accurately. As described above, the direction of gravity 101 can be detected more reliably by detecting the direction of gravity 101 based on the normal image 81 that captures a subject including a liquid puddle.

[0078] 16, a normal image 81 whose subject includes a puddle of liquid 131 is a normal image 81 in which the puddle of liquid 131 appears in any region of the normal image 81. As long as it is a puddle of liquid, it does not matter what region it is in the normal image 81. However, it is preferable to distinguish and recognize the puddle of liquid 131 present in the observation target from, for example, water droplets or the like adhering to the lens of an endoscope.

[0079] When detecting the direction of gravity 101 based on a normal image 81 captured of a subject including a puddle of liquid 131 using machine learning technology, it is preferable to use a learning model trained using a learning image in which the normal image 81 captured of a subject including a puddle of liquid 131 is associated with the direction of gravity 101 in this normal image 81.

[0080] Note that the direction of gravity 101 may be detected using at least two normal images 81 captured at different times of a subject including the puddle of liquid 131. This is because there is a possibility that the direction of gravity 101 can be detected more accurately by knowing the change in the puddle of liquid 131.

[0081] In addition, when detecting the gravity direction 101 based on the normal image 81, it is preferable that the normal image 81 is obtained by photographing a subject including a mucosa with at least a part thereof incised. A subject including an incised mucosa is photographed, for example, when a treatment tool is used to resect the mucosa, and when endoscopic submucosal dissection (ESD) or endoscopic mucosal resection (EMR) is performed. Since ESD is performed under an endoscope, it has the advantage of being minimally invasive, but the doctor who is the operator must perform ESD with limited information based on endoscopic images and the like.

[0082] ESD to remove a tumor is performed by the following steps: marking around the tumor, local injection into the submucosa, incision of the mucosa, dissection of the submucosa, and hemostasis. When dissecting the submucosa and dissecting the mucosa, for example, if the mucosa to be dissected is located near the top, the dissected mucosa may droop, making it impossible to see the back of the drooping mucosa, and making it impossible to see the area to be dissected later. Therefore, when dissecting the upper mucosa, by showing the direction of gravity, the dissection procedure can be devised in advance, and visibility can be secured until the latter half of the dissection.

[0083] The mucosa with at least a part incised is a mucosa including a part incised by a treatment tool or the like, and the extent or area of ​​incision does not matter. For example, in ESD, the mucosa is first incised, then the incision around the tumor is advanced, and finally the mucosa including the tumor is peeled off. Therefore, it is preferable that the normal image obtained by photographing a subject including a mucosa with at least a part incised is a normal image 81 that shows the stages of incision of the mucosa and peeling of the submucosa in the ESD procedure.

[0084] When detecting the direction of gravity 101 by the direction detection unit 73 for a normal image 81 obtained by photographing a subject including a mucous membrane with at least a portion thereof cut open, image processing and machine learning techniques can be used. With these techniques, the direction of gravity 101 can be detected by detecting the position, etc., of the cut mucous membrane in the subject captured in the normal image 81. With the use of machine learning techniques, the direction of gravity 101 can be detected more reliably and accurately based on the position, shape, etc. of the cut mucous membrane. As described above, the direction of gravity 101 can be detected more reliably by detecting the direction of gravity 101 based on a normal image 81 in which the subject includes a mucous membrane with at least a portion thereof cut open.

[0085] As shown in FIG. 17, a normal image 81 in which the subject includes a mucosa with at least a portion cut is a normal image 81 in which at least a portion of the mucosa 132 is shown in any region of the normal image 81. The mucosa 132 is a mucosa that has been cut by ESD, and the doctor advances the incision along a marker 134 that was made prior to the incision. After the incision, the submucosa 133 is exposed, and the cut mucosa 132 hangs down in the direction of gravity 101. As long as the subject has at least a portion of the mucosa 132 cut, the region in the normal image 81 does not matter, but it is important that the mucosa 132 has a portion of the mucosa 132 that hangs down in the direction of gravity 101. but This is preferable. Due to the shape of the mucous membrane 132, the mucous membrane 132 hangs down, and the direction of gravity 101 can be recognized based on the direction of the mucous membrane 132 in the normal image 81.

[0086] When detecting the direction of gravity 101 based on a normal image 81 of a subject including a mucous membrane 132 with at least a portion thereof incised using machine learning technology, it is preferable to use a learning model trained using a learning image in which the normal image 81 of a subject including the incised mucous membrane 132 is associated with the direction of gravity 101 in this normal image 81.

[0087] In addition, the normal image 81, which is an image of a subject including a mucous membrane that is at least partially incised, is different. photograph The direction of gravity 101 may be detected using at least two normal images 81 at a time point. This is because there is a possibility that the direction of gravity 101 can be detected more accurately by knowing the change in the position or shape of the incised mucosa.

[0088] In addition, when detecting the direction of gravity 101 based on the normal image 81, it is preferable that the normal image 81 is obtained by photographing a subject containing a discharged liquid. During an endoscopic examination, for example, a cleaning liquid for washing mucus may be discharged, a staining liquid may be applied, or a liquid may be discharged from the endoscope tip 12d. Specifically, a liquid such as a cleaning liquid is discharged using a forceps port 12h or an air / water supply port (not shown) provided on the endoscope tip 12d. When the discharged liquid hits an observation target, it then falls in the direction of gravity 101. Therefore, the direction of gravity 101 can be detected by the direction detection unit 73 for the normal image 81 obtained by photographing a subject containing the discharged liquid.

[0089] When the direction of gravity is detected by the direction detection unit 73 from a normal image 81 obtained by photographing a subject containing the discharged liquid, the direction or trajectory of the discharged liquid can be detected by image processing or machine learning technology. By using machine learning technology, the direction of gravity 101 can be detected more reliably and accurately. As described above, by detecting the direction of gravity 101 based on the normal image 81 capturing a subject containing the discharged liquid, the direction of gravity 101 can be detected more reliably.

[0090] As shown in Fig. 18, a normal image 81 including a subject containing discharged liquid 135 is a normal image 81 in which a subject containing discharged liquid 135 appears in any region of the normal image 81. As long as the subject contains discharged liquid 135, the region in the normal image 81 does not matter, but it is preferable that the normal image 81 be one in which the direction of the discharged liquid 135 after it hits the observation target is visible. Therefore, a normal image 81 in which the distance between the endoscope tip 12d and the observation target is appropriate is preferable. According to a normal image 81 in which the direction of the liquid 135 is visible, it is possible to accurately detect the direction of gravity 101.

[0091] When detecting the direction of gravity 101 based on a normal image 81 of a subject including a subject containing the ejected liquid 135 using machine learning technology, it is preferable to use a learning model trained using a learning image in which the normal image 81 of a subject including the ejected liquid 135 is associated with the direction of gravity 101 in this normal image 81.

[0092] Note that the gravity direction 101 may be detected using at least two normal images 81 captured at different times of the normal image 81 of the subject including the discharged liquid 135. This is because there is a possibility that the gravity direction 101 can be detected more accurately by knowing the change in the direction of the discharged liquid 135.

[0093] It is preferable that the direction display control unit 74 controls whether or not to display the gravity direction display 121. The control of whether or not to display the gravity direction display 121 can be controlled based on a user's instruction or the normal image 81.

[0094] When controlling whether to display the gravity direction display 121 based on a user's instruction, the control may be performed using a user interface such as the keyboard 16, the touch panel of the display 15, or a foot switch (not shown), or a device that performs voice recognition. When performing voice recognition, the control can be performed to switch between displaying and hiding the gravity direction display 121 by using a so-called wake word such as "gravity" and a voice instruction such as "display" or "stop."

[0095] When controlling whether or not to display the gravity direction display 121 based on the normal image 81, the display is controlled according to the subject appearing in the normal image 81, the shooting environment, or changes over time.

[0096] As shown in FIG. 19, the medical image processing device 17 includes an image analysis unit 141. The image analysis unit 141 analyzes the subject, the photographing environment, or the change over time in the normal image 81. For example, the subject in the normal image 81 is a landmark, blood, a lesion or the state of the lesion, or a treatment tool. The state of the lesion is, for example, a mucous membrane marked in ESD, a mucous membrane raised by local injection, or a state in which the submucosal layer 133 is photographed and the mucous membrane is peeled off. The photographing environment of the normal image 81 is the distance between the endoscope tip 12d and the observation target as the subject, the type of spectrum of the illumination light, or the zoom magnification. The change over time in the normal image 81 is based on whether the change over time is large or small. For example, in normal images 81 taken within a certain period of time, if the subject depicted in the normal images 81 is different throughout the image, it can be determined that the change over time is large and the endoscope is being moved, whereas if the subject depicted in the normal images 81 is almost the same, it can be determined that the change over time is small and the observation is being made without moving the endoscope.

[0097] The image analysis unit 141 performs an analysis based on the normal image 81 and grasps the above-mentioned contents. Then, when it is determined from the grasped contents that the situation is one in which displaying the gravity direction display 121 on the display 15 is to the user's advantage, it instructs the direction display control unit 74 to display the gravity direction display 121. On the other hand, when it is determined from the grasped contents that the situation is one in which displaying the gravity direction display 121 on the display 15 is to the user's disadvantage, it instructs the direction display control unit 74 not to display the gravity direction display 121. In this case, specifically, if the gravity direction display 121 is displayed, it instructs the direction display control unit 74 to stop displaying it. The direction display control unit 74 controls whether to display the gravity direction display 121 or not, based on the instruction from the image analysis unit 141.

[0098] The image analysis unit 141 performs an analysis based on the normal image 81 and controls whether or not to display the gravity direction display 121, so that the display of the gravity direction display 121 is automatically turned on and off. For example, when the image analysis unit 141 determines through analysis that bleeding has occurred in the observation subject and that bleeding has occurred in the subject of the normal image 81, the direction display control unit 74 controls not to display the gravity direction display 121. Then, when the image analysis unit 141 determines through analysis that bleeding has stopped in the observation subject and that the bleeding has disappeared in the subject of the normal image 81, the direction display control unit 74 controls to display the gravity direction display 121.

[0099] As shown in FIG. 20, the gravity direction display 121 is displayed in the normal image 81 while the user is observing. The gravity direction display 121 is a transparent arrow shape. During observation, bleeding occurs, and blood 151 is included in the normal image 81. In this case, the image analysis unit 141 performs an analysis based on the normal image 81 and detects blood 151, so the direction display control unit 74 controls not to display the gravity direction display 121. As a result, in the normal image 81 in which blood 151 is detected, the gravity direction display 121 automatically disappears, and if blood 151 is detected in the normal image 81, it is not continuously displayed. In this way, the gravity direction display 121 is automatically not displayed, so the user can concentrate on stopping the bleeding without the gravity direction display 121 in the normal image 81.

[0100] Furthermore, for example, when the image analysis unit 141 determines through analysis that mucosal incision by ESD has started, the direction display control unit 74 performs control to display the gravity direction display 121. When the image analysis unit 141 determines through analysis that the ESD incision has been completed, the direction display control unit 74 performs control not to display the gravity direction display 121.

[0101] As shown in FIG. 21, the user performs ESD without displaying the gravity direction display 121 in the normal image 81. In order to avoid the complexity of the figure, only some parts are labeled. After the ESD progresses and the marker 134 is applied to the mucosa by marking, the incision is started. When the incision is started, the image analysis unit 141 performs an analysis based on the normal image 81 and detects the incision of the mucosa, so the direction display control unit 74 controls to display the gravity direction display 121. The gravity direction display 121 is a frame of a specific color around the area displaying the normal image 81 that has been set in advance. This gravity direction display 121 is automatically displayed when the incision of the mucosa is detected. In this way, since the gravity direction display 121 is automatically displayed, the user can proceed with ESD while grasping the direction of the mucosa sagging due to ESD with the gravity direction display 121 displayed on the normal image 81.

[0102] As described above, the image analysis unit 141 performs analysis based on the normal image 81 and controls whether or not to display the gravity direction display 121, which is preferable because it allows the user to understand the gravity direction 101 while concentrating on operating the endoscope.

[0103] The display form of the gravity direction display 121 may be determined based on the normal image 81. The display form of the gravity direction display 121 refers to the form of the gravity direction display 121 itself, or the position of the gravity direction display 121 within the normal image 81. The form of the gravity direction display 121 itself refers to how the gravity direction display 121 is displayed, and includes, for example, a color, a symbol, a figure, or a combination of these. It also includes a display form such as blinking.

[0104] The image analysis unit 141 performs an analysis based on the normal image 81. Details of the analysis are the same as those of the analysis for controlling whether or not to display the gravity direction display 121 described above. Then, the display form of the gravity direction display 121 is determined based on the contents grasped by the analysis, and information on the determined display form is sent to the direction display control unit 74. Regarding the display position of the gravity direction display 121, a method of determining the display form in real time may be adopted, such as calculating an appropriate position of the normal image 81 and then appropriately displaying the gravity direction display 121 at that position.

[0105] As shown in Fig. 22, the medical image processing device 17 may include a body position information unit 161 that records information on the body position of a patient who is a subject of an endoscopic examination. The body position information unit 161 receives and records information on the body position of the patient from a function of grasping the shape of the endoscope, or a camera that captures the image of the patient installed in an endoscopic examination room. The body position of the patient is the position of the patient's body when the patient is undergoing an endoscopic examination. The information on the body position of the patient is, for example, information on the body position of the patient, such as how far the patient's body position has tilted forward or backward from a reference body position with the longitudinal direction of the patient's body as an axis.

[0106] As shown in Fig. 23, the posture information unit 161 grasps the position of a patient 171 in a standard posture on a bed 172 using a camera (not shown) installed in the direction of the patient's head, receives information on the patient's posture, and records it together with time data. Fig. 23 shows a schematic diagram of a patient photographed by a camera installed in the direction of the patient's head, and explains the patient's posture. For example, the information on the patient's posture may be obtained by measuring the position of the patient 171 when the patient 171 is tilted backward at an angle a from the standard posture at a certain time. 4 Also, at a different time, the patient 17 leaned forward at an angle b from the reference position. 3 When referring to the inclination of a patient, "front" refers to the front as seen from the patient, and "back" refers to the rear as seen from the patient.

[0107] By using the information on the patient's posture recorded in the posture information unit 161 to obtain more detailed information about the patient's posture, the precision and accuracy of the gravity direction 101 detected by the direction detection unit 73 can be further improved.

[0108] In the above embodiment, the present invention is applied to the processing of endoscopic images, but the present invention can also be applied to processor devices, medical image processing devices, medical image processing systems, etc. that process medical images other than endoscopic images.

[0109] 24, a part or all of the image processing unit 55 and / or the central control unit 58 of the endoscope system 10 can be provided in a diagnosis support device 610 that acquires an image captured by the endoscope 12, for example, directly from the endoscope system 10 or indirectly from a PACS (Picture Archiving and Communication Systems) 22. Similarly, a part or all of the medical image processing device 17 of the endoscope system 10 can be provided in a diagnosis support device 610 that acquires an image captured by the endoscope 12, for example, directly from the endoscope system 10 or indirectly from a PACS (Picture Archiving and Communication Systems) 22.

[0110] Furthermore, as shown in FIG. 25, a medical work support device 630 connected to various inspection devices including the endoscopic system 10, such as a first inspection device 621, a second inspection device 622, ..., an Nth inspection device 623, via a network 626, can be provided with part or all of the image processing unit 55 and / or central control unit 58 of the endoscopic system 10, or part or all of the medical image processing device 17.

[0111] In the above embodiment, the hardware structure of the processing units that execute various processes, such as the central control unit 58, image acquisition unit 51, DSP 52, noise reduction unit 53, memory 54, image processing unit 55, display control unit 56, and video signal generation unit 57 included in the processor device 14 including the light source processor and the first processor, and the central control unit 71, medical image acquisition unit 72, direction detection unit 73, direction display control unit 74, image analysis unit 141, and body position information unit 161 included in the medical image processing device 17 including the second processor, are various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacturing such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing various processes.

[0112] One processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with one processor. As an example of configuring multiple processing units with one processor, first, as represented by a computer such as a client or server, one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0113] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit in the form of a combination of circuit elements such as semiconductor elements. [Explanation of symbols]

[0114] 10 Endoscope System 12 Endoscopy 12a Insertion part 12b Operation section 12c Curved section 12d Tip 12e Angle knob 12f Zoom operation section 12g mode switch 12h forceps mouth 13 Light source device 14 Processor unit 15 Display 16 Keyboard 17 Medical image processing equipment 20 Light source section 20a V-LED 20b B-LED 20c G-LED 20d R-LED 21 Light source processor 22 PACS 30a illumination optical system 30b Imaging optical system 41 Light Guide 42 Lighting Lens 43 Objective Lens 44 Zoom Lens 45 Image sensor 46 CDS / AGC circuit 47 A / D Converter 51 Image acquisition unit 52 DSP 53 Noise reduction section 54 Memory 55 Image processing section 56 Display control unit 57 Video signal generator 58, 71 Central Control Unit 61 Normal image processing section 62 Special image processing section 72 Medical Image Acquisition Department 73 Direction detection unit 74 Directional display control unit 81 Normal Images 91 Pylorus 92 Lesser curve 93 Great Curve 94 Front wall 95 Back wall 96 Antrum 101 Gravity direction 111 fingers 112 Arrow 113 Gravity direction specification screen 113a Rotatable Display 114 Three-dimensional diagram of the stomach 115 Shooting position 116 Camera Position 121 Gravity direction display 122 Overlay Images 131 Liquid Pool 132 Mucosa 133 Submucosa 134 Marker 135 Liquid 141 Image Analysis Unit 151 Blood 161 Position Information Department 171 Patient in standard position 172 Beds 173 Patient leaning back 174 Patient leaning forward 610 Diagnostic Support Device 621 First Inspection Device 622 Second Inspection Equipment 623 Nth Inspection Device 626 Network 630 Medical Business Support Devices x,y direction a, b Inclination

Claims

1. A processor is provided. The processor, An endoscopic image is obtained by photographing a subject with an endoscope; Detecting a direction of gravity in the endoscopic image based on the endoscopic image; generating a gravity direction display indicating the gravity direction; Control is performed to display the gravity direction display on a display; The processor is a medical image processing device that detects the direction of gravity in the endoscopic image using the endoscopic image and a learning model that outputs the direction of gravity in the endoscopic image when the endoscopic image is input.

2. The medical imaging apparatus of claim 1 , wherein the subject includes a pool of liquid.

3. The medical image processing apparatus according to claim 1 , wherein the subject includes a mucous membrane at least part of which is incised.

4. The medical image processing apparatus according to claim 1 , wherein the subject includes a discharged liquid.

5. The processor generates a superimposed image by superimposing the gravity direction display on the endoscopic image, 5. The medical image processing apparatus according to claim 1, further comprising a control unit for controlling display of the superimposed image on the display.

6. 6. The medical image processing apparatus according to claim 1, wherein the processor controls the display of the gravity direction by at least one of a color, a symbol, and a graphic.

7. 7. The medical image processing apparatus according to claim 1, wherein the processor controls the display of the gravity direction by a color of a periphery of the endoscopic image.

8. 8. The medical image processing apparatus according to claim 1, wherein the processor controls whether or not the gravity direction display is displayed based on an instruction from a user.

9. The medical image processing apparatus according to claim 1 , wherein the processor controls whether or not to display the gravity direction display based on the endoscopic image.

10. The medical image processing apparatus according to claim 1 , wherein the processor determines a display mode of the gravity direction display based on the endoscopic image.

11. acquiring an endoscopic image by photographing a subject with an endoscope; detecting a direction of gravity in the endoscopic image based on the endoscopic image; generating a gravity direction display indicating the gravity direction; and performing control to display the gravity direction display on a display, A method for operating a medical image processing device that detects the direction of gravity in an endoscopic image using the endoscopic image and a learning model that outputs the direction of gravity in the endoscopic image when the endoscopic image is input.

12. A medical image processing apparatus according to any one of claims 1 to 10, An endoscope system comprising:

Images