Medical image processing device, operation method of medical image processing device, and endoscope system

Abstract

The present invention aims to provide a medical image processing device, an operating method of the medical image processing device, and an endoscope system that can notify omissions of observations at appropriate timing. A medical image processing device according to one embodiment of the present invention comprises: a memory that stores part information indicating a predetermined part of a subject to be imaged; a processor; and a notification unit, wherein the processor acquires a medical image of the subject, identifies the part of the subject shown in the medical image, compares the identified part with the part indicated by the part information, determines which part of the subject to be imaged has not been imaged, and notifies the notification unit of the result of the determination at an expected end timing when acquisition of the medical image of the part to be imaged is expected to be completed.

Description

Technical Field

[0001] The present invention relates to a medical image processing device, an operating method of the medical image processing device, and an endoscope system. Background Art

[0002] The image display device described in Patent Document 1 detects landmark images containing anatomical landmarks from endoscopic images. Furthermore, the landmark images are assigned to landmark portions of a virtual model corresponding to the target organ, and mapping is performed to allocate multiple endoscopic images to corresponding portions of the virtual model using their interconnected relationships. Based on the virtual model with the multiple endoscopic images allocated to each portion, a mapped image representing the imaged and unimaged areas of the target organ is generated and displayed on a monitor.

[0003] Previous technical literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-50890 Summary of the Invention

[0006] Technical issues to be solved by the invention

[0007] Screening endoscopes for diagnosing lesions have many observation areas, particularly in the upper endoscope, and are difficult to operate. This poses a problem of users (doctors) unfamiliar with endoscopic diagnosis missing observations. One possible solution to this problem is to prevent missed observations by using technology that automatically determines observed areas. However, constantly reporting the observed areas can hinder diagnosis. However, conventional techniques such as Patent Document 1, while performing missed observation evaluation, do not report the evaluation results at an appropriate time.

[0008] The present invention has been completed in view of this situation, and its purpose is to provide a medical image processing device, an operating method of the medical image processing device, and an endoscope system that can notify observation omissions at appropriate timing.

[0009] Means for solving technical problems

[0010] The medical image processing device involved in the first embodiment of the present invention comprises: a memory, which stores part information representing a plurality of predetermined parts of a subject to be photographed; a processor; and a notification unit, wherein the processor acquires a medical image of the subject, identifies the part of the subject reflected in the medical image, compares the identified part with the part represented by the part information, determines the part that has not been photographed among the plurality of parts to be photographed, and notifies the result of the determination through the notification unit at an expected end timing when the acquisition of the medical images of the plurality of parts to be photographed is expected to be completed.

[0011] In the medical image processing apparatus according to the second aspect, in the first aspect, the processor accepts a user operation indicating termination and notifies the result of determination using the timing at which the operation is accepted as the expected termination timing.

[0012] In the medical image processing apparatus according to the third aspect, in the first or second aspect, the processor reports the determination result using the timing at which the observation target portion in the recognition result changes from one organ to another organ as the expected end timing.

[0013] In the medical image processing device according to the fourth aspect, in any one of the first to third aspects, the processor reports the result of the determination using the timing at which the medical image of the subject at a predetermined portion is acquired as the expected end timing.

[0014] The medical image processing apparatus according to a fifth aspect is characterized in that, in the fourth aspect, the predetermined site is the esophagogastric junction.

[0015] The medical image processing device according to a sixth aspect is characterized in that, in the fourth aspect, the predetermined site is the throat.

[0016] The medical image processing device according to the seventh aspect is one of the fourth to sixth aspects, wherein the processor makes a determination at the timing of acquiring a medical image at a predetermined site.

[0017] The medical image processing apparatus according to an eighth aspect is configured such that, in any one of the first to seventh aspects, the processor reduces the effectiveness of the notification if a predetermined time has elapsed after the notification has been issued.

[0018] The medical image processing apparatus according to a ninth aspect is configured such that, in any one of the first to eighth aspects, the notification unit includes a display for displaying information on a screen and / or a speaker for outputting sound.

[0019] The medical image processing apparatus according to the tenth aspect is configured such that, in the ninth aspect, the processor performs notification by changing a display mode of information displayed on the display and / or a sound output mode output from the speaker.

[0020] In the medical image processing device involved in the eleventh embodiment, in the ninth or tenth embodiment, the processor performs notification by causing information that was not displayed on the screen before the notification to be displayed on a new screen on the display, and / or by causing the speaker to newly output a sound that was not output before the notification was started.

[0021] The medical image processing apparatus according to a twelfth aspect is configured such that, in any one of the ninth to eleventh aspects, the processor increases or decreases the notification power of the screen display on the display.

[0022] In the medical image processing device involved in the thirteenth method, in any one of the first to twelfth methods, the processor determines that the part is identified when at least one of the following items is satisfied: the subject appears in the medical image for more than a certain time; the subject appears in a certain area of ​​the medical image; and the subject is at a certain focus level or above in the medical image.

[0023] The operating method of the medical image processing device involved in the fourteenth embodiment of the present invention is an operating method of the following medical image processing device, which comprises: a memory, which stores part information representing a plurality of predetermined parts of the subject to be photographed; a processor; and a notification unit, wherein the processor executes: an image acquisition process, which acquires a medical image of the subject; a part recognition process, which recognizes the part of the subject reflected in the medical image; a judgment process, which compares the recognized part with the part represented by the part information, and determines the part that has not been photographed among the plurality of parts to be photographed; and a notification process, which notifies the judgment result through the notification unit at the expected end timing when the acquisition of the medical images of the plurality of parts to be photographed is expected to be completed.

[0024] Furthermore, the operating method according to the fourteenth aspect may also have the same structure as the second to thirteenth aspects. Furthermore, as aspects of the present invention, there are also provided a program for causing a computer to execute the operating method according to the present invention, and a non-transitory recording medium having a computer-readable code of the program recorded thereon.

[0025] An endoscope system according to a fifteenth embodiment of the present invention comprises: a medical image processing device according to any one of the first to thirteenth embodiments; and an endoscope inserted into a subject to be examined to capture medical images, and a processor acquiring the medical images captured by the endoscope.

[0026] In the endoscope system according to the sixteenth aspect, in the fifteenth aspect, the processor estimates the moving direction of the endoscope and notifies the result of the determination using the timing when the estimated moving direction changes to the backward direction as the expected end timing. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a diagram showing the configuration of an endoscope system according to the first embodiment.

[0028] Figure 2 This is another diagram showing the structure of the endoscope system.

[0029] Figure 3 This is a functional block diagram of the image processing unit.

[0030] Figure 4 It is a diagram showing information recorded in the recording unit.

[0031] Figure 5 This is a diagram showing an example structure of a convolutional neural network.

[0032] Figure 6 It is a diagram showing the state of convolution processing using filters.

[0033] Figure 7 It is a flowchart showing the steps of the medical image processing method involved in the first embodiment.

[0034] Figure 8 This is a diagram showing an example of a setting screen for notification conditions and notification method.

[0035] Figure 9 This is a diagram showing an example of a method of reporting a determination result.

[0036] Figure 10 This is another diagram showing an example of a method of notifying a determination result.

[0037] Figure 11 This is another diagram showing an example of a method of reporting a determination result.

[0038] Figure 12 This is another flowchart showing the steps of the medical image processing method involved in the first embodiment.

[0039] Figure 13 This is another flowchart showing the steps of the medical image processing method involved in the first embodiment. DETAILED DESCRIPTION

[0040] Hereinafter, embodiments of the medical image processing device, the operation method of the medical image processing device, and the endoscope system according to the present invention will be described in detail with reference to the accompanying drawings.

[0041] <First embodiment>

[0042] <Structure of the Endoscope System>

[0043] Figure 1 is an external view of the endoscope system 10 (endoscope system, medical image processing device), Figure 2 FIG is a block diagram showing the main structure of the endoscope system 10. Figure 1 、 2 As shown, the endoscope system 10 is composed of an endoscope observer 100 (medical image acquisition unit, endoscope observer), a processor 200 (medical image processing device, processor, medical image acquisition unit, part recognition unit, judgment unit, notification unit, notification control unit, operation acceptance unit, movement direction estimation unit), a light source device 300 (light source device), and a monitor 400 (display device, display).

[0044] <Structure of Endoscope>

[0045] The endoscope 100 includes a handheld operation unit 102 and an insertion unit 104 connected to the handheld operation unit 102. The operator (user) grasps the handheld operation unit 102 and operates it, inserting the insertion unit 104 into the body of a subject (living organism) for observation. The handheld operation unit 102 is also equipped with air and water supply buttons 141, a suction button 142, function buttons 143 assigned various functions, and a capture button 144 for receiving capture instructions (still image, moving image). The insertion unit 104 is composed, in order from the handheld operation unit 102 side, of a flexible portion 112, a curved portion 114, and a distal rigid portion 116. Specifically, the curved portion 114 is connected to the proximal end of the distal rigid portion 116, and the flexible portion 112 is connected to the proximal end of the curved portion 114. The handheld operation unit 102 is connected to the proximal end of the insertion unit 104. The user can bend the bending portion 114 by operating the hand operation unit 102, and change the direction of the top hard portion 116 upward, downward, left, and right. The top hard portion 116 is provided with a photographic optical system 130, an illumination unit 123, a forceps opening 126, etc. (see Figure 1 、 2 ).

[0046] During observation and treatment, the operation unit 208 (see Figure 2 ) operation, white light and / or narrowband light (one or more of red narrowband light, green narrowband light, blue narrowband light, and violet narrowband light) can be emitted from the illumination lenses 123A and 123B of the illumination unit 123. Furthermore, by operating the air and water supply button 141, cleansing water can be discharged from a water supply nozzle (not shown) to clean the imaging lens 132 (photographic lens, imaging unit) and the illumination lenses 123A and 123B of the imaging optical system 130. A conduit (not shown) is connected to the forceps opening 126 opened in the distal rigid portion 116. A treatment instrument (not shown) used for tumor removal, for example, is inserted through this conduit, allowing it to be inserted and removed as appropriate to perform necessary treatment on the subject.

[0047] like Figure 1 、 2As shown, a photographic lens 132 (photographic unit) is provided on the distal end surface 116A of the distal rigid portion 116. Behind the photographic lens 132, a CMOS (Complementary Metal-Oxide Semiconductor) type imaging element 134 (imaging element, image acquisition unit), a drive circuit 136, and an AFE 138 (AFE: Analog Front End, photographic unit) are provided. These components output image signals. The imaging element 134 is a color imaging element having a plurality of pixels composed of a plurality of light-receiving elements arranged in a matrix (two-dimensional arrangement) using a specific pattern arrangement (Bayer arrangement, X-Trans (registered trademark) arrangement, honeycomb arrangement, etc.). Each pixel of the imaging element 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (photodiode, etc.). The imaging optical system 130 can generate a color image based on pixel signals for red, green, and blue, or it can generate an image based on pixel signals for any one or two of these colors. Furthermore, the imaging element 134 can also be a CCD (Charge Coupled Device) type. Furthermore, each pixel of the imaging element 134 can also include a violet color filter corresponding to the violet light source 310V and / or an infrared filter corresponding to the infrared light source.

[0048] The optical image of the subject is formed by the imaging lens 132 onto the light-receiving surface (imaging surface) of the imaging element 134 and converted into an electrical signal. The electrical signal is then output to the processor 200 via a signal cable (not shown) and converted into an image signal. As a result, an endoscopic image (image, medical image) of the subject is displayed on the monitor 400 connected to the processor 200.

[0049] Furthermore, illumination lenses 123A and 123B of the illumination unit 123 are provided on the distal end surface 116A of the distal rigid portion 116, adjacent to the imaging lens 132. The emission end of a light guide 170, described later, is disposed behind the illumination lenses 123A and 123B. This light guide 170 is inserted through the insertion portion 104, the hand-side operation portion 102, and the universal cable 106, and the incident end of the light guide 170 is disposed within the light guide connector 108.

[0050] The user inserts or removes the endoscope 100 (insertion section 104) having the above structure into or from the body of the subject while performing imaging at a predetermined frame rate (which can be performed under the control of the medical image acquisition section 220), thereby sequentially capturing time-series images of the body.

[0051] <Structure of Light Source Device>

[0052] like Figure 2As shown, the light source device 300 comprises an illumination light source 310, an aperture 330, a condenser lens 340, and a light source control unit 350, and directs observation light into the light guide 170. The light source 310 includes a red light source 310R, a green light source 310G, a blue light source 310B, and a violet light source 310V, which respectively emit narrowband light of red, green, blue, and violet. The illumination intensity of the observation light from the light source 310 is controlled by the light source control unit 350, which can change (increase or decrease) the illumination intensity and stop the illumination as needed.

[0053] Light source 310 can emit red, green, blue, and violet narrowband light in any combination. For example, it can emit red, green, blue, and violet narrowband light simultaneously to irradiate white light (normal light) as observation light, or it can emit any one or two of these lights to irradiate narrowband light (special light). Light source 310 can also include an infrared light source that irradiates infrared light (an example of narrowband light). Alternatively, it is possible to utilize a light source that irradiates white light and filters that transmit white light and each narrowband light to irradiate white light or narrowband light as observation light.

[0054] <Light source wavelength>

[0055] Light source 310 can generate light in a white wavelength band, or can generate light in multiple wavelength bands as the white wavelength band, or can generate light in a specific wavelength band narrower than the white wavelength band. The specific wavelength band can be a blue wavelength band or a green wavelength band in the visible range, or a red wavelength band in the visible range. When the specific wavelength band is a blue wavelength band or a green wavelength band in the visible range, it can include a wavelength band between 390 nm and 450 nm, or between 530 nm and 550 nm, and have a peak wavelength within the wavelength band between 390 nm and 450 nm, or between 530 nm and 550 nm. Furthermore, when the specific wavelength band is a red wavelength band in the visible range, it can include a wavelength band between 585 nm and 615 nm, or between 610 nm and 730 nm, and have a peak wavelength within the wavelength band between 585 nm and 615 nm, or between 610 nm and 730 nm.

[0056] The specific wavelength band may include a wavelength band where the absorption coefficient differs between oxyhemoglobin and deoxyhemoglobin, and the light in the specific wavelength band may have a peak wavelength in the wavelength band where the absorption coefficient differs between oxyhemoglobin and deoxyhemoglobin. In this case, the specific wavelength band may include 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band between 600 nm and 750 nm, and the light in the specific wavelength band may have a peak wavelength in the wavelength band between 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band between 600 nm and 750 nm.

[0057] In addition, the wavelength band of the light generated by the light source 310 may also include a wavelength band of 790nm to 820nm or 905nm to 970nm, and the light generated by the light source 310 has a peak wavelength in the wavelength band of 790nm to 820nm or 905nm to 970nm.

[0058] Alternatively, light source 310 may include a light source that emits excitation light with a peak wavelength of 390 nm to 470 nm. In this case, medical images (medical images, in-vivo images) containing information about the fluorescence emitted by fluorescent substances within the subject (organism) can be acquired. Fluorescence pigments (such as fluorescein and acridine orange) may also be used to acquire fluorescence images.

[0059] The light source type (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode) etc.), wavelength, presence or absence of a filter, etc. of light source 310 are preferably configured according to the type of subject, the purpose of observation, etc. Furthermore, during observation, it is preferable to combine and / or switch the wavelength of the observation light according to the type, location, and purpose of observation of the subject. When switching the wavelength, for example, the wavelength of the irradiated light can be switched by rotating a disc-shaped filter (rotating color filter) disposed in front of the light source and provided with a filter that transmits or blocks light of a specific wavelength.

[0060] In addition, the imaging element used when implementing the present invention is not limited to a color imaging element in which a color filter is provided for each pixel like the imaging element 134, but may also be a monochrome imaging element. When using a monochrome imaging element, the wavelength of the observation light can be switched in sequence to perform imaging in a plane order (color order). For example, the wavelength of the emitted observation light can be switched in sequence between narrow-band light (purple, blue, green, red), or broadband light (white light) can be irradiated and the wavelength of the emitted observation light can be switched by rotating color filters (red, green, blue, purple, etc.). In addition, one or more narrow-band lights (green, blue, purple, etc.) can be irradiated and the wavelength of the emitted observation light can be switched by rotating color filters (green, blue, purple, etc.). The narrow-band light can be infrared light of two or more wavelengths with different wavelengths (first narrow-band light, second narrow-band light).

[0061] By connecting the optical connector 108 (refer to Figure 1 、 2 ) is connected to the light source device 300, and the observation light irradiated from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and is irradiated toward the observation range from the illumination lenses 123A and 123B.

[0062] <Processor Structure>

[0063] based on Figure 2 The structure of the processor 200 is described below. The processor 200 inputs the image signal output from the endoscope 100 via the image input controller 202, performs necessary image processing in the image processing unit 204 (medical image processing unit, processor), and outputs it via the video output unit 206. As a result, the observation image (in vivo image) is displayed on the monitor 400 (display device). These processes are performed under the control of the CPU 210 (CPU: Central Processing Unit, processor). The communication control unit 205 performs communication control regarding the acquisition of medical images, etc. with the hospital system (HIS: Hospital Information System) or the hospital LAN (Local Area Network) not shown in the figure, and / or external systems or networks.

[0064] <Functions of the Image Processing Unit>

[0065] Figure 3 This is a functional block diagram of the image processing unit 204. The image processing unit 204 includes a medical image acquisition unit 220 (medical image acquisition unit, image acquisition unit), a site identification unit 222 (site identification unit), a determination unit 224 (determination unit), a notification control unit 226 (notification control unit), an operation acceptance unit 227 (operation acceptance unit), a movement direction estimation unit 228 (movement direction estimation unit), a recording control unit 229 (recording control unit), and a display control unit 230 (display control unit). Details of the processing using these functions will be described later.

[0066] The image processing unit 204 utilizes the above-described functions to calculate the characteristic values ​​of medical images, perform processing to emphasize or reduce components in specific wavelength bands, and perform processing to emphasize or de-emphasize specific objects (such as regions of interest and blood vessels at a desired depth). The image processing unit 204 may also include a special light image acquisition unit that acquires a special light image containing information about a specific wavelength band based on a normal light image obtained by irradiating the image with light in the white wavelength band or by irradiating the image with light in multiple wavelength bands as the white wavelength band. In this case, the signal for the specific wavelength band can be obtained by performing calculations based on RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) color information contained in the normal light image. Furthermore, the image processing unit 204 may also include a characteristic value image generation unit that acquires and displays a characteristic value image as a medical image. This characteristic value image generation unit generates the characteristic value image by performing calculations based on at least one of a normal light image obtained by irradiating the image with light in the white wavelength band or by irradiating the image with light in multiple wavelength bands as the white wavelength band, and a special light image obtained by irradiating the image with light in the specific wavelength band. In addition, the above-mentioned processing is executed under the control of the CPU 210 .

[0067] <Implementation of various processor functions>

[0068] The functions of each part of the above-mentioned image processing unit 204 can be implemented using various processors and recording media. Among the various processors, there are general-purpose processors such as CPUs (Central Processing Units) that implement various functions by executing software (programs). In addition, among the various processors mentioned above, there are also processors specifically used for image processing, namely GPUs (Graphics Processing Units), and processors whose circuit structures can be changed after manufacturing FPGAs (Field Programmable Gate Arrays), namely programmable logic devices (PLDs). In the case of learning or recognizing images as in the present invention, a structure using a GPU is effective. Moreover, processors having circuit structures specially designed for executing specific processes such as ASICs (Application Specific Integrated Circuits), namely dedicated circuits, are also included in the various processors mentioned above.

[0069] The functions of each part can be implemented by one processor, or by multiple processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). In addition, multiple functions can also be implemented by one processor. As an example of implementing multiple functions with one processor, first, there is a form such as a computer, which uses a combination of one or more CPUs and software to form a processor, and implements the processor as multiple functions. Secondly, there is a form such as a system on chip (System On Chip: SoC), which uses a processor that implements the functions of the entire system with an IC (Integrated Circuit) chip. In this way, various functions are implemented using one or more of the above-mentioned various processors as a hardware structure. Moreover, more specifically, the hardware structure of these various processors is a circuit (circuitry) that combines circuit elements such as semiconductor elements. These circuits can also be circuits that use logical sum, logical product, logical negation, exclusive or, and logical operations combining them to implement the above-mentioned functions.

[0070] When the above-mentioned processor or circuit executes the software (program), the computer-readable code of the executed software (for example, the various processors or circuits constituting the image processing unit 204, and / or their combination) is pre-stored in a non-temporary recording medium such as ROM211 (ROM: Read Only Memory), and the computer refers to the software. The software pre-stored in the non-temporary recording medium includes a program for executing the operation method of the medical image processing device involved in the present invention and data used during execution (data related to the acquisition of medical images, specific data for notification conditions and notification methods, parameters used in the identification unit, etc.). Instead of storing the code in ROM211, it is also possible to store it in various non-temporary recording media such as optical magnetic recording devices and semiconductor memories. When software is used for processing, for example, RAM212 (RAM: Random Access Memory) is used as a temporary storage area, and for example, data stored in an EEPROM (Electrically Erasable and Programmable Read Only Memory) not shown in the figure can also be referenced. The recording unit 207 can also be used as a "non-temporary recording medium".

[0071] In addition, ROM 211 (ROM: Read Only Memory) is a non-volatile storage element (non-temporary recording medium) that stores computer-readable code of programs that cause CPU 210 and / or image processing unit 204 (computer) to execute various image processing methods (including the operating method of the medical image processing device involved in the present invention). RAM 212 (RAM: Random Access Memory) is a storage element used for temporary storage during various processing and can also be used as a buffer during image acquisition. The sound processing unit 209 outputs messages (sounds) related to medical image processing, part recognition, notification, etc. from the speaker 209A (notification unit, speaker) under the control of CPU 210 and image processing unit 204.

[0072] <Operation Section>

[0073] The operation unit 208 may be composed of a keyboard, mouse, or other device not shown in the figure. The user may designate the execution instructions of medical image processing or the conditions required for execution (for example, the setting of notification conditions and notification methods described later) via the operation unit 208 (operation receiving unit). The operation via the operation unit 208 includes the setting of notification conditions and notification methods (see Figure 8 ), indicating that the acquisition of medical images of multiple parts to be photographed is completed. The above-mentioned operation acceptance unit 227 accepts user operations via the operation unit 208, and processes the accepted operations in the CPU 210 and the image processing unit 204.

[0074] <Information recorded in the recording section>

[0075] like Figure 4 As shown, recording unit 207 (recording device, memory, non-transitory recording medium) records an endoscopic image 260 (endoscopic image, medical image, medical image), part information 262 (part information; information indicating multiple parts of the subject to be imaged), part recognition result 264 (recognition result of the part of the subject imaged in the medical image), and determination result 266 (determination result of the part not imaged among the multiple parts to be imaged). Part information 262 may be in an image format or in another format such as a list format consisting of text or numbers (e.g., "thoracic esophagus 1," "middle gastric body B," "lesser curvature of gastric angle") or the like.

[0076] <Neural Network-Based Recognition Unit>

[0077] In the first embodiment, the part recognition unit 222 can be configured using a learned model such as a neural network (a model learned using a set of images obtained by capturing images of a living body). The following describes a configuration for performing multi-class classification (each class corresponds to a different part) using a CNN (Convolutional Neural Network) as the neural network.

[0078] <Configuration Example of Identification Unit>

[0079] Figure 5 This is a diagram showing the structure of CNN232 (neural network). Figure 5 In the example shown in part (a), CNN232 includes an input layer 232A (input unit), an intermediate layer 232B and an output layer 232C. The input layer 232A inputs the endoscopic image (medical image) acquired by the medical image acquisition unit 220 and outputs a feature quantity. The intermediate layer 232B includes a convolution layer 234 and a pooling layer 235, which calculates other feature quantities by inputting the feature quantities output from the input layer 232A. These layers are formed into a structure in which a plurality of "nodes" are connected by "edges", and the weight coefficients applied to the input image are associated with the nodes and edges and are stored in a weight coefficient storage unit not shown. The value of the weight coefficient changes as learning progresses.

[0080] <Processing in the middle layer>

[0081] The intermediate layer 232B calculates the feature quantity through convolution operation and pooling processing. The convolution operation performed in the convolution layer 234 is a process of obtaining a feature map by using a convolution operation of a filter, and plays the role of extracting features such as edges from the image. By using the convolution operation of the filter, a "feature map" of one channel (one frame) is generated for one filter. When the size of the "feature map" is reduced (downscaling) by convolution, it becomes smaller as convolution is performed in each layer. The pooling processing performed in the pooling layer 235 is a process of reducing (or enlarging) the feature map output by the convolution operation to obtain a new feature map, and plays the role of providing robustness to prevent the extracted features from being affected by parallel movement, etc. The intermediate layer 232B can be composed of one or more layers that perform these processes. In addition, CNN232 can also be constructed without a pooling layer 235.

[0082] CNN232 can also be Figure 5 As shown in the example of part (b) of FIG, a fully connected layer 236 is included. The layer structure of CNN 232 is not limited to the case where the convolution layer 234 and the pooling layer 235 are repeated one by one, and a plurality of arbitrary layers (for example, the convolution layer 234) may be included consecutively.

[0083] Figure 6 Yes Figure 5 Schematic diagram of a structural example of the intermediate layer 232B of the CNN 232 shown. In the initial (first) convolutional layer of the intermediate layer 232B, a convolution operation is performed on an image group consisting of multiple medical images (an image group for learning during learning, and an image group for part recognition during part recognition) and a filter F1. The image group is composed of N frames (N channels) of image size with a height H and a width W. In the case of inputting an ordinary light image, the images constituting the image group are images of three channels: R (red), G (green), and B (blue). Since the image group is N channels (N frames), the filter F1 that performs the convolution operation with the image group has a filter size of 5 (5×5), for example, the filter size becomes 5×5×N. By using the convolution operation using the filter F1, a "feature map" of 1 channel (1 frame) is generated for 1 filter F1. For example, when the filter F2 used in the second convolutional layer has a size of 3 (3×3), the filter size becomes 3×3×M.

[0084] Similar to the first convolution layer, filters F2 to F n The size of the “feature map” in the nth convolutional layer is smaller than that in the second convolutional layer. This is because it is reduced by the convolutional layer or pooling layer up to the previous stage.

[0085] In the layer of the intermediate layer 232B, low-order feature extraction (edge ​​extraction, etc.) is performed in the convolution layer close to the input side, and high-order feature extraction (feature extraction related to the shape, structure, etc. of the identified object) is performed as it approaches the output side.

[0086] In addition to the convolutional layer 234 and the pooling layer 235, the intermediate layer 232B may also include a batch normalization layer. Batch normalization normalizes the distribution of data in small batches during learning, and helps to accelerate learning, reduce dependence on initial values, and prevent over-learning.

[0087] The output layer 232C outputs the feature values ​​calculated by the intermediate layer 232B in a format suitable for part recognition. The output layer 232C may also include a fully connected layer.

[0088] <Each process of the medical image processing method>

[0089] Figure 7 This is a flowchart showing an overview of the processing of the medical image processing method (operation method of the medical image processing device) involved in the first embodiment. In addition, it is assumed that the learning of the CNN 232 using the learning data has been performed.

[0090] <Setting of notification conditions and notification methods>

[0091] The image processing unit 204 (notification control unit 226) sets notification conditions and notification methods according to user operations via the operation unit 208 (step S100: notification condition setting process, notification method setting process). Figure 8 The illustrated screen 700 (displayed on the monitor 400 ) performs setting operations.

[0092] Screen 700 has areas 702 to 712 with radio buttons, an area 714 with a pull-down menu, and an area 716 with a numeric input field. The user can set whether to issue notifications (on or off; area 702) by operating the radio buttons. In addition, the user can set "whether to issue notifications" (area 702), "whether to issue notifications based on screen display" (area 704), "whether to display the notification screen in the initial state" (area 706; see Figure 10 、 11 For example), "Whether to make a notification based on sound output (sound signal)" (area 708). Furthermore, the user can set "Whether to make a notification at the timing of switching the organs of the subject shown in the medical image" (area 710) and "Whether to make a notification at the timing of capturing the medical image of the subject at a predetermined part (so-called "landmark")" (area 712) by operating the radio button. In addition, when the user turns on the radio button of area 712, he can select the part to be set as the landmark by operating the drop-down menu of area 714. Figure 8 In the example, the “esophagogastric junction (EGJ)” is selected as the landmark, but the landmark may be another site (eg, pharynx).

[0093] The above-mentioned "timing for switching the organs of the subject shown in the medical image" and "timing for capturing the medical image of the subject at a predetermined position" are examples of the "expected end timing" (the timing for expecting the completion of acquisition of medical images of multiple parts to be captured) in the present invention.

[0094] Furthermore, the user can set the "time elapsed from the start of notification to the end (from the notification state to the switching to the non-notification state)" ("pre-specified time") by inputting a value in area 716. After the time (number of seconds) inputted in area 716 has elapsed, the notification control unit 226 switches the notification by the monitor 400 and / or speaker 209A from the notification state to the non-notification state (stops or ends the notification). Figure 8In the example, the duration from the start of notification to its end is 1.0 second. Numerical input can also be performed by selecting a predetermined value from a drop-down menu. By switching to the non-notification state, assistance can be terminated as needed by the user, thereby suppressing excessive assistance. Furthermore, the notification control unit 226 can also reduce (decrease) the notification force after a specified period of time, in addition to or in lieu of the termination of notification.

[0095] In this way, in the endoscope system 10 (medical image processing device, endoscope system), the user can set the notification conditions and notification methods as needed, and the notification control unit 226 performs notification (assistance) according to the setting content, thereby suppressing excessive notifications. In addition, the above example is an example of setting, and other items (such as notifications based on light or vibration) can also be set. In addition, the setting of the notification conditions and notification methods can be performed not only at the beginning of medical image processing, but also at any time during the processing. Moreover, the setting of the notification conditions and notification methods can also be performed automatically by the endoscope system 10 without user operation.

[0096] <Acquisition of Endoscopic Images>

[0097] The medical image acquisition unit 220 acquires a time series of endoscopic images (medical images) (step S110: image acquisition process). The medical image acquisition unit 220 may acquire endoscopic images captured by the endoscope 100 or may acquire endoscopic images 260 recorded in the recording unit 207. When the medical image acquisition unit 220 acquires endoscopic images captured by the endoscope 100, the recording control unit 229 may record the acquired images as endoscopic images 260 in the recording unit 207.

[0098] <Identification of the Photographing Part>

[0099] The part recognition unit 222 (part recognition unit, processor) uses the above-mentioned CNN232 to recognize the part of the subject (the part that has been photographed) reflected in the endoscopic image obtained in step S110 (step S120: part recognition process). As parts, for example, in the case of the esophagus, the cervical esophagus, the chest esophagus, and the abdominal esophagus can be cited. The chest esophagus can be further divided into the upper chest esophagus, the middle chest esophagus, and the lower chest esophagus. In addition, in the case of the stomach, the cardia, the fornix (stomach fundus), the body of the stomach, the gastric angle, the vestibule, the anterior pylorus, and the pyloric ring can be cited. The body of the stomach can be further divided into the upper part, the middle part, and the lower part. In addition, it can also be divided into the lesser curvature, the anterior wall, the greater curvature, and the posterior wall in the circumferential direction.

[0100] The part identification unit 222 (processor) can also determine that the "part is identified" when at least one of the following conditions is met: a specific subject appears in the endoscopic image (medical image) for a period of time or longer; a specific subject appears in a specific area (e.g., the center) of the endoscopic image; a specific subject is captured in the endoscopic image at a size larger than a specific size; and a specific subject is in focus at a level larger than a specific level in the endoscopic image.

[0101] <Record of Photographed Areas>

[0102] The recording control unit 229 records the information of the imaged part (part identification result) as the part identification result 264 in the recording unit 207 (step S130: identification result recording step). The recording control unit 229 preferably records the part identification result in association with the endoscopic image. The recording control unit 229 may record the part identification result in the aforementioned list format.

[0103] <Determination of unimaged areas>

[0104] The determination unit 224 compares the part identified in step S130 (part identification result 264) with the part indicated by the part information 262, and determines which parts of the multiple parts to be imaged have not been imaged (unimaged parts) (step S140: determination step). The determination unit 224 can, for example, determine whether there are unimaged parts and / or which parts have not been imaged. The determination unit 224 can perform the determination each time one or more images are acquired, or each time a specified time period has elapsed.

[0105] <Notification of Determination Result of Expected End Timing>

[0106] The notification control unit 226 determines whether the expected end timing of the acquisition of the endoscopic images (medical images) of the plurality of parts to be photographed has arrived (step S150: notification process). For example, the timing of the change of the part to be observed in the recognition result from one organ to another (for example, the timing of the change from the esophagus to the stomach; ... Figure 8 In the example of setting in area 710), the timing of acquiring a medical image of the subject at a predetermined portion (the timing of acquiring an image of a landmark; in Figure 8In the example of , it is set in areas 712 and 714) as the expected end timing. In addition, the timing of receiving the user operation indicating the end of shooting for the determined range (the timing when the operation receiving unit 227 receives the user operation via the operation unit 208), that is, the timing when the user recognizes that "shooting is completed" can be set as the "expected end timing". In addition, in addition to these examples, the notification control unit 226 can also determine the arrival of the expected end timing based on the expiration of the determined number of shots or shooting time. In addition, in the "expected end timing", it is sufficient to expect the acquisition of the endoscopic image to be completed (as long as there is a possibility), and the acquisition is not actually completed.

[0107] Alternatively, the movement direction estimating unit 228 (movement direction estimating unit) may estimate the movement direction (insertion or removal) of the endoscope 100 based on, for example, the motion vector of the subject, and define the timing when the movement direction changes from insertion (forward direction) to removal (backward direction) as the "expected end timing." Alternatively, the "expected end timing" may be defined as the timing when the user reverses the orientation of the distal end portion of the endoscope 100 and "looks up" (the timing when the endoscope 100 appears in the endoscopic image).

[0108] When the expected end timing has arrived (YES in step S150), the notification control unit 226 notifies the judgment result through the monitor 400 (notification unit, display) and / or the speaker 209A (notification unit, speaker) (step S160: notification process). The notification method follows the content set in step S100 (refer to Figure 8 for example). Figure 9 This is a diagram showing an example of stomach-related notification. Figure 9 Part (a) is a diagram showing the state before the start of observation (all parts are not observed), and the entire area of ​​the image 800 showing the simplified diagram (schematic diagram) of the stomach is not colored (shaded). On the other hand, part (b) of the figure is a diagram showing the notification state when the expected end timing is expected. In image 802, the observed parts (fornix, body of stomach, vestibule, anterior pylorus, greater curvature of the gastric angle) are colored, and the unobserved parts (lesser curvature of the gastric angle) are not colored. In addition, a mark (circle 804) is added to surround the unobserved parts. The simplified diagram of the internal organs (esophagus, stomach, etc.) can be a three-dimensional model or an expanded diagram.

[0109] Furthermore, the notification control unit 226 may color the diagram in advance during normal observation and not color the observed areas (or make the colors lighter) during notification. Alternatively, the colors of the observed areas and the unobserved areas may be changed, or the colors may be changed at the timing of notification, or the unobserved areas may be made to flash.

[0110] Figure 10 It is a diagram showing a display example on the monitor 400 (display). Figure 10 Part (a) of the figure shows the state of normal observation (the state in which the expected end timing has not arrived), and only the endoscopic image 810 (normal observation image) is displayed. On the other hand, part (b) of the figure shows the state of notification of the expected end timing, and an image 802 representing an unobserved part is displayed superimposed (refer to Figure 9 ) of the endoscopic image 812. In addition, the notification control unit 226 may display the image 802 on a screen different from the endoscopic image.

[0111] exist Figure 10 In the illustrated embodiment, the notification control unit 226 displays information (image 802) that was not displayed on the screen before the notification was issued. However, in addition to or in place of the screen display, the notification control unit 226 may output a sound that was not outputted before the notification from the speaker 209A. For example, if there is a part that has not been imaged, the notification control unit 226 may output a warning sound such as a beep or a sound indicating that the part has not been imaged, such as "The lesser curvature of the gastric angle has not been imaged," during the notification.

[0112] In the endoscope system 10, the user can operate the endoscope 100 based on the notification indicating that imaging was not performed (observation was missed) to image (observe) the non-imaged portion, thereby preventing observation from being missed. In addition, since the endoscope system 10 issues the notification at an appropriate timing (expected end timing), there is no possibility of obstructing observation.

[0113] Furthermore, if the determination in step S140 indicates that there are no unimaged parts, the notification control unit 226 preferably provides a notification in a different manner than when there are unimaged parts, such as by coloring the entire diagram with the same color, enclosing the entire diagram with a circle (when displayed on the screen), or outputting a voice message such as "There are no unimaged parts." The notification control unit 226 may also reduce the intensity of the notification (e.g., by making the diagram smaller or lowering the volume) when there are no unimaged parts compared to when there are unimaged parts.

[0114] The CPU 110 and the image processing unit 204 repeat the processes of steps S110 to S160 until the observation is completed (during the period when the answer is NO in step S170 ).

[0115] As described above, according to the medical image processing device, the operating method of the medical image processing device, and the endoscope system involved in the present invention, it is possible to notify observation omissions at appropriate timing.

[0116] <Modification of Notification Method>

[0117] exist Figure 10 In the manner shown, the schematic diagram of the stomach (image 802) is not superimposed during normal observation (the screen display during normal observation is closed; during Figure 8 The radio button is turned off in area 706 of FIG), but Figure 11 As shown in part (a) of FIG, a simplified diagram can also be displayed in an overlapping manner during normal observation (for example Figure 9 The notification control unit 226 is as follows: Figure 11 As shown in part (b) of FIG. 8 , the image 802 is caused to flash on the screen of the monitor 400 at the time of notification (expected end timing). Figure 11 The illustrated method is a method of providing notification by changing the display method of information (image 800 ) already displayed on the monitor 400 (display).

[0118] The notification control unit 226 may also perform notification by changing the output method of the sound output from the speaker 209A in addition to or instead of displaying the information on the screen. For example, the notification control unit 226 can increase the notification effect by changing the content (message content), volume, pitch, and pattern of the sound output during normal observation and the sound output when the expected end time is reached.

[0119] Figure 11 Part (c) shows the situation during normal observation (image 800 is displayed; a cross is attached to icon 816, and there is no sound output), and part (d) of the figure shows the situation during notification (image 802 flashes and sound is output; icon 818 is displayed).

[0120] <Variations of Processing Procedures>

[0121] exist Figure 7 In the method shown in FIG. 1 , the non-imaged portion is continuously determined even outside the expected end timing. However, Figure 12 As shown, the determination unit 224 may also determine the unimaged part at the expected end timing (for example, the timing of receiving the user's instruction, the timing of the observed organ changing, the timing of acquiring the image of the landmark) (step S152: determination process). Figure 12 The flowchart is the same as that of Figure 7 The same parts are omitted here.

[0122] In addition, considering that “notification does not hinder observation or diagnosis”, it is also possible to Figure 13 As shown, after a certain time has passed since the notification, the notification control unit 226 reduces the notification force (including the end of the notification) (steps S162, S164: notification process, refer to Figure 8 area 716).

[0123] <Modification of Part Identification Method>

[0124] In the above-mentioned first embodiment, the case where the part recognition unit 222 uses CNN to perform part recognition is described, but part recognition is not limited to CNN, and general multi-category classification methods based on supervised learning such as support vector machine (SVM) and k-nearest neighbor method (k-NN) can also be used.

[0125] (Note)

[0126] In addition to the above-described embodiment and modifications, the following configurations are also included in the scope of the present invention.

[0127] (Note 1)

[0128] A medical image processing device, wherein:

[0129] The medical image analysis processing unit detects a region of interest as a region of interest based on the feature quantity of pixels of the medical image.

[0130] The medical image analysis result acquisition unit acquires the analysis result of the medical image analysis processing unit.

[0131] (Note 2)

[0132] A medical image processing device, wherein:

[0133] The medical image analysis and processing unit detects whether there is an object of attention based on the feature quantity of the pixels of the medical image.

[0134] The medical image analysis result acquisition unit acquires the analysis result of the medical image analysis processing unit.

[0135] (Note 3)

[0136] A medical image processing device, wherein:

[0137] The medical image analysis result acquisition unit acquires the analysis result of the medical image from the recording device.

[0138] The analysis result is either or both of the region of interest as the region of interest included in the medical image and the presence or absence of an object of interest.

[0139] (Note 4)

[0140] A medical image processing device, wherein:

[0141] A medical image is an ordinary light image obtained by irradiating light in a white wavelength band or irradiating light in a plurality of wavelength bands as light in a white wavelength band.

[0142] (Note 5)

[0143] A medical image processing device, wherein:

[0144] Medical images are images obtained by irradiating light of a specific wavelength band.

[0145] The specific band is a band narrower than the white band.

[0146] (Note 6)

[0147] A medical image processing device, wherein:

[0148] The specific band is either the blue or green band of the visible range.

[0149] (Note 7)

[0150] A medical image processing device, wherein:

[0151] The specific wavelength band includes a wavelength band of 390 nm to 450 nm or a wavelength band of 530 nm to 550 nm, and light in the specific wavelength band has a peak wavelength in the wavelength band of 390 nm to 450 nm or a wavelength band of 530 nm to 550 nm.

[0152] (Note 8)

[0153] A medical image processing device, wherein:

[0154] The specific band is the red band of the visible range.

[0155] (Note 9)

[0156] A medical image processing device, wherein:

[0157] The specific wavelength band includes a wavelength band of 585 nm to 615 nm or a wavelength band of 610 nm to 730 nm, and light in the specific wavelength band has a peak wavelength in the wavelength band of 585 nm to 615 nm or a wavelength band of 610 nm to 730 nm.

[0158] (Note 10)

[0159] A medical image processing device, wherein:

[0160] The specific wavelength band includes a wavelength band where the absorption coefficients of oxidized hemoglobin and reduced hemoglobin are different, and light in the specific wavelength band has a peak wavelength in the wavelength band where the absorption coefficients of oxidized hemoglobin and reduced hemoglobin are different.

[0161] (Note 11)

[0162] A medical image processing device, wherein:

[0163] The specific wavelength band includes 400±10nm, 440±10nm, 470±10nm, or a wavelength band of 600nm to 750nm, and the light in the specific wavelength band has a peak wavelength in the wavelength band of 400±10nm, 440±10nm, 470±10nm, or a wavelength band of 600nm to 750nm.

[0164] (Note 12)

[0165] A medical image processing device, wherein:

[0166] Medical images are images taken inside the body of a living being.

[0167] The in-vivo image contains information on fluorescence emitted by fluorescent substances in the body.

[0168] (Note 13)

[0169] A medical image processing device, wherein:

[0170] Fluorescence is obtained by irradiating a living body with excitation light having a peak value between 390 nm and 470 nm.

[0171] (Note 14)

[0172] A medical image processing device, wherein:

[0173] Medical images are images taken inside the body of a living being.

[0174] The specific wavelength band is the infrared light band.

[0175] (Note 15)

[0176] A medical image processing device, wherein:

[0177] The specific wavelength band includes a wavelength band of 790 nm to 820 nm or a wavelength band of 905 nm to 970 nm, and light in the specific wavelength band has a peak wavelength in the wavelength band of 790 nm to 820 nm or a wavelength band of 905 nm to 970 nm.

[0178] (Note 16)

[0179] A medical image processing device, wherein:

[0180] The medical image acquisition unit includes a special light image acquisition unit that acquires a special light image having information of a specific wavelength band based on an ordinary light image obtained by irradiating light of a white wavelength band or irradiating light of multiple wavelength bands as light of a white wavelength band.

[0181] Medical images are special light images.

[0182] (Note 17)

[0183] A medical image processing device, wherein:

[0184] The signal of a specific wavelength band is obtained by calculation based on RGB or CMY color information included in the normal light image.

[0185] (Note 18)

[0186] A medical image processing device, wherein:

[0187] A feature image generating unit is provided for generating a feature image by performing calculations based on at least one of a normal light image obtained by irradiating light of a white wavelength band or irradiating light of multiple wavelength bands as the white wavelength band, and a special light image obtained by irradiating light of a specific wavelength band.

[0188] Medical images are feature quantity images.

[0189] (Note 19)

[0190] An endoscope device, comprising:

[0191] The medical image processing device according to any one of Supplementary Notes 1 to 18; and

[0192] An endoscope acquires an image by irradiating at least either light in a white wavelength band or light in a specific wavelength band.

[0193] (Note 20)

[0194] A diagnostic aid device, wherein:

[0195] A medical image processing device having any one of Appendixes 1 to 18.

[0196] (Note 21)

[0197] A medical service auxiliary device, wherein:

[0198] A medical image processing device having any one of Appendixes 1 to 18.

[0199] While the embodiments and other examples of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0200] Explanation of symbols

[0201] 10 Endoscopic system

[0202] 100 Endoscope

[0203] 102 Hand operation unit

[0204] 104 Insertion

[0205] 106 General Cable

[0206] 108 optical connector

[0207] 112 Soft Department

[0208] 114 Bend

[0209] 116 top hard part

[0210] 116A top side end face

[0211] 123 Lighting Department

[0212] 123A Lighting Lens

[0213] 123B Lighting Lens

[0214] 126 Clamping mouth

[0215] 130 Photographic Optical System

[0216] 132 Photographic lens

[0217] 134 Camera Components

[0218] 136 drive circuit

[0219] 138 AFE

[0220] 141 Gas and water supply buttons

[0221] 142 Attraction Button

[0222] 143 function buttons

[0223] 144 Shoot button

[0224] 170 Light Guide

[0225] 200 processors

[0226] 202 Image Input Controller

[0227] 204 Image Processing Department

[0228] 205 Communication Control Department

[0229] 206 Video Output Unit

[0230] 207 Records Department

[0231] 208 Operation Department

[0232] 209 Sound Processing Department

[0233] 209A Speaker

[0234] 210 CPU

[0235] 211 ROM

[0236] 212 RAM

[0237] 220 Medical Image Acquisition Department

[0238] 222 Part Identification Department

[0239] 224 Judgment Department

[0240] 226 Notification Control Department

[0241] 227 Operation Acceptance Unit

[0242] 228 Movement Direction Estimation Unit

[0243] 229 Recording Control Department

[0244] 230 Display control unit

[0245] 232A Input Layer

[0246] 232B middle layer

[0247] 232C Output layer

[0248] 234 convolutional layers

[0249] 235 Pooling Layer

[0250] 236 fully connected layers

[0251] 260 Endoscopic Images

[0252] 262 Part Information

[0253] 264 Part Recognition Results

[0254] 266 Judgment Result

[0255] 300 Light Source Device

[0256] 310 light source

[0257] 310B blue light source

[0258] 310G green light source

[0259] 310R red light source

[0260] 310V purple light source

[0261] 330 aperture

[0262] 340 Condenser Lens

[0263] 350 Light source control unit

[0264] 400 monitors

[0265] 700 screens

[0266] 702 Area

[0267] 704 Area

[0268] 706 Area

[0269] 708 Area

[0270] 710 Area

[0271] 712 Area

[0272] 714 Area

[0273] 716 Area

[0274] 800 images

[0275] 802 images

[0276] 804 Circle

[0277] 810 Endoscopic images

[0278] 812 endoscopic images

[0279] 816 Icon

[0280] 818 Icon

[0281] F1 filter

[0282] F2 filter

[0283] S100~S170 Each step of the operation method of the medical image processing device

Claims

1. A medical image processing device comprising: a memory storing in advance part information indicating a plurality of parts of a subject to be photographed; processor; and Reporting Department, in, The processor acquiring a medical image of the subject, The part of the subject appearing in the medical image is identified as the part that has been photographed when at least one of the following conditions is met: a specific subject appears in the medical image for a period of time or longer; a specific subject appears in a specific area of ​​the medical image; and a specific subject appears in the medical image in a size larger than a specific size. The information of the photographed part is recorded in the memory. The information of the photographed part is compared with the part information representing the multiple parts to be photographed stored in the memory, and the information representing the photographed part and the information representing the unphotographed part among the multiple parts to be photographed are separately notified by the notification unit.

2. The medical image processing apparatus according to claim 1, wherein: The processor records information of the site identified as the photographed site in the memory in association with the medical image.

3. The medical image processing apparatus according to claim 1 or 2, wherein: The processor causes the information indicating the imaged region to be displayed on a display device together with the information indicating the unimaged region.

4. The medical image processing apparatus according to claim 1 or 2, wherein: The processor uses a schematic diagram to display information indicating the imaged portion and information indicating the unimaged portion on the same display screen as the medical image.

5. The medical image processing apparatus according to claim 4, wherein: The processor displays information indicating the imaged portion and information indicating the unimaged portion in different display modes in the schematic diagram.

6. The medical image processing apparatus according to claim 1 or 2, wherein: The processor uses a neural network to perform the image recognition.

7. The medical image processing apparatus according to claim 1 or 2, wherein: The processor compares the information of the part that has been photographed with the part information representing the multiple parts that should be photographed, determines the part that has not been photographed among the multiple parts that should be photographed, and reports the result of the determination through the notification unit at the expected end timing when the acquisition of the medical images of the multiple parts that should be photographed is expected to be completed.

8. The medical image processing apparatus according to claim 7, wherein: The processor accepting a user action indicating said termination, The result of the determination is reported using the timing at which the operation is accepted as the expected end timing.

9. The medical image processing apparatus according to claim 7, wherein: The processor reports the result of the determination by using the timing at which the observation target site in the result of the recognition changes from one organ to another organ as the expected end timing.

10. The medical image processing apparatus according to claim 7, wherein: The processor reports the result of the determination using the timing at which a medical image of the subject at a predetermined portion is acquired as the expected end timing.

11. The medical image processing apparatus according to claim 10, wherein: The predetermined site is the esophagogastric junction.

12. The medical image processing apparatus according to claim 10, wherein: The predetermined site is the throat.

13. The medical image processing apparatus according to claim 10, wherein: The processor performs the determination at a timing when the medical image at the predetermined site is acquired.

14. The medical image processing apparatus according to claim 1 or 2, wherein: The processor reduces the notification power of the notification when a predetermined time has elapsed after the notification is issued.

15. The medical image processing apparatus according to claim 1 or 2, wherein: The notification unit includes a display for displaying information on a screen and / or a speaker for outputting sound.

16. The medical image processing apparatus according to claim 15, wherein: The processor performs the notification by changing a display format of the information displayed on the display and / or an output format of the sound output from the speaker.

17. The medical image processing apparatus according to claim 15, wherein: The processor performs the notification by causing the display to display information that was not displayed on the screen before the notification is performed, and / or by causing the speaker to output a new sound that was not output before the notification is started.

18. The medical image processing apparatus according to claim 15, wherein: The processor increases or decreases the notification power of the screen display on the display.

19. An endoscope system, wherein: have: The medical image processing device according to any one of claims 1 to 18; and an endoscope inserted into a subject serving as the imaging object to capture the medical image, The processor acquires the medical image captured by the endoscope.

20. The endoscope system according to claim 19, wherein: The processor Comparing the information of the imaged part with the part information indicating the plurality of parts to be imaged, and determining the part that has not been imaged among the plurality of parts to be imaged, estimating the direction of movement of the endoscope, The result of the determination is reported after the estimated timing when the moving direction changes to the backward direction.

21. An operating method of a medical image processing device, the medical image processing device comprising: a memory storing part information indicating a plurality of predetermined parts of a subject to be photographed; a processor; and a notification unit, wherein: The processor performs: An image acquisition step of acquiring a medical image of the subject; A part recognition process for recognizing the part of the subject appearing in the medical image as the part that has been photographed when at least one of the following conditions is met: a specific subject appears in the medical image for a period of time or longer; a specific subject appears in a specific area of ​​the medical image; and a specific subject appears in the medical image at a size greater than a specific size; a recording step of recording information of the photographed part into a memory; The notification process compares the information of the part that has been photographed with the part information representing the multiple parts to be photographed stored in the memory, and uses the notification unit to distinguish between the information representing the part that has been photographed and the information representing the part that has not been photographed among the multiple parts to be photographed.

22. The operation method of the medical image processing device according to claim 21, wherein: The processor uses a neural network to perform the image recognition.

Images