Medical image processing device, operation method, program, and recording medium

Abstract

Provided are a medical image processing device, a medical image processing method, and a program which can reduce a deviation between recognition by a recognizer and recognition of a user. The medical image processing device is provided with a processor (210), a memory (207), and a monitor (400), wherein the processor (204) sequentially acquires a plurality of medical images in time series, recognizes a specific scene of the medical images on the basis of the acquired medical images, acquires the frequency of recognition of the recognized specific scene, and displays, on a monitor, a notification display that shows, according to the frequency of recognition, the degree of recognition varying in two or more steps.

Description

[Technical field]

[0001] The present invention relates to a medical image processing apparatus, a medical image processing method, and a program. [Background technology]

[0002] Diagnosis and examination using an endoscope device are performed by displaying an endoscopic image acquired by the endoscope device on a monitor and having a doctor observe the endoscopic image displayed on the monitor.

[0003] As described in Patent Document 1, a technique is known that assists a doctor's observation by performing image processing on an endoscopic image to estimate a lesion and superimposing a mark on the lesion and displaying it on a monitor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-87793 A Summary of the Invention [Problem to be solved by the invention]

[0005] Furthermore, in recent years, deep learning has made it possible to automatically recognize objects in images with high accuracy. One application of deep learning to an endoscope device is the use of a recognizer created by deep learning. Specifically, a recognizer created by deep learning automatically recognizes specific scenes from an endoscopic image, and presents the recognized specific scene to a user (such as a doctor) to inform them that the specific scene has been observed.

[0006] However, there may be a discrepancy between the recognizer's recognition of a specific scene and the user's actual observation. For example, even if a specific scene is captured instantaneously when the endoscope is moved quickly, the recognizer may recognize the specific scene, and in that case, the user is notified that the specific scene has been recognized. On the other hand, when a specific scene is captured instantaneously by the user, the user does not observe the specific scene. In this way, there may be a discrepancy between the recognizer's recognition of a specific scene and the user's actual observation.

[0007] Furthermore, since the recognizer notifies the user that a particular scene has already been observed, the user may see the notification and mistakenly recognize that the particular scene has been observed, even though the user has not actually observed the particular scene.

[0008] The present invention has been made in consideration of the above circumstances, and its purpose is to provide a medical image processing device, a medical image processing method, and a program that can reduce the discrepancy between the recognition by the recognizer and the recognition by the user. [Means for solving the problem]

[0009] A medical image processing device, which is one aspect of the present invention for achieving the above-mentioned object, is a medical image processing device equipped with a processor, which sequentially acquires a plurality of medical images in chronological order, recognizes specific scenes in the medical images based on the acquired medical images, acquires the recognition frequency of the recognized specific scenes, and displays on a monitor a notification display indicating the degree of recognition, which changes in two or more stages depending on the recognition frequency.

[0010] According to the present embodiment, a notification display is displayed on the monitor indicating the degree of recognition, which changes in two or more stages depending on the recognition frequency of a specific scene in a medical image, so that the user can be notified of the recognition frequency of a specific scene in the processor, thereby reducing the discrepancy between the user's observation of a specific scene and the processor's recognition of the specific scene.

[0011] Preferably, the processor causes the monitor to display different colors as the notification display depending on the recognition frequency.

[0012] Preferably, the processor causes the monitor to display different line types as the notification display depending on the recognition frequency.

[0013] Preferably, the processor causes the monitor to display, as a notification display, a different degree of blur in accordance with the recognition frequency.

[0014] Preferably, the processor obtains the recognition frequency based on the number of medical images of the recognized specific scene.

[0015] Preferably, the processor obtains the recognition frequency based on the number of chronologically consecutive medical images of the recognized specific scene.

[0016] Preferably, the processor recognizes the specific scene by calculating a feature amount of the medical image of the specific scene and classifying the medical image based on the feature amount of the specific scene.

[0017] Preferably, the processor calculates a feature amount of a medical image of a specific scene, and recognizes the specific scene based on a similarity between the feature amount and the medical image.

[0018] Preferably, when the recognition frequency is equal to or greater than a first threshold, the processor switches the notification display from an initial display to the first display and displays it on the monitor.

[0019] Preferably, when the recognition frequency is equal to or greater than a second threshold value that is greater than the first threshold value, the processor switches the notification display from the first display to a second display and displays the second display on the monitor.

[0020] Preferably, the processor displays a plurality of reference images, each of which includes a respective one of a plurality of specific scenes, on the monitor to provide the first display and the second display.

[0021] Preferably, the processor displays, on the monitor, a model image including parts corresponding to a plurality of specific scenes, and performs the first display and the second display by associating the recognized specific scenes with the model image.

[0022] Preferably, the processor causes the monitor to display a notification display superimposed on the model image, by matching the position where the specific scene is recognized with a position on the model image.

[0023] Preferably, when the specific scene has not yet been recognized, the processor causes the monitor to display an initial display different from the notification display.

[0024] Another aspect of the present invention is a medical image processing method that includes a medical image acquisition step of sequentially acquiring a plurality of medical images in a time series, a specific scene recognition step of recognizing a specific scene in the medical image based on the acquired medical image, a frequency acquisition step of acquiring the recognition frequency of the specific scene recognized by the specific scene recognition step, and a display control step of displaying on a monitor a notification display indicating the degree of recognition, which changes in two or more stages depending on the recognition frequency.

[0025] Another aspect of the present invention is a program that causes a computer to execute a medical image processing method including a medical image acquisition step of sequentially acquiring a plurality of medical images in chronological order, a specific scene recognition step of recognizing a specific scene in the medical image based on the acquired medical image, a frequency acquisition step of acquiring the recognition frequency of the specific scene recognized by the specific scene recognition step, and a display control step of displaying on a monitor a notification display indicating the degree of recognition, which changes in two or more stages depending on the recognition frequency. Effect of the Invention

[0026] According to the present invention, it is possible to reduce the discrepancy between recognition by a recognizer and recognition by a user. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is an external view of the endoscope system. [Diagram 2] FIG. 2 is a block diagram showing a configuration of a main part of the endoscope system. [Diagram 3] FIG. 3 is a functional block diagram of the medical image processing device in the image processing unit. [Figure 4] FIG. 4 is a diagram showing the configuration of a CNN. [Diagram 5] FIG. 5 is a schematic diagram showing a configuration example of the intermediate layer. [Figure 6] FIG. 6 is a flowchart showing a medical image processing method. [Figure 7] FIG. 7 is a flow diagram of acquiring the recognition frequency. [Figure 8] FIG. 8 is a diagram showing an example of a notification display. [Figure 9] FIG. 9 is a flow diagram of a medical image processing method. [Figure 10] FIG. 10 is a flow diagram of obtaining the recognition frequency. [Figure 11] FIG. 11 is a diagram showing a specific example of the first modified example 1. In FIG. [Figure 12] FIG. 12 is a diagram showing a specific example of the second modified example 1. In FIG. [Figure 13] FIG. 13 is a diagram showing a specific example of the third modified example 1. In FIG. [Figure 14] FIG. 14 is a diagram showing a specific example of the fourth modified example 1. In FIG. [Figure 15] FIG. 15 is a diagram showing a specific example of the fifth modified example 1. In FIG. [Figure 16] FIG. 16 is a diagram showing the first modified example 2. In FIG. [Figure 17] FIG. 17 is a diagram showing the second modified example 2. In FIG. [Figure 18] FIG. 18 is a diagram showing a case where display is performed on two monitors. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a medical image processing apparatus, a medical image processing method, and a program according to the present invention will now be described with reference to the accompanying drawings.

[0029] <Endoscope system configuration> Fig. 1 is an external view of an endoscope system 10, and Fig. 2 is a block diagram showing the configuration of the main parts of the endoscope system 10. As shown in Figs. 1 and 2, the endoscope system 10 is composed of an endoscope scope 100, an endoscope processor device 200, a light source device 300, and a monitor 400. The endoscope processor device 200 is equipped with the medical image processing device of the present invention.

[0030] <Configuration of an endoscope> The endoscope 100 includes a handheld operation unit 102 and an insertion section 104 connected to the handheld operation unit 102. The operator (user) holds and operates the handheld operation unit 102, and inserts the insertion section 104 into the body of a subject (living body) for observation. The handheld operation unit 102 is provided with an air / water supply button 141, a suction button 142, function buttons 143 to which various functions are assigned, and a shooting button 144 for receiving shooting instruction operations (still image, moving image). The insertion section 104 is composed of a flexible section 112, a bending section 114, and a tip rigid section 116, in that order from the handheld operation unit 102 side. That is, the bending section 114 is connected to the base end side of the tip rigid section 116, and the flexible section 112 is connected to the base end side of the bending section 114. The handheld operation unit 102 is connected to the base end side of the insertion section 104. A user can bend the bending section 114 by operating the hand operation section 102 to change the orientation of the tip rigid section 116 up, down, left, or right. The tip rigid section 116 is provided with an imaging optical system 130, an illumination section 123, a forceps port 126, etc. (see FIGS. 1 and 2).

[0031] During observation and treatment, white light and / or narrowband light (one or more of red narrowband light, green narrowband light, blue narrowband light, and purple narrowband light) can be irradiated from the illumination lenses 123A and 123B of the illumination unit 123 by operating the operation unit 208 (see FIG. 2). In addition, cleaning water can be discharged from a water supply nozzle (not shown) by operating the air / water supply button 141 to clean the photographing lens 132 of the photographing optical system 130 and the illumination lenses 123A and 123B. A conduit (not shown) is connected to the forceps port 126 opening at the tip rigid portion 116, and a treatment tool (not shown) for tumor removal or the like is inserted into this conduit, and the treatment tool can be advanced and retreated as appropriate to perform necessary treatment on the subject.

[0032] As shown in FIG. 1 and FIG. 2, a photographing lens 132 is disposed on the tip end surface 116A of the tip rigid portion 116. A CMOS (Complementary Metal-Oxide Semiconductor) type image sensor 134, a driving circuit 136, and an AFE 138 (AFE: Analog Front End) are disposed behind the photographing lens 132, and an image signal is output by these elements. The image sensor 134 is a color image sensor, and has a plurality of pixels constituted by a plurality of light receiving elements arranged in a matrix (two-dimensional array) in a specific pattern array (Bayer array, X-Trans (registered trademark) array, honeycomb array, etc.). Each pixel of the image sensor 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (photodiode, etc.). The photographing optical system 130 can generate a color image from pixel signals of three colors, red, green, and blue, or can generate an image from pixel signals of any one or two colors of red, green, and blue. The image sensor 134 may be of a CCD (Charge Coupled Device) type. Each pixel of the image sensor 134 may further include a purple color filter corresponding to the purple light source 310V and / or an infrared filter corresponding to the infrared light source.

[0033] An optical image of the subject is formed on the light receiving surface (imaging surface) of the image sensor 134 by the photographing lens 132, converted into an electric signal, and output to the endoscope processor device 200 via a signal cable (not shown) and converted into a video signal. As a result, an endoscopic image (medical image) of the subject is displayed on the screen of the monitor 400 connected to the endoscope processor device 200.

[0034] Furthermore, on the tip end surface 116A of the tip hard portion 116, illumination lenses 123A and 123B of the illumination portion 123 are provided adjacent to the photographing lens 132. An exit end of a light guide 170 (described later) is disposed behind the illumination lenses 123A and 123B. This light guide 170 is inserted through the insertion portion 104, the handheld operation portion 102, and the universal cable 106, and an entrance end of the light guide 170 is disposed in the light guide connector 108.

[0035] A user can capture time-series endoscopic images of the inside of a living body at a determined frame rate while inserting or removing the endoscope 100 having the above-described configuration into or from the living body.

[0036] <Light source configuration> 2, the light source device 300 is composed of an illumination light source 310, an aperture 330, a condenser lens 340, a light source control unit 350, etc., and causes observation light to enter the light guide 170. The light source 310 includes a red light source 310R, a green light source 310G, a blue light source 310B, and a purple light source 310V that respectively irradiate narrowband light of red, green, blue, and purple, and can irradiate narrowband light of red, green, blue, and purple. The illuminance of the observation light from the light source 310 is controlled by the light source control unit 350, and the illuminance of the observation light can be changed (increased or decreased) and the illumination can be stopped as necessary.

[0037] The light source 310 can emit narrowband light of red, green, blue, and purple in any combination. For example, narrowband light of red, green, blue, and purple can be emitted simultaneously to irradiate white light (normal light) as the observation light, or narrowband light (special light) can be irradiated by emitting one or two of them. The light source 310 may further include an infrared light source that irradiates infrared light (an example of narrowband light). In addition, white light or narrowband light may be irradiated as the observation light by using a light source that irradiates white light and a filter that transmits the white light and each narrowband light.

[0038] <Light source wavelength band> The light source 310 may be a light source that generates light in a white band or a plurality of wavelength bands as white band light, or may be a light source that generates light in a specific wavelength band narrower than the white wavelength band. The specific wavelength band may be a blue band or a green band in the visible range, or a red band in the visible range. When the specific wavelength band is a blue band or a green band in the visible range, it may include a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and may have a peak wavelength within the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. When the specific wavelength band is a red band in the visible range, it may include a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and the light in the specific wavelength band may have a peak wavelength within the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm.

[0039] The light in the specific wavelength band may include a wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different, and may have a peak wavelength in the wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different. In this case, the specific wavelength band may include a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm to 750 nm, and may have a peak wavelength in a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm to 750 nm.

[0040] Furthermore, the light generated by light source 310 may include a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm, and may have a peak wavelength in the wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.

[0041] The light source 310 may also include a light source that irradiates excitation light with a peak of 390 nm or more and 470 nm or less. In this case, an endoscopic image having information on the fluorescence emitted by a fluorescent substance in the subject (living body) can be obtained. When obtaining a fluorescent image, a dye agent for the fluoroscopy (fluorescein, acridine orange, etc.) may be used.

[0042] It is preferable to configure the type of light source (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), etc.), wavelength, presence or absence of a filter, etc. of the light source 310 according to the type and location of the subject, the purpose of observation, etc., and it is also preferable to combine and / or switch the wavelength of the observation light according to the type and location of the subject, the purpose of observation, etc., during observation. When switching the wavelength, the wavelength of the irradiated light may be switched, for example, by rotating a disk-shaped filter (rotary color filter) that is placed in front of the light source and has a filter that transmits or blocks light of a specific wavelength.

[0043] In addition, the imaging element used when carrying out the present invention is not limited to a color imaging element in which a color filter is arranged for each pixel, such as the imaging element 134, but may be a monochrome imaging element. When a monochrome imaging element is used, the wavelength of the observation light can be switched sequentially to capture images in a field sequential (color sequential) manner. For example, the wavelength of the emitted observation light may be switched sequentially between (purple, blue, green, red), or broadband light (white light) may be irradiated and the wavelength of the emitted observation light may be switched by a rotary color filter (red, green, blue, purple, etc.). Also, one or more narrowband lights (green, blue, purple, etc.) may be irradiated and the wavelength of the emitted observation light may be switched by a rotary color filter (green, blue, purple, etc.). The narrowband light may be infrared light of two or more different wavelengths (first narrowband light, second narrowband light).

[0044] By connecting the light guide connector 108 (see Figures 1 and 2) to the light source device 300, the observation light irradiated from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and is irradiated onto the observation range from the illumination lenses 123A and 123B.

[0045] <Configuration of the endoscope processor device> The configuration of the endoscope processor device 200 will be described with reference to Fig. 2. The endoscope processor device 200 receives an image signal output from the endoscope scope 100 via an image input controller 202, performs necessary image processing in an image processing unit 204, and outputs the signal via a video output unit 206. As a result, an endoscopic image is displayed on a monitor 400. These processes are performed under the control of a CPU 210 (CPU: Central Processing Unit). The CPU 210 functions as a processor of the medical image processing device. The communication control unit 205 controls communications regarding the acquisition of medical images between an in-hospital system (HIS: Hospital Information System) and an in-hospital LAN (Local Area Network), not shown, and / or an external system or network.

[0046] <Image processing unit functions> The image processing unit 204 can calculate the feature amount of the endoscopic image, emphasize or reduce components of a specific frequency band, and emphasize or make inconspicuous a specific target (such as a region of interest or blood vessels at a desired depth). The image processing unit 204 may include a special light image acquisition unit (not shown) that acquires a special light image having information of a specific wavelength band based on a normal light image acquired by irradiating light of a white band or light of a plurality of wavelength bands as light of a white band. In this case, a signal of a specific wavelength band can be acquired by calculation based on color information of RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) included in the normal light image. The image processing unit 204 may also include a feature image generation unit (not shown) that generates a feature image by calculation based on at least one of a normal light image acquired by irradiating light of a white band or light of a plurality of wavelength bands as light of a white band and a special light image acquired by irradiating light of a specific wavelength band, and acquire and display the feature image as an endoscopic image. The above-mentioned processing is performed under the control of the CPU 210.

[0047] Furthermore, the image processing unit 204 has the functions of a medical image processing apparatus as described below.

[0048] 3 is a functional block diagram of the medical image processing apparatus in the image processing unit 204. The image processing unit 204 includes a medical image acquisition unit 220, a specific scene recognition unit 222, a frequency acquisition unit 224, and a display control unit 226.

[0049] <Realization of functions using various processors> The functions of each part of the image processing unit 204 described above can be realized using various processors and recording media. The various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. The various processors described above also include a GPU (Graphics Processing Unit), which is a processor specialized for image processing, and a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacturing. When learning and recognizing images as in the present invention, a configuration using a GPU is effective. Furthermore, the various processors described above also include dedicated electric circuits, which are processors having a circuit configuration designed specifically for executing specific processing, such as an ASIC (Application Specific Integrated Circuit).

[0050] The functions of each part may be realized by one processor, or may be realized by multiple processors of the same type or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functions may be realized by one processor. As an example of configuring multiple functions by one processor, first, as represented by a computer, there is a form in which one processor is configured by a combination of one or more CPUs and software, and this processor realizes multiple functions. Second, as represented by a system on chip (SoC), there is a form in which a processor that realizes the functions of the entire system by one IC (Integrated Circuit) chip is used. In this way, various functions are configured by using one or more of the above-mentioned various processors as a hardware structure. Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) that combines circuit elements such as semiconductor elements. These electric circuits may be electric circuits that realize the above-mentioned functions using logical sum, logical product, logical negation, exclusive logical sum, and logical operations combining these.

[0051] When the above-mentioned processor or electric circuit executes the software (program), a code readable by a computer (for example, various processors and electric circuits constituting the image processing unit 204, and / or a combination thereof) of the software to be executed is stored in a non-temporary recording medium such as a ROM 211 (Read Only Memory), and the computer refers to the software. The software stored in the non-temporary recording medium includes a program for executing the operating method of the medical image processing device according to the present invention and data used during execution (data related to acquisition of medical images, data used to specify the notification conditions and notification modes, parameters used in the recognition unit, etc.). The code may be recorded in a non-temporary recording medium such as various magneto-optical recording devices and semiconductor memories instead of the ROM 211. When processing using the software, for example, a RAM 212 (RAM: Random Access Memory) is used as a temporary storage area, and data stored in, for example, an EEPROM (Electronically Erasable and Programmable Read Only Memory) not shown can also be referenced. The recording unit 207 may be used as the "non-temporary recording medium".

[0052] Furthermore, a ROM 211 (Read Only Memory) is a non-volatile storage element (non-temporary recording medium) that stores computer-readable code of a program that causes the CPU 210 and / or the image processing unit 204 to execute various image processing methods. A RAM 212 (Random Access Memory) is a storage element for temporary storage during various processes, and can also be used as a buffer when acquiring images. The audio processing unit 209 outputs voice and sound from a speaker 209A under the control of the CPU 210.

[0053] The operation unit 208 can be configured with devices such as a keyboard and a mouse (not shown), and a user can issue instructions to execute processing and specify conditions required for execution via the operation unit 208 .

[0054] The recording unit 207 records acquired endoscopic images and also records programs and information for controlling the medical image processing device. The recording unit 207 functions as a memory for the medical image processing device.

[0055] <Specific scene recognition using neural network> The specific scene recognition unit 222 in the image processing unit 204 described above is configured with a recognizer that recognizes a specific scene, which is configured using a trained model such as a neural network (a model trained using an image set consisting of images of a living body). Here, the specific scene is, for example, the esophagus, duodenum, cardia, pylorus, gastric angle, fundus, body, antrum, lesser curvature, greater curvature, etc. Below, a configuration in the case where a CNN (Convolutional Neural Network) is used as the neural network will be described.

[0056] <Example of recognition unit configuration> FIG. 4 is a diagram showing the configuration of the CNN 232 (neural network). In the example shown in part (a) of FIG. 4, the CNN 232 has an input layer 232A, an intermediate layer 232B, and an output layer 232C. The input layer 232A inputs an endoscopic image acquired by the medical image acquisition unit 220 and outputs a feature amount. The intermediate layer 232B includes a convolution layer 234 and a pooling layer 235, and inputs the feature amount output by the input layer 232A to calculate other feature amounts. These layers have a structure in which a plurality of "nodes" are connected by "edges", and weighting coefficients applied to the input image are associated with the nodes and edges and stored in a weighting coefficient storage unit (not shown). The value of the weighting coefficient changes as learning progresses.

[0057] <Processing in the middle layer> The intermediate layer 232B calculates features by convolution and pooling. The convolution performed in the convolution layer 234 is a process of acquiring a feature map by convolution using a filter, and plays a role in extracting features such as edge extraction from an image. The convolution using this filter generates a "feature map" of one channel (one sheet) for one filter. When downscaling is performed by convolution, the size of the "feature map" becomes smaller as convolution is performed in each layer. The pooling process performed in the pooling layer 235 is a process of reducing (or enlarging) the feature map output by the convolution to create a new feature map, and plays a role in providing robustness so that the extracted features are not affected by translation or the like. The intermediate layer 232B can be configured with one or more layers that perform these processes. Note that the CNN 232 may be configured without the pooling layer 235.

[0058] The CNN 232 may include a fully connected layer 236 as shown in the example of part (b) of Fig. 4. The layer configuration of the CNN 232 is not limited to the case where the convolutional layer 234 and the pooling layer 235 are repeated one by one, but may include any layer (for example, the convolutional layer 234) in a series of multiple layers.

[0059] FIG. 5 is a schematic diagram showing an example of the configuration of the intermediate layer 232B of the CNN 232 shown in FIG. 4. In the first (first) convolutional layer of the intermediate layer 232B, a convolutional operation is performed between an image set consisting of a plurality of endoscopic images and a filter F1. The image set is composed of N images (N channels) having an image size of H vertically and W horizontally. When a normal light image is input, the images constituting the image set are images of three channels, R (red), G (green), and B (blue). The filter F1 convolved with this image set has an image set of N channels (N images), so that, for example, in the case of a filter of size 5 (5×5), the filter size is a filter of 5×5×N. A "feature map" of one channel (one image) is generated for one filter F1 by a convolutional operation using this filter F1. For example, in the case of a filter of size 3 (3×3), the filter size of the filter F2 used in the second convolutional layer is 3×3×M.

[0060] Similar to the first convolutional layer, the second to nth convolutional layers use filters F2 to F n The size of the "feature map" in the nth convolutional layer is smaller than that of the "feature map" in the second convolutional layer because it has been downscaled by the previous convolutional layers or pooling layers.

[0061] Among the layers in intermediate layer 232B, low-order feature extraction (such as edge extraction) is performed in the convolutional layers closer to the input side, and higher-order feature extraction (extraction of features related to the shape, structure, etc. of the recognition target) is performed as the layers approach the output side.

[0062] The intermediate layer 232B may include a layer that performs batch normalization in addition to the convolution layer 234 and the pooling layer 235. The batch normalization process is a process that normalizes the distribution of data in units of mini-batches when performing learning, and plays a role in speeding up learning, reducing dependency on initial values, suppressing overlearning, etc.

[0063] The output layer 232C outputs the feature amount calculated by the intermediate layer 232B in a format suitable for specific scene recognition. The output layer 232C may include a fully connected layer.

[0064] <Each step of the medical image processing method> Next, a medical image processing method using a medical image processing device will be described. Note that, although the following describes a case where an endoscopic image acquired by an endoscope system 10 is processed as a medical image, the application of the present invention is not limited to this. For example, in the present invention, an endoscopic image acquired by another endoscope system 10 can also be processed as a medical image by a medical image processing device.

[0065] 6 is a flow chart showing the medical image processing method. Each step will be described below with reference to FIG.

[0066] Medical image acquisition step The medical image acquisition unit 220 sequentially acquires a plurality of medical images in chronological order (step S10). The medical image acquisition unit 220 may acquire medical images of the body cavity photographed by the endoscope 100, or may acquire medical images of the body cavity recorded in the recording unit 207.

[0067] Specific scene recognition step The specific scene recognition unit 222 recognizes a specific scene in the acquired medical image. The specific scene recognition unit 222 receives the medical image, inputs the medical image to the above-mentioned CNN, and gradually extracts features from the medical image. The specific scene recognition unit 222 finally recognizes the specific scene by performing class classification (for example, esophagus, duodenum, cardia, pylorus, gastric angle, fundus, gastric body, antrum, lesser curvature, greater curvature, and 11 other class classifications) using the features. Note that a detailed explanation of this recognition method is described in the following literature (B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In Neural Information Processing Systems (NIPS), pages 487-495, 2014. 1, 4, 6, 8)). The specific scene recognition unit 222 can also recognize a specific scene by comparing the finally extracted feature amount with a feature amount (e.g., a feature amount of the duodenum) previously stored in the recording unit 207 and calculating the similarity. A detailed explanation of this recognition method is described in the following document (FaceNet: A Unified Embedding for Face Recognition and Clustering https: / / arxiv.org / abs / 1503.03832).

[0068] If the specific scene recognition unit 222 cannot recognize a specific scene in the received medical image (No in step S11), the medical image acquisition unit 220 acquires the next medical image (step S10) if there is a next medical image (Yes in step S19). On the other hand, if the specific scene recognition unit 222 can recognize a specific scene in the medical image (Yes in step S11), the process proceeds to a frequency acquisition step (step S12) for updating the record of the recognition frequency.

[0069] Frequency acquisition step When the specific scene recognition unit 222 recognizes a specific scene in the medical image, the frequency acquisition unit 224 updates the record of the recognition frequency (step S12). For example, when the specific scene recognition unit 222 recognizes the duodenum as a specific scene, the recognition frequency of the duodenum is updated from 0 to 1. There are various possible modes for updating the recognition frequency, and for example, the recognition frequency may be updated from 0 to 64. The updating of the recognition frequency will be described in detail later.

[0070] Display control steps If the recognition frequency is equal to or greater than the first threshold (Yes in step S13), the initial display is being displayed on monitor 400, and the first display, which is one of the notification displays, has not yet been displayed on monitor 400 (No in step S14), the display control unit 226 changes the initial display to the first display (step S15).

[0071] On the other hand, if the recognition frequency is equal to or greater than the first threshold (Yes in step S13), the first display is already being displayed on the monitor 400 (Yes in step S14), and the recognition frequency is equal to or greater than the second threshold (Yes in step S16), the display control unit 226 changes the initial display to the second display. Note that the first threshold and the second threshold are appropriately set by the user. In this way, by gradually changing the display according to the recognition frequency of the specific scene by the specific scene recognition unit 222 and displaying the initial display, the first screen, or the second screen on the monitor 400, the user can know the recognition frequency.

[0072] If the recognition frequency is less than the first threshold (No in step S13), if the recognition frequency is less than the second threshold (No in step S16), or if the second display is already being displayed on the monitor 400 (Yes in step S17), and if there is a next medical image, the medical image acquisition unit 220 acquires the next medical image (step S10).

[0073] <Frequency acquisition step> Next, the frequency acquisition step performed by the frequency acquisition unit 224 will be described.

[0074] Fig. 7 is a flow diagram of acquiring the recognition frequency. Fig. 7 explains a case where the recognition frequency is updated according to the number of images in which a specific scene (e.g., the pylorus) is recognized in an endoscopic image. Note that n in the figure is the number of images in which a specific scene is recognized, and N is a frequency update threshold that is appropriately set by the user.

[0075] The initial state is n=0 (step S20). The medical image acquisition unit 220 receives a medical image (step S21). If the specific scene recognition unit 222 recognizes a specific scene in the medical image (Yes in step S22), the number of images n in which the specific scene is recognized is updated to n+1 (step S23). If the updated number of images n is equal to or greater than the frequency threshold N (Yes in step S24), the frequency acquisition unit 224 updates the recognition frequency (step S25). For example, if the frequency threshold is 5 (N=5) and the specific scene recognition unit 222 recognizes the pylorus in five medical images, the frequency acquisition unit 224 updates the recognition frequency of the pylorus. Then, the process proceeds to the display control steps (steps S13 to 18: step S26) described in FIG. 6.

[0076] If the specific scene recognition unit 222 does not recognize a specific scene in the acquired medical image (No in step S22), if the updated number of images n is smaller than the frequency threshold N (No in step S24), and if there is a next medical image (Yes in step S27) after the display control steps (steps S13 to S18) described in Figure 6, the medical image acquisition unit 220 acquires the next medical image.

[0077] The specific scene recognized by the specific scene recognition unit 222 may be one or more specific scenes. When the specific scene recognized by the specific scene recognition unit 222 is a multiple-scene scene, a frequency threshold is set for each specific scene, and the recognition frequency is updated for each specific scene.

[0078] <Notification display> Next, the notification display on the monitor 400 performed by the display control unit 226 will be described.

[0079] FIG. 8 is a diagram showing an example of a notification display on the monitor 400 performed by the display control unit 226. As shown in FIG.

[0080] 8, an endoscopic image (medical image) 301 captured in real time is displayed in the main display area of ​​the monitor 400. In addition, a notification display 303 consisting of a reference image is displayed in the sub display area of ​​the monitor 400.

[0081] The notification display 303 is made up of reference images (303A to 303I) of specific scenes 1 to 9.

[0082] For specific scenes 4, 8, and 9 (reference symbols 303D, 303H, and 303I) that have not yet been recognized by the specific scene recognition unit 222, only the frame line 305 is displayed as an initial display. In this way, by making the initial display different from the notification display, the user can know that the specific scene has not yet been recognized by the specific scene recognition unit 222. On the other hand, for specific scenes 1 to 3, 5, 6, and 7 (reference symbols 303A, 303B, 303C, 303E, 303F, 303H, and 303I), the recognition frequency is equal to or greater than the first threshold, and the first display (reference image 307 of the specific scene) is displayed. For specific scene 2, the recognition frequency is equal to or greater than the second threshold, and the second display (the line type of the frame line 305 is changed from a solid line to a dotted line) is displayed.

[0083] As described above, in this embodiment, the notification display is switched according to the recognition frequency of a specific scene in a medical image and displayed on the monitor 400, so that the user can know the recognition frequency of the specific scene in the medical image processing device (specific scene recognition unit 222). Therefore, this embodiment can reduce the discrepancy between the recognition of the medical image processing device (specific scene recognition unit 222) and the user's recognition.

[0084] Next, a modification of the above-described embodiment will be described.

[0085] <Modifications of each process of the medical image processing method> A modified example of each process of the medical image processing method will be described. In the above embodiment, the first display and the second display are displayed on the monitor 400. In this example, the third display or more are displayed, and a notification display is displayed on the monitor 400.

[0086] Fig. 9 is a flow diagram of the medical image processing method in this example. The value of m in Fig. 9 indicates the number of the screen in the notification display. A threshold value is set for each notification display. Therefore, the mth display is displayed when the recognition frequency is equal to or greater than the mth threshold value.

[0087] The initial state is m=1 (step S30). The medical image acquisition unit 220 acquires a plurality of medical images in chronological order (step S31). Thereafter, if the specific scene recognition unit 222 recognizes a specific scene in the acquired medical image (if Yes in step S32), the frequency acquisition unit 224 updates the record of the recognition frequency (step S33). Thereafter, if the recognition frequency is equal to or greater than the m-th threshold (if Yes in step S34) and the m-th display has not yet been displayed on the monitor 400 (if No in step S35), the display control unit 226 changes to the m-th display (step S36). On the other hand, if the m-th display has already been displayed on the monitor 400, m=m+1 (step S37).

[0088] If the specific scene recognition unit 222 fails to recognize a specific scene (No in step S32), and after changing to the mth display (step S36), if there is a next medical image (Yes in step S38), the medical image acquisition unit 220 acquires the medical image.

[0089] As described above, in this example, the change is not limited to the first and second display, but may be changed to the third display or higher. This allows the user to more accurately grasp the recognition frequency of a specific scene in the medical image processing device.

[0090] <Modification of Recognition Frequency Update> Next, a modified example of updating the recognition frequency will be described. In Fig. 7, an example of updating the recognition frequency based on the number of medical images in which a specific scene was recognized was described, but in this example, the recognition frequency is updated based on the number of recognized images that are continuous in time series. Note that n in the figure is the number of images in which a specific scene was recognized, and N is a frequency update threshold that is appropriately set by the user.

[0091] FIG. 10 is a flow diagram of acquisition of the recognition frequency in the frequency acquisition unit 224 of this embodiment.

[0092] The initial state is n=0 (step S40). The medical image acquisition unit 220 acquires medical images continuously in time series (step S41). If the specific scene recognition unit 222 recognizes a specific scene in a medical image (Yes in step S42), the number of images n in which a specific scene is recognized is updated to n+1 (step S44). On the other hand, if the specific scene recognition unit 222 cannot recognize a specific scene, the counter is set to n=0 (step S49), and if there is a next medical image, the next medical image is acquired.

[0093] If the updated number of images n in which the specific scene is recognized is equal to or greater than the frequency threshold value N (Yes in step S45), the recognition frequency is updated (step S46). Then, the display control steps (steps S13 to S18: step S47) described in Fig. 7 are performed. Then, if there is a next medical image (Yes in step S48), the process proceeds to processing the next endoscopic image.

[0094] As described above, in this example, when a specific scene is recognized in a predetermined number of consecutive medical images, the recognition frequency is updated. Therefore, the recognition result of the specific scene recognition unit 222 can be output by accurately reflecting what the user is observing.

[0095] <Modification 1 of Notification Display> Next, a first modified example of the notification display will be described. In this example, various modified examples will be described in which a reference image is used as the notification display.

[0096] 11 to 15 are diagrams illustrating specific examples of the first modified example 1 to the fifth modified example 1 of the notification display. In the diagrams, IMGB is an initial display, IMG1 is a first display to which the initial display is switched, and IMG2 is a second display to which the first display is switched.

[0097] In the first modified example 1 shown in FIG. 11, the color of the frame line changes according to the recognition frequency.

[0098] For example, in IMG B, the frame 311 is black, in IMG 1 the frame 313 is white, and in IMG 2 the frame 315 is red. In this way, a user who sees the notification display of this example can understand the recognition frequency of the specific scene recognition unit 222 from the color of the frame.

[0099] In the second modified example 1 shown in FIG. 12, the type (line type) of the frame line changes according to the recognition frequency.

[0100] For example, in IMG1, the frame line 323 is a roughly dotted line compared to the frame line 321, and in IMG2, the frame line 325 is a solid line. In this way, a user who sees the notification display of this example can understand the recognition frequency of the specific scene recognition unit 222 from the type of the frame line.

[0101] In the third modified example 1 shown in FIG. 13, the background color of the reference image changes according to the recognition frequency.

[0102] For example, in IMGB, the background 331 is white, in IMG1, the background 333 is gray, and in IMG2, the background 335 is black. In this way, a user who sees the notification display of this example can understand the recognition frequency of the specific scene recognition unit 222 from the background color.

[0103] In the fourth modification example 1 shown in FIG. 14, the color of the periphery of the reference image is changed.

[0104] For example, in IMGB, the periphery 341 of the reference image is white, in IMG1, the periphery 343 of the reference image is pink, and in IMG2, the periphery 345 of the reference image is red. In this way, a user who sees the notification display of this example can understand the recognition frequency of the specific scene recognition unit 222 from the color of the reference image.

[0105] In the fifth modified example 1 shown in FIG. 15, the image of the reference image gradually becomes clearer (the degree of blur changes).

[0106] For example, in IMGB, no reference image is displayed, and a plain image 351 is displayed, in IMG1, a blurred (blurred) reference image 353 is displayed, and in IMG2, a clear reference image 355 is displayed. In this way, a user who sees the notification display of this example can understand the recognition frequency of the specific scene recognition unit 222 from the clarity (blurredness) of the reference image.

[0107] <Modification 2 of Notification Display> Next, a description will be given of a second modified example of the notification display. In this example, the notification display is superimposed on a model image including parts corresponding to a plurality of specific scenes.

[0108] FIG. 16 is a diagram showing the first modified example 2. In this example, a model image 361 that imitates a stomach is displayed as a notification display. The model image 361 includes parts corresponding to a plurality of specific scenes. When each specific scene is recognized, a band display with a different color according to the recognition frequency is displayed as a notification display. In the case shown in FIG. 16, a band display 363A that is a first display is displayed for the specific scene A, and a band display 363D that is a first display is displayed for the specific scene D. In addition, a band display 363B that is a second display is displayed for the specific scene B, a band display 363C that is a second display is displayed for the specific scene C, and a band display 363E that is a second display is displayed for the specific scene E. Note that the band displays 363A and 363D that are the first display are displayed in the same color, and the band displays 363B, 363C, and 363E that are the second display are displayed in the same color. In this way, a user who sees the notification display of this example can grasp the recognition frequency of the specific scene recognition unit 222 by the color of the band display superimposed on the model image.

[0109] FIG. 17 is a diagram showing the second modified example 2. In this example, a model image 371 that imitates a stomach is displayed as a notification display. The model image 371 includes parts corresponding to a plurality of specific scenes. When each specific scene is recognized to be equal to or greater than a predetermined threshold, a recognition mark 373 is displayed as a notification display at a position corresponding to the recognized specific scene. The recognition marks 373 are displayed overlapping depending on the position of the specific scene, and the color of the overlapped portion of the recognition marks changes according to the degree of overlap. As a specific example, the recognition marks 373 are displayed so that they become gradually brighter each time they overlap. In this way, a user who sees the notification display of this example can grasp the recognition frequency of the specific scene recognition unit 222 by the display of the recognition marks 373 that are displayed on the model image 371 and change color each time they overlap.

[0110] <Modifications of monitor display> Next, a modified example of monitor display will be described. In this example, display is performed on two monitors, a first monitor and a second monitor.

[0111] FIG. 18 is a diagram showing a case where display is performed on two monitors (a first monitor and a second monitor).

[0112] The first monitor 400A displays a medical image 301 acquired in real time. The second monitor 400B displays the notification display 303 described in Fig. 8. By using two monitors in this way to display the notification display 303, it is possible to perform a display with improved visibility.

[0113] Although the present invention has been described above with reference to examples, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0114] 10: Endoscope system 100: Endoscope 102: Handheld operation unit 104: Insertion section 106: Universal cable 108: Light guide connector 112: Soft part 114: Curved section 116: Hard tip 116A:Tip side end surface 123: Lighting Department 123A: Lighting lens 123B: Lighting lenses 126: Forceps mouth 130: Photographing optical system 132: Shooting lens 134: Image sensor 136: Drive circuit 141: Air / water supply button 142: Suction button 143: Function button 144: Shooting button 170: Light guide 200: Endoscope processor device 202: Image input controller 204: Image processing unit 205: Communication control section 206: Video output section 207: Recording Department 208 :Operation unit 209: Audio processing unit 209A: Speaker 210: CPU 211 :ROM 212:RAM 220: Medical image acquisition unit 222: Specific scene recognition unit 224: Frequency acquisition unit 226: Display control unit

Claims

1. A medical image processing apparatus including a processor, The processor, Multiple medical images taken during endoscopic examinations are acquired in sequence in a time series. Based on the acquired medical images, a specific scene in which a region to be examined during the endoscopic examination is shown in the medical images is recognized; obtaining a recognition frequency of the recognized specific scene; displaying a notification display indicating the degree of recognition of the examination region on a monitor by changing the display mode in two or more stages based on the recognition frequency and a threshold value related to the recognition frequency; The processor obtains the recognition frequency based on the number of consecutive medical images of the recognized specific scene.

2. The medical image processing apparatus according to claim 1 , wherein the processor causes the monitor to display a different color as the notification display depending on the recognition frequency.

3. The medical image processing apparatus according to claim 1 , wherein the processor causes the monitor to display different types of lines as the notification display depending on the recognition frequency.

4. The medical image processing apparatus according to claim 1 , wherein the processor causes the monitor to display a different degree of blur as the notification display in accordance with the recognition frequency.

5. The medical image processing apparatus according to claim 1 , wherein the processor acquires the recognition frequency based on the number of chronologically consecutive medical images of the recognized specific scene.

6. The medical image processing device according to claim 1 , wherein the processor recognizes the specific scene by calculating features of the medical image of the specific scene and classifying the medical image based on the features of the specific scene.

7. The medical image processing apparatus according to claim 1 , wherein the processor calculates a feature amount of the medical image of the specific scene, and recognizes the specific scene based on a similarity between the feature amount and the medical image.

8. 6. The medical image processing device according to claim 1, wherein the processor switches the notification display from an initial display to a first display and displays it on the monitor when the recognition frequency is equal to or greater than a first threshold value which is a threshold value related to the recognition frequency.

9. The medical image processing device according to claim 8 , wherein the processor switches the notification display from the first display to a second display and displays it on the monitor when the recognition frequency is equal to or greater than a second threshold value that is greater than the first threshold value.

10. The medical image processing apparatus according to claim 9 , wherein the processor performs the first display and the second display by displaying a plurality of reference images having the respective specific scenes on the monitor.

11. The medical image processing device according to claim 9, wherein the processor displays a model image including parts corresponding to a plurality of the specific scenes on the monitor, and performs the first display and the second display by matching the specific scenes recognized to the model image.

12. The medical image processing apparatus according to claim 11 , wherein the processor causes the notification display to be superimposed on the model image and displayed on the monitor, by making the position at which the specific scene is recognized correspond to a position on the model image.

13. The medical image processing apparatus according to claim 1 , wherein the processor causes the monitor to display an initial display different from the notification display when the specific scene has not yet been recognized.

14. The medical image processing apparatus according to claim 1 , wherein the processor changes an overall display mode of the notification display to indicate a degree of recognition of the examination site.

15. The medical image processing apparatus according to claim 1 , wherein the specific scene is an esophagus, a duodenum, a cardia, a pylorus, a gastric angle, a gastric fundus, a gastric body, an antrum, a lesser curvature, or a greater curvature.

16. 1. A method of operating a medical imaging device having a processor, comprising: The processor performs: A medical image acquisition step of sequentially acquiring a plurality of medical images in time series taken during an endoscopic examination; a specific scene recognition step of recognizing a specific scene in which a region to be examined during the endoscopic examination is shown in the medical image based on the acquired medical image; a frequency acquisition step of acquiring a recognition frequency of the specific scene recognized by the specific scene recognition step; a display control step of displaying, on a monitor, a notification display indicating a degree of recognition of the examination region by changing a display mode in two or more stages based on the recognition frequency and a threshold value related to the recognition frequency; A method of operation comprising: An operational method in which, in the frequency acquisition step, the processor acquires the recognition frequency based on the number of consecutive medical images of the recognized specific scene.

17. A medical image acquisition step of sequentially acquiring a plurality of medical images in time series taken during an endoscopic examination; a specific scene recognition step of recognizing a specific scene in which a region to be examined during the endoscopic examination is shown in the medical image based on the acquired medical image; a frequency acquisition step of acquiring a recognition frequency of the specific scene recognized by the specific scene recognition step; a display control step of displaying, on a monitor, a notification display indicating a degree of recognition of the examination region by changing a display mode in two or more stages based on the recognition frequency and a threshold value related to the recognition frequency; A program for causing a computer to execute a medical image processing method including: The frequency acquisition step acquires the recognition frequency based on the number of consecutive medical images of the recognized specific scene.

18. A medical image processing apparatus including a processor, The processor, Multiple medical images are acquired in sequence over time, Recognizing a specific scene in the medical image based on the acquired medical image; obtaining a recognition frequency of the recognized specific scene; displaying on a monitor a notification display indicating a degree of recognition which changes in two or more stages according to the recognition frequency; A medical image processing apparatus, wherein the processor acquires the recognition frequency based on a number of chronologically consecutive medical images of the recognized specific scene.

19. 1. A method of operating a medical imaging device having a processor, comprising: The processor performs: A medical image acquisition step of sequentially acquiring a plurality of medical images in time series taken during an endoscopic examination; a specific scene recognition step of recognizing a specific scene in which a region to be examined during the endoscopic examination is shown in the medical image based on the acquired medical image; a frequency acquisition step of acquiring a recognition frequency of the specific scene recognized by the specific scene recognition step; a display control step of displaying, on a monitor, a notification display indicating a degree of recognition of the examination site by changing a display mode in two or more stages according to the recognition frequency; A method of operation comprising: In the frequency acquisition step, the recognition frequency is acquired based on the number of chronologically consecutive medical images of the recognized specific scene.

20. A medical image acquisition step of sequentially acquiring a plurality of medical images in time series taken during an endoscopic examination; a specific scene recognition step of recognizing a specific scene in which a region to be examined during the endoscopic examination is shown in the medical image based on the acquired medical image; a frequency acquisition step of acquiring a recognition frequency of the specific scene recognized by the specific scene recognition step; a display control step of displaying, on a monitor, a notification display indicating a degree of recognition of the examination site by changing a display mode in two or more stages according to the recognition frequency; A program for causing a computer to execute a medical image processing method including: The frequency acquisition step acquires the recognition frequency based on the number of chronologically consecutive medical images of the recognized specific scene.

21. A medical image processing apparatus including a processor, The processor, Multiple medical images taken during endoscopic examinations are acquired in sequence in a time series. Based on the acquired medical images, a specific scene in which a region to be examined during the endoscopic examination is shown in the medical images is recognized; obtaining a recognition frequency of the recognized specific scene; displaying a notification display indicating the degree of recognition of the examination region on a monitor by changing the display mode in two or more stages according to the recognition frequency; A medical image processing device, The processor, When the recognition frequency is equal to or greater than a first threshold, the notification display is switched from an initial display to a first display, and is displayed on the monitor; when the recognition frequency is equal to or greater than a second threshold value that is greater than the first threshold value, the notification display is switched from the first display to a second display and displayed on the monitor. Medical imaging equipment.

22. 1. A method of operating a medical imaging device having a processor, comprising: The processor performs: A medical image acquisition step of sequentially acquiring a plurality of medical images in time series taken during an endoscopic examination; a specific scene recognition step of recognizing a specific scene in which a region to be examined during the endoscopic examination is shown in the medical image based on the acquired medical image; a frequency acquisition step of acquiring a recognition frequency of the specific scene recognized by the specific scene recognition step; a display control step of displaying, on a monitor, a notification display indicating a degree of recognition of the examination site by changing a display mode in two or more stages according to the recognition frequency; A method of operation comprising: In the display control step, When the recognition frequency is equal to or greater than a first threshold, the notification display is switched from an initial display to a first display, and is displayed on the monitor; when the recognition frequency is equal to or greater than a second threshold value that is greater than the first threshold value, the notification display is switched from the first display to a second display and displayed on the monitor. How it works.

23. A medical image acquisition step of sequentially acquiring a plurality of medical images in time series taken during an endoscopic examination; a specific scene recognition step of recognizing a specific scene in which a region to be examined during the endoscopic examination is shown in the medical image based on the acquired medical image; a frequency acquisition step of acquiring a recognition frequency of the specific scene recognized by the specific scene recognition step; a display control step of displaying, on a monitor, a notification display indicating a degree of recognition of the examination site by changing a display mode in two or more stages according to the recognition frequency; A program for causing a computer to execute a medical image processing method including: In the display control step, When the recognition frequency is equal to or greater than a first threshold, the notification display is switched from an initial display to a first display, and is displayed on the monitor; when the recognition frequency is equal to or greater than a second threshold value that is greater than the first threshold value, the notification display is switched from the first display to a second display and displayed on the monitor. program.

24. A medical image processing apparatus including a processor, The processor, Multiple medical images taken during endoscopic examinations are acquired in sequence in a time series. Based on the acquired medical images, a specific scene in which a region to be examined during the endoscopic examination is shown in the medical images is recognized; obtaining a recognition frequency of the recognized specific scene; displaying a notification display indicating the degree of recognition of the examination region on a monitor by changing the display mode in two or more stages based on the recognition frequency and a threshold value related to the recognition frequency; The processor, when the recognition frequency is equal to or greater than a first threshold which is a threshold related to the recognition frequency, switches the notification display from an initial display to a first display and displays it on the monitor, and when the recognition frequency is equal to or greater than a second threshold which is greater than the first threshold, switches the notification display from the first display to a second display and displays it on the monitor.

25. The medical image processing apparatus according to claim 24 , wherein the processor performs the first display and the second display by displaying a plurality of reference images having a plurality of the specific scenes on the monitor.

26. The medical image processing device according to claim 24, wherein the processor displays a model image including parts corresponding to a plurality of the specific scenes on the monitor, and performs the first display and the second display by corresponding the specific scene recognized to the model image.

27. The medical image processing apparatus according to claim 26 , wherein the processor causes the notification display to be superimposed on the model image and displayed on the monitor, by making the position at which the specific scene is recognized correspond to a position on the model image.

28. 1. A method of operating a medical imaging device having a processor, comprising: The processor performs: A medical image acquisition step of sequentially acquiring a plurality of medical images in time series taken during an endoscopic examination; a specific scene recognition step of recognizing a specific scene in which a region to be examined during the endoscopic examination is shown in the medical image based on the acquired medical image; a frequency acquisition step of acquiring a recognition frequency of the specific scene recognized by the specific scene recognition step; a display control step of displaying, on a monitor, a notification display indicating a degree of recognition of the examination region by changing a display mode in two or more stages based on the recognition frequency and a threshold value related to the recognition frequency; A method of operation comprising: In the display control step, when the recognition frequency is equal to or greater than a first threshold which is a threshold related to the recognition frequency, the processor switches the notification display from an initial display to a first display and displays it on the monitor, and when the recognition frequency is equal to or greater than a second threshold which is greater than the first threshold, the processor switches the notification display from the first display to a second display and displays it on the monitor.

29. A medical image acquisition step of sequentially acquiring a plurality of medical images in time series taken during an endoscopic examination; a specific scene recognition step of recognizing a specific scene in which a region to be examined during the endoscopic examination is shown in the medical image based on the acquired medical image; a frequency acquisition step of acquiring a recognition frequency of the specific scene recognized by the specific scene recognition step; a display control step of displaying, on a monitor, a notification display indicating a degree of recognition of the examination region by changing a display mode in two or more stages based on the recognition frequency and a threshold value related to the recognition frequency; A program for causing a computer to execute a medical image processing method including: The display control step is a step of switching the notification display from an initial display to a first display and displaying it on the monitor when the recognition frequency is equal to or greater than a first threshold which is a threshold related to the recognition frequency, and switching the notification display from the first display to a second display and displaying it on the monitor when the recognition frequency is equal to or greater than a second threshold which is greater than the first threshold.

30. A non-transitory computer-readable recording medium having the program according to any one of claims 17, 20, 23, and 29 recorded thereon.

Images