Medical image processing device, medical image processing method, and program

Abstract

The present invention provides a medical image processing device, a medical image processing method, and a program capable of reducing the discrepancy between recognition by a recognizer and recognition by a user. The medical image processing device comprises a processor (210), a memory (207), and a monitor (400), wherein the processor (204) sequentially acquires a plurality of medical images in a time series, recognizes a specific scene in the medical image based on the acquired medical images, acquires a recognition frequency of the recognized specific scene, and displays a notification display on the monitor indicating a recognition degree that changes in two or more stages based on the recognition frequency.

Description

Technical Field

[0001] The present invention relates to a medical image processing device, a medical image processing method, and a program. Background Art

[0002] Diagnosis and examination using an endoscopic apparatus are performed by displaying an endoscopic image acquired by the endoscopic apparatus on a monitor and allowing a doctor to observe the endoscopic image displayed on the monitor.

[0003] As described in Patent Document 1, there is known a technique in which a lesion is estimated by performing image processing on an endoscopic image, and a mark is superimposed on the lesion to be displayed on a monitor to assist a doctor's observation.

[0004] Previous technical literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2011-87793 Summary of the Invention

[0007] Technical issues to be solved by the invention

[0008] Furthermore, in recent years, deep learning has enabled highly accurate automatic recognition of objects within images. One example of deep learning applications in endoscopic devices involves the use of a recognizer developed using deep learning. Specifically, a recognizer developed using deep learning automatically identifies specific scenes within endoscopic images, and then notifies the user (physician, etc.) that the scene has been observed by displaying the identified scene.

[0009] However, there are sometimes discrepancies between what the recognizer recognizes as a specific scene and what the user actually observes. For example, even when the endoscope is rapidly moved and a specific scene is captured momentarily, the recognizer may still recognize the scene, in which case the user is notified of the recognition. On the other hand, when the specific scene is captured momentarily, the user does not observe the scene. Thus, there are sometimes discrepancies between what the recognizer recognizes as a specific scene and what the user actually observes.

[0010] Furthermore, since the recognizer notifies the user that the specific scene has been observed, the user may mistakenly recognize that the specific scene has been observed by seeing the notification display even though the user has not actually observed the specific scene.

[0011] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a medical image processing apparatus, a medical image processing method, and a program capable of reducing the discrepancy between recognition by a recognizer and recognition by a user.

[0012] Means for solving technical problems

[0013] A medical image processing device as one embodiment of the present invention for achieving the above-mentioned purpose comprises a processor, wherein the processor sequentially acquires a plurality of medical images in a time series, identifies specific scenes of the medical images based on the acquired medical images, acquires a recognition frequency of the identified specific scenes, and displays a notification display on a monitor indicating a recognition degree that changes in two or more stages based on the recognition frequency.

[0014] According to this method, since a notification display indicating the degree of recognition that changes in two or more stages is displayed on the monitor based on the recognition frequency of a specific scene in a medical image, the recognition frequency of the specific scene in the processor can be notified to the user, and the discrepancy between the user's observation of the specific scene and the processor's recognition of the specific scene can be reduced.

[0015] Preferably, the processor displays different colors on the monitor as a notification display according to the recognition frequency.

[0016] Preferably, the processor displays different line types on the monitor as a notification display according to the recognition frequency.

[0017] Preferably, the processor displays different blur states on the monitor as a notification display according to the recognition frequency.

[0018] Preferably, the processor obtains the recognition frequency based on the number of medical images of the recognized specific scene.

[0019] Preferably, the processor acquires the recognition frequency based on the number of medical images consecutive in time series of the recognized specific scene.

[0020] Preferably, the processor calculates a feature value of the medical image of the specific scene, and identifies the specific scene by classifying the medical image based on the feature value of the specific scene.

[0021] Preferably, the processor calculates a feature value of a medical image of a specific scene, and identifies the specific scene based on a similarity between the feature value and the medical image.

[0022] Preferably, the processor switches the notification display from the initial display to the first display and displays the notification on the monitor when the recognition frequency is equal to or greater than a first threshold value.

[0023] Preferably, the processor switches the notification display from the first display to the second display and displays the notification on the monitor when the recognition frequency is equal to or higher than a second threshold value that is greater than the first threshold value.

[0024] Preferably, the processor displays a plurality of reference images corresponding to a plurality of specific scenes on the monitor to perform the first display and the second display.

[0025] Preferably, the processor displays a model image including parts corresponding to a plurality of specific scenes on the monitor, and performs the first display and the second display by making the recognized specific scene correspond to the model image.

[0026] Preferably, the processor makes the position where the specific scene is recognized correspond to the position on the model image, and causes the notification display to be superimposed on the model image and displayed on the monitor.

[0027] Preferably, the processor displays an initial display different from the notification display on the monitor when the specific scene has not yet been recognized.

[0028] As another embodiment of the present invention, a medical image processing method includes: a medical image acquisition step of acquiring a plurality of medical images in a time series; a specific scene recognition step of recognizing a specific scene of the medical image based on the acquired medical image; a frequency acquisition step of acquiring a recognition frequency of the specific scene recognized by the specific scene recognition step; and a display control step of displaying a notification display on a monitor indicating a recognition degree that changes in two or more stages according to the recognition frequency.

[0029] As another embodiment of the present invention, a program enables a computer to execute a medical image processing method, which includes: a medical image acquisition step of acquiring multiple medical images in a time series; a specific scene recognition step of identifying a specific scene of the medical image based on the acquired medical image; a frequency acquisition step of acquiring the recognition frequency of the specific scene recognized by the specific scene recognition step; and a display control step of displaying a notification display on a monitor indicating the recognition degree that changes in two or more stages according to the recognition frequency.

[0030] Effects of the Invention

[0031] According to the present invention, it is possible to reduce the discrepancy between recognition by the identifier and recognition by the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is an external view of the endoscope system.

[0033] Figure 2 This is a block diagram showing the main structure of an endoscope system.

[0034] Figure 3 This is a functional block diagram of a medical image processing device in the image processing unit.

[0035] Figure 4 It is a diagram showing the structure of CNN.

[0036] Figure 5 Schematic diagram showing a structural example of an intermediate layer.

[0037] Figure 6 This is a flowchart showing a medical image processing method.

[0038] Figure 7 This is a flow chart of how to obtain the identification frequency.

[0039] Figure 8 It is a diagram showing an example of notification display.

[0040] Figure 9 It is a flow chart of a medical image processing method.

[0041] Figure 10 This is a flow chart of how to obtain the identification frequency.

[0042] Figure 11 This is a diagram showing a specific example of the first modified example 1.

[0043] Figure 12 This is a diagram showing a specific example of the second modification example 1.

[0044] Figure 13 This is a diagram showing a specific example of the third modification example 1.

[0045] Figure 14 This is a diagram showing a specific example of the fourth modification example 1.

[0046] Figure 15 This is a diagram showing a specific example of the fifth modification example 1.

[0047] Figure 16 It is a diagram showing the first modified example 2.

[0048] Figure 17 It is a diagram showing the second modified example 2.

[0049] Figure 18 This figure shows a situation where display is performed on two monitors. DETAILED DESCRIPTION

[0050] Hereinafter, preferred embodiments of the medical image processing apparatus, medical image processing method, and program according to the present invention will be described with reference to the accompanying drawings.

[0051] <Structure of the Endoscope System>

[0052] Figure 1 is an external view of the endoscope system 10, Figure 2 FIG is a block diagram showing the main structure of the endoscope system 10. Figure 1 、 2As shown, the endoscope system 10 is composed of an endoscope 100, an endoscope processor 200, a light source device 300, and a monitor 400. The endoscope processor 200 is equipped with the medical image processing apparatus of the present invention.

[0053] <Structure of Endoscope>

[0054] The endoscope 100 includes a handheld operation unit 102 and an insertion unit 104 connected to the handheld operation unit 102. The operator (user) grasps the handheld operation unit 102 and operates it, inserting the insertion unit 104 into the body of a subject (living organism) for observation. The handheld operation unit 102 also includes air and water supply buttons 141, a suction button 142, function buttons 143 assigned various functions, and a capture button 144 for receiving capture instructions (still image, moving image). The insertion unit 104 consists of, in order from the handheld operation unit 102 side, a flexible portion 112, a curved portion 114, and a distal rigid portion 116. Specifically, the curved portion 114 is connected to the proximal end of the distal rigid portion 116, and the flexible portion 112 is connected to the proximal end of the curved portion 114. The handheld operation unit 102 is connected to the proximal end of the insertion unit 104. The user can bend the bending portion 114 by operating the hand operation part 102, and change the direction of the top hard part 116 upward, downward, left and right. The top hard part 116 is provided with a photographic optical system 130, an illumination part 123, a forceps opening 126, etc. (see Figure 1 、 2 ).

[0055] During observation and treatment, the operation unit 208 (see Figure 2 ) operation, white light and / or narrowband light (one or more of red narrowband light, green narrowband light, blue narrowband light, and violet narrowband light) can be emitted from the illumination lenses 123A and 123B of the illumination unit 123. Furthermore, by operating the air and water supply button 141, cleansing water can be discharged from a water supply nozzle (not shown) to clean the imaging lens 132 and illumination lenses 123A and 123B of the imaging optical system 130. A conduit (not shown) is connected to the forceps opening 126 opened in the distal rigid portion 116. A treatment instrument (not shown) used for tumor removal, for example, is inserted into this conduit, allowing it to be inserted and removed as appropriate to perform necessary treatment on the subject.

[0056] like Figure 1 and Figure 2As shown, a photographic lens 132 is provided on the distal end surface 116A of the distal rigid portion 116. Behind the photographic lens 132, a CMOS (Complementary Metal-Oxide Semiconductor) type imaging element 134, a driving circuit 136, and an AFE 138 (AFE: Analog Front End) are provided, and these elements output image signals. The imaging element 134 is a color imaging element having a plurality of pixels composed of a plurality of light-receiving elements arranged in a matrix (two-dimensional arrangement) using a specific pattern arrangement (Bayer arrangement, X-Trans (registered trademark) arrangement, honeycomb arrangement, etc.). Each pixel of the imaging element 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (photodiode, etc.). The photographic optical system 130 can generate a color image from pixel signals of three colors: red, green, and blue, or it can generate an image from pixel signals of one or two colors: red, green, and blue. Furthermore, the imaging element 134 may be a CCD (Charge Coupled Device) type. In addition, each pixel of the imaging element 134 may include a purple color filter corresponding to the purple light source 310V and / or an infrared filter corresponding to the infrared light source.

[0057] The optical image of the subject is formed by the imaging lens 132 onto the light-receiving surface (imaging surface) of the imaging element 134 and converted into an electrical signal. The electrical signal is then output to the endoscope processor 200 via a signal cable (not shown) and converted into a video signal. As a result, an endoscopic image (medical image) of the subject is displayed on the monitor 400 connected to the endoscope processor 200.

[0058] Furthermore, illumination lenses 123A and 123B of the illumination unit 123 are provided on the distal end surface 116A of the distal rigid portion 116, adjacent to the imaging lens 132. The emission end of a light guide 170, described later, is disposed behind the illumination lenses 123A and 123B. This light guide 170 is inserted through the insertion portion 104, the hand-side operation portion 102, and the universal cable 106, and the incident end of the light guide 170 is disposed within the light guide connector 108.

[0059] The user can sequentially capture time-series endoscopic images of the body by inserting or removing the endoscope 100 having the above configuration into or from the body of a subject while performing imaging at a predetermined frame rate.

[0060] <Structure of Light Source Device>

[0061] like Figure 2As shown, the light source device 300 comprises an illumination light source 310, an aperture 330, a condenser lens 340, and a light source control unit 350, and directs observation light into the light guide 170. The light source 310 includes a red light source 310R, a green light source 310G, a blue light source 310B, and a violet light source 310V, which respectively emit narrowband light of red, green, blue, and violet. The illumination intensity of the observation light from the light source 310 is controlled by the light source control unit 350, which can change (increase or decrease) the illumination intensity and stop the illumination as needed.

[0062] Light source 310 can emit any combination of red, green, blue, and violet narrowband light. For example, it can emit red, green, blue, and violet narrowband light simultaneously to illuminate white light (ordinary light) as observation light, or it can emit any one or two of these narrowband lights to illuminate narrowband light (special light). Light source 310 can also include an infrared light source that illuminates infrared light (an example of narrowband light). Alternatively, a light source that illuminates white light and filters that transmit white light and each narrowband light can be used to illuminate white light or narrowband light as observation light.

[0063] <Wavelength Band of Light Source>

[0064] Light source 310 can generate light in a white band, or generate light in multiple wavelength bands as white band light, or it can generate light in a specific wavelength band narrower than the white wavelength band. The specific wavelength band can be a blue band or a green band in the visible range, or a red band in the visible range. When the specific wavelength band is the blue band or the green band in the visible range, it can include a wavelength band between 390 nm and 450 nm, or between 530 nm and 550 nm, and have a peak wavelength within the wavelength band between 390 nm and 450 nm, or between 530 nm and 550 nm. Furthermore, when the specific wavelength band is the red band in the visible range, it can include a wavelength band between 585 nm and 615 nm, or between 610 nm and 730 nm, and have a peak wavelength within the wavelength band between 585 nm and 615 nm, or between 610 nm and 730 nm.

[0065] The specific wavelength band may include a wavelength band where the absorption coefficient differs between oxyhemoglobin and deoxyhemoglobin, and the light in the specific wavelength band may have a peak wavelength in the wavelength band where the absorption coefficient differs between oxyhemoglobin and deoxyhemoglobin. In this case, the specific wavelength band may include 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band between 600 nm and 750 nm, and the light in the specific wavelength band may have a peak wavelength in 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band between 600 nm and 750 nm.

[0066] In addition, the wavelength band of the light generated by the light source 310 may also include a wavelength band above 790nm and below 820nm, or above 905nm and below 970nm, and the light generated by the light source 310 has a peak wavelength in the wavelength band above 790nm and below 820nm, or above 905nm and below 970nm.

[0067] Alternatively, light source 310 may be configured to emit excitation light with a peak wavelength of 390 nm to 470 nm. In this case, an endoscopic image containing information about the fluorescence emitted by fluorescent substances within the subject (organism) can be obtained. Fluorescence pigments (such as fluorescein and acridine orange) may also be used to obtain fluorescence images.

[0068] The light source type (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode) etc.), wavelength, presence or absence of a filter, etc. of the light source 310 are preferably configured according to the type and location of the subject, the purpose of observation, etc. Furthermore, during observation, it is preferable to combine and / or switch the wavelength of the observation light according to the type and location of the subject, the purpose of observation, etc. When switching the wavelength, for example, a circular filter (rotating color filter) disposed in front of the light source and provided with a filter that transmits or blocks light of a specific wavelength can be rotated to switch the wavelength of the irradiated light.

[0069] In addition, the imaging element used when implementing the present invention is not limited to a color imaging element in which a color filter is provided for each pixel like the imaging element 134, but may also be a monochrome imaging element. When using a monochrome imaging element, the wavelength of the observation light can be switched in sequence to perform imaging in a plane order (color order). For example, the wavelength of the emitted observation light can be switched in sequence between narrow-band light (purple, blue, green, red), or broadband light (white light) can be irradiated and the wavelength of the emitted observation light can be switched by rotating color filters (red, green, blue, purple, etc.). In addition, one or more narrow-band lights (green, blue, purple, etc.) can be irradiated and the wavelength of the emitted observation light can be switched by rotating color filters (green, blue, purple, etc.). The narrow-band light can be infrared light of two or more wavelengths with different wavelengths (first narrow-band light, second narrow-band light).

[0070] By connecting the optical connector 108 (refer to Figure 1 、 2 ) is connected to the light source device 300, and the observation light irradiated from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and is irradiated toward the observation range from the illumination lenses 123A and 123B.

[0071] <Structure of Endoscope Processor Device>

[0072] based on Figure 2 The structure of the endoscope processor device 200 will be described. The endoscope processor device 200 receives the image signal output from the endoscope 100 via the image input controller 202, performs necessary image processing in the image processing unit 204, and outputs the signal via the video output unit 206. This displays the endoscopic image on the monitor 400. These processes are performed under the control of the CPU 210 (CPU: Central Processing Unit). Furthermore, the CPU 210 functions as a processor for the medical image processing device. The communication control unit 205 controls communications related to the acquisition of medical images, etc., with an in-hospital system (HIS: Hospital Information System) or a hospital LAN (Local Area Network) (not shown), and / or external systems or networks.

[0073] <Functions of the Image Processing Unit>

[0074] The image processing unit 204 can calculate characteristic quantities of endoscopic images, perform processing to emphasize or reduce components of specific wavelength bands, and perform processing to emphasize or de-emphasize specific objects (such as regions of interest or blood vessels at a desired depth). The image processing unit 204 may also include a special light image acquisition unit (not shown) that acquires a special light image containing information about a specific wavelength band based on a normal light image obtained by irradiating the image with light in a white wavelength band or by irradiating a plurality of wavelength bands as the white wavelength band. In this case, the signal of the specific wavelength band can be obtained through calculations based on the RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) color information contained in the normal light image. The image processing unit 204 may also include a feature image generation unit (not shown) that acquires and displays a feature image as an endoscopic image. The feature image generation unit generates the feature image by performing calculations based on at least one of a normal light image obtained by irradiating a white band of light, or a normal light image obtained by irradiating a plurality of wavelength bands as the white band of light, and a special light image obtained by irradiating a specific wavelength band of light. The above-described processing is performed under the control of the CPU 210.

[0075] Furthermore, as described below, the image processing unit 204 has various functions in the medical image processing apparatus.

[0076] Figure 3 2 is a functional block diagram of a medical image processing device in the image processing unit 204. The image processing unit 204 includes a medical image acquisition unit 220, a specific scene recognition unit 222, a frequency acquisition unit 224, and a display control unit 226.

[0077] <Implementation of various processor functions>

[0078] The functions of each part of the above-mentioned image processing unit 204 can be implemented using various processors and recording media. Among the various processors, there are general-purpose processors such as CPUs (Central Processing Units) that implement various functions by executing software (programs). In addition, among the various processors mentioned above, there are also processors specifically used for image processing, such as GPUs (Graphics Processing Units), FPGAs (Field Programmable Gate Arrays), and processors whose circuit structures can be changed after manufacturing, such as programmable logic devices (PLDs). In the case of learning or recognizing images as in the present invention, a structure using a GPU is effective. Moreover, processors having a circuit structure specially designed for executing specific processing such as ASICs (Application Specific Integrated Circuits), such as dedicated circuits, are also included in the various processors mentioned above.

[0079] The functions of each part can be implemented by one processor, or by multiple processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). In addition, multiple functions can also be implemented by one processor. As an example of implementing multiple functions with one processor, first, there is a form such as a computer, which uses a combination of one or more CPUs and software to form a processor, and implements the processor as multiple functions. Secondly, there is a form such as a system on chip (SoC), which uses a processor that implements the functions of the entire system with an IC (Integrated Circuit) chip. In this way, various functions are implemented using one or more of the above-mentioned various processors as a hardware structure. Moreover, more specifically, the hardware structure of these various processors is a circuit (circuitry) that combines circuit elements such as semiconductor elements. These circuits can also be circuits that use logical sum, logical product, logical negation, exclusive or, and logical operations combining them to implement the above-mentioned functions.

[0080] When the processors or circuits described above execute software (programs), computer-readable code for the executed software (e.g., the various processors or circuits, and / or combinations thereof, that comprise the image processing unit 204) is first stored in a non-transitory recording medium such as ROM 211 (ROM: Read Only Memory), and the computer references the software. The software pre-stored in the non-transitory recording medium includes a program for executing the operating method of the medical image processing apparatus according to the present invention and data used during execution (data related to medical image acquisition, specific data for notification conditions and notification methods, parameters used in the recognition unit, etc.). Instead of storing the code in ROM 211, the code may be stored in a non-transitory recording medium such as various magneto-optical recording devices or semiconductor memories. When processing using software, for example, RAM 212 (RAM: Random Access Memory) is used as a temporary storage area, and data stored in, for example, EEPROM (Electrically Erasable and Programmable Read Only Memory) (not shown) may also be referenced. The recording unit 207 may also serve as a "non-transitory recording medium."

[0081] ROM 211 (Read Only Memory) is a nonvolatile storage element (non-transitory recording medium) that stores computer-readable code for programs that cause CPU 210 and / or image processing unit 204 to execute various image processing methods. RAM 212 (Random Access Memory) is a storage element used for temporary storage during various processing operations and can also be used as a buffer for image acquisition. The audio processing unit 209 outputs sound and voice from speaker 209A under the control of CPU 210.

[0082] The operation unit 208 may be composed of a keyboard, a mouse, or other devices (not shown), and the user can issue an instruction to execute a process or designate conditions required for the execution via the operation unit 208 .

[0083] The recording unit 207 records the acquired endoscopic images. In addition, the recording unit 207 records programs and information for controlling the medical image processing apparatus. The recording unit 207 functions as a memory of the medical image processing apparatus.

[0084] <Neural network-based specific scene recognition unit>

[0085] The specific scene recognition unit 222 in the image processing unit 204 is comprised of a recognizer for recognizing specific scenes. This recognizer utilizes a learned model, such as a neural network (a model learned using a set of images obtained by capturing a living organism). Here, specific scenes include, for example, the esophagus, duodenum, cardia, pylorus, gastric angle, gastric fundus, gastric body, vestibule, lesser curvature, and greater curvature. The following describes the structure of a neural network using a CNN (Convolutional Neural Network).

[0086] <Configuration Example of Identification Unit>

[0087] Figure 4 This is a diagram showing the structure of CNN232 (neural network). Figure 4 In the example shown in part (a), CNN232 has an input layer 232A, an intermediate layer 232B, and an output layer 232C. The input layer 232A inputs the endoscopic image acquired by the medical image acquisition unit 220 and outputs the feature quantity. The intermediate layer 232B includes a convolution layer 234 and a pooling layer 235, which calculates other feature quantities by inputting the feature quantity output from the input layer 232A. These layers are formed into a structure in which multiple "nodes" are connected by "edges", and the weight coefficients applied to the input image are associated with the nodes and edges and stored in a weight coefficient storage unit not shown. The value of the weight coefficient changes as learning progresses.

[0088] <Processing of the middle layer>

[0089] The middle layer 232B calculates the feature quantity through convolution operation and pooling processing. The convolution operation performed in the convolution layer 234 is a process of obtaining a feature map by using a convolution operation of a filter, and plays the role of extracting features such as edges from the image. By using the convolution operation of the filter, a "feature map" of one channel (one sheet) is generated for one filter. When the size of the "feature map" is reduced (downscaling) by convolution, it becomes smaller as convolution is performed in each layer. The pooling processing performed in the pooling layer 235 is a process of reducing (or enlarging) the feature map output by the convolution operation to obtain a new feature map, and plays the role of providing robustness to prevent the extracted features from being affected by parallel movement, etc. The middle layer 232B can be composed of one or more layers that perform these processes. In addition, CNN232 can also be constructed without a pooling layer 235.

[0090] CNN232 can also be Figure 4 As shown in the example of part (b) of FIG, the fully connected layer 236 is included. The layer structure of CNN 232 is not limited to the case where the convolution layer 234 and the pooling layer 235 are repeated one by one, and a plurality of arbitrary layers (for example, the convolution layer 234) may be included consecutively.

[0091] Figure 5 Yes Figure 4 The diagram shows a schematic diagram of an example structure of the intermediate layer 232B of the CNN 232. In the initial (first) convolutional layer of the intermediate layer 232B, a convolution operation is performed on an image group consisting of multiple endoscopic images and a filter F1. The image group consists of N images (N channels) with an image size of height H and width W. When a normal light image is input, the images that constitute the image group are images of three channels: R (red), G (green), and B (blue). Since the image group consists of N channels (N images), the filter F1 that performs the convolution operation with the image group has a filter size of 5 (5×5), for example, 5×5×N. By performing the convolution operation using the filter F1, a "feature map" of one channel (one image) is generated for one filter F1. The filter F2 used in the second convolutional layer has a filter size of 3 (3×3), for example, 3×3×M.

[0092] Similar to the first convolution layer, filters F2 to F n The size of the feature map in the nth convolutional layer is smaller than that in the second convolutional layer because it is reduced by the previous convolutional layer or pooling layer.

[0093] In the layer of the intermediate layer 232B, low-order feature extraction (edge ​​extraction, etc.) is performed in the convolution layer close to the input side, and high-order feature extraction (feature extraction related to the shape, structure, etc. of the identified object) is performed as it approaches the output side.

[0094] In addition to the convolutional layer 234 and the pooling layer 235, the intermediate layer 232B may also include a batch normalization layer. Batch normalization normalizes the distribution of data in small batches during learning, and helps to accelerate learning, reduce dependence on initial values, and prevent over-learning.

[0095] The output layer 232C outputs the feature quantities calculated by the intermediate layer 232B in a form consistent with specific scene recognition. The output layer 232C may also include a fully connected layer.

[0096] <Each Process of Medical Image Processing Method>

[0097] Next, a medical image processing method using a medical image processing apparatus will be described. Furthermore, while the following description will focus on processing endoscopic images acquired by the endoscope system 10 as medical images, the application of the present invention is not limited thereto. For example, the present invention can also process endoscopic images acquired by other endoscope systems 10 as medical images using the medical image processing apparatus.

[0098] Figure 6 is a flowchart showing a medical image processing method. Figure 6 Each step is explained.

[0099] Medical image acquisition steps

[0100] The medical image acquisition unit 220 acquires multiple medical images in time series (step S10 ). The medical image acquisition unit 220 may acquire medical images of the body cavity captured by the endoscope 100 or medical images of the body cavity recorded in the recording unit 207 .

[0101] Specific scene recognition steps

[0102] The specific scene recognition unit 222 recognizes specific scenes in the acquired medical images. The specific scene recognition unit 222 receives the medical images, inputs the medical images into the above-mentioned CNN, and extracts feature quantities from the medical images step by step. The specific scene recognition unit 222 finally uses the feature quantities to recognize specific scenes by classifying them into categories (e.g., esophagus, duodenum, cardia, pylorus, gastric angle, gastric fundus, gastric body, vestibule, lesser curvature, greater curvature, and other 11 categories). In addition, a detailed description of the recognition method is recorded in the following literature (B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In Neural Information Processing Systems (NIPS), pages 487-495, 2014.1, 4, 6, 8). Alternatively, the specific scene recognition unit 222 can also recognize specific scenes by comparing the finally extracted feature values ​​with feature values ​​previously stored in the recording unit 207 (e.g., feature values ​​of the duodenum) and calculating similarity. A detailed description of this recognition technique is provided in the following document (FaceNet: A Unified Embedding for Face Recognition and Clustering https: / / arxiv.org / abs / 1503.03832).

[0103] If the specific scene recognition unit 222 cannot recognize the specific scene in the received medical image (No in step S11), the medical image acquisition unit 220 acquires the next medical image (step S10) when the next medical image is available (Yes in step S19). On the other hand, if the specific scene recognition unit 222 can recognize the specific scene in the medical image (Yes in step S11), the process proceeds to the frequency acquisition step (step S12) for updating the record of the recognition frequency.

[0104] Frequency acquisition steps

[0105] When the specific scene recognition unit 222 recognizes a specific scene in the medical image, the frequency acquisition unit 224 updates the record of the recognition frequency (step S12). For example, if the specific scene recognition unit 222 recognizes the duodenum as a specific scene, the recognition frequency of the duodenum is updated from 0 to 1. Various methods are possible for updating the recognition frequency, such as updating to a value between 0 and 64. The updating of the recognition frequency will be described in detail later.

[0106] Display control steps

[0107] When the recognition frequency is above the first threshold (Yes in step S13), the initial display is displayed on the monitor 400, and the first display as one of the notification displays has not yet been displayed on the monitor 400 (No in step S14), the display control unit 226 changes the initial display to the first display (step S15).

[0108] On the other hand, the display control unit 226 changes the initial display to the second display when the recognition frequency is above the first threshold (if "Yes" in step S13), the first display is already displayed on the monitor 400 (if "Yes" in step S14), and the recognition frequency is above the second threshold (if "Yes" in step S16). The first and second thresholds are set appropriately by the user. Thus, by displaying the initial display, the first display, or the second display on the monitor 400 in a stepwise manner according to the recognition frequency of the specific scene by the specific scene recognition unit 222, the user can be aware of the recognition frequency.

[0109] When the recognition frequency is lower than the first threshold (No in step S13), or the recognition frequency is lower than the second threshold (No in step S16), or the second display is already displayed on the monitor 400 (Yes in step S17), when there is a next medical image, the medical image acquisition unit 220 acquires the next medical image (step S10).

[0110] <Frequency Acquisition Steps>

[0111] Next, the frequency acquisition procedure performed by the frequency acquisition unit 224 will be described.

[0112] Figure 7 This is a flow chart of how to obtain the recognition frequency. Figure 7 In the figure, n is the number of images in which a specific scene (e.g., the pylorus) is recognized in an endoscopic image, and N is the frequency update threshold value, which is set appropriately by the user.

[0113] In the initial state, n is set to 0 (step S20). The medical image acquisition unit 220 receives the medical image (step S21). When the specific scene recognition unit 222 recognizes a specific scene in the medical image (Yes in step S22), the number of images n that recognize the specific scene is updated to n+1 (step S23). When the updated number of images n is greater than the frequency threshold N (Yes in step S24), the frequency acquisition unit 224 updates the recognition frequency (step S25). For example, when the frequency threshold is 5 (N=5), when the specific scene recognition unit 222 recognizes the pylorus in five medical images, the frequency acquisition unit 224 updates the recognition frequency of the pylorus. After that, the process is switched to the use Figure 6 The display control step (steps S13 to S18: step S26) has been described.

[0114] When the specific scene recognition unit 222 does not recognize a specific scene in the acquired medical image (No in step S22), or when the updated image number n is less than the frequency threshold N (No in step S24), or when the specific scene recognition unit 222 recognizes a specific scene in the acquired medical image (No in step S22), or when the updated image number n is less than the frequency threshold N (No in step S24), Figure 6 After the display control steps (steps S13 to S18 ) described above are performed, when there is a next medical image (if Yes in step S27 ), the medical image acquisition part 220 acquires the next medical image.

[0115] Furthermore, the specific scene recognition unit 222 may recognize one or more specific scenes. If the specific scene recognition unit 222 recognizes multiple specific scenes, a frequency threshold is set for each specific scene, and the recognition frequency is updated for each specific scene.

[0116] <Notification Display>

[0117] Next, the notification display on the monitor 400 by the display control unit 226 will be described.

[0118] Figure 8 This is a diagram showing an example of a notification display performed by the display control unit 226 on the monitor 400 .

[0119] exist Figure 8In the illustrated case, an endoscopic image (medical image) 301 captured in real time is displayed on the main display area of ​​the monitor 400. In addition, a notification display 303 composed of a reference image is displayed on the sub-display area of ​​the monitor 400.

[0120] The notification display 303 is composed of reference images (303A to 303I) of specific scenes 1 to 9.

[0121] In specific scenes 4, 8, and 9 (symbols 303D, 303H, and 303I) that have not yet been recognized by the specific scene recognition unit 222, the initial display is displayed and only the frame line 305 is displayed. In this way, by setting the initial display to a display different from the notification display, the user can know that the specific scene recognition unit 222 has not yet recognized the specific scene. On the other hand, in specific scenes 1 to 3, 5, 6, and 7 (symbols 303A, 303B, 303C, 303E, 303F, and 303G), the recognition frequency is greater than the first threshold value, and the first display (reference image 307 of the specific scene) is displayed. In addition, in specific scene 2, the recognition frequency is greater than the second threshold value, and the second display is displayed (the line type of the frame line 305 is changed from a solid line to a dotted line).

[0122] As described above, in this embodiment, the notification display is switched based on the recognition frequency of specific scenes in medical images and displayed on the monitor 400. This allows the user to know the recognition frequency of specific scenes in the medical image processing apparatus (specific scene recognition unit 222). Therefore, this embodiment can reduce the discrepancy between the recognition performed by the medical image processing apparatus (specific scene recognition unit 222) and the user's recognition.

[0123] Next, modifications of the above-described embodiment will be described.

[0124] <Modifications of Each Process of the Medical Image Processing Method>

[0125] In the above embodiment, the first and second displays are displayed on the monitor 400. However, in this example, the display is changed to the third display and the notification display is performed on the monitor 400.

[0126] Figure 9 This is a flowchart of the medical image processing method in this example. Figure 9 The value of m in represents the number of the screen in the notification display. In addition, a threshold value is set corresponding to each notification display. Therefore, when the recognition frequency is greater than the mth threshold value, the mth display is displayed.

[0127] In the initial state, m=1 (step S30). The medical image acquisition unit 220 sequentially acquires a plurality of medical images in a time series (step S31). Thereafter, when the specific scene recognition unit 222 can recognize a specific scene in the acquired medical image (when the answer is Yes in step S32), the frequency acquisition unit 224 updates the record of the recognition frequency (step S33). Thereafter, when the recognition frequency is above the mth threshold (when the answer is Yes in step S34) and the mth display has not yet been displayed on the monitor 400 (when the answer is No in step S35), the display control unit 226 changes to the mth display (step S36). On the other hand, when the mth display has already been displayed on the monitor 400, m=m+1 (step S37).

[0128] When the specific scene recognition unit 222 cannot recognize the specific scene (No in step S32), or after changing to the mth display (step S36), when there is the next medical image (Yes in step S38), the medical image acquisition unit 220 acquires the medical image.

[0129] As described above, in this example, the changes to the third display and above are not limited to the first and second displays. This allows the user to more accurately grasp the recognition frequency of specific scenes in the medical image processing apparatus.

[0130] <Modification of Updating Identification Frequency>

[0131] Next, a modification of the updating of the recognition frequency will be described. Figure 7 , described an example of updating the recognition frequency based on the number of medical images that recognized a specific scene. However, in this example, the recognition frequency is updated based on the number of consecutive recognized images in a time series. In the figure, n represents the number of images that recognized a specific scene, and N represents the frequency update threshold, which can be set appropriately by the user.

[0132] Figure 10 This is a flowchart of the acquisition of the identification frequency in the frequency acquisition unit 224 of this example.

[0133] In the initial state, n is set to 0 (step S40). The medical image acquisition unit 220 continuously acquires medical images in a time series (step S41). If the specific scene recognition unit 222 recognizes a specific scene in the medical image (Yes in step S42), the number of images in which the specific scene was recognized, n, is updated to n+1 (step S44). On the other hand, if the specific scene recognition unit 222 cannot recognize a specific scene, the counter is set to n=0 (step S49), and the next medical image is acquired when it is available.

[0134] If the updated number n of images that recognize a specific scene is greater than the frequency threshold N (if Yes in step S45), the recognition frequency is updated (step S46). Figure 7 The display control steps (steps S13 to S18: step S47) described in the previous section are performed. After that, when the next medical image is available (if Yes in step S48), the process proceeds to the next endoscopic image.

[0135] As described above, in this example, the recognition frequency is updated when a specific scene can be continuously recognized in a predetermined number of medical images. Therefore, the recognition result of the specific scene recognition unit 222 can be outputted with high accuracy reflecting the observation situation of the user.

[0136] <Modification Example 1 of Notification Display>

[0137] Next, a modification example 1 of the notification display will be described. In this example, various modifications using a reference image as the notification display will be described.

[0138] Along Figures 11 to 15 , which is a diagram illustrating specific examples of the first to fifth modified examples 1 of the notification display. IMGB in the diagram is an initial display, IMG1 is a first display switched from the initial display, and IMG2 is a second display switched from the first display.

[0139] exist Figure 11 In the first modified example 1 shown, the color of the frame line changes according to the recognition frequency.

[0140] For example, in IMGB, the frame line 311 is black, in IMG1, the frame line 313 is white, and in IMG2, the frame line 315 is red. Thus, a user viewing the notification display of this example can understand the recognition frequency of the specific scene recognition unit 222 by the color of the frame line.

[0141] exist Figure 12 In the second modified example 1 shown, the type (line type) of the frame line changes according to the recognition frequency.

[0142] For example, in IMGB, frame line 321 is a thin dotted line, in IMG1, frame line 323 is a dotted line thicker than frame line 321, and in IMG2, frame line 325 is a solid line. Thus, a user viewing the notification display of this example can understand the recognition frequency of the specific scene recognition unit 222 by the type of frame line.

[0143] exist Figure 13 In the third modified example 1 shown, the background color of the reference image changes according to the recognition frequency.

[0144] For example, in IMGB, background 331 is white, in IMG1, background 333 is gray, and in IMG2, background 335 is black. Thus, a user viewing the notification display of this example can understand the recognition frequency of the specific scene recognition unit 222 by the background color.

[0145] exist Figure 14 In the fourth modification example 1 shown, the color of the periphery of the reference image changes.

[0146] For example, in IMGB, the periphery 341 of the reference image is white, in IMG1, the periphery 343 of the reference image is pink, and in IMG2, the periphery 345 of the reference image is red. Thus, a user viewing the notification display of this example can understand the recognition frequency of the specific scene recognition unit 222 by the color of the periphery of the reference image.

[0147] exist Figure 15 In the fifth modification example 1 shown, the image of the reference image gradually becomes clearer (the degree of blurring changes).

[0148] For example, IMGB displays a blank image 351 instead of a reference image, IMG1 displays an unclear (blurred) reference image 353, and IMG2 displays a clear reference image 355. Thus, a user viewing the notification display of this example can understand the recognition frequency of the specific scene recognition unit 222 by the clarity (blurriness) of the reference image.

[0149] <Modification Example 2 of Notification Display>

[0150] Next, a modification example 2 of the notification display will be described. In this example, the notification display is superimposed on a model image including parts corresponding to a plurality of specific scenes.

[0151] Figure 16 : is a diagram showing the first variant example 2. In this example, it is a diagram showing a notification display on a model image 361 simulating a stomach. The model image 361 includes parts corresponding to a plurality of specific scenes. Moreover, when each specific scene is recognized, a band display is performed as a notification display, the color of which changes according to the recognition frequency. Figure 16In the example shown, band display 363A is displayed as the first display for specific scene A, and band display 363D is displayed as the first display for specific scene D. Furthermore, band display 363B is displayed as the second display for specific scene B, band display 363C is displayed as the second display for specific scene C, and band display 363E is displayed as the second display for specific scene E. Furthermore, band displays 363A and 363D, which serve as the first display, are displayed in the same color, while band displays 363B, 363C, and 363E, which serve as the second display, are displayed in the same color. Thus, a user viewing the notification display of this example can understand the recognition frequency of the specific scene recognition unit 222 by the color of the band display superimposed on the model image.

[0152] Figure 17 : is a figure showing the second variant example 2. In this example, it is a figure showing a notification display on a model image 371 simulating a stomach. The model image 371 includes parts corresponding to a plurality of specific scenes. Moreover, when each specific scene is recognized above a predetermined threshold, an identification mark 373 is displayed as a notification display at a position corresponding to the recognized specific scene. The identification mark 373 is displayed overlappingly according to the position of the specific scene, and the color of the part where the identification mark overlaps changes according to the degree of overlap. As a specific example, the identification mark 373 is displayed in a manner that gradually becomes brighter each time it overlaps. In this way, a user who sees the notification display of this example can grasp the recognition frequency of the specific scene recognition unit 222 by displaying the identification mark 373 displayed on the model image 371 and whose color changes each time it overlaps.

[0153] <Modification Example of Monitor Display>

[0154] Next, a modification of the monitor display will be described. In this example, the display is performed on two monitors, a first monitor and a second monitor.

[0155] Figure 18 This diagram shows the case where display is performed on two monitors (a first monitor and a second monitor).

[0156] On the first monitor 400A, the medical image 301 acquired in real time is displayed. On the second monitor 400B, Figure 8 By performing the notification display 303 using two monitors in this manner, it is possible to perform a display with improved visibility.

[0157] While the examples of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0158] Explanation of symbols

[0159] 10: Endoscopic system

[0160] 100: Endoscope

[0161] 102: Hand operation department

[0162] 104: Insertion

[0163] 106: Universal Cable

[0164] 108: Optical connector

[0165] 112: Soft Department

[0166] 114: Bend

[0167] 116: Top hard part

[0168] 116A: Top end face

[0169] 123: Lighting Department

[0170] 123A: Lighting lens

[0171] 123B: Lighting lens

[0172] 126: Clamping

[0173] 130: Photographic Optical System

[0174] 132: Photographic lens

[0175] 134: Camera element

[0176] 136: Driving circuit

[0177] 141: Gas and water supply buttons

[0178] 142: Attract button

[0179] 143: Function button

[0180] 144: Shoot button

[0181] 170: Light guide

[0182] 200: Endoscope processor device

[0183] 202: Image input controller

[0184] 204: Image Processing Department

[0185] 205: Communication Control Department

[0186] 206: Video output unit

[0187] 207: Records Department

[0188] 208: Operation Department

[0189] 209: Sound Processing Department

[0190] 209A: Speaker

[0191] 210: CPU

[0192] 211: ROM

[0193] 212: RAM

[0194] 220: Medical Image Acquisition Department

[0195] 222: Specific scene recognition department

[0196] 224: Frequency acquisition unit

[0197] 226: Display control unit

Claims

1. A medical image processing device comprising a processor, wherein: The processor Acquire multiple medical images in real time in time series. Based on the medical image acquired in real time, identifying a specific scene of the medical image, As the specific scene is sequentially recognized for the plurality of medical images acquired in real time, the number of times the specific scene is recognized in the plurality of medical images is obtained, and the recognition frequency of the specific scene is updated based on the number of times the specific scene is recognized. Based on the updated recognition frequency, a notification display indicating the recognition degree displayed on the monitor is changed in two or more stages to continuously update the display mode of the notification display.

2. The medical image processing apparatus according to claim 1, wherein: The processor displays different colors on the monitor as the notification display according to the recognition frequency.

3. The medical image processing apparatus according to claim 1 or 2, wherein: The processor displays different line types on the monitor as the notification display according to the recognition frequency.

4. The medical image processing apparatus according to claim 1 or 2, wherein: The processor displays different blur states on the monitor as the notification display according to the recognition frequency.

5. The medical image processing apparatus according to claim 1 or 2, wherein: The processor acquires the recognition frequency based on the number of the medical images of the recognized specific scene.

6. The medical image processing apparatus according to claim 1 or 2, wherein: The processor acquires the recognition frequency based on the number of the medical images of the recognized specific scene that are continuous in time series.

7. The medical image processing apparatus according to claim 1 or 2, wherein: The processor calculates a feature value of the medical image of the specific scene, and identifies the specific scene by classifying the medical image into a category based on the feature value of the specific scene.

8. The medical image processing apparatus according to claim 1 or 2, wherein: The processor calculates a feature value of the medical image of the specific scene, and identifies the specific scene based on a similarity between the feature value and the medical image.

9. The medical image processing apparatus according to claim 1 or 2, wherein: When the recognition frequency is equal to or greater than a first threshold, the processor switches the notification display from an initial display to a first display and displays the notification on the monitor.

10. The medical image processing apparatus according to claim 9, wherein: When the recognition frequency is equal to or greater than a second threshold value that is greater than the first threshold value, the processor switches the notification display from the first display to a second display and displays the notification on the monitor.

11. The medical image processing apparatus according to claim 10, wherein: The processor displays a plurality of reference images for each of the plurality of specific scenes on the monitor to perform the first display and the second display.

12. The medical image processing apparatus according to claim 10, wherein: The processor displays a model image including parts corresponding to the plurality of specific scenes on the monitor, and performs the first display and the second display by associating the recognized specific scene with the model image.

13. The medical image processing apparatus according to claim 12, wherein: The processor makes the position where the specific scene is recognized correspond to the position on the model image, and causes the notification display to be superimposed on the model image and displayed on the monitor.

14. The medical image processing apparatus according to claim 1 or 2, wherein: The processor displays an initial display different from the notification display on the monitor when the specific scene has not been recognized.

15. A medical image processing method, comprising: A medical image acquisition step, acquiring multiple medical images in real time and in time sequence; a specific scene recognition step of recognizing a specific scene of the medical image based on the medical image acquired in real time; a frequency updating step of sequentially identifying the specific scene with respect to the plurality of medical images acquired in real time, obtaining a number of times the specific scene is identified in the plurality of medical images, and updating a recognition frequency of the specific scene identified in the specific scene identifying step based on the number of times the specific scene is identified; as well as The display control step changes a notification display indicating the degree of recognition displayed on a monitor in two or more stages according to the updated recognition frequency to continuously update the display mode of the notification display.

16. A recording medium which is non-transitory and computer-readable, wherein: Computer instructions for causing a computer to execute the medical image processing method according to claim 15 are recorded.

Images