Medical image processing device, medical image processing method, and recording medium

Abstract

The present invention provides a medical image processing device, a medical image processing method, and a program that suppress the reduction in visibility of endoscopic images and effectively display an observation status display related to the comprehensiveness of observation. The medical image processing device is a medical image processing device comprising a processor (210) and a memory (207), wherein the processor (210) acquires a plurality of medical images in a time series, determines the observation status of a small area unit of an object based on the medical images, records the determination result in the memory (207), and causes a monitor (400) to display the observation status display of the object based on the determination result recorded in the memory (207) when the observation status of the object changes.

Description

Technical Field

[0001] The present invention relates to a medical image processing device, a medical image processing method and a program. Background Art

[0002] Conventionally, in inspections using endoscope systems, it has been required to fully observe the region such as organs to be inspected.

[0003] Patent Document 1 describes a technique for preventing missed imaging during an inspection using an endoscope system. In the technique described in Patent Document 1, a map image showing imaged and unimaged areas of an organ to be imaged is displayed on a monitor as a notification display.

[0004] Previous technical literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2018-50890 Summary of the Invention

[0007] Technical issues to be solved by the invention

[0008] Typically, on an endoscope system monitor, the main display area displays the endoscopic image captured in real time during the inspection. Therefore, if the main display area displays a notification display, as described in Patent Document 1, it will overlap with the endoscopic image, reducing the visibility of the endoscopic image. Furthermore, if the notification display is displayed in a sub-display area of ​​the monitor, the display area is small and the visibility of the notification display is reduced. On the other hand, while displaying the notification display on a sub-monitor separate from the main monitor is also a possibility, there is the problem of the user being unable to focus on the endoscopic image displayed on the main monitor during the inspection.

[0009] Patent Document 1 mentioned above does not mention a display method that takes into account the visibility of an endoscopic image or the visibility of a notification display (map image).

[0010] The present invention is completed in view of such circumstances, and its purpose is to provide a medical image processing device, medical image processing method and program that suppress the reduction in visibility of endoscopic images and effectively display the observation status related to the comprehensiveness of the observation.

[0011] Means for solving technical problems

[0012] A medical image processing device as one embodiment of the present invention for achieving the above-mentioned purpose comprises a processor and a memory, wherein the processor acquires a plurality of medical images in a time series, determines the observation state of a small area unit of the subject based on the medical images, records the determination result in the memory, and when the observation state of the subject changes, causes a monitor to display the observation state of the subject based on the determination result recorded in the memory.

[0013] According to this aspect, the observation state of the subject is displayed on the monitor when the observation state of the subject changes. Thus, by displaying the observation state of the subject at an appropriate timing, it is possible to suppress the influence on the observation of the endoscopic image and to display it efficiently.

[0014] Preferably, the processor turns off the observation status display displayed on the monitor after a predetermined time has elapsed.

[0015] Preferably, the apparatus further includes a user operation accepting unit, and the processor sets the observation state display to display or hide based on an instruction from the user operation accepting unit.

[0016] Preferably, when determining the observation state of the small area, the processor determines that the observation is completed if the observation of the small area is completed, and determines that the observation is incomplete if the observation is not completed.

[0017] Preferably, the processor causes the monitor to display the observation status by character information.

[0018] Preferably, the processor adds character information regarding whether observation of the small area unit of the object is completed or not, and displays the information as an observation status display.

[0019] Preferably, the processor displays the observation status display by a subject model schematically representing the subject.

[0020] Preferably, the processor adds information on whether observation of the small area unit of the object is completed or not to the object model, and displays the information as an observation status display.

[0021] Preferably, the processor displays only the completed observation of the small area unit of the object or only the incomplete observation as the observation status display.

[0022] Preferably, the processor causes the monitor to display the medical image and causes the observation status display to be displayed superimposed on the medical image.

[0023] Preferably, the processor is a monitor having a first display area and a second display area smaller than the first display area, so that the first display area and the second display area display the observation status in different ways.

[0024] Preferably, the processor causes the second display area to always display the observation status display.

[0025] Preferably, the processor is a monitor having a third display area different from the first display area and the second display area, and the third display area displays the medical image.

[0026] A medical image processing method as another embodiment of the present invention is a medical image processing method of a medical image processing device having a processor and a memory, wherein the processor executes: a medical image acquisition step of acquiring a plurality of medical images in a time series; an observation state determination step of determining the observation state of a small area unit of the subject based on the medical image; a recording step of recording the determination result in the memory; and a display step of causing a monitor to display the observation state of the subject based on the determination result recorded in the memory at the moment when the observation state of the subject changes.

[0027] A program as another embodiment of the present invention is a program for causing a medical image processing device having a processor and a memory to execute a medical image processing method, wherein in the medical image processing method, the processor executes: a medical image acquisition step of acquiring multiple medical images in a time series; an observation state determination step of determining the observation state of a small area unit of the subject based on the medical image; a recording step of recording the determination result in a memory; and a display step of causing a monitor to display the observation state of the subject based on the determination result recorded in the memory at the moment when the observation state of the subject changes.

[0028] Effects of the Invention

[0029] According to the present invention, the observation state of the subject is displayed on the monitor when the observation state of the subject changes. Therefore, by displaying the observation state of the subject at an appropriate timing, the influence on the observation of the endoscopic image can be suppressed and the display can be performed effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is an external view of the endoscope system.

[0031] Figure 2 This is a block diagram showing the main structure of an endoscope system.

[0032] Figure 3 This is a functional block diagram of the medical image processing device in the image processing unit.

[0033] Figure 4 This is a diagram showing main information recorded in the recording unit.

[0034] Figure 5 It is a diagram showing the structure of a neural network.

[0035] Figure 6 Schematic diagram showing a structural example of an intermediate layer.

[0036] Figure 7 is a flowchart representing a medical image processing method.

[0037] Figure 8 This is a diagram showing an example of an observation status display.

[0038] Figure 9 It is a diagram showing a first modification of the observation state display.

[0039] Figure 10 It is a diagram showing a second modification of the observation state display.

[0040] Figure 11 It is a diagram showing a third modification of the observation state display.

[0041] Figure 12 It is a diagram showing a fourth modification of the observation state display.

[0042] Figure 13 It is a diagram showing a fifth modification of the observation state display.

[0043] Figure 14 It is a diagram showing a sixth modification of the observation state display.

[0044] Figure 15 This is a diagram illustrating an example of a monitor having a main display area and a sub display area.

[0045] Figure 16 This is a diagram for explaining an example in which different observation status displays are displayed in the main display area and the sub display area.

[0046] Figure 17 This is a diagram for explaining an example in which different observation status displays are displayed in the main display area and the sub display area. DETAILED DESCRIPTION

[0047] Below, preferred embodiments of the medical image processing device, medical image processing method and program involved in the present invention are described with reference to the accompanying drawings.

[0048] <Structure of the Endoscope System>

[0049] Figure 1 is an external view of the endoscope system 10, Figure 2FIG is a block diagram showing the main structure of the endoscope system 10. Figure 1 、 2 As shown, the endoscope system 10 is composed of an endoscope 100, an endoscope processor 200, a light source device 300, and a monitor 400. The endoscope processor 200 is equipped with the medical image processing device of the present invention.

[0050] <Structure of Endoscope>

[0051] The endoscope 100 includes a handheld operation unit 102 and an insertion unit 104 connected to the handheld operation unit 102. The operator (user) grasps the handheld operation unit 102 and operates it, inserting the insertion unit 104 into the body of a subject (living organism) for observation. The handheld operation unit 102 also includes air and water supply buttons 141, a suction button 142, function buttons 143 assigned various functions, and a capture button 144 for receiving capture instructions (still image, moving image). The insertion unit 104 consists of, in order from the handheld operation unit 102 side, a flexible portion 112, a curved portion 114, and a distal rigid portion 116. Specifically, the curved portion 114 is connected to the proximal end of the distal rigid portion 116, and the flexible portion 112 is connected to the proximal end of the curved portion 114. The handheld operation unit 102 is connected to the proximal end of the insertion unit 104. The user can bend the bending portion 114 by operating the hand operation part 102, and change the direction of the top hard part 116 upward, downward, left and right. The top hard part 116 is provided with a photographic optical system 130, an illumination part 123, a forceps opening 126, etc. (see Figure 1 、 2 ).

[0052] During observation and treatment, the operation unit 208 (see Figure 2 ) operation, white light and / or narrowband light (one or more of red narrowband light, green narrowband light, blue narrowband light, and violet narrowband light) can be emitted from the illumination lenses 123A and 123B of the illumination unit 123. Furthermore, by operating the air and water supply button 141, cleansing water can be discharged from a water supply nozzle (not shown) to clean the imaging lens 132 and illumination lenses 123A and 123B of the imaging optical system 130. A conduit (not shown) is connected to the forceps opening 126 opened in the distal rigid portion 116. A treatment instrument (not shown) used for tumor removal, for example, is inserted into this conduit, allowing it to be inserted and removed as appropriate to perform necessary treatment on the subject.

[0053] like Figure 1 and Figure 2As shown, a photographic lens 132 is provided on the distal end surface 116A of the distal rigid portion 116. Behind the photographic lens 132, a CMOS (Complementary Metal-Oxide Semiconductor) type imaging element 134, a driving circuit 136, and an AFE 138 (AFE: Analog Front End) are provided, and these elements output image signals. The imaging element 134 is a color imaging element having a plurality of pixels composed of a plurality of light-receiving elements arranged in a matrix (two-dimensional arrangement) using a specific pattern arrangement (Bayer arrangement, X-Trans (registered trademark) arrangement, honeycomb arrangement, etc.). Each pixel of the imaging element 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (photodiode, etc.). The photographic optical system 130 can generate a color image from pixel signals of three colors: red, green, and blue, or it can generate an image from pixel signals of one or two colors: red, green, and blue. Furthermore, the imaging element 134 may be a CCD (Charge Coupled Device) type. In addition, each pixel of the imaging element 134 may include a purple color filter corresponding to the purple light source 310V and / or an infrared filter corresponding to the infrared light source.

[0054] The optical image of the subject is formed by the imaging lens 132 onto the light-receiving surface (imaging surface) of the imaging element 134 and converted into an electrical signal. The electrical signal is then output to the endoscope processor 200 via a signal cable (not shown) and converted into a video signal. As a result, the endoscopic image (medical image) of the subject is displayed on the monitor 400 connected to the endoscope processor 200.

[0055] Furthermore, illumination lenses 123A and 123B of the illumination unit 123 are provided on the distal end surface 116A of the distal rigid portion 116, adjacent to the imaging lens 132. The emission end of a light guide 170, described later, is disposed behind the illumination lenses 123A and 123B. This light guide 170 is inserted through the insertion portion 104, the hand-side operation portion 102, and the universal cable 106, and the incident end of the light guide 170 is disposed within the light guide connector 108.

[0056] The user can sequentially capture time-series endoscopic images of the body by inserting or removing the endoscope 100 having the above configuration into or from the body of a subject while performing imaging at a predetermined frame rate.

[0057] <Structure of Light Source Device>

[0058] like Figure 2As shown, the light source device 300 comprises an illumination light source 310, an aperture 330, a condenser lens 340, and a light source control unit 350, and directs observation light into the light guide 170. The light source 310 includes a red light source 310R, a green light source 310G, a blue light source 310B, and a violet light source 310V, which respectively emit narrowband light of red, green, blue, and violet. The illumination intensity of the observation light from the light source 310 is controlled by the light source control unit 350, which can change (increase or decrease) the illumination intensity and stop the illumination as needed.

[0059] Light source 310 can emit any combination of red, green, blue, and violet narrowband light. For example, it can emit red, green, blue, and violet narrowband light simultaneously to illuminate white light (ordinary light) as observation light, or it can emit any one or two of these narrowband lights to illuminate narrowband light (special light). Light source 310 can also include an infrared light source that illuminates infrared light (an example of narrowband light). Alternatively, a light source that illuminates white light and filters that transmit white light and each narrowband light can be used to illuminate white light or narrowband light as observation light.

[0060] <Light source wavelength>

[0061] Light source 310 can generate light in a white wavelength band, or can generate light in multiple wavelength bands as the white wavelength band, or can generate light in a specific wavelength band narrower than the white wavelength band. The specific wavelength band can be a blue wavelength band or a green wavelength band in the visible range, or a red wavelength band in the visible range. When the specific wavelength band is a blue wavelength band or a green wavelength band in the visible range, it can include a wavelength band between 390 nm and 450 nm, or between 530 nm and 550 nm, and have a peak wavelength within the wavelength band between 390 nm and 450 nm, or between 530 nm and 550 nm. Furthermore, when the specific wavelength band is a red wavelength band in the visible range, it can include a wavelength band between 585 nm and 615 nm, or between 610 nm and 730 nm, and have a peak wavelength within the wavelength band between 585 nm and 615 nm, or between 610 nm and 730 nm.

[0062] The specific wavelength band may include a wavelength band where the absorption coefficient differs between oxyhemoglobin and deoxyhemoglobin, and the light in the specific wavelength band may have a peak wavelength in the wavelength band where the absorption coefficient differs between oxyhemoglobin and deoxyhemoglobin. In this case, the specific wavelength band may include 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band between 600 nm and 750 nm, and the light in the specific wavelength band may have a peak wavelength in the wavelength band between 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band between 600 nm and 750 nm.

[0063] In addition, the wavelength band of the light generated by the light source 310 may also include a wavelength band of 790nm to 820nm or 905nm to 970nm, and the light generated by the light source 310 has a peak wavelength in the wavelength band of 790nm to 820nm or 905nm to 970nm.

[0064] Alternatively, light source 310 may be configured to emit excitation light with a peak wavelength of 390 nm to 470 nm. In this case, an endoscopic image containing information about the fluorescence emitted by fluorescent substances within the subject (organism) can be obtained. Fluorescence pigments (such as fluorescein and acridine orange) may also be used to obtain fluorescence images.

[0065] The light source type (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode) etc.), wavelength, presence or absence of a filter, etc. of the light source 310 are preferably configured according to the type and location of the subject, the purpose of observation, etc. Furthermore, during observation, it is preferable to combine and / or switch the wavelength of the observation light according to the type and location of the subject, the purpose of observation, etc. When switching the wavelength, for example, a circular filter (rotating color filter) disposed in front of the light source and provided with a filter that transmits or blocks light of a specific wavelength can be rotated to switch the wavelength of the irradiated light.

[0066] In addition, the imaging element used when implementing the present invention is not limited to a color imaging element in which a color filter is provided for each pixel like the imaging element 134, but may also be a monochrome imaging element. When using a monochrome imaging element, the wavelength of the observation light can be switched in sequence to capture images in a plane order (color order). For example, the wavelength of the emitted observation light can be switched in sequence between narrow-band light (purple, blue, green, red), or broadband light (white light) can be irradiated and the wavelength of the emitted observation light can be switched by rotating color filters (red, green, blue, purple, etc.). In addition, one or more narrow-band lights (green, blue, purple, etc.) can be irradiated and the wavelength of the emitted observation light can be switched by rotating color filters (green, blue, purple, etc.). The narrow-band light can be infrared light of two or more wavelengths with different wavelengths (first narrow-band light, second narrow-band light).

[0067] By connecting the optical connector 108 (refer to Figure 1 、 2 ) is connected to the light source device 300, and the observation light irradiated from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and is irradiated toward the observation range from the illumination lenses 123A and 123B.

[0068] <Structure of Endoscope Processor Device>

[0069] based on Figure 2 The structure of the endoscope processor device 200 will be described. The endoscope processor device 200 inputs the image signal output from the endoscope viewer 100 via the image input controller 202, performs necessary image processing in the image processing unit 204, and outputs it via the video output unit 206. As a result, the endoscopic image is displayed on the monitor 400. These processes are performed under the control of the CPU 210 (CPU: Central Processing Unit). In addition, the CPU 210 functions as a processor of the medical image processing device. The communication control unit 205 controls communication with the hospital system (HIS: Hospital Information System) or the hospital LAN (Local Area Network) (not shown), and / or external systems or networks regarding the acquisition of medical images.

[0070] <Functions of the Image Processing Unit>

[0071] The image processing unit 204 can calculate characteristic quantities of endoscopic images, perform processing to emphasize or reduce components in specific wavelength bands, and perform processing to emphasize or de-emphasize specific objects (such as regions of interest or blood vessels at a desired depth). The image processing unit 204 may also include a special light image acquisition unit (not shown) that acquires a special light image containing information about a specific wavelength band based on a normal light image obtained by irradiating the image with light in a white wavelength band or by irradiating a plurality of wavelength bands as the white wavelength band. In this case, the signal for the specific wavelength band can be obtained through calculations based on the RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) color information contained in the normal light image. The image processing unit 204 may also include a feature image generation unit (not shown) that acquires and displays a feature image as an endoscopic image. The feature image generation unit generates the feature image by performing a calculation based on at least one of a normal light image obtained by irradiating light in a white wavelength band or a normal light image obtained by irradiating light in multiple wavelength bands as the white wavelength band, and a special light image obtained by irradiating light in a specific wavelength band. The above processing is performed under the control of the CPU 210.

[0072] Furthermore, as described below, the image processing unit 204 has various functions in the medical image processing apparatus.

[0073] Figure 3 2 is a functional block diagram of a medical image processing device in the image processing unit 204. The image processing unit 204 includes a medical image acquisition unit 220, an observation state determination unit 222, and a display control unit 224.

[0074] <Implementation of various processor functions>

[0075] The functions of each part of the above-mentioned image processing unit 204 can be implemented using various processors and recording media. Among the various processors, there are general-purpose processors such as CPUs (Central Processing Units) that implement various functions by executing software (programs). In addition, among the various processors mentioned above, there are also processors specifically used for image processing, such as GPUs (Graphics Processing Units), FPGAs (Field Programmable Gate Arrays), and processors whose circuit structures can be changed after manufacturing, such as programmable logic devices (PLDs). In the case of learning or recognizing images as in the present invention, a structure using a GPU is effective. Moreover, processors having a circuit structure specially designed for executing specific processing such as ASICs (Application Specific Integrated Circuits), such as dedicated circuits, are also included in the various processors mentioned above.

[0076] The functions of each part can be implemented by one processor, or by multiple processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). In addition, multiple functions can also be implemented by one processor. As an example of using one processor to constitute multiple functions, first, there is a form such as a computer, which uses a combination of one or more CPUs and software to constitute a processor, and the processor is used to implement multiple functions. Secondly, there is a form such as a system on chip (System On Chip: SoC), which uses a processor that implements the functions of the entire system with an IC (Integrated Circuit) chip. In this way, various functions are constituted using one or more of the above-mentioned various processors as a hardware structure. Moreover, more specifically, the hardware structure of these various processors is a circuit (circuitry) that combines circuit elements such as semiconductor elements. These circuits can also be circuits that use logical sum, logical product, logical negation, exclusive or, and logical operations combining them to implement the above-mentioned functions.

[0077] When the above-mentioned processor or circuit executes the software (program), the computer-readable code of the executed software (for example, the various processors or circuits constituting the image processing unit 204, and / or their combination) is first stored in a non-temporary recording medium such as ROM211 (ROM: ReadOnly Memory), and the computer refers to the software. The software pre-stored in the non-temporary recording medium includes a program for executing the medical image processing method of the medical image processing device involved in the present invention and data used during execution. Instead of recording the code in ROM211, it is also possible to record it in various non-temporary recording media such as magneto-optical recording devices and semiconductor memories. When the software is used for processing, for example, RAM212 (RAM: Random Access Memory) is used as a temporary storage area. In addition, for example, data stored in an EEPROM (Flectrically Erasable and Programmable Read 0nly Memory) not shown in the figure can also be referred to. The recording unit 207 can also be used as a "non-temporary recording medium".

[0078] ROM 211 (Read Only Memory) is a nonvolatile storage element (non-transitory recording medium) that stores computer-readable code for programs that cause CPU 210 and / or image processing unit 204 to execute various image processing methods. RAM 212 (Random Access Memory) is a storage element used for temporary storage during various processing operations and can also be used as a buffer for image acquisition. The audio processing unit 209 outputs sound and voice from speaker 209A under the control of CPU 210.

[0079] The operation unit 208 may be composed of a keyboard, a mouse, or other devices (not shown), and the user can issue an instruction to execute a process or designate conditions required for the execution via the operation unit 208 .

[0080] <Information recorded in the recording section>

[0081] Figure 4 2 is a diagram showing the main information recorded in the recording unit 207. The recording unit (memory) 207 records a medical image (endoscopic image) 260, a determination result 262 of the observation state determination unit 222, and the like. In addition, in an examination performed using the endoscope system 10, information related to a series of small areas to be observed is recorded. Here, the small areas of the subject are, for example, various parts of the internal organs. Specifically, when an examination is performed to observe all parts of the stomach, the small areas are the cardia, the fundus, the anterior part, the corpus (upper, middle, lower part), the vestibule, the anterior wall, the posterior wall, the greater curvature, and the lesser curvature.

[0082] <Neural Network-Based Recognition Unit>

[0083] The observation status determination unit 222 within the image processing unit 204 includes a discriminator capable of identifying small areas of the subject, constructed using a learned model such as a neural network (a model learned using a set of images obtained by imaging a living organism). Furthermore, the observation status determination unit 222 determines whether observation of the small area has been completed based on factors such as the position of the small area identified by the discriminator and the number of endoscopic images in which the small area was identified. The following describes the structure of the discriminator included in the observation status determination unit 222 when a CNN (Convolutional Neural Network) is used as the neural network.

[0084] <Structure Example of Identifier>

[0085] Figure 5 This is a diagram showing the structure of CNN232 (neural network). Figure 5 In the example shown in part (a), CNN232 has an input layer 232A, an intermediate layer 232B and an output layer 232C. The input layer 232A inputs the endoscopic image acquired by the medical image acquisition unit 220 and outputs the feature quantity. The intermediate layer 232B includes a convolution layer 234 and a pooling layer 235, which calculates other feature quantities by inputting the feature quantities output from the input layer 232A. These layers are formed into a structure in which a plurality of "nodes" are connected by "edges", and the weight coefficients applied to the input image are associated with the nodes and edges and are stored in a weight coefficient storage unit not shown. The value of the weight coefficient changes as learning progresses.

[0086] <Processing of the middle layer>

[0087] The middle layer 232B calculates the feature quantity through convolution operation and pooling processing. The convolution operation performed in the convolution layer 234 is a process of obtaining a feature map by using a convolution operation of a filter, and plays the role of extracting features such as edges from the image. By using the convolution operation of the filter, a "feature map" of one channel (one sheet) is generated for one filter. When the size of the "feature map" is reduced (downscaling) by convolution, it becomes smaller as convolution is performed in each layer. The pooling processing performed in the pooling layer 235 is a process of reducing (or enlarging) the feature map output by the convolution operation to obtain a new feature map, and plays the role of providing robustness to prevent the extracted features from being affected by parallel movement, etc. The middle layer 232B can be composed of one or more layers that perform these processes. In addition, CNN232 can also be constructed without a pooling layer 235.

[0088] CNN232 can also be Figure 5As shown in the example of part (b) of FIG, the fully connected layer 236 is included. The layer structure of CNN 232 is not limited to the case where the convolution layer 234 and the pooling layer 235 are repeated one by one, and a plurality of arbitrary layers (for example, the convolution layer 234) may be included consecutively.

[0089] Figure 6 Yes Figure 5 The diagram shows a schematic diagram of an example structure of the intermediate layer 232B of the CNN 232. In the initial (first) convolutional layer of the intermediate layer 232B, a convolution operation is performed on an image group consisting of multiple endoscopic images and a filter F1. The image group consists of N images (N channels) with an image size of height H and width W. When a normal light image is input, the images that constitute the image group are images of three channels: R (red), G (green), and B (blue). Since the image group consists of N channels (N images), the filter F1 that performs the convolution operation with the image group has a filter size of 5 (5×5), for example, 5×5×N. By performing the convolution operation using the filter F1, a "feature map" of one channel (one image) is generated for one filter F1. The filter F2 used in the second convolutional layer has a filter size of 3 (3×3), for example, 3×3×M.

[0090] Similar to the first convolution layer, filters F2 to F n The size of the feature map in the nth convolutional layer is smaller than that in the second convolutional layer because it is reduced by the previous convolutional layer or pooling layer.

[0091] In the layer of the intermediate layer 232B, low-order feature extraction (edge ​​extraction, etc.) is performed in the convolution layer close to the input side, and high-order feature extraction (feature extraction related to the shape, structure, etc. of the identified object) is performed as it approaches the output side.

[0092] In addition to the convolutional layer 234 and the pooling layer 235, the intermediate layer 232B may also include a batch normalization layer. Batch normalization normalizes the distribution of data in small batches during learning, and helps to accelerate learning, reduce dependence on initial values, and prevent over-learning.

[0093] The output layer 232C outputs the feature quantity calculated by the intermediate layer 232B in a form suitable for recognition. The output layer 232C may also include a fully connected layer.

[0094] <Each process of the medical image processing method>

[0095] Next, a medical image processing method using the medical image processing device is described.

[0096] Figure 7 is a flowchart showing a medical image processing method. Figure 7 Each step will be described. In addition, the following describes a case where observation of the small areas, namely, area 1, area 2, and area 3, is performed in area units when examining an organ A as a subject.

[0097] (Medical image acquisition steps)

[0098] The medical image acquisition unit 220 sequentially acquires multiple medical images of organ A in a time series (step S10). Furthermore, the recording unit 207 records that observations of regions 1, 2, and 3 of organ A have been performed. Initially, regions 1, 2, and 3 are recorded as incomplete.

[0099] (Observation status determination step)

[0100] The observation status determination unit 222 determines the observation status of Zone 1, Zone 2, and Zone 3 of organ A based on the acquired medical image (step S11). The observation status determination unit 222 identifies Zone 1, Zone 2, or Zone 3 in the medical image. Then, based on the identification result, the observation status of Zone 1, Zone 2, and Zone 3 is determined. For example, if Zone 1 is identified as the center of the medical image in ten consecutive medical images in a time series, the observation status determination unit 222 determines that the observation of Zone 1 is complete.

[0101] (Record steps)

[0102] The recording unit 207 records the result of the determination made by the observation state determination unit 222 (step S12). At the start of the inspection (initial state), areas 1, 2, and 3 are recorded as observation incomplete. However, when the observation state determination unit 222 determines that the observation of each area is completed, the record is updated to observation completed.

[0103] (Show steps)

[0104] The display control unit 224 determines whether the observation status of the subject has changed (step S13). Then, when it is determined that the observation status of the subject has changed, the display control unit 224 displays the subject observation status display 501 on the monitor 400 (step S14). Here, the moment the observation status of the subject has changed refers to the moment when a change in the observation status of the plurality of predetermined small areas for observation, recorded in the recording unit 207, is recorded in the recording unit 207. For example, this may be the case where the recording unit 207 records areas 1, 2, and 3 as being "uncompleted," but the observation status determination unit 222 determines that area 1 is "completed," and the observation status of area 1 changes to "completed" in the recording unit 207. Furthermore, the subject observation status display 501 notifies the user of the observation status of the small areas constituting the subject being observed. By viewing the observation status display 501, the user can confirm whether the small areas of the subject being observed can be fully observed.

[0105] Figure 8 1 is a diagram showing an example of an observation status display 501 of the organ A displayed on the monitor 400 .

[0106] exist Figure 8 In the illustrated case, an endoscopic image 503 is displayed on the entire monitor 400. Furthermore, when area 3, as recorded by the recording unit 207, is updated from observation uncompleted to observation completed, the display control unit 224 causes the observation status display 501 to be superimposed on the endoscopic image 503 and displayed on the monitor 400. The observation status display 501 is a list display with text information indicating areas that have been observed and areas that have not been observed. In the observation status display 501, areas 1 and 3, which have been observed (represented as "area" in the figure), are listed under "Complete," while area 2, which has not been observed, is listed under "Not Completed."

[0107] Back to Figure 7 , the display control unit 224 then continues to display the observation status display 501 until a predetermined display time has elapsed. The display time can be appropriately set by the user. If the user can confirm the observation status, it is preferred that the observation status display 501 be turned off so that the endoscopic image 503 can be observed. Therefore, it is preferred to set the display time based on the time during which the user can confirm the observation status. For example, the display time can be set to 10 seconds or 30 seconds. After the predetermined time has elapsed, the display control unit 224 turns off the observation status display 501 (step S15).

[0108] Afterwards, the medical image acquisition unit 220 determines whether the observation of all small area units is completed (step S16). Since the observation of area 2 has not been completed, a medical image is further acquired (step S10).

[0109] As described above, in the present invention, when observing small areas of organ A, namely, areas 1, 2, and 3, observation status display 501 is performed at the point in time when the observation status of organ A changes. Thus, observation status display 501 is performed when necessary, and in other cases, the display is controlled, thereby suppressing the influence on the observation of the endoscopic image and enabling efficient display.

[0110] <Modification of Observation Status Display>

[0111] In use Figure 8 While the example described above illustrates an observation status display 501 that displays text information indicating areas where observation has been completed and areas where observation has not been completed, examples of observation status display 501 are not limited thereto. The display format of observation status display 501 is not particularly limited, as long as it can inform the user of the observation status of the subject being observed using text or graphics. A specific example of observation status display 501 will be described below.

[0112] Figure 9 This figure shows a first variation of observation status display 501. This example shows observation status display 501 with text information. This example displays only the areas for which observation has not yet been completed. Specifically, if area 2 is not yet completed, area 2 and the text information are displayed below "Not Completed."

[0113] In this way, by displaying the observation status 501 of only the small area where observation is not completed on the monitor 400, the user can clearly identify the small area where observation is not completed, and can achieve comprehensive observation. Figure 9 In the example, an example of displaying a small area in which observation is not completed in the observation status display 501 is described, but a small area in which observation is completed may be displayed in the observation status display 501. In this case, the user can clearly identify the small area in which observation is completed.

[0114] Figure 10 This is a diagram showing a second modification of the observation status display 501. This example is an observation status display 501 of character information. In the observation status display 501 of this example, all the small areas (areas 1 to 5) of the subject that constitutes the observation object are displayed as a list. Moreover, in the observation status display 501, the character color is changed to display the areas where observation is completed and the areas where observation is not completed. Specifically, in the observation status display 501, areas 3 and 5 are displayed with characters of the same color because observation is not completed, and areas 1, 2, and 4 are displayed with characters of the same color because observation is completed. In this way, all the small areas are displayed in a list, and the information of whether observation is completed or not is assigned to characters in small area units, so that the user can fully identify the areas where observation is completed or not.

[0115] Figure 11 This is a diagram showing a modified example 3 of the observation status display 501. This example is an observation status display 501 based on character information. The observation status display 501 of this example displays all the small areas (areas 1 to 5) of the subject that constitutes the observation object as a list. Moreover, in the observation status display 501, "○" or "×" is displayed next to the characters of the areas where observation is completed and observation is not completed. Specifically, in the observation status display 501, areas 3 and 5 are assigned "×" because observation is not completed, and areas 1, 2, and 4 are assigned "○" because observation is completed. In this way, all the small areas are displayed in a list, and the information of whether observation is completed or not is displayed next to the characters in small area units, so that the user can fully identify the areas where observation is completed or not.

[0116] Figure 12 : is a diagram showing a modified example 4 of the observation status display 501. This example is an observation status display 501 based on character information. The observation status display 501 of this example shows the proportion of incomplete observation as the observation status display 501. Here, the proportion of incomplete observation refers to the proportion of small areas where observation is incomplete among a plurality of small areas predetermined for observation. In addition, the calculation of this proportion can be calculated using the number of small areas or the area of ​​the small areas. In addition, in Figure 12 In the example shown, the ratio is expressed in characters, but the ratio may be expressed in a graphic such as a bar graph. In this way, by indicating the ratio of the observation not completed, the user can clearly know the area where the observation is not completed.

[0117] Figure 13 : is a diagram showing a modified example 5 of the observation status display 501. This example is based on the observation status display 501 of the subject model M that schematically represents the subject. The subject model M is a diagram that schematically represents the stomach of the subject to be observed. The subject model M assigns different colors to the area 513 where observation has been completed and the area 511 where observation has not been completed in the small area unit. In this way, by giving the subject model M information on whether observation has been completed or not, the user can identify the position of the small area where observation has been completed and the position of the small area where observation has not been completed in the subject. In addition, in Figure 13 In the example shown, observation completion and observation incomplete information are given by changing the color of the area of ​​the subject model M, but the present invention is not limited to this example. For example, information on observation completion and observation incomplete may be given to the subject model M by changing the color density.

[0118] Figure 14: is a diagram showing a modification example 6 of the observation status display 501. This example is an observation status display 501 based on a subject model M that schematically represents the subject. In this example, only the small areas where observation is completed are shown in the subject model M. Moreover, when the observation of all the small areas is completed, the entire subject model M (stomach) is displayed. In this way, by giving the subject model M information on whether the observation is completed or not, the user can identify the positions of the small areas where the observation is completed and the positions of the small areas where the observation is not completed in the subject. In addition, in Figure 13 and Figure 14 In the embodiment, a cutout view of the stomach as the subject is used as the subject model M, but the present invention is not limited thereto. For example, the subject model M may be an expanded view of the stomach as the subject.

[0119] <Modification Example of Monitor>

[0120] In the above description, an example of the monitor 400 having only the main display area is described, but the present invention is not limited thereto.

[0121] Figure 15 This is a diagram for explaining an example of a monitor 401 having a main display area and a sub display area.

[0122] like Figure 15 As shown, the monitor 401 has a main display area 402 and a sub-display area 404. The main display area 402 displays the endoscopic image 503 captured by the endoscope 100 in real time. The sub-display area 404 is set smaller than the main display area 402 and displays information such as imaging conditions, date, and patient information. Figure 15 The information displayed in the sub-display area 404 is omitted in the figure. In the monitor 401, the observation state display 501 is performed in the main display area 402 and the sub-display area 404. In the main display area 402, as described above, the observation state display 501 is displayed when the observation state of the subject changes, and the display ends when the display time has passed. Figure 15 , a situation where the observation status display 501 of the main display area 402 is not displayed is illustrated.

[0123] Observation status display 501B is always displayed in sub-display area 404. Since sub-display area 404 has a smaller display area, observation status display 501B is also smaller than observation status display 501A displayed in main display area 402. However, it serves as an aid for comprehensive observation. Furthermore, the main display area 402 and sub-display area 404 can display the same observation status display or different observation status displays.

[0124] Figure 16 and Figure 17 This is a diagram for explaining an example in which different observation status displays are displayed in the main display area and the sub display area.

[0125] exist Figure 16 In the example shown, a detailed observation state display 501A showing a list of small areas is displayed in the main display area 402. In addition, an observation state display 501B of the subject model M is displayed in the sub-display area 404.

[0126] exist Figure 17 In the example shown, detailed observation status display 501A showing a list of small areas is displayed in main display area 402. Observation status display 501B showing the percentage of uncompleted observations is displayed in sub-display area 404.

[0127] like Figure 16 and Figure 17 As shown, in the main display area 402, an observation status display 501A representing small areas in a list is displayed, which allows the user to grasp the observation status of the small area units in detail when the observation status of the subject changes. As a result, the user can grasp the detailed observation status of the subject as the number of small areas whose observations have been completed increases. In addition, the subject model M or the observation status display 501B indicating the proportion of incomplete observation is always displayed in the sub-display area 404. As a result, even if the observation status display 501A is not displayed in the main display area 402, the user can roughly grasp the observation status of the subject. In addition, in the above example, the monitor 401 having the main display area (first display area) 402 and the sub-display area (second display area) 404 is described, but the observation status display 501 can also be displayed on a monitor having a third display area, for example. In addition, the observation status display 501 can also be displayed on multiple monitors.

[0128] <Other Examples>

[0129] In the above description, a method of displaying the observation status display 501 during a predetermined display time has been described. However, the method of hiding the observation status display 501 that was once displayed is not limited to this.

[0130] For example, the display control unit 224 may also be based on the user's operation via the handy operation unit 102 (user operation receiving unit) (see Figure 1 ) to hide the observation status display 501. Furthermore, the display control unit 224 may also display the observation status display 501, which was once hidden, again based on a command input by the user via the handy operation unit 102 (user operation receiving unit). In this way, the user can control the display and hiding of the observation status display 501 via the handy operation unit 102, thereby enabling the user to confirm the observation status display at a desired timing.

[0131] While the examples of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0132] Explanation of symbols

[0133] 10: Endoscopic system

[0134] 100: Endoscope

[0135] 102: Hand operation department

[0136] 104: Insertion

[0137] 106: Universal Cable

[0138] 108: Optical connector

[0139] 112: Soft Department

[0140] 114: Bend

[0141] 116: Top hard part

[0142] 116A: Top end face

[0143] 123: Lighting Department

[0144] 123A: Lighting lens

[0145] 123B: Lighting lens

[0146] 126: Clamping

[0147] 130: Photographic Optical System

[0148] 132: Photographic lens

[0149] 134: Camera element

[0150] 136: Driving circuit

[0151] 141: Gas and water supply buttons

[0152] 142: Attract button

[0153] 143: Function button

[0154] 144: Shoot button

[0155] 170: Light guide

[0156] 200: Endoscope processor device

[0157] 202: Image input controller

[0158] 204: Image Processing Department

[0159] 205: Communication Control Department

[0160] 206: Video output unit

[0161] 207: Records Department

[0162] 208: Operation Department

[0163] 209: Sound Processing Department

[0164] 209A: Speaker

[0165] 210: CPU

[0166] 211: ROM

[0167] 212: RAM

[0168] 220: Medical Image Acquisition Department

[0169] 222: Observation status determination unit

[0170] 224: Display control unit

[0171] 232A: Input layer

[0172] 232B: Middle layer

[0173] 232C: Output layer

[0174] 234: Convolutional layer

[0175] 235: Pooling layer

[0176] 236: Fully connected layer

[0177] 300: Light source device

[0178] 310: Light Source

[0179] 310B: Blue light source

[0180] 310G: Green light source

[0181] 310R: Red light source

[0182] 310V: Purple light source

[0183] 330: Aperture

[0184] 340: Focusing lens

[0185] 350: Light source control unit

[0186] 400: Monitor

Claims

1. A medical image processing device comprising a processor and a memory, wherein: The processor Acquire multiple medical images in time series, In the medical image, a plurality of small areas with different predetermined positions within the organ of the subject are identified using a learned model that has been learned in a manner of identifying each small area unit of the plurality of small areas. Based on the result of the recognition, an observation state is determined. In the observation state determination, for each of the plurality of small areas of the subject, if the observation is completed, the observation is determined to be completed; if the observation is not completed, the observation is determined to be incomplete. displaying an observation status on a monitor, in which a user is informed, based on a result of the determination, whether the observation status of each of the plurality of small areas of the subject is observation incomplete or observation completed; recording the result of the determination in the memory, When the observation status of any one of the multiple small areas of the subject changes from observation incomplete to observation completed, the observation status display is updated based on the result of the judgment recorded in the memory to notify that the observation status of the small area is observation completed.

2. A medical image processing device comprising a processor and a memory, wherein: The processor Acquire multiple medical images in time series, Based on the medical image, the observation status of each small area unit of the subject is determined. In the determination of the observation status, if the observation of the small area unit is completed, it is set as observation completed; if the observation is not completed, it is set as observation incomplete. recording the result of the determination in the memory, When the observation state of the small area of ​​the object changes from observation incomplete to observation completed, the monitor is caused to display an observation state display, in which the user is informed of the observation state of the small area unit of the object based on the result of the determination recorded in the memory. It also includes a user operation receiving unit, The processor sets the observation status display to display or non-display based on an instruction from the user operation accepting unit.

3. A medical image processing device comprising a processor and a memory, wherein: The processor Acquire multiple medical images in time series, Based on the medical image, the observation status of each small area unit of the subject is determined. In the determination of the observation status, if the observation of the small area unit is completed, it is set as observation completed; if the observation is not completed, it is set as observation incomplete. recording the result of the determination in the memory, When the observation state of the small area of ​​the object changes from observation incomplete to observation completed, the monitor is caused to display an observation state display, in which the user is informed of the observation state of the small area unit of the object based on the result of the determination recorded in the memory. The processor turns off the observation status display displayed on the monitor after a predetermined time has elapsed.

4. A medical image processing device comprising a processor and a memory, wherein: The processor Acquire multiple medical images in time series, Based on the medical image, the observation status of each small area unit of the subject is determined. In the determination of the observation status, if the observation of the small area unit is completed, it is set as observation completed; if the observation is not completed, it is set as observation incomplete. recording the result of the determination in the memory, When the observation state of the small area of ​​the object changes from observation incomplete to observation completed, the monitor is caused to display an observation state display, in which the user is informed of the observation state of the small area unit of the object based on the result of the determination recorded in the memory. The processor causes the monitor to display the medical image and causes the observation status display to be displayed superimposed on the medical image.

5. A medical image processing device comprising a processor and a memory, wherein: The processor Acquire multiple medical images in time series, Based on the medical image, the observation status of each small area unit of the subject is determined. In the determination of the observation status, if the observation of the small area unit is completed, it is set as observation completed; if the observation is not completed, it is set as observation incomplete. recording the result of the determination in the memory, When the observation state of the small area of ​​the object changes from observation incomplete to observation completed, the monitor is caused to display an observation state display, in which the user is informed of the observation state of the small area unit of the object based on the result of the determination recorded in the memory. The processor is the monitor having a first display area and a second display area smaller than the first display area, and causes the first display area and the second display area to display the observation status in different ways, The first display area is caused to display the medical image, and the second display area is caused to display the observation status display.

6. The medical image processing apparatus according to any one of claims 1, 2, 4, and 5, wherein: The processor turns off the observation status display displayed on the monitor after a predetermined time has elapsed.

7. The medical image processing apparatus according to any one of claims 1, 3 to 5, wherein: It also includes a user operation receiving unit, The processor sets the observation status display to display or non-display based on an instruction from the user operation accepting unit.

8. The medical image processing apparatus according to any one of claims 1 to 5, wherein: The processor causes the monitor to display the observation status display using character information.

9. The medical image processing apparatus according to claim 8, wherein: The processor adds information on whether observation of the small area unit of the object is completed or not to the character information, and causes the monitor to display the information as the observation status display.

10. The medical image processing apparatus according to any one of claims 1 to 5, wherein: The processor displays the observation status display by a subject model schematically representing the subject.

11. The medical image processing apparatus according to claim 10, wherein: The processor adds information on whether observation of the small area unit of the object is completed or not to the object model, and causes the monitor to display the information as the observation status display.

12. The medical image processing apparatus according to any one of claims 1 to 5, wherein: The processor causes the monitor to display as the observation status display only the observation of the small area unit of the object is completed or only the incomplete display.

13. The medical image processing apparatus according to any one of claims 1 to 3 and 5, wherein: The processor causes the monitor to display the medical image and causes the observation status display to be displayed superimposed on the medical image.

14. The medical image processing apparatus according to any one of claims 1 to 4, wherein: The processor is the monitor having a first display area and a second display area smaller than the first display area, and causes the first display area and the second display area to display the observation status display in different ways.

15. The medical image processing apparatus according to claim 14, wherein: The processor causes the first display area to display the observation status display when the observation status changes, and causes the second display area to always display the observation status display.

16. The medical image processing apparatus according to claim 5 or 15, wherein: The processor is the monitor having a third display area different from the first display area and the second display area, and causes the third display area to display the medical image.

17. The medical image processing apparatus according to claim 5 or 15, wherein: The first display area is caused to display the medical image, and the second display area is caused to display the observation status display.

18. The medical image processing apparatus according to claim 5, wherein: The observation state display is composed of a subject model schematically representing the subject.

19. The medical image processing apparatus according to claim 5, wherein: The observation status display is composed of character information.

20. The medical image processing apparatus according to claim 19, wherein: The observation status display is composed of the character information indicating the percentage of incomplete observation.

21. The medical image processing apparatus according to any one of claims 2 to 5, wherein: The processor In the medical image, a plurality of predetermined small areas of the subject are identified using a learned model that has been learned in a manner of identifying each of the plurality of small areas as a unit of the small area. The observation state is determined based on the result of the identification.

22. A medical image processing device comprising a processor and a memory, wherein: The processor Acquire multiple medical images in time series, In the medical image, a plurality of predetermined small areas of the subject are identified using a learned model that has been learned in a manner of identifying each of the plurality of small areas. Based on the result of the identification, the observation status of each of the small area units is determined. In the determination of the observation status, if the observation of the small area unit of the subject is completed, it is determined that the observation is completed; if the observation is not completed, it is determined that the observation is not completed. displaying an observation status display on a monitor, in which the observation status of the small area unit of the object is notified to a user based on the result of the determination, recording the result of the determination in the memory, updating the observation status display based on the result of the determination recorded in the memory when the observation status of the small area of ​​the object changes from observation not completed to observation completed, The processor causes the monitor to display the medical image and causes the observation status display to be displayed superimposed on the medical image.

23. A medical image processing method for a medical image processing device, the medical image processing device comprising a processor and a memory, wherein: Executed by the processor: A medical image acquisition step, acquiring a plurality of medical images in time series; A recognition step, in the medical image, recognizing a plurality of small areas with different predetermined positions within the organ of the subject using a learned model that has been learned in a manner of recognizing each small area unit of the plurality of small areas, an observation status determination step of determining an observation status based on the result of the recognition, wherein, in the observation status determination, for each of the plurality of small areas of the subject, if the observation is completed, the observation is determined to be completed; otherwise, the observation is determined to be incomplete; a display step of displaying an observation status on a monitor, wherein, in the observation status display, a user is informed of whether the observation status of each of the plurality of small areas of the object is observation incomplete or observation completed based on the result of the determination; a recording step of recording the result of the determination in the memory; as well as An updating step, in which, when the observation status of any one of the multiple small areas of the subject changes from observation incomplete to observation completed, the observation status display is updated based on the result of the judgment recorded in the memory so as to notify that the observation status of the small area is observation completed.

24. A recording medium which is non-transitory and computer-readable, wherein: A program for causing a computer to execute the medical image processing method according to claim 23 is recorded.

Images