Medical image processing device, medical image processing method, and program

Abstract

To provide a medical image processing device, a medical image processing method, and a program, which can reduce a gap between a recognition of a recognition device and a recognition of a user.SOLUTION: A medical image processing device includes a processor (210) and a memory (207). The processor (210) obtains a plurality of medical images in a time series, determines an observation state in a unit of a small region of the subject based on the medical image, records the result of the determination in the memory (207), and displays, on the monitor (400), the observation state display of the subject based on the determination result recorded in the memory (207) at the time where a change occurs in the observation state of the subject.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a medical image processing device, a medical image processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, in examinations performed using endoscope systems, there has been a demand for comprehensive observation of an area of ​​an organ or the like that is the subject of examination.

[0003] Patent Document 1 describes a technology aimed at preventing missed images during an examination using an endoscope system. In the technology described in Patent Document 1, a map image showing the already-imaged and unimaged areas of the organ to be imaged is displayed on a monitor as a notification display. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-50890 A Summary of the Invention [Problem to be solved by the invention]

[0005] Here, in general, an endoscopic image captured in real time during an examination is displayed in a main display area on the monitor of an endoscope system. Therefore, if a notification display as described in Patent Document 1 is displayed in the main display area, it is superimposed on the endoscopic image, and the visibility of the endoscopic image is reduced. If the notification display is displayed in a sub-display area of ​​the monitor, the display area is small and the visibility of the notification display is reduced. On the other hand, it is also possible to display the notification display on a sub-monitor separate from the main monitor, but this has the problem that the user cannot focus on the endoscopic image displayed on the main monitor during an examination.

[0006] The above-mentioned Patent Document 1 does not mention a display mode that takes into consideration the visibility of an endoscopic image or the visibility of a notification display (map image).

[0007] The present invention has been made in consideration of the above circumstances, and its object is to provide a medical image processing device, a medical image processing method, and a program that effectively display an observation status indication regarding the comprehensiveness of observation while suppressing a decrease in visibility of endoscopic images. [Means for solving the problem]

[0008] A medical image processing device, which is one aspect of the present invention for achieving the above-mentioned object, is a medical image processing device equipped with a processor and a memory, in which the processor acquires a plurality of medical images in chronological order, judges the observation state of each small area of ​​the subject based on the medical images, records the judgment results in memory, and, at the point in time when a change occurs in the observation state of the subject, displays on a monitor an observation state display of the subject based on the judgment results recorded in the memory.

[0009] According to this aspect, when a change occurs in the observation state of the subject, the observation state display of the subject is displayed on the monitor. This makes it possible to effectively display the observation state display of the subject while minimizing the influence on the observation of the endoscopic image by displaying the observation state display of the subject at an appropriate timing.

[0010] Preferably, the processor causes the observation status display displayed on the monitor to disappear after a predetermined time has elapsed.

[0011] Preferably, the apparatus further includes a user operation receiving section, and the processor displays or hides the observation state display based on an instruction from the user operation receiving section.

[0012] Preferably, when determining the observation state of the small area, the processor determines that observation is completed if observation of the small area is completed, and determines that observation is incomplete if observation is incomplete.

[0013] Preferably, the processor causes the monitor to display the observation status indication using text information.

[0014] Preferably, the processor adds information regarding whether observation of each small area of ​​the subject has been completed or not to text information and displays it as an observation status indication.

[0015] Preferably, the processor causes the observation state display to be displayed using an object model that typically represents an object.

[0016] Preferably, the processor provides the object model with information regarding whether observation of each small area of ​​the object has been completed or not, and displays the information as an observation status display.

[0017] Preferably, the processor displays, as an observation status display, an indication of only whether observation of the small area unit of the subject has been completed or not.

[0018] Preferably, the processor causes the monitor to display a medical image and to display an observation status indication superimposed on the medical image.

[0019] Preferably, the processor is a monitor having a first display area and a second display area smaller than the first display area, and causes the observation status display to be displayed in different manners in the first display area and the second display area.

[0020] Preferably, the processor causes the second display area to constantly display the observation status display.

[0021] Preferably, the processor causes a monitor having a third display area different from the first display area and the second display area to display the medical image in the third display area.

[0022] Another aspect of the present invention is a medical image processing method for a medical image processing device having a processor and a memory, in which the processor executes a medical image acquisition step of acquiring a plurality of medical images in chronological order, an observation state determination step of determining the observation state of each small area of ​​the subject based on the medical images, a recording step of recording the determination result in memory, and a display step of displaying on a monitor an observation state display of the subject based on the determination result recorded in memory when a change occurs in the observation state of the subject.

[0023] Another aspect of the present invention is a program that causes a medical image processing device having a processor and a memory to execute a medical image processing method, in which the processor causes the medical image processing method to execute a medical image acquisition step of acquiring a plurality of medical images in chronological order, an observation state determination step of determining the observation state of each small area of ​​the subject based on the medical images, a recording step of recording the determination result in memory, and a display step of displaying on a monitor an observation state display of the subject based on the determination result recorded in memory when a change occurs in the observation state of the subject. Effect of the Invention

[0024] According to the present invention, the observation state display of the subject is displayed on the monitor when a change occurs in the observation state of the subject. Therefore, by displaying the observation state display of the subject at an appropriate timing, it is possible to display it effectively while minimizing the impact on the observation of the endoscopic image. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is an external view of the endoscope system. [Diagram 2] FIG. 2 is a block diagram showing a configuration of a main part of the endoscope system. [Diagram 3] FIG. 3 is a functional block diagram of the medical image processing device in the image processing unit. [Figure 4] FIG. 4 is a diagram showing main information recorded in the recording section. [Diagram 5]FIG. 5 is a diagram showing a configuration of a neural network. [Figure 6] FIG. 6 is a schematic diagram showing a configuration example of the intermediate layer. [Figure 7] FIG. 7 is a flow chart illustrating a medical image processing method. [Figure 8] FIG. 8 is a diagram showing an example of an observation state display. [Figure 9] FIG. 9 is a diagram showing a first modified example of the observation state display. [Figure 10] FIG. 10 is a diagram showing a second modified example of the observation state display. [Figure 11] FIG. 11 is a diagram showing a third modified example of the observation state display. [Figure 12] FIG. 12 is a diagram showing a fourth modified example of the observation state display. [Figure 13] FIG. 13 is a diagram showing a fifth modified example of the observation state display. [Figure 14] FIG. 14 is a diagram showing a sixth modified example of the observation state display. [Figure 15] FIG. 15 is a diagram illustrating an example of a monitor having a main display area and a sub-display area. [Figure 16] FIG. 16 is a diagram for explaining an example in which different observation state displays are displayed in the main display area and the sub-display area. [Figure 17] FIG. 17 is a diagram for explaining an example in which different observation state displays are displayed in the main display area and the sub-display area. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of a medical image processing apparatus, a medical image processing method, and a program according to the present invention will now be described with reference to the accompanying drawings.

[0027] <Endoscope system configuration> Fig. 1 is an external view of an endoscope system 10, and Fig. 2 is a block diagram showing the configuration of the main parts of the endoscope system 10. As shown in Figs. 1 and 2, the endoscope system 10 is composed of an endoscope scope 100, an endoscope processor device 200, a light source device 300, and a monitor 400. The endoscope processor device 200 is equipped with the medical image processing device of the present invention.

[0028] <Configuration of an endoscope> The endoscope 100 includes a handheld operation unit 102 and an insertion section 104 connected to the handheld operation unit 102. The operator (user) holds and operates the handheld operation unit 102, and inserts the insertion section 104 into the body of a subject (living body) for observation. The handheld operation unit 102 is provided with an air / water supply button 141, a suction button 142, function buttons 143 to which various functions are assigned, and a shooting button 144 for receiving shooting instruction operations (still image, moving image). The insertion section 104 is composed of a flexible section 112, a bending section 114, and a tip rigid section 116, in that order from the handheld operation unit 102 side. That is, the bending section 114 is connected to the base end side of the tip rigid section 116, and the flexible section 112 is connected to the base end side of the bending section 114. The handheld operation unit 102 is connected to the base end side of the insertion section 104. A user can bend the bending section 114 by operating the hand operation section 102 to change the orientation of the tip rigid section 116 up, down, left, or right. The tip rigid section 116 is provided with an imaging optical system 130, an illumination section 123, a forceps port 126, etc. (see FIGS. 1 and 2).

[0029] During observation and treatment, white light and / or narrowband light (one or more of red narrowband light, green narrowband light, blue narrowband light, and violet narrowband light) can be irradiated from the illumination lenses 123A and 123B of the illumination unit 123 by operating the operation unit 208 (see FIG. 2). In addition, cleaning water can be discharged from a water supply nozzle (not shown) by operating the air / water supply button 141 to clean the photographing lens 132 of the photographing optical system 130 and the illumination lenses 123A and 123B. A conduit (not shown) is connected to the forceps port 126 opening at the tip hard portion 116, and a treatment tool (not shown) for tumor removal or the like is inserted into this conduit, and the treatment tool can be advanced and retreated as appropriate to perform necessary treatment on the subject.

[0030] As shown in FIG. 1 and FIG. 2, a photographing lens 132 is disposed on the tip end surface 116A of the tip rigid portion 116. A CMOS (Complementary Metal-Oxide Semiconductor) type image sensor 134, a driving circuit 136, and an AFE 138 (AFE: Analog Front End) are disposed behind the photographing lens 132, and an image signal is output by these elements. The image sensor 134 is a color image sensor, and has a plurality of pixels constituted by a plurality of light receiving elements arranged in a matrix (two-dimensional array) in a specific pattern array (Bayer array, X-Trans (registered trademark) array, honeycomb array, etc.). Each pixel of the image sensor 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (photodiode, etc.). The photographing optical system 130 can generate a color image from pixel signals of three colors, red, green, and blue, or can generate an image from pixel signals of any one or two colors of red, green, and blue. The image sensor 134 may be of a CCD (Charge Coupled Device) type. Each pixel of the image sensor 134 may further include a purple color filter corresponding to the purple light source 310V and / or an infrared filter corresponding to the infrared light source.

[0031] An optical image of the subject is formed on the light receiving surface (imaging surface) of the image sensor 134 by the photographing lens 132, converted into an electric signal, and output to the endoscope processor device 200 via a signal cable (not shown) and converted into a video signal. As a result, an endoscopic image (medical image) of the subject is displayed on the screen of the monitor 400 connected to the endoscope processor device 200.

[0032] Furthermore, on the tip end surface 116A of the tip hard portion 116, illumination lenses 123A and 123B of the illumination portion 123 are provided adjacent to the photographing lens 132. An exit end of a light guide 170 (described later) is disposed behind the illumination lenses 123A and 123B. This light guide 170 is inserted through the insertion portion 104, the handheld operation portion 102, and the universal cable 106, and an entrance end of the light guide 170 is disposed in the light guide connector 108.

[0033] A user can capture time-series endoscopic images of the inside of a living body at a determined frame rate while inserting or removing the endoscope 100 having the above-described configuration into or from the living body.

[0034] <Light source configuration> 2, the light source device 300 is composed of a light source 310 for illumination, an aperture 330, a condenser lens 340, a light source control unit 350, etc., and causes observation light to enter the light guide 170. The light source 310 is equipped with a red light source 310R, a green light source 310G, a blue light source 310B, and a purple light source 310V that irradiate narrowband light of red, green, blue, and purple, respectively, and can irradiate narrowband light of red, green, blue, and purple. The illuminance of the observation light from the light source 310 is controlled by the light source control unit 350, and the illuminance of the observation light can be changed (increased or decreased) and the illumination can be stopped as necessary.

[0035] The light source 310 can emit narrowband light of red, green, blue, and purple in any combination. For example, narrowband light of red, green, blue, and purple can be emitted simultaneously to irradiate white light (normal light) as the observation light, or narrowband light (special light) can be irradiated by emitting one or two of them. The light source 310 may further include an infrared light source that irradiates infrared light (an example of narrowband light). In addition, white light or narrowband light may be irradiated as the observation light by using a light source that irradiates white light and a filter that transmits the white light and each narrowband light.

[0036] <Light source wavelength band> The light source 310 may be a light source that generates light in a white band, or light in a plurality of wavelength bands as white band light, or may be a light source that generates light in a specific wavelength band narrower than the white wavelength band. The specific wavelength band may be a blue band or a green band in the visible range, or a red band in the visible range. When the specific wavelength band is a blue band or a green band in the visible range, it may include a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and may have a peak wavelength within the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. When the specific wavelength band is a red band in the visible range, it may include a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and the light in the specific wavelength band may have a peak wavelength within the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm.

[0037] The light in the specific wavelength band may include a wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different, and may have a peak wavelength in the wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different. In this case, the specific wavelength band may include a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm to 750 nm, and may have a peak wavelength in a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm to 750 nm.

[0038] Furthermore, the light generated by light source 310 may include a wavelength band of 790 nm to 820 nm, or 905 nm to 970 nm, and may have a peak wavelength in the wavelength band of 790 nm to 820 nm, or 905 nm to 970 nm.

[0039] The light source 310 may also include a light source that irradiates excitation light with a peak of 390 nm or more and 470 nm or less. In this case, an endoscopic image having information on the fluorescence emitted by a fluorescent substance in the subject (living body) can be obtained. When obtaining a fluorescent image, a dye agent for the fluoroscopy (fluorescein, acridine orange, etc.) may be used.

[0040] The type of light source (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), etc.), wavelength, presence or absence of a filter, etc. of the light source 310 are preferably configured according to the type and location of the subject, the purpose of observation, etc., and it is also preferable to combine and / or switch the wavelength of the observation light according to the type and location of the subject, the purpose of observation, etc., during observation. When switching the wavelength, the wavelength of the irradiated light may be switched, for example, by rotating a disk-shaped filter (rotary color filter) that is placed in front of the light source and has a filter that transmits or blocks light of a specific wavelength.

[0041] In addition, the imaging element used when carrying out the present invention is not limited to a color imaging element in which a color filter is arranged for each pixel, such as the imaging element 134, but may be a monochrome imaging element. When a monochrome imaging element is used, the wavelength of the observation light can be switched sequentially to capture images in a frame sequential (color sequential) manner. For example, the wavelength of the emitted observation light may be switched sequentially between (purple, blue, green, red), or broadband light (white light) may be irradiated and the wavelength of the emitted observation light may be switched by a rotary color filter (red, green, blue, purple, etc.). Also, one or more narrowband lights (green, blue, purple, etc.) may be irradiated and the wavelength of the emitted observation light may be switched by a rotary color filter (green, blue, purple, etc.). The narrowband light may be infrared light of two or more different wavelengths (first narrowband light, second narrowband light).

[0042] By connecting the light guide connector 108 (see Figures 1 and 2) to the light source device 300, the observation light irradiated from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and is irradiated onto the observation range from the illumination lenses 123A and 123B.

[0043] <Configuration of the endoscope processor device> The configuration of the endoscope processor device 200 will be described with reference to Fig. 2. The endoscope processor device 200 receives an image signal output from the endoscope scope 100 via an image input controller 202, performs necessary image processing in an image processing unit 204, and outputs the signal via a video output unit 206. As a result, an endoscopic image is displayed on a monitor 400. These processes are performed under the control of a CPU 210 (CPU: Central Processing Unit). The CPU 210 functions as a processor of the medical image processing device. The communication control unit 205 controls communications regarding the acquisition of medical images between an in-hospital system (HIS: Hospital Information System) and an in-hospital LAN (Local Area Network) (not shown), and / or an external system or network.

[0044] <Image processing unit functions> The image processing unit 204 can calculate the feature amount of the endoscopic image, emphasize or reduce components of a specific frequency band, and emphasize or make inconspicuous a specific target (such as a region of interest or blood vessels at a desired depth). The image processing unit 204 may include a special light image acquisition unit (not shown) that acquires a special light image having information of a specific wavelength band based on a normal light image acquired by irradiating light of a white band or light of a plurality of wavelength bands as light of a white band. In this case, a signal of a specific wavelength band can be acquired by calculation based on color information of RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) included in the normal light image. The image processing unit 204 may also include a feature image generation unit (not shown) that generates a feature image by calculation based on at least one of a normal light image acquired by irradiating light of a white band or light of a plurality of wavelength bands as light of a white band and a special light image acquired by irradiating light of a specific wavelength band, and acquire and display the feature image as an endoscopic image. The above-mentioned processing is performed under the control of the CPU 210.

[0045] Furthermore, the image processing unit 204 has the functions of a medical image processing apparatus as described below.

[0046] 3 is a functional block diagram of the medical image processing apparatus in the image processing unit 204. The image processing unit 204 includes a medical image acquisition unit 220, an observation state determination unit 222, and a display control unit 224.

[0047] <Realization of functions using various processors> The functions of each part of the image processing unit 204 described above can be realized using various processors and recording media. The various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. The various processors described above also include a GPU (Graphics Processing Unit), which is a processor specialized for image processing, and a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacture. When learning and recognizing images as in the present invention, a configuration using a GPU is effective. Furthermore, the various processors described above also include dedicated electric circuits, which are processors having a circuit configuration designed specifically for executing specific processing, such as an ASIC (Application Specific Integrated Circuit).

[0048] The functions of each part may be realized by one processor, or may be realized by multiple processors of the same type or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functions may be realized by one processor. As an example of configuring multiple functions by one processor, first, as represented by a computer, there is a form in which one processor is configured by a combination of one or more CPUs and software, and this processor realizes multiple functions. Second, as represented by a system on chip (SoC), there is a form in which a processor that realizes the functions of the entire system by one IC (Integrated Circuit) chip is used. In this way, various functions are configured by using one or more of the above-mentioned various processors as a hardware structure. Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) that combines circuit elements such as semiconductor elements. These electric circuits may be electric circuits that realize the above-mentioned functions using logical sum, logical product, logical negation, exclusive logical sum, and logical operations combining these.

[0049] When the above-mentioned processor or electric circuit executes software (program), a code readable by a computer (for example, various processors and electric circuits constituting the image processing unit 204, and / or a combination thereof) of the software to be executed is stored in a non-transient recording medium such as a ROM 211 (Read Only Memory), and the computer refers to the software. The software stored in the non-transient recording medium includes a program for executing the medical image processing method of the medical image processing device according to the present invention and data used during execution. The code may be recorded in a non-transient recording medium such as various magneto-optical recording devices and semiconductor memories, instead of the ROM 211. When processing using the software, for example, a RAM 212 (RAM: Random Access Memory) is used as a temporary storage area, and data stored in, for example, an EEPROM (Electronically Erasable and Programmable Read Only Memory) not shown can also be referenced. The recording unit 207 may be used as the "non-transient recording medium".

[0050] Further, the ROM 211 (Read Only Memory) is a non-volatile storage element (non-temporary recording medium) that stores computer-readable code of a program that causes the CPU 210 and / or the image processing unit 204 to execute various image processing methods. The RAM 212 (Random Access Memory) is a storage element for temporary storage during various processes, and can also be used as a buffer when acquiring images. The audio processing unit 209 outputs voice and sound from a speaker 209A under the control of the CPU 210.

[0051] The operation unit 208 can be configured with devices such as a keyboard and a mouse (not shown), and a user can issue instructions to execute processing and specify conditions required for execution via the operation unit 208 .

[0052] <Information recorded in the recorder> 4 is a diagram showing main information recorded in the recording unit 207. The recording unit (memory) 207 records a medical image (endoscopic image) 260, a determination result 262 of the observation state determination unit 222, and the like. Also, information on a series of small regions to be observed in an examination performed using the endoscope system 10 is recorded. Here, the small regions of the subject are, for example, each part of an organ. Specifically, when an examination is performed to observe all parts of the stomach, the small regions are the cardia, fundus, angle of the stomach, body of the stomach (upper, middle, lower), vestibule, anterior wall, posterior wall, greater curvature, and lesser curvature.

[0053] <Neural network recognition section> The observation state determination unit 222 in the image processing unit 204 described above includes a recognizer that is configured using a trained model such as a neural network (a model trained using an image set consisting of images of a living body) and that can recognize small areas of a subject. The observation state determination unit 222 then determines whether or not the observation of the small area is complete for each small area based on the position of the small area recognized by the recognizer, the number of endoscopic images in which the small area is recognized, and the like. The following describes the configuration of the recognizer included in the observation state determination unit 222 when a CNN (Convolutional Neural Network) is used as the neural network.

[0054] <Example of recognizer configuration> FIG. 5 is a diagram showing the configuration of the CNN 232 (neural network). In the example shown in part (a) of FIG. 5, the CNN 232 has an input layer 232A, an intermediate layer 232B, and an output layer 232C. The input layer 232A inputs an endoscopic image acquired by the medical image acquisition unit 220 and outputs a feature amount. The intermediate layer 232B includes a convolution layer 234 and a pooling layer 235, and inputs the feature amount output by the input layer 232A to calculate other feature amounts. These layers have a structure in which a plurality of "nodes" are connected by "edges," and weighting coefficients applied to the input image are associated with the nodes and edges and stored in a weighting coefficient storage unit (not shown). The value of the weighting coefficient changes as learning progresses.

[0055] <Processing in the middle layer> The intermediate layer 232B calculates features by convolution and pooling. The convolution performed in the convolution layer 234 is a process of acquiring a feature map by convolution using a filter, and plays a role in extracting features such as edge extraction from an image. The convolution using this filter generates a "feature map" of one channel (one sheet) for one filter. When downscaling is performed by convolution, the size of the "feature map" becomes smaller as convolution is performed in each layer. The pooling process performed in the pooling layer 235 is a process of reducing (or enlarging) the feature map output by the convolution to create a new feature map, and plays a role in providing robustness so that the extracted features are not affected by translation or the like. The intermediate layer 232B can be configured with one or more layers that perform these processes. Note that the CNN 232 may be configured without the pooling layer 235.

[0056] The CNN 232 may include a fully connected layer 236 as shown in the example of part (b) of Fig. 5. The layer configuration of the CNN 232 is not limited to the case where the convolutional layer 234 and the pooling layer 235 are repeated one by one, but may include any layer (for example, the convolutional layer 234) in a series of multiple layers.

[0057] 6 is a schematic diagram showing a configuration example of the intermediate layer 232B of the CNN 232 shown in FIG. 5. In the first (first) convolutional layer of the intermediate layer 232B, an image set consisting of a plurality of endoscopic images and a filter F 1 The image set is composed of N images (N channels) with an image size of H vertically and W horizontally. When normal light images are input, the images that make up the image set are images of three channels: R (red), G (green), and B (blue). The filter F that is convolved with this image set is 1 Since the image set has N channels (N images), for example, in the case of a filter of size 5 (5 × 5), the filter size will be 5 × 5 × N. This filter F 1A single filter F 1 A one-channel (one-image) “feature map” is generated for each filter F used in the second convolutional layer. 2 For example, for a filter of size 3 (3×3), the filter size will be 3×3×M.

[0058] Similar to the first convolutional layer, the second to nth convolutional layers use a filter F 2 ~F n The size of the "feature map" in the nth convolutional layer is smaller than that of the "feature map" in the second convolutional layer because it has been downscaled by the previous convolutional layers or pooling layers.

[0059] Among the layers in intermediate layer 232B, low-order feature extraction (such as edge extraction) is performed in the convolutional layers closer to the input side, and higher-order feature extraction (extraction of features related to the shape, structure, etc. of the recognition target) is performed as the layers approach the output side.

[0060] The intermediate layer 232B may include a layer that performs batch normalization in addition to the convolution layer 234 and the pooling layer 235. The batch normalization process is a process that normalizes the distribution of data in units of mini-batches when performing learning, and plays a role in speeding up learning, reducing dependency on initial values, suppressing overlearning, etc.

[0061] The output layer 232C outputs the feature amount calculated by the intermediate layer 232B in a format suitable for recognition. The output layer 232C may include a fully connected layer.

[0062] <Each step of the medical image processing method> Next, a medical image processing method using the medical image processing apparatus will be described.

[0063] Fig. 7 is a flowchart showing a medical image processing method. Each step will be described below with reference to Fig. 6. Note that the following describes a case where small areas, Area 1, Area 2, and Area 3, are observed in area units when examining an organ A, which is a subject.

[0064] (Medical image acquisition step) The medical image acquisition unit 220 sequentially acquires a plurality of medical images of the organ A in chronological order (step S10). Note that the recording unit 207 records that observations of areas 1, 2, and 3 of the organ A are to be performed, and that the observations of areas 1, 2, and 3 are not yet completed as the initial state.

[0065] (Observation state determination step) The observation state determination unit 222 determines the observation states of areas 1, 2, and 3 of organ A based on the acquired medical images (step S11). The observation state determination unit 222 recognizes area 1, area 2, or area 3 in the medical images. Then, based on the recognition results, the observation state determination unit 222 determines the observation states of areas 1, 2, and 3. For example, when area 1 is recognized as the center of 10 chronologically consecutive medical images, the observation state determination unit 222 determines that observation of area 1 has been completed.

[0066] (Recording Step) The recording unit 207 records the result of the judgment made by the observation state judgment unit 222 (step S12). At the start of the inspection (initial state), areas 1, 2, and 3 are recorded as observation incomplete, but when the observation state judgment unit 222 judges that the observation of each area is completed, the record is updated to observation completed.

[0067] (Display Step) The display control unit 224 judges whether or not a change has occurred in the observation state of the subject (step S13). Then, when the display control unit 224 judges that a change has occurred in the observation state of the subject, it displays the observation state display 501 of the subject on the monitor 400 (step S14). Here, the time when a change has occurred in the observation state of the subject is when a change has been recorded in the recording unit 207 in the observation state of a plurality of small area units in which observation is to be performed and recorded in the recording unit 207. For example, the observation state display 501 is when the observation state of area 1, area 2, and area 3 has been recorded in the recording unit 207 as incomplete, but the observation state determination unit 222 has determined that observation has been completed for area 1, and the observation state of area 1 has changed to observation completion in the recording unit 207. The observation state display 501 of the subject is a display that notifies the user of the observation state of the small area units constituting the subject to be observed. By looking at the observation state display 501, the user can check whether the small areas to be observed have been comprehensively observed.

[0068] FIG. 8 is a diagram showing an example of an observation state display 501 of an organ A displayed on the monitor 400.

[0069] In the case shown in Fig. 8, an endoscopic image 503 is displayed on the entire surface of the monitor 400. Then, at the point in time when area 3 is updated from incomplete observation to complete observation in the record of the recorder 207, the display controller 224 causes the monitor 400 to display an observation status display 501 superimposed on the endoscopic image 503. The observation status display 501 is a list display having text information indicating the areas where observation has been completed and the areas where observation has not been completed. In the observation status display 501, the areas where observation has been completed (referred to as area in the figure) 1 and area 3 are listed under "Complete", and the area where observation has not been completed is listed under "Not yet".

[0070] Returning to FIG. 7, the display control unit 224 then continues to display the observation state display 501 until a predetermined display time has elapsed. The display time can be set appropriately by the user. It is preferable that the observation state display 501 is hidden and the endoscopic image 503 can be observed once the user is able to confirm the observation state, and therefore it is preferable that the display time is set based on the time during which the user can confirm the observation state. For example, the display time can be set to 10 seconds or 30 seconds. Thereafter, the display control unit 224 hides the observation state display 501 after the predetermined time has elapsed (step S15).

[0071] Thereafter, the medical image acquisition section 220 determines whether or not the observation of all small area units has been completed (step S16), and since the observation of area 2 has not yet been completed, further medical images are acquired (step S10).

[0072] As described above, in the present invention, when observing Area 1, Area 2, and Area 3, which are small regions of Organ A, the observation state display 501 is performed at the point in time when a change occurs in the observation state of Organ A. As a result, by performing the observation state display 501 when necessary and refraining from displaying it in other cases, it is possible to effectively display the image while suppressing the influence on the observation of the endoscopic image.

[0073] <Modification of Observation State Display> 8, a case has been described in which an observation status display 501 is displayed that displays areas that have been observed and areas that have not been observed using text information, but the example of the observation status display 501 is not limited to this. The display form of the observation status display 501 is not particularly limited as long as it can notify the user of the observation status of the subject to be observed using text and / or graphics. A specific example of the observation status display 501 will be described below.

[0074] 9 is a diagram showing modified example 1 of the observation status display 501. In this example, the observation status display 501 is text information. In this example, the observation status display 501 displays only areas for which observation has not been completed. Specifically, when observation of area 2 has not been completed, the text information "Area 2" is written below "Not yet".

[0075] In this way, by displaying the observation status display 501 having only uncompleted small areas on the monitor 400, the user can clearly recognize the uncompleted small areas, and comprehensive observation can be realized. Note that, in the example of Fig. 9, an example has been described in which uncompleted small areas are displayed on the observation status display 501, but completed observation areas may also be displayed on the observation status display 501. In this case, the user can clearly recognize the completed observation areas.

[0076] FIG. 10 is a diagram showing a second modified example of the observation status display 501. This example is an observation status display 501 of text information. In the observation status display 501 of this example, all of the small areas (areas 1 to 5) constituting the subject to be observed are displayed as a list. In the observation status display 501, areas where observation has been completed and areas where observation has not been completed are displayed with different text colors. Specifically, in the observation status display 501, areas 3 and 5 are not yet observed, so they are displayed with text of the same color, and areas 1, 2, and 4 are not yet observed, so they are displayed with text of the same color. In this way, all of the small areas are displayed in a list, and information on whether observation has been completed or not is added to text on a small area-by-small area basis, so that the user can comprehensively recognize the areas where observation has been completed or not completed.

[0077] FIG. 11 is a diagram showing a third modified example of the observation status display 501. This example is an observation status display 501 using text information. In the observation status display 501 of this example, all of the small areas (areas 1 to 5) constituting the subject to be observed are displayed as a list. In the observation status display 501, an "o" or an "x" is displayed next to the letters of the areas for which observation has been completed and for which observation has not been completed. Specifically, in the observation status display 501, since areas 3 and 5 are not yet observed, an "x" is added, and since areas 1, 2, and 4 are observed, an "o" is added. In this way, all of the small areas are displayed in a list, and information on whether observation has been completed or not is displayed next to the letters for each small area, so that the user can comprehensively recognize the areas for which observation has been completed or not completed.

[0078] FIG. 12 is a diagram showing a fourth modified example of the observation status display 501. In this example, the observation status display 501 is based on text information. In this example, the observation status display 501 shows the percentage of incomplete observation as the observation status display 501. The percentage of incomplete observation here refers to the percentage of incomplete observation small areas among the multiple small areas to be observed. This percentage may be calculated based on the number of small areas or the area of ​​the small areas. In the example shown in FIG. 12, the percentage is shown by text, but it may also be shown using a graphic such as a bar display indicating the percentage. In this way, by showing the percentage of incomplete observation, the user can clearly know the areas where observation is incomplete.

[0079] FIG. 13 is a diagram showing a modified example 5 of the observation state display 501. This example is an observation state display 501 by a subject model M which is a schematic representation of a subject. The subject model M is a schematic diagram showing a stomach which is a subject to be observed. In the subject model M, different colors are assigned to the observation completed area 513 and the observation incomplete area 511 in small area units. In this way, by assigning information on the observation completion and the observation incomplete to the subject model M, the user can recognize the position of the observation completed small area and the position of the observation incomplete small area in the subject. Note that, in the example shown in FIG. 13, the information on the observation completion and the observation incomplete is assigned by changing the color of the area of ​​the subject model M, but is not limited to this example. For example, information on the observation completion and the observation incomplete may be assigned to the subject model M by changing the color density.

[0080] FIG. 14 is a diagram showing a sixth modified example of the observation status display 501. This example is an observation status display 501 using a subject model M that is a schematic representation of a subject. In this example, only small areas that have been observed are shown in the subject model M. Then, when the observation of all small areas is completed, the entire subject model M (stomach) is displayed. In this way, by adding information on whether observation has been completed or not to the subject model M, the user can recognize the positions of small areas in the subject that have been observed and the positions of small areas in which observation has not been completed. Note that in FIGS. 13 and 14, a cutaway view of the stomach, which is the subject, is used as the subject model M, but this is not limited to this. For example, the subject model M may be a development view of the stomach, which is the subject.

[0081] <Monitor Variations> In the above description, an example of the monitor 400 having only a main display area has been described, but the present invention is not limited to this. Modifications of the monitor will be described below.

[0082] FIG. 15 is a diagram illustrating an example of a monitor 401 having a main display area and a sub-display area.

[0083] As shown in FIG. 15, the monitor 401 has a main display area 402 and a sub-display area 404. In the main display area 402, an endoscopic image 503 captured by the endoscope 100 is displayed in real time. The sub-display area 404 is set smaller than the main display area 402, and displays information such as the imaging conditions, date, and patient information. Note that in FIG. 15, the illustration of the information displayed in the sub-display area 404 is omitted. In the monitor 401, the observation state display 501 is displayed in the main display area 402 and the sub-display area 404. In the main display area 402, the observation state display 501 is displayed when a change occurs in the observation state of the subject as described above, and the display ends when the display time has elapsed. Note that FIG. 15 illustrates a case where the observation state display 501 in the main display area 402 is not displayed.

[0084] The observation state display 501B is always displayed in the sub-display area 404. Since the display area of ​​the sub-display area 404 is small, the observation state display 501B is also smaller than the observation state display 501A displayed in the main display area 402, but it serves as an aid for comprehensive observation. The main display area 402 and the sub-display area 404 may display the same observation state display, or different observation state displays.

[0085] 16 and 17 are diagrams for explaining an example in which different observation state displays are displayed in the main display area and the sub-display area.

[0086] 16, a detailed observation state display 501A in which small regions are displayed as a list is displayed in the main display area 402. In addition, in the sub-display area 404, an observation state display 501B of the subject model M is displayed.

[0087] 17, a detailed observation status display 501A in which small areas are displayed in a list is displayed in the main display area 402. In addition, an observation status display 501B indicating the percentage of incomplete observations is displayed in the sub-display area 404.

[0088] As shown in FIG. 16 and FIG. 17, in the main display area 402, an observation status display 501A is displayed in a list of small areas, which allows the user to grasp the observation status of each small area in detail, when the observation status of the subject changes. This allows the user to grasp the detailed observation status of the subject when the number of small areas that have been observed increases. In addition, in the sub-display area 404, an observation status display 501B showing the subject model M and the ratio of incomplete observation is always displayed. This allows the user to roughly grasp the observation status of the subject even when the observation status display 501A is not displayed in the main display area 402. In the above example, the monitor 401 having the main display area (first display area) 402 and the sub-display area (second display area) 404 has been described, but the observation status display 501 may be displayed on a monitor having, for example, a third display area. In addition, the observation status display 501 may be displayed on multiple monitors.

[0089] <Other examples> In the above description, a mode in which the observation status display 501 is displayed for a predetermined display time period has been described, but the mode in which the observation status display 501 once displayed is made non-displayable is not limited to this.

[0090] For example, the display control unit 224 may hide the observation state display 501 based on a command input by the user via the local operation unit 102 (user operation reception unit) (see FIG. 1). Furthermore, the display control unit 224 may display the observation state display 501, which has been hidden once, again based on a command input by the user via the local operation unit 102 (user operation reception unit). In this way, the user can check the observation state display at a desired timing by controlling the display and non-display of the observation state display 501 via the local operation unit 102.

[0091] Although the present invention has been described above with reference to examples, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0092] 10: Endoscope system 100: Endoscope 102: Handheld operation unit 104: Insertion section 106: Universal cable 108: Light guide connector 112: Soft part 114: Curved section 116: Hard tip 116A:Tip side end surface 123: Lighting Department 123A: Lighting lens 123B: Lighting lenses 126: Forceps mouth 130: Photographing optical system 132: Shooting lens 134: Image sensor 136: Drive circuit 141: Air / water supply button 142: Suction button 143: Function button 144: Shooting button 170: Light guide 200: Endoscope processor device 202: Image input controller 204: Image processing unit 205: Communication control section 206: Video output section 207: Recording Department 208 :Operation unit 209: Audio processing unit 209A: Speaker 210: CPU 211 :ROM 212:RAM 220: Medical image acquisition unit 222: Observation state determination unit 224: Display control unit 232A: Input layer 232B: Middle layer 232C: Output layer 234: Convolutional layer 235: Pooling layer 236 :Fully connected layer 300: Light source device 310 :Light source 310B: Blue light source 310G: Green light source 310R: Red light source 310V: Violet light source 330: Aperture 340: Condenser lens 350: Light source control unit 400: Monitor

Claims

1. A medical imaging device comprising a processor and a memory, The processor, Acquire multiple medical images in time series, determining an observation state of a small area of ​​the subject based on the medical image; Recording the result of the determination in the memory; When a change occurs in the observation state of the object, an observation state display of the object based on the result of the determination recorded in the memory is displayed on a monitor.

1. A medical image processing device, comprising: The processor is configured to hide the observation state display on the monitor after a predetermined time has elapsed.

2. A medical imaging device comprising a processor and a memory, The processor, Acquire multiple medical images in time series, determining an observation state of a small area of ​​the subject based on the medical image; Recording the result of the determination in the memory; When a change occurs in the observation state of the object, an observation state display of the object based on the result of the determination recorded in the memory is displayed on a monitor.

1. A medical image processing device, comprising: A user operation receiving unit is further provided, The processor displays or hides the observation state display based on a command from the user operation reception unit.

3. A medical imaging device comprising a processor and a memory, The processor, Acquire multiple medical images in time series, determining an observation state of a small area of ​​the subject based on the medical image; Recording the result of the determination in the memory; When a change occurs in the observation state of the object, an observation state display of the object based on the result of the determination recorded in the memory is displayed on a monitor.

1. A medical image processing device, comprising: The processor displays the medical image on the monitor and displays the observation state display superimposed on the medical image.

4. A medical imaging device comprising a processor and a memory, The processor, Acquire multiple medical images in time series, determining an observation state of a small area of ​​the subject based on the medical image; Recording the result of the determination in the memory; When a change occurs in the observation state of the object, an observation state display of the object based on the result of the determination recorded in the memory is displayed on a monitor.

1. A medical image processing device, comprising: the processor causes the monitor to have a first display area and a second display area smaller than the first display area, and causes the first display area and the second display area to display the observation state display in different manners; A medical image processing apparatus that displays the medical image in the first display area and displays the observation status display in the second display area.

5. The medical image processing apparatus according to claim 2 , wherein the processor makes the observation state display displayed on the monitor non-display after a predetermined time has elapsed.

6. A user operation receiving unit is further provided, The medical image processing apparatus according to claim 1 , wherein the processor displays or hides the observation state display based on a command from the user operation reception unit.

7. The medical image processing apparatus according to claim 1 , wherein the processor causes the monitor to display the medical image and the observation state display to be superimposed on the medical image.

8. A medical image processing device according to any one of claims 1 to 3, wherein the processor is a monitor having a first display area and a second display area smaller than the first display area, and the observation status display is displayed in different manners in the first display area and the second display area.

9. The medical image processing device according to any one of claims 1 to 8, wherein the processor, when determining the observation state of the small region, determines that observation is completed if observation of the small region is completed, and determines that observation is incomplete if observation is incomplete.

10. The medical image processing apparatus according to claim 1 , wherein the processor causes the monitor to display the observation state indication by text information.

11. The medical image processing apparatus according to claim 10 , wherein the processor adds information regarding whether observation of the small area of ​​the subject is complete or not to the character information and displays the character information as the observation status display.

12. The medical image processing apparatus according to claim 1 , wherein the processor displays the observation state display using a subject model that typically represents the subject.

13. The medical image processing apparatus according to claim 12 , wherein the processor adds information regarding whether observation of the unit of the small region of the subject is completed or not to the subject model, and displays the information as the observation status display.

14. The medical image processing apparatus according to claim 9 , wherein the processor displays, as the observation state display, an indication of only whether observation of the small area unit of the subject has been completed or not.

15. The medical image processing apparatus according to claim 4 or 8, wherein the processor constantly displays the observation state display in the second display area.

16. The medical image processing apparatus according to claim 4 , 8 or 15 , wherein the processor is a monitor having a third display area different from the first display area and the second display area, and causes the medical image to be displayed in the third display area.

17. A medical image processing method for a medical image processing device having a processor and a memory, comprising: The processor, A medical image acquisition step of acquiring a plurality of medical images in time series; an observation state determination step of determining an observation state of a small region of a subject based on the medical image; a recording step of recording the result of the determination in the memory; a display step of displaying, on a monitor, an observation state display of the object based on the result of the judgment recorded in the memory, when a change occurs in the observation state of the object; A medical image processing method comprising the steps of: The medical image processing method further includes a step executed by the processor of hiding the observation status display displayed on the monitor after a predetermined time has elapsed.

18. A medical image processing method for a medical image processing device having a processor and a memory, comprising: The processor, A medical image acquisition step of acquiring a plurality of medical images in time series; an observation state determination step of determining an observation state of a small region of a subject based on the medical image; a recording step of recording the result of the determination in the memory; a display step of displaying, on a monitor, an observation state display of the object based on the result of the judgment recorded in the memory, when a change occurs in the observation state of the object; A medical image processing method comprising the steps of: The medical image processing method, wherein in the display step, the processor displays or hides the observation state display based on an instruction from a user operation receiving unit.

19. A medical image processing method for a medical image processing device having a processor and a memory, comprising: The processor, A medical image acquisition step of acquiring a plurality of medical images in time series; an observation state determination step of determining an observation state of a small region of a subject based on the medical image; a recording step of recording the result of the determination in the memory; a display step of displaying, on a monitor, an observation state display of the object based on the result of the judgment recorded in the memory, when a change occurs in the observation state of the object; A medical image processing method comprising the steps of: A medical image processing method, wherein in the display step, the processor causes the monitor to display the medical image and the observation state display to be superimposed on the medical image.

20. A medical image processing method for a medical image processing device having a processor and a memory, comprising: The processor, A medical image acquisition step of acquiring a plurality of medical images in time series; an observation state determination step of determining an observation state of a small region of a subject based on the medical image; a recording step of recording the result of the determination in the memory; a display step of displaying, on a monitor, an observation state display of the object based on the result of the judgment recorded in the memory, when a change occurs in the observation state of the object; A medical image processing method comprising the steps of: In the display step, the processor the monitor having a first display area and a second display area smaller than the first display area, the first display area and the second display area being caused to display the observation state display in different manners; A medical image processing method comprising: displaying the medical image in the first display area; and displaying the observation status indication in the second display area.

21. A program for causing a medical image processing device having a processor and a memory to execute a medical image processing method, The processor, A medical image acquisition step of acquiring a plurality of medical images in time series; an observation state determination step of determining an observation state of a small region of a subject based on the medical image; a recording step of recording the result of the determination in the memory; a display step of displaying, on a monitor, an observation state display of the object based on the result of the judgment recorded in the memory, when a change occurs in the observation state of the object; A program for executing a medical image processing method, comprising: A program for causing the processor to execute a medical image processing method, the program further comprising a step of hiding the observation status display displayed on the monitor after a predetermined time has elapsed.

22. A program for causing a medical image processing device having a processor and a memory to execute a medical image processing method, The processor, A medical image acquisition step of acquiring a plurality of medical images in time series; an observation state determination step of determining an observation state of a small region of a subject based on the medical image; a recording step of recording the result of the determination in the memory; a display step of displaying, on a monitor, an observation state display of the object based on the result of the judgment recorded in the memory, when a change occurs in the observation state of the object; A program for executing a medical image processing method, comprising: The processor is a program for causing the processor to perform a medical image processing method in which, in the display step, the observation state display is displayed or hidden based on an instruction from a user operation receiving unit.

23. A program for causing a medical image processing device having a processor and a memory to execute a medical image processing method, The processor, A medical image acquisition step of acquiring a plurality of medical images in time series; an observation state determination step of determining an observation state of a small region of a subject based on the medical image; a recording step of recording the result of the determination in the memory; a display step of displaying, on a monitor, an observation state display of the object based on the result of the judgment recorded in the memory, when a change occurs in the observation state of the object; A program for executing a medical image processing method, comprising: The program causes the processor to execute a medical image processing method, in which the processor causes the monitor to display the medical image and the observation state display to be superimposed on the medical image in the display step.

24. A program for causing a medical image processing device having a processor and a memory to execute a medical image processing method, The processor, A medical image acquisition step of acquiring a plurality of medical images in time series; an observation state determination step of determining an observation state of a small region of a subject based on the medical image; a recording step of recording the result of the determination in the memory; a display step of displaying, on a monitor, an observation state display of the object based on the result of the judgment recorded in the memory, when a change occurs in the observation state of the object; A program for executing a medical image processing method, comprising: In the display step, the processor The monitor has a first display area and a second display area smaller than the first display area, and displays the observation state display in different manners in the first display area and the second display area; A program for executing a medical image processing method, which displays the medical image in the first display area and displays the observation state display in the second display area.

25. A non-transitory computer-readable recording medium having the program according to any one of claims 21 to 24 recorded thereon.

Images