Medical image processing apparatus, medical image processing method, program, and recording medium

Abstract

Provided are a medical image processing apparatus, a medical image processing method, a program, and a recording medium capable of reducing deviation between recognition of a recognizer and recognition of a user.SOLUTION: A medical image processing device is a medical image processing device including a processor (210) and a memory (207), in which the processor (210) acquires a plurality of medical images in time series, determines an observation state of a subject in units of small regions on the basis of the medical images, records a result of the determination in the memory (207), and displays an observation state display of the subject based on the result of the determination recorded in the memory (207) on a monitor (400) when a change occurs in the observation state of the subject.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a medical image processing device, a medical image processing method, a program, and a recording medium. [Background technology]

[0002] 2. Description of the Related Art Conventionally, in examinations performed using an endoscope system, there has been a demand for comprehensive observation of an area of ​​an organ or the like to be examined.

[0003] Patent Document 1 describes a technology aimed at preventing missed images during an examination using an endoscopic system. In the technology described in Patent Document 1, a map image showing the already-photographed and unphotographed areas of the organ to be photographed is displayed on a monitor as a notification display. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-50890 A Summary of the Invention [Problem to be solved by the invention]

[0005] Typically, the monitor of an endoscope system displays an endoscopic image captured in real time during an examination in the main display area. Therefore, if a notification display such as that described in Patent Document 1 is displayed in the main display area, it will be superimposed on the endoscopic image, reducing the visibility of the endoscopic image. Furthermore, if the notification display is displayed in a sub-display area of ​​the monitor, the display area is small, reducing the visibility of the notification display. While it is conceivable to display the notification display on a sub-monitor separate from the main monitor, this presents a problem in that the user cannot focus on the endoscopic image displayed on the main monitor during an examination.

[0006] The above-mentioned Patent Document 1 does not mention a display mode that takes into consideration the visibility of the endoscopic image or the visibility of the notification display (map image).

[0007] The present invention has been made in consideration of these circumstances, and its purpose is to provide a medical image processing device, a medical image processing method, and a program that effectively display an observation status indication regarding the comprehensiveness of observation while suppressing a decrease in visibility of endoscopic images. [Means for solving the problem]

[0008] One aspect of the present invention for achieving the above-mentioned object is a medical image processing device that includes a processor and a memory, in which the processor acquires multiple medical images in chronological order, determines the observation state of each small area of ​​the subject based on the medical images, records the results of the determination in memory, and, when a change occurs in the observation state of the subject, displays on a monitor an observation state display of the subject based on the results of the determination recorded in memory.

[0009] According to this aspect, the observation state display of the subject is displayed on the monitor when a change occurs in the observation state of the subject. This allows the observation state display of the subject to be displayed at an appropriate timing, thereby enabling effective display while minimizing the impact on the observation of the endoscopic image.

[0010] Preferably, the processor makes the observation status display displayed on the monitor disappear after a predetermined time has elapsed.

[0011] Preferably, the apparatus further includes a user operation accepting section, and the processor displays or hides the observation state display based on an instruction from the user operation accepting section.

[0012] Preferably, when determining the observation state of the small area, the processor determines that observation is complete if observation of the small area is complete, and determines that observation is incomplete if observation is incomplete.

[0013] Preferably, the processor causes the monitor to display the observation status indication using text information.

[0014] Preferably, the processor adds information regarding whether observation of each small area of ​​the subject has been completed or not to text information and displays it as an observation status display.

[0015] Preferably, the processor displays the observation state display using an object model that schematically represents the object.

[0016] Preferably, the processor provides the object model with information regarding whether observation of each small region of the object has been completed or not, and displays the information as an observation status display.

[0017] Preferably, the processor displays, as an observation status display, an indication of whether observation of each small area of ​​the subject has been completed or not.

[0018] Preferably, the processor causes the monitor to display a medical image and an observation status display superimposed on the medical image.

[0019] Preferably, the processor is a monitor having a first display area and a second display area smaller than the first display area, and causes the first display area and the second display area to display the observation status display in different manners.

[0020] Preferably, the processor causes the second display area to constantly display the observation status display.

[0021] Preferably, the processor causes a monitor having a third display area different from the first display area and the second display area to display the medical image in the third display area.

[0022] Another aspect of the present invention is a medical image processing method for a medical image processing device equipped with a processor and a memory, in which the processor executes a medical image acquisition step of acquiring a plurality of medical images in chronological order, an observation state determination step of determining the observation state of each small area of ​​the subject based on the medical images, a recording step of recording the determination results in memory, and a display step of displaying on a monitor an observation state display of the subject based on the determination results recorded in memory when a change occurs in the observation state of the subject.

[0023] Another aspect of the present invention is a program that causes a medical image processing device having a processor and a memory to execute a medical image processing method, wherein the processor causes the medical image processing method to execute the following steps: a medical image acquisition step of acquiring multiple medical images in chronological order; an observation state determination step of determining the observation state of each small area of ​​the subject based on the medical images; a recording step of recording the determination results in memory; and a display step of displaying on a monitor an observation state display of the subject based on the determination results recorded in memory when a change occurs in the observation state of the subject. [Effects of the Invention]

[0024] According to the present invention, the observation state display of the subject is displayed on the monitor when a change occurs in the observation state of the subject. Therefore, by displaying the observation state display of the subject at an appropriate timing, it is possible to display the image effectively while minimizing the impact on the observation of the endoscopic image. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is an external view of the endoscope system. [Figure 2] FIG. 2 is a block diagram showing the configuration of the main parts of the endoscope system. [Figure 3] FIG. 3 is a functional block diagram of the medical image processing apparatus in the image processing unit. [Figure 4] FIG. 4 is a diagram showing main information recorded in the recording unit. [Figure 5]FIG. 5 is a diagram showing the configuration of a neural network. [Figure 6] FIG. 6 is a schematic diagram showing an example of the configuration of the intermediate layer. [Figure 7] FIG. 7 is a flowchart illustrating a medical image processing method. [Figure 8] FIG. 8 is a diagram showing an example of the observation state display. [Figure 9] FIG. 9 is a diagram showing a first modification of the observation state display. [Figure 10] FIG. 10 is a diagram showing a second modification of the observation state display. [Figure 11] FIG. 11 is a diagram showing a third modification of the observation state display. [Figure 12] FIG. 12 is a diagram showing a fourth modified example of the observation state display. [Figure 13] FIG. 13 is a diagram showing a fifth modified example of the observation state display. [Figure 14] FIG. 14 is a diagram showing a sixth modification of the observation state display. [Figure 15] FIG. 15 is a diagram illustrating an example of a monitor having a main display area and a sub-display area. [Figure 16] FIG. 16 is a diagram illustrating an example in which different observation state displays are displayed in the main display area and the sub-display area. [Figure 17] FIG. 17 is a diagram illustrating an example in which different observation state displays are displayed in the main display area and the sub-display area. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, preferred embodiments of a medical image processing apparatus, a medical image processing method, a program, and a recording medium according to the present invention will be described with reference to the accompanying drawings.

[0027] <Configuration of endoscope system> Fig. 1 is an external view of an endoscope system 10, and Fig. 2 is a block diagram showing the configuration of the main parts of the endoscope system 10. As shown in Figs. 1 and 2, the endoscope system 10 is composed of an endoscope 100, an endoscope processor device 200, a light source device 300, and a monitor 400. The endoscope processor device 200 is equipped with the medical image processing device of the present invention.

[0028] <Configuration of an endoscope> The endoscope 100 includes a handheld control unit 102 and an insertion section 104 connected to the handheld control unit 102. The surgeon (user) holds and operates the handheld control unit 102, and inserts the insertion section 104 into the body of a subject (living organism) for observation. The handheld control unit 102 is also provided with an air / water supply button 141, a suction button 142, function buttons 143 to which various functions can be assigned, and an image capture button 144 for receiving image capture instructions (still image, moving image). The insertion section 104 is composed of, in order from the handheld control unit 102 side, a flexible section 112, a bending section 114, and a distal end rigid section 116. That is, the bending section 114 is connected to the proximal end side of the distal end rigid section 116, and the flexible section 112 is connected to the proximal end side of the bending section 114. The handheld control unit 102 is connected to the proximal end side of the insertion section 104. The user can bend the bending section 114 by operating the hand operation section 102, thereby changing the orientation of the tip rigid section 116 up, down, left, or right. The tip rigid section 116 is provided with an imaging optical system 130, an illumination section 123, a forceps port 126, etc. (see FIGS. 1 and 2).

[0029] During observation and treatment, white light and / or narrowband light (one or more of red narrowband light, green narrowband light, blue narrowband light, and violet narrowband light) can be emitted from illumination lenses 123A and 123B of illumination unit 123 by operating operation unit 208 (see FIG. 2). In addition, by operating air / water supply button 141, cleaning water can be discharged from a water supply nozzle (not shown) to clean imaging lens 132 of imaging optical system 130 and illumination lenses 123A and 123B. A conduit (not shown) communicates with forceps port 126 opening at tip rigid portion 116, and a treatment tool (not shown) for tumor removal or the like can be inserted into this conduit and moved forward and backward as appropriate to perform the necessary treatment on the subject.

[0030] As shown in FIGS. 1 and 2, a photographing lens 132 is disposed on the distal end surface 116A of the distal end rigid portion 116. A CMOS (Complementary Metal-Oxide Semiconductor) image sensor 134, a driver circuit 136, and an AFE (Analog Front End) 138 are disposed behind the photographing lens 132, and these elements output an image signal. The image sensor 134 is a color image sensor and includes a plurality of pixels each composed of a plurality of light receiving elements arranged in a matrix (two-dimensional array) in a specific pattern array (e.g., Bayer array, X-Trans (registered trademark) array, honeycomb array, etc.). Each pixel of the image sensor 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (e.g., a photodiode). The photographing optical system 130 can generate a color image from pixel signals of three colors (red, green, and blue), or can generate an image from pixel signals of any one or two colors of red, green, and blue. The image sensor 134 may be a CCD (Charge Coupled Device) type. Each pixel of the image sensor 134 may further include a purple color filter corresponding to a purple light source 310V and / or an infrared filter corresponding to an infrared light source.

[0031] An optical image of the subject is formed on the light receiving surface (imaging surface) of the imaging element 134 by the photographing lens 132, converted into an electrical signal, and output to the endoscope processor device 200 via a signal cable (not shown), where it is converted into a video signal. As a result, an endoscopic image (medical image) of the subject is displayed on the screen of the monitor 400 connected to the endoscope processor device 200.

[0032] Furthermore, illumination lenses 123A and 123B of the illumination unit 123 are provided adjacent to the photographing lens 132 on the distal end surface 116A of the distal end hard portion 116. An exit end of a light guide 170 (described later) is disposed behind the illumination lenses 123A and 123B. This light guide 170 is inserted through the insertion portion 104, the handheld operation unit 102, and the universal cable 106, and the entrance end of the light guide 170 is disposed inside the light guide connector 108.

[0033] The user can sequentially capture time-series endoscopic images of the inside of a living body at a determined frame rate while inserting or removing the endoscope 100 having the above-described configuration into or from the living body.

[0034] <Configuration of light source device> 2, light source device 300 is composed of an illumination light source 310, an aperture 330, a condenser lens 340, a light source control unit 350, etc., and causes observation light to enter light guide 170. Light source 310 is equipped with red light source 310R, green light source 310G, blue light source 310B, and purple light source 310V, which emit narrowband light of red, green, blue, and purple, respectively, and is capable of emitting narrowband light of red, green, blue, and purple. The illuminance of the observation light emitted by light source 310 is controlled by light source control unit 350, which can change (increase or decrease) the illuminance of the observation light or stop illumination as necessary.

[0035] The light source 310 can emit narrowband light of red, green, blue, and purple in any combination. For example, it can simultaneously emit narrowband light of red, green, blue, and purple to irradiate white light (normal light) as observation light, or it can emit narrowband light (special light) by emitting one or two of the colors. The light source 310 may further include an infrared light source that irradiates infrared light (an example of narrowband light). Alternatively, white light or narrowband light may be irradiated as observation light by using a light source that irradiates white light and a filter that transmits the white light and each narrowband light.

[0036] <Light source wavelength band> Light source 310 may be a light source that generates white light, or light in multiple wavelength bands as white light, or a light source that generates light in a specific wavelength band narrower than the white wavelength band. The specific wavelength band may be the blue or green band in the visible range, or the red band in the visible range. If the specific wavelength band is the blue or green band in the visible range, it may include a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and have a peak wavelength within the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. If the specific wavelength band is the red band in the visible range, it may include a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and the light in the specific wavelength band may have a peak wavelength within the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm.

[0037] The light in the specific wavelength band may include a wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin differ, and may have a peak wavelength in the wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin differ. In this case, the specific wavelength band may include a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm to 750 nm, and may have a peak wavelength in a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm to 750 nm.

[0038] Furthermore, the light emitted by light source 310 may include a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm, and may have a peak wavelength in the wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.

[0039] Furthermore, the light source 310 may be equipped with a light source that emits excitation light having a peak wavelength of 390 nm or more and 470 nm or less. In this case, an endoscopic image containing information on the fluorescence emitted by fluorescent substances in the subject (living body) can be acquired. When acquiring a fluorescent image, a dye agent for fluoroscopy (fluorescein, acridine orange, etc.) may be used.

[0040] The type of light source (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), etc.), wavelength, presence or absence of a filter, etc. of the light source 310 are preferably configured according to the type and location of the subject, the purpose of observation, etc., and during observation, it is preferable to combine and / or switch the wavelength of the observation light according to the type and location of the subject, the purpose of observation, etc. When switching wavelengths, the wavelength of the irradiated light may be switched, for example, by rotating a disk-shaped filter (rotary color filter) placed in front of the light source and equipped with a filter that transmits or blocks light of a specific wavelength.

[0041] Furthermore, the imaging element used in carrying out the present invention is not limited to a color imaging element such as the imaging element 134, in which a color filter is provided for each pixel, but may also be a monochrome imaging element. When a monochrome imaging element is used, imaging can be performed in a frame sequential (color sequential) manner by sequentially switching the wavelength of the observation light. For example, the wavelength of the emitted observation light may be sequentially switched between (purple, blue, green, red), or broadband light (white light) may be irradiated and the wavelength of the emitted observation light may be switched using a rotary color filter (red, green, blue, purple, etc.). Alternatively, one or more narrowband lights (green, blue, purple, etc.) may be irradiated and the wavelength of the emitted observation light may be switched using a rotary color filter (green, blue, purple, etc.). The narrowband light may be infrared light of two or more different wavelengths (first narrowband light, second narrowband light).

[0042] By connecting the light guide connector 108 (see Figures 1 and 2) to the light source device 300, the observation light emitted from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and is then irradiated onto the observation range from the illumination lenses 123A and 123B.

[0043] <Configuration of endoscope processor device> The configuration of the endoscope processor device 200 will be described with reference to Fig. 2. The endoscope processor device 200 receives an image signal output from the endoscope 100 via an image input controller 202, performs necessary image processing in an image processing unit 204, and outputs the signal via a video output unit 206. This results in an endoscopic image being displayed on the monitor 400. These processes are performed under the control of a CPU 210 (Central Processing Unit). The CPU 210 functions as a processor of the medical image processing device. A communication control unit 205 controls communications regarding the acquisition of medical images, etc., between an in-hospital system (HIS: Hospital Information System) and an in-hospital LAN (Local Area Network), not shown, and / or external systems or networks.

[0044] <Image processing unit functions> The image processing unit 204 can calculate the feature values ​​of the endoscopic image, emphasize or reduce components in specific frequency bands, and emphasize or reduce the prominence of specific targets (regions of interest, blood vessels at a desired depth, etc.). The image processing unit 204 may also include a special light image acquisition unit (not shown) that acquires a special light image having information about a specific wavelength band based on a normal light image acquired by irradiating the normal light image with white light or light in multiple wavelength bands as white light. In this case, the signal of the specific wavelength band can be obtained by calculation based on RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) color information contained in the normal light image. The image processing unit 204 may also include a feature image generation unit (not shown) that generates a feature image by calculation based on at least one of a normal light image acquired by irradiating the normal light image with white light or light in multiple wavelength bands as white light and a special light image acquired by irradiating light in a specific wavelength band, and acquires and displays the feature image as an endoscopic image. The above-mentioned processing is performed under the control of the CPU 210.

[0045] Furthermore, the image processing unit 204 has the functions of a medical image processing apparatus as described below.

[0046] 3 is a functional block diagram of the medical image processing apparatus in the image processing unit 204. The image processing unit 204 includes a medical image acquisition unit 220, an observation state determination unit 222, and a display control unit 224.

[0047] <Functional implementation using various processors> The functions of each unit of the image processing unit 204 described above can be realized using various processors and recording media. The various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. The various processors also include a GPU (Graphics Processing Unit), which is a processor specialized for image processing, and a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacturing. When performing image learning and recognition, as in the present invention, a configuration using a GPU is effective. Furthermore, the various processors described above also include dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing specific processing.

[0048] The functions of each unit may be realized by a single processor, or by multiple processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functions may be realized by a single processor. Examples of multiple functions configured by a single processor include: a first configuration, as typified by a computer, in which a single processor is configured by combining one or more CPUs and software, and this processor realizes multiple functions; a second configuration, as typified by a system-on-chip (SoC), in which a processor is used to realize the functions of the entire system on a single IC (Integrated Circuit) chip; and various functions are thus configured as hardware structures using one or more of the various processors described above. Furthermore, the hardware structures of these various processors are, more specifically, electrical circuits combining circuit elements such as semiconductor devices. These electrical circuits may be electrical circuits that realize the above-mentioned functions using logical operations such as logical sum, logical product, logical negation, exclusive OR, and combinations of these.

[0049] When the processor or electrical circuit described above executes software (programs), computer-readable code for the software to be executed (e.g., various processors and electrical circuits constituting the image processing unit 204, and / or a combination thereof) is stored in a non-transitory recording medium such as a ROM 211 (Read Only Memory), and the computer references the software. The software stored in the non-transitory recording medium includes a program for executing the medical image processing method of the medical image processing device according to the present invention and data used during execution. The code may be recorded in a non-transitory recording medium such as various types of magneto-optical recording devices or semiconductor memory, instead of the ROM 211. When processing using the software, for example, a RAM 212 (Random Access Memory) is used as a temporary storage area, and data stored in, for example, an EEPROM (Electronically Erasable and Programmable Read Only Memory), not shown, may also be referenced. The recording unit 207 may also be used as a "non-transitory recording medium."

[0050] Furthermore, ROM 211 (Read Only Memory) is a non-volatile storage element (non-temporary recording medium) that stores computer-readable code of programs that cause the CPU 210 and / or image processing unit 204 to execute various image processing methods. RAM 212 (Random Access Memory) is a storage element for temporary storage during various processes, and can also be used as a buffer when acquiring images. Audio processing unit 209 outputs voice and sound from speaker 209A under the control of CPU 210.

[0051] The operation unit 208 can be configured with devices such as a keyboard and a mouse (not shown), and the user can issue instructions to execute processing and specify conditions required for execution via the operation unit 208.

[0052] <Information recorded in the recording unit> 4 is a diagram showing the main information recorded in the recording unit 207. The recording unit (memory) 207 records a medical image (endoscopic image) 260, a determination result 262 from the observation state determination unit 222, and the like. In addition, information relating to a series of small regions to be observed in an examination performed using the endoscope system 10 is recorded. Here, the small regions of the subject are, for example, various parts of an organ. Specifically, in an examination to observe all parts of the stomach, the small regions are the cardia, fundus, angle of the stomach, body (upper, middle, and lower parts), vestibule, anterior wall, posterior wall, greater curvature, and lesser curvature.

[0053] <Neural network recognition section> The observation state determination unit 222 in the image processing unit 204 described above includes a recognizer that is configured using a trained model such as a neural network (a model trained using an image set consisting of images of a living body) and that can recognize small regions of a subject. The observation state determination unit 222 then determines for each small region whether observation of that small region has been completed based on the position of the small region recognized by the recognizer, the number of endoscopic images in which that small region is recognized, etc. Below, the configuration of the recognizer included in the observation state determination unit 222 when a CNN (Convolutional Neural Network) is used as the neural network will be described.

[0054] <Example of recognizer configuration> FIG. 5 is a diagram showing the configuration of a CNN 232 (neural network). In the example shown in part (a) of FIG. 5, the CNN 232 has an input layer 232A, an intermediate layer 232B, and an output layer 232C. The input layer 232A inputs an endoscopic image acquired by the medical image acquisition unit 220 and outputs features. The intermediate layer 232B includes a convolutional layer 234 and a pooling layer 235, and inputs the features output by the input layer 232A to calculate other features. These layers have a structure in which multiple "nodes" are connected by "edges," and weighting coefficients to be applied to the input image are associated with the nodes and edges and stored in a weighting coefficient storage unit (not shown). The values ​​of the weighting coefficients change as learning progresses.

[0055] <Processing in the middle layer> The intermediate layer 232B calculates features through convolution and pooling. The convolution performed in the convolution layer 234 is a process for obtaining a feature map through convolution using a filter, and is responsible for extracting features such as edge extraction from an image. The convolution using this filter generates one channel (one image) of a "feature map" for each filter. When downscaling is performed by convolution, the size of the "feature map" becomes smaller as convolution is performed in each layer. The pooling process performed in the pooling layer 235 reduces (or enlarges) the feature map output by the convolution to create a new feature map, and is responsible for providing robustness to the extracted features so that they are not affected by translation, etc. The intermediate layer 232B can be composed of one or more layers that perform these processes. Note that the CNN 232 may be configured without the pooling layer 235.

[0056] The CNN 232 may include a fully connected layer 236, as shown in the example of part (b) of Fig. 5. The layer configuration of the CNN 232 is not limited to a case where the convolutional layer 234 and the pooling layer 235 are repeated one by one, but may include any layer (for example, the convolutional layer 234) multiple times in succession.

[0057] FIG. 6 is a schematic diagram showing an example of the configuration of the hidden layer 232B of the CNN 232 shown in FIG. 5. In the first convolutional layer of the hidden layer 232B, a convolutional operation is performed between an image set consisting of multiple endoscopic images and a filter F1. The image set consists of N images (N channels) with an image size of H vertically and W horizontally. When a normal light image is input, the images constituting the image set are images of three channels: R (red), G (green), and B (blue). The filter F1 convolved with this image set has N channels (N images). For example, in the case of a filter of size 5 (5 × 5), the filter size is 5 × 5 × N. The convolutional operation using this filter F1 generates a one-channel (one-image) "feature map" for each filter F1. The filter F2 used in the second convolutional layer has a filter size of 3 × 3 × M, for example, in the case of a filter of size 3 (3 × 3).

[0058] Similar to the first convolutional layer, the second to nth convolutional layers perform convolutional operations using filters F2 to Fn. The size of the "feature map" in the nth convolutional layer is smaller than that of the "feature map" in the second convolutional layer because it has been downscaled by the previous convolutional layers or pooling layers.

[0059] Among the layers in the intermediate layer 232B, low-level feature extraction (edge ​​extraction, etc.) is performed in the convolutional layers closer to the input side, and higher-level feature extraction (extraction of features related to the shape, structure, etc. of the recognition target) is performed as the layers approach the output side.

[0060] The intermediate layer 232B may include a layer that performs batch normalization in addition to the convolutional layer 234 and the pooling layer 235. The batch normalization process normalizes the distribution of data in units of mini-batches when performing learning, and plays a role in accelerating learning, reducing dependency on initial values, and suppressing overlearning.

[0061] The output layer 232C outputs the feature amounts calculated by the intermediate layer 232B in a format suitable for recognition. The output layer 232C may include a fully connected layer.

[0062] <Each step in the medical image processing method> Next, a medical image processing method using the medical image processing apparatus will be described.

[0063] Fig. 7 is a flowchart showing a medical image processing method. Each step will be explained below with reference to Fig. 6. Note that the following describes a case where small areas, Area 1, Area 2, and Area 3, are observed on an area-by-area basis when examining organ A, which is the subject.

[0064] (Medical image acquisition step) The medical image acquisition unit 220 sequentially acquires a plurality of medical images of organ A in chronological order (step S10). Note that the recording unit 207 records that observations of areas 1, 2, and 3 of organ A will be performed, and that the initial state is that observations of areas 1, 2, and 3 are incomplete.

[0065] (Observation state determination step) The observation state determination unit 222 determines the observation states of areas 1, 2, and 3 of organ A based on the acquired medical images (step S11). The observation state determination unit 222 recognizes area 1, area 2, or area 3 in the medical images. Then, based on the recognition results, it determines the observation states of areas 1, 2, and 3. For example, if area 1 is recognized as the center of 10 chronologically consecutive medical images, the observation state determination unit 222 determines that observation of area 1 has been completed.

[0066] (Recording step) The recording unit 207 records the result of the determination by the observation state determination unit 222 (step S12). At the start of the inspection (initial state), areas 1, 2, and 3 are recorded as not having been observed, but when the observation state determination unit 222 determines that the observation of each area has been completed, the record is updated to indicate that observation has been completed.

[0067] (Display step) The display control unit 224 determines whether a change has occurred in the observation state of the subject (step S13). Then, when the display control unit 224 determines that a change has occurred in the observation state of the subject, it displays the observation state display 501 of the subject on the monitor 400 (step S14). Here, the point in time when a change has occurred in the observation state of the subject is when a change in the observation state is recorded in the recording unit 207 in multiple small area units where observation is scheduled to be performed, which are recorded in the recording unit 207. For example, this is when the recording unit 207 records areas 1, 2, and 3 as incomplete observation, but the observation state determination unit 222 determines that observation of area 1 has been completed, and the observation state of area 1 has been completed in the recording unit 207. The observation state display 501 of the subject is a display that notifies the user of the observation state of the small area units that make up the subject to be observed. By looking at the observation state display 501, the user can confirm whether the small areas to be observed have been comprehensively observed.

[0068] FIG. 8 is a diagram showing an example of an observation state display 501 of organ A displayed on the monitor 400.

[0069] In the case shown in Fig. 8, an endoscopic image 503 is displayed across the entire surface of the monitor 400. Then, at the point in time when the record of the recording unit 207 updates area 3 from incomplete observation to complete observation, the display control unit 224 displays an observation status display 501 on the monitor 400, superimposed on the endoscopic image 503. The observation status display 501 is a list display having text information indicating the areas for which observation has been completed and the areas for which observation has not been completed. In the observation status display 501, the areas for which observation has been completed (referred to as area in the figure) 1 and area 3 are listed under "completed," and the area for which observation has not been completed is listed under "not yet."

[0070] Returning to FIG. 7, the display control unit 224 then continues to display the observation status display 501 until a predetermined display time has elapsed. The display time can be set appropriately by the user. It is preferable that the observation status display 501 is hidden once the user can confirm the observation status, allowing the endoscopic image 503 to be observed, and therefore it is preferable that the display time be set based on the time during which the user can confirm the observation status. For example, the display time can be set to 10 seconds or 30 seconds. Thereafter, the display control unit 224 hides the observation status display 501 after the predetermined time has elapsed (step S15).

[0071] Thereafter, the medical image acquisition unit 220 determines whether or not the observation of all small area units has been completed (step S16), and since the observation of area 2 has not yet been completed, further medical images are acquired (step S10).

[0072] As described above, in the present invention, when observing Area 1, Area 2, and Area 3, which are small regions of Organ A, the observation status display 501 is displayed at the point in time when a change occurs in the observation status of Organ A. As a result, by displaying the observation status display 501 when necessary and refraining from displaying it in other cases, it is possible to display the endoscopic image effectively while minimizing its impact on observation.

[0073] <Modification of observation state display> 8, a case has been described in which an observation status display 501 is displayed that displays areas that have been observed and areas that have not yet been observed using text information, but the example of the observation status display 501 is not limited to this. The display form of the observation status display 501 is not particularly limited as long as it can use text and graphics to notify the user of the observation status of the subject being observed. Specific examples of the observation status display 501 are described below.

[0074] FIG. 9 is a diagram showing a first modified example of the observation status display 501. In this example, the observation status display 501 is text information. In this example, the observation status display 501 displays only areas for which observation has not been completed. Specifically, if observation of area 2 has not been completed, the text information "Area 2" is written below "Not yet."

[0075] In this way, by displaying the observation status display 501 on the monitor 400, which includes only small areas for which observation has not been completed, the user can clearly recognize the small areas for which observation has not been completed, thereby achieving comprehensive observation. Note that in the example of Figure 9, an example was described in which small areas for which observation has not been completed are displayed in the observation status display 501, but small areas for which observation has been completed may also be displayed in the observation status display 501. In this case, the user can clearly recognize the small areas for which observation has been completed.

[0076] FIG. 10 is a diagram showing a second modified example of the observation status display 501. In this example, the observation status display 501 is text information. In this example, the observation status display 501 displays all of the small areas (areas 1 to 5) that make up the subject to be observed as a list. In the observation status display 501, areas where observation has been completed and areas where observation has not been completed are displayed with different colored text. Specifically, in the observation status display 501, areas 3 and 5 are not yet observed, so they are displayed with text of the same color, and areas 1, 2, and 4 are observed, so they are displayed with text of the same color. In this way, all of the small areas are displayed in a list, and information on whether observation has been completed or not is added to text on a small area-by-small area basis, so the user can comprehensively recognize the areas where observation has been completed and areas where observation has not been completed.

[0077] FIG. 11 is a diagram showing a third modified example of the observation status display 501. This example is an observation status display 501 using text information. In this example, the observation status display 501 displays all of the small areas (areas 1 to 5) that make up the subject to be observed as a list. In the observation status display 501, an "o" or an "x" is displayed next to the letters of areas for which observation has been completed and areas for which observation has not been completed. Specifically, in the observation status display 501, areas 3 and 5 are incomplete because observation has not been completed, and areas 1, 2, and 4 are complete because observation has been completed, and an "o" is displayed. In this way, all of the small areas are displayed in a list, and information on whether observation has been completed or not is displayed next to the letters for each small area, allowing the user to comprehensively recognize areas for which observation has been completed or not.

[0078] FIG. 12 is a diagram showing a fourth modified example of the observation status display 501. In this example, the observation status display 501 uses text information. In this example, the observation status display 501 shows the percentage of incomplete observation as the observation status display 501. Here, the percentage of incomplete observation is the percentage of incomplete observation small areas among the multiple small areas to be observed. Note that this percentage may be calculated using the number of small areas or the area of ​​the small areas. Furthermore, in the example shown in FIG. 12, the percentage is shown using text, but it may also be shown using a graphic such as a bar display indicating the percentage. In this way, by showing the percentage of incomplete observation, the user can clearly see the areas where observation is incomplete.

[0079] FIG. 13 is a diagram showing a fifth modified example of the observation status display 501. This example shows the observation status display 501 using a subject model M that schematically represents a subject. The subject model M is a diagram that schematically illustrates a stomach, which is the subject to be observed. In the subject model M, different colors are assigned to the observed regions 513 and the incomplete regions 511 in small region units. By assigning information on whether observation is complete or incomplete to the subject model M in this way, the user can recognize the positions of the observed small regions and the incomplete small regions in the subject. Note that in the example shown in FIG. 13, the information on whether observation is complete or incomplete is assigned by changing the color of the region of the subject model M, but this is not limiting. For example, information on whether observation is complete or incomplete may be assigned to the subject model M by changing the color density.

[0080] FIG. 14 is a diagram showing a sixth modified example of the observation status display 501. This example shows the observation status display 501 using a subject model M that schematically represents the subject. In this example, only small areas that have been observed are displayed in the subject model M. When observation of all small areas is complete, the entire subject model M (stomach) is displayed. In this way, by assigning information on whether observation has been completed or not to the subject model M, the user can recognize the positions of small areas in the subject that have been observed and the positions of small areas in which observation has not been completed. Note that in FIGS. 13 and 14, the subject model M is a cross-section view of the stomach, which is the subject, but this is not limited to this. For example, the subject model M may be a development view of the stomach, which is the subject.

[0081] <Modification of monitor> In the above description, an example of the monitor 400 having only a main display area has been described, but the present invention is not limited to this. Modifications of the monitor will be described below.

[0082] FIG. 15 is a diagram illustrating an example of a monitor 401 having a main display area and a sub-display area.

[0083] As shown in FIG. 15 , the monitor 401 has a main display area 402 and a sub-display area 404. The main display area 402 displays an endoscopic image 503 captured by the endoscope 100 in real time. The sub-display area 404 is set smaller than the main display area 402, and displays information such as the imaging conditions, date, and patient information. Note that FIG. 15 omits the illustration of this information displayed in the sub-display area 404. On the monitor 401, the observation status display 501 is displayed in the main display area 402 and the sub-display area 404. As explained above, the observation status display 501 is displayed in the main display area 402 when a change occurs in the observation status of the subject, and the display ends when the display time has elapsed. Note that FIG. 15 illustrates a case where the observation status display 501 in the main display area 402 is not displayed.

[0084] The observation status display 501B is always displayed in the sub-display area 404. Because the display area of ​​the sub-display area 404 is small, the observation status display 501B is also smaller than the observation status display 501A displayed in the main display area 402, but it serves as an aid for performing comprehensive observations. The main display area 402 and the sub-display area 404 may display the same observation status display, or different observation status displays.

[0085] 16 and 17 are diagrams illustrating an example in which different observation state displays are displayed in the main display area and the sub-display area.

[0086] 16, a detailed observation status display 501A showing a list of small areas is displayed in the main display area 402. In addition, an observation status display 501B of the subject model M is displayed in the sub-display area 404.

[0087] 17, a detailed observation status display 501A showing a list of small areas is displayed in the main display area 402. In addition, an observation status display 501B showing the percentage of incomplete observations is displayed in the sub-display area 404.

[0088] As shown in FIGS. 16 and 17 , the main display area 402 displays an observation status display 501A, which shows a list of small areas, allowing the user to grasp the observation status of each small area in detail, when the observation status of the subject changes. This allows the user to grasp the detailed observation status of the subject when the number of small areas for which observation has been completed increases. In addition, the sub-display area 404 always displays an observation status display 501B, which shows the subject model M and the percentage of uncompleted observations. This allows the user to grasp the general observation status of the subject even when the observation status display 501A is not displayed in the main display area 402. Note that in the above example, the monitor 401 having the main display area (first display area) 402 and the sub-display area (second display area) 404 has been described. However, the observation status display 501 may also be displayed on a monitor having, for example, a third display area. Furthermore, the observation status display 501 may be displayed on multiple monitors.

[0089] <Other examples> In the above description, the observation status display 501 is displayed for a predetermined display time period, but the manner in which the observation status display 501 is hidden after it has been displayed is not limited to this.

[0090] For example, the display control unit 224 may hide the observation status display 501 based on a command input by the user via the local operation unit 102 (user operation reception unit) (see FIG. 1). Furthermore, the display control unit 224 may re-display the observation status display 501, which has been temporarily hidden, based on a command input by the user via the local operation unit 102 (user operation reception unit). In this way, the user can control the display and non-display of the observation status display 501 using the local operation unit 102, allowing the user to check the observation status display at a desired timing.

[0091] Although examples of the present invention have been described above, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the present invention. [Explanation of symbols]

[0092] 10: Endoscope system 100: Endoscope 102: Handheld operation unit 104: Insertion section 106: Universal cable 108: Light guide connector 112: Soft part 114: Curved section 116: Hard tip 116A:Tip side end surface 123: Lighting Department 123A: Lighting lens 123B: Lighting lenses 126: Forceps opening 130: Photographing optical system 132: Photo lens 134: Image sensor 136: Drive circuit 141: Air / water supply button 142: Suction button 143: Function button 144: Shooting button 170: Light guide 200: Endoscope processor device 202: Image input controller 204: Image processing unit 205: Communication control section 206: Video output section 207: Recording Department 208 :Operation unit 209: Audio processing unit 209A: Speaker 210:CPU 211 :ROM 212:RAM 220: Medical image acquisition unit 222: Observation state determination unit 224: Display control unit 232A: Input layer 232B: Middle layer 232C: Output layer 234: Convolutional layer 235: Pooling layer 236 :Fully connected layer 300: Light source device 310 :Light source 310B: Blue light source 310G: Green light source 310R: Red light source 310V: Violet light source 330: Aperture 340: Condenser lens 350: Light source control unit 400: Monitor

Claims

1. 1. A medical imaging device comprising a processor, The processor: Acquire multiple medical images in time series, Based on the medical image, an observation state is recognized for each of a plurality of small regions that are predetermined and located differently from one another within the organ that is the subject; Based on the recognition result, if observation of each of the plurality of small regions is completed, the observation is determined to be complete, and if observation is incomplete, the observation is determined to be incomplete; displaying information indicating that the observation state of each of the plurality of small regions is "observation completed" on the subject model that schematically represents the organ based on the result of the determination; when the observation state of any one of the plurality of small regions changes from the observation incomplete to the observation completed, updating the object model by reflecting information indicating that the observation state of the one small region is observation completed. Medical imaging equipment.

2. The medical image processing apparatus according to claim 1 , wherein the processor makes the information displayed on the subject model invisible after a predetermined time has elapsed.

3. A user operation receiving unit is further provided, The medical image processing apparatus according to claim 1 , wherein the processor displays or hides the information based on an instruction from the user operation accepting unit.

4. The medical image processing apparatus according to claim 1 , wherein the processor displays the information on a monitor as text information.

5. The medical image processing apparatus according to claim 4 , wherein the processor adds information regarding whether observation of the small area of ​​the subject has been completed or not to the character information and displays the information.

6. The medical image processing apparatus according to claim 1 , wherein the processor displays, on a monitor as the information, an indication of whether observation of the small area of ​​the subject has been completed or not.

7. The medical image processing apparatus according to claim 1 , wherein the processor displays the medical image on a monitor and displays the information superimposed on the medical image.

8. 8. A medical image processing device according to claim 1, wherein the processor is a monitor having a first display area and a second display area smaller than the first display area, and the processor displays the information in different manners in the first display area and the second display area.

9. The medical image processing apparatus according to claim 8 , wherein the processor constantly displays the information in the second display area.

10. The medical image processing apparatus according to claim 9 , wherein the processor is a monitor having a third display area different from the first display area and the second display area, and causes the medical image to be displayed in the third display area.

11. The medical image processing apparatus according to claim 1 , wherein the processor displays the object model by using a straight line to distinguish between the observation-completed region and the incomplete region.

12. 12. The medical image processing device according to claim 1, wherein the processor recognizes, in the medical image, a plurality of small regions at different positions within the organ of the subject, using a trained model that has been trained to recognize each of the plurality of small regions as a small region unit.

13. A medical image processing method for a medical image processing device including a processor, comprising: The processor: acquiring a plurality of medical images in time series; a step of recognizing an observation state for each of a plurality of small regions, each of which is located at a different position from one another, in the organ as the subject based on the medical image; determining that the observation is complete when the observation of each of the plurality of small regions is completed based on the recognition result, and determining that the observation is incomplete when the observation is incomplete; displaying information indicating that the observation state of each of the plurality of small regions is "observation complete" on the subject model that schematically represents an organ based on the result of the determination; when the observation state of any one of the plurality of small regions changes from the observation incomplete to the observation completed, updating the object model by reflecting information indicating that the observation state of the one small region is observation completed; A medical image processing method comprising:

14. A program for causing a medical image processing device having a processor to execute a medical image processing method, acquiring a plurality of medical images in time series; a step of recognizing an observation state for each of a plurality of small regions, each of which is located at a different position from one another, in the organ as the subject based on the medical image; determining that the observation is complete when the observation of each of the plurality of small regions is completed based on the recognition result, and determining that the observation is incomplete when the observation is incomplete; displaying information indicating that the observation state of each of the plurality of small regions is "observation complete" on the subject model that schematically represents an organ based on the result of the determination; when the observation state of any one of the plurality of small regions changes from the observation incomplete to the observation completed, updating the object model by reflecting information indicating that the observation state of the one small region is observation completed; A program causing the processor to execute a medical image processing method comprising:

15. A non-transitory computer-readable recording medium on which the program according to claim 14 is recorded.

Images