Medical image processing device, medical image processing method, program, and recording medium

Abstract

A medical image processing device, a medical image processing method, a program, and a recording medium are provided that can reduce the discrepancy between recognition by a recognizer and recognition by a user. [Solution] The medical image processing device is a medical image processing device equipped with a processor (210) and a memory (207), in which the processor (210) acquires a plurality of medical images in chronological order, determines the observation state of each small area of ​​the subject based on the medical images, records the determination results in the memory (207), and, at the point in time when a change occurs in the observation state of the subject, displays on the monitor (400) an observation state display of the subject based on the determination results recorded in the memory (207).

Description

Technical Field

[0006] , ,

[0005] ,

[0001] The present invention relates to a medical image processing apparatus, a medical image processing method, a program, and a recording medium.

Background Art

[0002] Conventionally, in an examination performed using an endoscope system, it has been required to comprehensively observe regions such as organs of an examination target.

[0003] Patent Document 1 describes a technique for preventing imaging omission in an examination using an endoscope system. In the technique described in Patent Document 1, a map image representing the already imaged region and the non-imaged region of the imaging target organ is displayed on a monitor as a notification display.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Here, usually, an endoscope image being taken in real time during an examination is displayed on the main display area of the monitor of the endoscope system. Therefore, when a notification display as described in Patent Document 1 is displayed in the main display area, it will be superimposed on the endoscope image, and the visibility of the endoscope image will be reduced. Also, when the notification display is displayed in the sub-display area of the monitor, the display area is small and the visibility of the notification display is reduced. On the other hand, although it is also conceivable to perform the notification display on a sub-monitor different from the main monitor, there is a problem that the user cannot focus on the endoscope image displayed on the main monitor during the examination.

[0006] The aforementioned Patent Document 1 does not mention any display modes that take into account the visibility of endoscopic images or the visibility of notification displays (map images).

[0007] This invention has been made in view of these circumstances, and its purpose is to provide a medical image processing device, a medical image processing method, and a program that effectively display an observation status indicator regarding the comprehensiveness of the observation while suppressing a decrease in the visibility of endoscopic images. [Means for solving the problem]

[0008] A medical image processing apparatus, which is one aspect of the present invention for achieving the above objective, is a medical image processing apparatus comprising a processor and memory, wherein the processor acquires a plurality of medical images in a time series, determines the observation state of a small area of ​​a subject based on the medical images, records the result of the determination in memory, and, when a change occurs in the observation state of the subject, displays a display of the observation state of the subject based on the determination result recorded in memory on a monitor.

[0009] According to this embodiment, when a change occurs in the observation state of the subject, the observation state of the subject is displayed on the monitor. This allows for effective display of the observation state of the subject at an appropriate time, while minimizing the impact on the observation of the endoscopic image.

[0010] Preferably, the processor hides the observation status display shown on the monitor after a predetermined time has elapsed.

[0011] Preferably, the system further includes a user operation reception unit, and the processor displays or hides the observation status display based on a command from the user operation reception unit.

[0012] Preferably, when the processor determines the observation status of a small region, it determines that the observation is complete if the observation of the small region is finished, and that the observation is not yet complete if the observation is not yet finished.

[0013] Preferably, the processor displays the observed status on the monitor using character information.

[0014] Preferably, the processor adds information regarding the completion or incompleteness of the observation of a small area unit of the subject to the character information and displays it as an observation status indicator.

[0015] Preferably, the processor displays the observation status using a subject model that schematically represents the subject.

[0016] Preferably, the processor provides the subject model with information regarding the completion or incompleteness of the observation of a small area unit of the subject, and displays it as an observation status indicator.

[0017] Preferably, the processor displays, as an observation status indicator, only the completion or incompleteness of the observation of a small area unit of the subject.

[0018] Preferably, the processor displays a medical image on the monitor and overlays an observation status display onto the medical image.

[0019] Preferably, the processor is a monitor having a first display area and a second display area smaller than the first display area, and the observation status is displayed in the first display area and the second display area in different ways.

[0020] Preferably, the processor causes the observation status to be displayed at all times in the second display area.

[0021] Preferably, the processor is a monitor having a third display area different from the first and second display areas, and the medical image is displayed in the third display area.

[0022] Another aspect of the present invention, a medical image processing method, is a medical image processing method for a medical image processing apparatus including a processor and a memory. The processor executes: a medical image acquisition step of acquiring a plurality of medical images in time series; an observation state determination step of determining an observation state of a small region unit of a subject based on the medical images; a recording step of recording the determination result in the memory; and a display step of causing a monitor to display an observation state display of the subject based on the determination result recorded in the memory when a change occurs in the observation state of the subject.

[0023] Another aspect of the present invention, a program, is a program for causing a medical image processing apparatus including a processor and a memory to execute a medical image processing method. The processor executes: a medical image acquisition step of acquiring a plurality of medical images in time series; an observation state determination step of determining an observation state of a small region unit of a subject based on the medical images; a recording step of recording the determination result in the memory; and a display step of causing a monitor to display an observation state display of the subject based on the determination result recorded in the memory when a change occurs in the observation state of the subject.

Advantages of the Invention

[0024] According to the present invention, since the observation state display of the subject is displayed on the monitor when a change occurs in the observation state of the subject, by displaying the observation state display of the subject at an appropriate timing, it is possible to effectively perform the display while suppressing the influence on the observation of the endoscopic image.

Brief Description of the Drawings

[0025] [Figure 1] FIG. 1 is an external view of an endoscopic system. [Figure 2] FIG. 2 is a block diagram showing a main configuration of the endoscopic system. [Figure 3] FIG.  3 is a functional block diagram of a medical image processing apparatus in an image processing unit. [Figure 4] FIG. 4 is a diagram showing main information recorded in a recording unit. [Figure 5]Figure 5 shows the structure of a neural network. [Figure 6] Figure 6 is a schematic diagram showing an example of the configuration of the intermediate layer. [Figure 7] Figure 7 is a flowchart showing a medical image processing method. [Figure 8] Figure 8 shows an example of an observation status display. [Figure 9] Figure 9 shows a modified example 1 of the observation state display. [Figure 10] Figure 10 shows a modified example 2 of the observation status display. [Figure 11] Figure 11 shows a modified example 3 of the observation status display. [Figure 12] Figure 12 shows a modified example 4 of the observation state display. [Figure 13] Figure 13 shows a modified example 5 of the observation status display. [Figure 14] Figure 14 shows a modified example 6 of the observation state display. [Figure 15] Figure 15 illustrates an example of a monitor having a main display area and a sub-display area. [Figure 16] Figure 16 illustrates an example of displaying different observation statuses in the main display area and the sub-display area. [Figure 17] Figure 17 illustrates an example of displaying different observation statuses in the main display area and the sub-display area. [Modes for carrying out the invention]

[0026] Hereinafter, preferred embodiments of the medical image processing apparatus, medical image processing method, program, and recording medium according to the present invention will be described with reference to the attached drawings.

[0027] <Configuration of the Endoscope System> Figure 1 is an external view of the endoscope system 10, and Figure 2 is a block diagram showing the main components of the endoscope system 10. As shown in Figures 1 and 2, the endoscope system 10 consists of an endoscope scope 100, an endoscope processor device 200, a light source device 300, and a monitor 400. The endoscope processor device 200 is equipped with the medical image processing device of the present invention.

[0028] <Configuration of Endoscope Scope> The endoscope scope 100 comprises a handheld control unit 102 and an insertion unit 104 connected to the handheld control unit 102. The operator (user) grasps and operates the handheld control unit 102 and inserts the insertion unit 104 into the body of the subject (living body) for observation. The handheld control unit 102 is also equipped with an air supply / water supply button 141, a suction button 142, function buttons 143 to which various functions can be assigned, and a shooting button 144 that accepts shooting instruction operations (still image, moving image). The insertion unit 104 is composed of a flexible part 112, a bending part 114, and a rigid tip part 116, in that order from the handheld control unit 102 side. That is, the bending part 114 is connected to the proximal end of the rigid tip part 116, and the flexible part 112 is connected to the proximal end of the bending part 114. The handheld control unit 102 is connected to the proximal end of the insertion unit 104. The user can change the orientation of the hard tip 116 up, down, left, and right by operating the hand-held control unit 102 to bend the curved section 114. The hard tip 116 is equipped with an imaging optical system 130, an illumination unit 123, a forceps channel 126, etc. (see Figures 1 and 2).

[0029] During observation and treatment, the operating unit 208 (see Figure 2) can be used to irradiate the illumination unit 123 with white light and / or narrowband light (one or more of red narrowband light, green narrowband light, blue narrowband light, and violet narrowband light). Additionally, by operating the air / water supply button 141, cleaning water can be released from a water supply nozzle (not shown) to clean the imaging lens 132 of the imaging optical system 130 and the illumination lenses 123A and 123B. A conduit (not shown) is connected to the forceps channel 126, which opens at the hard tip 116. A treatment instrument (not shown) for tumor removal, etc., is inserted through this conduit and can be moved forward and backward as needed to perform the necessary treatment on the subject.

[0030] As shown in Figures 1 and 2, a photographic lens 132 is disposed on the tip end face 116A of the hard tip portion 116. Behind the photographic lens 132 are a CMOS (Complementary Metal-Oxide Semiconductor) type image sensor 134, a drive circuit 136, and an AFE 138 (AFE: Analog Front End), and these elements output an image signal. The image sensor 134 is a color image sensor and comprises multiple pixels composed of multiple light-receiving elements arranged in a matrix (two-dimensional array) with a specific pattern arrangement (Bayer array, X-Trans® array, honeycomb array, etc.). Each pixel of the image sensor 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (photodiode, etc.). The photographic optical system 130 can generate a color image from pixel signals of three colors: red, green, and blue, or it can generate an image from pixel signals of any one or two of the red, green, and blue colors. The image sensor 134 may also be a CCD (Charge Coupled Device) type. Furthermore, each pixel of the image sensor 134 may be further equipped with a purple color filter corresponding to a purple light source 310V, and / or an infrared filter corresponding to an infrared light source.

[0031] The optical image of the subject is formed on the light-receiving surface (imaging surface) of the image sensor 134 by the imaging lens 132 and converted into an electrical signal, which is output to the endoscope processor device 200 via a signal cable (not shown) and converted into a video signal. As a result, the endoscopic image (medical image) of the subject is displayed on the monitor 400 connected to the endoscope processor device 200.

[0032] Furthermore, on the tip end face 116A of the hard tip portion 116, illumination lenses 123A and 123B of the illumination portion 123 are provided adjacent to the photographic lens 132. Behind the illumination lenses 123A and 123B, the ejection end of a light guide 170, which will be described later, is positioned. This light guide 170 is inserted into the insertion portion 104, the handheld operation portion 102, and the universal cable 106, and the incident end of the light guide 170 is positioned inside the light guide connector 108.

[0033] The user can sequentially capture a time-series of endoscopic images of the body by inserting or withdrawing the endoscope scope 100, configured as described above, into or from the body of the subject, while taking images at a predetermined frame rate.

[0034] <Configuration of the light source device> As shown in Figure 2, the light source device 300 consists of a light source 310 for illumination, an aperture 330, a focusing lens 340, and a light source control unit 350, and directs observation light onto the light guide 170. The light source 310 is equipped with a red light source 310R, a green light source 310G, a blue light source 310B, and a violet light source 310V, which emit narrowband light of red, green, blue, and violet, respectively, and can emit narrowband light of red, green, blue, and violet. The illuminance of the observation light from the light source 310 is controlled by the light source control unit 350, which can change (increase or decrease) the illuminance of the observation light as needed, and can also stop the illumination.

[0035] The light source 310 can emit red, green, blue, and violet narrowband light in any combination. For example, it is possible to emit red, green, blue, and violet narrowband light simultaneously to irradiate with white light (normal light) as observation light, or to emit one or two of them to irradiate with narrowband light (special light). The light source 310 may further include an infrared light source that emits infrared light (an example of narrowband light). Alternatively, a light source that emits white light and a filter that transmits white light and each of the narrowband lights may be used to irradiate with white light or narrowband light as observation light.

[0036] <Light source wavelength range> The light source 310 may be a light source that generates light in the white band, or a light source that generates light in multiple wavelength bands as white band light, or a light source that generates light in a specific wavelength band narrower than the white wavelength band. The specific wavelength band may be the blue band or green band of the visible range, or the red band of the visible range. If the specific wavelength band is the blue band or green band of the visible range, it may include a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and may have a peak wavelength within the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. If the specific wavelength band is the red band of the visible range, it may include a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and the light in the specific wavelength band may have a peak wavelength within the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm.

[0037] The light in the specific wavelength band described above may include wavelength bands in which oxyhemoglobin and deoxyhemoglobin have different absorption coefficients, and may have a peak wavelength in the wavelength band in which oxyhemoglobin and deoxyhemoglobin have different absorption coefficients. In this case, the specific wavelength band may include wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm, and may have a peak wavelength in the wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm.

[0038] Furthermore, the light generated by the light source 310 may include a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm, and may have a peak wavelength in the wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.

[0039] Furthermore, the light source 310 may be equipped with a light source that emits excitation light with a peak wavelength of 390 nm to 470 nm. In this case, it is possible to acquire endoscopic images that contain fluorescence information emitted by fluorescent substances in the subject (living body). When acquiring fluorescence images, fluorescent dyes (fluorescein, acridine orange, etc.) may be used.

[0040] The type of light source (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), etc.), wavelength, presence or absence of a filter, etc. of the light source 310 are preferably configured according to the type, part, purpose of observation, etc. Furthermore, when observing, it is preferable to combine and / or switch the wavelength of the observation light according to the type, part, purpose of observation, etc. When switching wavelengths, for example, the wavelength of the irradiated light may be switched by rotating a disc-shaped filter (rotary color filter) that is placed in front of the light source and has a filter that transmits or blocks light of a specific wavelength.

[0041] Furthermore, the image sensor used when implementing the present invention is not limited to a color image sensor with a color filter arranged for each pixel, such as image sensor 134, but may also be a monochrome image sensor. When using a monochrome image sensor, imaging can be performed sequentially by switching the wavelength of the observation light. For example, the wavelength of the emitted observation light may be switched sequentially between (purple, blue, green, red), or broadband light (white light) may be irradiated and the wavelength of the emitted observation light may be switched using a rotary color filter (red, green, blue, purple, etc.). Alternatively, one or more narrowband lights (green, blue, purple, etc.) may be irradiated and the wavelength of the emitted observation light may be switched using a rotary color filter (green, blue, purple, etc.). The narrowband lights may be two or more infrared lights with different wavelengths (first narrowband light, second narrowband light).

[0042] By connecting the light guide connector 108 (see Figures 1 and 2) to the light source device 300, the observation light emitted from the light source device 300 is transmitted via the light guide 170 to the illumination lenses 123A and 123B, and then illuminates the observation area from the illumination lenses 123A and 123B.

[0043] <Configuration of the Endoscope Processor System> The configuration of the endoscope processor unit 200 will be explained based on Figure 2. The endoscope processor unit 200 receives the image signal output from the endoscope scope 100 via the image input controller 202, performs the necessary image processing in the image processing unit 204, and outputs it via the video output unit 206. As a result, the endoscope image is displayed on the monitor 400. These processes are performed under the control of the CPU 210 (CPU: Central Processing Unit). The CPU 210 functions as the processor of the medical image processing device. The communication control unit 205 controls communication regarding the acquisition of medical images, etc., with the hospital information system (HIS), the hospital LAN (Local Area Network), and / or external systems and networks (not shown).

[0044] <Image Processing Functions> The image processing unit 204 can calculate feature quantities of the endoscopic image, process to enhance or reduce components in a specific frequency band, and process to enhance or make less prominent specific targets (areas of interest, blood vessels at a desired depth, etc.). The image processing unit 204 may also include a special light image acquisition unit (not shown) that acquires a special light image having information in a specific wavelength band based on a normal light image obtained by irradiating with white light, or light of multiple wavelength bands as white light. In this case, the signal in the specific wavelength band can be obtained by calculations based on RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) color information contained in the normal light image. Furthermore, the image processing unit 204 may also include a feature quantity image generation unit (not shown) that generates a feature quantity image by calculations based on at least one of a normal light image obtained by irradiating with white light, or light of multiple wavelength bands as white light, and a special light image obtained by irradiating with light of a specific wavelength band, and acquire and display the feature quantity image as an endoscopic image. The above processing is performed under the control of the CPU 210.

[0045] Furthermore, the image processing unit 204 has the following functions in the medical image processing device.

[0046] Figure 3 is a functional block diagram of the medical image processing device in the image processing unit 204. The image processing unit 204 comprises a medical image acquisition unit 220, an observation state determination unit 222, and a display control unit 224.

[0047] <Realization of functions by various processors> The functions of each part of the image processing unit 204 described above can be realized using various processors and recording media. These various processors include, for example, a CPU (Central Processing Unit), a general-purpose processor that executes software (programs) to realize various functions. Furthermore, these various processors also include a GPU (Graphics Processing Unit), a processor specialized for image processing, and a Programmable Logic Device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacturing. In cases of image learning and recognition as in the present invention, a configuration using a GPU is effective. Additionally, dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to execute particular processing, are also included in these various processors.

[0048] The functions of each part may be realized by a single processor, or by multiple processors of the same or different types (for example, multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Furthermore, multiple functions may be realized by a single processor. Examples of configuring multiple functions with a single processor include, firstly, as exemplified by computers, where one or more CPUs and software are combined to form a single processor, and this processor realizes multiple functions; and secondly, as exemplified by System-on-a-Chip (SoC), where a processor is used that realizes the functions of the entire system on a single IC (Integrated Circuit) chip. Thus, various functions are configured as hardware structures using one or more of the aforementioned processors. More specifically, the hardware structures of these various processors are electrical circuits combining circuit elements such as semiconductor elements. These electrical circuits may be electrical circuits that realize the aforementioned functions using logical operations such as logical OR, logical AND, logical negation, exclusive OR, and combinations thereof.

[0049] When the aforementioned processor or electrical circuit executes software (program), it stores a code readable by the computer (for example, the various processors and electrical circuits constituting the image processing unit 204, and / or a combination thereof) of the software to be executed in a non-temporary recording medium such as ROM 211 (ROM: Read Only Memory), and the computer refers to that software. The software stored in the non-temporary recording medium includes a program for executing the medical image processing method of the medical image processing apparatus according to the present invention and data used during execution. The code may be recorded in a non-temporary recording medium such as various magneto-optical recording devices or semiconductor memory instead of ROM 211. When processing using software, for example, RAM 212 (RAM: Random Access Memory) may be used as a temporary storage area, and data stored in an EEPROM (Electronically Erasable and Programmable Read Only Memory), not shown, may also be referred to. The recording unit 207 may be used as the "non-temporary recording medium".

[0050] Furthermore, ROM211 (ROM: Read Only Memory) is a non-volatile memory element (non-temporary recording medium) that stores computer-readable code for programs that cause the CPU210 and / or image processing unit204 to execute various image processing methods. RAM212 (RAM: Random Access Memory) is a memory element for temporary storage during various processes and can also be used as a buffer when acquiring images. The audio processing unit209 outputs audio and sound from speaker209A under the control of the CPU210.

[0051] The control unit 208 can be configured with devices such as a keyboard or mouse (not shown), and the user can issue execution instructions and specify the conditions necessary for execution via the control unit 208.

[0052] <Information recorded in the recording unit> Figure 4 shows the main information recorded in the recording unit 207. The recording unit (memory) 207 records medical images (endoscopic images) 260, the judgment results 262 of the observation status determination unit 222, etc. It also records information about a series of small regions to be observed during an examination performed using the endoscopic system 10. Here, the small regions of the subject are, for example, the various parts of an organ. Specifically, when performing an examination to observe the entire internal structure of the stomach, the small regions are the cardia, gastric fundus, gastric angle, gastric body (upper, middle, lower), antrum, anterior wall, posterior wall, greater curvature, and lesser curvature.

[0053] <Recognition unit using a neural network> The observation state determination unit 222 in the image processing unit 204 described above includes a recognition unit that can recognize small regions of a subject, configured using a trained model such as a neural network (a model trained using an image set composed of images of living organisms). The observation state determination unit 222 then determines whether the observation of a small region is complete or not on a small region basis, based on the position of the small region recognized by the recognition unit and the number of endoscopic images in which that small region was recognized. The configuration of the recognition unit included in the observation state determination unit 222 when a CNN (Convolutional Neural Network) is used as the neural network will be described below.

[0054] <Example of a recognition device configuration> Figure 5 shows the structure of a CNN232 (neural network). In the example shown in part (a) of Figure 5, the CNN232 has an input layer 232A, an intermediate layer 232B, and an output layer 232C. The input layer 232A takes an endoscopic image acquired by the medical image acquisition unit 220 as input and outputs feature quantities. The intermediate layer 232B includes a convolutional layer 234 and a pooling layer 235, and calculates other feature quantities by taking the feature quantities output by the input layer 232A as input. These layers have a structure in which multiple "nodes" are connected by "edges," and the weight coefficients applied to the input image are associated with the nodes and edges and stored in a weight coefficient memory unit (not shown). The values ​​of the weight coefficients change as learning progresses.

[0055] <Processing in the intermediate layer> The hidden layer 232B calculates features through convolution and pooling operations. The convolution operation performed in the convolution layer 234 is a process that obtains feature maps using a filter, and is responsible for feature extraction such as edge extraction from images. This convolution operation using a filter generates one channel (one image) of "feature map" for each filter. The size of the "feature map" decreases as convolution is performed in each layer if it is downscaled by convolution. The pooling operation performed in the pooling layer 235 is a process that shrinks (or expands) the feature map output by the convolution operation to create a new feature map, and is responsible for providing robustness so that the extracted features are not affected by translation, etc. The hidden layer 232B can be composed of one or more layers that perform these operations. Note that CNN232 may be composed without the pooling layer 235.

[0056] CNN232 may include a fully connected layer 236, as shown in the example in part (b) of Figure 5. The layer configuration of CNN232 is not limited to a sequence of one convolutional layer 234 and one pooling layer 235; any of the layers (for example, multiple convolutional layers 234) may be included consecutively.

[0057] Figure 6 is a schematic diagram showing an example of the configuration of the intermediate layer 232B of the CNN232 shown in Figure 5. In the first (1st) convolutional layer of the intermediate layer 232B, a convolution operation is performed between an image set composed of multiple endoscopic images and a filter F1. The image set consists of N images (N channels) with an image size of H vertically and W horizontally. When a normal optical image is input, the images that make up the image set are 3-channel images of R (red), G (green), and B (blue). Since the image set has N channels (N images), the filter F1 that is convolved with this image set will have a filter size of 5 × 5 × N if, for example, the filter is of size 5 (5 × 5). Through the convolution operation using this filter F1, a "feature map" of 1 channel (1 image) is generated for one filter F1. The filter F2 used in the second convolutional layer will have a filter size of 3 × 3 × M if, for example, the filter is of size 3 (3 × 3).

[0058] Similar to the first convolutional layer, the second through nth convolutional layers perform convolutional operations using filters F2 to Fn. The reason the size of the "feature map" in the nth convolutional layer is smaller than the size of the "feature map" in the second convolutional layer is because it has been downscaled by the preceding convolutional or pooling layers.

[0059] In the intermediate layer 232B, the convolutional layers closer to the input side perform low-order feature extraction (such as edge extraction), while higher-order feature extraction (extraction of features related to the shape, structure, etc. of the recognition target) is performed as you move closer to the output side.

[0060] Furthermore, the hidden layer 232B may include a layer that performs batch normalization in addition to the convolutional layer 234 and the pooling layer 235. Batch normalization is a process that normalizes the distribution of data in units of minibatches during training, and plays a role in speeding up training, reducing dependence on initial values, and suppressing overfitting.

[0061] The output layer 232C outputs the feature quantities calculated by the hidden layer 232B in a format suitable for recognition. The output layer 232C may also include a fully connected layer.

[0062] <Each step in medical image processing methods> Next, we will explain a medical image processing method using a medical image processing device.

[0063] Figure 7 is a flowchart of the medical image processing method. Each step will be explained below in accordance with Figure 6. In the following, we will explain the case where, when examining organ A, the subject of the examination, the observation of small regions, Area 1, Area 2, and Area 3, is performed area by area.

[0064] (Medical image acquisition steps) The medical image acquisition unit 220 sequentially acquires multiple medical images of organ A in chronological order (step S10). The recording unit 207 records that observations of areas 1, 2, and 3 of organ A have been performed, and initially, areas 1, 2, and 3 are recorded as incomplete.

[0065] (Observation status determination step) The observation state determination unit 222 determines the observation state of area 1, area 2, and area 3 of organ A based on the acquired medical images (step S11). The observation state determination unit 222 recognizes area 1, area 2, or area 3 in the medical images. Based on the recognition result, it determines the observation state of area 1, area 2, and area 3. For example, if area 1 is recognized as being in the center of 10 consecutive medical images in chronological order, the observation state determination unit 222 determines that observation of area 1 is complete.

[0066] (Recording step) The recording unit 207 records the result of the determination made by the observation status determination unit 222 (step S12). At the start of the inspection (initial state), areas 1, 2, and 3 are recorded as not yet observed. However, if the observation status determination unit 222 determines that each area has been observed, the record is updated to "observation completed".

[0067] (Display step) The display control unit 224 determines whether or not there has been a change in the observation state of the subject (step S13). Then, when the display control unit 224 determines that there has been a change in the observation state of the subject, it displays the observation state display 501 of the subject on the monitor 400 (step S14). Here, the point at which a change in the observation state of the subject occurs is when a change in the observation state is recorded in the recording unit 207 for one of the multiple small area units that are scheduled to be observed, as recorded in the recording unit 207. For example, this occurs when the recording unit 207 records that areas 1, 2, and 3 are not yet observed, but the observation state determination unit 222 determines that area 1 is now observed, and the observation state of area 1 changes to observed in the recording unit 207. The observation state display 501 of the subject is a display that informs the user of the observation state of the small area units that make up the subject being observed. By looking at the observation state display 501, the user can confirm whether the small areas of the subject being observed have been observed comprehensively.

[0068] Figure 8 shows an example of the observation status display 501 of organ A as displayed on monitor 400.

[0069] In the case shown in Figure 8, the endoscopic image 503 is displayed across the entire surface of the monitor 400. When the recording unit 207 updates area 3 from "observation incomplete" to "observation complete," the display control unit 224 overlays the observation status display 501 onto the endoscopic image 503 and displays it on the monitor 400. The observation status display 501 is a list display containing text information indicating areas that have been observed and areas that have not been observed. In the observation status display 501, areas 1 and 3 (labeled "area" in the figure) that have been observed are listed under "Complete," and area 2 that has not been observed is listed under "Not."

[0070] Returning to Figure 7, the display control unit 224 then continues to display the observation status indicator 501 until a predetermined display time has elapsed. The display time can be set as appropriate by the user. Preferably, the display time is set based on the time the user can confirm the observation status, as it is desirable that the display time be hidden once the user can confirm the observation status, allowing the user to observe the endoscope image 503. For example, the display time can be set to 10 seconds or 30 seconds. After the predetermined time has elapsed, the display control unit 224 hides the observation status indicator 501 (step S15).

[0071] Subsequently, the medical image acquisition unit 220 determines whether observation of all small region units has been completed (step S16), and since area 2 has not yet been observed, it acquires further medical images (step S10).

[0072] As described above, in the present invention, when observing the sub-regions of organ A, namely area 1, area 2, and area 3, the observation status display 501 is performed when a change occurs in the observation state of organ A. This allows for effective display while suppressing the impact on the observation of endoscopic images by performing the observation status display 501 when necessary and refraining from displaying it otherwise.

[0073] <Variations of the display of the observation status> In the example shown in Figure 8, we described a case where an observation status display 501 is shown, which displays the areas that have been observed and the areas that have not yet been observed using text information. However, the examples of the observation status display 501 are not limited to this. The display format of the observation status display 501 is not particularly limited as long as it can inform the user of the observation status of the subject being observed using text or graphics. Specific examples of the observation status display 501 are described below.

[0074] Figure 9 shows a modified example 1 of the observation status display 501. This example is an observation status display 501 with text information. In this example, the observation status display 501 displays only areas where observation is incomplete. Specifically, if area 2 is not yet observed, the text information "Area 2" is written below "Incomplete".

[0075] In this way, by displaying an observation status indicator 501 on the monitor 400 that shows only small areas that have not yet been observed, the user can clearly recognize the small areas that have not yet been observed, and comprehensive observation can be achieved. In the example in Figure 9, an example of displaying small areas that have not yet been observed on the observation status indicator 501 has been described, but small areas that have been observed may also be displayed on the observation status indicator 501. In this case, the user can clearly recognize the small areas that have been observed.

[0076] Figure 10 shows a modified example 2 of the observation status display 501. This example is an observation status display 501 with text information. In this example, the observation status display 501 displays all the small regions (areas 1 to 5) that make up the subject being observed as a list. The observation status display 501 then displays areas that have been observed and areas that have not been observed by changing the color of the text. Specifically, in the observation status display 501, areas 3 and 5 are not yet observed so they are displayed with the same color text, and areas 1, 2, and 4 are completed so they are displayed with the same color text. In this way, all the small regions are displayed as a list, and by adding information about whether the observation is complete or not to the text for each small region, the user can comprehensively recognize which areas have been observed and which have not been observed.

[0077] Figure 11 shows a modified example 3 of the observation status display 501. This example is an observation status display 501 using text information. In this example, the observation status display 501 displays all the small regions (areas 1 to 5) that make up the subject being observed as a list. The observation status display 501 displays "○" or "×" next to the text indicating areas that have been observed and areas that have not been observed. Specifically, in the observation status display 501, areas 3 and 5 are marked with "×" because they have not been observed, and areas 1, 2, and 4 are marked with "○" because they have been observed. In this way, all small regions are displayed as a list, and information on whether an area has been observed or not is displayed next to the text for each small region, allowing the user to comprehensively recognize areas that have been observed or not.

[0078] Figure 12 shows a modified example 4 of the observation status display 501. This example uses text information for the observation status display 501. In this example, the observation status display 501 shows the percentage of incomplete observation. Here, the percentage of incomplete observation is the percentage of sub-regions that have not been observed out of multiple sub-regions that are scheduled for observation. This percentage may be calculated using the number of sub-regions or the area of ​​the sub-regions. In the example shown in Figure 12, the percentage is shown in text, but the percentage may also be shown using a graphic such as a bar display. By showing the percentage of incomplete observation in this way, the user can clearly see the areas that have not been observed.

[0079] Figure 13 shows a modified example 5 of the observation status display 501. This example is an observation status display 501 using a subject model M that schematically represents the subject. The subject model M is a schematic representation of the stomach, which is the subject of observation. In the subject model M, different colors are used for areas 513 that have been observed and areas 511 that have not been observed, within the sub-region units. By providing information on observation completion and observation completion to the subject model M in this way, the user can recognize the location of the sub-regions that have been observed and the sub-regions that have not been observed within the subject. In the example shown in Figure 13, information on observation completion and observation completion is provided by changing the color of the regions in the subject model M, but the example is not limited to this example. For example, information on observation completion and observation completion may be provided to the subject model M by changing the density of the colors.

[0080] Figure 14 shows a modified example 6 of the observation status display 501. This example is an observation status display 501 using a subject model M that schematically represents the subject. In this example, only the small regions where observation is complete are shown in the subject model M. When observation of all small regions is complete, the entire subject model M (stomach) is displayed. In this way, by adding information about observation completion and observation incomplete to the subject model M, the user can recognize the location of the small regions where observation is complete and the location of the small regions where observation is incomplete in the subject. Note that in Figures 13 and 14, the subject model M is a cross-section of the stomach, which is the subject, but it is not limited to this. For example, the subject model M may be an unfolded view of the stomach, which is the subject.

[0081] The above description described an example of a monitor 400 having only a main display area, but the present invention is not limited to this. Modifications of the monitor are described below.

[0082] Figure 15 illustrates an example of a monitor 401 having a main display area and a sub-display area.

[0083] As shown in Figure 15, the monitor 401 has a main display area 402 and a sub-display area 404. The main display area 402 displays the endoscopic image 503 taken by the endoscope scope 100 in real time. The sub-display area 404 is set to be smaller than the main display area 402 and displays information such as shooting conditions, date, and patient information. Note that in Figure 15, the illustration of this information displayed in the sub-display area 404 is omitted. In the monitor 401, the observation status display 501 is displayed in both the main display area 402 and the sub-display area 404. In the main display area 402, as explained above, the observation status display 501 is displayed when a change occurs in the observation status of the subject, and the display ends when the display time has elapsed. Note that Figure 15 illustrates the case when the observation status display 501 in the main display area 402 is not displayed.

[0084] In the sub-display area 404, the observation status display 501B is always displayed. Because the sub-display area 404 has a small display area, the observation status display 501B is smaller than the observation status display 501A displayed in the main display area 402, but it serves as an aid for comprehensive observation. Note that the same observation status display may be displayed in the main display area 402 and the sub-display area 404, or different observation status displays may be displayed.

[0085] Figures 16 and 17 illustrate an example of displaying different observation statuses in the main display area and the sub-display area.

[0086] In the example shown in Figure 16, the main display area 402 shows a detailed observation status display 501A with sub-areas listed. In addition, the sub-display area 404 shows the observation status display 501B of the subject model M.

[0087] In the example shown in Figure 17, the main display area 402 shows a detailed observation status display 501A, which lists the sub-regions. In addition, the sub-display area 404 shows an observation status display 501B, which indicates the percentage of observations that are not yet complete.

[0088] As shown in Figures 16 and 17, the main display area 402 displays an observation status display 501A, which shows a list of small areas, allowing for a detailed understanding of the observation status of each small area, whenever the observation status of the subject changes. This allows the user to grasp the detailed observation status of the subject when the number of small areas that have been observed increases. In addition, the sub-display area 404 constantly displays an observation status display 501B, which shows the subject model M and the percentage of incomplete observations. This allows the user to grasp the observation status of the subject in general terms, even when the observation status display 501A is not displayed in the main display area 402. In the example described above, a monitor 401 having a main display area (first display area) 402 and a sub-display area (second display area) 404 has been described, but the observation status display 501 may also be displayed on a monitor having, for example, a third display area. Furthermore, the observation status display 501 may be displayed on multiple monitors.

[0089] <Other examples> The above description concerns a mode in which the observation status display 501 is displayed for a predetermined period of time. However, the mode in which the displayed observation status display 501 is hidden is not limited to this.

[0090] For example, the display control unit 224 may hide the observation status display 501 based on a command entered by the user via the manual operation unit 102 (user operation reception unit) (see Figure 1). Furthermore, the display control unit 224 may redisplay the observation status display 501, which has been hidden, based on a command entered by the user via the manual operation unit 102 (user operation reception unit). In this way, by controlling the display and hiding of the observation status display 501 using the manual operation unit 102, the user can check the observation status display at a desired timing.

[0091] Although examples of the present invention have been described above, it goes without saying that the present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention. [Explanation of Symbols]

[0092] 10: Endoscopy System 100: Endoscope 102: Handheld control unit 104: Insertion part 106: Universal Cable 108: Light guide connector 112: Soft part 114: Curved section 116: Hardened tip 116A: Tip side end surface 123: Lighting Department 123A: Illumination lens 123B: Illumination lens 126: Forceps channel 130: Imaging Optical System 132: Photographing lens 134: Image sensor 136: Drive Circuit 141: Air / Water Supply Button 142: Suction button 143: Function button 144: Shoot button 170: Light Guide 200: Endoscope processor device 202: Image Input Controller 204: Image Processing Unit 205: Communication Control Unit 206: Video output section 207: Records Department 208 :Operation unit 209: Audio Processing Unit 209A: Speaker 210: CPU 211 :ROM 212: RAM 220: Medical Image Acquisition Department 222: Observation status determination unit 224: Display Control Unit 232A: Input layer 232B: Middle layer 232C: Output layer 234: Convolutional layer 235: Pooling layer 236 :Fully connected layer 300: Light source device 310 :Light source 310B: Blue light source 310G: Green light source 310R: Red light source 310V: Violet light source 330: Aperture 340: Focusing lens 350: Light source control unit 400: Monitor

Claims

1. A medical image processing device equipped with a processor, The aforementioned processor, By acquiring multiple medical images in chronological order, Based on the aforementioned medical images, the observation state is recognized for each of several predetermined sub-regions located at different positions within the organ that is the subject of the study. Based on the results of the recognition, if the observation of each of the multiple sub-regions is completed, it is determined that the observation is complete; if the observation is not completed, it is determined that the observation is incomplete. For a subject model schematically representing an organ, information indicating that the observation status of each of the multiple sub-regions is complete is displayed based on the result of the above determination. When the observation status of any one of the multiple sub-regions changes from "observation incomplete" to "observation completed," information indicating that the observation status of the one sub-region is complete is reflected and updated in the region of the subject model that corresponds to the part of the organ within the one sub-region. The processor determines whether the observation of one sub-region is complete based on the number of consecutive images in which one of the plurality of sub-regions is recognized in the time-series medical images, and the position of the one sub-region in the medical images. Medical image processing equipment.

2. The medical image processing apparatus according to claim 1, wherein the processor hides the information displayed on the subject model after a predetermined time has elapsed.

3. It also includes a user operation reception section, The medical image processing apparatus according to claim 1 or 2, wherein the processor displays or hides the information based on a command from the user operation reception unit.

4. The medical image processing apparatus according to any one of claims 1 to 3, wherein the processor displays the information on a monitor as text information.

5. The medical image processing apparatus according to claim 4, wherein the processor adds information regarding the completion or incompleteness of the observation of the small region unit of the subject to the character information and displays it as such information.

6. The medical image processing apparatus according to any one of claims 1 to 5, wherein the processor displays on the monitor as information only whether the observation of the small region unit of the subject is complete or incomplete.

7. The medical image processing apparatus according to any one of claims 1 to 6, wherein the processor causes the medical image to be displayed on a monitor and the information to be superimposed on the medical image.

8. The medical image processing apparatus according to any one of claims 1 to 7, wherein the processor is a monitor having a first display area for displaying the medical image and a second display area smaller than the first display area for displaying the subject model, and the information is displayed in the first display area and the second display area in different manners.

9. The medical image processing apparatus according to claim 8, wherein the processor continuously displays the information in the second display area.

10. The medical image processing apparatus according to claim 9, wherein the processor is a monitor having a third display area different from the first display area and the second display area, and the medical image is displayed in the third display area.

11. The medical image processing apparatus according to any one of claims 1 to 10, wherein the processor displays the observed region and the unobserved region in the subject model with a straight line.

12. The medical image processing apparatus according to any one of claims 1 to 11, wherein the processor recognizes a plurality of predetermined subregions located at different relative positions within the organs of the subject in the medical image using a trained model that has been trained to recognize each of the plurality of subregions as a subregion unit.

13. A medical image processing method for a medical image processing device equipped with a processor, The aforementioned processor, Steps include acquiring multiple medical images in chronological order, Based on the aforementioned medical image, the steps include recognizing the observation state for each of several predetermined sub-regions located at different positions within the organ that is the subject, Based on the results of the recognition, the observation is determined to be complete if the observation of each of the multiple sub-regions is completed, and to be determined to be incomplete if the observation is not completed. The steps include displaying information indicating that the observation status of each of the multiple sub-regions is complete, based on the result of the determination, for a subject model that schematically represents an organ, When the observation status of any one of the plurality of subregions changes from "observation incomplete" to "observation completed," the following steps are performed: updating the region on the subject model that corresponds to the part of the organ within the subregion by reflecting information indicating that the observation status of the one subregion is complete; The determination step involves determining whether the observation of one sub-region is complete based on the number of images in which one of the multiple sub-regions is recognized consecutively in the time-series medical images, and the position of the one sub-region in the medical images. Medical image processing methods.

14. A program that causes a medical image processing device equipped with a processor to execute a medical image processing method, Steps include acquiring multiple medical images in chronological order, Based on the aforementioned medical image, the steps include recognizing the observation state for each of several predetermined sub-regions located at different positions within the organ that is the subject, Based on the results of the recognition, the observation is determined to be complete if the observation of each of the multiple sub-regions is completed, and to be determined to be incomplete if the observation is not completed. The steps include displaying information indicating that the observation status of each of the multiple sub-regions is complete, based on the result of the determination, for a subject model that schematically represents an organ, When the observation status of any one of the plurality of sub-regions changes from "observation incomplete" to "observation completed," the step of updating the region on the subject model that corresponds to the part of the organ within the sub-region by reflecting information indicating that the observation status of the one sub-region is complete, A program that causes the processor to execute a medical image processing method including, The determination step involves determining whether the observation of one sub-region is complete based on the number of images in which one of the multiple sub-regions is recognized consecutively in the time-series medical images, and the position of the one sub-region in the medical images. program.

15. A non-temporary and computer-readable recording medium on which the program described in claim 14 is recorded.

Images