Medical image processing apparatus, medical image processing method, and recording medium

Abstract

The invention provides a medical image processing device, a medical image processing method, and a recording medium, which can effectively display observation state display related to the comprehensiveness of observation while suppressing the visibility reduction of endoscopic images. A medical image processing device is provided with a processor (210) and a memory (207), the processor (210) acquiring a plurality of medical images in time series, determining the observation state of a small area unit of a subject on the basis of the medical images, recording the determination result in the memory (207), and at the time when the observation state of the subject changes, determining the observation state of the small area unit of the subject. A monitor (400) displays an observation state display of the subject based on the result of the determination recorded in the memory (207).

Description

[0001] This application is a divisional application of application number "202180009915.7", filed on January 25, 2021, entitled "Medical Image Processing Apparatus, Medical Image Processing Method and Recording Medium". Technical Field

[0002] This invention relates to medical image processing apparatus, medical image processing methods and programs. Background Technology

[0003] Traditionally, examinations using endoscopic systems have required a comprehensive view of the organs and other areas of the patient being examined.

[0004] Patent Document 1 describes a technique for preventing missed images during examinations using an endoscopic system. In the technique described in Patent Document 1, a monitor displays a map image showing the areas of the organ being photographed and the areas not photographed as a notification display.

[0005] Previous technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2018-50890 Summary of the Invention

[0008] The technical problem to be solved by the invention

[0009] Here, typically, the endoscopic image captured in real-time during the examination is displayed on the main display area of ​​the endoscope system's monitor. Therefore, when the notification display, as described in Patent Document 1, is displayed on the main display area, it overlaps with the endoscopic image, reducing the visibility of the endoscopic image. Furthermore, when the notification display is displayed on a sub-display area of ​​the monitor, the display area is small, and the visibility of the notification display is reduced. On the other hand, while displaying the notification on a sub-monitor different from the main monitor is also considered, there is a problem that the user cannot focus on the endoscopic image displayed on the main monitor during the examination.

[0010] Patent document 1 mentioned above does not mention a display method that takes into account the visibility of the endoscopic image or the visibility of the notification display (map image).

[0011] The present invention was made in view of the following circumstances, and its object is to provide a medical image processing apparatus, medical image processing method and program that suppresses the reduction of visibility of endoscopic images and effectively displays the observation status display related to the comprehensiveness of the observation.

[0012] means for solving technical problems

[0013] A medical image processing apparatus according to one aspect of the present invention for achieving the above-mentioned objectives includes a processor and a memory, wherein the processor acquires multiple medical images in a time sequence, determines the observation state of a small region unit of the subject based on the medical images, records the determination result in the memory, and causes a monitor to display the observation state of the subject based on the determination result recorded in the memory when the observation state of the subject changes.

[0014] According to this method, when the observation status of the subject changes, the observation status of the subject is displayed on the monitor. Therefore, by displaying the observation status of the subject at appropriate times, the influence on the observation of the endoscopic image can be suppressed, and the display can be performed effectively.

[0015] Preferably, the processor sets the observation status displayed on the monitor to non-display after a predetermined time has elapsed.

[0016] Preferably, it also includes a user operation receiving unit, and the processor sets the observation status display to display or not display based on instructions from the user operation receiving unit.

[0017] Preferably, when determining the observation status of a small region, the processor determines that the observation is complete if the observation of the small region is completed, and determines that the observation is incomplete if the observation is not completed.

[0018] Preferably, the processor causes the monitor to display the observation status display via character information.

[0019] Preferably, the processor assigns character information related to whether the observation of a small area unit of the subject is completed or not, and displays it as an observation status display.

[0020] Preferably, the processor displays the observation status by schematically representing a subject model of the subject.

[0021] Preferably, the processor assigns information related to whether the observation of a small region unit of the subject is complete or incomplete to the subject model and displays it as an observation status display.

[0022] Preferably, the processor displays only the observed small area units of the subject as completed or only the incomplete display as the observation status display.

[0023] Preferably, the processor causes the monitor to display the medical image and causes the observation status display to be overlaid on the medical image.

[0024] Preferably, the processor is a monitor having a first display area and a second display area smaller than the first display area, such that the first display area and the second display area display the observation status in different ways.

[0025] Preferably, the processor ensures that the second display area always displays the observation status display.

[0026] Preferably, the processor is a monitor having a third display area different from the first and second display areas, so that the third display area displays medical images.

[0027] Another aspect of the present invention is a medical image processing method of a medical image processing apparatus having a processor and a memory, wherein the processor performs: a medical image acquisition step, acquiring multiple medical images in a time sequence; an observation state determination step, determining the observation state of a small region unit of a subject based on the medical images; a recording step, recording the determination result in the memory; and a display step, causing a monitor to display the observation state of the subject based on the determination result recorded in the memory when the observation state of the subject changes.

[0028] Another aspect of the present invention is a program that enables a medical image processing apparatus equipped with a processor and a memory to execute a medical image processing method, wherein the processor executes: a medical image acquisition step, acquiring multiple medical images in a time sequence; an observation state determination step, determining the observation state of a small region unit of the subject based on the medical images; a recording step, recording the determination result in a memory; and a display step, causing a monitor to display the observation state of the subject based on the determination result recorded in the memory when the observation state of the subject changes.

[0029] Invention Effects

[0030] According to the present invention, since the observation status of the subject is displayed on the monitor at the moment when the observation status of the subject changes, the influence on the observation of the endoscopic image can be suppressed and the display can be performed effectively by displaying the observation status of the subject at the appropriate time. Attached Figure Description

[0031] Figure 1 This is an external view of the endoscope system.

[0032] Figure 2 This is a block diagram showing the main structural components of an endoscope system.

[0033] Figure 3 This is a functional block diagram of the medical image processing device in the image processing department.

[0034] Figure 4 It is a diagram that represents the main information recorded in the recording section.

[0035] Figure 5 It is a diagram representing the structure of a neural network.

[0036] Figure 6 This is a schematic diagram illustrating a structural example of an intermediate layer.

[0037] Figure 7 This is a flowchart illustrating a medical image processing method.

[0038] Figure 8 This is a diagram showing an example of the observed state.

[0039] Figure 9 This is a diagram showing a variation of Example 1, displaying the observation state.

[0040] Figure 10 This is a diagram representing a variation of Example 2, showing the observed state.

[0041] Figure 11 This is a diagram showing the modified example 3 as displayed in the observation state.

[0042] Figure 12 This is a diagram representing the modified example 4, showing the observation state.

[0043] Figure 13 This is a diagram representing the modified example 5, showing the observation state.

[0044] Figure 14 This is a diagram representing a variation of Example 6, showing the observed state.

[0045] Figure 15 This diagram illustrates an example of a monitor with a main display area and sub-display areas.

[0046] Figure 16 This diagram illustrates examples of displaying different viewing states in the main display area and the sub-display area.

[0047] Figure 17 This diagram illustrates examples of displaying different viewing states in the main display area and the sub-display area. Detailed Implementation

[0048] The preferred embodiments of the medical image processing apparatus, medical image processing method and program involved in the present invention will now be described with reference to the accompanying drawings.

[0049] <Structure of an Endoscopic System>

[0050] Figure 1 This is an external view of the endoscope system 10. Figure 2This is a block diagram showing the main structural components of the endoscope system 10. For example... Figure 1 , 2 As shown, the endoscope system 10 comprises an endoscope observer 100, an endoscope processor device 200, a light source device 300, and a monitor 400. Furthermore, the endoscope processor device 200 is equipped with the medical image processing device of the present invention.

[0051] <Structure of the Endoscopic Observation Device>

[0052] The endoscope observation device 100 includes a handheld operating unit 102 and an insertion unit 104 connected to the handheld operating unit 102. The operator (user) holds the handheld operating unit 102 and operates it to insert the insertion unit 104 into the body of the subject (organism) for observation. Additionally, the handheld operating unit 102 is equipped with an air / water supply button 141, a suction button 142, function buttons 143 assigned various functions, and a recording button 144 for receiving recording instructions (still image, moving image). The insertion unit 104, starting from the handheld operating unit 102 side, consists of a flexible part 112, a curved part 114, and a rigid tip part 116. Specifically, the curved part 114 is connected to the base of the rigid tip part 116, and the flexible part 112 is connected to the base of the curved part 114. The handheld operating unit 102 is connected to the base of the insertion unit 104. The user can bend the bending part 114 by operating the hand-held operating unit 102, and change the orientation of the top rigid part 116 by moving it up, down, left, and right. The top rigid part 116 houses a photographic optical system 130, an illumination unit 123, a clamping mouth 126, etc. (see reference). Figure 1 , 2 ).

[0053] During observation and handling, the operation unit 208 (see reference) is used. Figure 2 The operation allows white light and / or narrowband light (one or more of red, green, blue, and purple narrowband light) to be emitted from the illumination lenses 123A and 123B of the illumination unit 123. Additionally, cleaning water can be discharged from a water nozzle (not shown) via the operation of the air / water supply button 141 to clean the imaging lens 132 and illumination lenses 123A and 123B of the imaging optical system 130. A (not shown) conduit is connected to the forceps opening 126 of the rigid end portion 116. A (not shown) treatment instrument for removing tumors, etc., is inserted into this conduit, allowing for appropriate insertion and removal to perform necessary treatment on the subject.

[0054] like Figure 1 and Figure 2As shown, a photographic lens 132 is mounted on the top side face 116A of the top rigid portion 116. A CMOS (Complementary Metal-Oxide Semiconductor) type image sensor 134, a driving circuit 136, and an AFE 138 (AFE: Analog Front End) are mounted inside the photographic lens 132, and image signals are output using these elements. The image sensor 134 is a color image sensor, comprising multiple pixels composed of multiple light-receiving elements arranged in a matrix (two-dimensional arrangement) using a specific pattern (Bayer arrangement, X-Trans arrangement, honeycomb arrangement, etc.). Each pixel of the image sensor 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (photodiode, etc.). The photographic optical system 130 can generate a color image from pixel signals of red, green, and blue, or it can generate an image from pixel signals of any one or two of the three colors of red, green, and blue. Furthermore, the image sensor 134 can also be a CCD (Charge Coupled Device) type. In addition, each pixel of the image sensor 134 can also have a violet color filter corresponding to the violet light source 310V and / or an infrared filter corresponding to the infrared light source.

[0055] The optical image of the subject is imaged onto the light-receiving surface (image-capturing surface) of the imaging element 134 using the photographic lens 132 and converted into an electrical signal. This signal is then output to the endoscope processor device 200 via a signal cable (not shown) and converted into a video signal. As a result, the endoscopic image (medical image) of the subject is displayed on a monitor 400 connected to the endoscope processor device 200.

[0056] Additionally, on the top side face 116A of the top rigid portion 116, illumination lenses 123A and 123B are provided adjacent to the photographic lens 132 and have illumination portions 123. The emission end of a light guide 170 (described later) is provided on the inner side of the illumination lenses 123A and 123B. The light guide 170 is inserted into the insertion portion 104, the hand operation portion 102, and the universal cable 106. The incident end of the light guide 170 is disposed in the light guide connector 108.

[0057] While inserting or removing the endoscopic observer 100 of the above structure into or from the biological body being examined, the user takes photographs at a fixed frame rate, thereby enabling the sequential capture of time-series endoscopic images within the biological body.

[0058] <Structure of the Light Source Device>

[0059] like Figure 2As shown, the light source device 300 comprises a light source 310 for illumination, an aperture 330, a condenser lens 340, and a light source control unit 350, etc., so that the observation light is incident on the light guide 170. The light source 310 is equipped with a red light source 310R, a green light source 310G, a blue light source 310B, and a violet light source 310V, which respectively illuminate narrow bands of red, green, blue, and violet light, and can illuminate narrow bands of red, green, blue, and violet light. The illuminance of the observation light of the light source 310 is controlled by the light source control unit 350, which can change (increase or decrease) the illuminance of the observation light and stop the illumination as needed.

[0060] The light source 310 can emit light by arbitrarily combining narrowband lights of red, green, blue, and violet. For example, it can simultaneously emit narrowband lights of red, green, blue, and violet to illuminate white light (ordinary light) as observation light, or it can emit narrowband light (special light) by emitting any one or two types of light. The light source 310 can also be equipped with an infrared light source that illuminates infrared light (an example of narrowband light). Alternatively, it can use a light source that illuminates white light and filters that transmit white light and each narrowband light to illuminate either white light or narrowband light as observation light.

[0061] <wavelength band of light source>

[0062] Light source 310 can be a light source that generates light in the white wavelength range, or a light source that generates light in multiple wavelength ranges as white wavelength light, or a light source that generates light in a specific wavelength range narrower than the white wavelength range. The specific wavelength range can be the blue or green wavelength range within the visible range, or the red wavelength range within the visible range. When the specific wavelength range is the blue or green wavelength range within the visible range, it can include wavelengths between 390 nm and 450 nm, or between 530 nm and 550 nm, and has a peak wavelength within the wavelength range between 390 nm and 450 nm, or between 530 nm and 550 nm. Furthermore, when the specific wavelength range is the red wavelength range within the visible range, it can include wavelengths between 585 nm and 615 nm, or between 610 nm and 730 nm, and the light in the specific wavelength range has a peak wavelength within the wavelength range between 585 nm and 615 nm, or between 610 nm and 730 nm.

[0063] The aforementioned specific wavelength bands may also include bands with different absorption coefficients in oxidized hemoglobin and deoxyhemoglobin, and the light of the specific wavelength band has peak wavelengths in the bands with different absorption coefficients in oxidized hemoglobin and deoxyhemoglobin. In this case, the specific wavelength bands may include bands of 400±10 nm, 440±10 nm, 470±10 nm, or bands of 600 nm to 750 nm, and the light of the specific wavelength band has peak wavelengths in the bands of 400±10 nm, 440±10 nm, 470±10 nm, or bands of 600 nm to 750 nm.

[0064] In addition, the light generated by the light source 310 may also include a wavelength range of 790nm to 820nm or 905nm to 970nm, and the light generated by the light source 310 has a peak wavelength in the wavelength range of 790nm to 820nm or 905nm to 970nm.

[0065] Alternatively, the light source 310 may also be equipped with an excitation light source that illuminates a peak wavelength of 390 nm or higher and 470 nm or lower. In this case, endoscopic images containing information about the fluorescence emitted by fluorescent substances within the subject (organism) can be acquired. When acquiring fluorescence images, fluorescent dyes (fluorescein, acridine orange, etc.) can also be used.

[0066] The type of light source 310 (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), wavelength, and presence or absence of filters are preferably configured according to the type and location of the subject, the purpose of observation, etc. Furthermore, during observation, it is preferable to combine and / or switch the wavelength of the observation light according to the type and location of the subject, the purpose of observation, etc. When switching wavelengths, for example, the wavelength of the illuminated light can be switched by rotating a circular plate-shaped filter (rotating color filter) positioned in front of the light source and equipped with filters that transmit or block light of a specific wavelength.

[0067] Furthermore, the imaging element used in implementing this invention is not limited to a color imaging element like imaging element 134, which provides a color filter for each pixel; it can also be a monochrome imaging element. When using a monochrome imaging element, the wavelengths of the observation light can be switched sequentially to capture images in a surface order (color order). For example, the wavelengths of the emitted observation light can be switched sequentially between narrowband lights (violet, blue, green, red), or broadband light (white light) can be irradiated and the wavelengths of the emitted observation light can be switched by rotating the color filters (red, green, blue, violet, etc.). Alternatively, one or more narrowband lights (green, blue, violet, etc.) can be irradiated and the wavelengths of the emitted observation light can be switched by rotating the color filters (green, blue, violet, etc.). The narrowband light can be infrared light of two or more wavelengths with different wavelengths (first narrowband light, second narrowband light).

[0068] By using the optical guide connector 108 (reference) Figure 1 , 2 The light source device 300 is connected to the observation light source device 300. The observation light emanating from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and then irradiates the observation area from the illumination lenses 123A and 123B.

[0069] <Structure of the Endoscopic Processor Device>

[0070] based on Figure 2 The structure of the endoscope processor device 200 will be described. The endoscope processor device 200 receives the image signal output from the endoscope observer 100 via the image input controller 202, performs necessary image processing in the image processing unit 204, and outputs the signal via the video output unit 206. The endoscope image is then displayed on the monitor 400. This processing is performed under the control of the CPU 210 (Central Processing Unit). Furthermore, the CPU 210 functions as the processor of the medical image processing device. The communication control unit 205 performs communication control regarding the acquisition of medical images, etc., with a hospital information system (HIS) or a hospital LAN (Local Area Network) not shown, and / or external systems or networks.

[0071] <Functions of the Image Processing Unit>

[0072] The image processing unit 204 is capable of calculating feature quantities of endoscopic images, emphasizing or reducing components of specific wavelength bands, emphasizing specific objects (areas of interest, blood vessels at desired depths, etc.) or making them less prominent. The image processing unit 204 may also include a special light image acquisition unit (not shown), which acquires a special light image with information of specific wavelength bands based on a normal light image obtained by illuminating light of the white wavelength band or light of multiple wavelength bands as white wavelength band light. In this case, the signal of the specific wavelength band can be obtained by calculation based on the color information of RGB (R: red, G: green, B: blue) or CMY (C: blue-green, M: magenta, Y: yellow) contained in the normal light image. Alternatively, the image processing unit 204 may also include a feature quantity image generation unit (not shown), which acquires and displays a feature quantity image as an endoscopic image. This feature quantity image generation unit generates the feature quantity image by calculation based on at least one of a normal light image obtained by illuminating light of the white wavelength band or light of multiple wavelength bands as white wavelength band light, and a special light image obtained by illuminating light of a specific wavelength band. Furthermore, the above processing is performed under the control of CPU210.

[0073] Furthermore, as described below, the image processing unit 204 has all the functions of a medical image processing device.

[0074] Figure 3 This is a functional block diagram of the medical image processing apparatus in the image processing unit 204. The image processing unit 204 includes a medical image acquisition unit 220, an observation state determination unit 222, and a display control unit 224.

[0075] <Implementation of the functions of various processors>

[0076] The functions of each part of the image processing unit 204 described above can be implemented using various processors and recording media. Among these processors are, for example, general-purpose processors (CPUs) that implement various functions by executing software (programs). Additionally, among these processors are processors specifically designed for image processing, such as GPUs (Graphics Processing Units), FPGAs (Field Programmable Gate Arrays), and other processors whose circuit structures can be modified after manufacturing—that is, programmable logic devices (PLDs). In the case of image learning or recognition as described in this invention, the use of a GPU structure is effective. Furthermore, processors with circuit structures specifically designed for performing specific processing such as ASICs (Application Specific Integrated Circuits), i.e., dedicated circuits, are also included among these various processors.

[0077] The functions of each part can be implemented by a single processor, or by multiple processors of the same or different types (e.g., multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Furthermore, a single processor can implement multiple functions. As examples of using a single processor to implement multiple functions, firstly, there are forms such as computers, where a combination of one or more CPUs and software is used to construct a single processor, which then implements multiple functions. Secondly, there are forms such as Systems on a Chip (SoC), where a single IC (Integrated Circuit) chip is used to implement the overall system functions. In this way, one or more of the above-mentioned processors are used as hardware structures to construct various functions. More specifically, the hardware structures of these various processors are circuits that combine semiconductor elements and other circuit components. These circuits can also be circuits that implement the above functions using logical operations such as AND, product, NOT, XOR, and combinations thereof.

[0078] When the aforementioned processor or circuit executes the software (program), the computer (e.g., various processors or circuits constituting the image processing unit 204, and / or combinations thereof) stores the readable code of the executed software in a non-transitory recording medium such as ROM 211 (ReadOnly Memory), and the computer refers to this software. The software pre-stored in the non-transitory recording medium includes a program for executing the medical image processing method of the medical image processing apparatus according to the present invention and data used during execution. Alternatively, the code may not be recorded in ROM 211, but rather in a non-transitory recording medium such as various optical-magnetic recording devices or semiconductor memories. When processing using software, for example, RAM 212 (Random Access Memory) may be used as a temporary storage area, and data stored in EEPROM (Electrically Erasable and Programmable Read Only Memory, not shown) may also be referenced. The recording unit 207 may also be used as a "non-transitory recording medium".

[0079] Additionally, ROM 211 (Read Only Memory) is a non-volatile storage element (non-temporary recording medium) that stores computer-readable code that enables the CPU 210 and / or image processing unit 204 to execute various image processing methods. RAM 212 (Random Access Memory) is a storage element used for temporary storage during various processing steps, and can also be used as a buffer during image acquisition. The sound processing unit 209 outputs sound and voice from the speaker 209A under the control of the CPU 210.

[0080] The operation unit 208 may consist of devices such as a keyboard or mouse (not shown), and the user can execute instructions or specify the conditions required for execution through the operation unit 208.

[0081] <Information recorded in the records department>

[0082] Figure 4 This is a diagram representing the main information recorded in the recording unit 207. The recording unit (memory) 207 records medical images (endoscopic images) 260, the determination result 262 of the observation status determination unit 222, and so on. Furthermore, during examinations performed using the endoscope system 10, information related to a series of small areas to be observed is recorded. Here, the small areas of the subject are, for example, various parts of organs. Specifically, when examining all parts of the stomach, the small areas are the cardia, fundus, angular region, body (upper, middle, and lower parts), vestibule, anterior wall, posterior wall, greater curvature, and lesser curvature.

[0083] <Neural Network-Based Recognition Department>

[0084] The observation state determination unit 222 in the image processing unit 204 described above includes a recognizer that can identify small regions of the subject, constructed using a learned model such as a neural network (a model learned using a group of images obtained from photographs of an organism). Furthermore, the observation state determination unit 222 determines whether the observation of a small region has been completed by measuring the location of the small region identified by the recognizer and the number of endoscopic images that identified that small region. The structure of the recognizer included in the observation state determination unit 222 when using a CNN (Convolutional Neural Network) as the neural network will be described below.

[0085] <Example of a Recognizer Structure>

[0086] Figure 5 This is a diagram representing the structure of a CNN232 (neural network). Figure 5 In the example shown in part (a), the CNN232 has an input layer 232A, an intermediate layer 232B, and an output layer 232C. The input layer 232A receives the endoscopic image acquired by the medical image acquisition unit 220 and outputs feature values. The intermediate layer 232B includes a convolutional layer 234 and a pooling layer 235, which compute other feature values ​​by inputting the feature values ​​output from the input layer 232A. These layers are structured as multiple "nodes" connected by "edges," and the weight coefficients applied to the input image are associated with the nodes and edges and stored in a weight coefficient storage unit (not shown). The values ​​of the weight coefficients change as learning progresses.

[0087] <Processing of the intermediate layer>

[0088] Intermediate layer 232B computes feature values ​​through convolution and pooling operations. The convolution operation performed in convolutional layer 234 is a process that uses a filter to obtain feature maps, responsible for extracting features such as edges from the image. By using the convolution operation of this filter, a "feature map" is generated for each filter channel (one image). The size of the "feature map" decreases as convolution is performed in each layer, being downscaled. The pooling process performed in pooling layer 235 is a process that downscales (or upscales) the feature map output by the convolution operation to obtain a new feature map, providing robustness to prevent the extracted features from being affected by parallel shifts, etc. Intermediate layer 232B can consist of one or more layers performing these processes. Alternatively, CNN232 can also be configured without pooling layer 235.

[0089] CNN232 can also be like Figure 5The example shown in part (b) includes a fully connected layer 236. The layer structure of CNN232 is not limited to the case where convolutional layers 234 and pooling layers 235 are repeated one after another, but can also include multiple arbitrary layers (e.g., convolutional layers 234) consecutively.

[0090] Figure 6 It means Figure 5 The diagram illustrates an example of the structure of the intermediate layer 232B of the CNN232. In the initial (first) convolutional layer of the intermediate layer 232B, a convolution operation is performed between an image group consisting of multiple endoscopic images and a filter F1. The image group consists of N images (N channels) with image dimensions H and W. In the case of a normal light image as input, the images constituting the image group are images with three channels: R (red), G (green), and B (blue). Since the image group has N channels (N images), the filter F1 used for the convolution operation with this image group has a filter size of 5×5×N, for example, in the case of a filter with a size of 5 (5×5). By using the convolution operation of this filter F1, a "feature map" of one channel (one image) is generated for one filter F1. The filter F2 used in the second convolutional layer has a filter size of 3×3×M, for example, in the case of a filter with a size of 3 (3×3).

[0091] Similar to the first convolutional layer, convolution operations using filters F2 to Fn are performed in the second to nth convolutional layers. The size of the "feature map" in the nth convolutional layer is smaller than the size of the "feature map" in the second convolutional layer because it is reduced by the convolutional or pooling layers up to the previous layers.

[0092] In the intermediate layer 232B, low-order feature extraction (edge ​​extraction, etc.) is performed in the convolutional layers near the input side, and high-order feature extraction (feature extraction related to the shape, structure, etc. of the object to be identified) is performed as the layers move closer to the output side.

[0093] In addition to the convolutional layer 234 and the pooling layer 235, the intermediate layer 232B may also include a layer for batch normalization. Batch normalization is a process that normalizes the distribution of data in small batches during learning, and it serves to speed up learning, reduce dependence on initial values, and suppress overlearning.

[0094] The output layer 232C outputs the feature values ​​calculated by the intermediate layer 232B in a form that conforms to recognition. The output layer 232C may also include fully connected layers.

[0095] <Various Processing Steps in Medical Image Processing>

[0096] Next, the medical image processing method using a medical image processing device will be explained.

[0097] Figure 7 This is a flowchart illustrating a medical image processing method. The following follows... Figure 7 Each step will be explained. In addition, the following will explain the observation of the organ A, which is the subject, as a small area, divided into three zones: zone 1, zone 2, and zone 3.

[0098] (Medical image acquisition steps)

[0099] The medical image acquisition unit 220 acquires multiple medical images of organ A in a time sequence (step S10). In addition, the recording unit 207 records the observation of organ A in regions 1, 2 and 3, and records regions 1, 2 and 3 as incomplete observation as the initial state.

[0100] (Observation of status determination steps)

[0101] The observation state determination unit 222 determines the observation state of region 1, region 2, and region 3 of organ A based on the acquired medical images (step S11). The observation state determination unit 222 identifies region 1, region 2, or region 3 in the medical images. Then, based on its identification results, it determines the observation state of region 1, region 2, and region 3. For example, when region 1 is identified as the center of the medical images in ten consecutive medical images in a time sequence, the observation state determination unit 222 determines that the observation of region 1 is complete.

[0102] (Record the steps)

[0103] The recording unit 207 records the result of the determination in the observation status determination unit 222 (step S12). At the start of the inspection (initial state), zones 1, 2, and 3 are recorded as observation incomplete, but when the observation status determination unit 222 determines that the observation of each zone is complete, the record is updated to observation complete.

[0104] (Steps shown)

[0105] The display control unit 224 determines whether the observation state of the subject has changed (step S13). Then, when it is determined that the observation state of the subject has changed, the display control unit 224 displays the observation state display 501 of the subject on the monitor 400 (step S14). Here, the moment when the observation state of the subject changes refers to the situation where changes in the observation state are recorded in the recording unit 207 among the predetermined multiple small area units for observation. For example, it means that in the recording unit 207, areas 1, 2, and 3 are recorded as not having completed observation, but the observation state determination unit 222 determines that area 1 has completed observation, and the observation state of area 1 changes to completed observation in the recording unit 207. In addition, the observation state display 501 of the subject refers to a display that informs the user of the observation state of the small area units of the subject constituting the object of observation. By viewing the observation state display 501, the user can confirm whether the small area of ​​the object of observation can be fully observed.

[0106] Figure 8 This is a diagram showing an example of the observation status display 501 of organ A on monitor 400.

[0107] exist Figure 8 In the case shown, the endoscopic image 503 is displayed on the entire surface of the monitor 400. Furthermore, during the recording by the recording unit 207, at the moment when area 3 updates from "observation incomplete" to "observation complete," the display control unit 224 overlays the observation status display 501 onto the endoscopic image 503 and displays it on the monitor 400. The observation status display 501 is a list display of character information indicating areas where observation is complete and areas where observation is incomplete. In the observation status display 501, areas 1 and 3 where observation is complete (referred to as "area" in the figure) are listed under "Complete," and area 2 where observation is incomplete is listed under "Not Completed."

[0108] Back Figure 7 Then, the display control unit 224 continues to display the observation status display 501 until a predetermined display time has elapsed. The display time can be appropriately set by the user. If the user can confirm the observation status, it is preferable that the observation status display 501 is not displayed, thus enabling the observation of the endoscopic image 503. Therefore, it is preferable to set the display time based on the time during which the user can confirm the observation status. For example, the display time can be set to 10 seconds or 30 seconds. After that, the display control unit 224 sets the observation status display 501 to be off after the predetermined time has elapsed (step S15).

[0109] Afterwards, the medical image acquisition unit 220 determines whether the observation of all small region units has ended (step S16). Since region 2 has not yet been observed, the medical image is further acquired (step S10).

[0110] As described above, in this invention, when observing small areas of organ A, namely areas 1, 2, and 3, an observation status display 501 is performed at the moment when the observation status of organ A changes. Thus, the observation status display 501 is performed only when necessary; otherwise, by controlling the display, the influence on the observation of the endoscopic image can be suppressed, and the display can be performed effectively.

[0111] <Examples of variations shown in the observation status>

[0112] In use Figure 8 The example described illustrates the case where the observation status display 501 shows the area where observation is completed and the area where observation is incomplete, but examples of observation status display 501 are not limited to this. There are no particular restrictions on the display method for observation status display 501, as long as it can use characters or graphics to inform the user of the observation status of the observed subject. Specific examples of observation status display 501 will be described below.

[0113] Figure 9 This is a diagram representing a variation of observation status display 501, example 1. This example shows observation status display 501 for character information. In this example, observation status display 501 only shows areas where observation is incomplete. Specifically, when area 2 is incomplete, area 2 and the character information are displayed below "Incomplete".

[0114] Thus, by displaying the observation status 501 on the monitor 400, showing only small areas where observation is incomplete, the user can clearly identify these small areas and achieve comprehensive observation. Furthermore, in Figure 9 The example illustrates how a small area where observation is incomplete is displayed in the observation status display 501. However, it is also possible to display a small area where observation is complete in the observation status display 501, in which case the user can clearly identify the small area where observation is complete.

[0115] Figure 10 This is a diagram illustrating a variation of the observation status display 501, example 2. This example is an observation status display 501 displaying character information. In this example's observation status display 501, all small areas (areas 1-5) constituting the observed object are displayed as a list. Furthermore, in the observation status display 501, the character color is changed to indicate areas where observation is complete or incomplete. Specifically, in the observation status display 501, areas 3 and 5 are displayed with characters of the same color because observation is incomplete, while areas 1, 2, and 4 are displayed with characters of the same color because observation is complete. In this way, by displaying all small areas as a list and assigning characters to indicate whether observation is complete or incomplete within each small area unit, the user can comprehensively identify areas where observation is complete or incomplete.

[0116] Figure 11 This is a diagram illustrating a variation of the observation status display 501, example 3. This example is an observation status display 501 based on character information. The observation status display 501 in this example displays a list of all the small regions (areas 1-5) constituting the observed object. Furthermore, in the observation status display 501, a "○" or "×" is displayed next to the characters for regions where observation is complete or incomplete. Specifically, in the observation status display 501, areas 3 and 5 are marked with "×" because observation is incomplete, while areas 1, 2, and 4 are marked with "○" because observation is complete. In this way, by displaying a list of all small regions and showing the information of observation completion or incompleteness next to the characters for each small region, the user can comprehensively identify regions where observation is complete or incomplete.

[0117] Figure 12 This is a diagram representing a variation of observation status display 501, example 4. This example is an observation status display 501 based on character information. In this example, observation status display 501 represents the proportion of incomplete observations. Here, the proportion of incomplete observations refers to the proportion of incomplete small areas among multiple small areas designated for observation. Furthermore, this proportion can be calculated using either the number of small areas or the area of ​​the small areas. Additionally, in... Figure 12 In the example shown, the scale is represented by characters, but it can also be represented by graphics such as bar charts. This way, by showing the scale of the incomplete observation, the user can clearly see the area that is not being observed.

[0118] Figure 13 This is a diagram representing a variation of the observation status display 501. This example is based on an observation status display 501 that schematically represents a subject model M. The subject model M is a diagram schematically representing the stomach of the subject being observed. Different colors are assigned to the observed regions 513 (completed observation) and 511 (incomplete observation) within the small region units of the subject model M. Thus, by assigning observation completion and incomplete observation information to the subject model M, the user can identify the positions of the completed and incomplete small regions within the subject. Furthermore, in Figure 13 In the example shown, the information of observation completion and incompleteness is conveyed by changing the color of a region of the subject model M, but this is not limited to this example. For instance, information related to observation completion and incompleteness can also be conveyed to the subject model M by changing the color intensity.

[0119] Figure 14This is a variation of Example 6, representing the observation status display 501. This example is based on the observation status display 501 of a subject model M, which schematically represents the subject. In this example, only small areas where observation is complete are shown in the subject model M. Furthermore, when observation of all small areas is complete, the entire subject model M (stomach) is displayed. Thus, by assigning information of observation completion and incomplete observation to the subject model M, the user can identify the locations of completed and incomplete small areas within the subject. Furthermore, in Figure 13 and Figure 14 In this context, a cutaway view of the stomach as the subject is used as the subject model M, but it is not limited to this. For example, the subject model M can also be an unfolded view of the stomach as the subject.

[0120] <Example of a monitor>

[0121] In the above description, an example of a monitor 400 having only a main display area was described, but the present invention is not limited thereto. Hereinafter, variations of the monitor will be described.

[0122] Figure 15 This diagram illustrates an example of a monitor 401 having a main display area and sub-display areas.

[0123] like Figure 15 As shown, the monitor 401 has a main display area 402 and a sub-display area 404. The main display area 402 displays endoscopic images 503 captured by the endoscopic observer 100 in real time. The sub-display area 404 is smaller than the main display area 402 and displays information such as imaging conditions, date, and patient information. Furthermore, in... Figure 15 The illustration of this information displayed in sub-display area 404 is omitted. In monitor 401, observation status display 501 is displayed in main display area 402 and sub-display area 404. In main display area 402, as described above, observation status display 501 is displayed when the observation status of the subject changes, and the display ends when the display time has elapsed. Furthermore, in... Figure 15 The image shows the observation status of the main display area 402, where display 501 is not displayed.

[0124] The observation status display 501B is always displayed in the sub-display area 404. Because the sub-display area 404 has a smaller display area, the observation status display 501B is also smaller than the observation status display 501A displayed in the main display area 402, but it serves as an auxiliary tool for comprehensive observation. Furthermore, the same observation status display or different observation status displays can be shown in both the main display area 402 and the sub-display area 404.

[0125] Figure 16 and Figure 17 This diagram illustrates examples of displaying different viewing states in the main display area and sub-display areas.

[0126] exist Figure 16 In the example shown, a detailed observation status display 501A, representing small areas in a list, is displayed in the main display area 402. Additionally, an observation status display 501B of the subject model M is displayed in the sub-display area 404.

[0127] exist Figure 17 In the example shown, the main display area 402 displays a detailed observation status 501A, showing small areas in a list format. Additionally, the sub-display area 404 displays an observation status 501B indicating the percentage of observations that are not yet complete.

[0128] like Figure 16 and Figure 17 As shown, in the main display area 402, an observation status display 501A, which lists the observation status of small area units in detail, is displayed as the observation status of the subject changes. This allows the user to grasp the detailed observation status of the subject as the observed small area increases. Additionally, an observation status display 501B, indicating the proportion of the subject model M or the scale of the incomplete observation, is always displayed in the sub-display area 404. Therefore, even when the observation status display 501A is not displayed in the main display area 402, the user can still roughly grasp the observation status of the subject. Furthermore, in the above example, a monitor 401 with a main display area (first display area) 402 and a sub-display area (second display area) 404 has been described, but the observation status display 501 can also be displayed on a monitor with a third display area, for example. Additionally, the observation status display 501 can be displayed on multiple monitors.

[0129] <Other examples>

[0130] The above description describes the method of displaying the observation status 501 during a predetermined display period, but the method of setting the once-displayed observation status 501 to be non-displayed is not limited to this.

[0131] For example, the display control unit 224 can also be based on the user's operation via the handheld operation unit 102 (user operation receiving unit) (see reference). Figure 1 The observation status display 501 can be set to off-screen by the user's input command via the handheld operation unit 102 (user operation receiving unit). Furthermore, the display control unit 224 can also re-display the previously off-screen observation status display 501 based on a command input by the user via the handheld operation unit 102 (user operation receiving unit). In this way, the user can control the display and off-screen status of the observation status display 501 via the handheld operation unit 102, allowing the user to check the observation status display at the desired time.

[0132] The examples of the present invention have been described above, but the present invention is not limited to the embodiments described above. Of course, various modifications can be made without departing from the spirit of the present invention.

[0133] Symbol Explanation

[0134] 10: Endoscopic System

[0135] 100: Endoscopic Observation Device

[0136] 102: Hands-on Operations Department

[0137] 104: Insertion Section

[0138] 106: General purpose cable

[0139] 108: Optical Wire Connector

[0140] 112: Soft parts

[0141] 114: Bend

[0142] 116: Hardened apex

[0143] 116A: Top side face

[0144] 123: Lighting Department

[0145] 123A: Illumination lens

[0146] 123B: Illumination lens

[0147] 126: Pliers

[0148] 130: Photographic Optical System

[0149] 132: Photographic Lens

[0150] 134: Camera element

[0151] 136: Drive circuit

[0152] 141: Gas and water supply buttons

[0153] 142: Attraction Button

[0154] 143: Function Buttons

[0155] 144: Camera button

[0156] 170: Optical guide

[0157] 200: Endoscopic Processor Device

[0158] 202: Image Input Controller

[0159] 204: Image Processing Department

[0160] 205: Communications Control Department

[0161] 206: Video Output Department

[0162] 207: Recording Department

[0163] 208: Operations Department

[0164] 209: Sound Processing Department

[0165] 209A: Speaker

[0166] 210: CPU

[0167] 211: ROM

[0168] 212: RAM

[0169] 220: Medical Image Acquisition Department

[0170] 222: Observation Status Judgment Department

[0171] 224: Display Control Unit

[0172] 232A: Input Layer

[0173] 232B: Intermediate Layer

[0174] 232C: Output Layer

[0175] 234: Convolutional Layer

[0176] 235: Pooling layer

[0177] 236: Fully Connected Layer

[0178] 300: Light source device

[0179] 310: Light source

[0180] 310B: Blue light source

[0181] 310G: Green Light Source

[0182] 310R: Red light source

[0183] 310V: Purple light source

[0184] 330: Aperture

[0185] 340: Condensing Lens

[0186] 350: Light Source Control Department

[0187] 400: Monitor

Claims

1. A medical image processing device, comprising a processor, wherein, The processor Acquire multiple medical images in a time series. The observation state is identified based on each of a plurality of small regions located at predetermined positions within the organs of the subject, as described in the medical image. Based on the identification results, for each of the plurality of small regions, observation is determined to be complete if observation is completed, and incomplete if observation is incomplete. For a subject model schematically representing an organ, based on the determination result, information indicating that the observation status of each of the plurality of small regions is complete is displayed. When the observation state of any one of the multiple small regions changes from "observation incomplete" to "observation complete", the information indicating that the observation state of the small region is "observation complete" will be reflected in the subject model for updating.

2. The medical image processing apparatus according to claim 1, wherein, The processor sets the information to be displayed on the subject model to be undisplayed after a predetermined time.

3. The medical image processing apparatus according to claim 1 or 2, wherein, It also has a user operation receiving unit. The processor sets the information to be displayed or not displayed based on instructions from the user operation receiving unit.

4. The medical image processing apparatus according to claim 1 or 2, wherein, The processor causes the monitor to display the information via character information.

5. The medical image processing apparatus according to claim 4, wherein, The processor assigns information related to whether the observation of the small region unit of the subject is completed or not to the character information, and displays it as such information.

6. The medical image processing apparatus according to claim 1, wherein, The processor either completes the observation of the small area unit of the subject or displays the incomplete display as information on the monitor.

7. The medical image processing apparatus according to claim 1, wherein, The processor causes the monitor to display the medical image and overlays the information onto the medical image for display.

8. The medical image processing apparatus according to claim 1, wherein, The processor is a monitor having a first display area and a second display area smaller than the first display area, enabling the first display area and the second display area to display the information in different ways.

9. The medical image processing apparatus according to claim 8, wherein, The processor ensures that the second display area always displays the information.

10. The medical image processing apparatus according to claim 9, wherein, The processor is a monitor having a third display area different from the first and second display areas, which displays the medical image in the third display area.

11. The medical image processing apparatus according to claim 1, wherein, The processor identifies multiple small regions within the organs of the subject in the medical image that are located at predetermined positions and are distinct from each other using a learned model that has been trained to identify each small region unit of the multiple small regions.

12. A medical image processing method using a medical image processing apparatus, the medical image processing apparatus comprising a processor, wherein, The processor executes: Medical image acquisition steps, acquiring multiple medical images in a time series; and The observation state determination step identifies the observation state based on the medical image, according to each of a plurality of small regions located at predetermined, distinct positions within the organs of the subject. The processor Based on the identification results, for each of the plurality of small regions, observation is determined to be complete if observation is completed, and incomplete if observation is incomplete. For a subject model schematically representing an organ, based on the determination result, information indicating that the observation status of each of the plurality of small regions is complete is displayed. When the observation state of any one of the multiple small regions changes from "observation incomplete" to "observation complete", the information indicating that the observation state of the small region is "observation complete" will be reflected in the subject model for updating.

13. A recording medium that is non-transitory and computer-readable, wherein, The document contains a program for causing a computer to perform the medical image processing method of claim 12.

Images