Medical image processing apparatus, medical image processing method, and program

Abstract

Provided are a medical image processing device, a medical image processing method, and a program with which it becomes possible to efficiently acquire a desired medical image. In a medical image processing device (14) comprising a processor (41), the processor (41) performs: a first determining process of determining, on the basis of an acquired medical image, whether any portion, from among a plurality of portions subject to image capture, is included in the medical image; a second determining process of determining, when it is determined in the first determining process that a portion from among the plurality of portions subject to image capture is included in the medical image, whether the medical image is an image that satisfies a determination criteria for the portion included in the medical image; and a display control process of causing, when it is determined in the second determining process that the medical image is an image that satisfies a determination criteria for the portion included in the medical image, a display unit to display a notification display indicating that an image that satisfies a determination criteria for the portion included in the medical image has been captured.

Description

Technical Field

[0006] , ,

[0005] ,

[0007] , ,

[0001] The present invention relates to a medical image processing apparatus, a medical image processing method, and a program.

Background Art

[0002] In recent years, an endoscope system has been used to photograph a subject, and an examination is performed based on the acquired medical image. Various techniques for assisting a user have been proposed when photographing with an endoscope system.

[0003] In the technique described in Patent Document 1, a technique for notifying the recognition result of a medical image according to the operation of an operator (user) of an endoscope system has been proposed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Here, photographing of an organ or the like, which is a subject by an endoscope system, needs to be performed according to photographing conditions, composition, etc. defined manually or the like, and it is difficult depending on the photographing target site. In particular, for a user with little experience, it is very difficult to photograph a desired medical image, and it may also be difficult to determine whether an appropriate medical image defined manually or the like has been photographed.

[0006] The present invention has been made in view of such circumstances, and an object thereof is to provide a medical image processing apparatus, a medical image processing method, and a program capable of efficiently acquiring a desired medical image.

Means for Solving the Problems

[0007] A medical image processing apparatus, which is one aspect of the present invention for achieving the above objective, is a medical image processing apparatus equipped with a processor, wherein the processor performs: a first determination process that determines whether any of a plurality of target areas is included in the medical image based on the acquired medical image; a second determination process that determines whether the medical image is an image that satisfies the criteria for determining an area to be included in the medical image, if the first determination process determines that any of the plurality of target areas is included in the medical image; and a display control process that causes a notification display on the display unit to indicate that an image satisfying the criteria for determining an area to be included in the medical image has been taken, if the second determination process determines that the medical image is an image that satisfies the criteria for determining an area to be included in the medical image.

[0008] According to this embodiment, when the second determination process determines that the medical image satisfies the determination criteria for the part of the medical image contained in the image, a notification display indicating that an image satisfying the determination criteria for the part of the medical image contained in the image has been captured is displayed on the display unit. As a result, this embodiment can notify the user that a medical image satisfying the determination criteria set for each part has been captured, and the desired medical image can be efficiently acquired.

[0009] Preferably, the display control process maintains the display of the notification display until notification displays are shown for all of the multiple target areas being photographed.

[0010] Preferably, the second determination process performs a determination based on different determination criteria for each of the multiple target areas being photographed.

[0011] Preferably, each of the second determination processes is performed based on multiple indicators and on the determination results of the multiple indicators.

[0012] Preferably, the multiple indicators include at least one of the following: blur detection, brightness detection, boundary visibility detection, cardia visibility detection, cardia distance detection, peristalsis detection, fold detection, treatment detection, and composition detection.

[0013] Preferably, the second determination process is performed by different determination devices for each of the multiple target areas being photographed.

[0014] Preferably, the first decision process is performed based on the decision result of a first classifier composed of a convolutional neural network, and at least part of the second decision process is performed by inputting intermediate features obtained by the first classifier into the second classifier, and the second classifier outputting a decision result.

[0015] Preferably, the medical image is captured using a first light source or a second light source. The first determination process, when determining a medical image captured using the first light source, makes a determination on multiple target areas. When determining a medical image captured using the second light source, it makes a determination on a selected target area from among the multiple target areas.

[0016] Preferably, the medical image is an endoscopic image taken with an endoscope, and the display control process displays a schematic diagram of a tubular organ captured by the endoscope on the display unit, and displays a notification on the schematic diagram.

[0017] Preferably, the display control process causes the location of multiple target areas to be photographed to be displayed on the schematic diagram at their corresponding positions.

[0018] Preferably, if the display control process determines, through the second determination process, that the medical image is an image that satisfies the determination criteria for the body part included in the medical image, it will perform a notification display by changing the display format of the body part display.

[0019] Preferably, the display control process causes a guide display to be shown in a schematic diagram to guide the acquisition of medical images using an endoscope scope.

[0020] Preferably, the guide indicator is in the shape of a stick, where the direction of the stick indicates the shooting direction of the endoscope, and the length of the stick indicates the distance of the endoscope from the subject.

[0021] Preferably, the medical image is taken using the first light source or the second light source, and in the display control process, when a medical image taken using the first light source is acquired, a schematic diagram is displayed in the first display mode, and when a medical image taken using the second light source is acquired, a schematic diagram is displayed in the second display mode.

[0022] Preferably, in the display control process, for all of a plurality of imaging target sites, when it is determined by the second determination process that the medical image is an image that satisfies the determination criteria for the site included in the medical image, information indicating that all imaging of the imaging target sites has been completed is displayed on the display unit.

[0023] Preferably, when it is determined by the second determination process that the medical image is an image that satisfies the determination criteria for the site included in the medical image, the processor performs a storage process of storing the medical image in the memory.

[0024] Preferably, the plurality of imaging target sites are selected from at least the esophagogastric junction, the small curvature J-turn immediately below the cardia, the large curvature U-turn immediately below the cardia, the gastric angular region or the small curvature posterior wall J-turn from the lower body, the pyloric ring from the front of the pyloric ring, and the downward view of the large curvature of the lower body.

[0025] Preferably, the plurality of imaging target sites are selected from at least the rectum, anus, splenic flexure, hepatic flexure, duodenal inlet, and ileocecal valve.

[0026] Another aspect of the present invention is a medical image processing method for a medical image processing apparatus including a processor. The method includes: a first determination step of determining, by the processor, whether any one of a plurality of imaging target sites is included in an acquired medical image; a second determination step of determining, when it is determined in the first determination step that any one of the plurality of imaging target sites is included in the medical image, whether the medical image is an image that satisfies the determination criteria for the site included in the medical image; and a display control step of causing the display unit to display a notification indicating that an image that satisfies the determination criteria for the site included in the medical image has been captured, when it is determined in the second determination step that the medical image is an image that satisfies the determination criteria for the site included in the medical image.

[0027] Another aspect of the present invention is a program for causing a medical image processing apparatus including a processor to execute a medical image processing method. The program causes the processor to perform: a first determination step of determining, based on an acquired medical image, whether any one of a plurality of imaging target sites is included in the medical image; a second determination step of determining, when it is determined in the first determination step that any one of the plurality of imaging target sites is included in the medical image, whether the medical image is an image that satisfies the determination criteria for the site included in the medical image; and a display control step of causing the display unit to display a notification indicating that an image that satisfies the determination criteria for the site included in the medical image has been captured, when it is determined in the second determination step that the medical image is an image that satisfies the determination criteria for the site included in the medical image.

Advantages of the Invention

[0028] According to the present invention, when it is determined in the second determination process that the medical image is an image that satisfies the determination criteria for the site included in the medical image, a notification indicating that an image that satisfies the determination criteria for the site included in the medical image has been captured is displayed on the display unit. Therefore, it is possible to notify the user that a medical image that satisfies the determination criteria set for each site has been captured, and it is possible to efficiently acquire a desired medical image.

Brief Description of the Drawings

[0029] [Figure 1] Figure 1 is a schematic diagram showing the overall configuration of the endoscope system. [Figure 2] Figure 2 is a block diagram showing the configuration of the light source device. [Figure 3] Figure 3 is a block diagram showing an embodiment of a medical image processing device. [Figure 4] Figure 4 is a flowchart illustrating a medical image processing method. [Figure 5] Figure 5 shows a specific example of the configuration of the first and second classifiers. [Figure 6] Figure 6 is a diagram illustrating the data input to the first and second classifiers. [Figure 7] Figure 7 illustrates specific examples of each indicator of the second classifier. [Figure 8] Figure 8 shows a schematic diagram displayed on the display unit. [Figure 9] Figure 9 is a diagram illustrating the notification display in a schematic diagram. [Figure 10] Figure 10 is a diagram illustrating the complete information displayed on the display unit. [Figure 11] Figure 11 illustrates the highlighting of body parts. [Figure 12] Figure 12 illustrates an example of a schematic diagram. [Figure 13] Figure 13 shows the inspection screen displayed on the display unit. [Figure 14] Figure 14 is a diagram illustrating the areas to be photographed when performing a colonoscopy. [Figure 15] Figure 15 is a flowchart illustrating a medical image processing method. [Figure 16] Figure 16 illustrates schematic diagrams of two different light sources. [Figure 17] Figure 17 illustrates the first example of a display variation. [Figure 18] Figure 18 illustrates a second example of the display variation. [Figure 19] Figure 19 illustrates a third example of the display variation. [Figure 20] Figure 20 illustrates a fourth example of the display variation. [Modes for carrying out the invention]

[0030] Preferred embodiments of the medical image processing apparatus, medical image processing method, and program according to the present invention will be described below with reference to the attached drawings.

[0031] [Overall configuration of an endoscopy system including medical image processing equipment] Figure 1 is a schematic diagram showing the overall configuration of an endoscope system including a medical image processing device according to the present invention. The medical images input to the medical image processing device 14 are captured by the endoscope system described below. In the following description, the medical image processing device 14 included in the endoscope system will be described, but the embodiments of the present invention are not limited thereto. For example, the medical image processing device 14 may receive medical images captured by an endoscope system separate from the medical image processing device 14.

[0032] As shown in Figure 1, the endoscope system 9 comprises an endoscope scope 10 which is an electronic endoscope, a light source device 11, an endoscope processor device 12, a display device 13, a medical image processing device 14, an operation unit 15, and a display unit 16.

[0033] The endoscope scope 10 is used to capture a time-series of medical images including the subject image, and is, for example, a scope for the lower or upper gastrointestinal tract. This endoscope scope 10 has an insertion section 20 that is inserted into the subject (e.g., the stomach or large intestine) and has a tip and a base, a handheld control section 21 connected to the base end of the insertion section 20 and grasped by the physician performing various operations, and a universal cord 22 connected to the handheld control section 21.

[0034] The insertion section 20 is formed in a long, slender shape with a small diameter throughout. The insertion section 20 is composed of a flexible section 25 that is flexible from the base end to the tip end, a curved section 26 that can be bent by the operation of the handheld operation section 21, and a tip section 27 that houses an imaging optical system (objective lens) and an image sensor 28, etc. (not shown).

[0035] The image sensor 28 is either a CMOS (complementary metal oxide semiconductor) or CCD (charge coupled device) type image sensor. Image light from the area to be observed is incident on the imaging surface of the image sensor 28 through an observation window (not shown) opened on the tip surface of the tip portion 27, and an objective lens (not shown) positioned behind this observation window. The image sensor 28 captures (converts into an electrical signal) the image light from the area to be observed that has been incident on its imaging surface and outputs an imaging signal.

[0036] The handheld control unit 21 is equipped with various operating members that are operated by the user (a person who operates the endoscope system 9, such as a physician). Specifically, the handheld control unit 21 is equipped with two types of bending operation knobs 29 used for bending the bending section 26, an air supply / water supply button 30 for air supply / water supply operation, and a suction button 31 for suction operation. The handheld control unit 21 is also equipped with a still image capture instruction unit 32 for giving instructions for capturing still images 39 of the area to be observed, and a treatment instrument introduction port 33 for inserting a treatment instrument (not shown) into a treatment instrument insertion passage (not shown) that runs through the insertion section 20.

[0037] The universal cord 22 is a connecting cord for connecting the endoscope scope 10 to the light source device 11. This universal cord 22 contains a light guide 35, a signal cable 36, and a fluid tube (not shown) which are inserted through the insertion section 20. The end of the universal cord 22 is provided with a connector 37a for connection to the light source device 11, and a connector 37b that branches off from this connector 37a and connects to the endoscope processor device 12.

[0038] By connecting connector 37a to light source device 11, the light guide 35 and fluid tube (not shown) are inserted into light source device 11. As a result, the necessary illumination light, water, and gas are supplied from light source device 11 to endoscope scope 10 via the light guide 35 and fluid tube (not shown). Consequently, illumination light is irradiated from the illumination window (not shown) on the tip surface of tip 27 toward the area to be observed. In addition, in response to the pressing of the aforementioned air / water supply button 30, gas or water is sprayed from the air / water supply nozzle (not shown) on the tip surface of tip 27 toward the observation window (not shown) on the tip surface.

[0039] By connecting connector 37b to the endoscope processor device 12, the signal cable 36 and the endoscope processor device 12 are electrically connected. As a result, the imaging signal of the area under observation is output from the image sensor 28 of the endoscope scope 10 to the endoscope processor device 12 via the signal cable 36, and a control signal is output from the endoscope processor device 12 to the endoscope scope 10.

[0040] The light source device 11 supplies illumination light to the light guide 35 of the endoscope scope 10 via the connector 37a. The illumination light is selected from various wavelength bands depending on the purpose of observation, such as white light (light in the white wavelength band or light in multiple wavelength bands), light in one or more specific wavelength bands, or a combination thereof. The light source device 11 will be described in detail later.

[0041] The endoscope processor device 12 controls the operation of the endoscope scope 10 via the connector 37b and the signal cable 36. The endoscope processor device 12 also generates an image (also called "video 38") consisting of a time-series of frame images 38a including the subject image, based on the imaging signal acquired from the image sensor 28 of the endoscope scope 10 via the connector 37b and the signal cable 36. Furthermore, when the still image capture instruction unit 32 is operated on the handheld control unit 21 of the endoscope scope 10, the endoscope processor device 12 acquires a still image 39 of one frame image 38a from the video 38 in parallel with the generation of the video 38, according to the timing of the capture instruction. In this description, the medical image consists of the above-mentioned still image 39 and frame image 38a.

[0042] Furthermore, if the video 38 and still image 39 are images obtained using light in the specific wavelength band (special light) described above, then both are special light images. The endoscope processor device 12 then outputs the generated video 38 and still image 39 to the display device 13 and the medical image processing device 14.

[0043] Furthermore, the endoscope processor device 12 may generate (acquire) a special light image having information of the specific wavelength band described above, based on the normal light image obtained with the white light described above. In this case, the endoscope processor device 12 functions as a special light image acquisition unit. The endoscope processor device 12 then obtains a signal of the specific wavelength band by performing calculations based on the color information of red, green, and blue [RGB (Red, Green, Blue)] or cyan, magenta, and yellow [CMY (Cyan, Magenta, Yellow)] contained in the normal light image.

[0044] Furthermore, the endoscope processor device 12 may generate a feature image, such as a known oxygen saturation image, based on, for example, at least one of the normal light image obtained with the white light described above and the special light image obtained with the light of a specific wavelength band (special light) described above. In this case, the endoscope processor device 12 functions as a feature image generation unit. The frame images 38a or still images 39 that constitute the video 38, which includes the in vivo image, normal light image, special light image, and feature image, are all medical images obtained by photographing or measuring the human body for the purpose of image-based examination.

[0045] The display device 13 is connected to the endoscope processor device 12 and functions as a display unit 16 that displays video 38 and still images 39 input from the endoscope processor device 12. The user performs operations such as advancing and retracting the insertion unit 20 while checking the video 38 displayed on the display device 13, and if a lesion or other abnormality is found in the target area or the area being observed, the user operates the still image capture instruction unit 32 to capture a still image of the area being observed, and also performs procedures such as diagnosis and biopsy. Similarly, the video 38 and still images 39 are also displayed on the display unit 16 connected to the medical image processing device 14, which will be described later.

[0046] The endoscopic system 9 shown in Figure 1 is a specific example, and other embodiments of endoscopic systems may be used. For example, the processing using trained models such as the first classifier 42 (Figure 3) and the second classifier 43 (Figure 3) described below may be mounted in a dedicated AI (Artificial Intelligence) box and connected to the endoscopic system. Alternatively, the endoscopic system may be connected to a dedicated server via a network, and medical image processing may be performed on that dedicated server.

[0047] <Configuration of the light source device> Figure 2 is a block diagram showing the configuration of the light source device 11.

[0048] As shown in Figure 2, the light source device 11 consists of a light source 310 for illumination, an aperture 330, a focusing lens 340, and a light source control unit 350, and directs observation light onto the light guide 35. The light source 310 is equipped with a red light source 310R, a green light source 310G, a blue light source 310B, and a violet light source 310V, which emit narrowband light of red, green, blue, and violet, respectively, and can emit narrowband light of red, green, blue, and violet. The illuminance of the observation light from the light source 310 is controlled by the light source control unit 350, which can change (increase or decrease) the illuminance of the observation light as needed, and can also stop the illumination.

[0049] The light source 310 can emit red, green, blue, and violet narrowband light in any combination. For example, it is possible to emit red, green, blue, and violet narrowband light simultaneously to irradiate with white light (normal light) as observation light, or to emit one or two of them to irradiate with narrowband light (special light). The light source 310 may further include an infrared light source that emits infrared light (an example of narrowband light). Alternatively, a light source that emits white light and a filter that transmits white light and each of the narrowband lights may be used to irradiate with white light and narrowband light as observation light.

[0050] <Wavelength band of the light source> The light source 310 may be a light source that generates light in the white band, or a light source that generates light in multiple wavelength bands as white band light, or a light source that generates light in a specific wavelength band narrower than the white wavelength band. The specific wavelength band may be the blue band or green band of the visible range, or the red band of the visible range. If the specific wavelength band is the blue band or green band of the visible range, it may include a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and may have a peak wavelength within the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. If the specific wavelength band is the red band of the visible range, it may include a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and the light in the specific wavelength band may have a peak wavelength within the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm.

[0051] The light in the specific wavelength band described above may include wavelength bands in which oxyhemoglobin and deoxyhemoglobin have different absorption coefficients, and may have a peak wavelength in the wavelength band in which oxyhemoglobin and deoxyhemoglobin have different absorption coefficients. In this case, the specific wavelength band may include wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm, and may have a peak wavelength in the wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm.

[0052] Furthermore, the light generated by the light source 310 may include a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm, and may have a peak wavelength in the wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.

[0053] Furthermore, the light source 310 may be equipped with a light source that emits excitation light with a peak wavelength of 390 nm to 470 nm. In this case, it is possible to acquire endoscopic images that contain fluorescence information emitted by fluorescent substances in the subject (living body). When acquiring fluorescence images, fluorescent dyes (fluorescein, acridine orange, etc.) may be used.

[0054] The type of light source (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), etc.), wavelength, presence or absence of a filter, etc. of the light source 310 are preferably configured according to the type, part, purpose of observation, etc. Furthermore, when observing, it is preferable to combine and / or switch the wavelength of the observation light according to the type, part, purpose of observation, etc. When switching wavelengths, for example, the wavelength of the irradiated light may be switched by rotating a disc-shaped filter (rotary color filter) that is placed in front of the light source and has a filter that transmits or blocks light of a specific wavelength.

[0055] Furthermore, the image sensor 28 used in implementing the present invention is not limited to a color image sensor in which a color filter is provided for each pixel, but may also be a monochrome image sensor. When a monochrome image sensor is used, imaging can be performed sequentially by switching the wavelength of the observation light. For example, the wavelength of the emitted observation light may be switched sequentially between (purple, blue, green, red), or broadband light (white light) may be irradiated and the wavelength of the emitted observation light may be switched by a rotary color filter (red, green, blue, purple, etc.). Alternatively, one or more narrowband lights (green, blue, purple, etc.) may be irradiated and the wavelength of the emitted observation light may be switched by a rotary color filter (green, blue, purple, etc.). The narrowband lights may be two or more infrared lights with different wavelengths (first narrowband light, second narrowband light).

[0056] [Medical Image Processing Equipment] Figure 3 is a block diagram showing an embodiment of the medical image processing device 14. The medical image processing device 14 acquires a medical image and performs a first determination process and a second determination process based on the acquired medical image. The medical image processing device 14 then performs a display control process to display a notification on the display unit 16 based on the determination result of the second determination process.

[0057] The medical image processing device 14 is composed of, for example, a computer. The operating unit 15 includes a keyboard and mouse connected to the computer by wire or wireless connection, as well as buttons provided on the handheld operating unit 21 of the endoscope scope 10, and the display unit 16 uses various monitors such as an LCD monitor that can be connected to the computer.

[0058] The medical image processing device 14 consists of a medical image acquisition unit 40, a CPU (Central Processing Unit) 41, a first determination unit 42, a second determination unit 43, a display control unit 46, an audio control unit 47, and a memory 48. The processing of each unit is realized by one or more processors. Here, the processor may consist of the CPU 41, or it may consist of one or more CPUs not shown.

[0059] The CPU 41 operates based on the operating system stored in the memory 48, including various programs such as the medical image processing program according to the present invention, and comprehensively controls the medical image acquisition unit 40, the first determination unit 42, the second determination unit 43, the display control unit 46, and the sound control unit 47, and also functions as a part of each of these units.

[0060] The medical image acquisition unit 40 acquires medical images. The medical image acquisition unit 40 acquires medical images, including the subject image, from the endoscope processor device 12 (Figure 1) using an image input / output interface (not shown) that is connected to the endoscope processor device 12 by wire or wireless connection. For example, if the still image 39 described above is taken while a video 38 is being recorded with the endoscope scope 10, the medical image acquisition unit 40 acquires the still image 39 from the endoscope processor device 12. The medical image acquisition unit 40 may also acquire the frame images 38a that constitute the video 38 as medical images.

[0061] The first determination unit 42 performs a first determination process based on the medical image acquired by the medical image acquisition unit 40 to determine whether any of the pre-set target areas are included in the medical image. Here, the target areas are areas that must be intentionally photographed and recorded during the examination. Furthermore, the target areas are areas that need to be photographed by intentionally manipulating the endoscope scope 10. These target areas are relatively difficult to photograph, and it is also difficult for the user to determine whether they have been photographed properly.

[0062] Multiple imaging sites are set depending on the purpose of the examination. For example, the imaging sites may be set to be between 5 and 15 parts, preferably between 5 and 10 parts. For example, when examining the inside of the stomach with the endoscope system 9, the imaging sites may be set to "esophagogastric junction," "lesser curvature J-turn just below the cardia (imaging by J-turn operation)," "greater curvature U-turn just below the cardia (imaging by U-turn operation)," "posterior wall J-turn of the lesser curvature from the gastric angle or lower body," "from the anterior part of the pyloric ring to the pyloric ring," and "view of the greater curvature from the lower body." Note that "posterior wall J-turn of the lesser curvature from the gastric angle or lower body" may be changed to "lower body lesser curvature J-turn" considering that it may not always be possible to image the gastric angle and it may not be possible to guarantee that the posterior wall is being imaged. Also, "from the anterior part of the pyloric ring to the pyloric ring" may be changed to "panoramic view of the antrum," which prioritizes whether the antrum can be imaged from above rather than pinpointing the pyloric ring. Furthermore, "viewing the lower body and greater curvature" may also be referred to as "viewing the greater curvature," emphasizing the ability to photograph the greater curvature with its folds open, rather than being limited to the lower body. Also, for example, when examining the inside of the large intestine with endoscopic system 9, the target areas for imaging are set to "rectum," "anus," "splenic flexure," "hepatic flexure," "duodenal opening," and "ileocecal region."

[0063] The first classifier 42 recognizes the target area included in the input medical image using various methods. For example, the first classifier 42 recognizes the target area included in the input medical image using a trained model (recognition model) composed of a convolutional neural network or the like. The first classifier 42 learns images (medical images) to recognize pre-set target areas, generates a trained model, and uses that trained model to determine whether or not the medical image contains the set target area.

[0064] The first classifier 42 may classify or recognize the input medical image based on similarity. When the first classifier 42 classifies the medical image to recognize the target area, the technology described in the reference (B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In Neural Information Processing Systems (NIPS), pages 487-495, 2014. 1, 4, 6, 8) can be used. Alternatively, when the first classifier 42 recognizes the target area based on the similarity of the features of the medical image, the technology described in the reference (FaceNet: A Unified Embedding for Face Recognition and Clustering https: / / arxiv.org / abs / 1503.03832) can be used.

[0065] The second judging device 43 determines whether the medical image meets the criteria for a part to be included in a medical image, when the first judging device 42 determines that any of the multiple target areas to be photographed are included in the medical image. In other words, the second judging device 43 determines whether the target area to be photographed in the medical image is appropriately captured as the desired medical image. A medical image in which the target area to be photographed is captured as desired is a good image, an image suitable for diagnosis.

[0066] A second judging device 43 is provided for each area being photographed, and each second judging device 43 can perform judgments based on different judgment criteria. Furthermore, the second judging device 43 is performed using multiple indicators, and the judgment is made based on the judgment results of multiple indicators. A detailed explanation of the first judging device 42 and the second judging device 43 will be given later.

[0067] If the display control unit 46 determines in the second determination process that the medical image satisfies the determination criteria for the part of the medical image contained in the image, it causes the display unit 16 to display a notification indicating that an image satisfying the determination criteria for the part of the medical image contained in the image has been captured. Note that the display performed by the display control unit 46 on the display unit 16 may also be performed on the display device 13.

[0068] The audio control unit 47 plays a notification sound from the speaker 17. For example, the notification sound is played from the speaker 17 in conjunction with the notification display shown by the display control unit 46 described above.

[0069] Memory 48 includes flash memory, ROM (Read-only Memory), RAM (Random Access Memory), and a hard disk drive. The flash memory, ROM, and hard disk drive are non-volatile memories that store the operating system, various programs such as the medical image processing program according to the present invention, and captured still images 39 and video 38. The RAM is a volatile memory that allows high-speed reading and writing of data and functions as a temporary storage area for various programs stored in the non-volatile memory and as a work area for the CPU 41.

[0070] <Medical Image Processing Methods> Next, a medical image processing method using the medical image processing device 14 will be described. For example, the medical image processing method is performed by the processor of the medical image device executing a program.

[0071] Figure 4 is a flowchart showing a medical image processing method using the medical image processing device 14.

[0072] First, the medical image acquisition unit 40 acquires a medical image (for example, a still image 39) from the endoscope processor device 12 (step S10). Then, the first determination unit 42 performs a first determination process based on the acquired medical image (step S11: first determination step). Specifically, the first determination unit 42 determines whether or not the medical image contains any of the multiple target areas. If the first determination unit 42 determines that the medical image does not contain any of the target areas (in the case of No in step S11), the second determination process in the subsequent second determination unit 43 is not performed.

[0073] On the other hand, if the first determination device 42 determines that the medical image includes any of the target areas to be photographed (in the case of Yes in step S11), the second determination device 43 performs a second determination process based on the medical image for which the first determination was made (step S12: second determination step). Specifically, the second determination device 43 determines whether or not the area determined by the first determination device 42 meets the set determination criteria using a set index. If the second determination device 43 determines that the determination criteria are not met (in the case of No in step S12), no notification is displayed on the display unit 16.

[0074] On the other hand, if the second judge 43 determines that the medical image meets the criteria, the display control unit 46 displays a notification on the display unit 16 (step S13: display control step). The still image 39 is acquired based on the instructions of the still image capture instruction unit 32, and the above-described processing is performed sequentially on the acquired still image 39.

[0075] The first determination process, the second determination process, and the display control process described above will be explained in detail below.

[0076] <First and second decision processes> First, we will explain the first determination process performed in the first determination step and the second determination process performed in the second determination step.

[0077] Figure 5 shows a specific example configuration of a first determination unit 42 that performs the first determination process and a second determination unit 43 that performs the second determination process.

[0078] The first classifier 42 is a pre-trained model composed of a CNN and has undergone machine learning in advance. Six locations inside the stomach (see Figures 7 and 8) are set as the target areas for imaging. The first classifier 42 determines whether or not the medical image contains these six locations inside the stomach. Specifically, the first classifier 42 determines whether or not the medical image contains the following target areas: "esophagogastric junction," "lesser curvature J-turn just below the cardia (imaging by J-turn operation)," "greater curvature U-turn just below the cardia (imaging by U-turn operation)," "posterior wall J-turn of the lesser curvature from the gastric angle or lower body," "from the anterior part of the pylorus to the pylorus," and "view looking down at the greater curvature of the lower body." The first classifier 42 has been trained using training data consisting of medical images of the six locations inside the stomach so that it can recognize each of the aforementioned target areas.

[0079] For example, the first classifier 42 recognizes the target area of ​​the medical image based on the classification score for each of the six target areas mentioned above. The first classifier 42 determines that the target area with the highest classification score among the classification scores output for each target area has been recognized. The first classifier 42 is set to a threshold, and if the classification score is below the threshold, it determines that none of the target areas are included in the medical image.

[0080] When the first classifier 42 recognizes that the medical image contains a target area, a second classification is performed by the second classifiers 43a to 43g corresponding to the recognized target area. For example, if the first classifier 42 determines that the input medical image contains the "esophagogastric junction," the second classifier 43a performs the second classification process. Similarly, if the first judging device 42 determines that the medical image includes a "lesser curvature J-turn directly below the cardia (imaging by J-turn operation)", the second judging device 43b performs a second judging process. If the first judging device 42 determines that the medical image includes a "greater curvature U-turn directly below the cardia (imaging by U-turn operation)", the second judging device 43c performs a second judging process. If the first judging device 42 determines that the medical image includes a "posterior wall J-turn of the lesser curvature from the gastric angle or lower body", the second judging device 43d performs a second judging process. If the first judging device 42 determines that the medical image includes "from the anterior part of the pylorus to the pylorus", the second judging device 43e performs a second judging process. If the first judging device 42 determines that the medical image includes a "view of the greater curvature of the lower body", the second judging device 43f performs a second judging process. Thus, the second judging devices 43a to 43f are provided for each imaging target area judged by the first judging device 42. Each of the second judging devices 43a to 43f can perform judgments using multiple indicators and different judgment criteria.

[0081] Figure 6 is a diagram illustrating the data input to the first determination device 42 and the second determination device 43.

[0082] The first classifier 42 receives a medical image (still image 39) acquired by the medical image acquisition unit 40 as input. The first classifier 42 performs calculations in each layer of the CNN based on the input medical image to generate intermediate features of the input medical image. The first classifier 42 then outputs a classification result based on the intermediate features.

[0083] Similar to the first classifier 42, the second classifier 43 receives the medical image acquired by the medical image acquisition unit 40 as input. The second classifier 43 also receives the intermediate features generated by the first classifier 42 as input. The second classifier 43 outputs a judgment result based on the medical image and the intermediate features. Thus, the first classifier 42 outputs a judgment result based on the medical image, while the second classifier 43 outputs a judgment result based on both the medical image and the intermediate features.

[0084] Figure 7 illustrates specific examples of each indicator of the second classifier 43a to 43f.

[0085] Each of the second classifiers 43a to 43f outputs a judgment result based on multiple indicators. The indicator for the second classifier 43a is indicated by code 52, the indicator for the second classifier 43b is indicated by code 53, the indicator for the second classifier 43c is indicated by code 54, the indicator for the second classifier 43d is indicated by code 55, the indicator for the second classifier 43e is indicated by code 56, and the indicator for the second classifier 43f is indicated by code 57. Code 51 indicates that the second judgment process is not performed if the first classifier 42 determines that the medical image does not include any of the target areas.

[0086] In code 52, the second determination process is shown for the case where the first determination unit 42 determines that the medical image contains the "esophagogastric junction". In this case, the second determination unit 43a performs the second determination process based on the indices of blur / blur detection, brightness detection, and boundary visibility detection. Here, blur / blur detection determines whether the medical image is blurred and / or blurred based on the input medical image. Blur / blur detection is performed by a blur / blur determination unit, which uses known techniques such as the Fast Fourier Transform (FFT) to determine the blur / blur of the medical image. Brightness detection is performed based on the input medical image, for example, based on the luminance of the medical image. Boundary visibility detection is performed by a boundary visibility determination unit of a trained model composed of a CNN. The boundary visibility determination unit determines whether the junction between the stomach and esophagus is visible in the medical image based on the input intermediate features. The second judge 43a determines whether the medical image meets predetermined criteria for each of the following indicators: blur detection, brightness detection, and boundary visibility detection. Here, the criteria are set as threshold values ​​for the sum, average, or weighted average of the detection results of each indicator.

[0087] In reference to 53, the second determination process is shown for the case where the first determination unit 42 determines that the medical image includes a "lesser curve J-turn just below the cardia (imaging by J-turn operation)". In this case, the second determination unit 43b performs the second determination process based on the following indicators: blur detection, brightness detection, cardia visibility detection, and cardia distance detection. Cardia visibility detection and cardia distance detection are performed by pre-trained cardia visibility determination and cardia distance determination models composed of CNNs. The cardia visibility determination determines whether the cardia is visible in the medical image based on the input intermediate features. Cardia distance determination determines the distance from the tip of the endoscope scope 10 (imaging surface of the image sensor 28) to the cardia in the medical image based on the input intermediate features.

[0088] In code 54, the second determination process is shown for the case where the first determination unit 42 determines that the medical image contains a "great curve U-turn directly below the cardia". In this case, the second determination unit 43c performs the second determination process based on the following indicators: blur detection, brightness detection, treatment detection, and composition detection. Treatment detection and composition detection are performed by a treatment determination unit and a composition determination unit, which are trained models composed of a CNN. The treatment determination unit determines whether or not water, residue, or foam has accumulated in the area included in the medical image based on the input intermediate features. The composition determination unit also determines the composition of the medical image based on the input intermediate features. For example, the composition determination unit determines whether or not the target area is in the center of the medical image.

[0089] Reference numeral 55 indicates the second determination process when the first determination device 42 determines that the medical image includes a "J-turn in the posterior wall of the lesser curvature from the gastric angle or lower body." In this case, the second determination device 43d performs the second determination process based on indices for blur detection, brightness detection, and composition detection.

[0090] In code 56, the second determination process is shown for the case where the first determination unit 42 determines that the medical image includes "the anterior part of the pylorus to the pylorus." In this case, the second determination unit 43e performs the second determination based on the following indicators: blur detection, brightness detection, peristalsis detection, and composition detection. Here, peristalsis detection is performed by a peristalsis determination unit, which is a trained model composed of a CNN. The peristalsis determination unit determines whether or not there is peristalsis in the target area of ​​the medical image based on the input intermediate features.

[0091] In code 57, the second determination process is shown for the case where the first determination unit 42 determines that the medical image includes a "view of the greater curvature of the lower body." In this case, the second determination unit 43f performs the second determination process based on the following indicators: blur detection, brightness detection, treatment detection, composition detection, and fold detection. Here, fold detection is performed by a fold determination unit, which is a trained model composed of a CNN. The fold determination unit determines whether or not folds are extended in the target area of ​​the medical image based on the input intermediate features.

[0092] Furthermore, blur detection is performed using a common criterion in the second classifiers 43a to 43f, while the detection of other indicators is performed individually and independently in the second classifiers 43a to 43f. In addition, the above explanation describes an example of obtaining intermediate features from the first classifier 42, but it is not limited to this. For example, the second classifiers 43a to 43f may calculate and detect features individually.

[0093] As described above, the second judgment process combines judgments using a trained model (boundary visibility judgment, cardia visibility judgment, cardia distance judgment, treatment judgment, composition judgment, cervix judgment, and fold judgment) with judgments using pixel values ​​without a trained model (blur / blur judgment and brightness judgment). By switching between judgments using a trained model and judgments using pixel values ​​without a trained model, depending on the indicators that make up the second judgment process, the judgment accuracy and speed can be optimized. Furthermore, the judgment accuracy can be improved by performing judgments using multiple trained models in the second judgment process.

[0094] As explained above, the first determination process determines whether any of the target areas are included in the medical image. Then, the second determination process, based on the result of the first determination process, determines whether the desired medical image has been captured based on different indicators for each target area and according to the respective determination criteria.

[0095] The above-described explanations of the first and second determination processes are specific examples and are not limiting. The first and second determination processes may be implemented in the following ways.

[0096] The above example describes a case in which the second determination process is performed on all target areas determined to be included in the medical image in the first determination process, but the present invention is not limited to this example. For example, the second determination process may be performed on only a portion of the target areas determined to be included in the medical image in the first determination process. Specifically, the first determination process determines whether any of the 13 target areas are included in the medical image, and the second determination process is performed when it is determined that six predetermined target areas are included in the medical image. In this way, by setting the system to perform the second determination process on areas where, for example, medical images suitable for diagnosis are required, the desired medical image can be efficiently acquired.

[0097] The example above described a case where the criteria for brightness determination differ for each body part being photographed. However, the criteria for brightness determination may be the same for all body parts being photographed. By setting different criteria for brightness determination for each body part being photographed, it becomes possible to determine, for example, whether the brightness is appropriate for the diagnosis of each body part being photographed.

[0098] <Display control processing> Next, we will explain the display control process performed in the display control step.

[0099] Figure 8 shows a schematic diagram displayed on the display unit 16 by the display control unit 46.

[0100] Schematic diagram 101 is a model image of a tubular organ captured by an endoscope scope 10. Schematic diagram 101 consists of a stomach portion 105, an esophageal portion 103, and a duodenal portion 107, with the esophageal portion 103 and the duodenal portion 107 connected to the stomach portion 105. Schematic diagram 101 shows corresponding location indicators 109A to 109F, which represent the modeled areas to be photographed. Here, region indicator 109A schematically represents the "esophagogastric junction," region indicator 109B schematically represents the "lesser curvature J-turn just below the cardia," region indicator 109C schematically represents the "greater curvature U-turn just below the cardia," region indicator 109D schematically represents the "post-wall J-turn of the lesser curvature from the gastric angle or lower body," region indicator 109E schematically represents the "anterior part of the pyloric ring to the pyloric ring," and region indicator 109F schematically represents the "view looking down at the greater curvature of the lower body."

[0101] Furthermore, location indicators 109A to 109F display guide indicators that guide the imaging of the endoscope scope 10. Specifically, location indicator 109A displays guide indicator 111A, location indicator 109B displays guide indicator 111B, location indicator 109C displays guide indicator 111C, location indicator 109D displays guide indicator 111D, location indicator 109E displays guide indicator 111E, and location indicator 109F displays guide indicator 111F. Each of the guide indicators 111A to 111F is stick-shaped, and the direction of the stick shape indicates the imaging direction of the endoscope scope 10. In addition, the length of the stick shape indicates the distance from the target area to the endoscope scope 10. Specifically, the opposite end of each stick-shaped part indicator 109A to 109F of the guide indicators 111A to 111F indicates an imaging position (position of the endoscope tip 27) where images that meet the judgment criteria in the second judgment process are easily obtained. When the endoscope scope 10 tip 27 is positioned at the opposite end of the stick-shaped part indicators 109A to 109F and imaging is performed, a desired medical image (for example, one with a composition suitable for diagnosis) can be obtained. In this way, by indicating a position where a medical image that meets the judgment criteria in the composition judgment of the second judgment process can be taken, it is possible to assist in obtaining a medical image suitable for diagnosis.

[0102] When the endoscope system 9 takes an image of the area to be photographed, the display control unit 46 displays the schematic diagram 101 on the display unit 16.

[0103] Figure 9 is a diagram illustrating the notification display in a schematic diagram.

[0104] The display control unit 46 displays a notification on the display unit 16 if it determines in the second judgment process that the image meets the judgment criteria. The display control unit 46 performs the notification by changing the display format of the part displays 109A to 109F in schematic diagram 101. The notification is performed by changing the display format of the part display corresponding to the part that met the judgment criteria in the second judgment process among the part displays 109A to 109F. In the illustrated case, the notification is performed by changing the color of the part displays 109A to 109F. The display control unit 46 also hides the guide displays corresponding to the part displays 109A to 109F on which the notification was displayed, since the acquisition of the desired medical image has been completed. However, the display of the guide displays corresponding to the part displays 109A to 109F on which the notification was displayed may be maintained. Furthermore, various forms of notification can be adopted. For example, the part displays may be blinked or lit as a notification. In addition, along with the notification display, the audio control unit 47 may output a notification sound through the speaker 17.

[0105] As described above, in this embodiment, the first determination process determines whether the medical image includes any of the multiple target areas, the second determination determines whether the criteria for the area included in the medical image are met, and if the medical image meets the criteria, a notification is displayed on the display unit 16. This allows the user to efficiently acquire the desired medical image that meets the criteria for the target area.

[0106] <Example 1> Next, a modification 1 of this embodiment will be described. In this example, when all target areas have been photographed, information indicating that the photography of all target areas has been completed (completion information) is displayed on the display unit 16. By displaying the completion information on the display unit 16, the user can easily recognize that the photography of the target areas has been completed.

[0107] Figure 10 is a diagram illustrating the complete information displayed on the display unit 16.

[0108] Reference numeral 60 indicates an example of the display on the display unit 16 at the start of an endoscopic examination of the stomach 105. The display unit 16 shows a schematic diagram 101, which has region indicators 109A to 109F for the areas to be photographed. In the case of reference numeral 60, since there are no areas to be photographed that have been photographed yet, all region indicators 109A to 109F are shown as "not yet photographed".

[0109] In reference numeral 61, an example of a display showing that the "esophagogastric junction," "J-turn of the posterior wall of the lesser curvature from the gastric angle or lower body," and "from the anterior part of the pylorus to the pylorus" during an intermediate stage of endoscopic examination of the stomach 105 meet the second judgment criterion. In schematic diagram 101 displayed on the display unit 16, the display format of the part indicators 109A, 109D, 109E, and 109F has been changed, and a notification display is being provided. The display control unit 46 maintains the display of the notification display until notification displays are shown for all of the multiple target areas. This allows the user to accurately grasp the progress of imaging of the target areas. Note that schematic diagram 101 may be turned on or off on the display unit 16 in response to the detection of events such as still image acquisition or lesion detection. If schematic diagram 101 is turned on during the examination, the display of schematic diagram 101 should begin after the notification display has been shown for the parts that meet the second judgment criterion from the start of the examination until the point in time when schematic diagram 101 is turned on.

[0110] Reference numeral 62 indicates an example of the display on the display unit 16 at the stage when imaging of all target areas has been completed. The display unit 16 displays complete information 113 indicating that imaging of all target areas has been completed when the second judgment process determines that the medical images meet the judgment criteria for all target areas. Here, the complete information 113 can be displayed in various forms. In the illustration, the complete information 113 is indicated by a checkbox. In addition, a display mode may be adopted in which the brightness of the notification displays of area indicators 109A to 109F is reduced when the complete information 113 is illuminated. Furthermore, after a certain period of time has elapsed since the display of the complete information 113, the area indicators 109A to 109F may be stopped and only the complete information 113 may be displayed.

[0111] As explained above, in this example, when the imaging of all target areas is completed, the completion information 113 is displayed on the display unit 16. This allows the user to easily recognize that the imaging of all target areas has been completed.

[0112] <Modification 2> Next, a modified example 2 of this embodiment will be described. In this example, newly captured areas are highlighted.

[0113] Figure 11 illustrates the highlighting of body parts.

[0114] In schematic diagram 101, area indicators 109A, 109E, and 109F have completed the acquisition of medical images that meet the criteria, and notification displays are shown. Area indicator 109D is a area where the acquisition of a newly acquired image that meets the criteria has been completed, and area indicator 109D corresponding to this target area is highlighted. Note that the notification displays for area indicators 109A, 109E, and 109F and the highlight display for area indicator 109D have different display modes. For example, the notification display and the highlight display have different display modes in terms of color and brightness.

[0115] In this way, by highlighting the area from which a medical image meeting the new criteria has been captured, users can easily recognize the newly captured target area. The duration of the highlight display is determined by a certain period of time, the time until the next medical image (still image 39) is captured, the time until the first judgment process determines that the next target area is included in the medical image, or the time until the second judgment process determines that the next criteria are met, or a combination of these times.

[0116] <Variation 3> Next, a third modified example of this embodiment will be described. In this example, in the first determination process, notification is also provided for the area corresponding to the area to be photographed that is determined to be included in the medical image.

[0117] Figure 12 is a diagram illustrating an example of the display shown in schematic diagram 101.

[0118] In schematic diagram 101, the first determination display 115D is displayed in the area display 109D. The first determination display 115D is displayed so as to surround the outline of the area display 109D. This first determination display 115D is displayed when the first determination device 42 determines that the area to be photographed corresponding to the area display 109D is included in the medical image. Similarly, for the other area displays 109A to 109C, 109E, and 109F, the first determination display is also displayed when it is determined that the area to be photographed is included in the medical image. In this way, by displaying the first determination display 115D, which indicates that it was detected in the first determination process, the user can recognize that at least the corresponding area to be photographed is included in the medical image.

[0119] <Modification 4> Next, a fourth modification of this embodiment will be described. In this example, schematic diagram 101 is displayed superimposed on the inspection screen.

[0120] Figure 13 shows the inspection screen displayed on the display unit 16.

[0121] The examination screen 117 is a screen displayed on the display device 13 and / or display unit 16 when a user is performing an examination using the endoscopy system 9. The user can easily obtain necessary information by checking the examination screen 117. The examination screen 117 consists of a live view display 119, subject information 115, and recorded images 113A to 113C. The live view display 119 displays real-time video 38 captured by the endoscope scope 10. The user operates the endoscope scope 10 while observing the live view display 119. The subject information 115 shows the name, age, date of birth, gender, ID number, etc., of the subject undergoing the endoscopic examination. Recorded images 113A to 113C display still images 39 acquired by the user when the still image capture instruction unit 32 instructs the user to capture images while the user is capturing video 38.

[0122] Schematic diagram 101 is superimposed on the examination screen 117. In the illustration, schematic diagram 101 is superimposed on the lower right of the examination screen 117. In schematic diagram 101, the area display 109D indicates that the previous imaging was completed and the notification display is maintained. The live view display 119 displays the target area for imaging (view of the greater curvature of the lower body) corresponding to area display 109F. When an appropriate medical image of the target area for imaging corresponding to area display 109F is acquired, a notification is displayed on area display 109F. In this way, by superimposing schematic diagram 101 on the examination screen 117, the user can efficiently proceed with imaging of the target area while observing the live view display 119.

[0123] <Modification 5> Next, a fifth modification of this embodiment will be described. In the above example, the present invention was described as being applied to a stomach examination, but in this example, the present invention is applied to a colon examination.

[0124] Figure 14 is a diagram illustrating the areas to be photographed when performing a colonoscopy.

[0125] In Figure 14, a schematic diagram of the large intestine 121 shows location indicators 123A to 123E and guide indicators 125A to 125E indicating the areas to be photographed. In this example, the areas to be photographed are set as "rectum" (indicated by symbol 125A-1) (corresponding location indicator 123A), "anus" (indicated by symbol 125A-2) (corresponding location indicator 123A), "splenic flexure" (corresponding location indicator 123B), "hepatic flexure" (corresponding location indicator 123C), "duodenal entrance" (corresponding location indicator 123D), and "ileocecal region" (corresponding location indicator 123D). In addition, arrow-shaped guide indicators 125A to 125E are shown for each area to be photographed. The arrow-shaped guide indicators 125A to 125E allow the user to recognize the direction of photography of the endoscope scope 10. Guide indicator 125A-1 indicates the direction of photography for "rectum". Guide indicator 125A-2 indicates the direction of photography for "anus". Guide display 125B indicates the imaging direction for imaging the "splenic flexure". Guide display 125C indicates the imaging direction for imaging the "hepatic flexure". Guide display 125D indicates the imaging direction for imaging the "ileocecal region". Guide display 125E indicates the imaging direction for imaging the "duodenal entrance". In this way, the present invention can also be applied to the large intestine by setting the target imaging area. In this example as well, as in the embodiments described above, if the second judgment process determines that the judgment criteria are met, a notification will be displayed in the area displays 123A to 123E, and this notification will be maintained until imaging of all target imaging areas is completed.

[0126] <Variation 6> Next, we will explain a modified example 6. In the example described above, a still image 39 was input as a medical image, and a first and second judgment process was performed based on that still image 39. In this example, a frame image 38a constituting a video 38 is input as a medical image, and a first and second judgment process is performed on that frame image 38a. In this example, the frame image 38a that satisfies the judgment criteria of the second judgment process is stored in memory 48 as a recorded image.

[0127] Figure 15 is a flowchart illustrating a medical image processing method using the medical image processing device 14.

[0128] First, the medical image acquisition unit 40 acquires a frame image 38a (step S20). Then, the first determination unit 42 performs a first determination process based on the acquired frame image 38a (step S21: first determination step). Specifically, the first determination unit 42 determines whether or not the frame image 38a contains any of the multiple target areas. If the first determination unit 42 determines that the frame image 38a does not contain any of the target areas (in the case of No in step S21), the second determination unit 43 does not perform a determination.

[0129] On the other hand, if the first determination unit 42 determines that the frame image 38a has any of the target areas to be photographed (in the case of Yes in step S21), the second determination unit 43 performs a second determination process based on the frame image 38a for which the first determination was made (step S22: second determination step). Specifically, the second determination unit 43 determines whether or not the area determined by the first determination unit 42 meets the determination criteria set for that area. If the second determination unit 43 determines that the determination criteria are not met (in the case of No in step S22), no notification is displayed on the display unit 16.

[0130] On the other hand, if the second determination unit 43 determines that the frame image 38a meets the determination criteria, the display control unit 46 displays a notification on the display unit 16 (step S23: display control process). Also, if the second determination process determines that the determination criteria are met, the processor stores the frame image 38a in the memory 48 as a recorded image (step S24: storage process). The frame images 38a are acquired sequentially in chronological order, and the above processing is performed sequentially on the acquired frame images 38a.

[0131] As explained above, in this example, a frame image 38a is input, and a first and second judgment are performed based on the input frame image 38a. In the second judgment, frame images 38a that meet the judgment criteria are stored as still images in memory 48. This makes it possible to efficiently acquire medical images that meet the judgment criteria.

[0132] <Example 7> Next, a modification 7 of this embodiment will be described. In this example, the display mode of the schematic diagram is changed in accordance with the change in the light source.

[0133] Figure 16 illustrates schematic diagrams of two different light sources.

[0134] The display of schematic diagram 101 and schematic diagram 131 is switched in accordance with the switching of the light source. Alternatively, the display of schematic diagram 101 and schematic diagram 131 may be switched according to the light source of the medical image (the light source used when the medical image was taken).

[0135] Schematic Figure 101 (First Display Mode) is displayed on the display unit 16 when the first light source is used. Here, the first light source is a white light source, which emits broadband light. The white light source is a light source used in examinations using a normal endoscopy system 9. Schematic Figure 131 (Second Display Mode) is displayed on the display unit 16 when a medical image from the second light source is acquired. Here, the second light source is a special light source, which emits narrowband light. The special light source is a light source used for observing a specific area or a specific lesion.

[0136] Schematic diagram 101, displayed when the first light source is used, shows six body part indicators 109A to 109F, similar to the explanation above (see Figure 8). When the first light source is used, the first and second judgment processes are performed on the target body parts corresponding to body part indicators 109A to 109F. In the diagram, notification displays are shown for body part indicators 109A and 109C.

[0137] When the second light source is used, the target area for imaging becomes a single location: the "esophagogastric junction." In the schematic diagram 131 displayed when the second light source is used, only the area indicator 109A corresponding to the "esophagogastric junction" is displayed. In the schematic diagram 131, the stomach region 105, which is an area where no target area is set for imaging, is displayed in a different color (e.g., gray) from other areas. In addition, in the schematic diagram 131, area indicators 109B to 109F, which are not target areas for imaging with the second light source, are not displayed. As a result, the schematic diagram 131 can show the range of the target area for imaging with the second light source (the range where the first and second judgment processes are performed).

[0138] In the example described above, five of the six target areas were only considered for evaluation when imaged with the first light source, while the remaining target area ("esophagogastric junction") was considered for evaluation when imaged with either the first or second light source. In other words, this example described a case where the number of target areas is reduced when medical images are taken with a narrowband light source compared to a broadband light source. However, if an appropriate light source is determined for a target area, the target area may be set according to the light source.

[0139] Next, we will explain the display variations of schematic diagrams 101 and 131, which are displayed according to the type of light source described above.

[0140] Figure 17 illustrates the first example of a display variation.

[0141] In this example, in schematic diagram 131 explained in Figure 16, area indicators 109B to 109F, which are outside the range of the area to be imaged, are also displayed. Specifically, in schematic diagram 131, area indicators 109A to 109F are displayed in the same way as in schematic diagram 101. On the other hand, in schematic diagram 131, the stomach area 105, which is an area where no area to be imaged is set, is displayed in a different color (e.g., gray) from the other areas, indicating that the stomach area 105 is outside the range of the area to be imaged. In this example, by displaying all area indicators 109A to 109F, the notification display when the first light source is used can be maintained. For example, the notification display for area indicator 109C in schematic diagram 131 is maintained because the notification display is maintained when imaging is performed with the first light source and a medical image that meets the judgment criteria is obtained.

[0142] Figure 18 illustrates a second example of the display variation.

[0143] In this example, the range of the area to be imaged is indicated by the display manner of the part indicators 109A to 109F. Specifically, in schematic diagram 131, part indicator 109A is shown with a solid line, and part indicators 109B to 109F are shown with dotted lines. This makes it possible to indicate that the area including part indicator 109A (esophagus 103) is the range of the area to be imaged, and that the area including part indicators 109B to 109F (stomach 105) is outside the range of the area to be imaged.

[0144] Figure 19 illustrates a third example of the display variation.

[0145] In this example, the range of the area to be imaged, which changes depending on the light source, is shown by changing the display of the range of the area to be imaged. Specifically, when the first light source is used, the areas to be imaged are the "esophagogastric junction," the "lesser curvature J-turn just below the cardia," and the "greater curvature U-turn just below the cardia." In schematic diagram 101, the range 133(A) including the area displays 109A~109C corresponding to the "esophagogastric junction," the "lesser curvature J-turn just below the cardia," and the "greater curvature U-turn just below the cardia," and the other ranges 133(B) are displayed in different colors. Furthermore, when the second light source is used, the areas to be imaged are the "posterior wall J-turn of the lesser curvature from the gastric angle or lower body," the "anterior part of the pylorus to the pylorus," and the "view of the greater curvature of the lower body." In schematic diagram 131, the range 133(B), which includes the area indicators 109D~109F corresponding to "J-turn of the posterior wall of the lesser curvature from the gastric angle or lower body," "from the anterior part of the pylorus to the pylorus," and "viewing down the greater curvature of the lower body," is displayed in a different color from the other range 133(A). This allows schematic diagrams 101 and 131 to show the range of the area to be photographed, which changes depending on the light source.

[0146] Figure 20 illustrates a fourth example of the display variation.

[0147] This example describes the process of acquiring medical images that meet the criteria for judgment using both the first and second light sources. Specifically, in this example, the target area for imaging, the "esophagogastric junction," is captured using both the first and second light sources, and images that meet the criteria for judgment in the second judgment process are acquired.

[0148] In schematic diagram 101, which is displayed when the first light source is used, the location indicator 109A is displayed as a notification. That is, the acquisition of a medical image that meets the criteria for the "esophagogastric junction" (the target area corresponding to location indicator 109A) has been completed with the first light source. Subsequently, when the light source is switched from the first light source to the second light source, schematic diagram 131 is displayed. In schematic diagram 131, location indicator 109A is displayed within the range of the target area, as explained in Figure 16. However, since the acquisition of a medical image that meets the criteria for the "esophagogastric junction" using the second light source has not yet been completed, the location indicator 109A in schematic diagram 131 is not displayed as a notification. Subsequently, when the acquisition of a medical image that meets the criteria for the "esophagogastric junction" using the second light source is completed, the location indicator 109A is displayed as a notification. This allows for the efficient acquisition of medical images that meet the criteria for the "esophagogastric junction" using both the first and second light sources.

[0149] <Other> In the above embodiment, the hardware structure of the processing unit (e.g., medical image acquisition unit 40, first determination unit 42, second determination unit 43, display control unit 46, audio control unit 47) that performs various processes is a variety of processors as shown below. These various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units; a Programmable Logic Device (PLD), such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacturing; and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which has a circuit configuration specifically designed to perform a particular process.

[0150] A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different type (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers such as client and server computers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of System-on-a-Chip (SoC) systems. Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.

[0151] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit composed of circuit elements such as semiconductor devices.

[0152] Each of the above-described configurations and functions can be appropriately implemented using any hardware, software, or a combination thereof. For example, the present invention can also be applied to a program that causes a computer to execute the above-described processing steps (processing procedures), a computer-readable recording medium (non-temporary recording medium) that records such a program, or a computer on which such a program can be installed.

[0153] Although examples of the present invention have been described above, it goes without saying that the present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention. [Explanation of Symbols]

[0154] 9: Endoscopy System 10: Endoscope 11:Light source device 12: Endoscope processor device 13:Display device 14: Medical image processing equipment 15:Operation unit 16: Display section 17: Speaker 20: Insertion part 21: Handheld control unit 22: Universal Code 25: Soft part 26: Curved section 27:Tip 28: Image sensor 29: Curved operating knob 30: Air / Water Supply Button 31: Suction button 32: Still image shooting instruction unit 33: Instrument entry port 35: Light Guide 36: Signal Cable 37a: Connector 37b: Connector 38: Video 38a: Frame image 39: Still image 40: Medical Image Acquisition Unit 41: CPU 42:First judger 43:Second determiner 46: Display Control Unit 47: Audio Control Unit 48: Memory

Claims

1. In a medical image processing device equipped with a processor, The aforementioned processor, A first determination process determines whether any of the multiple target areas are included in the medical image based on the acquired medical image, If, in the first determination process, it is determined that any of the multiple target areas is included in the medical image, a second determination process is performed to determine whether the medical image is an image that satisfies the criteria for determining which areas are included in the medical image. In the second determination process, if it is determined that the medical image is an image that satisfies the determination criteria for the part of the medical image, a display control process is performed to display a notification on the display unit indicating that an image satisfying the determination criteria for the part of the medical image has been captured. Perform The second determination process is a medical image processing device that determines the medical image based on the determination criteria associated with each of the plurality of target areas to be photographed.

2. The medical image processing apparatus according to claim 1, wherein the display control process maintains the display of the notification display until the notification display is displayed for all of the plurality of target areas.

3. The medical image processing apparatus according to claim 1, wherein the second determination process is performed by making a determination based on different determination criteria for each of the multiple target areas to be photographed.

4. Each of the second determination processes is performed based on a plurality of indicators, and is performed based on the determination results in the plurality of indicators, as described in claim 1.

5. The medical image processing apparatus according to claim 4, wherein the plurality of indicators include at least one of the following: blur detection, brightness detection, boundary visibility detection, cardia visibility detection, cardia distance detection, peristalsis detection, fold detection, treatment detection, and composition detection.

6. The medical image processing apparatus according to claim 1, wherein the second determination process is performed by determining each of the multiple target areas for imaging using a different determination device.

7. The first determination process is performed based on the determination result of a first determination unit composed of a convolutional neural network. The medical image processing apparatus according to claim 1, wherein at least a portion of the second determination process is performed by inputting intermediate feature quantities obtained in the first determination device to the second determination device, and the second determination device outputs a determination result.

8. The aforementioned medical images are captured using either the first or second light source. The medical image processing apparatus according to claim 1, wherein the first determination process, when determining the medical image captured using the first light source, performs a determination with respect to the plurality of target areas to be photographed, and when determining the medical image captured using the second light source, performs a determination with respect to a selected target area from the plurality of target areas to be photographed.

9. The aforementioned medical image is an endoscopic image taken with an endoscope. The medical image processing apparatus according to claim 1, wherein the display control process causes a schematic diagram of a tubular organ captured by the endoscope scope to be displayed on the display unit, and the notification display is displayed on the schematic diagram.

10. The medical image processing apparatus according to claim 9, wherein the display control process causes the part indications of the plurality of target areas to be photographed to be displayed on the schematic diagram at corresponding positions.

11. The medical image processing apparatus according to claim 10, wherein the display control process, when the second determination process determines that the medical image is an image that satisfies the determination criteria for a part included in the medical image, causes the notification display to be performed by changing the display format of the part display for the part.

12. The medical image processing apparatus according to claim 9, wherein the display control process causes a guide display to be shown on the schematic diagram to guide the imaging of the endoscope scope that captures the medical image.

13. The medical image processing apparatus according to claim 12, wherein the guide display is in the shape of a stick, the direction of the stick shape indicates the shooting direction of the endoscope scope, and the length of the stick shape indicates the distance of the endoscope scope from the subject.

14. The aforementioned medical images are captured using either the first or second light source. The medical image processing apparatus according to claim 9, wherein the display control process displays the schematic diagram in a first display mode when the medical image captured using the first light source is acquired, and displays the schematic diagram in a second display mode when the medical image captured using the second light source is acquired.

15. The medical image processing apparatus according to claim 1, wherein the display control process, with respect to all of the plurality of target areas for imaging, determines by the second determination process that the medical image is an image that satisfies the determination criteria for the area in which the medical image is included, and displays information on the display unit indicating that imaging of all target areas for imaging has been completed.

16. The aforementioned processor, The medical image processing apparatus according to claim 1, wherein, if the second determination process determines that the medical image is an image that satisfies the determination criteria for the part of the medical image that is included in the medical image, the apparatus performs a storage process to store the medical image in memory.

17. The medical image processing apparatus according to claim 1, wherein the plurality of imaging target areas are selected from at least the esophagogastric junction, the lesser curvature J-turn directly below the cardia, the greater curvature U-turn directly below the cardia, the posterior wall J-turn of the lesser curvature from the gastric angle or the lower body, from the anterior part of the pyloric ring to the pyloric ring, and looking down at the greater curvature of the lower body.

18. The medical image processing apparatus according to claim 1, wherein the plurality of target areas for imaging are selected from at least the rectum, anus, splenic flexure, hepatic flexure, duodenal opening, and ileocecal region.

19. A medical image processing method for a medical image processing device equipped with a processor, The aforementioned processor, A first determination step, based on the acquired medical image, determines whether any of the multiple target areas are included in the medical image. If, in the first determination step, it is determined that any of the multiple target areas is included in the medical image, a second determination step is performed to determine whether the medical image is an image that satisfies the criteria for determining which areas are included in the medical image. If, in the second determination step, the medical image is determined to be an image that satisfies the determination criteria for the part of the medical image, a display control step is performed to cause a notification display on the display unit indicating that an image satisfying the determination criteria for the part of the medical image has been captured. A medical image processing method comprising the second determination step, in which the medical image is determined based on the determination criteria associated with each of the plurality of target areas to be photographed.

20. A program that causes a medical image processing device equipped with a processor to execute a medical image processing method, The aforementioned processor, A first determination step, based on the acquired medical image, determines whether any of the multiple target areas are included in the medical image. If, in the first determination step, it is determined that any of the multiple target areas is included in the medical image, a second determination step is performed to determine whether the medical image is an image that satisfies the criteria for determining which areas are included in the medical image. If, in the second determination step, the medical image is determined to be an image that satisfies the determination criteria for the part of the medical image, the system will perform a display control step that causes a notification display on the display unit indicating that an image satisfying the determination criteria for the part of the medical image has been captured. The program instructs the second determination step to determine the medical image based on the determination criteria associated with each of the multiple target areas for imaging.

21. A non-temporary and computer-readable recording medium on which the program described in claim 20 is recorded.

Images