Medical image processing apparatus and medical image processing method

Abstract

The present application provides a medical image processing device, a medical image processing method and a program capable of effectively acquiring a desired medical image. In a medical image processing device (14) provided with a processor (41), the processor (41) performs: first determination processing of determining whether any of a plurality of photographing target sites is included in an acquired medical image based on the medical image; second determination processing of determining whether the medical image is an image satisfying a determination criterion for a site included in the medical image when it is determined in the first determination processing that any of the plurality of photographing target sites is included in the medical image; and display control processing of causing a display section to display a notification display indicating that an image satisfying the determination criterion for the site included in the medical image is photographed when it is determined in the second determination processing that the medical image is an image satisfying the determination criterion for the site included in the medical image.

Description

Technical Field

[0001] This invention relates to medical image processing apparatus, medical image processing methods and programs. Background Technology

[0002] In recent years, examinations have been conducted by using endoscopic systems to photograph the patient and performing medical examinations based on the acquired images. Furthermore, various techniques have been proposed to assist users during endoscopic imaging.

[0003] The technology described in Patent Document 1 proposes a technique for notifying the recognition results of medical images based on the actions of the operator (user) of the endoscope system.

[0004] Previous technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2020-146202 Summary of the Invention

[0007] The technical problem to be solved by the invention

[0008] Here, imaging of organs and other objects using an endoscopic system requires adherence to the imaging conditions and composition specified in the manual, which can present challenges depending on the location of the object being imaged. Especially for inexperienced users, capturing the desired medical images can be very difficult, and it can also be challenging to determine whether the appropriate medical images specified in the manual have been captured.

[0009] The present invention was made in view of the following circumstances, and its object is to provide a medical image processing apparatus, medical image processing method and program that can effectively acquire desired medical images.

[0010] means for solving technical problems

[0011] To achieve the above objectives, a medical image processing apparatus according to one aspect of the present invention includes a processor, wherein the processor performs: a first determination process, which determines, based on an acquired medical image, whether any one of a plurality of photographed object parts is included in the medical image; a second determination process, which, when the first determination process determines that any one of the plurality of photographed object parts is included in the medical image, determines whether the medical image is an image that satisfies the determination criteria for the part contained in the medical image; and a display control process, which, when the second determination process determines that the medical image is an image that satisfies the determination criteria for the part contained in the medical image, causes a display unit to display a notification indicating that an image satisfying the determination criteria for the part contained in the medical image has been captured.

[0012] According to this method, when the second determination process determines that a medical image meets the determination criteria for the part contained in the medical image, the display unit displays a notification indicating that an image meeting the determination criteria for the part contained in the medical image has been captured. Therefore, this method can notify the user that a medical image meeting the determination criteria set for each part has been captured, and can effectively acquire the desired medical image.

[0013] Preferably, in the display control process, the notification display is maintained until the notification display is displayed for all parts of the multiple subject parts.

[0014] Preferably, in the second determination process, the determination is made based on different determination criteria for each of the multiple subject parts.

[0015] Preferably, the second determination process is performed based on multiple indicators and based on the determination results of the multiple indicators.

[0016] Preferably, the multiple indicators include at least one of the following: blur jitter determination, brightness determination, boundary visibility determination, cardia visibility determination, cardia distance determination, peristalsis determination, wrinkle determination, processing determination, and composition determination.

[0017] Preferably, in the second determination process, a determination is made by using a different determiner for each of the multiple parts of the photographed object.

[0018] Preferably, the first determination process is performed based on the determination result of the first determiner composed of a convolutional neural network, and at least a portion of the second determination process is performed by inputting intermediate feature quantities obtained in the first determiner into the second determiner, and the second determiner outputs the determination result.

[0019] Preferably, the medical image is captured using a first light source or a second light source. In the first determination process, if the medical image is captured using the first light source, multiple subject areas are determined. If the medical image is captured using the second light source, a selected subject area from the multiple subject areas is determined.

[0020] Preferably, the medical image is an endoscopic image taken by an endoscopic observer. In the display control processing, the display unit displays a schematic diagram of the lumen organ taken by the endoscopic observer, and a notification display is shown on the schematic diagram.

[0021] Preferably, in the display control processing, the parts of multiple shooting objects are displayed at corresponding positions on the schematic diagram.

[0022] Preferably, in the display control processing, when the second determination process determines that the medical image is an image that meets the determination criteria for the part contained in the medical image, a notification is displayed by changing the display mode of the part.

[0023] Preferably, in the display control processing, a schematic diagram is used to guide the imaging of the endoscopic observer that captures medical images.

[0024] Preferably, the guide is displayed in the shape of a bar, the direction of the bar indicating the shooting direction of the endoscope, and the length of the bar indicating the distance of the endoscope from the subject.

[0025] Preferably, the medical image is captured using a first light source or a second light source. In the display control processing, when a medical image captured using the first light source is acquired, a schematic diagram is displayed in a first display mode; when a medical image captured using the second light source is acquired, a schematic diagram is displayed in a second display mode.

[0026] Preferably, in the display control processing, when the second determination process determines that all of the multiple subject areas are images that meet the determination criteria for the subject areas included in the medical image, the display unit displays information indicating that all subject areas have been captured.

[0027] Preferably, when the second determination process determines that the medical image is an image that meets the determination criteria for the part contained in the medical image, the processor performs storage processing to store the medical image in the memory.

[0028] Preferably, the multiple subject areas are selected from at least the esophagogastric junction, the lesser curvature J-shaped bend directly below the cardia, the greater curvature U-shaped bend directly below the cardia, the lesser curvature J-shaped bend on the posterior wall from the gastric angle or the lower part of the gastric body, and the greater curvature viewed from the anterior part of the pyloric ring to the pyloric ring and the lower part of the gastric body.

[0029] Preferably, the multiple imaging subjects are selected from at least the rectum, anus, splenic flexure, hepatic flexure, duodenal inlet, and ileocecal junction.

[0030] Another aspect of the present invention is a medical image processing method of the medical image processing apparatus equipped with a processor, wherein the processor performs: a first determination step, based on an acquired medical image, determining whether any one of a plurality of photographed object parts is included in the medical image; a second determination step, when it is determined in the first determination step that any one of the plurality of photographed object parts is included in the medical image, determining whether the medical image is an image that satisfies the determination criteria for the part contained in the medical image; and a display control step, when it is determined in the second determination step that the medical image is an image that satisfies the determination criteria for the part contained in the medical image, causing the display unit to display a notification indicating that an image satisfying the determination criteria for the part contained in the medical image has been captured.

[0031] Another aspect of the present invention is a program that causes a medical image processing apparatus equipped with a processor to execute a medical image processing method, wherein the processor executes: a first determination step, based on an acquired medical image, determining whether any one of a plurality of photographed object parts is included in the medical image; a second determination step, when it is determined in the first determination step that any one of the plurality of photographed object parts is included in the medical image, determining whether the medical image is an image that satisfies the determination criteria for the part contained in the medical image; and a display control step, when it is determined in the second determination step that the medical image is an image that satisfies the determination criteria for the part contained in the medical image, causing the display unit to display a notification indicating that an image satisfying the determination criteria for the part contained in the medical image has been captured.

[0032] Invention Effects

[0033] According to the present invention, when the medical image is determined to be an image that meets the determination criteria for the part contained in the medical image during the second determination process, a notification display indicating that an image that meets the determination criteria for the part contained in the medical image has been captured is displayed on the display unit. Therefore, the user can be notified that a medical image that meets the determination criteria set for each part has been captured, and the desired medical image can be effectively acquired. Attached Figure Description

[0034] Figure 1 It is a schematic diagram showing the overall structure of the endoscope system.

[0035] Figure 2 It is a block diagram showing the structure of the light source device.

[0036] Figure 3 This is a block diagram illustrating an implementation of a medical image processing device.

[0037] Figure 4 This is a flowchart illustrating a medical image processing method.

[0038] Figure 5 This is a diagram showing a specific structural example of the first and second decision units.

[0039] Figure 6 This diagram illustrates the data input to the first and second decision units.

[0040] Figure 7 This diagram illustrates specific examples of the various indicators of the second decision-maker.

[0041] Figure 8 This is a diagram showing a schematic representation of what is displayed on the display unit.

[0042] Figure 9 This diagram illustrates the notification display shown in the illustration.

[0043] Figure 10 This is a diagram illustrating the completion information displayed on the display unit.

[0044] Figure 11 This diagram illustrates the highlighted display of specific body parts.

[0045] Figure 12 This is a diagram illustrating a schematic diagram.

[0046] Figure 13 This is a diagram showing the inspection screen displayed on the display unit.

[0047] Figure 14 This is an illustration of the areas to be photographed during a colon examination.

[0048] Figure 15 This is a flowchart illustrating a medical image processing method.

[0049] Figure 16 This diagram illustrates two different light sources.

[0050] Figure 17 This is a diagram illustrating the first example of the displayed change.

[0051] Figure 18 This is a diagram illustrating the second example of the change.

[0052] Figure 19 This is a diagram illustrating the third example of the displayed changes.

[0053] Figure 20 This is a diagram illustrating the fourth example of the displayed changes. Detailed Implementation

[0054] The preferred embodiments of the medical image processing apparatus, medical image processing method and program involved in the present invention will now be described with reference to the accompanying drawings.

[0055] [The overall structure of an endoscope system, including medical image processing equipment]

[0056] Figure 1 This is a schematic diagram showing the overall structure of an endoscope system including the medical image processing apparatus according to the present invention. The medical images input to the medical image processing apparatus 14 are captured by the endoscope system described below. Furthermore, in the following description, the medical image processing apparatus 14 included in the endoscope system will be described, but embodiments of the present invention are not limited thereto. For example, medical images captured by an endoscope system separate from the medical image processing apparatus 14 may also be input to the medical image processing apparatus 14.

[0057] like Figure 1 As shown, the endoscope system 9 includes an endoscope observer 10 as an electronic endoscope, a light source device 11, an endoscope processor device 12, a display device 13, a medical image processing device 14, an operation unit 15, and a display unit 16.

[0058] The endoscope observer 10 captures time-series medical images including images of the subject, for example, an observer for the lower or upper digestive tract. The endoscope observer 10 has an insertion part 20 inserted into the subject (e.g., the stomach or large intestine) and having a front end and a base end; a hand-held operating part 21 connected to the base end of the insertion part 20 and for the operator (i.e., physician) to hold and perform various operations; and a universal tether 22 connected to the hand-held operating part 21.

[0059] The insertion part 20 has a small diameter and is formed into a long strip. The insertion part 20 is composed of a flexible soft part 25, a bending part 26 that can be bent by the operation of the hand operation part 21, and a front end part 27 that houses a camera optical system (objective lens) and camera element 28 (not shown) connected from its base end side to the front end side.

[0060] The imaging element 28 is a CMOS (complementary metal oxide semiconductor) or CCD (charge coupled device) type imaging element. The image light of the observed area passes through an observation window (not shown) that opens at the front end face of the front end 27, and an objective lens (not shown) disposed behind the observation window, and enters the imaging surface of the imaging element 28. The imaging element 28 captures the image light of the observed area incident on the imaging surface (converts it into an electrical signal), and outputs an imaging signal.

[0061] The handheld control unit 21 is equipped with various operating components operated by the user (such as a doctor operating the endoscope system 9). Specifically, the handheld control unit 21 is equipped with two bending operation buttons 29 for bending operation of the bending section 26, an air and water supply button 30 for air and water supply operation, and a suction button 31 for suction operation. In addition, the handheld control unit 21 is equipped with a still image shooting indicator 32 for shooting a still image 39 of the observed area, and a treatment device inlet 33 for inserting a treatment device (not shown) into the treatment device insertion path (not shown) in the insertion section 20.

[0062] The universal plug rope 22 is a connecting plug rope used to connect the endoscope observer 10 to the light source device 11. The universal plug rope 22 contains a light guide 35, a signal cable 36, and a fluid tube (not shown) that passes through the insertion part 20. Additionally, at one end of the universal plug rope 22 are a connector 37a that connects to the light source device 11 and a connector 37b that branches off from the connector 37a and connects to the endoscope processor device 12.

[0063] By connecting connector 37a to the light source device 11, light guide 35 and fluid tube (not shown) are inserted into the light source device 11. Thus, necessary illumination light, water, and gas are supplied to the endoscope observer 10 from the light source device 11 via light guide 35 and fluid tube (not shown). As a result, illumination light is projected onto the observed area from the illumination window (not shown) on the front end face of the front end 27. Furthermore, upon pressing the gas / water supply button 30, gas or water is sprayed from the gas / water supply nozzle (not shown) on the front end face of the front end 27 towards the observation window (not shown) on the front end face.

[0064] By connecting connector 37b to endoscope processor device 12, signal cable 36 is electrically connected to endoscope processor device 12. Thus, an image signal of the observed part is output from the camera element 28 of endoscope observer 10 to endoscope processor device 12 via signal cable 36, and a control signal is output from endoscope processor device 12 to endoscope observer 10.

[0065] The light source device 11 supplies illumination light to the light guide 35 of the endoscope observer 10 via connector 37a. The illumination light is selected from various wavelength bands corresponding to the observation purpose, such as white light (light in the white wavelength band or light in multiple wavelength bands), light in one or more specific wavelength bands, or combinations thereof. The light source device 11 will be described in detail below.

[0066] The endoscope processor device 12 controls the operation of the endoscope observer 10 via connector 37b and signal cable 36. Furthermore, based on the imaging signal acquired from the imaging element 28 of the endoscope observer 10 via connector 37b and signal cable 36, the endoscope processor device 12 generates an image (also referred to as "moving image 38") consisting of a time-series of frame images 38a containing images of the subject. Moreover, when the still image capturing instruction unit 32 is operated via the handheld operation unit 21 of the endoscope observer 10, the endoscope processor device 12 simultaneously acquires a still image 39 corresponding to the timing of the capturing instruction, along with the generation of the moving image 38. In this description, the medical image consists of the aforementioned still image 39 and frame images 38a.

[0067] Furthermore, when the moving image 38 and the still image 39 are images obtained using light (special light) of the aforementioned specific wavelength band, they are both special light images. Moreover, the endoscope processor device 12 outputs the generated moving image 38 and still image 39 to the display device 13 and the medical image processing device 14.

[0068] Furthermore, the endoscope processor device 12 can also generate (acquire) a special light image with information of the specific wavelength band described above, based on the ordinary light image obtained by the white light described above. In this case, the endoscope processor device 12 functions as a special light image acquisition unit. Moreover, the endoscope processor device 12 obtains a signal of the specific wavelength band by performing calculations based on the color information of red, green, and blue [RGB (Red, Green, Blue)] or cyan, magenta, and yellow [CMY (Cyan, Magenta, Yellow)] contained in the ordinary light image.

[0069] Furthermore, the endoscope processor device 12 can generate a known characteristic image, such as an oxygen saturation image, based on at least one of a normal light image obtained by the aforementioned white light and a special light image obtained by the aforementioned light of a specific wavelength band (special light). In this case, the endoscope processor device 12 functions as a characteristic image generation unit. Moreover, the frame image 38a or still image 39 constituting the dynamic frame 38, which includes the aforementioned in vivo image, normal light image, special light image, and characteristic image, are all medical images obtained by imagerizing the results of photographing or measuring a human body for the purpose of image-based examination.

[0070] The display device 13 is connected to the endoscope processor device 12 and functions as a display unit 16 that displays the dynamic image 38 and still image 39 input from the endoscope processor device 12. While checking the dynamic image 38 displayed on the display device 13, the user performs operations such as advancing and retracting the insertion unit 20. When a lesion is found at the target site or the observed site, the user operates the still image capture indicator 32 to perform still image capture of the observed site. Furthermore, diagnostic procedures, biopsies, and other treatments can be performed. Similarly, the dynamic image 38 and still image 39 are also displayed on the display unit 16, which is connected to the medical image processing device 14 described later.

[0071] also, Figure 1 The endoscope system 9 shown is a specific example, but other endoscope systems can also be used. For example, the first decision-maker 42 as described below ( Figure 3 ) and the second decision unit 43 ( Figure 3 The processing of the learned model can also be installed on a dedicated AI (Artificial Intelligence) box and connected to the endoscope system. Alternatively, the endoscope system can connect to a dedicated server via a network for medical image processing.

[0072] <Structure of the Light Source Device>

[0073] Figure 2 This is a block diagram showing the structure of the light source device 11.

[0074] like Figure 2 As shown, the light source device 11 comprises a light source 310 for illumination, an aperture 330, a condenser lens 340, and a light source control unit 350, etc., allowing observation light to be incident on the light guide 35. The light source 310 is equipped with a red light source 310R, a green light source 310G, a blue light source 310B, and a violet light source 310V, which respectively illuminate narrow bands of red, green, blue, and violet light, and can illuminate narrow bands of red, green, blue, and violet light. The illuminance of the observation light based on the light source 310 is controlled by the light source control unit 350, which can change (increase or decrease) the illuminance of the observation light and stop illumination as needed.

[0075] The light source 310 can emit narrowband light of red, green, blue, and violet in any combination. For example, it can emit narrowband light of red, green, blue, and violet simultaneously to illuminate white light (ordinary light) as observation light, or it can illuminate narrowband light (special light) by emitting any one or two types of light. The light source 310 may also be equipped with an infrared light source to illuminate infrared light (an example of narrowband light). Alternatively, it can illuminate white light and narrowband light as observation light by using a light source that illuminates white light and filters that allow white light and each narrowband light to pass through.

[0076] <Wavelength band of the light source>

[0077] Light source 310 can be a light source that generates light in the white frequency band, or a light source that generates light in multiple wavelength bands as the white frequency band, or a light source that generates light in a specific wavelength band narrower than the white wavelength band. The specific wavelength band can be the blue or green band of the visible range, or the red band of the visible range. When the specific wavelength band is the blue or green band of the visible range, it can also include wavelength bands between 390nm and 450nm, or between 530nm and 550nm, and the light in the specific wavelength band has a peak wavelength within the wavelength band between 390nm and 450nm or between 530nm and 550nm. Alternatively, when the specific wavelength band is the red band of the visible region, it can also include wavelength bands between 585nm and 615nm, or between 610nm and 730nm, and the light in the specific wavelength band has a peak wavelength within the wavelength band between 585nm and 615nm or between 610nm and 730nm.

[0078] The aforementioned specific wavelength band may also include wavelength bands with different absorption coefficients in oxyhemoglobin and deoxyhemoglobin, and the light in the specific wavelength band has a peak wavelength in the wavelength band with different absorption coefficients in oxyhemoglobin and deoxyhemoglobin. In this case, the specific wavelength band may also include wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm, and the light in the specific wavelength band has a peak wavelength in the wavelength band of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm.

[0079] In addition, the wavelength band of the light generated by the light source 310 may also include wavelength bands of 790nm to 820nm or 905nm to 970nm, and the light generated by the light source 310 has a peak wavelength in the wavelength bands of 790nm to 820nm or 905nm to 970nm.

[0080] Alternatively, the light source 310 may also be equipped with an excitation light source that illuminates a peak wavelength of 390 nm or higher and 470 nm or lower. In this case, endoscopic images containing information about the fluorescence emitted by fluorescent substances within the subject (organism) can be acquired. When acquiring fluorescence images, fluorescent dyes (fluorescein, acridine orange, etc.) can also be used.

[0081] The type of light source 310 (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode) etc.), wavelength, and presence or absence of filters are preferably configured according to the type, part, and observation purpose of the subject. Furthermore, it is preferable that the wavelength of the observation light is combined and / or switched according to the type, part, and observation purpose of the subject during observation. When switching wavelengths, for example, the wavelength of the illuminated light can be switched by rotating a disc-shaped filter (rotating color filter) positioned in front of the light source and equipped with filters that transmit or block light of specific wavelengths.

[0082] Furthermore, the imaging element 28 used in implementing this invention is not limited to a color imaging element with a color filter for each pixel, but can also be a monochrome imaging element. When using a monochrome imaging element, the wavelengths of the observation light can be switched sequentially and images can be taken in a surface order (color order). For example, the wavelengths of the emitted observation light can be switched sequentially between (violet, blue, green, red), or broadband light (white light) can be irradiated, and the wavelengths of the emitted observation light can be switched by rotating the color filter (red, green, blue, violet, etc.). Alternatively, one or more narrowband lights (green, blue, violet, etc.) can be irradiated, and the wavelengths of the emitted observation light can be switched by rotating the color filter (green, blue, violet, etc.). The narrowband light can also be infrared light of two or more wavelengths with different wavelengths (first narrowband light, second narrowband light).

[0083] [Medical Image Processing Device]

[0084] Figure 3 This is a block diagram illustrating an embodiment of the medical image processing apparatus 14. The medical image processing apparatus 14 acquires medical images and performs first determination processing and second determination processing based on the acquired medical images. Furthermore, the medical image processing apparatus 14 performs display control processing to cause the display unit 16 to display a notification display based on the determination result of the second determination processing.

[0085] The medical image processing device 14 is, for example, a computer. The operation unit 15 includes, in addition to a keyboard and mouse that are wired or wirelessly connected to the computer, buttons that are located on the hand operation unit 21 of the endoscope observer 10. The display unit 16 uses various monitors such as an LCD monitor that can be connected to a computer.

[0086] The medical image processing device 14 comprises a medical image acquisition unit 40, a CPU (Central Processing Unit) 41, a first determination unit 42, a second determination unit 43, a display control unit 46, a voice control unit 47, and a memory 48. The processing of each unit is implemented by one or more processors. Here, the processor may be CPU 41, or it may be one or more CPUs (not shown).

[0087] The CPU 41 operates based on various programs, including the operating system stored in the memory 48 and the medical image processing program involved in this invention, and uniformly controls the medical image acquisition unit 40, the first decision unit 42, the second decision unit 43, the display control unit 46, and the voice control unit 47. In addition, it functions as a part of these units.

[0088] The medical image acquisition unit 40 acquires medical images. The medical image acquisition unit 40 is used in conjunction with the endoscope processor device 12. Figure 1 The medical image acquisition unit 40 acquires medical images, including images of the subject, from the endoscope processor device 12 via a wired or wireless image input / output interface (not shown). For example, when the endoscope observer 10 captures a still image 39 as described earlier during the capture of a dynamic image 38, the medical image acquisition unit 40 acquires the still image 39 from the endoscope processor device 12. Alternatively, the medical image acquisition unit 40 can also acquire frame images 38a constituting the dynamic image 38 as medical images.

[0089] The first determiner 42 performs a first determination process based on the medical images acquired by the medical image acquisition unit 40 to determine whether any one of a plurality of pre-set target areas is included in the medical images. Here, the target area refers to the area that must be intentionally photographed and recorded during the examination. In addition, the target area is the area that needs to be photographed to perform the desired operation of the endoscope 10. The target area is an area that is relatively difficult to photograph, and also an area where it is difficult to determine whether the user has photographed it appropriately.

[0090] The number of imaging sites can be set depending on the purpose of the examination. For example, the number of imaging sites can be set to 5 to 15, preferably 5 to 10. For example, when examining the inside of the stomach using the endoscope system 9, the imaging sites can be set as "esophagogastric junction", "small bend J-shaped turn directly below the cardia (based on J-shaped turn operation)", "large bend U-shaped turn directly below the cardia (based on U-shaped turn operation)", "small bend posterior wall J-shaped turn from the gastric angle or lower part of the gastric body", "from the anterior part of the pyloric ring to the pyloric ring", and "great bend viewed from the lower part of the gastric body". In addition, "small bend posterior wall J-shaped turn from the gastric angle or lower part of the gastric body" can be set to "small bend J-shaped turn from the lower part of the gastric body" because it is not always possible to photograph the gastric angle and posterior wall. In addition, "from the anterior part of the pyloric ring to the pyloric ring" can be set to "panoramic view of the vestibule" which places more emphasis on whether the vestibule can be photographed from above compared to photographing the vestibule from a target point. Furthermore, "view of the greater curvature from the lower part of the stomach body" is not limited to the lower part of the stomach body; it can also be set to "view of the greater curvature from above" that emphasizes the opening of the folds. In addition, for example, when examining the inside of the large intestine using the endoscope system 9, the target areas are set as "rectum", "anus", "splenic flexure", "hepatic flexure", "duodenal inlet", and "ileocecal junction".

[0091] The first determiner 42 identifies the subject area contained in the input medical image using various methods. For example, the first determiner 42 identifies the subject area contained in the input medical image using a learned model (recognition model) composed of a convolutional neural network, etc. The first determiner 42 learns an image (medical image) to identify a pre-defined subject area, generates a learned model, and uses the learned model to determine whether the medical image contains the pre-defined subject area.

[0092] The first decision-maker 42 can also determine and recognize the input medical image based on classification or similarity. When the first decision-maker 42 classifies the medical image and identifies the subject area, it can utilize the techniques described in the literature (B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. In Neural Information Processing Systems (NIPS), pages 487-495, 2014.1, 4, 6, 8). Furthermore, when the first decision-maker 42 identifies the subject area based on the similarity of the feature quantities of the medical image, it can utilize the techniques described in the literature (FaceNet: A Unified Embedding for Face Recognition and Clustering https: / / arxiv.org / abs / 1503.03832).

[0093] When the first determiner 42 determines that any one of the multiple subject areas is included in the medical image, the second determiner 43 determines whether the medical image meets the criteria for determining whether the subject area is included in the medical image. That is, the second determiner 43 determines whether the subject area has been appropriately captured as the desired medical image in the medical image. A medical image in which the subject area has been captured as desired is a good image and is suitable for diagnosis.

[0094] The second determiner 43 is configured for each part of the photographed object, and each second determiner 43 can make a determination based on different determination criteria. Furthermore, the determination of the second determiner 43 is based on multiple indicators and the determination results of these multiple indicators are used for the final determination. A detailed description of the first determiner 42 and the second determiner 43 will follow.

[0095] If, during the second determination process, the medical image is determined to be an image that meets the determination criteria for the part contained in the medical image, the display control unit 46 causes the display unit 16 to display a notification indicating that an image that meets the determination criteria for the part contained in the medical image has been captured. Furthermore, the display performed by the display unit 16 via the display control unit 46 can also be performed on the display device 13.

[0096] The voice control unit 47 plays a notification sound from the speaker 17. For example, when a notification is displayed according to the notification display in the display control unit 46 described above, a notification sound is played from the speaker 17.

[0097] The memory 48 includes flash memory, ROM (Read-only Memory), RAM (Random Access Memory), and a hard disk drive. Flash memory, ROM, and the hard disk drive are non-volatile memories that store the operating system, various programs such as the medical image processing program involved in this invention, and captured still images 39 and moving images 38. RAM, on the other hand, is a volatile memory that functions as an area temporarily storing various programs stored in non-volatile memory and as the operating area of ​​the CPU 41, allowing for fast data read and write operations.

[0098] <Medical Image Processing Methods>

[0099] Next, a medical image processing method using the medical image processing apparatus 14 will be described. For example, the medical image processing method is performed by executing a program through the processor of the medical image processing apparatus 14.

[0100] Figure 4 This is a flowchart illustrating a medical image processing method using the medical image processing device 14.

[0101] First, the medical image acquisition unit 40 acquires a medical image (e.g., still image 39) from the endoscope processor device 12 (step S10). Then, the first determiner 42 performs a first determination process based on the acquired medical image (step S11: first determination process). Specifically, the first determiner 42 determines whether the medical image contains any of the multiple target areas. When the first determiner 42 determines that the medical image does not contain any target area (No in step S11), the second determination process in the subsequent second determiner 43 is not performed.

[0102] On the other hand, when the first determiner 42 determines that the medical image contains any part of the photographed object (Yes in step S11), the second determiner 43 performs a second determination process based on the medical image that has undergone the first determination process (step S12: second determination process). Specifically, the second determiner 43 determines whether the determination criteria set for the part determined by the first determiner 42 are met according to the set indicators. When the second determiner 43 determines that the determination criteria are not met (No in step S12), no notification is displayed on the display unit 16.

[0103] On the other hand, when the second determiner 43 determines that the medical image meets the determination criteria, the display control unit 46 displays a notification on the display unit 16 (step S13: display control process). Furthermore, the still image 39 is acquired based on the instruction of the still image acquisition instruction unit 32, and the acquired still image 39 is processed in the above manner.

[0104] The first determination process, the second determination process, and the display control process described above will be explained in detail below.

[0105] <First Judgment Processing and Second Judgment Processing>

[0106] First, the first determination process performed in the first determination step and the second determination process performed in the second determination step will be explained.

[0107] Figure 5 This is a diagram illustrating a specific structural example of the first determiner 42 performing the first determination process and the second determiner 43 performing the second determination process.

[0108] The first decision-maker 42 is a learned model composed of a CNN, which has undergone machine learning beforehand. Six locations inside the stomach are designated as the subjects of the imaging (see reference). Figure 7 and Figure 8 The first determiner 42 determines whether six locations inside the stomach are included in the medical image. Specifically, the first determiner 42 determines whether the following locations are included in the medical image: the "esophagogastric junction," the "J-shaped bend directly below the cardia (based on J-shaped bend operation)," the "U-shaped bend directly below the cardia (based on U-shaped bend operation)," the "J-shaped bend on the posterior wall of the lesser bend from the gastric angle or lower part of the gastric body," the "from the anterior part of the pyloric ring to the pyloric ring," and the "greater bend viewed from the lower part of the gastric body." Furthermore, the first determiner 42 learns from training data consisting of medical images capturing the six locations inside the stomach to identify each of the aforementioned locations.

[0109] For example, the first determiner 42 identifies the subject area of ​​the medical image based on a classification score for each of the six subject areas mentioned above. The first determiner 42 identifies the subject area with the highest classification score among the classification scores output for each subject area. Furthermore, a threshold is set in the first determiner 42; when the classification score is below the threshold, it is determined that no subject area is included in the medical image.

[0110] When the first determiner 42 identifies that the medical image contains the area of ​​the subject being photographed, a second determination process is performed in the second determiners 43a to 43f corresponding to the identified area of ​​the subject being photographed. For example, when the first determiner 42 determines that the input medical image contains the "esophagogastric junction", the second determination process is performed by the second determiner 43a. Similarly, when the first determiner 42 determines that the medical image contains a "small J-shaped bend directly below the cardia (based on J-shaped bend operation)," a second determination process is performed by the second determiner 43b. When the first determiner 42 determines that the medical image contains a "large U-shaped bend directly below the cardia (based on U-shaped bend operation)," a second determination process is performed by the second determiner 43c. When the first determiner 42 determines that the medical image contains a "small posterior wall J-shaped bend from the gastric angle or lower part of the gastric body," a second determination process is performed by the second determiner 43d. When the first determiner 42 determines that the medical image contains "from the anterior part of the pyloric ring to the pyloric ring," a second determination process is performed by the second determiner 43e. When the first determiner 42 determines that the medical image contains "the large curvature viewed from the lower part of the gastric body," a second determination process is performed by the second determiner 43f. Thus, the second determiners 43a to 43f are configured for each part of the subject being photographed, as determined by the first determiner 42. Furthermore, the second determiners 43a to 43f can make determinations based on multiple indicators and different determination criteria.

[0111] Figure 6 This diagram illustrates the data input to the first decision unit 42 and the second decision unit 43.

[0112] A medical image (still image 39) acquired by the medical image acquisition unit 40 is input to the first determiner 42. Based on the input medical image, the first determiner 42 performs calculations in each layer constituting the CNN to generate intermediate feature values ​​of the input medical image. Furthermore, the first determiner 42 outputs a determination result based on the intermediate feature values.

[0113] Similar to the first determiner 42, the medical image acquired by the medical image acquisition unit 40 is input to the second determiner 43. Additionally, intermediate feature values ​​generated by the first determiner 42 are input to the second determiner 43. The second determiner 43 outputs a determination result based on the medical image and the intermediate feature values. Thus, the first determiner 42 outputs a determination result based on the medical image, and the second determiner 43 outputs a determination result based on the medical image and the intermediate feature values.

[0114] Figure 7 This diagram illustrates specific examples of the various indicators of the second decision makers 43a to 43f.

[0115] The second decision-makers 43a to 43f output decision results based on multiple indicators. The indicator for the second decision-maker 43a is represented by symbol 52, the indicator for the second decision-maker 43b by symbol 53, the indicator for the second decision-maker 43c by symbol 54, the indicator for the second decision-maker 43d by symbol 55, the indicator for the second decision-maker 43e by symbol 56, and the indicator for the second decision-maker 43f by symbol 57. Furthermore, symbol 51 indicates that no second decision processing is performed when the first decision-maker 42 determines that the medical image does not contain any part of the photographed object.

[0116] The symbol 52 denotes the second determination process when the first determiner 42 determines that the medical image contains the "esophagogastric junction". In this case, the second determiner 43a performs the second determination process based on indicators of blur and jitter determination, brightness determination, and boundary visibility determination. Here, blur and jitter determination is based on the input medical image to determine whether the medical image has blur and / or jitter. Blur and jitter determination is performed by a blur and jitter determiner, using known techniques such as Fast Fourier Transform (FFT) to determine the blur and jitter of the medical image. In addition, brightness determination is based on the input medical image, for example, based on the luminance of the medical image. In addition, boundary visibility determination is performed using a boundary visibility determiner based on a learned model composed of a CNN. The boundary visibility determiner determines whether the junction of the stomach and esophagus can be seen in the medical image using the input intermediate feature quantities. The second determiner 43a determines whether the determination results of each indicator of blur and jitter determination, brightness determination, and boundary visibility determination of the medical image meet the prescribed determination criteria. Here, as a judgment criterion, a threshold is set for the value obtained by adding up the judgment results of each indicator, the average value, or the weighted average value.

[0117] Symbol 53 denotes the second determination process when the first determiner 42 determines that the medical image contains a "small J-shaped bend directly below the cardia (based on J-shaped bend operation)". In this case, the second determiner 43b performs the second determination process based on indicators such as blur / jitter determination, brightness determination, cardia visibility determination, and cardia distance determination. The cardia visibility determination and cardia distance determination are performed using a cardia visibility determiner and a cardia distance determiner based on a learned model constructed from a CNN. The cardia visibility determiner determines whether the cardia can be seen in the medical image based on the input intermediate feature value. The cardia distance determiner determines the distance from the front end of the endoscope observer 10 (the imaging surface of the imaging element 28) to the cardia in the medical image based on the input intermediate feature value.

[0118] Symbol 54 denotes the second determination process when the first determiner 42 determines that the medical image contains a "large U-shaped bend directly below the cardia". In this case, the second determiner 43c performs the second determination process based on indicators of blur / jitter determination, brightness determination, processing determination, and composition determination. The processing determination and composition determination are performed using a processing determiner and a composition determiner based on a learned model constructed from a CNN. The processing determiner determines whether water, debris, or bubbles are accumulated in the part of the medical image based on the input intermediate feature values. In addition, the composition determiner determines the composition of the medical image based on the input intermediate feature values. For example, the composition determiner determines whether the subject part is centered in the medical image.

[0119] Symbol 55 indicates a second determination process when the first determiner 42 determines that the medical image contains a "J-shaped bend from the posterior wall of the small curve of the stomach angle or lower part of the stomach body". In this case, the second determiner 43d performs the second determination process based on indicators of blur jitter determination, brightness determination, and composition determination.

[0120] Symbol 56 denotes the second determination process when the first determiner 42 determines that the medical image contains "from the anterior part of the pyloric ring to the pyloric ring". In this case, the second determiner 43e performs the second determination process based on indicators of blur jitter determination, brightness determination, creep determination, and composition determination. Here, creep determination is performed using a creep determiner based on a learned model constructed from a CNN. The creep determiner determines whether creep exists in the subject area of ​​the medical image based on the input intermediate feature values.

[0121] Symbol 57 denotes the second determination process when the first determiner 42 determines that the medical image contains the "greater curvature of the lower part of the stomach viewed from above." In this case, the second determiner 43f performs the second determination process based on indicators such as blur / jitter determination, brightness determination, processing determination, composition determination, and wrinkle determination. Here, wrinkle determination is performed using a wrinkle determiner based on a learned model constructed from a CNN. The wrinkle determiner determines whether wrinkles at the object location in the medical image extend based on the input intermediate feature values.

[0122] Furthermore, the fuzziness jitter determination is performed on a common benchmark in the second determiners 43a-43f, while the determination of other indicators is performed independently in the second determiners 43a-43f. Additionally, the above description illustrates an example of obtaining intermediate feature quantities from the first determiner 42, but it is not limited to this. For example, the second determiners 43a-43f may also calculate feature quantities and make determinations separately.

[0123] As described above, in the second determination process, determinations using the learned model (boundary visibility determination, gastric gate visibility determination, gastric gate distance determination, handling determination, composition determination, creep determination, and wrinkle determination) and determinations using pixel values ​​without using the learned model (blur jitter determination and brightness determination) are combined. By separating the determinations using the learned model from the determinations using pixel values ​​without using the learned model according to the metrics constituting the second determination process, determination accuracy or determination speed can be optimized. Furthermore, in the second determination process, by utilizing multiple learned models for determination, determination accuracy can be improved.

[0124] As described above, in the first determination process, it is determined whether any of the target body parts are included in the medical image. Furthermore, in the second determination process, based on the result of the first determination process, and based on different indicators for each target body part, it is determined whether the desired medical image was captured according to various determination criteria.

[0125] Furthermore, the above descriptions of the first and second determination processes are specific examples and are not limited to them. The first and second determination processes can also be implemented in the following ways.

[0126] The above example illustrates a second determination process performed on all the subject areas determined to be included in the medical image during the first determination process; however, the present invention is not limited to this example. For instance, the second determination process can also be performed on a portion of the subject areas determined to be included in the medical image during the first determination process. Specifically, in the first determination process, it is determined whether any one of the 13 subject areas is included in the medical image, and the second determination process is performed when it is determined that the subject areas of six predetermined areas are included in the medical image. Thus, for example, by setting the second determination process to be performed on the areas for which a medical image suitable for diagnosis needs to be obtained, the desired medical image can be obtained effectively.

[0127] In the examples above, the criterion for brightness determination was explained as being different for each part of the subject being photographed. However, the criterion for brightness determination can also be common to all parts of the subject being photographed. Furthermore, by setting a brightness determination criterion for each part of the subject being photographed, it is possible to determine, for example, whether the brightness is suitable for diagnosis of each part of the subject being photographed.

[0128] <Display Control Processing>

[0129] Next, the display control processing performed in the display control process will be explained.

[0130] Figure 8 This is a schematic diagram showing the display on the display unit 16 via the display control unit 46.

[0131] Schematic diagram 101 is a model image of a luminal organ taken by the endoscopic observer 10. Schematic diagram 101 consists of a stomach 105, an esophagus 103, and a duodenum 107, with the esophagus 103 and duodenum 107 connected to the stomach 105. In schematic diagram 101, part numbers 109A to 109F are shown at corresponding positions to represent the modeled object. Here, part number 109A schematically represents the "esophagogastric junction", part number 109B schematically represents the "J-shaped bend of the lesser curvature directly below the cardia", part number 109C schematically represents the "U-shaped bend of the greater curvature directly below the cardia", part number 109D schematically represents the "J-shaped bend of the posterior wall of the lesser curvature from the gastric angle or lower part of the gastric body", part number 109E schematically represents "from the anterior part of the pyloric ring to the pyloric ring", and part number 109F schematically represents the "greater curvature viewed from the lower part of the gastric body".

[0132] Additionally, in location displays 109A to 109F, guidance displays for taking images using the endoscope observer 10 are shown. Specifically, guidance display 111A is shown on location display 109A, guidance display 111B is shown on location display 109B, guidance display 111C is shown on location display 109C, guidance display 111D is shown on location display 109D, guidance display 111E is shown on location display 109E, and guidance display 111F is shown on location display 109F. Guidance displays 111A to 111F are each bar-shaped, and the direction of the bar shape indicates the imaging direction of the endoscope observer 10. Furthermore, the length of the bar shape indicates the distance between the endoscope observer 10 and the target area. Specifically, the front end of the opposite side of each bar-shaped location display 109A to 109F indicates the imaging position (the position of the front end 27 of the endoscope) where an image satisfying the judgment criteria can be easily obtained during the second judgment process. When the front end 27 of the endoscope 10 is positioned at the rod-shaped portion and the image is taken from the opposite side of the front end (109A-109F), a desired medical image (e.g., a composition suitable for diagnosis) can be acquired. Thus, in the composition determination of the second determination process, by indicating the position where a medical image that meets the determination criteria can be captured, the acquisition of a medical image suitable for diagnosis can be assisted.

[0133] When the subject area is photographed through the endoscope system 9, the display control unit 46 causes the display unit 16 to display the schematic diagram 101.

[0134] Figure 9 This diagram illustrates the notification display shown in the illustration.

[0135] When an image is determined to meet the determination criteria in the second determination process, the display control unit 46 causes the display unit 16 to display a notification indicating this situation. The display control unit 46 performs the notification display by changing the display mode of the part displays 109A to 109F in the schematic diagram 101. The notification display is performed by changing the display mode of the part displays 109A to 109F that correspond to the part that meets the determination criteria in the second determination process. In the illustrated case, the notification display is performed by changing the color of the part displays 109A to 109F. In addition, since the acquisition of the desired medical image has been completed, the display control unit 46 hides the guide display corresponding to the part displays 109A to 109F that have been notified. Alternatively, the display of the guide display corresponding to the part displays 109A to 109F that have been notified can also be maintained. In addition, the notification display can take various forms. For example, as a notification display, the part display can also be blinked or lit. Alternatively, a notification sound can be output by the voice control unit 47 through the speaker 17 along with the notification display.

[0136] As described above, in this embodiment, the first determination process determines whether any one of the multiple subject areas is included in the medical image. The second determination process determines whether the medical image meets the determination criteria for the included subject area. Furthermore, if the medical image meets the determination criteria, a notification is displayed on the display unit 16. Thus, the user can effectively acquire the desired medical image that meets the determination criteria for the subject area being photographed.

[0137] <Variation Example 1>

[0138] Next, a variation of this embodiment, Example 1, will be described. In this example, after all the subject areas have been photographed, information indicating that the photographing of all subject areas has been completed (completion information) is displayed on the display unit 16. By displaying the completion information on the display unit 16, the user can easily identify the completion status of the photographing of the subject areas.

[0139] Figure 10 This diagram illustrates the completion information displayed on the display unit 16.

[0140] The symbol 60 indicates an example of a display unit 16 at the beginning stage of an endoscopic examination of the stomach 105. The display unit 16 shows a schematic diagram 101, which includes part displays 109A to 109F indicating the area to be photographed. In the case indicated by symbol 60, since there is no area to be photographed yet, part displays 109A to 109F are all un-photographed displays.

[0141] Symbol 61 indicates a display example of a state where the "esophagogastric junction," "J-shaped bend in the posterior wall of the lesser curvature from the gastric angle or lower part of the gastric body," and "from the anterior part of the pyloric ring to the pyloric ring" meet the judgment criteria in the second judgment process during an endoscopic examination of the stomach 105. In the schematic diagram 101 displayed on the display unit 16, notifications are displayed by changing the display modes of location displays 109A, 109D, 109E, and 109F. Furthermore, the display control unit 46 maintains the notification display until notifications are displayed for all locations of the multiple imaging targets. This allows the user to reliably monitor the progress of imaging of the imaging targets. Additionally, the schematic diagram 101 can be turned on / off to the display unit 16 based on events such as still image acquisition or lesion detection. When the schematic diagram 101 is opened during the examination, the display of the schematic diagram 101 begins from the start of the examination until the moment the schematic diagram 101 is opened, displaying notifications for locations that meet the judgment criteria in the second judgment process.

[0142] The display unit 16, indicated by symbol 62, shows a stage where the imaging of all subject areas has been completed. When all subject areas are determined by the second determination process to be images that meet the determination criteria for medical imaging, completion information 113 indicating that the imaging of all subject areas has been completed is displayed on the display unit 16. The completion information 113 can be displayed in various ways. In the illustrated case, the completion information 113 is represented by a checkbox. Alternatively, the brightness of the notification displays for the subject areas 109A to 109F can be reduced as the completion information 113 is illuminated. Furthermore, after a certain period of time following the display of the completion information 113, the displays for the subject areas 109A to 109F can be stopped, and only the completion information 113 is displayed.

[0143] As described above, in this example, when all parts of the subject have been photographed, completion information 113 is displayed on the display unit 16. This allows the user to easily identify the completion status of photographing all parts of the subject.

[0144] <Variation Example 2>

[0145] Next, a variation of this embodiment, example 2, will be described. In this example, the part of the newly photographed subject is highlighted.

[0146] Figure 11 This diagram illustrates the highlighted display of specific body parts.

[0147] In schematic diagram 101, location displays 109A, 109E, and 109F have completed the acquisition of medical images that meet the judgment criteria and are displayed as notifications. Additionally, location display 109D represents a newly acquired location for which an image meeting the judgment criteria has been captured; the location display 109D corresponding to this acquired location is highlighted. Furthermore, the notification displays of location displays 109A, 109E, and 109F and the highlight display of location display 109D use different display methods. For example, the notification display and the highlight display use different colors and brightness levels.

[0148] In this way, by highlighting the newly captured medical image that meets the judgment criteria, the user can easily identify the newly captured object area. Furthermore, the duration of the highlighting is determined by a certain time, the time until the next medical image (still image 39) is captured, the time until the first judgment process determines that the next captured object area is included in the medical image, or the time until the second judgment process determines that the next judgment criteria are met, or a combination of these times.

[0149] <Variation Example 3>

[0150] Next, a variation of this embodiment, 3, will be described. In this example, during the first determination process, a notification is also displayed for the part corresponding to the part of the object being photographed that is determined to be included in the medical image.

[0151] Figure 12 This is a diagram illustrating a display example of schematic diagram 101.

[0152] In the part display 109D of schematic diagram 101, a first determination display 115D is shown. The first determination display 115D is displayed in a manner that surrounds the part display 109D. The first determination display 115D is displayed when the first determiner 42 determines that the subject part corresponding to the part display 109D is included in the medical image. Similarly, in the other part displays 109A-109C, 109E, and 109F, the first determination display is performed when it is determined that the subject part is included in the medical image. Thus, by displaying the first determination display 115D indicating the situation detected in the first determination process, the user can identify at least the corresponding subject part being included in the medical image.

[0153] <Variation Example 4>

[0154] Next, a variation of this embodiment, example 4, will be described. In this example, schematic diagram 101 is displayed overlaid on the inspection screen.

[0155] Figure 13This is a diagram showing the inspection screen displayed on the display unit 16.

[0156] The examination screen 117 is the screen displayed on the display device 13 and / or display unit 16 when the user performs an examination through the endoscope system 9. By checking the examination screen 117, the user can easily obtain the necessary information. The examination screen 117 consists of a live view display 119, subject information 115, and recorded images 113A to 113C. The live view display 119 displays a real-time dynamic image 38 captured by the endoscope observer 10. The user operates the endoscope observer 10 while observing the live view display 119. The subject information 115 indicates the name, age, date of birth, gender, and ID number of the subject undergoing the endoscopy. The recorded images 113A to 113C display still images 39 acquired by the user when the user instructs the still image capture instruction unit 32 to capture the dynamic image 38.

[0157] Illustration 101 is overlaid on examination screen 117. In the illustrated case, illustration 101 is overlaid on the lower right of examination screen 117. In illustration 101, the body part display 109D shows the last completed image and maintains the notification display. The subject body part (lower curvature of the stomach body viewed from above) corresponding to body part display 109F is displayed on live view display 119. When an appropriate medical image of the subject body part corresponding to body part display 109F is acquired, a notification is displayed on body part display 109F. In this way, by overlaying illustration 101 on examination screen 117, the user can effectively advance the image capture of the subject body part while observing live view display 119.

[0158] <Variation Example 5>

[0159] Next, a variation of this embodiment, example 5, will be described. In the examples described above, the application of the present invention to the examination of the stomach was explained; however, in this example, the present invention will be applied to the examination of the large intestine.

[0160] Figure 14 This is an illustration of the areas to be photographed during a colon examination.

[0161] exist Figure 14In the schematic diagram 121 of the large intestine, location displays 123A-123E and guide displays 125A-125E are shown to indicate the sites to be photographed. In this example, the sites to be photographed are designated as "rectum" (represented by symbol 125A-2) (corresponding location display 123A), "anus" (represented by symbol 125A-1) (corresponding location display 123A), "splenic flexure" (corresponding location display 123B), "hepatic flexure" (corresponding location display 123C), "duodenal inlet" (corresponding location display 123E), and "ileocecal junction" (corresponding location display 123D). Additionally, each site to be photographed is indicated by an arrow-shaped guide display 125A-125E. The user can identify the photographing direction of the endoscopic observer 10 using the arrow-shaped guide displays 125A-125E. Guide display 125A-2 indicates the photographing direction for the "rectum." Guide display 125A-1 indicates the photographing direction for the "anus." Guide display 125B indicates the imaging direction for the "splenic flexure". Guide display 125C indicates the imaging direction for the "hepatic flexure". Guide display 125D indicates the imaging direction for the "ileocecal junction". Guide display 125E indicates the imaging direction for the "duodenal inlet". This allows the target imaging sites to be set and the invention to be applied even within the large intestine. Furthermore, in this example, similar to the embodiment described above, when the second determination process determines that the determination criteria are met, a notification is displayed in site displays 123A to 123E, and this notification is maintained until imaging of all target imaging sites is completed.

[0162] <Variation Example 6>

[0163] Next, variation 6 will be described. In the above example, an example was described where a still image 39 was input as a medical image, and a first determination process and a second determination process were performed based on the still image 39. In this example, a frame image 38a constituting the dynamic image 38 was input as a medical image, and the first determination process and the second determination process were performed on the frame image 38a. Moreover, in this example, the frame image 38a that satisfies the determination criteria of the second determination process is stored as a recorded image in the memory 48.

[0164] Figure 15 This is a flowchart illustrating a medical image processing method using the medical image processing device 14.

[0165] First, the medical image acquisition unit 40 acquires frame image 38a (step S20). Then, the first determiner 42 performs a first determination process based on the acquired frame image 38a (step S21: first determination process). Specifically, the first determiner 42 determines whether the frame image 38a contains any of the multiple subject areas. When the first determiner 42 determines that the frame image 38a does not contain any subject area (No in step S21), the determination by the second determiner 43 is not performed.

[0166] On the other hand, when the first determiner 42 determines that the frame image 38a has any part of the subject area (when Yes is in step S21), the second determiner 43 performs a second determination process based on the frame image 38a that has undergone the first determination process (step S22: second determination process). Specifically, the second determiner 43 determines whether the determination criteria set for the part determined by the first determiner 42 are met. When the second determiner 43 determines that the determination criteria are not met (when No is in step S22), no notification is displayed on the display unit 16.

[0167] On the other hand, when the second determiner 43 determines that the frame image 38a meets the determination criteria, the display control unit 46 notifies the display unit 16 (step S23: display control process). Additionally, when the second determination process determines that the determination criteria are met, the processor stores the frame image 38a as a recorded image in the memory 48 (step S24: storage process). Furthermore, the frame images 38a are acquired continuously in a time sequence, and the above-described processing is performed sequentially on each acquired frame image 38a.

[0168] As explained above, in this example, a frame image 38a is input, and a first determination process and a second determination process are performed based on the input frame image 38a. Furthermore, in the second determination process, the frame image 38a that meets the determination criteria is stored as a still image in the memory 48. Thus, medical images that meet the determination criteria can be effectively acquired.

[0169] <Variation Example 7>

[0170] Next, a variation of this embodiment, example 7, will be described. In this example, the display method of the schematic diagram is changed according to the change of the light source.

[0171] Figure 16 This diagram illustrates two different light sources.

[0172] The display of diagrams 101 and 131 is switched depending on the switching of the light source. Alternatively, the display of diagrams 101 and 131 can also be switched according to the light source of the medical image (the light source used when capturing the medical image).

[0173] When the first light source is used, schematic diagram 101 (first display mode) is displayed on display unit 16. Here, the first light source is a white light source, which is a light source that emits broadband light. The white light source is the light source used in examinations using a conventional endoscope system 9. When a medical image of the second light source is acquired, schematic diagram 131 (second display mode) is displayed on display unit 16. Here, the second light source is a special light source, which is a light source that emits narrowband light. The special light source is the light source used when observing specific areas or specific lesions.

[0174] In the schematic diagram 101 shown when using the first light source, the description above is consistent with that (see...). Figure 8 Similarly, the six parts are displayed as 109A to 109F. When the first light source is used, the first determination process and the second determination process are performed on the subject parts corresponding to the parts displayed as 109A to 109F. In the illustrated case, a notification display is made in parts display as 109A and parts display as 109C.

[0175] When using the second light source, the target area is the "esophagogastric junction". Furthermore, in schematic diagram 131 showing the second light source, only the area corresponding to the "esophagogastric junction" (display 109A) is shown. Additionally, in schematic diagram 131, the stomach 105, which is not a designated target area, is displayed in a different color (e.g., gray) than other areas. Furthermore, in schematic diagram 131, areas not designated as target areas by the second light source (displays 109B to 109F) are hidden. Thus, schematic diagram 131 shows the range of the target area in the second light source (the range for the first and second determination processes).

[0176] In the above example, it was explained that five of the six subject areas were considered only when photographed by the first light source, while the remaining subject area ("esophagogastric junction") was considered when photographed by either the first or second light source. That is, this example illustrates a reduction in the number of subject areas when photographing medical images using a narrowband light source compared to a broadband light source. However, if an appropriate light source is determined among the subject areas, the subject areas can also be set according to the light source.

[0177] Next, the display changes of schematic diagrams 101 and 131, which are shown according to the types of light sources mentioned above, will be explained.

[0178] Figure 17 This is a diagram illustrating the first example of the displayed change.

[0179] In this example, Figure 16In the illustrated diagram 131, areas outside the target area are also shown as displays 109B to 109F. Specifically, in diagram 131, similar to diagram 101, areas 109A to 109F are shown. On the other hand, diagram 131 shows a situation where the stomach 105, which is an area where no target area is defined, is displayed in a different color (e.g., gray) than other areas, thus placing the stomach 105 outside the target area. Furthermore, in this example, by displaying all areas 109A to 109F, a notification display can be maintained when using the first light source. For example, the notification display in area display 109C in diagram 131 is to maintain a notification display when taking pictures using the first light source and acquiring medical images that meet the judgment criteria.

[0180] Figure 18 This is a diagram illustrating the second example of the change.

[0181] In this example, the range of the subject area is shown by displaying parts 109A to 109F. Specifically, in schematic diagram 131, part 109A is represented by a solid line, and parts 109B to 109F are represented by dashed lines. Thus, it can be shown that the area including part 109A (esophagus 103) is the range of the subject area, and it can be shown that the area including parts 109B to 109F (stomach 105) is outside the range of the subject area.

[0182] Figure 19 This is a diagram illustrating the third example of the displayed changes.

[0183] In this example, the range of the subject area is shown by changing the display of the range of the subject area according to the light source. Specifically, when using the first light source, the subject area is shown as the "esophagogastric junction", the "small J-shaped bend directly below the cardia", and the "large U-shaped bend directly below the cardia". Moreover, in schematic diagram 101, the colors of ranges 133(A) and other ranges 133(B) corresponding to the "esophagogastric junction", the "small J-shaped bend directly below the cardia", and the "large U-shaped bend directly below the cardia" are changed. In addition, when using the second light source, the subject area is shown as the "small J-shaped bend on the posterior wall of the small bend from the gastric angle or the lower part of the gastric body", the "from the anterior part of the pyloric ring to the pyloric ring", and the "large curvature viewed from the lower part of the gastric body". Furthermore, in schematic diagram 131, the colors of ranges 133(B) and other ranges 133(A) corresponding to the areas of "J-shaped bend in the posterior wall of the lesser curvature of the stomach angle or lower part of the stomach body", "from the anterior part of the pyloric ring to the pyloric ring", and "greater curvature of the lower part of the stomach body viewed from above" are changed. Thus, in schematic diagrams 101 and 131, the range of the subject area that changes according to the light source can be shown.

[0184] Figure 20 This is a diagram illustrating the fourth example of the displayed changes.

[0185] This example describes a case where medical images that meet the judgment criteria are acquired using both a first light source and a second light source. Specifically, in this example, images of the "esophagogastric junction," the site to be photographed, are captured using both the first and second light sources, and these images meet the judgment criteria in the second judgment process.

[0186] In schematic diagram 101, displayed when using the first light source, location display 109A provides a notification. That is, in the first light source, the acquisition of a medical image that meets the criteria for the "esophagogastric junction" (the imaging target location corresponding to location display 109A) is completed. Afterwards, when switching from the first light source to the second light source, schematic diagram 131 is displayed. Schematic diagram 131 is as follows... Figure 16 The area displayed 109A is within the range of the object being photographed, as explained. Furthermore, since the acquisition of a medical image satisfying the "esophagogastric junction" determination criterion using the second light source is not yet complete, no notification is displayed in the area display 109A shown in schematic diagram 131. Later, when the acquisition of a medical image satisfying the "esophagogastric junction" determination criterion using the second light source is completed, a notification is displayed in the area display 109A. Therefore, it is possible to effectively acquire medical images satisfying the "esophagogastric junction" determination criterion using both the first and second light sources.

[0187] <Other>

[0188] In the above embodiments, the hardware structure of the processing unit (e.g., medical image acquisition unit 40, first determiner 42, second determiner 43, display control unit 46, voice control unit 47) that performs various processes is as shown below. These processors include general-purpose processors that execute software (programs) and function as various processing units, such as CPUs (Central Processing Units); processors with customizable circuit structures, such as FPGAs (Field Programmable Gate Arrays), which are programmable logic devices (PLDs); and processors with circuit structures specifically designed for performing specific processes, such as ASICs (Application Specific Integrated Circuits), which are dedicated circuits.

[0189] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of CPU and FPGA). Alternatively, a single processor can constitute multiple processing units. Examples of a single processor constituting multiple processing units include, firstly, client or server computers, which use a combination of one or more CPUs and software to form a single processor that functions as multiple processing units. Secondly, there are systems-on-a-chip (SoCs), which use a single integrated circuit (IC) chip to implement the overall system functionality including multiple processing units. In this way, various processing units can be constructed using one or more of the aforementioned processors as hardware structures.

[0190] Furthermore, more specifically, the hardware structure of these various processors is a circuit composed of circuit elements such as semiconductor components.

[0191] The above-described structures and functions can be suitably implemented by any hardware, software, or a combination of both. For example, the present invention can also be applied to a program that causes a computer to perform the above-described processing steps (processing sequence), a computer-readable recording medium (non-transitory recording medium) containing such a program, or a computer on which such a program can be installed.

[0192] The examples of the present invention have been described above, but the present invention is not limited to the embodiments described above. Of course, various modifications can be made without departing from the spirit of the present invention.

[0193] Symbol Explanation

[0194] 9: Endoscopic System

[0195] 10: Endoscopic Observation Device

[0196] 11: Light source device

[0197] 12: Endoscopic processor device

[0198] 13: Display device

[0199] 14: Medical image processing device

[0200] 15: Operations Department

[0201] 16: Display Section

[0202] 17: Speaker

[0203] 20: Insertion section

[0204] 21: Hands-on Operations Department

[0205] 22: Universal Rope

[0206] 25: Soft parts

[0207] 26: Curved section

[0208] 27: Front end

[0209] 28: Camera components

[0210] 29: Bend the operating button

[0211] 30: Gas and water supply buttons

[0212] 31: Attraction Button

[0213] 32: Still Image Shooting Instruction Section

[0214] 33: Treatment device inlet

[0215] 35: Optical guide

[0216] 36: Signal cable

[0217] 37a: Connector

[0218] 37b: Connector

[0219] 38: Dynamic Images

[0220] 38a: Frame Image

[0221] 39: Still Image

[0222] 40: Medical Image Acquisition Unit

[0223] 41: CPU

[0224] 42: First Decision Maker

[0225] 43: Second Decision Maker

[0226] 46: Display Control Unit

[0227] 47: Voice Control Department

[0228] 48: Memory

Claims

1. A medical image processing device, comprising a processor, wherein, The processor performs: The first determination process involves determining, based on the acquired medical image, whether any one of the multiple photographed object parts is included in the medical image. The second determination process involves determining whether the medical image is an image that satisfies the determination criteria for the parts contained in the medical image when it is determined in the first determination process that any of the multiple photographed object parts is included in the medical image. as well as In the display control processing, when the second determination process determines that the medical image is an image that meets the determination criteria for the part contained in the medical image, the display unit displays a notification indicating that an image that meets the determination criteria for the part contained in the medical image has been captured. The second determination process determines the medical image based on the determination criteria established for each of the multiple photographed object parts.

2. The medical image processing device according to claim 1, wherein, In the display control process, the notification display is maintained until the notification display is shown for all of the plurality of subject parts.

3. The medical image processing apparatus according to claim 1 or 2, wherein, In the second determination process, a determination is made based on the determination criteria that are different for each of the plurality of photographed object parts.

4. The medical image processing apparatus according to claim 1 or 2, wherein, The second determination process is performed based on multiple indicators and based on the determination results among the multiple indicators.

5. The medical image processing device according to claim 4, wherein, The plurality of indicators include at least one of the following: blur jitter determination, brightness determination, boundary visibility determination, cardia visibility determination, cardia distance determination, peristalsis determination, wrinkle determination, processing determination, and composition determination.

6. The medical image processing apparatus according to claim 1 or 2, wherein, In the second determination process, a determination is made by using a different determiner for each of the plurality of subject parts.

7. The medical image processing apparatus according to claim 1 or 2, wherein, The first determination process is performed based on the determination result of a first determiner composed of a convolutional neural network. At least a portion of the second determination process is performed by inputting the intermediate feature quantity obtained in the first determiner into the second determiner, and the second determiner outputs the determination result.

8. The medical image processing apparatus according to claim 1 or 2, wherein, The medical images are captured using a first light source or a second light source. In the first determination process, if the medical image is captured using the first light source, the plurality of subject areas are determined; if the medical image is captured using the second light source, the selected subject area among the plurality of subject areas is determined.

9. The medical image processing apparatus according to claim 1 or 2, wherein, The medical images are endoscopic images taken by an endoscopic observer. In the display control process, the display unit displays a schematic diagram of the lumen organ taken by the endoscopic observer, and the notification display is displayed on the schematic diagram.

10. The medical image processing apparatus according to claim 9, wherein, In the display control process, on the schematic diagram, the parts of the plurality of photographed objects are displayed at corresponding positions.

11. The medical image processing apparatus according to claim 10, wherein, In the display control process, when the second determination process determines that the medical image is an image that meets the determination criteria for the part contained in the medical image, the notification is displayed by changing the display mode of the part.

12. The medical image processing apparatus according to claim 9, wherein, In the display control process, the schematic diagram is made to display a guide for the endoscope to capture the medical images.

13. The medical image processing apparatus according to claim 12, wherein, The guide is displayed in the shape of a bar, the direction of which indicates the shooting direction of the endoscope, and the length of which indicates the distance of the endoscope from the subject.

14. The medical image processing apparatus according to claim 9, wherein, The medical images are captured using a first light source or a second light source. In the display control process, when the medical image captured using the first light source is acquired, the schematic diagram is displayed in a first display mode; when the medical image captured using the second light source is acquired, the schematic diagram is displayed in a second display mode.

15. The medical image processing apparatus according to claim 1 or 2, wherein, In the display control process, when the second determination process determines that all of the plurality of photographed object parts are images that satisfy the determination criteria of the parts included in the medical image, the display unit displays information indicating that all photographed object parts have been photographed.

16. The medical image processing apparatus according to claim 1 or 2, wherein, When the second determination process determines that the medical image is an image that meets the determination criteria for the part contained in the medical image, the processor performs a storage process to store the medical image in the memory.

17. The medical image processing apparatus according to claim 1 or 2, wherein, The multiple camera locations are selected from at least the esophagogastric junction, the small J-shaped bend directly below the cardia, the large U-shaped bend directly below the cardia, the small J-shaped bend on the posterior wall of the gastric angle or lower part of the gastric body, and the large bend viewed from the anterior part of the pyloric ring to the pyloric ring and lower part of the gastric body.

18. The medical image processing apparatus according to claim 1 or 2, wherein, The multiple camera sites are selected from at least the rectum, anus, splenic flexure, hepatic flexure, duodenal inlet, and ileocecal junction.

19. A medical image processing method, wherein the medical image processing device includes a processor, wherein, Performed via the processor: The first determination step involves determining, based on the acquired medical image, whether any one of the multiple photographed object parts is included in the medical image. In the second determination process, when it is determined in the first determination process that any of the multiple photographed object parts is included in the medical image, it is determined whether the medical image is an image that meets the determination criteria for the parts contained in the medical image. as well as In the display control process, when it is determined in the second determination process that the medical image is an image that meets the determination criteria for the part contained in the medical image, the display unit displays a notification indicating that an image that meets the determination criteria for the part contained in the medical image has been captured. In the second determination process, the medical image is determined based on the determination criteria established for each of the multiple photographed object parts.

20. A recording medium that is non-transitory and computer-readable, wherein, The document contains a program that causes a computer to execute the medical image processing method of claim 19.

Images