Image processing device, its operating method, and endoscope system

Abstract

To provide an image processing device capable of outputting information for observation support, in accordance with an observation object included in a medical image, an operation method of the device, and an endoscope system.SOLUTION: A processor provided in an image processing device performs control of acquiring a medical image, inputting the medical image into an observation object identification algorithm to output observation object identification information indicating whether being an observation support object portion or not, selecting one specific observation support algorithm from a plurality of observation support algorithms based on the observation object identification information, inputting the medical image into the specific observation support algorithm to output observation support information, and reporting the observation support information.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an image processing device that provides observation support, an operation method thereof, and an endoscope system. [Background technology]

[0002] Patent Document 1 discloses an endoscopic image processing system that displays a guide for observation points, and states that "when the image area B10 specified by the area specification data acquired by the AI ​​processing module 22 corresponds to the observation point, the output processing module 24 displays a guide as follows. That is, the output processing module 24 superimposes aiming displays F1, F2, and F3 on the endoscopic image, and when the image area B10 reaches a predetermined size within the frame of the aiming display F2 displayed at a predetermined position, the endoscopic image is converted into a still image G4, and the still image G4 is rotated and reduced while being framed out."

[0003] Furthermore, Patent Document 2 discloses an operation assistance device that generates and displays operation assistance information indicating the next action to be performed, and states that "the operation assistance device 20 can display operation assistance information by placing an indicator on the endoscopic image." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-48927 [Patent Document 2] Patent Publication No. 2021-61911 Summary of the Invention [Problem to be solved by the invention]

[0005] In endoscopic examinations, the structure of the object of observation shown in medical images can vary greatly depending on the observation site. For this reason, when performing endoscopic examinations across multiple sites with significantly different internal structures, if there is only one type of trained model that outputs operation support information such as position information for the tip of the endoscope and the route to take in response to input medical images, it may not be sufficient to output information to support observation appropriate for each observation site.

[0006] Furthermore, in endoscopic examinations, there are situations in which medical images unsuitable for observation are acquired, and it is desirable to notify the user, who is the surgeon, not only in situations in which medical images are acquired during normal observation, but also in situations in which such medical images unsuitable for observation are acquired.

[0007] An object of the present invention is to provide an image processing device, an operating method thereof, and an endoscope system that can output information for assisting observation according to an observation target included in a medical image. [Means for solving the problem]

[0008] The image processing device of the present invention includes a processor that acquires a medical image and inputs the medical image into an observation object identification algorithm, thereby outputting observation object identification information indicating that a region included in the medical image that requires observation assistance or that the medical image is not subject to observation assistance, selects one specific observation assistance algorithm from a plurality of observation assistance algorithms based on the observation object identification information, and inputs the medical image into the specific observation assistance algorithm, thereby outputting the observation assistance information and performing control to notify the observation assistance information.

[0009] The region to be subjected to observation support is preferably the nasal cavity, nasopharynx, oropharynx, hypopharynx, larynx, trachea, or esophagus.

[0010] The observation support information is preferably notified by a guide image that displays the observation support information and / or a sound that notifies the observation support information.

[0011] The observation support information is preferably endoscope position determination information indicating whether the position of the endoscope is appropriate or inappropriate, endoscope operation information indicating how to operate the endoscope, subject position information prompting a change or confirmation of the subject's position, and / or observation support stop information indicating that observation support is to be stopped.

[0012] The endoscope operation information is the movement direction and / or movement amount of the tip of the endoscope, and the movement direction is preferably rightward, leftward, upward, downward, backward, forward, rightward rotation, or leftward rotation.

[0013] The processor preferably controls the display of an endoscope operation support diagram on the guide image as the endoscope operation information.

[0014] The processor preferably controls switching of the display of the endoscope operation information based on the endoscope position determination information.

[0015] Preferably, the specific observation support algorithm includes an operation support algorithm, and the processor outputs endoscope operation information or observation support stop information by inputting a medical image to the operation support algorithm.

[0016] The processor preferably inputs the most recent medical images into the operation assistance algorithm.

[0017] The operation assistance algorithm is preferably a trained model that outputs endoscope operation information and / or subject position information.

[0018] The specific observation support algorithm preferably includes a position determination algorithm and an operation support algorithm, and the position determination algorithm is a trained model that outputs endoscope position determination information when a medical image is input, and the operation support algorithm is a trained model that outputs endoscope operation information and / or subject position information when a medical image is input.

[0019] The processor preferably inputs the most recent medical images into the position determination algorithm and the navigation assistance algorithm, respectively.

[0020] The observation object identification algorithm is preferably a trained model trained using training images including medical images in which the observation object includes the nasal cavity, nasopharynx, oropharynx, hypopharynx, larynx, trachea, or esophagus.

[0021] The processor preferably controls switching between the presence and absence of notification of observation support information.

[0022] When the observation support target area output by the observation target identification algorithm is the nasal cavity, the processor preferably inputs a medical image into the operation support algorithm, outputs insertion area information indicating the upper route insertion area and / or lower route insertion area contained in the medical image, which are areas suitable for inserting an endoscope, calculates insertion route information, which is coordinate information for the area, width and / or center position of the upper route insertion area and / or lower route insertion area, based on the insertion area information, and controls the display of the insertion route information on the guide image as observation support information.

[0023] It is preferable that the processor performs control to display the upper route insertion area or the lower route insertion area on the guide image in a different display mode based on the insertion route information.

[0024] When the observation support target area output by the observation target identification algorithm is the nasal cavity, the processor preferably inputs a medical image into the operation support algorithm, outputs insertion area information indicating the upper route insertion area and / or lower route insertion area contained in the medical image, which are areas suitable for inserting an endoscope, calculates insertion route information, which is coordinate information for the area, width and / or center position of the upper route insertion area and / or lower route insertion area, based on the insertion area information, and outputs endoscope operation information using the insertion route information.

[0025] When the observation support target area output by the observation target identification algorithm is the nasopharynx, the processor preferably inputs a medical image into the operation support algorithm to output nasopharynx position information indicating that the area is in an appropriate position or in an inappropriate direction position such as an inappropriate right position, an inappropriate left position, an inappropriate upper position, an inappropriate lower position, a deep inappropriate position, or a deep inappropriate position, and outputs endoscope operation information based on the nasopharynx position information.

[0026] The operation assistance algorithm is preferably a trained model trained using training images including medical images associated with nasopharynx position information.

[0027] When the observation support target area output by the observation target identification algorithm is the oropharynx and the position determination algorithm to which a medical image is input outputs that the endoscope position determination information is inappropriate, the processor preferably inputs the medical image to the operation assistance algorithm, outputs oropharyngeal region information indicating the glottis region and / or epiglottis region contained in the medical image, calculates oropharyngeal region calculation information, which is coordinate information for the area, width and / or center position of the glottis region and epiglottis region, based on the oropharyngeal region information, and outputs endoscope operation information using the oropharyngeal region calculation information.

[0028] The position determination algorithm is preferably a trained model that is trained using training images in which medical images and endoscope position determination information are associated with each other.

[0029] When the observation support target area output by the observation target identification algorithm is the hypopharynx and the position determination algorithm to which a medical image is input outputs that the endoscope position determination information is inappropriate, the processor preferably inputs the medical image into the operation assistance algorithm, outputs hypopharynx area information indicating the glottis area and / or vocal fold area contained in the medical image, calculates hypopharynx area calculation information, which is the area, width and / or coordinate information of the center position of the glottis area and the length of the vocal fold area, based on the hypopharynx area information, and outputs endoscope operation information using the hypopharynx area calculation information.

[0030] The position determination algorithm is a trained model that is trained using training images in which medical images and endoscope position determination information are associated with each other. child It is preferable that:

[0031] When the observation support target area output by the observation target identification algorithm is the esophagus or trachea, the processor preferably inputs a medical image into the operation support algorithm and outputs endoscope operation information instructing the withdrawal of the endoscope.

[0032] When the observation target identification algorithm outputs "not subject to observation support" as the observation target identification information, the processor preferably outputs observation support stop information by inputting the medical image into the operation support algorithm.

[0033] It is preferable that the observation object identification algorithm is a trained model that is trained to output the observation object identification information as not being subject to observation support when a medical image containing foreign objects such as food or saliva, blur, blurring, or halation is input.

[0034] The method of operating the image processing device of the present invention includes the steps of acquiring a medical image, inputting the medical image into an observation object identification algorithm to output observation object identification information indicating that a region included in the medical image that is subject to observation assistance or that the medical image is not subject to observation assistance, selecting one specific observation assistance algorithm from a plurality of observation assistance algorithms based on the observation object identification information, inputting the medical image into the specific observation assistance algorithm to output the observation assistance information, and controlling the notification of the observation assistance information.

[0035] An endoscope system of the present invention includes the image processing device described above, a light source device that emits illumination light, and an endoscope that captures medical images. [Effects of the Invention]

[0036] According to the present invention, it is possible to output information for assisting observation according to an observation target included in a medical image. [Brief explanation of the drawings]

[0037] [Figure 1] FIG. 2 is a block diagram showing functions of the image processing device. [Figure 2] FIG. 1 is a schematic diagram of an endoscope system. [Figure 3] FIG. 2 is a block diagram showing functions of an observation support information generation unit. [Figure 4] FIG. 10 is an explanatory diagram showing the output of observation support information using the latest medical image as input. [Figure 5] FIG. 10 is an image diagram showing an example of a guide image indicating "appropriate" endoscope position determination information when an oropharyngeal image is used as input. [Figure 6] FIG. 10 is an image diagram showing an example of a guide image indicating "inappropriate" endoscope position determination information when an oropharyngeal image is used as input. [Figure 7] 10 is an image diagram showing an example of a guide image indicating observation support stop information. FIG. [Figure 8] FIG. 1 is an explanatory diagram showing swallowing. [Figure 9] FIG. 10 is an explanatory diagram showing an appropriate observation position. [Figure 10] FIG. 10 is an explanatory diagram showing a case where an endoscope is erroneously inserted into the esophagus. [Figure 11] FIG. 10 is an explanatory diagram showing a case where an endoscope is erroneously inserted into the trachea. [Figure 12] FIG. 2 is an explanatory diagram showing the insertion route of the endoscope as seen from the side of the subject. [Figure 13] FIG. 2 is an explanatory diagram showing an insertion route of an endoscope as viewed from the front of a subject. [Figure 14] FIG. 10 is an image diagram showing an example of an upper route image. [Figure 15] FIG. 10 is an image diagram showing an example of a lower route image. [Figure 16] FIG. 10 is an image diagram showing an example of a front nose image. [Figure 17] FIG. 10 is an image diagram showing an example of an upper route insertion area. [Figure 18] 10 is an image diagram showing an example of a guide image that displays an insertion route guide mark and an image center guide mark when an upper route image is input. FIG. [Figure 19] 10 is an image diagram showing an example of a guide image in which a guide arrow is used to instruct an operation in the upward direction when an upper route image is input. FIG. [Figure 20] 10 is an image diagram showing an example of a guide image that uses an icon to instruct an operation in the upward direction when an upper route image is input. FIG. [Figure 21] 10 is an image diagram showing an example of a guide image that uses an icon to instruct an operation in the lower left direction when the upper route image is used as an input. FIG. [Figure 22] 10 is an image diagram showing an example of a guide image that uses an endoscope operation support diagram to instruct an operation in the lower left direction when an upper route image is input. FIG. [Figure 23] 10 is an image diagram showing an example of a guide image that uses an endoscope operation support diagram to instruct left rotation when an upper route image is input. FIG. [Figure 24] FIG. 10 is an image diagram showing an example of a lower route insertion area. [Figure 25] 10 is an image diagram showing an example in which an upper route insertion area and a lower route insertion area are detected. FIG. [Figure 26] 10 is an image diagram showing an example of a guide image displaying an insertion route guide mark and an image center guide mark when an upper route insertion area and a lower route insertion area are detected. FIG. [Figure 27] 10 is an image diagram showing an example of a guide image that gives operation instructions when an upper route insertion area and a lower route insertion area are detected. FIG. [Figure 28] FIG. 2 is an explanatory diagram showing the height of the nasopharynx, oropharynx, and hypopharynx. [Figure 29] FIG. 10 is an image diagram showing an example of a nasopharynx image at an appropriate observation position. [Figure 30] FIG. 10 is an image diagram showing an example of a guide image when an image of the nasopharynx at an appropriate observation position is used as input. [Figure 31] FIG. 10 is an image diagram showing an example of a nasopharynx image at an inappropriate observation position. [Figure 32] FIG. 10 is an image diagram showing an example of a guide image that uses an icon to instruct an operation in the forward direction when a nasopharynx image is used as input. [Figure 33] FIG. 10 is an image diagram showing an example of a guide image that uses an endoscopic operation support diagram to instruct an operation in the forward direction when a nasopharynx image is used as input. [Figure 34] FIG. 1 is an image diagram showing an example of an oropharynx image in an appropriate viewing position. [Figure 35] FIG. 10 is an image diagram showing an example of an oropharynx image in an inappropriate observation position. [Figure 36] FIG. 10 is an image diagram showing an example of a guide image when an oropharyngeal image at an appropriate observation position is used as input. [Figure 37] FIG. 10 is an image diagram showing an example of an oropharyngeal image in which the glottis region and the epiglottis region are detected. [Figure 38] FIG. 10 is an image diagram showing an example of a guide image that uses an icon to instruct an operation in the forward direction when an oropharyngeal image is used as input. [Figure 39] 10 is an image diagram showing an example of a guide image indicating subject position information when an oropharyngeal image is used as input. FIG. [Figure 40] FIG. 10 is an image diagram showing an example of a hypopharyngeal image at an appropriate observation position. [Figure 41] FIG. 10 is an image diagram showing an example of a hypopharyngeal image at an inappropriate observation position. [Figure 42] 10 is an image diagram showing an example of a guide image when a hypopharyngeal image at an appropriate observation position is used as input. FIG. [Figure 43] FIG. 10 is an image diagram showing an example of a hypopharynx image in which the glottis region and the vocal fold region are detected. [Figure 44] FIG. 10 is an image diagram showing an example of a guide image that uses icons to instruct operations in the forward direction when a hypopharyngeal image is used as input. [Figure 45] FIG. 10 is an image diagram showing an example of an esophageal image. [Figure 46] FIG. 10 is an image diagram showing an example of a tracheal image. [Figure 47] FIG. 10 is an image diagram showing an example of a guide image that uses icons to instruct operations in the forward direction when an esophageal image is used as an input. [Figure 48] 10A and 10B are image diagrams showing examples of guide images that display a warning. [Figure 49] 1 is a flowchart illustrating a method of operating the image processing device. DETAILED DESCRIPTION OF THE INVENTION

[0038] 1, the image processing device 10 receives medical images from a modality 20 that captures and transmits the medical images to the image processing device 10 or from a database 11 in which the medical images are stored. The image processing device 10, the modality 20, and the database 11 are connected to each other so that they can communicate with each other via a wired connection or wirelessly via a network. The network may be the Internet, a LAN (Local Area Network), or the like.

[0039] The image processing device 10 is connected to a user interface 12 via a wired or wireless connection. The user interface 12 is an input device, such as a keyboard, mouse, microphone, foot switch, touchpad, tablet, or touch pen, that accepts input operations for setting functions from a user. The user interface 12 is also an output device, such as a display, head-mounted display, or speaker, that receives notification instructions from the image processing device 10 and notifies the user. The user interface 12 may be connected to both the image processing device 10 and a processor device 23, which will be described later. In other words, the user interface 12 is an input / output device that has a function of accepting input instructions to the image processing device 10 or the processor device 23 and a function of issuing output instructions from the image processing device 10 or the processor device 23.

[0040] 1, the image processing device 10 includes an image acquisition unit 30, an observation target identification unit 40, an observation support algorithm selection unit 50, an observation support information generation unit 60, and a notification control unit 70. The image processing device 10 receives medical images, performs image processing on the medical images in almost real time, outputs observation support information, and transmits it to a user interface 12. As will be described in detail later, the image processing device 10 identifies the region of the observation target included in the medical image, identifies that the medical image is not subject to observation support and outputs observation target identification information, selects an observation support algorithm according to the observation target identification information, inputs the medical image into the observation support algorithm, outputs observation support information, and performs control to notify users such as doctors of the observation support information via the user interface 12.

[0041] The image processing device 10 is a computer equipped with a processor. A control unit (not shown) configured by the processor runs programs relating to various processes or controls that are stored in a program storage memory (not shown) provided in the image processing device 10, thereby realizing the functions of an image acquisition unit 30, an observation target identification unit 40, an observation support algorithm selection unit 50, an observation support information generation unit 60, and a notification control unit 70 in the image processing device 10.

[0042] In this embodiment, the image processing device 10 uses an endoscope system 21 as the modality 20 and is suitable for use in image processing of medical images, which are endoscopic images. The endoscope system 21 obtains endoscopic images by having a user insert an endoscope into the living body of a subject and capture an image of the object of observation that becomes observable when illuminated with illumination light. The captured endoscopic image is transmitted to the image processing device 10. The user is a doctor who operates the endoscope, and is also called an operator.

[0043] 2, the endoscope system 21 includes an endoscope 22, a processor device 23, and a light source device 24. The endoscope 22 is optically connected to the light source device 24 and electrically connected to the processor device 23. These connections are not limited to being wired, and may be wireless. Note that the endoscope system 21 may also include an image processing device 10 in addition to the endoscope 22, the processor device 23, and the light source device 24.

[0044] The endoscope 22 has an insertion section 22a that is inserted into the body of a subject, an operation section 22b provided at the base end of the insertion section 22a, a bending section 22c provided at the tip side of the insertion section 22a, and a tip section 22d. The bending section 22c is bent by operating an angle knob 22e of the operation section 22b. The tip section 22d is directed in a desired direction by the bending operation of the bending section 22c.

[0045] The tip portion 22d irradiates the observation object with illumination light generated from the light source device 24 and receives reflected light from the observation object to photograph the observation object. The operation portion 22b is provided with an angle knob 22e and a still image capture command switch 22f. The still image capture command switch 22f is used to command the capture of a still image of the observation object. The endoscope 22 may be a flexible video endoscope equipped with an imaging sensor at the tip portion 22d, an endoscope equipped with a zoom mechanism, an endoscope in which an imaging sensor is connected to a fiberscope with a narrow diameter in the insertion portion 22a, or an endoscope in which an external imaging sensor is connected to a rigid endoscope.

[0046] Routes for inserting an endoscope into a subject include transnasal insertion, which involves insertion through the nose, nasal cavity, and nasopharynx into the oropharynx, hypopharynx, esophagus, stomach, and duodenum; oral insertion, which involves insertion through the mouth, oral cavity, and into the oropharynx, hypopharynx, esophagus, stomach, duodenum, and small intestine; and lower insertion, which involves insertion through the anus into the rectum, colon, and small intestine. The image processing device 10 of this embodiment is particularly suitable when the endoscope 22 included in the endoscope system 21 is a flexible endoscope that is inserted transnasally into the subject. Note that, in this specification, the term "subject" refers to a subject into which the endoscope 22 is inserted. The term "observation target" refers to a subject that is included in the field of view of the endoscope 22 and appears in a medical image.

[0047] Note that a still image acquisition instruction may be issued using an operating device other than the still image acquisition instruction switch 22f. For example, a foot switch, a foot pedal, a touch panel, a gesture input device, an eye-gaze input device, or the like may be connected to the processor device 23 as the user interface 12, and a still image acquisition instruction may be issued when a specific operation is performed by the user on the user interface 12. Also, a still image acquisition instruction may be issued when a specific voice input to the processor device 23 via a microphone as the user interface 12 is recognized.

[0048] The processor device 23 controls the turning on and off of the illumination light for the light source device 24, controls the operation of the endoscope 22, controls the shooting by the endoscope 22, controls image processing such as correction processing and enhancement processing for the endoscopic images shot by the endoscope 22, and controls the transmission of the endoscopic images to the image processing device 10.

[0049] The light source device 24 includes a light source section and a light source control section. The light source section emits illumination light. The light source control section controls the operation of the light source section. The light source section is a light source such as a laser diode, an LED (Light Emitting Diode), a xenon lamp, or a halogen lamp, and emits illumination light to illuminate the observation object. The light source section may be built into the endoscope 22. The light source control section controls the on / off and light emission amount of each light source that constitutes the light source section.

[0050] An illumination optical system and an imaging optical system are provided at the tip 22d of the endoscope 22. Illumination light emitted by the light source is emitted from the tip 22d through an illumination lens of the illumination optical system. The imaging optical system has an objective lens and an imaging sensor. Reflected light from the observation object illuminated with the illumination light is incident on the imaging sensor via the objective lens. As a result, an image of the observation object is formed on the imaging sensor, and an image signal is output from the imaging sensor. The imaging sensor may be provided at the tip 22d, may be connected to the endoscope 22, which is a fiberscope, or may be externally attached to the endoscope 22, which is a rigid endoscope.

[0051] The imaging sensor is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge-Coupled Device) sensor, or the like. The processor device 23 generates a medical image based on the image signal output by the imaging sensor. The medical image generated by the processor device 23 is output to the image acquisition unit 30 of the image processing device 10. The medical images transmitted from the endoscope system 21 to the image processing device 10 are a series of time-series continuous moving images captured during an endoscopic examination. Note that the medical images may be transmitted from the endoscope system 21 to the database 11 for storage, and then transmitted from the database 11 to the image processing device 10.

[0052] The database 11 is a storage, a file server, a cloud storage, etc. that stores medical images such as endoscopic images. The database 11 may be part of a system that directly or indirectly cooperates with the image processing device 10, such as a hospital information system (so-called HIS (Hospital Information Systems)) or a PACS (Picture Archiving and Communication Systems).

[0053] The following describes the flow in which the image processing device 10 performs image processing on a medical image, outputs observation support information, and notifies the user via the user interface 12. The image acquisition unit 30 acquires medical images by receiving them from the endoscope system 21, which is the modality 20, or the database 11. The image acquisition unit 30 transmits the medical images to the observation target identification unit 40.

[0054] The observation target identification unit 40 inputs the medical image into an observation target identification algorithm provided in the observation target identification unit 40, and outputs observation target identification information. The observation target identification algorithm identifies an observation support target portion, which is a portion of the observation target included in the medical image, and outputs observation target identification information indicating the observation support target portion. Furthermore, when the observation target identification unit 40 identifies that the medical image is not subject to observation support, it outputs observation target identification information of "not subject to observation support." Note that, in this specification calligraphy In this case, outputting various information includes generating various information.

[0055] The observation assistance target region is a portion of the anatomical region of the subject's body that is the target of observation. Specific examples of the observation assistance target region include the nasal cavity, nasopharynx, oropharynx, hypopharynx, larynx, trachea, main bronchi, lobar bronchi, segmental bronchi, subsegmental bronchi, oral cavity, esophagus, stomach, duodenum, jejunum, ileum, cecum, ascending colon, transverse colon, descending colon, sigmoid colon, and rectum. The nasopharynx is also called the nasopharynx. When the observation target identification algorithm identifies an observation target included in a medical image as one of the observation assistance target regions, it outputs the observation assistance target region as observation target identification information.

[0056] In this embodiment, it is preferable that the nasal cavity, nasopharynx, oropharynx, hypopharynx, larynx, trachea, or esophagus be output as the observation assistance target region. In this case, the observation target identification algorithm outputs "not subject to observation assistance" for input medical images in which a region other than these regions is the observation target. In this way, when the observation target regions are limited to certain regions, if the observation target identification algorithm identifies the nasopharynx as the observation target, it outputs "nasopharynx" as the observation target identification information, and if it identifies the descending colon as the observation target in the medical image, it outputs "not subject to observation assistance" as the observation target identification information.

[0057] The larynx is included in medical images in which the observation object is the oropharynx or hypopharynx (see Figures 34 and 40, etc., described later). When the observation object identification algorithm identifies the "larynx," it may be configured to output "oropharynx" or "hypopharynx." Furthermore, when the observation object identification algorithm identifies the trachea, main bronchi, lobar bronchi, segmental bronchi, or subsegmental bronchi, it may be configured to output "trachea." Furthermore, as described later, the regions output to the observation object identification unit 40 as the observation support target regions, namely, the nasal cavity, nasopharynx, oropharynx, hypopharynx, larynx, trachea, or esophagus, may be further narrowed to a more limited range of these regions.

[0058] When the observation object identification algorithm identifies an observation object contained in a medical image as not a region requiring observation assistance, it outputs "not suitable for observation assistance" as the observation object identification information. Medical images for which the observation object identification algorithm outputs "not suitable for observation assistance" include medical images in which the observation object is a region other than a region requiring observation assistance, as well as medical images in which the observation object contains foreign matter such as food or saliva, medical images in which the observation object is blurred, medical images in which the observation object is out of focus, and medical images in which the observation object contains halation. The observation object identification algorithm outputs "not suitable for observation assistance" for input of such medical images that are not suitable for observation. Note that food as a foreign body refers to residual food or swallowed food for examination. Saliva as a foreign body refers to saliva accumulated in anatomical regions such as the pyriform sinus.

[0059] The observed object identification algorithm and the position determination algorithm and operation support algorithm described later may be machine learning algorithms or algorithms for identifying medical images by pattern matching. Machine learning algorithms include decision trees, support vector machines, random forests, regression analysis, supervised learning, semi-unsupervised learning, unsupervised learning, reinforcement learning, deep learning, deep reinforcement learning, neural networks, convolutional neural networks, and generative adversarial networks.

[0060] This embodiment is suitable when the observation target identification algorithm as machine learning possessed by the image processing device 10, and the position determination algorithm and operation assistance algorithm described later are convolutional neural networks. Also, a part of the position determination algorithm and operation assistance algorithm described later may be a convolutional neural network. In this specification, when the observation target identification algorithm and the observation assistance algorithm described later are machine learning, the observation target identification algorithm or observation assistance algorithm before learning is called a learning model, and the observation target identification algorithm or observation assistance algorithm after learning is called a trained model.

[0061] Image data used to train a learning model is called training images. Training images are also called teacher data. Training images include correct answers such as correct answers for the area to be observed with observation assistance, correct answers for the observation target in an "appropriate" observation position (described later), and correct answers for the classification of specific areas included in the observation target.

[0062] The observation object identification algorithm is preferably a trained model trained using training images including medical images in which the observation object includes the nasal cavity, nasopharynx, oropharynx, hypopharynx, larynx, trachea, or esophagus. For example, a convolutional neural network such as VGGNet (e.g., VGG16 or VGG19) or ResNet is suitable.

[0063] When supervised learning or semi-supervised learning is applied as the observation object identification algorithm, the correct image is a medical image depicting the nasal cavity, nasopharynx, oropharynx, or hypopharynx that a doctor has determined to be suitable for observation. Correct images that are not targets for observation assistance include images taken outside the subject's body, images in which a doctor has determined that the observation object contains foreign matter such as food or saliva, images in which a doctor or an existing device has performed a blur / blur determination and determined that the observation object contains blur or blur, images that are too close to the observation object and therefore do not capture the characteristic structures necessary for identifying the observation assistance object, images in which the observation object is too dark to be distinguished, and images in which the observation object contains a lot of halation. Halation refers to areas in the image that are washed out due to strong light, such as reflected light from the observation object, entering the image sensor, resulting in a blown-out image that is unsuitable for observation. Furthermore, a medical image determined by a doctor or an existing device to be in the middle of a specific action, such as swallowing or coughing, may also be considered a correct image that is not targets for observation assistance. Furthermore, medical images in which regions other than the nasal cavity, nasopharynx, oropharynx, hypopharynx, larynx, trachea, or esophagus are to be observed may be included in the correct images that are not the subject of observation support.

[0064] The observation support algorithm selection unit 50 selects one specific observation support algorithm from multiple observation support algorithms based on the observation target identification information. The observation support algorithm selection unit 50 transmits specific observation support algorithm selection information, which is information indicating which specific observation support algorithm has been selected, to the observation support information generation unit 60. The observation support algorithm is an algorithm that outputs observation support information, which will be described later, when the observation support information generation unit 60 inputs a medical image.

[0065] The observation support algorithms include a nasal cavity observation support algorithm, a nasopharynx observation support algorithm, a oropharynx observation support algorithm, a hypopharynx observation support algorithm, an incorrect insertion observation support algorithm, and an observation support algorithm for objects not subject to observation support. The observation support algorithm selection unit 50 selects one of these observation support algorithms as the specific observation support algorithm.

[0066] It is preferable that which observation support algorithm the observation support algorithm selection unit 50 selects as the specific observation support algorithm based on the observation target identification information is set in advance. Specifically, observation target identification information indicating a region that requires observation support or that is not subject to observation support is associated with an observation support algorithm and stored, and the observation support algorithm selection unit 50 selects the specific observation support algorithm based on this association. For example, if the region that requires observation support as observation target identification information is the "nasal cavity," the observation support algorithm selection unit 50 selects the nasal cavity observation support algorithm as the specific observation support algorithm and transmits information instructing the use of the nasal cavity observation support algorithm to the observation support information generation unit 60 as specific observation support algorithm selection information.

[0067] Furthermore, when observation object identification information of "not subject to observation support" is output, the observation support algorithm selection unit 50 selects an observation support algorithm not subject to observation support as the specific observation support algorithm, and transmits information instructing the use of the observation support algorithm not subject to observation support to the observation support information generation unit 60 as specific observation support algorithm selection information.

[0068] The observation support algorithm is included in the observation support information generation unit 60. As shown in Fig. 3, the observation support information generation unit 60 includes a nasal cavity observation support information generation unit 60a, a nasopharynx observation support information generation unit 60b, a oropharynx observation support information generation unit 60c, a hypopharynx observation support information generation unit 60d, an incorrect insertion observation support information generation unit 60e, and an observation support information generation unit for non-observation support targets 60f.

[0069] The nasal cavity observation support information generation unit 60a has a nasal cavity observation support algorithm. The nasopharynx observation support information generation unit 60b has a nasopharynx observation support algorithm. The oropharynx observation support information generation unit 60c has an oropharynx observation support algorithm. The hypopharynx observation support information generation unit 60d has a hypopharynx observation support algorithm. The incorrect insertion observation support information generation unit 60e has an incorrect insertion observation support algorithm. The non-observation support target observation support information generation unit 60f has a non-observation support target observation support algorithm.

[0070] If the region requiring observation support includes the digestive tract, the observation support information generating unit 60 may be provided with a stomach observation support information generating unit or the like, and may be provided with a stomach observation support algorithm as the observation support algorithm. In this case, the observation support algorithm selecting unit 50 can select the stomach observation support algorithm as the specific observation support algorithm.

[0071] The observation support information generator 60 inputs the medical image into the observation support algorithm selected as the specific observation support algorithm and outputs observation support information. The observation support information includes information for instructing the user (the surgeon) to operate the modality 20, information indicating that observation support has been stopped, information for giving instructions to the subject, etc.

[0072] Specifically, this information includes endoscope position determination information indicating whether the position of the tip 22d of the endoscope 22 is appropriate or inappropriate, endoscope operation information indicating how to operate the endoscope 22, subject position information prompting the subject to change or confirm their position, and observation support stop information indicating the stop of observation support. The observation support information includes insertion region information, insertion route information, nasopharynx position information, oropharynx region information, oropharynx region calculation information, hypopharyngeal region information, and hypopharyngeal region calculation information, which will be described later. The details of each piece of observation support information will be described later. The observation support information generation unit 60 transmits the output observation support information to the notification control unit 70.

[0073] The observation support information generation unit 60 may transmit the same medical image as the medical image transmitted to the observation target identification unit 40, or may transmit a medical image different from the medical image input to the observation target identification unit 40. When transmitting a medical image different from the medical image transmitted to the observation target identification unit 40 to the observation support information generation unit 60, the most recent medical image is input to the observation support algorithm. This case will be explained using FIG. 4.

[0074] The image acquisition unit 30 acquires medical image 31 at time T(-1), medical image 32 at time T(0), and medical image 33 at time T(1) in chronological order in the direction of the arrow indicating time in Figure 4. The medical image 31 is the latest medical image at time T(-1), the medical image 32 at time T(0), and the medical image 33 at time T(1).

[0075] At time T(0), a medical image 32 is transmitted to the observation object identification unit 40 and input to the observation object identification algorithm. Observation object identification information 41 output by the observation object identification algorithm is transmitted to the observation support algorithm selection unit 50, which transmits specific observation support algorithm selection information 51 based on the observation object identification information to the observation support information generation unit 60. At this point in time, it is assumed that time has passed from time T(0) to time T(1), and a medical image 33 has been acquired. In this case, the observation support information generation unit 60 inputs the medical image 33, which is the latest medical image, to the specific observation support algorithm. The observation support information 61 output by the specific observation support algorithm is transmitted to the notification control unit 70.

[0076] In this way, by inputting the latest medical images into the observation assistance algorithm, it is possible to notify observation assistance information in almost real time. Furthermore, the most recently acquired medical images (latest medical images) may be input into the position determination algorithm and operation assistance algorithm, which will be described later. During an endoscopic examination, medical images are acquired continuously within a very short period of time, and therefore, the observation targets included in a medical image determined as "appropriate" or "inappropriate" by the position determination algorithm and a medical image that is the target of operation assistance usually do not change significantly, except in special cases such as when the subject is performing a specific action, such as swallowing or coughing.

[0077] The notification control unit 70 controls the notification of observation support information. The observation support information is notified to the user via the user interface 12 as a guide image displayed on the display or as sound emitted from a speaker. When a guide image is to be displayed, the notification control unit 70 generates a guide image using the observation support information and controls the display of the guide image. It also controls the generation of sound instruction information using the observation support information and the notification of the observation support information as sound from a speaker. Note that the notification method may be a combination of displaying a guide image and sound instruction information. The notification method is not limited to this.

[0078] As a specific example, an example in which endoscope position determination information indicating whether the position of the tip portion 22d of the endoscope 22 is appropriate or inappropriate is displayed on the guide image will be described with reference to FIGS. 5, 6 and 7. FIG.

[0079] In this example, the observation target identification algorithm outputs "oropharynx" as the region requiring observation support. In this case, the observation support algorithm selection unit 50 selects the oropharynx observation support algorithm as the specific observation support algorithm. The oropharynx observation support algorithm processes medical images in which the observation target is the oropharynx (hereinafter referred to as oropharynx images), and outputs endoscope position determination information indicating whether the position of the tip 22d of the endoscope 22 is "appropriate" or "inappropriate" as observation support information. If the oropharynx observation support algorithm outputs endoscope position determination information that is "appropriate," the notification control unit 70 generates a guide image 71 shown in FIG. 5 and displays it on the display.

[0080] The notification control unit 70 generates a guide image 71 by superimposing observation support information on the medical image input to the observation support algorithm or the latest medical image, or by displaying these medical images and the observation support information side by side. In the example of the guide image 71 shown in Fig. 5, a medical image 72 and an observation support information display field 73 are provided. In the example shown in Fig. 5, a message indicating that the endoscope position determination information is "appropriate" is displayed in the observation support information display field 73.

[0081] 5 includes anatomical structures in the oropharynx, such as the base of the tongue Rl, epiglottis Eg, glottis Rg, vocal folds Vof, pyriform sinuses Ps, and posterior pharyngeal wall Pw. Details of the type of medical image that is output as "appropriate" will be described later.

[0082] On the other hand, when the oropharyngeal observation support algorithm outputs endoscope position determination information of "inappropriate," indicating that the position of the tip 22d of the endoscope 22 is inappropriate, the notification control unit 70 generates a guide image 71 shown in Fig. 6 and displays it on the display. An observation support information display field 73 is provided. In the example of the guide image 71 shown in Fig. 6, a message indicating the endoscope position determination information of "inappropriate" is displayed in the observation support information display field 73. Note that the medical image 72 (oropharyngeal image) of the guide image 71 shown in Fig. 6 is an image that is too close to the larynx, and is therefore determined to be "inappropriate."

[0083] When the oropharyngeal observation support algorithm outputs endoscope position determination information of "appropriate" or "inappropriate," the notification control unit 70 may control the speaker to emit audio instructions such as audio messages such as "appropriate position" or "inappropriate position," or alarm sounds associated with "appropriate" or "inappropriate." These audio messages may also be emitted simultaneously with the guide image 71 shown in FIG. 6. Alternatively, the system may be configured in advance not to emit an audio message only when the oropharyngeal observation support algorithm outputs endoscope position determination information of "appropriate" or "inappropriate."

[0084] Below, we will explain how to notify observation support information when the observation target identification algorithm outputs observation target identification information of "not subject to observation support." In this case, the observation support algorithm selection unit 50 selects the not-subject-to-observation-support observation support algorithm as the specific observation support algorithm. The not-subject-to-observation-support observation support algorithm performs image processing on medical images identified as "not subject to observation support" (images not subject to observation support), and outputs observation support stop information, which indicates that observation support should be stopped, as observation support information.

[0085] In this case, the notification control unit 70 generates a guide image 71 shown in Fig. 7. In the example of the guide image 71 shown in Fig. 7, a message indicating observation support stop information of "CAD OFF", which indicates that observation support (Computer Aided Diagnosis / Detection, CAD) is stopped, is displayed in the observation support information display field 73. Note that the guide image 71 shown as an example in Fig. 7 is a guide image generated by the notification control unit 70 as a result of the medical image 72 being blurred, and therefore "not subject to observation support" being output by the observation target identification algorithm, and the observation support stop information being output by the not-subject-to-observation-support observation support algorithm.

[0086] With the above configuration, it is possible to output observation support information according to the region of the observation target contained in the medical image sent from the modality 20 or the database 11. Furthermore, it is possible to identify medical images that are not suitable for observation as not subject to observation support, and to select a specific observation support algorithm according to the scene in which such medical images were acquired, and then output the observation support information.

[0087] Furthermore, in this embodiment, the observation target identification algorithm can identify the nasal cavity, nasopharynx, oropharynx, hypopharynx, larynx, trachea, or esophagus as the region requiring observation assistance. This configuration is particularly suitable for swallowing endoscopy, which requires observation at multiple observation regions while holding the endoscope 22 in an appropriate position.

[0088] Swallowing endoscopy is one of the methods used to evaluate swallowing disorders (swallowing function evaluation tests). Swallowing refers to the series of actions involved in placing food or drink in the mouth, chewing, swallowing, and sending it down the esophagus, as shown in Figure 8. In the example of normal swallowing shown in Figure 8, the process transitions from the "oral phase" (Figure 8(A)) in which food F is transported from the oral cavity to the pharynx primarily by movement of the tongue To, to the "pharyngeal phase" (Figure 8(B)) in which food F is transported from the pharynx to the esophagus Es by the swallowing reflex, to the "esophageal phase" (Figure 8(C)) in which food F is transported from the esophagus Es to the stomach by esophageal peristalsis. During swallowing, the epiglottis Eg, which acts as a lid on the trachea Tr, closes the entrance to the trachea Tr (glottis Vc) by reflex movement to direct food F toward the esophagus Es and prevent it from entering the trachea Tr. In addition, the soft palate (Sp), which is the roof of the oral cavity, also moves backward to close the oral and nasal passages, preventing food from entering the mouth. thing Prevent F from entering the nasal cavity.

[0089] Dysphagia refers to a condition in which food or drink cannot be swallowed properly due to aging, neurological disorders, or other factors. In swallowing endoscopy, it is recommended to observe the swallowing state from multiple observation positions to determine whether or not a swallowing disorder has occurred and to investigate the cause of the swallowing disorder. The multiple observation positions refer to three positions of the endoscope 22 for observing the nasopharynx, oropharynx, and hypopharynx, as shown in Figures 9(A), 9(B), and 9(C). When the endoscope 22 is inserted transnasally and the nasopharynx, oropharynx, and hypopharynx are observed in this order, the tip 22d is moved in the order of Figures 9(A), 9(B), and 9(C). The observation position for the nasopharynx is shown in Figure 9(A). The observation position for the oropharynx is shown in Figure 9(B). The observation position for the hypopharynx is shown in Figure 9(C).

[0090] For an appropriate diagnosis, it is necessary to position the tip 22d at each site involved in swallowing, as shown in Figures 9(A), 9(B), and 9(C). Textbooks, guidelines, and the like indicate appropriate positions for each of these sites. However, the appropriate procedure for operating the endoscope 22 to position the tip 22d at that position depends on the physician's experience, and the appropriate method for operating the endoscope 22 also varies depending on the subject. For this reason, it is difficult for anyone other than an expert to position the endoscope 22 at the appropriate observation position and to approach the appropriate observation position.

[0091] This embodiment can automatically provide support for determining whether the tip portion 22d is appropriately positioned for observing the nasopharynx, oropharynx, or hypopharynx, and also for operating the endoscope 22 to move it to the appropriate observation position for each site. Therefore, even a doctor with little experience in swallowing endoscopy can be provided with observation support so that they can perform observations according to the guidelines.

[0092] Thus, in a swallowing endoscopy, the determination of whether the current position is appropriate and the content of the operation guidance for the endoscope 22 to the appropriate position vary depending on the region the user wishes to observe. Therefore, it is preferable to select a different observation support algorithm for each region, as in this embodiment. If a single observation support algorithm is used in a swallowing endoscopy, a medical image showing an appropriate position for observing the oropharynx may be deemed inappropriate for observing the hypopharynx because the glottis is too far away. Similarly, a medical image showing an appropriate position for observing the hypopharynx may be deemed inappropriate for observing the oropharynx because the glottis is too close. In this embodiment, a specific observation support algorithm is selected from multiple observation support algorithms after identifying the observation target, thereby preventing inconsistencies in the determination results that would occur if a single observation support algorithm were used.

[0093] Furthermore, by selecting a specific observation assistance algorithm after identifying the observation target, it is possible to automatically switch between guiding the endoscope 22 forward in a direction for close-up observation of the glottis (rearward) and guiding the endoscope 22 backward in a direction for distant-view observation of the glottis (frontward), depending on whether the observation objective is the oropharynx or the hypopharynx. This embodiment makes it possible to provide the most appropriate observation assistance according to the user's observation objective.

[0094] Furthermore, in addition to the period during which the tip 22d is finely adjusted to an appropriate position for observing the nasopharynx, oropharynx, or hypopharynx, automatic switching to the most appropriate observation support algorithm can be performed during the period during which the tip 22d is in the nasal cavity until the endoscope 22 is moved from the anterior nares to the nasopharynx.

[0095] 10 and 11, if the endoscope 22 is accidentally inserted into the esophagus Es (FIG. 10) or trachea Tr (FIG. 11), it is necessary to immediately pull the endoscope 22 back to the pharynx. Even in such a case, the most appropriate observation assistance algorithm can be automatically switched to provide observation assistance for withdrawing the endoscope 22.

[0096] The observation object identification algorithm for identifying the observation assistance target region is preferably a trained model trained using medical images depicting the nasal cavity, nasopharynx, oropharynx, or hypopharynx, which a physician has determined to be suitable for observation, as correct answer images. Furthermore, when generating such a trained model, it is more preferable to use, as training images, a series of video images in which the observation target changes from the nasal cavity, nasopharynx, oropharynx, to the hypopharynx in the order in which the endoscope 22 is inserted transnasally. It is more preferable that the training images include a series of video images in which the observation target changes from the hypopharynx, oropharynx, nasopharynx, to the nasal cavity in the order in which the endoscope 22 is removed transorally. It is even more preferable that the training images include a series of video images in which the observation target changes from the oral cavity, oropharynx, to the hypopharynx in the order in which the endoscope 22 is inserted transorally. In this way, generating a trained model using training images in which the order in which the endoscope 22 passes through is weighted chronologically, thereby improving the accuracy of the observation object identification algorithm in identifying the observation assistance target region.

[0097] An example of the observation support information output by the observation support algorithm is described below. The observation support information includes endoscope position determination information, endoscope operation information, subject position information, and observation support stop information.

[0098] The endoscope position determination information is information that indicates whether the position of the tip 22d of the endoscope 22 is "appropriate" or "inappropriate." The endoscope position determination information is information that is output by an oropharynx position determination algorithm and a hypopharynx position determination algorithm, which will be described later, among the observation support algorithms. Specific examples of notification of endoscope position determination information in the oropharynx and hypopharynx will be described later.

[0099] The endoscope operation information is information indicating an operation method of the endoscope. Specifically, it is information indicating the movement direction and / or movement amount of the tip 22d of the endoscope 22. The movement direction of the tip 22d includes rightward, leftward, upward, downward, backward, forward, rightward rotation, or leftward rotation. The observation support algorithms that output endoscope operation information are a nasal cavity operation support algorithm, a nasopharynx operation support algorithm, a oropharynx operation support algorithm, a hypopharynx operation support algorithm, and a misinsertion operation support algorithm, which will be described later. Specific examples of notification of endoscope operation information for the nasal cavity, nasopharynx, oropharynx, hypopharynx, esophagus, and trachea will be described later.

[0100] The subject position information is information that prompts the user to change or confirm the subject's recommended position to obtain medical images suitable for observation. Specifically, the information instructs the user to change or confirm the subject's position or posture, such as the angle of flexion (forward bending), extension (retroflexion), or rotation of the head and neck, the angle of the jaw relative to the trunk, and the orientation of the trunk. The observation support algorithms that output the subject position information are the oropharyngeal operation support algorithm and the hypopharyngeal operation support algorithm, which will be described later. During endoscopic examinations, the surgeon may not be able to maintain the tip 22d in an appropriate observation position solely through operation of the endoscope 22. Therefore, by reporting the subject position information, the physician can adjust the subject's posture appropriately, or the subject can be encouraged to adjust their own posture, thereby assisting in moving the tip 22d to an appropriate observation position.

[0101] The observation support stop information is information indicating that observation support will be stopped. The observation support algorithm that outputs the observation support stop information is an algorithm for supporting operations outside the observation support target, which will be described later. The observation support stop information is notified by displaying a message indicating the observation support stop information, such as "CAD OFF," on the guide image (see Fig. 7). The notification is also made by stopping the notification of observation support information, such as endoscope position determination information, endoscope operation information, and subject position information.

[0102] Below, we will explain, with specific examples, the observation support algorithm used, the observation support information output by the observation support algorithm, and the method of notifying the observation support information when the area identified from the medical image to be subjected to observation support is the nasal cavity, nasopharynx, oropharynx, hypopharynx, esophagus, or trachea, and also when the medical image is identified as not being subjected to observation support.

[0103] First, the observation support algorithm will be described. The observation support algorithm includes a position determination algorithm and an operation support algorithm, and the algorithm to be used is switched depending on the specific observation support algorithm selected according to the observation target identification information. Specifically, if the observation target identification information is output as "nasal cavity," "nasopharynx," "esophagus," "trachea," or "not subject to observation support," the operation support algorithm is used as the observation support algorithm. On the other hand, if the observation target identification information is output as "oropharynx" or "hypopharynx," the position determination algorithm and the operation support algorithm are used as the observation support algorithm. Each part will be described in detail below.

[0104] When the region to be assisted with observation is the nasal cavity, the nasal cavity observation support algorithm provided in the nasal cavity observation support information generating unit 60a is selected as the specific observation support algorithm. The nasal cavity observation support algorithm is an operation support algorithm. The operation support algorithm used when the region to be assisted with observation is the nasal cavity is called the nasal cavity operation support algorithm.

[0105] When the region requiring observation assistance is the nasopharynx, the nasopharynx observation assistance algorithm provided in the nasopharynx observation assistance information generating unit 60b is selected as the specific observation assistance algorithm. The nasopharynx observation assistance algorithm is an operation assistance algorithm. The operation assistance algorithm used when the region requiring observation assistance is the nasopharynx is called the nasopharynx operation assistance algorithm.

[0106] When the region requiring observation assistance is the oropharynx, the oropharynx observation assistance algorithm provided in the oropharynx observation assistance information generating unit 60c is selected as the specific observation assistance algorithm. The oropharynx observation assistance algorithm is composed of a combination of a position determination algorithm and an operation assistance algorithm. The position determination algorithm used when the region requiring observation assistance is the oropharynx is called the oropharynx position determination algorithm. The operation assistance algorithm used when the region requiring observation assistance is the oropharynx is called the oropharynx operation assistance algorithm.

[0107] When the region to be assisted in observation is the hypopharynx, the hypopharynx observation support algorithm provided in the hypopharynx observation support information generating unit 60d is selected as the specific observation support algorithm. The hypopharynx observation support algorithm is composed of a combination of a position determination algorithm and an operation support algorithm. The position determination algorithm used when the region to be assisted in observation is the oropharynx is called the hypopharynx position determination algorithm. When the region to be assisted in observation is the oropharynx, the position determination algorithm used is called the hypopharynx position determination algorithm. under The manipulation support algorithm used in the case of the pharynx is called the hypopharyngeal manipulation support algorithm.

[0108] When the observation support target region is the esophagus or trachea, the misinsertion observation support algorithm provided in the misinsertion observation support information generating unit 60e is selected as the specific observation support algorithm. The misinsertion observation support algorithm is an operation support algorithm. The operation support algorithm used when the observation support target region is the trachea or esophagus is called the misinsertion operation support algorithm.

[0109] When a medical image is identified as not being subject to observation support, the observation support algorithm for not being subject to observation support provided in the non-observation support target observation support information generating unit 60f is selected as the specific observation support algorithm. The observation support algorithm for not being subject to observation support is an operation support algorithm. The operation support algorithm used when a medical image is identified as not being subject to observation support is called the non-observation support target operation support algorithm.

[0110] The observation support information output by the nasal cavity operation support algorithm and the notification of the observation support information are described below. The nasal cavity operation support algorithm is an operation support algorithm that provides observation support by indicating an appropriate insertion route for the endoscope 22 from the anterior nares to the nasopharynx.

[0111] There are two insertion routes for the transnasal endoscope 22. One is an upper route 90 that runs from the anterior nares through the area of ​​the common nasal meatus surrounded by the middle turbinate Mt, the nasal septum Ns, and the inferior turbinate It to the posterior nares, as shown in Figure 12. The other is an inferior route 91 that runs from the anterior nares through the area of ​​the common nasal meatus surrounded by the inferior turbinate It, the nasal septum Ns, and the floor of the nasal cavity Nf to the posterior nares, as shown in Figure 12.

[0112] When the human face Fa is viewed from the front, the region into which the endoscope 22 is inserted in the superior route is a region 90a surrounded by the middle turbinate Mt, the nasal septum Ns, and the inferior turbinate It, as shown in Fig. 13. The region into which the endoscope 22 is inserted in the inferior route is a region 91a surrounded by the inferior turbinate It, the nasal septum Ns, and the nasal cavity floor Nf, as shown in Fig. 13. In Fig. 13, the region considered to be the "nasal cavity" through which the endoscope 22 may pass is indicated by a dotted pattern. The nasal cavity includes the common nasal meatus and the superior, middle, and inferior nasal meatus, which are continuous with the common nasal meatus and are not suitable for inserting the endoscope 22.

[0113] FIG. 14 shows an example of a medical image taken while passing through the upper route (hereinafter referred to as upper route image 100). FIG. 15 shows an example of a medical image taken while passing through the lower route (hereinafter referred to as lower route image 110). FIG. 16 shows an example of a medical image (hereinafter referred to as anterior nose image 120) in which both the upper route insertion area and the lower route insertion area, which will be described later, are included in the observation target. The anterior nose image is a medical image taken at a position close to the anterior nostrils within the nasal cavity area.

[0114] 14, 15, and 16 are examples of medical images of a subject's left nasal cavity, with the upper side of the page representing the subject's head, the lower side representing the subject's tail, the right side representing the subject's left side, and the left side representing the subject's right side. The trained model of the observation object identification algorithm is trained using an upper route image 100 as shown in FIG. 14, a lower route image 110 as shown in FIG. 15, and an anterior nose image 120 as shown in FIG. 16 as training images for the "nasal cavity." Furthermore, the trained model of the observation object identification algorithm is trained using medical images (upper route images and / or lower route images) of the right and left nasal cavities taken while passing through the upper and lower routes, respectively.

[0115] The following describes the case where an upper route image is input to the nasal cavity manipulation support algorithm. When an upper route image such as that shown in Fig. 14 is input, the nasal cavity manipulation support algorithm first outputs, as insertion region information, an upper route insertion region 92, which is surrounded by the middle turbinate Mt, the nasal septum Ns, and the inferior turbinate It, as shown in Fig. 17, and is an area suitable for inserting the endoscope 22.

[0116] The nasal cavity manipulation assistance algorithm that outputs the upper root insertion region is preferably a trained model that performs segmentation on medical images and outputs the upper root insertion region. The trained model that outputs the upper root insertion region is preferably generated by training using training images including correct images in which the upper root insertion region has been classified in advance by a doctor. The training model used to generate such a trained model is preferably a Pyramid Scene Parsing Network (PSPnet), but other training models suitable for segmentation, such as a Spatial Pyramid Pooling Network (SPPnet), may also be used. The nasal cavity manipulation assistance algorithm may also be a trained model generated by training a training model that applies unsupervised learning.

[0117] Next, the nasal cavity manipulation support algorithm outputs insertion route information using insertion area information indicating the upper route insertion area. The insertion route information is information used to smoothly insert the insertion section 22a of the endoscope 22 into the body. Specifically, when an upper route image is input to the nasal cavity manipulation support algorithm, the insertion route information is coordinate information of the center position of the upper route insertion area. Alternatively, the insertion route information may be the area or width of the upper route insertion area.

[0118] The coordinates of the center position of the upper route insertion area are the center of gravity or geometric center of the upper route insertion area. The geometric center of the upper route insertion area can be calculated, for example, by taking the coordinates of any number of points from the pixels that make up the perimeter of the upper route insertion area and using a general method for calculating the geometric center of a polygon. For example, if the number of pixels in the upper route insertion area is k, and an arbitrary point in the upper route insertion area is set as the origin and the coordinates of the i-th pixel are (x_i, y_i), the geometric center is calculated by dividing the sum of all coordinates by the number of pixels k. That is, the calculations can be made as follows: (x coordinate of the geometric center of the upper route insertion area) = Σ(x_i) / k, (y coordinate of the geometric center of the upper route insertion area) = Σ(y_i) / k. The method for calculating the coordinate information of the center position of the upper route insertion area is not limited to this. For example, the upper route insertion area can be surrounded by a rectangle and the coordinates of the center position of that rectangle can be used. The coordinates of any two points may be taken from the pixels that make up the upper route insertion area, and the midpoint of the longest distance between the two points may be used as the coordinates of the center position.

[0119] The area of ​​the upper route insertion area is calculated by the method of calculating the area of ​​the area classified as the upper route insertion area by segmentation. The width of the upper route insertion area is the longest distance between the coordinates of any two points from the pixels that make up the upper route insertion area.

[0120] An example of notifying observation support information will be described below. First, an example will be described in which the insertion route information is the coordinates of the center position of the upper route insertion area. The nasal cavity operation support algorithm outputs an upper route insertion area 92 as shown in FIG. 17, and then outputs the coordinates of the center position of the upper route insertion area 92 as insertion route information and transmits it to the notification control unit 70. The notification control unit 70 generates a guide image 71 as shown in FIG. 18 and controls its display on the screen.

[0121] 18, an insertion route guide mark 94 indicating the coordinates of the center position of the upper route insertion area 92, an image center guide mark 95 (a cross mark) indicating the center position of the upper route image 100, and a guide frame 96 that makes it easier to view the image center guide mark 95 are displayed. In this case, the user can position the tip 22d at an ideal insertion position by operating the endoscope 22 so as to align the position of the insertion route guide mark 94 with the image center guide mark 95.

[0122] The ideal insertion position of the upper route is a position where the center position of the upper route insertion area is near the center position of the medical image and the width of the upper route insertion area is arranged so that it extends in the vertical direction of the medical image. Specifically, in the medical image observed in real time, the ideal insertion position is the position of the tip 22d where the middle turbinate Mt and the nasal septum Ns are on the left and right of the medical image and the inferior turbinate It is below the medical image.

[0123] When the tip portion 22d of the endoscope 22 is positioned at an ideal insertion position where the insertion route guide mark 94 and the image center guide mark 95 overlap, the user can smoothly insert the tip portion 22d by advancing the tip portion 22d toward the back of the nasal cavity (pushing the tip portion 22d in). Therefore, by displaying a guide image 71 displaying the insertion route guide mark 94 and the image center guide mark 95 as shown in FIG. 18, observation assistance can be provided to help the user smoothly insert the tip portion 22d. As a result, the pain experienced by the subject when inserting the endoscope 22 can be reduced.

[0124] In addition, although the upper route insertion area 92 is depicted in FIG. 18, the upper route insertion area 92 may or may not be displayed on the guide image 71. Also, the image center guide mark 95 is displayed at a fixed position on the guide image 71. In this case, the notification control unit 70 notifies the insertion route information transmitted from the nasal cavity operation support algorithm as observation support information. In addition to the upper route insertion area, the guide image 71 also displays the lower route insertion area, the glottis area, the epiglottis area, and the glottis area, which will be described later, as insertion area information. band The pleat area may be displayed. Also, the display and non-display of the insertion area information in the guide image 71 may be switched by operating a notification display switching button or the operation unit 22b, which will be described later.

[0125] The nasal cavity manipulation support algorithm may generate endoscope manipulation information by using insertion route information, which is the coordinates of the center position of the upper route insertion region, and may report the endoscope manipulation information as observation support information. In this case, the nasal cavity manipulation support algorithm outputs a vector indicating the movement direction and movement amount for moving the tip portion 22d as endoscope manipulation information. The endoscope manipulation information as a vector is the direction and distance from the coordinates of the center position of the upper route insertion region 92 to the coordinates of the center position of the upper route image 100. The vector is calculated by calculating the difference between the information on the coordinates of the center position of the upper route insertion region 92 and the information on the coordinates of the center position of the upper route image 100. The endoscope manipulation information output by the nasal cavity manipulation support algorithm is transmitted to the notification control unit 70.

[0126] In this case, the notification control unit 70 generates a guide image 71 as shown in Fig. 19 and controls the display to display it on the display. In the example of the guide image 71 shown in Fig. 19, an image center guide mark 95, a guide frame 96, and a guide arrow 102 are included. andis displayed. The guide arrow 102 is an arrow-shaped display that is generated based on endoscope operation information and indicates the direction and amount of movement for moving the tip portion 22d from the image center guide mark 95 (coordinates of the center position of the upper route insertion area) to the image center guide mark 95 (coordinates of the center position of the upper route image). The guide arrow 102 may be displayed larger or smaller than the guide image 71 shown in FIG. 19 to indicate whether the amount of movement of the tip portion 22d is large or small. For example, the length of the guide arrow 102 can indicate the magnitude of the amount of movement. By displaying the guide image 71 as shown in FIG. 19, the guide arrow can indicate to the user the direction and amount of movement of the tip portion 22d.

[0127] Another example of notifying the endoscope operation information output by the nasal cavity operation support algorithm will be described. For example, when an upper route image as shown in FIG. 14 is input to the nasal cavity operation support algorithm, the notification control unit 70 generates a guide image 71 as shown in FIG. 20. In the example of the guide image 71 shown in FIG. 20, guide direction display icons 101a, 101b, 101c, and 101d are displayed around the upper route image 100. In the guide image 71, the movement direction of the distal end portion 22d is indicated, for example, by changing the display mode of each guide direction display icon depending on the ideal insertion route. Specifically, when the movement direction of the distal end portion 22d is output as "upward" as the endoscope operation information, the guide direction display icon 101d is displayed in a different color from the guide direction display icons 101a, 101b, and 101c, as shown in the example of FIG. 20. Note that in the example shown in FIG. 20, the color of the guide direction display icon differs depending on whether or not there is a diagonal line, indicating that the endoscope 22 is to be operated in the upward direction.

[0128] In addition, when the moving direction of the distal end portion 22d is set to "downward and leftward" as the endoscope operation information, the guide direction display icon 101 for the leftward direction is displayed as in the example of the guide image 71 shown in FIG. c and downward guide direction display icon 101 b and the right guide direction display icon 101 aThe guide direction display icons 101d for the left and right directions are displayed in different colors. The display manner of the guide direction display icons is not limited to this.

[0129] Furthermore, as the endoscope operation information, an endoscope operation support diagram showing the operation of the angle knob 22e for bending and moving the distal end portion 22d may be displayed in the guide image 71. In the guide image 71 shown in Fig. 22, an endoscope operation support diagram 103a is displayed showing the operation of the angle knob 22e for moving the distal end portion 22d in a "downward and leftward" direction. Note that the endoscope operation support diagram 103a shown in Fig. 22 shows an example of an operation instruction to rotate counterclockwise the outer lever 103b for bending the bending portion 22c in the left-right direction and the inner lever 103c for bending the bending portion 22c in the up-down direction.

[0130] Furthermore, the guide image 71 shown in FIG. 23 displays an endoscope operation support diagram 103a showing the operation of the angle knob 22e for "rotating left" as the movement direction of the tip portion 22d. The endoscope operation support diagram 103a in the guide image 71 shown in FIG. 23 is drawn so that the angle knob 22e is positioned on the left side as seen by the surgeon. Note that the guide image may display a combination of guide direction display icons, arrow displays, guide arrows, and endoscope operation support diagrams. With the above configuration, the ideal endoscope insertion route in the nasal cavity can be automatically notified to the user. Note that the movement direction and amount of the endoscope 22 may also be notified by voice.

[0131] An example of the presentation of observation support information when the insertion route information is the area or width of the upper route insertion area will be described below. In this case, the display mode of the upper route insertion area may be changed according to the insertion route information output by the nasal cavity operation support algorithm. For example, a first threshold for displaying the insertion area and a second threshold for displaying the insertion area that is smaller than the first threshold for displaying the insertion area are set for the area of ​​the upper route insertion area. As the display mode of the upper route insertion area of ​​the guide image 71 (for example, the display mode of the upper route insertion area 92 in FIG. 18), if the area of ​​the upper route insertion area is larger than the first threshold for displaying the insertion area, it is displayed in blue; if it is equal to or smaller than the first threshold for displaying the insertion area and larger than the second threshold for displaying the insertion area, it is displayed in yellow; and if it is equal to or smaller than the second threshold for displaying the insertion area, it is displayed in red. In other words, the display is switched from blue to yellow to red in descending order of the area of ​​the upper route insertion area. The first threshold for displaying the insertion area and the second threshold for displaying the insertion area can be set to any value. In addition, of The display mode is not limited to this. Furthermore, a first threshold value for displaying the insertion area and a second threshold value for displaying the insertion area may be set for the width of the upper route insertion area. In this case, the notification control unit 70 notifies the insertion route information transmitted from the nasal cavity operation support algorithm as observation support information.

[0132] The upper route images acquired during insertion of the endoscope 22 may include images for which the nasal cavity manipulation assistance algorithm cannot output an upper route insertion area. For example, due to a large deviation from the ideal insertion position, the upper route images acquired in real time may not show the upper route insertion area, or may show the upper route insertion area but be too small. The nasal cavity manipulation assistance algorithm may be trained using training images in which such upper route images that do not show the upper route insertion area or that show an upper route insertion area that is too small are associated with the movement direction or movement amount of the tip portion 22d to the ideal insertion position (i.e., endoscope manipulation information). In this case, the nasal cavity manipulation assistance algorithm, to which an upper route image that does not show the upper route insertion area or that shows an upper route insertion area that is too small, is input, outputs endoscope manipulation information.

[0133] The nasal cavity manipulation support algorithm that outputs the upper route insertion area may be the first nasal cavity manipulation support algorithm, and the nasal cavity manipulation support algorithm that outputs endoscope manipulation information by inputting an upper route image that does not show the upper route insertion area or an upper route image in which the upper route insertion area is too small may be the second nasal cavity manipulation support algorithm. The latest medical images may be input to the first nasal cavity manipulation support algorithm and / or the second nasal cavity manipulation support algorithm.

[0134] Furthermore, in training the nasal cavity manipulation assistance algorithm as a learning model, it is preferable to use training images in which the orientation (up, down, left, right) of the upper root image input to the nasal cavity manipulation assistance algorithm has been determined in advance. Furthermore, in training the nasal cavity manipulation assistance algorithm, it is preferable that the parameter update be performed taking into account the orientation. Furthermore, when associating the training image with the ideal insertion position, it is preferable that the association be performed taking into account the orientation. The same applies to the lower root image and the anterior nose image described below.

[0135] The above configuration allows surgeons who are unfamiliar with inserting a transnasal endoscope to confirm that they have selected an easy route for inserting the endoscope. Furthermore, the upper route insertion area varies from person to person depending on the internal structure of the subject's nasal cavity. With the above configuration, when a subject has a small upper route insertion area, the surgeon can be prompted to take care regarding endoscope insertion and select an insertion route that will reduce the subject's pain associated with endoscope insertion.

[0136] The following describes the case where a lower route image is input to the nasal cavity manipulation support algorithm. When a lower route image such as that shown in Fig. 15 is input, the nasal cavity manipulation support algorithm first outputs, as insertion region information, a lower route insertion region 93 that is surrounded by the inferior turbinate It, the nasal septum Ns, and the nasal cavity floor Nf, as shown in Fig. 24, and is suitable for inserting the endoscope 22. The lower route insertion region 93, like the upper route insertion region (see Fig. 17), is a region that is classified by the nasal cavity manipulation support algorithm as a learned model that performs segmentation.

[0137] Next, the nasal cavity operation support algorithm outputs insertion route information using insertion area information indicating the lower route insertion area. When a lower route image is input to the nasal cavity operation support algorithm, the insertion route information is coordinate information of the center position of the lower route insertion area. It may also be the area or width of the lower route insertion area. The coordinates of the center position of the lower route insertion area are the center of gravity or geometric center of the lower route insertion area. The method for calculating the center position of the lower route insertion area is the same as the method for calculating the center position of the upper route insertion area, so it will be omitted here. The method for calculating the area or width of the lower route insertion area is also the same as the method for calculating the center position of the upper route insertion area. Area or width The calculation method is the same as that of

[0138] The method of notifying the observation support information when a lower route image is input to the nasal cavity manipulation support algorithm is the same as when an upper route image is input to the nasal cavity manipulation support algorithm, and therefore a detailed example will be omitted. Briefly, when the insertion route information is set to the coordinates of the center position of the lower route insertion area, the insertion route information is used as the observation support information, and control is performed to display a guide image 71 on which an insertion route guide mark 94 and an image center guide mark 95 are displayed, as shown in Fig. 18 .

[0139] The ideal insertion position of the lower route is a position where the center of the lower route insertion area is near the center of the medical image and the width of the lower route insertion area is extended in the vertical direction of the medical image. Specifically, in the medical image observed in real time, the ideal insertion position is the position of the tip 22d where the inferior turbinate It and the nasal septum Ns are on the left and right of the medical image and the floor of the nasal cavity Nf is below the medical image.

[0140] In addition, when the insertion route information is the coordinates of the center position of the lower route insertion area, the nasal cavity operation support algorithm may output endoscope operation information by using the insertion route information. The endoscope operation information may include an image center guide mark 95 and a guide arrow 102 as shown in FIG. but The information is notified as a guide image 71 displayed on the screen, a guide image 71 on which a guide direction display icon is displayed as shown in Fig. 20, or a guide image 71 on which an endoscope operation support diagram is displayed as shown in Fig. 21 and Fig. 22. The method of outputting endoscope operation information is the same as when an upper route image is input to the nasal cavity operation support algorithm.

[0141] When the insertion route information is the area or width of the lower route insertion area, or when a lower route image that does not show the lower route insertion area or a lower route image in which the lower route insertion area is too small is input into the nasal cavity operation support algorithm, the method of outputting and notifying the observation support information is the same as when an upper route image is input into the nasal cavity operation support algorithm.

[0142] The following describes the case where a front nose image is input to the nasal cavity manipulation support algorithm. When a front nose image 120 such as that shown in FIG. 16 is input, the nasal cavity manipulation support algorithm first outputs, as insertion region information, an upper route insertion region 92 surrounded by the middle turbinate Mt, the nasal septum Ns, and the inferior turbinate It, which are suitable for inserting the endoscope 22, and an lower route insertion region 93 surrounded by the inferior turbinate It, the nasal septum Ns, and the nasal cavity floor Nf, as shown in FIG. 25. In the example of the front nose image 120 shown in FIG. 25, the region connected to the common nasal meatus (see FIG. 13) on the opposite side of the nasal cavity lateral wall Nw (i.e., on the nasal septum Ns side) is detected as the upper route insertion region 92. That is, when the front nose image 120 is input, the nasal cavity manipulation support algorithm outputs regions suitable for inserting the endoscope 22 for both the upper and lower routes. In this case, the upper route insertion region 92 and the lower route insertion region 93 are classified into different classes as a result of segmentation.

[0143] Next, the nasal cavity operation support algorithm uses the classified insertion area information (upper route insertion area and lower route insertion area) in the input anterior nose image to output insertion route information for each of the upper route insertion area and the lower route insertion area. The method for outputting the insertion route information is the same as when using the upper route image 100 or the lower route image 110, so it will be omitted here.

[0144] Next, the nasal cavity operation assistance algorithm outputs insertion route information for each of the upper route insertion region and the lower route insertion region. Furthermore, endoscope operation information is output using the insertion route information. The output method of the insertion route information and endoscope operation information and its preferred embodiments are similar to the output method of the insertion route information and endoscope operation information using the upper route image or the lower route image, and therefore will not be described further.

[0145] An example of notification of observation support information when an anterior nasal image is input to the nasal cavity operation support algorithm will be described. When notifying insertion route information as observation support information, the notification control unit 70 generates a guide image 71 as shown in Fig. 26. In the example of the guide image 71 shown in Fig. 26, an insertion route guide mark 94a indicating the coordinates of the center position of the upper route insertion area 92, a lower route insertion area 93, and an insertion route guide mark 94b indicating the coordinates of the center position of the upper route insertion area 93 are generated. 3 An insertion route guide mark 94b indicating the coordinates of the center position of the anterior nose image 120, an image center guide mark 95 indicating the center position of the anterior nose image 120, and a guide frame 96 are displayed. In this case, the user can position the tip 22d at an ideal insertion position by operating the endoscope 22 so as to align the position of the insertion route guide mark 94a or the insertion route guide mark 94b with the image center guide mark 95.

[0146] Furthermore, when notifying endoscope operation information as observation support information, the notification control unit 70 generates a guide image 71 as exemplified in Fig. 27. In the example of the guide image 71 shown in Fig. 27, the detected upper route insertion region 92 and lower route insertion region 93 are superimposed and displayed on the front nose image 120. In addition, an upper route operation display field 121 and a lower route operation display field 122 are displayed outside the front nose image 120. In the example shown in Fig. 27, the upper route operation display field 121 displays the movement direction of the tip 22d in the "lower left direction" as endoscope operation information for the upper route insertion region 92. In addition, the lower route operation display field 122 displays the movement direction of the tip 22d in the "upper right direction" as endoscope operation information for the lower route insertion region 93.

[0147] In this case, the display mode of the upper route insertion region and the lower route insertion region may be changed according to the insertion route information output by the nasal cavity operation assistance algorithm. Specifically, a first threshold value for displaying the insertion region and a second threshold value for displaying the insertion region are set for the area of ​​the upper route insertion region, and the display color of the upper route insertion region is switched from blue to yellow to red in descending order of area. Similarly, a first threshold value for displaying the insertion region and a second threshold value for displaying the insertion region are set for the area of ​​the lower route insertion region, and the display color of the upper route insertion region is switched from blue to yellow to red in descending order of area. In the example shown in FIG. 27, the different display colors of the lower route insertion region 93 and the upper route insertion region 92 are indicated by different hatching (diagonal lines). This configuration allows a surgeon who is unfamiliar with inserting a transnasal endoscope to select an insertion route that facilitates endoscope insertion.

[0148] The observation support information output by the nasopharyngeal operation support algorithm and the notification of the observation support information are described below. The nasopharyngeal operation support algorithm is an operation support algorithm that provides operation support by indicating whether the tip portion 22d is in an appropriate observation position in the nasopharynx and further indicating how to operate the endoscope 22 so that the tip portion 22d is in an appropriate observation position.

[0149] The nasopharynx is the region from the choanal cavity and pharyngeal vault Fp to the base of the uvula. As shown in FIG. 28, the nasopharynx Ep extends from the base of the skull to the transition between the hard and soft palates Sp, the oropharynx Mp extends from the transition between the hard and soft palates Sp to the bottom of the epiglottic vallecula Ev, and the hypopharynx Hp extends from the bottom of the epiglottic vallecula Ev to the lower edge of the cricoid cartilage Cc. The larynx La is located at the level of the lingual surface of the epiglottis Eg to the lower edge of the cricoid cartilage Cc, and is the region indicated by the diagonal lines, surrounded by the epiglottis Eg, thyroid cartilage Tc, and the left and right arytenoid cartilages. When the larynx is observed with an endoscope 22, the glottis Vc, indicated by the dotted line in FIG. 28, can be seen.

[0150] FIG. 29 shows an example of a medical image (hereinafter referred to as a nasopharyngeal image) in which the observation target is the nasopharynx. The example shown in FIG. 29 is an example of a nasopharyngeal image 130 in which the tip 22d is positioned at an appropriate observation position (see FIG. 9(A)). An appropriate observation position in the nasopharynx is a position where the posterior wall Pwe of the nasopharynx, the left and right side walls Lw, and the inferior wall, i.e., the nasal cavity floor Nf and the soft palate Sp, can be observed. Note that the nasal cavity floor Nf and the soft palate Sp are continuous, and when inserting or removing the endoscope 22, the nasal cavity floor Nf can be observed on the near side (nasal cavity side) and the soft palate Sp can be observed on the far side (pharyngeal side). At this position, the movement of the soft palate during speech and swallowing can be observed, allowing evaluation of the velopharyngeal closure function during speech and swallowing. The orientation of the example of the nasopharynx image 130 shown in FIG. 29 is such that the top of the page is the cranial side, the bottom of the page is the caudal side, the front of the page is the ventral side, and the back of the page is the dorsal side.

[0151] The observation object identification algorithm as a trained model is trained using the nasopharynx image shown in Figure 29 as the training image of the "nasopharynx."

[0152] The following describes the case where a nasopharyngeal image is input to the nasopharyngeal manipulation support algorithm. First, when a nasopharyngeal image such as that shown in Fig. 29 is input, the nasopharyngeal manipulation support algorithm outputs nasopharyngeal position information indicating that the position of tip 22d is in an appropriate position or an inappropriate direction position such as "right inappropriate position," "left inappropriate position," "upper inappropriate position," "lower inappropriate position," "rear inappropriate position," or "front inappropriate position."

[0153] "Right inappropriate position" refers to nasopharynx position information in which the position of the tip 22d is too far to the right. Similarly, if the position of the tip 22d is too far to the left, "left inappropriate position" is output; if it is too far up, "upper inappropriate position" is output; if it is too far down, "lower inappropriate position" is output; if it is too far back (toward the pharynx), "deep inappropriate position" is output; and if it is too far forward (toward the nasal cavity), "close inappropriate position" is output. Inappropriate directional positions are not limited to these, and the nasopharynx manipulation support algorithm outputs multiple categories of nasopharynx position information, such as "left-lower inappropriate position" and "right-upper back inappropriate position." The nasopharynx manipulation support algorithm may also output multiple categories of nasopharynx position information, such as "left inappropriate position" and "close inappropriate position."

[0154] The nasopharyngeal manipulation assistance algorithm that outputs nasopharyngeal position information is preferably a trained model trained using training images including nasopharyngeal images associated with nasopharyngeal position information. Convolutional neural networks such as VGGNet (e.g., VGG16, VGG19) and ResNet are suitable as training models used to generate the nasopharyngeal manipulation assistance algorithm as a trained model. The nasopharyngeal manipulation assistance algorithm may also be an algorithm that outputs nasopharyngeal position information by pattern matching with training images.

[0155] Next, the nasopharyngeal manipulation assistance algorithm outputs endoscope manipulation information based on the nasopharyngeal position information. If the nasopharyngeal position information indicates "in an appropriate position," the nasopharyngeal manipulation assistance algorithm outputs endoscope manipulation information to "stop endoscope 22." On the other hand, if the nasopharyngeal position information indicates "in an inappropriate orientation position," the nasopharyngeal manipulation assistance algorithm outputs endoscope manipulation information indicating a movement direction of tip portion 22d that moves tip portion 22d in the opposite direction to the inappropriate direction indicated by the inappropriate orientation position.

[0156] An example of notification of endoscope operation information output by the nasopharyngeal operation support algorithm will be described. For example, when a nasopharyngeal image 130 as shown in Fig. 29 is input to the nasopharyngeal operation support algorithm, the notification control unit 70 generates a guide image 71 as shown in Fig. 30. In the example of the guide image 71 shown in Fig. 30, an observation support information display field 73 is provided in a position different from the nasopharyngeal image 130, and the message "appropriate: still" is displayed in the observation support information display field 73. In this case, the nasopharyngeal position information output is "in appropriate position."

[0157] Instead of a message, a mark indicating "in the appropriate position" may be displayed to instruct the tip portion 22d to stop. For example, when "in the appropriate position," a green frame is displayed around the nasopharynx image 130. By displaying such a guide image 71, the surgeon can confirm that the tip portion 22d is in the appropriate observation position. Furthermore, when the nasopharynx position information "in the appropriate position" is output, the notification control unit 70 may issue a voice message from the speaker saying, "It is in the appropriate position. Please stop."

[0158] An example of notification of endoscope operation information when the endoscope is in an "inappropriate direction" will be described below. For example, a nasopharyngeal image 130 as shown in Fig. 31 is input to the nasopharyngeal operation support algorithm. In the nasopharyngeal image 130 shown in Fig. 31, the posterior wall Pwe of the nasopharyngeal is largely visible, the side wall Lw is slightly visible on the left side of the nasopharyngeal image 130, and the soft palate Sp is slightly visible on the lower side. right The side wall on the other side (i.e., the left side of the subject) is not visible, and the tip 22d is positioned too far back, resulting in an "inappropriately far back" image. In such a case, it is necessary to pull the tip 22d forward. When the nasopharynx image shown in FIG. 31 is input to the nasopharynx operation assistance algorithm, the nasopharynx position information is output as "inappropriately far back."

[0159] In this case, the notification control unit 70 generates a guide image 71 as shown in Fig. 32. In the example of the guide image 71 shown in Fig. 32, guide direction display icons 101a, 101b, 101c, and 101d indicating up, down, left, and right directions are displayed around the nasopharynx image 130. In addition, a guide direction display icon 101e indicating the back direction and a guide direction display icon 101f indicating the front direction are superimposed on the nasopharynx image 130. Furthermore, a guide direction display icon 101g indicating a left turn and a guide direction display icon 101h indicating a right turn are displayed in the guide image 71. In the guide image 71 exemplified in Fig. 32, the presence or absence of diagonal lines indicates that the guide direction display icon 101f indicating the front direction and the other guide direction display icons 101a, 101b, 101c, 101d, 101e, 101g, and 101h have different display modes (colors), indicating that the endoscope 22 is operated in the front direction.

[0160] The display mode of the guide direction display icon is not limited to this. For example, the guide direction display icon may be an upward direction, a downward direction, a leftward direction, a rightward direction, a backward direction, a hand before The guide direction display icons for turning right and turning left may all be displayed on the guide image, or only a part of them may be displayed on the guide image. Also, a message such as "Too far back: Pull back the endoscope" may be displayed on the guide image 71.

[0161] 33, an endoscope operation support image 103a showing an operation for withdrawing the endoscope 22 toward the user may be displayed on the guide image 71. As audio instruction information, an audio message such as "Please withdraw the endoscope" may be output from a speaker.

[0162] The above configuration can assist inexperienced surgeons in observing the nasopharynx by indicating whether the endoscope is in an appropriate position for observing the nasopharynx, and by providing guidance on how to operate the endoscope if the endoscope is in an inappropriate position for observation, thereby reducing the burden on the examinee.

[0163] The function of the oropharynx position determination algorithm, the observation support information output by the oropharynx operation support algorithm, and the notification of the observation support information are described below. The oropharynx position determination algorithm is a position determination algorithm for glottal distant view observation that indicates whether the tip 22d is in an appropriate observation position in the oropharynx. The oropharynx operation support algorithm is an operation support algorithm that provides operation support by indicating how to operate the endoscope 22 so that the tip 22d is in an appropriate observation position based on the output result of the oropharynx position determination algorithm.

[0164] An example of an oropharyngeal image is shown in FIG. 34. FIG. 34 is an example of a medical image 72 displayed on the guide image 71 illustrated in FIG. 5. The oropharyngeal image 140 shown in FIG. 34 is a medical image in which the tip 22d is positioned at an appropriate observation position. An appropriate observation position in the oropharynx is a position where the posterior pharyngeal wall Pw, the base of the tongue RI, and the entire larynx can be observed. Specific components of the larynx that can be observed include the glottis Rg, vocal folds Vof, vestibular folds Vef (also known as false vocal cords), epiglottis Eg, and pyriform sinuses Ps. The glottis Rg is the space between the left and right vocal folds Vof. Furthermore, when performing a closer observation, the aryepiglottic folds Af, the cuneiform tubercle Cut, and the lesser angle tubercle Cot, which will be described later, can be clearly observed outside the vestibular folds Vef. In this position, it is possible to observe from a distance the difference between the left and right movements of the pharyngeal wall during breathing, speaking, and swallowing, and to evaluate whether there are any abnormalities. Also, as with observations in the hypopharynx, it is possible to evaluate whether there is any accumulation of saliva, residual food, etc.

[0165] On the other hand, the oropharyngeal image 140 shown in Fig. 35 is a medical image in which the position of the tip 22d is inappropriate. Fig. 35 is an example of a medical image 72 displayed on the guide image 71 exemplified in Fig. 6. The oropharyngeal image 140 shown in Fig. 35 is an image that is too close to the larynx. Whether or not the tip 22d is too close to the larynx is determined by the oropharyngeal operation assistance algorithm.

[0166] The orientation of the example oropharynx image 140 shown in Figures 34 and 35 is such that the top of the page is the dorsal side, the bottom is the ventral side, the front of the page is the cranial side, and the back of the page is the caudal side. The target identification algorithm as a trained model is trained by using oropharynx images such as those shown in Figures 34 and 35 as training images for "oropharynx."

[0167] The following describes the process flow for outputting observation support information when an oropharyngeal image is input to the oropharyngeal observation support algorithm (oropharyngeal position determination algorithm and oropharyngeal operation support algorithm). First, the oropharyngeal position determination algorithm to which an oropharyngeal image such as that shown in Figure 34 or 35 is input outputs endoscope position determination information indicating whether the position of the tip 22d is "appropriate" or "inappropriate."

[0168] When an oropharyngeal image such as that shown in Fig. 34 is input, the oropharyngeal position determination algorithm outputs, as endoscope position determination information, that the position of tip portion 22d is "appropriate." On the other hand, when an oropharyngeal image such as that shown in Fig. 35 is input, the oropharyngeal position determination algorithm outputs, as endoscope position determination information, that the position of tip portion 22d is "inappropriate."

[0169] The oropharyngeal position determination algorithm that outputs endoscope position determination information is preferably a trained model trained using training images including oropharyngeal images associated with endoscope position determination information that is either "appropriate" or "inappropriate." When supervised or semi-supervised learning is applied to the training model, an oropharyngeal image such as that shown in FIG. 34 may be used as the "appropriate" correct image. Convolutional neural networks such as VGGNet (e.g., VGG16, VGG19) and ResNet are suitable as training models used to generate the oropharyngeal manipulation assistance algorithm as a trained model. The oropharyngeal manipulation assistance algorithm may also be an algorithm that outputs endoscope position determination information by pattern matching with training images.

[0170] Next, the oropharyngeal operation support algorithm outputs endoscope operation information based on the endoscope position determination information. If the endoscope position determination information is “appropriate,” the oropharyngeal operation support algorithm outputs endoscope operation information to “stop the endoscope 22.” In this case, the notification control unit 70 generates a guide image 71 as shown in FIG. 36. In the example of the guide image 71 shown in FIG. 36, an observation support information display field 73 is provided at a position different from the oropharyngeal image 140, and the observation support information display field 73 displays a message “Appropriate: Stop” indicating that the observation position is appropriate. Note that instead of a message, a mark indicating “appropriate” that instructs the tip portion 22d to stop may be displayed. For example, if the position is “appropriate,” a green frame is displayed around the oropharyngeal image 140. In this case, the notification control unit 70 controls the display of the endoscope operation information to be switched so that if the position is “inappropriate,” the mark indicating “appropriate” in the guide image 71 is hidden and a guide direction display icon, etc., is displayed.

[0171] Furthermore, when the endoscope position determination information of "appropriate" is output, the notification control unit 70 may issue a voice message from a speaker saying, "The position is appropriate. Please stay still." By providing such a guide image or voice notification, the surgeon can confirm that the tip portion 22d is in an appropriate observation position of the oropharynx.

[0172] Below, we will explain a method for outputting endoscope operation information by the oropharyngeal operation assistance algorithm and an example of reporting endoscope operation information when the endoscope position determination information is "inappropriate." It is preferable to input the latest medical image into the oropharyngeal operation assistance algorithm (see Figure 4). Note that the same medical image as that input into the oropharyngeal position determination algorithm may be input into the oropharyngeal operation assistance algorithm.

[0173] The oropharyngeal manipulation assistance algorithm first outputs oropharyngeal region information. Specifically, when an oropharyngeal image 140 such as that shown in FIG. 35 is input, the oropharyngeal manipulation assistance algorithm outputs a glottis region 141 and an epiglottis region 142 as oropharyngeal region information, as shown in FIG. 37. The glottis region may be the region of the glottis Rg during breathing, or the region of the glottis Rg during phonation, as shown in FIG. 37. band It can be the area consisting of the folds Vof and the glottis Rg, or the left and right glottis Rg excluding the glottis Rg. band It may be a region that is a fold Vof.

[0174] The oropharyngeal manipulation assistance algorithm that outputs oropharyngeal region information is preferably a trained model that outputs the glottis and epiglottis regions by segmenting medical images. The trained model is generated by training oropharyngeal images in which the glottis and epiglottis regions have been classified in advance by a doctor as training images, which are correct images. The training model used to generate such a trained model is preferably PSPnet, but other training models suitable for segmentation, such as SPPnet and Segnet, may also be used. The oropharyngeal manipulation assistance algorithm may also be a trained model generated by training a training model that applies unsupervised learning.

[0175] Next, the oropharyngeal operation assistance algorithm uses the oropharyngeal region information to output oropharyngeal region calculation information. The oropharyngeal region calculation information is information used to output endoscope operation information that positions the tip 22d at an appropriate observation position. The appropriate observation position is information that is attached to an "appropriate" oropharyngeal image when used as a learning image.

[0176] The oropharynx region calculation information is coordinate information of the area, width and / or center position of the glottis region and the epiglottis region. The areas of the glottis and epiglottis regions are calculated by a method of calculating the area of ​​the regions classified into these regions by segmentation. The width of the glottis region or the width of the epiglottis region refers to the distance between any two points of the pixels that make up the glottis or epiglottis region. The width of the glottis region is calculated by the distance between the left and right glottis regions. band It may be the distance between the folds Vof, or the distance between the dorsal and ventral points of the glottis Rg region. The coordinate information of the center positions of the glottis region and the epiglottis region is the center of gravity or geometric center of these regions. The method for calculating the center positions of the glottis region and the epiglottis region is the same as the method for calculating the center position of the upper route insertion region in the upper route image of the nasal cavity, and therefore will not be described here.

[0177] The oropharyngeal operation support algorithm outputs endoscope operation information using oropharyngeal region calculation information. Specifically, a first threshold for glottis distant view observation is set for the area of ​​the glottis or epiglottis region, and if the area of ​​the glottis or epiglottis region is greater than the first threshold for glottis distant view observation, endoscope operation instruction information is output indicating a "forward" direction of movement of the tip portion 22d. Furthermore, a second threshold for glottis distant view observation is set for the area of ​​the glottis or epiglottis region, and if the area of ​​the glottis or epiglottis region is less than the second threshold for glottis distant view observation, endoscope operation instruction information is output indicating a "rearward" direction of movement of the tip portion 22d. The first threshold for glottis distant view observation and the second threshold for glottis distant view observation are values ​​that can be set arbitrarily, and may be values ​​set for the areas of the glottis region and the epiglottis region, respectively, or may be values ​​assigned to either one of them. It should be noted that the width of the glottis or epiglottis region may be used instead of the area of ​​the glottis or epiglottis region, and endoscope operation information may be output based on a combination of these.

[0178] Alternatively, the center positions of the glottis and epiglottis regions output using an oropharyngeal image input into the oropharyngeal manipulation assistance algorithm may be compared with the center positions of the glottis and epiglottis regions in an oropharyngeal image in which the position of tip 22d is "appropriate," and endoscope manipulation information may be output based on the difference. In this case, for example, the difference between the center positions of the glottis and epiglottis regions in an "inappropriate" oropharyngeal image and the center positions of the glottis and epiglottis regions in an "appropriate" oropharyngeal image may be calculated as a vector indicating the direction and amount of movement of tip 22d of endoscope 22, and this may be used as endoscope manipulation information.

[0179] In addition, a current position vector may be generated by connecting the center position of the glottis area and the center position of the epiglottis area output using the oropharyngeal image input into the oropharyngeal operation assistance algorithm, and an appropriate position vector may be generated by connecting the center position of the glottis area and the center position of the epiglottis area in the oropharyngeal image in which the position of tip 22d is ``appropriate,'' and the movement direction, movement distance, and rotation angle of tip 22d may be output as endoscope operation instruction information based on the difference between the current position vector and the appropriate position vector.

[0180] An example of how endoscope operation information is reported when the endoscope position determination information is "inappropriate" will be described below. For example, suppose that an oropharyngeal operation support algorithm receives an oropharyngeal image 140 as shown in Fig. 35 and outputs endoscope operation instruction information indicating "deeper" as the movement direction of tip portion 22d.

[0181] In this case, the notification control unit 70 generates a guide image 71 as shown in Fig. 38. In the example of the guide image 71 shown in Fig. 38, guide direction display icons 101a, 101b, 101c, and 101d indicating the up, down, left, and right directions are displayed around the oropharynx image 140. In addition, guide direction display icons 101e and 101f indicating the back and front directions are superimposed on the oropharynx image 140. In the guide image 71 shown in Fig. 38, the display manner (color) of the guide direction display icon 101f indicating the front direction is different from the other guide direction display icons 101a, 101b, 101c, 101d, and 101e, which is indicated by the presence or absence of a diagonal line, and indicates that the endoscope 22 is to be operated in the front direction. Note that the display manner of the guide direction display icons is not limited to this. For example, the guide image 71 may display a guide direction display icon 101g indicating a left turn and a guide direction display icon 101h indicating a right turn, as shown in FIG.

[0182] The guide image 71 also includes, as endoscope operation information, an endoscope operation support diagram showing the operation of the angle knob 22e for bending and moving the tip portion 22d or the direction of withdrawal or insertion of the endoscope 22. table It may be possible to display an image of the oropharynx (see FIG. 33, etc.) similar to the example of the nasopharyngeal image 130, and therefore the illustration is omitted. The above configuration can assist an inexperienced surgeon in observing the oropharynx by showing whether the endoscope is in an appropriate observation position for the oropharynx, and by providing instructions on how to operate the endoscope if the endoscope is in an inappropriate observation position. As a result, the burden on the examinee can be reduced.

[0183] Furthermore, the oropharyngeal operation support algorithm outputs subject body position information using oropharyngeal region calculation information. For example, when an oropharyngeal image 140 such as that shown in FIG. 35 is input, the oropharyngeal operation support algorithm outputs subject body position information such as "The head is bent forward. Check that the head is not extended." In this case, the notification control unit 70 generates a guide image 71 such as that shown in FIG. 39. In the example of the guide image 71 shown in FIG. 39, a message urging the subject to change their body position, "Subject's body position: Pay attention to the head extension position," is displayed in the observation support information display field 73. Furthermore, as audio instruction information, an audio message such as "The head may be bent forward. Pay attention to the head extension position," may be issued from a speaker.

[0184] The oropharyngeal manipulation support algorithm that outputs subject position information is preferably a trained model that has been trained using training images in which information about the subject's posture, such as "head bent forward," is added to an oropharyngeal image 140 such as that shown in Fig. 35. The oropharyngeal manipulation support algorithm that outputs endoscope operation information and the oropharyngeal manipulation support algorithm that outputs subject position information may be the same trained model or different trained models. When these are different trained models, the oropharyngeal manipulation support algorithm includes a first oropharyngeal manipulation support algorithm that outputs endoscope operation information and a second oropharyngeal manipulation support algorithm that outputs subject position information, and the latest oropharyngeal images are input to the first and second oropharyngeal manipulation support algorithms, respectively.

[0185] The function of the hypopharyngeal position determination algorithm, the observation support information output by the hypopharyngeal operation support algorithm, and the notification of the observation support information will be described below. The hypopharyngeal position determination algorithm is a position determination algorithm for glottal close-up observation that indicates whether the tip 22d is in an appropriate observation position in the hypopharynx. The hypopharyngeal operation support algorithm is an operation support algorithm that provides operation support by indicating how to operate the endoscope 22 so that the tip 22d is in an appropriate observation position based on the output result of the hypopharyngeal position determination algorithm.

[0186] FIG. 40 shows an example of a medical image (hereinafter referred to as a hypopharyngeal image) in which the observation target is the hypopharynx. The hypopharyngeal image 150 shown in FIG. 40 is a hypopharyngeal image in which the tip 22d is positioned at an appropriate observation position. An appropriate observation position in the hypopharynx is a position where the entire arytenoid region can be observed. Specifically, it is a position where the glottis Rg, left and right vocal folds Vof, left and right vestibular folds Vef, left and right aryepiglottic folds Af, left and right cuneiform tubercles Cut, left and right lesser tubercles Cot, left and right pyriform sinuses Ps, etc. can be clearly observed. In addition, from this position, the posterior pharyngeal wall Pw and the tracheal surface of the epiglottis Eg can be observed above and below the hypopharyngeal image. From this position, close-up observation of the arytenoid region during breathing, phonation, and swallowing can be performed, allowing evaluation of the movement of the arytenoid region, the presence or absence of organic abnormalities, and the presence or absence of accumulation of saliva, residual food, etc.

[0187] On the other hand, the hypopharynx image 150 shown in Fig. 41 is an example of the hypopharynx image 150 in which the position of the tip 22d is inappropriate. band In this image, the entire arytenoid folds Vof cannot be seen, and the entire arytenoid cannot be clearly observed. The orientation of the example oropharyngeal image 140 shown in Figures 40 and 41 is such that the upper side of the paper is the dorsal side, the lower side of the paper is the ventral side, the front side of the paper is the cranial side, and the back side of the paper is the caudal side.

[0188] The observation object identification algorithm as a trained model is trained by using hypopharynx images such as those shown in Figures 40 and 41 as training images for the "hypopharynx."

[0189] The following describes the process flow for outputting observation support information when a hypopharyngeal image is input to the hypopharyngeal observation support algorithm (the hypopharyngeal position determination algorithm and the hypopharyngeal operation support algorithm). First, the hypopharyngeal position determination algorithm to which a hypopharyngeal image such as that shown in Figures 40 and 41 is input outputs endoscope position determination information indicating whether the position of the tip 22d is "appropriate" or "inappropriate."

[0190] When a hypopharyngeal image such as that shown in Fig. 40 is input, the hypopharyngeal position determination algorithm outputs, as endoscope position determination information, that the position of tip portion 22d is "appropriate." On the other hand, when a hypopharyngeal image such as that shown in Fig. 41 is input, the hypopharyngeal position determination algorithm outputs, as endoscope position determination information, that the position of tip portion 22d is "inappropriate."

[0191] The hypopharyngeal position determination algorithm that outputs endoscope position determination information is preferably a trained model trained using training images including hypopharyngeal images associated with endoscope position determination information that is either "appropriate" or "inappropriate." When supervised learning or semi-supervised learning is applied to the training model, a hypopharyngeal image such as that shown in FIG. 40 may be used as the "appropriate" correct answer image. Convolutional neural networks such as VGGNet (e.g., VGG16, VGG19) and ResNet are suitable as training models used to generate the hypopharyngeal manipulation assistance algorithm as a trained model. The hypopharyngeal manipulation assistance algorithm may also be an algorithm that outputs endoscope position determination information by pattern matching with training images.

[0192] Next, the hypopharyngeal operation assistance algorithm outputs endoscope operation information based on the endoscope position determination information. If the endoscope position determination information is "appropriate," the hypopharyngeal operation assistance algorithm outputs endoscope operation information to "stop the endoscope 22." In this case, the notification control unit 70 generates a guide image 71 as shown in FIG. 42. In the example of the guide image 71 shown in FIG. 42, an observation assistance information display field 73, which is provided in a position different from the hypopharyngeal image 150, displays a message "appropriate: stop," indicating that the observation position is appropriate. Note that instead of a message, a mark indicating "appropriate" that instructs the tip 22d to stop may be displayed. For example, if the position is "appropriate," a green frame is displayed around the hypopharyngeal image 150.

[0193] Furthermore, when the endoscope position determination information of "appropriate" is output, the notification control unit 70 may issue a voice message from a speaker saying, "The position is appropriate. Please stay still." By providing such a guide image or voice notification, the surgeon can confirm that the tip portion 22d is in an appropriate observation position of the hypopharynx.

[0194] Below, we will explain a method for outputting endoscope operation information by the hypopharyngeal operation assistance algorithm and an example of reporting endoscope operation information when the endoscope position determination information is "inappropriate." It is preferable to input the latest medical image into the hypopharyngeal operation assistance algorithm (see Figure 4). Note that the same medical image as that input into the hypopharyngeal position determination algorithm may also be input into the hypopharyngeal operation assistance algorithm.

[0195] The hypopharyngeal manipulation assistance algorithm first outputs hypopharyngeal region information. When a hypopharyngeal image 150 such as that shown in Fig. 41 is input, the hypopharyngeal manipulation assistance algorithm outputs a glottis region 151 and left and right vocal fold regions 152a, 152b as hypopharyngeal region information, as shown in Fig. 43. As with the oropharynx, the glottis region 151 detected by the hypopharyngeal manipulation assistance algorithm is not limited to the example shown in Fig. 43.

[0196] The hypopharyngeal manipulation assistance algorithm that outputs hypopharyngeal region information is preferably a trained model that outputs the glottis and vocal fold regions by segmenting medical images. The trained model is generated by training hypopharyngeal images in which the glottis and vocal fold regions have been classified in advance by a doctor as training images, which are correct images. The left and right vocal fold regions may be classified into different classes or into the same class.

[0197] As a learning model to be applied to generate such a trained model, it is preferable to apply PSPnet, but other learning models suitable for segmentation such as SPPnet and Segnet may also be used. Note that the hypopharyngeal manipulation assistance algorithm may be a trained model generated by training a learning model to which unsupervised learning is applied.

[0198] Next, the hypopharyngeal operation assistance algorithm uses the hypopharyngeal region information to output hypopharyngeal region calculation information. The hypopharyngeal region calculation information is information used to output endoscope operation information that positions the tip 22d at an appropriate observation position. The appropriate observation position is information that is attached to an "appropriate" hypopharyngeal image when used as a learning image.

[0199] The hypopharyngeal region calculation information includes coordinate information of the area, width and / or center position of the glottis region, and voice The area of ​​the glottis region is calculated by calculating the area of ​​each of these regions classified by segmentation. The width of the glottis region refers to the distance between any two points contained in the pixels that make up the glottis region, and is the same as the width of the glottis region in the oropharynx, and is calculated by dividing the area of ​​the glottis region into two parts, left and right. band The distance between the folds Vof is not limited.

[0200] The length of the vocal fold region is the longest distance between any two points included in the pixels that make up the vocal fold region. For example, it is the distance (length) between the end of the dorsal (esophageal) vocal fold and the end of the ventral (epiglottis) vocal fold. The length of the vocal fold region may be calculated for both the left and right vocal fold regions, and the length of one of the vocal fold regions may be output as the hypopharynx region calculation information, or the length of both vocal fold regions may be output as the hypopharynx region calculation information.

[0201] The coordinate information of the center position of the glottis region is the center of gravity or geometric center of these regions. The method for calculating the center position of the glottis region is the same as the method for calculating the center position of the upper route insertion region in the upper route image of the nasal cavity, so a detailed description is omitted.

[0202] The hypopharyngeal operation assistance algorithm outputs endoscope operation information using hypopharyngeal region calculation information. Specifically, a first threshold for glottis close-up observation is set for the area of ​​the glottis region. If the area of ​​the glottis region is greater than the first threshold for glottis close-up observation, endoscope operation instruction information is output indicating a "progression" direction for the movement of the tip 22d. Furthermore, a second threshold for glottis close-up observation is set for the area of ​​the glottis region. If the area of ​​the glottis region is less than the second threshold for glottis close-up observation, endoscope operation instruction information is output indicating a "retraction" direction for the movement of the tip 22d. The first threshold for glottis close-up observation and the second threshold for glottis close-up observation are arbitrarily set values, and may be values ​​set separately for the area of ​​the glottis region and the area of ​​the vocal folds, or a value assigned to either one. Note that the width of the glottis region may be used instead of the area of ​​the glottis region, and endoscope operation information may be output based on a combination of these.

[0203] Furthermore, thresholds such as a first threshold for glottis close-up observation and a second threshold for glottis close-up observation may be set for the length of the vocal cords, and endoscope operation instruction information may be output according to the length of the vocal cords.

[0204] Alternatively, the center position of the glottis region output using the hypopharyngeal image input to the hypopharyngeal manipulation assistance algorithm may be compared with the center position of the glottis region in a hypopharyngeal image in which the position of the tip 22d is "appropriate," and endoscope manipulation information may be output based on the difference. In this case, for example, the difference between the center position of the glottis region in an "inappropriate" hypopharyngeal image and the center position of the glottis region in an "appropriate" hypopharyngeal image may be calculated as a vector indicating the movement direction, movement amount, and rotation angle of the tip 22d of the endoscope 22, and used as endoscope manipulation information.

[0205] An example of how endoscope operation information is reported when the endoscope position determination information is "inappropriate" will be described below. For example, suppose that a hypopharyngeal operation assistance algorithm, to which a hypopharyngeal image 150 as shown in Fig. 41 is input, outputs endoscope operation instruction information indicating "deeper" as the movement direction of the tip portion 22d.

[0206] In this case, the notification control unit 70 generates a guide image 71 as shown in Fig. 44. In the example of the guide image 71 shown in Fig. 44, guide direction display icons 101a, 101b, 101c, and 101d indicating the up, down, left, and right directions are displayed around the hypopharynx image 150. In addition, guide direction display icons 101e and 101f indicating the back and front directions are superimposed on the hypopharynx image 150. In the guide image 71 exemplified in Fig. 44, the guide direction display icon 101f indicating the front direction has a different display mode (color) from the other guide direction display icons 101a, 101b, 101c, 101d, and 101e, which is indicated by the presence or absence of a diagonal line, and indicates that the endoscope 22 is to be operated in the front direction. Note that the display mode of the guide direction display icons is not limited to this. For example, the guide image 71 may display a guide direction display icon 101g indicating a left turn and a guide direction display icon 101h indicating a right turn, as shown in FIG.

[0207] The guide image 71 also includes, as endoscope operation information, an endoscope operation support diagram showing the operation of the angle knob 22e for bending and moving the tip portion 22d or the direction of withdrawal or insertion of the endoscope 22. table The above configuration can assist in the observation of the hypopharynx by showing an inexperienced surgeon whether the endoscope is in an appropriate position for observing the hypopharynx, and by showing how to operate the endoscope if the endoscope is in an inappropriate position for observation. As a result, the burden on the examinee can be reduced.

[0208] The hypopharyngeal operation support algorithm also outputs subject position information. The method of reporting subject position information is similar to the method of reporting subject position information in the oropharynx example, and therefore will not be described here. The hypopharyngeal operation support algorithm is preferably a trained model that is trained using training images in which information about the subject's posture is added to hypopharyngeal images 150. The hypopharyngeal operation support algorithm that outputs endoscope operation information and the hypopharyngeal operation support algorithm that outputs subject position information may be the same trained model or different trained models. If these are different trained models, the system includes a first hypopharyngeal operation support algorithm that outputs endoscope operation information and a second hypopharyngeal operation support algorithm that outputs subject position information, and the latest hypopharyngeal images are input to the first hypopharyngeal operation support algorithm and the second hypopharyngeal operation support algorithm, respectively.

[0209] The observation support information and notification of the observation support information output by the incorrect insertion operation support algorithm will be described below. The incorrect insertion operation support algorithm is an operation support algorithm that provides operation support by issuing an operation instruction to pull out the endoscope 22 when the endoscope 22 is mistakenly inserted into the esophagus or trachea (see FIGS. 10 and 11).

[0210] Figure 45 shows an example of a medical image (esophagus image 160) in which the observation object is the esophagus when the endoscope 22 is erroneously inserted to the position shown in Figure 10. Figure 46 shows an example of a medical image (trachea image 170) in which the observation object is the trachea when the endoscope 22 is erroneously inserted to the position shown in Figure 11. The observation object identification algorithm as a trained model is trained by using an esophagus image such as that shown in Figure 45 as a training image for the "esophagus" and a trachea image such as that shown in Figure 46 as a training image for the "trachea."

[0211] The incorrect insertion operation assistance algorithm outputs endoscope operation information instructing the withdrawal of the endoscope when an esophageal image such as that shown in FIG. 45 or a tracheal image such as that shown in FIG. 46 is input.

[0212] An example of notification of endoscope operation information in this case will be described. For example, when an esophageal image such as that shown in FIG. 45 is input to the incorrect insertion operation assistance algorithm, the notification control unit 70 generates a guide image 71 such as that shown in FIG. 47. In the example of the guide image 71 shown in FIG. 47, guide direction display icons 101a, 101b, 101c, and 101d indicating the upper, lower, left, and right directions are displayed around the esophageal image 160. In addition, guide direction display icons 101e and 101f indicating the back and front directions are superimposed on the oropharyngeal image 140. In the guide image 71 illustrated in FIG. 47, the guide direction display icon 101f indicating the front direction has a different display mode (color) from the other guide direction display icons 101a, 101b, 101c, 101d, and 101e, as indicated by the presence or absence of a diagonal line, indicating that the endoscope 22 is to be operated in the front direction. Note that the display mode of the guide direction display icons is not limited to this.

[0213] 48, a warning display area 161 may be provided around the esophagus image 160 to notify the surgeon that the distal end 22d has reached the esophagus, urging the surgeon to withdraw the endoscope 22. In the example shown in FIG. 48, the message "Esophagus: Please withdraw the endoscope" is displayed. Furthermore, when the incorrect insertion operation assistance algorithm outputs endoscope operation information instructing the surgeon to withdraw the endoscope, the notification control unit 70 may control the speaker to emit a voice message saying "Please withdraw the endoscope" as voice instruction information. With the above configuration, when the image processing device 10 recognizes a region where the endoscope needs to be withdrawn immediately, it is possible to prompt the surgeon to withdraw the endoscope.

[0214] The following describes the notification of observation assistance information output by the algorithm for assisting operations outside of observation assistance targets. During a swallowing endoscopy, after a region requiring observation assistance has been identified, it may become impossible to identify the region requiring observation assistance due to the inflow of foreign matter such as food or saliva, or contraction of the pharynx caused by swallowing or coughing. In such cases, the algorithm for assisting observation outside of observation assistance targets outputs information to stop observation assistance.

[0215] When observation support stop information is output, the notification control unit 70 temporarily hides the endoscope operation information that was displayed on the guide image immediately before. For example, the guide direction display icons that were displayed immediately before are hidden. Also, the display mode of all guide direction display icons may be the same to indicate that the operation instructions have stopped. Also, a message "CAD OFF" may be displayed on the guide image to notify the surgeon that the operation support has been temporarily stopped (see FIG. 7).

[0216] When a medical image of the boundary region between the nasal cavity and nasopharynx, the nasopharynx and oropharynx, or the oropharynx and hypopharynx as the observation target is input to the observation target identification algorithm, the output may be set to "not subject to observation assistance." By excluding the boundary region from the observation support target and narrowing the range in which the oropharynx observation support algorithm or the hypopharyngeal observation support algorithm is selected as the specific observation support algorithm, it is possible to prevent the execution of processing that causes inconsistencies in image processing and to prevent the implementation of observation support that confuses the surgeon.

[0217] It is also preferable to be able to switch whether or not to notify the observation support information. For example, when a guide image on which a guide direction display icon is displayed is displayed, a notification switching instruction may be sent to the notification control unit 70 by operating the operation unit 22b, foot switch, etc. of the endoscope 22, so that the guide direction display icon of the guide image is hidden. Also, when the guide direction display icon of the guide image is hidden, the guide direction display icon may be displayed again by again operating the operation unit 22b, foot switch, etc. of the endoscope 22. Furthermore, a notification display switching button may be provided as a GUI (Graphical User Interface) on the guide image, so that the presence or absence of a notification of the observation support information can be switched. With this configuration for switching whether or not to notify the observation support information, a notification can be given only when the surgeon needs the notification.

[0218] A series of steps in the operating method of the image processing device 10 of this embodiment will be described using the flowchart in FIG. 49. First, the image acquisition unit 30 acquires a medical image from the modality 20 or the database 11 (step ST101). Next, the observation target identification unit 40 inputs the medical image into an observation target identification algorithm, thereby outputting observation target identification information (step ST102). Next, the observation support algorithm selection unit 50 selects one specific observation support algorithm from multiple observation support algorithms based on the observation target identification information (step ST103). Next, the observation support information generation unit 60 inputs the medical image into the specific observation support algorithm, thereby outputting observation support information (step ST104). Finally, the notification control unit 70 performs control to notify the user of the observation support information, such as generating a guide image, audio instruction information, etc. (step ST105). The observation support information is notified via a user interface such as a display or speaker.

[0219] In the above embodiment, the image processing device 10 has a hardware structure in which processing units that perform various processes, such as the image acquisition unit 30, observation target identification unit 40, observation support algorithm selection unit 50, observation support information generation unit 60, and notification control unit 70, are made up of various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, which is a processor with a circuit configuration designed specifically for performing various processes.

[0220] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which one processor is configured with a combination of one or more CPUs and software, as typified by client or server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a System on Chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0221] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit formed by combining circuit elements such as semiconductor elements, and the hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). [Explanation of symbols]

[0222] 10 Image processing device 11 Database 12 User Interface 20 Modalities 21 Endoscopy System 22 Endoscopy 22a Insertion part 22b Operation section 22c curved section 22d Tip 22e Angle Knob 22f Still image acquisition instruction switch 23 Processor unit 24 Light source device 30 Image acquisition unit 31, 32, 33, 72 Medical Imaging 40 Observation object identification unit 41 Observation target identification information 50 Observation support algorithm selection section 51 Specific observation support algorithm selection information 60 Observation support information generation unit 60a Nasal cavity observation support information generation unit 60b Nasopharyngeal observation support information generation unit 60c Oropharyngeal observation support information generation unit 60d Hypopharynx observation support information generation unit 60e Misinsertion observation support information generation unit 60f Observation support information generation unit for non-observation support 61 Observation Support Information 70 Notification control section 71 Guide Images 73 Observation support information display column 90 Upper Route 90a Upper Route Area 91 Lower Route 91a Lower Route Area 92 Upper root insertion area 93 Lower Route Insertion Area 94, 94a, 94b Insertion route guide marks 95 Image center guide mark 96 Guide frame 100 Upper Route Image 101a, 101b, 101c, 101d, 101e, 101f, 101g, 101h Guide direction display icon 102 Guide Arrow 103a Endoscope operation support diagram 103b, 103c lever 110 Lower Route Image 120 Frontal Nose Images 121 Upper route operation display 122 Lower route operation display column 130 Nasopharyngeal Images 140 Oropharynx Images 141, 151 glottic region 142 Epiglottis Region 150 Hypopharynx Images 152a, 152b vocal fold area 160 Esophageal Images 161 Warning display field 170 Tracheal Images Af aryepiglottic fold Cc cricoid cartilage Cot nodules Cut cuneiform tuberosity Eg epiglottis Ep nasopharynx Es esophagus Ev epiglottic vallecula F Food Fa face Fp Pharyngeal vault Hp hypopharynx It inferior nasal turbinate La larynx Lw side wall Mp oropharynx Mt middle nasal turbinate Nf Nasal Floor Ns nasal septum Nw nasal cavity side wall Ps piriform pit Pw Posterior wall of pharynx Pwe Posterior wall of the nasopharynx Rg glottic fissure Rl Tongue base Sp soft palate Tc thyroid cartilage Tr trachea Tongue Vc glottis Vef vestibular folds Vof vocal folds

Claims

1. A processor is provided, The processor: Acquire medical images; inputting the medical image into an observation object identification algorithm to output observation object identification information indicating a site that requires observation assistance included in the medical image or that the medical image is not subject to observation assistance; selecting one specific observation support algorithm from a plurality of observation support algorithms based on the observation target identification information; inputting the medical image into the specific observation support algorithm to output observation support information; performing control to notify the observation support information; the specific observation support algorithm includes an operation support algorithm, The processor: When the observation assistance target region output by the observation target identification algorithm is the nasal cavity, By inputting the medical image into the operation support algorithm, insertion area information indicating an upper route insertion area and / or a lower route insertion area included in the medical image, which is an area suitable for inserting an endoscope, is output; Calculating insertion route information, which is coordinate information of the area, width and / or center position of the upper route insertion area and / or the lower route insertion area, based on the insertion area information; an image processing device that performs control to display the insertion route information on a guide image as the observation support information;

2. The processor: The image processing apparatus according to claim 1 , wherein control is performed to display the upper route insertion area or the lower route insertion area on the guide image in a different display mode based on the insertion route information.

3. A processor is provided, The processor: Acquire medical images; inputting the medical image into an observation object identification algorithm to output observation object identification information indicating a site that requires observation assistance included in the medical image or that the medical image is not subject to observation assistance; selecting one specific observation support algorithm from a plurality of observation support algorithms based on the observation target identification information; inputting the medical image into the specific observation support algorithm to output observation support information; performing control to notify the observation support information; the specific observation support algorithm includes an operation support algorithm, The processor: When the observation assistance target region output by the observation target identification algorithm is the nasal cavity, By inputting the medical image into the operation support algorithm, insertion area information indicating an upper route insertion area and / or a lower route insertion area included in the medical image, which is an area suitable for inserting an endoscope, is output; Calculating insertion route information, which is coordinate information of the area, width and / or center position of the upper route insertion area and / or the lower route insertion area, based on the insertion area information; an image processing device that uses the insertion route information to output endoscope operation information that indicates a method of operating the endoscope;

4. A processor is provided, The processor: Acquire medical images; inputting the medical image into an observation object identification algorithm to output observation object identification information indicating a site that requires observation assistance included in the medical image or that the medical image is not subject to observation assistance; selecting one specific observation support algorithm from a plurality of observation support algorithms based on the observation target identification information; inputting the medical image into the specific observation support algorithm to output observation support information; performing control to notify the observation support information; the specific observation support algorithm includes an operation support algorithm, The processor: When the observation assistance target site output by the observation target identification algorithm is the nasopharynx, By inputting the medical image into the operation support algorithm, nasopharynx position information indicating an appropriate position or an inappropriate direction position such as a right inappropriate position, a left inappropriate position, an upper inappropriate position, a lower inappropriate position, a rear inappropriate position, or a front inappropriate position is output; An image processing device that outputs endoscope operation information indicating an operation method for the endoscope based on the nasopharynx position information.

5. The image processing device according to claim 4 , wherein the operation assistance algorithm is a trained model trained using training images including the medical image associated with the nasopharynx position information.

6. A processor is provided, The processor: Acquire medical images; inputting the medical image into an observation object identification algorithm to output observation object identification information indicating a site that requires observation assistance included in the medical image or that the medical image is not subject to observation assistance; selecting one specific observation support algorithm from a plurality of observation support algorithms based on the observation target identification information; inputting the medical image into the specific observation support algorithm to output observation support information; performing control to notify the observation support information; the specific observation support algorithm includes a position determination algorithm and an operation support algorithm; The processor: When the observation support target site output by the observation target identification algorithm is the oropharynx, and the position determination algorithm to which the medical image is input outputs "inappropriate" as endoscope position determination information indicating whether the position of the endoscope is appropriate or inappropriate, inputting the medical image into the operation support algorithm to output oropharyngeal region information included in the medical image and indicating a glottis region and / or an epiglottis region; calculating oropharyngeal region calculation information, which is coordinate information of the area, width, and / or center position of the glottis region and the epiglottis region, based on the oropharyngeal region information; An image processing device that uses the oropharyngeal region calculation information to output endoscope operation information indicating how to operate the endoscope or subject position information that prompts the subject to change or confirm their position.

7. A processor is provided, The processor: Acquire medical images; inputting the medical image into an observation object identification algorithm to output observation object identification information indicating a site that requires observation assistance included in the medical image or that the medical image is not subject to observation assistance; selecting one specific observation support algorithm from a plurality of observation support algorithms based on the observation target identification information; inputting the medical image into the specific observation support algorithm to output observation support information; performing control to notify the observation support information; the specific observation support algorithm includes a position determination algorithm; The processor: When the observation support target site output by the observation target identification algorithm is the oropharynx, and the position determination algorithm to which the medical image is input outputs endoscope position determination information indicating whether the position of the endoscope is appropriate or inappropriate, an image processing device that outputs the endoscope operation information for stopping the endoscope as endoscope operation information indicating a method of operating the endoscope;

8. The image processing device according to claim 6 , wherein the position determination algorithm is a trained model that is trained using training images in which the medical image and the endoscope position determination information are associated with each other.

9. A processor is provided, The processor: Acquire medical images; inputting the medical image into an observation object identification algorithm to output observation object identification information indicating a site that requires observation assistance included in the medical image or that the medical image is not subject to observation assistance; selecting one specific observation support algorithm from a plurality of observation support algorithms based on the observation target identification information; inputting the medical image into the specific observation support algorithm to output observation support information; performing control to notify the observation support information; the specific observation support algorithm includes a position determination algorithm and an operation support algorithm; The processor: When the observation support target site output by the observation target identification algorithm is the hypopharynx, and the position determination algorithm to which the medical image is input outputs "inappropriate" as endoscope position determination information indicating whether the position of the endoscope is appropriate or inappropriate, inputting the medical image into the operation support algorithm to output hypopharyngeal region information indicating the glottis region and / or the vocal fold region included in the medical image; Calculating hypopharyngeal region calculation information, which is the area, width, and / or coordinate information of the center position of the glottis region and the length of the vocal fold region, based on the hypopharyngeal region information; An image processing device that uses the hypopharyngeal region calculation information to output endoscope operation information indicating how to operate the endoscope or subject position information that prompts the subject to change or confirm their position.

10. A processor, The processor: Acquire medical images; inputting the medical image into an observation object identification algorithm to output observation object identification information indicating a site that requires observation assistance included in the medical image or that the medical image is not subject to observation assistance; selecting one specific observation support algorithm from a plurality of observation support algorithms based on the observation target identification information; inputting the medical image into the specific observation support algorithm to output observation support information; performing control to notify the observation support information; the specific observation support algorithm includes a position determination algorithm; The processor: When the observation support target site output by the observation target identification algorithm is the hypopharynx, and the position determination algorithm to which the medical image is input outputs endoscope position determination information indicating whether the position of the endoscope is appropriate or inappropriate, an image processing device that outputs the endoscope operation information for stopping the endoscope as endoscope operation information indicating a method of operating the endoscope;

11. The image processing device according to claim 9 , wherein the position determination algorithm is a trained model that is trained using training images in which the medical image and the endoscope position determination information are associated with each other.

12. A processor, The processor: Acquire medical images; inputting the medical image into an observation object identification algorithm to output observation object identification information indicating a site that requires observation assistance included in the medical image or that the medical image is not subject to observation assistance; selecting one specific observation support algorithm from a plurality of observation support algorithms based on the observation target identification information; inputting the medical image into the specific observation support algorithm to output observation support information; performing control to notify the observation support information; the specific observation support algorithm includes an operation support algorithm, The processor: When the observation assistance target region output by the observation target identification algorithm is the esophagus or the trachea, an image processing device that inputs the medical image into the operation support algorithm, and outputs the endoscope operation information indicating a method of operating the endoscope, the endoscope operation information instructing the withdrawal of the endoscope;

13. A processor, The processor: Acquire medical images; inputting the medical image into an observation object identification algorithm to output observation object identification information indicating a site that requires observation assistance included in the medical image or that the medical image is not subject to observation assistance; selecting one specific observation support algorithm from a plurality of observation support algorithms based on the observation target identification information; inputting the medical image into the specific observation support algorithm to output observation support information; performing control to notify the observation support information; the specific observation support algorithm includes an operation support algorithm, The processor: When the observation target identification algorithm outputs the observation target identification information as not being subject to observation support, an image processing device that outputs observation support stop information for stopping observation support by inputting the medical image into the operation support algorithm;

14. 14. The image processing device according to claim 13, wherein the observation object identification algorithm is a trained model that is trained to output, when the medical image includes a foreign object such as food or saliva, blur, blurring, or halation, as the observation object identification information, a value indicating that the medical image is not subject to observation support.

15. The image processing device according to claim 1 , wherein the notification of the observation support information is performed by a guide image that displays the observation support information and / or a sound that notifies the observation support information.

16. the endoscope operation information is a movement direction and / or a movement amount of a distal end portion of the endoscope, 13. The image processing device according to claim 3, wherein the movement direction is rightward, leftward, upward, downward, backward, forward, rightward turning, or leftward turning.

17. The image processing device according to claim 16 , wherein the processor performs control to display an endoscope operation support diagram on the guide image as the endoscope operation information.

18. The image processing device according to claim 6 , wherein the processor controls switching of the display of the endoscope operation information based on endoscope position determination information.

19. 15. The image processing device according to claim 1, wherein the observation object identification algorithm is a trained model trained using training images including medical images in which the observation object includes a nasal cavity, an nasopharynx, an oropharynx, a hypopharynx, a larynx, a trachea, or an esophagus.

20. The processor: The image processing device according to claim 1 , wherein control is performed to switch between whether or not to notify the observation support information.

21. acquiring a medical image; a step of inputting the medical image into an observation object identification algorithm to output observation object identification information indicating a site that requires observation assistance included in the medical image or that the medical image is not subject to observation assistance; selecting one specific observation support algorithm from a plurality of observation support algorithms based on the observation target identification information; outputting observation support information by inputting the medical image into the specific observation support algorithm; and performing control to notify the observation support information, the specific observation support algorithm includes an operation support algorithm, When the observation assistance target region output by the observation target identification algorithm is the nasal cavity, By inputting the medical image into the operation support algorithm, insertion area information indicating an upper route insertion area and / or a lower route insertion area included in the medical image, which is an area suitable for inserting an endoscope, is output; Calculating insertion route information, which is coordinate information of the area, width and / or center position of the upper route insertion area and / or the lower route insertion area, based on the insertion area information; An operating method of an image processing device that controls displaying the insertion route information on a guide image as the observation support information.

22. An image processing device according to any one of claims 1 to 14; a light source device that emits illumination light; an endoscope that captures the medical image.

Images