Medical assistance device, endoscope, medical assistance method, and program

Abstract

Provided is a medical assistance device provided with a processor that performs: a process for acquiring intestinal direction-related information relating to the intestinal direction of a duodenum on the basis of geometric characteristic information capable of specifying the geometric characteristics of the duodenum inserted into an endoscope observer; and outputting the related information of the intestinal direction.

Description

Technical Field

[0001] The technology of the present invention relates to a medical support device, an endoscope, a medical support method, and a program. Background Art

[0002] In Japanese Patent Laid-Open No. 2020-62218, a learning device is disclosed, which includes: an acquisition unit that acquires a plurality of pieces of information obtained by associating an image of the major duodenal papilla of the bile duct with information indicating a cannula insertion method as a method of inserting a catheter into the bile duct; a learning unit that machine-learns information indicating the cannula insertion method as training data based on the image of the major duodenal papilla of the bile duct; and a storage unit that associates and stores the result of machine learning performed by the learning unit with information indicating the cannula insertion method. Summary of the Invention

[0003] An embodiment of the technology of the present invention provides a medical support device, an endoscope, a medical support method, and a program that can enable a user to easily grasp how much the posture of an endoscope viewer is deviated from the intestinal direction of the duodenum.

[0004] Means for Solving the Technical Problem

[0005] The first aspect of the technology of the present invention is a medical support device, which includes a processor that performs the following processing: acquiring intestinal direction-related information related to the intestinal direction of the duodenum based on geometric characteristic information capable of determining the geometric characteristics of the duodenum into which the endoscope viewer is inserted; and outputting the intestinal direction-related information.

[0006] In the medical support device according to the second aspect of the technology of the present invention, which is based on the first aspect, the intestinal direction-related information includes offset amount information indicating the offset amount between the posture of the endoscope viewer and the intestinal direction.

[0007] In the medical support device according to the third aspect of the technology of the present invention, which is based on the first or second aspect, the geometric characteristic information includes an intestinal wall image obtained by photographing the intestinal wall of the duodenum using a camera provided in the endoscope viewer, and the processor acquires the intestinal direction-related information by performing a first image recognition process on the intestinal wall image.

[0008] In the medical support device according to the fourth aspect of the technology of the present invention, which is based on any one of the first to third aspects, outputting the intestinal direction-related information includes displaying the intestinal direction-related information on a first screen.

[0009] In the fifth aspect related to the technology of the present invention, in the medical support device related to any one of the first to fourth aspects, the intestinal direction-related information includes first direction information capable of determining a first direction that intersects the intestinal direction at a specified angle.

[0010] In the sixth aspect related to the technology of the present invention, in the medical support device related to the fifth aspect, the first direction information is obtained by performing a second image recognition process on an intestinal wall image obtained by photographing the intestinal wall of the duodenum using a camera provided in the endoscope viewer.

[0011] In the seventh aspect related to the technology of the present invention, in the medical support device related to the sixth aspect, the first direction information is information obtained with a reliability equal to or higher than a threshold value by performing an image recognition process in the AI method as the second image recognition process.

[0012] In the eighth aspect related to the technology of the present invention, in the medical support device related to any one of the first to eighth aspects, the processor acquires posture information capable of determining the posture of the endoscope viewer in a state where the endoscope viewer is inserted into the duodenum. The intestinal direction-related information includes posture adjustment support information for supporting the adjustment of the posture. The posture adjustment support information is information set according to the offset between the intestinal direction and the posture determined by the posture information.

[0013] In the ninth aspect related to the technology of the present invention, in the medical support device related to any one of the first to eighth aspects, the intestinal direction-related information includes condition information indicating a condition for making the optical axis direction of the camera provided in the endoscope viewer coincide with a second direction that intersects the intestinal direction at a specified angle by changing the posture of the endoscope viewer.

[0014] In the tenth aspect related to the technology of the present invention, in the medical support device related to the ninth aspect, the condition includes an operation condition related to an operation performed on the endoscope viewer to make the optical axis direction coincide with the second direction.

[0015] In the eleventh aspect related to the technology of the present invention, in the medical support device related to any one of the first to tenth aspects, when the optical axis direction of the camera provided in the endoscope viewer coincides with a third direction that intersects the intestinal direction at a specified angle, the intestinal direction-related information includes notification information for notifying the content that the optical axis direction coincides with the third direction.

[0016] In the 12th mode related to the technology of the present invention, in the medical support device related to any one of the 1st to 11th modes, the processor performs the following processing: detecting the duodenal papilla region by performing the 3rd image recognition processing on the intestinal wall image, the intestinal wall image being obtained by photographing the intestinal wall of the duodenum using a camera provided in the endoscope viewer; displaying the duodenal papilla region on the 2nd screen; and displaying the papilla orientation information indicating the orientation of the duodenal papilla region and obtained based on the intestinal tract direction-related information on the 2nd screen.

[0017] In the 13th mode related to the technology of the present invention, in the medical support device related to any one of the 1st to 12th modes, the processor performs the following processing: detecting the duodenal papilla region by performing the 4th image recognition processing on the intestinal wall image, the intestinal wall image being obtained by photographing the intestinal wall of the duodenum using a camera provided in the endoscope viewer; displaying the duodenal papilla region on the 3rd screen; and displaying the traveling direction information indicating the traveling direction of the tube leading to the opening of the duodenal papilla region and obtained based on the intestinal tract direction-related information on the 3rd screen.

[0018] In the 14th mode related to the technology of the present invention, in the medical support device related to the 13th mode, the tube is a bile duct or a pancreatic duct.

[0019] In the 15th mode related to the technology of the present invention, in the medical support device related to any one of the 1st to 14th modes, the geometric characteristic information includes depth information indicating the depth of the duodenum, and the intestinal tract direction-related information is obtained based on the depth information.

[0020] The 16th mode related to the technology of the present invention is an endoscope, which includes: the medical support device related to any one of the 1st to 15th modes; and an endoscope viewer.

[0021] The 17th mode related to the technology of the present invention is a medical support method, which includes the following steps: obtaining intestinal tract direction-related information related to the intestinal tract direction of the duodenum based on geometric characteristic information capable of determining the geometric characteristics of the duodenum inserted into the endoscope viewer; and outputting the intestinal tract direction-related information.

[0022] The 18th mode related to the technology of the present invention is a program for causing a computer to execute a process including the following steps: obtaining intestinal tract direction-related information related to the intestinal tract direction of the duodenum based on geometric characteristic information capable of determining the geometric characteristics of the duodenum inserted into the endoscope viewer; and outputting the intestinal tract direction-related information. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1It is a conceptual diagram showing an example of the way of using a duodenoscope system.

[0024] Figure 2 It is a conceptual diagram showing an example of the overall structure of a duodenoscope system.

[0025] Figure 3 It is a block diagram showing an example of the hardware structure of the electrical system of a duodenoscope system.

[0026] Figure 4 It is a conceptual diagram showing an example of the way of using a duodenoscope.

[0027] Figure 5 It is a block diagram showing an example of the hardware structure of the electrical system of an image processing device.

[0028] Figure 6 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope body, an image acquisition unit, an image recognition unit, and an export unit.

[0029] Figure 7 It is a conceptual diagram showing an example of the correlation among a display device, an image acquisition unit, an image recognition unit, an export unit, and a display control unit.

[0030] Figure 8 It is a flowchart showing an example of the process of medical support processing.

[0031] Figure 9 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope body, an image acquisition unit, an image recognition unit, and an export unit.

[0032] Figure 10 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope body, an image acquisition unit, an image recognition unit, and an export unit.

[0033] Figure 11 It is a conceptual diagram showing an example of the correlation among a display device, an image recognition unit, an export unit, and a display control unit.

[0034] Figure 12 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an export unit.

[0035] Figure 13 It is a conceptual diagram showing an example of the correlation among a display device, an image recognition unit, an export unit, and a display control unit.

[0036] Figure 14 It is a conceptual diagram showing an example of the correlation among a display device, an export unit, and a display control unit.

[0037] Figure 15It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope main body, an image acquisition unit, an image recognition unit, and an export unit.

[0038] Figure 16 It is a conceptual diagram showing an example of the correlation among a display device, an image recognition unit, an export unit, and a display control unit.

[0039] Figure 17 It is a conceptual diagram showing an example of the way to align the endoscope viewer with the papilla.

[0040] Figure 18 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope main body, an image acquisition unit, an image recognition unit, and an export unit.

[0041] Figure 19 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope main body, an image acquisition unit, an image recognition unit, and an export unit.

[0042] Figure 20 It is a conceptual diagram showing an example of the correlation among a display device, an image recognition unit, an export unit, and a display control unit.

[0043] Figure 21 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope main body, an image acquisition unit, an image recognition unit, and an export unit.

[0044] Figure 22 It is a conceptual diagram showing an example of the correlation among a display device, an image recognition unit, an export unit, and a display control unit.

[0045] Figure 23 It is a conceptual diagram showing an example of the correlation among a display device, an image recognition unit, an export unit, and a display control unit.

[0046] Figure 24 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an export unit.

[0047] Figure 25 It is a conceptual diagram showing an example of the correlation among a display device, an export unit, and a display control unit.

[0048] Figure 26 It is a flowchart showing an example of the process of medical support processing.

[0049] Figure 27 It is a conceptual diagram showing an example of the correlation among a display device, an export unit, and a display control unit.

[0050] Figure 28It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an export unit.

[0051] Figure 29 It is a conceptual diagram showing an example of the correlation among a display device, an export unit, and a display control unit.

[0052] Figure 30 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an export unit.

[0053] Figure 31 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an export unit.

[0054] Figure 32 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an export unit.

[0055] Figure 33 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an export unit.

[0056] Figure 34 It is a conceptual diagram showing an example of the correlation among a display device, an export unit, and a display control unit.

[0057] Figure 35 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an export unit.

[0058] Figure 36 It is a conceptual diagram showing an example of the correlation among a display device, an export unit, and a display control unit.

[0059] Figure 37 It is a conceptual diagram showing an example of the correlation among a display device, an export unit, and a display control unit. Detailed implementation manners

[0060] Hereinafter, an example of an implementation manner of a medical support device, an endoscope, a medical support method, and a program related to the technology of the present invention will be described with reference to the accompanying drawings.

[0061] First, the terms used in the following description will be explained.

[0062] The CPU is the abbreviation of "Central Processing Unit: Central Processor". The GPU is the abbreviation of "Graphics Processing Unit: Graphics Processor". The RAM is the abbreviation of "Random Access Memory: Random Access Memory". The NVM is the abbreviation of "Non-volatile memory: Non-volatile Memory". The EEPROM is the abbreviation of "Electrically Erasable Programmable Read-Only Memory: Electrically Erasable Programmable Read-Only Memory". The ASIC is the abbreviation of "Application Specific Integrated Circuit: Application Specific Integrated Circuit". The PLD is the abbreviation of "Programmable Logic Device: Programmable Logic Device". The FPGA is the abbreviation of "Field-Programmable Gate Array: Field Programmable Gate Array". The SoC is the abbreviation of "System-on-a-chip: System-on-a-chip". The SSD is the abbreviation of "Solid State Drive: Solid State Drive". The USB is the abbreviation of "Universal Serial Bus: Universal Serial Bus". The HDD is the abbreviation of "Hard Disk Drive: Hard Disk Drive". The EL is the abbreviation of "Electro-Luminescence: Electro-Luminescence". The CMOS is the abbreviation of "Complementary Metal Oxide Semiconductor: Complementary Metal Oxide Semiconductor". The CCD is the abbreviation of "Charge Coup]ed Device: Charge Coupled Device". The AI is the abbreviation of "Artificia] Intelligence: Artificial Intelligence". The BLI is the abbreviation of "Blue Light Imaging: Blue Light Imaging". The LCI is the abbreviation of "Linked Color Imaging: Linked Color Imaging". The I / F is the abbreviation of "Interface: Interface". The FIFO is the abbreviation of "First In First Out: First In First Out". The ERCP is the abbreviation of "Endoscopic Retrograde Cholangio-Pancreatography: Endoscopic Retrograde Cholangio-Pancreatography". The TOF is the abbreviation of "Time of Flight: Time of Flight".

[0063] <First Embodiment>

[0064] As an example, such as Figure 1As shown, the duodenoscope system 10 includes a duodenoscope 12 and a display device 13. The duodenoscope 12 is used by a doctor 14 during an endoscopic examination. The duodenoscope 12 is communicably connected to a communication device (not shown), and the information obtained by the duodenoscope 12 is sent to the communication device. The communication device receives the information sent from the duodenoscope 12 and performs processing (for example, processing recorded in an electronic medical record, etc.) using the received information.

[0065] The duodenoscope 12 includes an endoscope viewer 18. The duodenoscope 12 is a device for diagnosing and treating an observation object 21 (for example, the duodenum) contained in the body of a subject 20 (for example, a patient) using the endoscope viewer 18. The observation object 21 is the object observed by the doctor 14. The endoscope viewer 18 is inserted into the body of the subject 20. The duodenoscope 12 causes the endoscope viewer 18 inserted into the body of the subject 20 to photograph the observation object 21 in the body of the subject 20, and performs various medical treatments on the observation object 21 as needed. The duodenoscope 12 is an example of the "endoscope" related to the technology of the present invention.

[0066] The duodenoscope 12 obtains and outputs an image representing the internal morphology by photographing the inside of the body of the subject 20. In the present embodiment, the duodenoscope 12 is an endoscope having an optical imaging function, and the optical imaging function photographs the reflected light obtained by irradiating light in the body and reflected by the observation object 21.

[0067] The duodenoscope 12 includes a control device 22, a light source device 24, and an image processing device 25. The control device 22 and the light source device 24 are provided on a cart 34. A plurality of them are arranged in the vertical direction on the cart 34, and the image processing device 25, the control device 22, and the light source device 24 are arranged from the lower-stage cart to the upper-stage cart. And, a display device 13 is provided on the topmost stage of the cart 34.

[0068] The control device 22 is a device for controlling the entire duodenoscope 12. And, the image processing device 25 is a device for performing image processing on the images photographed by the duodenoscope 12 under the control of the control device 22.

[0069] The display device 13 displays various information including images (for example, images that have undergone image processing by the image processing device 25). As an example of the display device 13, a liquid crystal display or an EL display, etc. can be cited. And, instead of or together with the display device 13, a tablet terminal with a display can be used.

[0070] A plurality of screens are arranged and displayed on the display device 13. Figure 1In the example shown, screens 36, 37, and 38 are shown. An endoscopic image 40 obtained by the duodenoscope 12 is displayed on screen 36. An observation object 21 is reflected in the endoscopic image 40. The endoscopic image 40 is an image obtained by photographing the observation object 21 by a camera 48 (refer to Figure 2 ) provided in the endoscopic viewer 18 inside the body of the subject 20. As the observation object 21, the intestinal wall of the duodenum can be cited. Hereinafter, for the sake of convenience of explanation, an endoscopic image 40 obtained by photographing the intestinal wall of the duodenum as the observation object 21, that is, an intestinal wall image 41 will be described. In addition, the duodenum is merely an example, and any area that can be photographed by the duodenoscope 12 is acceptable. As an area that can be photographed by the duodenoscope 12, for example, the esophagus or the stomach can be cited. The intestinal wall image 41 is an example of the "intestinal wall image" and "geometric characteristic information" related to the technology of the present invention.

[0071] A moving image composed of multiple frames of the intestinal wall image 41 is displayed on screen 36. That is, on screen 36, multiple frames of the intestinal wall image 41 are displayed at a predetermined frame rate (for example, dozens of frames per second).

[0072] As an example, as shown in Figure 2 , the duodenoscope 12 includes an operation unit 42 and an insertion unit 44. The insertion unit 44 is locally bent by operating the operation unit 42. The insertion unit 44 is inserted while bending according to the shape of the observation object 21 (for example, the shape of the duodenum) in accordance with the operation of the doctor 14 on the operation unit 42.

[0073] A camera 48, a lighting device 50, a treatment opening 51, and an erecting mechanism 52 are provided at the front end portion 46 of the insertion unit 44. The camera 48 and the lighting device 50 are provided on the side surface of the front end portion 46. That is, the duodenoscope 12 becomes a side-viewing endoscope. Thereby, it is easy to observe the intestinal wall of the duodenum.

[0074] The camera 48 is a device that obtains the intestinal wall image 41 as a medical image by photographing inside the body of the subject 20. As an example of the camera 48, a CMOS camera can be cited. However, this is merely an example, and other types of cameras such as a CCD camera can also be used. The camera 48 is an example of the "camera" related to the technology of the present invention.

[0075] The lighting device 50 has a lighting window 50A. The lighting device 50 irradiates light through the lighting window 50A. As the types of light irradiated from the lighting device 50, for example, visible light (e.g., white light, etc.) and non-visible light (e.g., near-infrared light, etc.) can be cited. Further, the lighting device 50 irradiates special light through the lighting window 50A. As the special light, for example, BLI light and / or LCI light can be cited. The camera 48 optically photographs the inside of the subject 20 while the inside of the subject 20 is irradiated with light from the lighting device 50.

[0076] The treatment opening 51 serves as an instrument protruding port for protruding the treatment instrument 54 from the front end portion 46, a suction port for sucking blood, body dirt, etc., and a fluid delivery port for delivering fluid.

[0077] According to the operation of the doctor 14, the treatment instrument 54 protrudes from the treatment opening 51. The treatment instrument 54 is inserted into the insertion portion 44 through the treatment instrument insertion port 58. The treatment instrument 54 passes through the inside of the insertion portion 44 via the treatment instrument insertion port 58 and protrudes into the body of the subject 20 from the treatment opening 51. In Figure 2 In the example shown, as the treatment instrument 54, a cannula protrudes from the treatment opening 51. The cannula is merely an example of the treatment instrument 54. As another example of the treatment instrument 54, a papillotome or a snare, etc. can be cited.

[0078] The erecting mechanism 52 changes the protruding direction of the treatment instrument 54 protruding from the treatment opening 51. The erecting mechanism 52 includes a guide member 52A. By raising the guide member 52A with respect to the protruding direction of the treatment instrument 54, the protruding direction of the treatment instrument 54 is changed along the guide member 52A. Thereby, it becomes easy to make the treatment instrument 54 protrude toward the intestinal wall. In Figure 2 In the example shown, by the erecting mechanism 52, the protruding direction of the treatment instrument 54 is changed to a direction orthogonal to the advancing direction of the front end portion 46. The erecting mechanism 52 is operated by the doctor 14 via the operation portion 42. Thereby, the degree of change in the protruding direction of the treatment instrument 54 can be adjusted.

[0079] The endoscope viewer 18 is connected to the control device 22 and the light source device 24 via a general-purpose cord 60. A display device 13 and a receiving device 62 are connected to the control device 22. The receiving device 62 receives an instruction from a user (e.g., the doctor 14) and outputs the received instruction as an electric signal. In Figure 2 In the example shown, as an example of the receiving device 62, a keyboard can be cited. However, this is merely an example, and the receiving device 62 can also be a mouse, a touch panel, a foot switch, and / or a microphone, etc.

[0080] The control device 22 controls the entire duodenoscope 12. For example, the control device 22 controls the light source device 24, or transmits and receives various signals to and from the camera 48. The light source device 24 emits light under the control of the control device 22 and supplies light to the illumination device 50. A light guide is built into the illumination device 50, and the light supplied from the light source device 24 is irradiated from the illumination windows 50A and 50B through the light guide. The control device 22 causes the camera 48 to perform imaging, obtains the intestinal wall image 41 (refer to Figure 1 ) from the camera 48 and outputs it to a specified output destination (for example, the image processing device 25).

[0081] The image processing device 25 is communicably connected to the control device 22, and the image processing device 25 performs image processing on the intestinal wall image 41 output from the control device 22. Details of the image processing in the image processing device 25 will be described later. The image processing device 25 outputs the intestinal wall image 41 on which the image processing has been performed to a specified output destination (for example, the display device 13). In addition, here, an example of the method in which the intestinal wall image 41 output from the control device 22 is output to the display device 13 via the image processing device 25 has been described, but this is merely an example. It may also be a method in which the control device 22 is connected to the display device 13, and the intestinal wall image 41 on which the image processing has been performed by the image processing device 25 is displayed on the display device 13 via the control device 22.

[0082] As an example, as Figure 3 shown, the control device 22 includes a computer 64, a bus 66, and an external I / F 68. The computer 64 includes a processor 70, a RAM 72, and an NVM 74. The processor 70, the RAM 72, the NVM 74, and the external I / F 68 are connected to the bus 66.

[0083] For example, the processor 70 has a CPU and a GPU, and controls the entire control device 22. The GPU operates under the control of the CPU and undertakes various processes of the graphics system and operations using neural networks, etc. In addition, the processor 70 may be one or more CPUs integrated with GPU functions, or may be one or more CPUs without integrated GPU functions.

[0084] The RAM 72 is a memory that temporarily stores information and is used as a working memory by the processor 70. The NVM 74 is a non-volatile storage device that stores various programs and various parameters, etc. As an example of the NVM 74, a flash memory (for example, EEPROM and / or SSD) can be cited. In addition, the flash memory is merely an example, and it may be other non-volatile storage devices such as an HDD, or a combination of two or more non-volatile storage devices.

[0085] The external I / F 68 is responsible for the transmission and reception of various information between a device existing outside the control device 22 (hereinafter, also referred to as an "external device") and the processor 70. As an example of the external I / F 68, a USB interface can be cited.

[0086] Connected to the external I / F 68 is a camera 48 as one of the external devices. The external I / F 68 is responsible for the transmission and reception of various information between the camera 48 provided in the endoscope viewer 18 and the processor 70. The processor 70 controls the camera 48 via the external I / F 68. Also, the processor 70 acquires, via the external I / F 68, an intestinal wall image 41 obtained by photographing the inside of the subject 20 with the camera 48 provided in the endoscope viewer 18 (see Figure 1 ).

[0087] Connected to the external I / F 68 is a light source device 24 as one of the external devices. The external I / F 68 is responsible for the transmission and reception of various information between the light source device 24 and the processor 70. The light source device 24 supplies light to the illumination device 50 under the control of the processor 70. The illumination device 50 irradiates the light supplied from the light source device 24.

[0088] Connected to the external I / F 68 is a receiving device 62 as one of the external devices. The processor 70 acquires, via the external I / F 68, an instruction received by the receiving device 62 and executes processing corresponding to the acquired instruction.

[0089] Connected to the external I / F 68 is an image processing device 25 as one of the external devices. The processor 70 outputs the intestinal wall image 41 to the image processing device 25 via the external I / F 68.

[0090] In the treatment of the duodenum using an endoscope, a treatment called ERCP (endoscopic retrograde cholangiopancreatography) examination is sometimes performed. As an example, as Figure 4 shown, in the ERCP examination, for example, first the duodenoscope 12 is inserted into the duodenum J via the esophagus and the stomach. At this time, the insertion state of the duodenoscope 12 can be confirmed by X-ray imaging. Also, the distal end portion 46 of the duodenoscope 12 reaches near the duodenal papilla N (hereinafter, also simply referred to as "papilla N") existing on the intestinal wall of the duodenum J.

[0091] In an ERCP examination, for example, a cannula 54A is inserted from the papilla N. Here, the papilla N is a portion that protrudes from the intestinal wall of the duodenum J, and the openings of the ends of the bile duct T (e.g., the common bile duct, intrahepatic bile ducts, cystic duct) and the pancreatic duct S are present in the papillary elevation NA of the papilla N. Under the state of injecting a contrast agent into the bile duct T and the pancreatic duct S etc. via the cannula 54A from the opening of the papilla N, X-ray imaging is performed. Thus, in an ERCP examination, various surgical procedures including inserting a duodenoscope 12 into the duodenum J, confirming the position, orientation, and type of the papilla N, and further inserting a treatment instrument (e.g., a cannula) into the papilla N are included. Therefore, the doctor 14 needs to operate the duodenoscope 12 and observe the state of the target site according to each surgical procedure.

[0092] For example, in the case of inserting the duodenoscope 12 into the duodenum J, if the endoscope viewer 18 of the duodenoscope 12 is in a state of being inclined with respect to the intestinal direction, the papilla N will be visually recognized in an inclined state, so it is possible to misidentify the traveling directions of the bile duct T and the pancreatic duct S from the papilla N. Therefore, it is necessary to know to what extent the posture of the endoscope viewer 18 is inclined with respect to the intestinal direction within the duodenum J.

[0093] Therefore, in view of this situation, in order to support the implementation of medical treatment for the duodenum including ERCP examination, medical support processing is performed by the processor 82 of the image processing device 25.

[0094] As an example, as Figure 5 shown, the image processing device 25 includes a computer 76, an external I / F 78, and a bus 80. The computer 76 includes a processor 82, an NVM 84, and a RAM 81. The processor 82, the NVM 84, the RAM 81, and the external I / F 78 are connected to the bus 80. The computer 76 is an example of the "medical support device" and "computer" related to the technology of the present invention. The processor 82 is an example of the "processor" related to the technology of the present invention.

[0095] In addition, the hardware structure of the computer 76 (i.e., the processor 82, the NVM 84, and the RAM 81) is basically the same as that of the computer 64 shown in Figure 3 so the description related to the hardware structure of the computer 76 is omitted here. And the role of the external I / F 78 in the image processing device 25 for transceiver of information with the outside is basically the same as the role of the external I / F 68 in the control device 22 shown in Figure 3 so the description is omitted here.

[0096] A medical support program 84A is stored in the NVM 84. The medical support program 84A is an example of the "program" related to the technology of the present invention. The processor 82 reads the medical support program 84A from the NVM 84 and executes the read medical support program 84A on the RAM 81. The medical support process according to the present embodiment is realized by the processor 82 operating as an image acquisition unit 82A, an image recognition unit 82B, a derivation unit 82C, and a display control unit 82D according to the medical support program 84A executed on the RAM 81.

[0097] A learned model 84B is stored in the NVM 84. In the present embodiment, the image recognition unit 82B performs AI-based image recognition processing as object detection image recognition processing. The learned model 84B is optimized by performing machine learning on a neural network in advance.

[0098] As an example, as Figure 6 shown, the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48 in units of one frame, and the intestinal wall image 41 is generated by the camera 48 shooting at a shooting frame rate (for example, dozens of frames per second).

[0099] The image acquisition unit 82A holds the time-series image group 89. The time-series image group 89 is a plurality of intestinal wall images 41 showing the time series of the observation object 21. In the time-series image group 89, for example, it contains a specified number of frames (for example, a number of frames preset in the range of dozens to hundreds of frames) of intestinal wall images 41. Each time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0100] Here, an example of the method of holding and updating the time-series image group 89 by the image acquisition unit 82A is given, but this is only an example. For example, the time-series image group 89 can also be held and updated in a memory connected to the processor 82, such as the RAM 81.

[0101] The image recognition unit 82B performs image recognition processing on the time-series image group 89 using the learned model 84B. By performing the image recognition processing, the intestinal tract direction CD included in the observation object 21 is detected. Here, the intestinal tract direction CD refers to the lumen direction of the duodenum. Here, the detection of the intestinal tract direction means a process of storing in the memory in a state where the intestinal tract direction information 90 (for example, position coordinates indicating the direction in which the duodenum extends), which is information capable of determining the intestinal tract direction CD, is associated with the intestinal wall image 41. The intestinal tract direction information 90 is an example of the "intestinal tract direction-related information" related to the technology of the present invention.

[0102] The learned model 84B is obtained by optimizing a neural network through machine learning of the neural network using training data. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image equivalent to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be the subject of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, it is an annotation that can determine the intestinal tract direction CD.

[0103] Here, as an example of the annotation in the correct answer data, an annotation of the intestinal tract direction CD based on the fold shape of the intestinal tract shown in the intestinal wall image 41 can be cited (e.g., a line segment connecting the centers of the arcs of the fold shape is used as the annotation of the intestinal tract direction CD). And, as another annotation in the correct answer data, when the intestinal wall image 41 is a depth image, an annotation based on depth information can be cited (e.g., an annotation that sets the direction in which the depth in the depth direction represented by the depth information increases as the intestinal tract direction CD).

[0104] In addition, here, an example of the method in which only one learned model 84B is used by the image recognition unit 82B is given, but this is only one example. For example, it can also be set that the learned model 84B selected from a plurality of learned models 84B is used by the image recognition unit 82B. At this time, each learned model 84B is created by performing specific machine learning according to the surgical procedure of the ERCP examination (e.g., the position of the duodenoscope 12 relative to the papilla N, etc.), and it is only necessary to select the learned model 84B corresponding to the current surgical procedure of the ERCP examination and use it by the image recognition unit 82B.

[0105] The image recognition unit 82B inputs the intestinal wall image 41 obtained from the image acquisition unit 82A into the learned model 84B. Thereby, the learned model 84B outputs intestinal tract direction information 90 corresponding to the input intestinal wall image 41. The image recognition unit 82B acquires the intestinal tract direction information 90 output from the learned model 84B.

[0106] The derivation unit 82C derives the offset amount of the endoscope viewer 18 with respect to the intestinal tract direction CD (hereinafter, simply referred to as "offset amount"). Here, the offset amount refers to the degree of deviation between the posture of the endoscope viewer 18 and the intestinal tract direction CD. Specifically, the offset amount refers to the offset amount between the direction along the imaging surface of the imaging element of the camera 48 provided on the endoscope viewer 18 (e.g., the up and down directions in the viewing angle) and the intestinal tract direction CD. And, since the camera 48 is provided at the front end portion 46, the offset amount can also be said to be the angle between the longitudinal direction SD of the front end portion 46 (e.g., the central axis direction when the front end portion 46 is cylindrical) and the intestinal tract direction CD.

[0107] The derivation unit 82C acquires the intestinal tract direction information 90 from the image recognition unit 82B. Further, the derivation unit 82C acquires the posture information 91 from the optical fiber sensor 18A provided in the endoscope viewer 18. The posture information 91 is information indicating the posture of the endoscope viewer 18. The optical fiber sensor 18A is a sensor arranged along the longitudinal direction inside the endoscope viewer 18 (for example, the insertion portion 44 and the distal end portion 46). By using the optical fiber sensor 18A, the posture of the endoscope viewer 18 (for example, the inclination of the distal end portion 46 with respect to a reference position (for example, the straight state of the endoscope viewer 18)) can be detected. At this time, for example, a known posture detection technique of an endoscope such as Japanese Patent Publication No. 6797834 can be appropriately used. The posture information 91 is an example of the "posture information" related to the technology of the present invention.

[0108] Further, here, a posture detection technique using the optical fiber sensor 18A is given as an example, but this is merely an example. For example, a so-called electromagnetic navigation method may be used to detect the inclination of the distal end portion 46 of the endoscope viewer 18. At this time, for example, a known posture detection technique of an endoscope such as Japanese Patent Publication No. 6534193 can be appropriately used.

[0109] The derivation unit 82C uses the intestinal tract direction information 90 and the posture information 91 to derive information indicating an offset amount, that is, offset amount information 93. In Figure 6 the example shown, the angle A is shown as the offset amount information 93. The derivation unit 82C derives the offset amount using, for example, an offset calculation formula (not shown). The offset calculation formula is a calculation formula in which the position coordinates of the intestinal tract direction CD represented by the intestinal tract direction information 90 and the position coordinates of the longitudinal direction SD of the distal end portion 46 represented by the posture information 91 are used as independent variables, and the angle formed by the intestinal tract direction CD and the longitudinal direction SD of the distal end portion 46 is used as the dependent variable. The offset amount information 93 is an example of the "offset amount information" related to the technology of the present invention.

[0110] As an example, as Figure 7As shown, the display control unit 82D acquires the intestinal wall image 41 from the image acquisition unit 82A. Also, the display control unit 82D acquires the intestinal tract direction information 90 from the image recognition unit 82B. Further, the display control unit 82D acquires the offset information 93 from the derivation unit 82C. The display control unit 82D generates an operation instruction image 93A for aligning the longitudinal direction SD of the distal end portion 46 with the intestinal tract direction CD according to the offset represented by the offset information 93. The operation instruction image 93A is, for example, an arrow indicating the operation direction of the distal end portion 46 where the offset becomes smaller. The display control unit 82D generates a display image 94 including the intestinal wall image 41, the intestinal tract direction CD represented by the intestinal tract direction information 90, and the operation instruction image 93A, and outputs it to the display device 13. Specifically, the display control unit 82D controls the GUI (Graphical User Interface) for displaying the display image 94, causing the display device 13 to display the screen 36. The screen 36 is an example of the "first screen" related to the technology of the present invention. The operation instruction image 93A is an example of the "posture adjustment support information" related to the technology of the present invention.

[0111] In addition, here, an example of a method of enabling the user to grasp the offset by displaying the operation instruction image 93A on the screen 36 has been described, but the technology of the present invention is not limited thereto. For example, a message (not shown) indicating the operation content for reducing the offset can be displayed on the screen 36. As an example of the message, there can be cited "Please tilt the distal end portion of the duodenoscope 10 degrees toward the back side" and the like. It can also be notified to the user through a sound output device such as a speaker.

[0112] By visually recognizing the screen 36 of the display device 13, the user can grasp the intestinal tract direction CD. And by visually recognizing the operation instruction image 93A displayed on the screen 36, the user can grasp the operation for reducing the offset between the distal end portion 46 of the endoscope viewer 18 and the intestinal tract direction CD.

[0113] Next, with reference to Figure 8 , the operation of the duodenoscope system 10 related to the part of the technology of the present invention will be described.

[0114] In Figure 8 , an example of the process of the medical support process performed by the processor 82 is shown. Figure 8 The process of the medical support process shown is an example of the "medical support method" related to the technology of the present invention.

[0115] In Figure 8In the medical support process shown, first, in step ST10, the image acquisition unit 82A determines whether one frame of shooting has been performed by the camera 48 provided in the endoscope viewer 18. In step ST10, if one frame of shooting has not been performed by the camera 48, it is determined as negative, and the determination in step ST10 is performed again. In step ST10, if one frame of shooting has been performed by the camera 48, it is determined as positive, and the medical support process proceeds to step ST12.

[0116] In step ST12, the image acquisition unit 82A acquires one frame of intestinal wall image 41 from the camera 48 provided in the endoscope viewer 18. After performing the process of step ST12, the medical support process proceeds to step ST14.

[0117] In step ST14, the image recognition unit 82B detects the intestinal tract direction CD by performing AI-based image recognition processing on the intestinal wall image 41 acquired in step ST12 (i.e., image recognition processing using the learned model 84B). After performing the process of step ST14, the medical support process proceeds to step ST16.

[0118] In step ST16, the derivation unit 82C acquires the posture information 91 from the optical fiber sensor 18A of the endoscope viewer 18. After performing the process of step ST16, the medical support process proceeds to step ST18.

[0119] In step ST18, the derivation unit 82C derives an offset based on the intestinal tract direction CD obtained by the image recognition unit 82B in step ST14 and the posture information 91 acquired in step ST16. Specifically, the derivation unit 82C derives the angle between the intestinal tract direction CD and the longitudinal direction SD of the distal end portion 46 represented by the posture information 91. After performing the process of step ST18, the medical support process proceeds to step ST20.

[0120] In step ST20, the display control unit 82D generates a display image 94 in which the intestinal wall image 41 is superimposed with the intestinal tract direction CD and the operation instruction image 93A corresponding to the offset derived in step ST18. After performing the process of step ST20, the medical support process proceeds to step ST22.

[0121] In step ST22, the display control unit 82D outputs the display image 94 generated in step ST20 to the display device 13. After performing the process of step ST22, the medical support process proceeds to step ST24.

[0122] In step ST24, the display control unit 82D determines whether the condition for ending the medical support process is satisfied. As an example of the condition for ending the medical support process, a condition for giving an instruction to end the medical support process to the duodenoscope system 10 can be cited (for example, the condition that an instruction to end the medical support process is received by the receiving device 62).

[0123] In step ST24, when the condition for ending the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST10. In step ST24, when the condition for ending the medical support process is satisfied, the determination is positive, and the medical support process ends.

[0124] As described above, in the duodenoscope system 10 according to the first embodiment, in the image recognition unit 82B of the processor 82, an image recognition process is performed on the intestinal wall image 41, and as a result of the image recognition process, the intestinal tract direction CD in the intestinal wall image 41 is detected. Then, the intestinal tract direction information 90 indicating the intestinal tract direction CD is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes the intestinal tract direction CD superimposed on the intestinal wall image 41. Thus, the user can recognize the intestinal tract direction CD, and according to this configuration, it is possible to make it easy for the user to grasp how much the posture of the endoscope viewer 18 is deviated from the intestinal tract direction CD.

[0125] Moreover, in the duodenoscope system 10 according to the first embodiment, the deviation amount information 93 is derived in the derivation unit 82C. The deviation amount information 93 indicates the deviation amount between the posture of the endoscope viewer 18 and the intestinal tract direction CD. The deviation amount information 93 is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes a display based on the deviation amount information 93. Thus, the user can recognize the deviation amount between the posture of the endoscope viewer 18 and the intestinal tract direction CD, and according to this configuration, it is possible to make it easy for the user to grasp how much the posture of the endoscope viewer 18 is deviated from the intestinal tract direction CD.

[0126] Furthermore, in the duodenoscope system 10 according to the first embodiment, in the image recognition unit 82B, by performing an image recognition process on the intestinal wall image 41, the intestinal tract direction information 90 indicating the intestinal tract direction CD can be obtained. Thus, compared with the case where the user designates the intestinal tract direction CD for the intestinal wall image 41 by visual observation, the intestinal tract direction information 90 with higher accuracy can be obtained.

[0127] Moreover, in the duodenoscope system 10 according to the first embodiment, the intestinal tract direction information 90 is output to the display device 13 by the display control unit 82D. In the display device 13, the intestinal tract direction CD is displayed on the screen 36. Thus, it is possible for the user to easily grasp visually how much the posture of the endoscope viewer 18 is deviated from the intestinal tract direction CD.

[0128] Moreover, in the duodenoscope system 10 according to the first embodiment, the posture information 91 is obtained from the optical fiber sensor 18A by the derivation unit 82C, and the posture information 91 is information capable of determining the posture of the endoscope viewer 18. In the derivation unit 82C, the offset information 93 is generated based on the posture information 91 and the intestinal tract direction information 90. Further, in the display control unit 82D, the operation instruction image 93A indicating the operation direction for reducing the offset is generated based on the offset information 93. The display control unit 82D outputs the operation instruction image 93A to the display device 13, and in the display device 13, the operation instruction image 93A is superimposed and displayed on the intestinal wall image 41. Thus, in a state where the endoscope viewer 18 is inserted into the duodenum, it is easy to set the posture of the endoscope viewer 18 relative to the intestinal tract direction CD to a posture desired by the user. For example, by performing an operation of changing the posture of the endoscope viewer 18 in the direction shown in the operation instruction image 93A, the user can make the intestinal tract direction CD closer to the posture of the endoscope viewer 18.

[0129] In addition, in the above-described first embodiment, an example of a method of detecting the intestinal tract direction CD by image recognition processing based on the AI method has been described, but the technology of the present invention is not limited thereto. For example, the intestinal tract direction CD can be detected by image recognition processing based on the pattern matching method. In this case, for example, it may be the following method: detecting a region representing the folds of the intestinal tract (i.e., the fold region) included in the intestinal wall image 41, and inferring the intestinal tract direction based on the arc shape of the fold region (for example, inferring the line connecting the centers of the arcs as the intestinal tract direction).

[0130] (First modification example)

[0131] In the above-described first embodiment, an example of a method of detecting the intestinal tract direction CD using the intestinal wall image 41 not including depth information has been described, but the technology of the present invention is not limited thereto. In this first modification example, in the image recognition unit 82B, the derivation of the intestinal tract direction using the intestinal wall image 41 as a depth image is performed. As an example, as Figure 9As shown, the intestinal wall image 41 is a depth image having depth information 41A as pixel values. The depth information 41A is information representing the depth (i.e., the distance to the intestinal wall) of the duodenum as the subject. Regarding the depth of the duodenum, for example, it is obtained by distance measurement in a so-called TOF method using a distance measurement sensor mounted on the distal end portion 46. The image recognition unit 82B acquires the intestinal wall image 41 from the image acquisition unit 82A. The depth information 41A is an example of the "depth information" related to the technology of the present invention.

[0132] The image recognition unit 82B derives the intestinal tract direction information 90 based on the depth information 41A represented by the intestinal wall image 41. The image recognition unit 82B, for example, uses the intestinal tract direction calculation formula 82B1 to derive the intestinal tract direction information 90. The intestinal tract direction calculation formula 82B1 is, for example, a calculation formula in which the depth of the depth represented by the depth information 41A is the independent variable and the set of position coordinates of the axis representing the intestinal tract direction CD is the dependent variable. Thus, the intestinal tract direction information 90 is obtained based on the depth information 41A of the intestinal wall image 41.

[0133] As described above, in the duodenoscope system 10 according to the first modification example, the intestinal wall image 41 has the depth information 41A representing the depth of the duodenum, and the intestinal tract direction information 90 is obtained based on the depth information 41A. The intestinal tract direction CD is the direction along the depth direction in the lumen of the duodenum. And, the depth information 41A reflects the depth of the lumen of the duodenum. Therefore, the intestinal tract direction CD is derived based on the depth information 41A, and thus, compared with the case where the depth information 41A is not considered, the intestinal tract direction information 90 representing the intestinal tract direction CD with higher representation accuracy can be obtained.

[0134] (Second Modification Example)

[0135] In the above-described first embodiment, an example of a method of obtaining the intestinal tract direction CD by image recognition processing of the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In the second modification example, a direction that intersects the intestinal tract direction CD at a predetermined angle (hereinafter, also simply referred to as "predetermined direction") can be obtained.

[0136] As an example, as Figure 10 shown, each time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0137] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 into the learned model 84C. As a result, the learned model 84C outputs the vertical direction information 97 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the vertical direction information 97 output from the learned model 84C. Here, the vertical direction information 97 is information (for example, a set of position coordinates indicating an axis orthogonal to the intestinal tract direction CD) that can determine the direction VD (hereinafter, also simply referred to as "vertical direction VD") orthogonal to the intestinal tract direction CD.

[0138] In the image recognition process using the learned model 84C, the reliability of a specific result is calculated based on the result of determining the direction orthogonal to the intestinal tract direction CD. Here, the reliability is a statistical measure indicating the reliability of the specific result. The reliability is, for example, the score of the activation function (for example, softmax function, etc.) input to the output layer of the learned model 84C. The vertical direction information 97 output from the learned model 84C has a score equal to or higher than a threshold value (for example, 0.9 or higher).

[0139] In addition, in the present embodiment, "vertical" not only means completely vertical, but also means vertical in the sense including errors generally allowed in the technical field to which the technology of the present invention belongs and not violating the gist of the technology of the present invention. And here, as the specified angle with respect to the intestinal tract direction CD, the vertical direction with respect to the intestinal tract direction CD is cited, but the technology of the present invention is not limited thereto. For example, the specified angle can be 45 degrees, 60 degrees, or 80 degrees.

[0140] The learned model 84C is obtained by optimizing a neural network through machine learning using training data. The training data is a plurality of data (that is, multi-frame data) obtained by establishing a correspondence between example data and correct answer data. The example data is, for example, an image (for example, an image corresponding to the intestinal wall image 41) obtained by photographing a part (for example, the inner wall of the duodenum) that may be an object of ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, it is an annotation that can determine the vertical direction VD.

[0141] The derivation unit 82C derives the degree of coincidence between the specified direction and the direction of the optical axis of the camera 48. The coincidence between the specified direction and the direction of the optical axis means that the direction in which the camera 48 is oriented is the same as the direction preset by the user. That is, it means a state in which the front end 46 of the camera 48 is not in a direction that the user does not expect (for example, a direction inclined with respect to the intestinal tract direction CD).

[0142] Therefore, the derivation unit 82C acquires the vertical direction information 97. And the derivation unit 82C acquires the optical axis information 48A from the camera 48 of the endoscope viewer 18. The optical axis information 48A is information capable of determining the optical axis of the optical system of the camera 48. And the derivation unit 82C generates the consistency information 99 by comparing the direction represented by the vertical direction information 97 with the direction of the optical axis represented by the optical axis information 48A. The consistency information 99 is information indicating the degree of coincidence between the direction of the optical axis and a specified direction (for example, the angle formed by the direction of the optical axis and the specified direction). In addition, in the present embodiment, "coincidence" means not only complete coincidence, but also coincidence in the sense including errors generally allowed in the technical field to which the technology of the present invention belongs and not violating the gist of the technology of the present invention. The vertical direction information 97 is an example of the "first direction information" and the "intestinal direction-related information" related to the technology of the present invention.

[0143] Moreover, the derivation unit 82C determines whether the direction of the optical axis coincides with the specified direction. When the direction of the optical axis coincides with the specified direction, the derivation unit 82C generates the notification information 100. The notification information 100 is information for notifying the user of the content that the direction of the optical axis coincides with the specified direction (for example, text indicating that the direction of the optical axis coincides with the specified direction).

[0144] As an example, as Figure 11 shown, the display control unit 82D acquires the vertical direction information 97 from the image recognition unit 82B. And the display control unit 82D acquires the consistency information 99 from the derivation unit 82C. The display control unit 82D generates an operation instruction image 93B (for example, an arrow indicating the operation direction) for making the direction of the optical axis coincide with the specified direction according to the degree of coincidence between the direction of the optical axis represented by the consistency information 99 and the specified direction. And the display control unit 82D generates a display image 94 including the vertical direction VD represented by the vertical direction information 97, the operation instruction image 93B, and the intestinal wall image 41, and outputs it to the display device 13. In Figure 11 the example shown, on the display device 13, the intestinal wall image 41 with the vertical direction VD and the operation instruction image 93B superimposed and displayed is shown on the screen 36. The vertical direction VD is an example of the "first direction", the "second direction", and the "third direction" related to the technology of the present invention. The operation instruction image 93B is an example of the "condition information" related to the technology of the present invention.

[0145] And when the direction of the optical axis coincides with the specified direction, the derivation unit 82C outputs the notification information 100 to the display control unit 82D instead of the consistency information 99. At this time, the display control unit 82D generates a display image 94 including notifying the user of the content that the direction of the optical axis represented by the notification information 100 coincides with the specified direction instead of the operation instruction image 93B. In Figure 11In the example shown, an example is shown in the display device 13 where a message "The optical axis coincides with the vertical direction" is displayed on the screen 37. The notification information 100 is an example of the "notification information" related to the technology of the present invention.

[0146] In addition, here, an example of a method of displaying a message based on the notification information 100 in the display device 13 has been described, but this is merely an example. For example, marks such as circular marks based on the notification information 100 can be displayed. Also, it can be a method of outputting the notification information 100 to a sound output device such as a speaker instead of or together with the display device 13.

[0147] As described above, in the duodenoscope system 10 according to the second modification example, the vertical direction information 97, which is information capable of determining the direction orthogonal to the intestinal tract direction CD, is derived in the derivation unit 82C. The vertical direction information 97 is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes the vertical direction VD represented by the vertical direction information 97. Thus, the user can recognize the direction that intersects the intestinal tract direction CD at a specified angle.

[0148] Moreover, in the duodenoscope system 10 according to the second modification example, in the image recognition unit 82B, the vertical direction information 97 representing the vertical direction VD can be obtained by performing an image recognition process on the intestinal wall image 41. Thus, compared with the case where the user visually observes and designates the vertical direction VD for the intestinal wall image 41, the vertical direction information 97 with higher accuracy can be obtained.

[0149] Furthermore, in the duodenoscope system 10 according to the second modification example, in the image recognition process using the learned model 84C in the image recognition unit 82B, the vertical direction information 97 is obtained with a reliability above a threshold value. Thus, in the image recognition process using the learned model 84C in the image recognition unit 82B, compared with the case where no threshold value is set for the reliability, the vertical direction information 97 with higher accuracy can be obtained.

[0150] Also, in the duodenoscope system 10 according to the second modified example, the optical axis information 48A is obtained from the camera 48 by the extraction unit 82C. And in the extraction unit 82C, the coincidence information 99 is generated based on the optical axis information 48A and the vertical direction information 97. Further, in the display control unit 82D, a display image 94 is generated based on the coincidence information 99 and output to the display device 13. The display image 94 includes a display related to the degree of coincidence between the direction of the optical axis indicated by the coincidence information 99 and a specified direction. Thereby, the user can grasp the degree to which the optical axis of the camera 48 is deviated from the vertical direction VD. For example, when the optical axis coincides with the vertical direction VD, the camera 48 is highly likely to be facing the intestinal wall of the duodenum. By maintaining the posture of the endoscope viewer 18 in this state, it is easy to detect the papilla N existing on the intestinal wall of the duodenum, and further, it is also easy to make the camera 48 face the papilla N.

[0151] Also, in the duodenoscope system 10 according to the second modified example, in the display control unit 82D, an operation instruction image 93B for making the direction of the optical axis coincide with the specified direction is generated based on the coincidence information 99. The display control unit 82D outputs the operation instruction image 93B to the display device 13, and in the display device 13, the operation instruction image 93B is superimposed and displayed on the intestinal wall image 41. Thereby, the user can grasp the operation required to make the optical axis direction of the camera 48 coincide with the vertical direction VD.

[0152] Also, in the duodenoscope system 10 according to the second modified example, in the extraction unit 82C, it is determined whether the direction of the optical axis coincides with the specified direction. When the direction of the optical axis coincides with the specified direction, the extraction unit 82C generates a notification information 100. In the display control unit 82D, a display image 94 is generated based on the notification information 100 and output to the display device 13. The display image 94 includes a display of the content indicating that the direction of the optical axis coincides with the specified direction represented by the notification information 100. Thereby, the user can perceive that the direction of the optical axis coincides with the specified direction.

[0153] (Third Modified Example)

[0154] In the above first embodiment, an example of obtaining the intestinal tract direction CD by image recognition processing of the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In this third modified example, the traveling direction TD of the bile duct is obtained based on the intestinal tract direction CD.

[0155] As an example, as Figure 12 shown, every time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0156] The image recognition unit 82B performs nipple detection processing on the time-series image group 89 using the learned model 84D. The image recognition unit 82B obtains the time-series image group 89 from the image acquisition unit 82A, and inputs the obtained time-series image group 89 into the learned model 84D. As a result, the learned model 84D outputs nipple region information 95 corresponding to the input time-series image group 89. The image recognition unit 82B obtains the nipple region information 95 output from the learned model 84D. Here, the nipple region information 95 includes information (such as coordinates and ranges within the image) that can identify the nipple region N1 in the intestinal wall image 41 showing the nipple N.

[0157] The learned model 84D is obtained by optimizing a neural network through machine learning of the neural network using training data. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The training data is, for example, an image (such as an image corresponding to the intestinal wall image 41) obtained by photographing a part (such as the inner wall of the duodenum) that may be the subject of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation that can identify the nipple region N1 can be cited.

[0158] The derivation unit 82C derives travel direction information 96 as information representing the travel direction TD of the bile duct. The travel direction information 96 includes information (such as position coordinates indicating the direction of bile duct extension) that can determine the direction in which the bile duct extends. The derivation unit 82C obtains the nipple region information 95 from the image recognition unit 82B. And, the derivation unit 82C obtains the intestinal direction information 90 obtained through image recognition processing using the learned model 84B (refer to Figure 6 ). And, the derivation unit 82C derives the travel direction information 96 based on the intestinal direction information 90 and the nipple region information 95. The derivation unit 82C derives the travel direction TD, for example, according to a specified azimuth relationship between the intestinal direction CD and the travel direction TD. Specifically, when the intestinal direction CD is set to the 6 o'clock direction, the derivation unit 82C derives the travel direction TD as the 11 o'clock to 12 o'clock direction. Moreover, the derivation unit 82C uses the nipple region N1 represented by the nipple region information 95 as the starting point of the travel direction TD.

[0159] As an example, as Figure 13 shown, the display control unit 82D obtains the travel direction information 96 from the derivation unit 82C. And, the display control unit 82D obtains the nipple region information 95 from the image recognition unit 82B. The display control unit 82D generates an image for display on the display unit 81 by superimposing the travel direction information 96 and the nipple region information 95 on the intestinal wall image 41 obtained from the image acquisition unit 82A (refer to Figure 6)The obtained intestinal wall image 41 is superimposed with a display image 94 showing the traveling direction TD represented by the traveling direction information 96 and the nipple region N1 represented by the nipple region information 95, and is output to the display device 13. In the display device 13, the intestinal wall image 41 with the superimposed traveling direction TD is displayed on the screen 36.

[0160] As described above, in the duodenoscope system 10 according to the third modification, in the image recognition unit 82B, the nipple detection process is performed using the learned model 84D. The nipple region information 95 is obtained through the nipple detection process. Further, in the image recognition unit 82B, the intestinal tract direction information 90 can be obtained by performing the image recognition process using the learned model 84A. The derivation unit 82C derives the traveling direction information 96 based on the intestinal tract direction information 90 and the nipple region information 95. Then, the display control unit 82D outputs the display image 94 to the display device 13. The display image 94 includes the nipple region N1 represented by the nipple region information 95 and the traveling direction TD of the bile duct represented by the traveling direction information 96. In the display device 13, the nipple region N1 and the traveling direction TD of the bile duct are displayed on the screen 36. Thus, it is possible to make it easier for the user observing the nipple N through the screen 36 to visually grasp the traveling direction TD of the bile duct.

[0161] For example, in an ERCP examination, the camera 48 is sometimes directed at the nipple N. At this time, by utilizing the traveling direction of the bile duct or the pancreatic duct, it is easy to grasp the posture of the endoscope viewer 18. Also, when inserting a treatment instrument into the nipple N, by grasping the traveling direction of the bile duct or the pancreatic duct, it is easy to perform the intubation operation on the bile duct or the pancreatic duct within the nipple N.

[0162] (Fourth Modification)

[0163] In the above-described first embodiment, an example of obtaining the intestinal tract direction CD through the image recognition process of the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In the fourth modification, the orientation of the nipple bulge NA in the nipple N (hereinafter, also simply referred to as "nipple orientation ND") is obtained based on the intestinal tract direction CD.

[0164] In the image recognition unit 82B, the intestinal tract direction information 90 and the nipple region information 95 can be obtained by performing the image recognition process on the intestinal wall image 41 (see Figure 12 ). As an example, as Figure 14As shown, the derivation unit 82C generates nipple orientation information 102 based on intestinal tract direction information 90 and nipple region information 95. The nipple orientation information 102 is information capable of determining the nipple orientation ND (for example, the orientation in which the nipple bulge NA faces the treatment instrument). The nipple orientation ND is obtained, for example, as a tangent line at the nipple bulge NA in the traveling direction TD of the bile duct. Therefore, the derivation unit 82C derives the traveling direction TD of the bile duct based on the intestinal tract direction CD indicated by the intestinal tract direction information 90, and further derives the direction of the tangent line at the nipple bulge NA as the nipple orientation ND based on the traveling direction TD.

[0165] The display control unit 82D obtains the nipple orientation information 102 from the derivation unit 82C. The display control unit 82D generates a display image 94 in which the nipple orientation ND indicated by the nipple orientation information 102 and the nipple region N1 indicated by the nipple region information 95 are superimposed and displayed on the intestinal wall image 41 obtained from the image acquisition unit 82A (refer to Figure 6 ), and outputs it to the display device 13. In the display device 13, the intestinal wall image 41 with the nipple orientation ND superimposed and displayed is displayed on the screen 36.

[0166] In addition, here, an example of a method in which the nipple orientation ND is displayed as an arrow is given, but this is just an example. The nipple orientation ND can be a method of representing the direction by text.

[0167] As described above, in the duodenoscope system 10 according to the fourth modification example, in the image recognition unit 82B, nipple detection processing (refer to Figure 12 ) is performed to obtain the nipple region information 95. And, in the image recognition unit 82B, by using the learned model 84B (refer to Figure 6 ) for image recognition processing, the intestinal tract direction information 90 can be obtained. The derivation unit 82C derives the nipple orientation information 102 based on the intestinal tract direction information 90. And, the display control unit 82D outputs the display image 94 to the display device 13. The display image 94 includes the nipple region N1 indicated by the nipple region information 95 and the nipple orientation ND indicated by the nipple orientation information 102. In the display device 13, the nipple region N1 and the nipple orientation ND are displayed on the screen 36. Thus, it is possible to make it easy for the user observing the nipple N through the screen 36 to visually grasp the nipple orientation ND.

[0168] For example, in an ERCP examination, the camera 48 is sometimes directed at the nipple N. At this time, by using the nipple orientation ND, it is easy to grasp the posture of the endoscope viewer 18. And, when inserting the treatment instrument into the nipple N, by grasping the nipple orientation ND, it is possible to direct the treatment instrument at the nipple N and easily insert the treatment instrument into the nipple N.

[0169] <Second Embodiment>

[0170] In the above-described first embodiment, an example of obtaining the intestinal tract direction CD through image recognition processing of the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In the present second embodiment, the intestinal wall image 41 is an image obtained by photographing the intestinal wall including the papilla N, and the elevation direction RD of the papilla N is obtained through image recognition processing of the intestinal wall image 41.

[0171] For example, in ERCP examination, the camera 48 is sometimes directed at the elevation direction RD of the papilla N. Thereby, it is easy to infer the traveling directions of the bile duct T and the pancreatic duct S extending from the papilla N, or it is easy to insert a treatment instrument (for example, a cannula) into the papilla N. Therefore, in the present second embodiment, the elevation direction RD of the papilla N is obtained through image recognition processing of the intestinal wall image 41.

[0172] As an example, as Figure 15 shown, every time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0173] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 to the learned model 84E. Thereby, the learned model 84E outputs the elevation direction information 104 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the elevation direction information 104 output from the learned model 84E. Here, the elevation direction information 104 is information capable of determining the elevation direction of the papilla N (for example, a set of position coordinates of the axis representing the elevation direction RD).

[0174] The learned model 84E is obtained by optimizing a neural network through machine learning of the neural network using training data. The training data is a plurality of data (that is, multi-frame data) obtained by establishing a correspondence between example data and correct answer data. The training data is, for example, an image (for example, an image equivalent to the intestinal wall image 41) obtained by photographing a part (for example, the inner wall of the duodenum) that may be the subject of ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the elevation direction RD of the papilla N can be cited.

[0175] Here, the raising direction RD of the nipple N is determined, for example, as the direction from the top of the nipple prominence NA of the nipple N toward the top of the encircling fold H1. This is because, according to medical diagnostic opinions, the raising direction RD of the nipple N generally coincides with the direction from the top of the nipple prominence NA toward the top of the encircling fold H1. Here, in the nipple N, there are a plurality of folds (for example, folds H1 to H3) around the raised part. The encircling fold H1 is the fold closest to the nipple prominence NA. Therefore, as an example of the annotation in the correct solution data, an annotation can be cited in which the direction passing through the top of the encircling fold H1 is set as the raising direction RD.

[0176] The deriving unit 82C derives the degree of coincidence between the raising direction RD and the direction of the optical axis of the camera 48. The coincidence between the raising direction RD and the direction of the optical axis means that the direction toward which the camera 48 faces is directly opposite to the nipple N. That is, it means a state in which the front end portion 46 provided with the camera 48 is not in a direction that the user does not expect (for example, a direction in which the nipple N is inclined with respect to the raising direction RD).

[0177] Therefore, the deriving unit 82C obtains the raising direction information 104 from the image recognition unit 82B. And the deriving unit 82C obtains the optical axis information 48A from the camera 48 of the endoscope viewer 18. And the deriving unit 82C generates the coincidence information 103 by comparing the direction represented by the vertical direction information 97 with the direction of the optical axis represented by the optical axis information 48A. The coincidence information 103 is information indicating the degree of coincidence between the direction of the optical axis and the raising direction RD (for example, the angle formed by the direction of the optical axis and the raising direction RD).

[0178] As an example, as Figure 16 shown, the display control unit 82D obtains the raising direction information 104 from the image recognition unit 82B. And the display control unit 82D obtains the coincidence information 103 from the deriving unit 82C. The display control unit 82D generates an operation instruction image 93C (for example, an arrow indicating the operation direction) for making the direction of the optical axis coincide with the raising direction RD according to the degree of coincidence between the direction of the optical axis represented by the coincidence information 103 and the raising direction RD. And the display control unit 82D generates a display image 94 including the raising direction RD represented by the raising direction information 104, the operation instruction image 93C, and the intestinal wall image 41, and outputs it to the display device 13. In Figure 16 the example shown, on the display device 13, the intestinal wall image 41 with the raising direction RD and the operation instruction image 93C superimposed and displayed on the screen 36 is shown.

[0179] As an example, as Figure 17As shown, doctor 14 operates the endoscope viewer 18 to bring the optical axis of the camera 48 closer to the bulging direction RD. Thereby, the intestinal wall image 41 when the nipple N is facing the camera 48 can be obtained. Therefore, it is easy to infer the traveling directions of the bile duct T and the pancreatic duct S extending from the nipple N, or it is easy to insert a treatment instrument (for example, a cannula) into the nipple N.

[0180] As described above, in the duodenoscope system 10 according to the second embodiment, in the image recognition unit 82B of the processor 82, image recognition processing is performed on the intestinal wall image 41. As a result of the image recognition processing, the bulging direction RD of the nipple N in the intestinal wall image 41 is detected. And the bulging direction information 104 indicating the bulging direction RD is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes the bulging direction RD superimposed on the intestinal wall image 41. In this way, the bulging direction RD is displayed on the screen 36 in the display device 13. Thereby, the user observing the intestinal wall image 41 can visually grasp the bulging direction RD of the nipple N.

[0181] And, in the duodenoscope system 10 according to the second embodiment, in the image recognition unit 82B, the bulging direction information 104 is obtained based on the intestinal wall image 41. The bulging direction information 104 is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes a display based on the bulging direction information 104. Thereby, the user observing the intestinal wall image 41 can visually grasp the bulging direction RD of the nipple N.

[0182] And, in the duodenoscope system 10 according to the second embodiment, in the display control unit 82D, the display image 94 is generated. The display image 94 includes an image of an arrow indicating the bulging direction RD. Thereby, the bulging direction RD of the nipple N can be visualized and the user observing the intestinal wall image 41 can visually grasp it.

[0183] Also, in the duodenoscope system 10 according to the second embodiment, in the derivation unit 82C, the optical axis information 48A is obtained from the camera 48. Also, in the derivation unit 82C, the consistency information 103 is generated based on the optical axis information 48A and the bulge direction information 104. In the display control unit 82D, a display image 94 is generated based on the consistency information 103 and output to the display device 13. The display image 94 includes a display related to the degree of coincidence between the direction of the optical axis indicated by the consistency information 103 and the bulge direction RD. Thereby, the user observing the intestinal wall image 41 can visually grasp the degree of coincidence between the bulge direction RD of the papilla N and the optical axis direction. For example, when the optical axis coincides with the bulge direction RD, it is highly likely that the camera 48 is facing the papilla N directly. By maintaining the posture of the endoscope viewer 18 in this state, it is easy to observe the papilla N, and further, it is easy to insert the treatment instrument into the papilla N.

[0184] Also, in the duodenoscope system 10 according to the second embodiment, in the image recognition process in the image recognition unit 82B, the bulge direction RD is determined as the direction from the top of the papilla bulge NA of the papilla N toward the surrounding fold H1. And the display image 94 generated in the display control unit 82D is output to the display device 13. The bulge direction RD is included in the display image 94. Thereby, the user observing the intestinal wall image 41 can visually grasp the direction from the opening of the papilla bulge NA to the top of the surrounding fold H1. As a result, it is possible to easily determine the traveling direction TD of the bile duct leading to the opening of the papilla N.

[0185] Also, in the duodenoscope system 10 according to the second embodiment, in the image recognition process in the image recognition unit 82B, the bulge direction RD is determined as the direction from the top of the papilla bulge NA of the papilla N toward the surrounding fold H1. And the display image 94 generated in the display control unit 82D is output to the display device 13. An image of an arrow indicating the bulge direction RD is included in the display image 94. Thereby, the user observing the intestinal wall image 41 can visually grasp the direction from the opening of the papilla bulge NA to the top of the surrounding fold H1. As a result, it is possible to easily determine the traveling direction TD of the bile duct leading to the opening of the papilla N.

[0186] Also, in the duodenoscope system 10 according to the second embodiment, in the image recognition unit 82B, by performing an image recognition process on the intestinal wall image 41, the bulge direction information 104 indicating the bulge direction RD can be obtained. Thereby, compared with the case where the user designates the bulge direction RD for the intestinal wall image 41 by visual observation, the bulge direction information 104 with high accuracy can be obtained.

[0187] (Fifth Modification Example)

[0188] In the above-described second embodiment, an example of a method in which the elevation direction RD is determined to be the direction from the top of the nipple elevation NA toward the top of the circumferential fold H1 has been described, but the technology of the present invention is not limited thereto. The elevation direction RD is determined according to the patterns of the plurality of folds H1 to H3.

[0189] As an example, as Figure 18 shown, the image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 to the learned model 84E. Thereby, the learned model 84E outputs the elevation direction information 104 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the elevation direction information 104 output from the learned model 84E.

[0190] Here, the elevation direction RD of the nipple N is determined, for example, as the direction passing through the top of the circumferential fold H1. According to medical diagnosis opinions, the elevation direction RD of the nipple N sometimes coincides with the direction passing through the top of the circumferential fold H1. Therefore, as an example of the annotation in the correct answer data, an annotation in which the direction passing through the top of the circumferential fold H1 is set as the elevation direction RD can be cited.

[0191] In addition, here, an example of determining the elevation direction RD as the direction passing through the top of the circumferential fold H1 has been described, but this is merely an example. The elevation direction RD can be determined as the direction passing through at least one of the tops of the plurality of folds H1 to H3.

[0192] As described above, in the duodenoscope system 10 according to the fifth modification example, in the image recognition process in the image recognition unit 82B, the elevation direction RD is determined according to the patterns of the plurality of folds H1 to H3. And, the display image 94 generated in the display control unit 82D is output to the display device 13. The elevation direction RD is included in the display image 94. Thereby, the user observing the intestinal wall image 41 can visually recognize the direction passing through the top of the circumferential fold H1 of the nipple elevation NA as the elevation direction RD.

[0193] (Sixth Modification Example)

[0194] In the above-described second embodiment, an example of a method in which the elevation direction RD is determined to be the direction from the top of the nipple elevation NA toward the top of the circumferential fold H1 has been described, but the technology of the present invention is not limited thereto. In the sixth modification example, the elevation direction RD is determined according to the nipple elevation NA and the plurality of folds H1 to H3.

[0195] As an example, as Figure 19As shown, the image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 into the learned model 84E. Thereby, the learned model 84E outputs the bulge direction information 104 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the bulge direction information 104 output from the learned model 84E.

[0196] Here, the bulge direction RD of the nipple N is determined, for example, as the direction passing through the tops of the circumferential folds H1, H2, and H3 from the top of the nipple bulge NA. According to medical diagnostic opinions, the bulge direction RD of the nipple N sometimes coincides with the direction passing through the tops of the circumferential folds H1, H2, and H3 from the top of the nipple bulge NA. Therefore, as an example of the annotation in the correct answer data, an annotation can be cited in which the direction passing through the tops of the circumferential folds H1, H2, and H3 from the top of the nipple bulge NA is set as the bulge direction RD.

[0197] As described above, in the duodenoscope system 10 according to the sixth modification, in the image recognition process in the image recognition unit 82B, the bulge direction RD is determined based on the nipple bulge NA and the plurality of folds H1 to H3. And, the display image 94 generated in the display control unit 82D is output to the display device 13. The bulge direction RD is included in the display image 94. Thereby, the user observing the intestinal wall image 41 can visually grasp the direction passing through the top of the nipple bulge NA and the tops of the plurality of folds H1 to H3 as the bulge direction RD.

[0198] (Seventh Modification)

[0199] In the above-described second embodiment, an example of a method of obtaining the bulge direction RD by image recognition processing of the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In this seventh modification, the traveling direction TD of the bile duct is obtained based on the bulge direction RD.

[0200] In the image recognition unit 82B, by performing image recognition processing on the intestinal wall image 41, the bulge direction information 104 and the nipple region information 95 can be obtained (refer to Figure 12 and Figure 15 ). As an example, as Figure 20 shown, the derivation unit 82C derives the traveling direction information 96 based on the bulge direction information 104. The traveling direction TD of the bile duct has a predetermined azimuth relationship with the bulge direction RD of the nipple N. Specifically, when the bulge direction RD is set as the 12 o'clock direction, the derivation unit 82C derives the traveling direction TD as the 11 o'clock direction.

[0201] The display control unit 82D acquires the traveling direction information 96 from the derivation unit 82C. The display control unit 82D generates a display image 94 in which the traveling direction TD indicated by the traveling direction information 96 is superimposed on the intestinal wall image 41 acquired from the image acquisition unit 82A (reference Figure 6 ), and outputs it to the display device 13. In the display device 13, the intestinal wall image 41 with the traveling direction TD superimposed is displayed on the screen 36.

[0202] As described above, in the duodenoscope system 10 according to the seventh modification example, in the derivation unit 82C, the traveling direction information 96 is obtained based on the bulge direction information 104. Thus, since the traveling direction information 96 is obtained from the bulge direction information 104, it is easier to determine the traveling direction TD than in the case where the traveling direction information 96 is obtained by image recognition processing.

[0203] Moreover, in the duodenoscope system 10 according to the seventh modification example, in the display control unit 82D, the display image 94 is generated. The display image 94 includes an image indicating the traveling direction TD. Thereby, the user observing the intestinal wall image 41 can visually grasp the traveling direction TD of the bile duct.

[0204] (Eighth Modification Example)

[0205] In the above-described second embodiment, an example of a method of obtaining the bulge direction RD of the papilla N by image recognition processing of the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In the eighth modification example, by image recognition processing of the intestinal wall image 41, the direction MD of the surface in which the opening exists in the papilla N (hereinafter, also simply referred to as "surface direction MD") can be obtained.

[0206] As an example, as Figure 21 shown, each time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0207] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 to the learned model 84F. Thereby, the learned model 84F outputs the surface direction information 106 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the surface direction information 106 output from the learned model 84F. Here, the surface direction information 106 is information capable of determining the surface direction MD (for example, a set of position coordinates of an axis indicating the surface direction MD).

[0208] The learned model 84F is obtained by optimizing a neural network through machine learning using training data on the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image equivalent to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be the subject of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the surface direction MD can be cited.

[0209] The derivation unit 82C derives the relative angle between the surface P provided with the opening K of the nipple N and the posture of the endoscope viewer 18. That the relative angle between the surface P provided with the opening K of the nipple N and the posture of the endoscope viewer 18 is close to 0 means that it is close to the state where the camera 48 is directly facing the nipple N. Therefore, the derivation unit 82C acquires the surface direction information 106 from the image recognition unit 82B. And, the derivation unit 82C acquires the posture information 91 from the optical fiber sensor 18A of the endoscope viewer 18. And, the derivation unit 82C generates the relative angle information 108 by comparing the surface orientation of the surface having the opening K represented by the surface direction information 106 with the posture of the endoscope viewer 18 represented by the posture information 91. The relative angle information 108 is information representing the angle A formed by the surface P and the posture of the endoscope viewer 18 (e.g., the imaging surface of the camera 48).

[0210] As an example, as Figure 22 shown, the display control unit 82D acquires the surface direction information 106 from the image recognition unit 82B. And, the display control unit 82D acquires the relative angle information 108 from the derivation unit 82C. The display control unit 82D generates an operation instruction image 93D (e.g., an arrow indicating the operation direction) for making the camera 48 directly face the nipple N according to the angle represented by the relative angle information 108. And, the display control unit 82D generates a display image 94 including the surface direction MD represented by the surface direction information 106, the operation instruction image 93D, and the intestinal wall image 41, and outputs it to the display device 13. In Figure 22 the example shown, on the display device 13, the intestinal wall image 41 with the surface direction MD and the operation instruction image 93D superimposed and displayed is shown on the screen 36.

[0211] As described above, in the duodenoscope system 10 according to the eighth modification example, in the image recognition unit 82B of the processor 82, image recognition processing is performed on the intestinal wall image 41. As a result of the image recognition processing, the surface direction MD of the papilla N in the intestinal wall image 41 is detected. And the surface direction information 106 indicating the surface direction MD is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes the surface direction MD superimposed on the intestinal wall image 41. In this way, the surface direction MD is displayed on the screen 36 in the display device 13. Thereby, the user observing the intestinal wall image 41 can visually grasp the surface direction MD of the papilla N.

[0212] Moreover, in the duodenoscope system 10 according to the eighth modification example, the posture information 91 is obtained from the optical fiber sensor 18A by the derivation unit 82C, and the posture information 91 is information capable of determining the posture of the endoscope viewer 18. And in the derivation unit 82C, relative angle information 108 is generated based on the posture information 91 and the surface direction information 106. Further, in the display control unit 82D, an operation instruction image 93D for making the camera 48 face the papilla N directly is generated based on the relative angle information 108. The display control unit 82D outputs the operation instruction image 93D to the display device 13, and in the display device 13, the operation instruction image 93D is superimposed and displayed on the intestinal wall image 41. Thereby, in a state where the endoscope viewer 18 is inserted into the duodenum, it is easy to set the posture of the endoscope viewer 18 with respect to the surface direction MD of the papilla N to a posture desired by the user.

[0213] (Ninth Modification Example)

[0214] In the above-described second embodiment, an example of a method of displaying the elevation direction RD obtained by image recognition processing of the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In the ninth modification example, the papilla surface image 93E is displayed.

[0215] As an example, as Figure 23 shown, the display control unit 82D obtains the elevation direction information 104 from the image recognition unit 82B. The display control unit 82D generates the papilla surface image 93E based on the elevation direction RD indicated by the elevation direction information 104. The papilla surface image 93E is an image capable of determining a plane that intersects the elevation direction RD at a specified angle (for example, 90 degrees). Further, the display control unit 82D adjusts the papilla surface image 93E to a size and shape corresponding to the papilla region N1 according to the papilla region information 95 obtained in the image recognition unit 82B. And the display control unit 82D generates the operation instruction image 93C.

[0216] Further, the display control unit 82D generates a display image 94 including the nipple surface image 93E, the operation instruction image 93C, and the intestinal wall image 41, and outputs the same to the display device 13. In Figure 23 In the example shown, in the display device 13, an intestinal wall image 41 with the nipple surface image 93E and the operation instruction image 93C superimposed and displayed thereon is shown on the screen 36.

[0217] As described above, in the duodenoscope system 10 according to the ninth modification, in the display control unit 82D, the nipple surface image 93E is generated based on the bulge direction information 104. The display control unit 82D outputs the nipple surface image 93E to the display device 13, and in the display device 13, the nipple surface image 93E is superimposed and displayed on the intestinal wall image 41. Thereby, it is easy for a user observing the intestinal wall image 41 to visually predict the position of the opening included in the nipple N.

[0218] <Third Embodiment>

[0219] In the above-described first embodiment, an example of obtaining the intestinal tract direction CD through image recognition processing of the intestinal wall image 41 is given, and in the above-described second embodiment, an example of obtaining the bulge direction RD through image recognition processing of the intestinal wall image 41 is given, but the technology of the present invention is not limited thereto. In the third embodiment, the traveling direction TD of the bile duct T is obtained through image recognition processing of the intestinal wall image 41.

[0220] For example, in an ERCP examination, a treatment instrument (for example, a cannula) is sometimes inserted into the nipple N, and further, the treatment instrument is inserted into the bile duct T or the pancreatic duct S inside the nipple N. At this time, in the intestinal wall image 41, it is difficult to grasp the traveling direction of the bile duct T or the pancreatic duct S existing inside the nipple N. Therefore, in the third embodiment, the traveling direction of the bile duct T or the pancreatic duct S is obtained through image recognition processing of the intestinal wall image 41. Further, hereinafter, for the sake of convenience of explanation, the case of the bile duct T will be described as an example.

[0221] As an example, as Figure 24 shown, every time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0222] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 to the learned model 84G. Thereby, the learned model 84G outputs traveling direction information 96 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the traveling direction information 96 output from the learned model 84E.

[0223] The learned model 84G is obtained by optimizing a neural network through machine learning using training data on the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be an object of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the traveling direction TD can be cited.

[0224] Here, the traveling direction TD of the bile duct T is determined, for example, as the direction passing through the tops of a plurality of folds of the papilla N. This is because, according to medical diagnostic opinions, the traveling direction of the bile duct T sometimes coincides with the line connecting the tops of the folds. Therefore, as an example of the annotation in the correct answer data, an annotation that sets the direction passing through the tops of the folds of the papilla N as the traveling direction TD of the bile duct T can be cited.

[0225] Then, the acquired time-series image group 89 is input into the learned model 84H. As a result, the learned model 84H outputs diverticulum region information 110 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the diverticulum region information 110 output from the learned model 84H. The diverticulum region information 110 is information (coordinates indicating the size and position of the diverticulum) capable of determining the region representing the diverticulum existing in the papilla N. Here, the diverticulum is a region where a part of the papilla N protrudes outward in a sac shape into the duodenum.

[0226] The learned model 84H is obtained by optimizing a neural network through machine learning using training data on the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be an object of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the region representing the diverticulum can be cited

[0227] The derivation unit 82C derives a manner for displaying the traveling direction TD. For example, the traveling direction TD is determined in a manner that avoids the diverticulum. This is because, according to medical diagnostic opinions, the traveling direction TD sometimes forms while avoiding the diverticulum. Therefore, the derivation unit 82C changes the display manner of the traveling direction TD according to the diverticulum region information 110. Specifically, the derivation unit 82C changes the part that intersects with the diverticulum represented by the diverticulum region information 110 to a manner that avoids the diverticulum in the traveling direction TD represented by the traveling direction information 96. In this way, the derivation unit 82C generates display manner information 112 representing the changed display manner of the traveling direction TD.

[0228] As an example, as Figure 25 shown, the display control unit 82D acquires the display mode information 112 from the derivation unit 82C. The display control unit 82D generates a display image 94 including the changed traveling direction TD and the intestinal wall image 41 represented by the display mode information 112, and outputs it to the display device 13. In Figure 25 the example shown, in the display device 13, the intestinal wall image 41 with the changed traveling direction TD superimposed and displayed on the screen 36 is shown.

[0229] Next, with reference to Figure 26 , the operation of the duodenoscope system 10 related to the part involved in the technology of the present invention will be described.

[0230] In Figure 26 , an example of the process of the medical support process performed by the processor 82 is shown.

[0231] In Figure 26 the medical support process shown, first, in step ST110, the image acquisition unit 82A determines whether one frame of shooting has been performed by the camera 48 provided in the endoscope viewer 18. In step ST10, if one frame of shooting has not been performed by the camera 48, it is determined as negative, and the determination in step ST110 is performed again. In step ST110, if one frame of shooting has been performed by the camera 48, it is determined as positive, and the medical support process proceeds to step ST112.

[0232] In step ST112, the image acquisition unit 82A acquires one frame of the intestinal wall image 41 from the camera 48 provided in the endoscope viewer 18. After performing the process of step ST112, the medical support process proceeds to step ST114.

[0233] In step ST114, the image recognition unit 82B detects the traveling direction TD by performing AI-based image recognition processing (i.e., image recognition processing using the learned model 84G) on the intestinal wall image 41 acquired in step ST112. After performing the process of step ST114, the medical support process proceeds to step ST116.

[0234] In step ST116, the image recognition unit 82B detects the diverticulum area by performing AI-based image recognition processing (i.e., image recognition processing using the learned model 84H) on the intestinal wall image 41 acquired in step ST112. After performing the process of step ST116, the medical support process proceeds to step ST118.

[0235] In step ST118, the derivation unit 82C changes the display mode of the traveling direction TD based on the traveling direction TD obtained by the image recognition unit 82B in step ST114 and the diverticulum region obtained by the image recognition unit 82B in step ST116. Specifically, the derivation unit 82C changes the display mode of the traveling direction TD so as to avoid the diverticulum region. After performing the process of step ST118, the medical support process proceeds to step ST120.

[0236] In step ST120, the display control unit 82D generates a display image 94 in which the traveling direction TD whose display mode has been changed by the derivation unit 82C in step ST118 is superimposed on the intestinal wall image 41. After performing the process of step ST120, the medical support process proceeds to step ST122.

[0237] In step ST122, the display control unit 82D outputs the display image 94 generated in step ST120 to the display device 13. After performing the process of step ST122, the medical support process proceeds to step ST124.

[0238] In step ST124, the display control unit 82D determines whether the condition for ending the medical support process is satisfied. As an example of the condition for ending the medical support process, a condition for giving an instruction to end the medical support process to the duodenoscope system 10 can be cited (for example, the condition that the instruction to end the medical support process is received by the receiving device 62).

[0239] In step ST124, when the condition for ending the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST110. In step ST124, when the condition for ending the medical support process is satisfied, the determination is affirmative, and the medical support process ends.

[0240] As described above, in the duodenoscope system 10 according to the present third embodiment, in the image recognition unit 82B of the processor 82, an image recognition process is performed on the intestinal wall image 41, and as a result of the image recognition process, the traveling direction TD of the bile duct in the intestinal wall image 41 is detected. And the traveling direction information 96 indicating the traveling direction TD is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes the traveling direction TD superimposed on the intestinal wall image 41. Thus, the traveling direction TD is displayed on the screen 36 in the display device 13. Thereby, the user observing the intestinal wall image 41 can visually grasp the traveling direction TD of the bile duct.

[0241] Also, in the duodenoscope system 10 according to the third embodiment, in the image recognition unit 82B, image recognition processing is performed on the intestinal wall image 41 to obtain diverticulum region information 110. In the derivation unit 82C, display mode information 112 is generated based on the traveling direction information 96 and the diverticulum region information 110. And the display mode information 112 indicating the changed traveling direction TD is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes the changed traveling direction TD superimposed on the intestinal wall image 41. Thus, the changed traveling direction TD is displayed on the screen 36 in the display device 13. Thereby, the user observing the intestinal wall image 41 can visually grasp the traveling direction TD of the bile duct changed according to the presence of the diverticulum. For example, it is possible to suppress the occurrence of a situation where the user observing the intestinal wall image 41 visually misgrasps the traveling direction TD of the bile duct leading to the opening of the papilla N due to the presence of the diverticulum.

[0242] Also, in the duodenoscope system 10 according to the third embodiment, in the derivation unit 82C, the display mode information 112 indicates the traveling direction TD changed in a manner of avoiding the diverticulum in the traveling direction TD indicated by the traveling direction information 96. And the changed traveling direction TD is displayed on the screen 36 in the display device 13. Thereby, the user observing the intestinal wall image 41 can visually grasp the traveling direction TD of the bile duct changed in a manner of avoiding the diverticulum.

[0243] In addition, in the above third embodiment, as an example of the method of changing the display mode of the traveling direction TD of the bile duct, the method of avoiding the diverticulum is given, but the technology of the present invention is not limited thereto. For example, in the traveling direction TD of the bile duct, the area intersecting with the diverticulum can be set not to be displayed, or the area intersecting with the diverticulum can be set as a dotted line or semi-transparent.

[0244] Also, in the above third embodiment, an example of the method of detecting the diverticulum in the intestinal wall image 41 through image recognition processing and changing the display mode of the traveling direction TD according to the diverticulum is given, but the technology of the present invention is not limited thereto. For example, it may also be a method of not detecting the diverticulum.

[0245] (The tenth modification example)

[0246] In the above third embodiment, an example of the method of displaying the traveling direction TD of the bile duct while avoiding the diverticulum is given, but the technology of the present invention is not limited thereto. In the tenth modification example of the present invention, when the traveling direction TD of the bile duct intersects with the diverticulum, the user is notified of this content.

[0247] As an example, as Figure 27As shown, the derivation unit 82C obtains the traveling direction information 96 and the diverticulum region information 110 from the image recognition unit 82B. The derivation unit 82C determines the positional relationship between the diverticulum and the traveling direction TD based on the diverticulum region information 110 and the traveling direction information 96. Specifically, the derivation unit 82C compares the traveling direction TD represented by the traveling direction information 96 with the position and size of the diverticulum represented by the diverticulum region information 110 to determine whether the diverticulum intersects the traveling direction TD. Further, when the derivation unit 82C determines that the traveling direction TD and the diverticulum have an intersecting positional relationship, it generates the notification information 114.

[0248] The derivation unit 82C outputs the notification information 114 to the display control unit 82D. At this time, the display control unit 82D generates a display image 94 that includes content for notifying the user that the diverticulum intersects the traveling direction TD represented by the notification information 114. In Figure 27 the example shown, an example is shown in the display device 13 where a message "Diverticulum intersects the traveling direction" is displayed on the screen 37.

[0249] As described above, in the duodenoscope system 10 according to the present 10th modification example, in the derivation unit 82C, the positional relationship between the diverticulum and the traveling direction TD is determined based on the diverticulum region information 110 and the traveling direction information 96, and the notification information 114 is generated based on the determination result. In the display control unit 82D, the display image 94 is generated based on the notification information 114 and output to the display device 13. The display image 94 includes a display of the content that the diverticulum intersects the traveling direction represented by the notification information 114. Thereby, the user can be made aware that the diverticulum intersects the traveling direction. For example, it is possible to suppress a situation where a user observing the intestinal wall image 41 visually misgrasps the traveling direction TD of the bile duct leading to the opening of the papilla N due to the presence of the diverticulum.

[0250] <Fourth Embodiment>

[0251] In the above-described First Embodiment to the above-described Third Embodiment, examples of methods for determining information related to living tissues such as the intestinal tract direction CD, the papilla N, and the traveling direction TD of the bile duct by performing image recognition processing on the intestinal wall image 41 have been described, but the technology of the present invention is not limited thereto. In the present Fourth Embodiment, the relationship between the treatment instrument and the living tissue is determined by performing image recognition processing on the intestinal wall image 41.

[0252] For example, in an ERCP examination, various procedures using a treatment instrument are sometimes performed on the papilla N (for example, inserting a cannula into the papilla N). At this time, the positional relationship between the papilla N and the treatment instrument affects the success of the surgical procedure. For example, when the advancing direction of the treatment instrument is not consistent with the orientation ND of the papilla, the treatment instrument cannot properly enter the papilla N, making it difficult to succeed in the surgical procedure. Therefore, in the present fourth embodiment, the positional relationship between the treatment instrument and the papilla N is determined by image recognition processing of the intestinal wall image 41.

[0253] As an example, as Figure 28 shown, every time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0254] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 into the learned model 84I. As a result, the learned model 84I outputs the positional relationship information 116 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the positional relationship information 116 output from the learned model 84I. Here, the positional relationship information 116 is information capable of determining the position of the papilla N and the position of the treatment instrument (for example, the distance and angle between the position of the papilla N and the position of the tip of the treatment instrument).

[0255] The learned model 84I is obtained by optimizing a neural network through machine learning using training data. The training data is a plurality of data (that is, multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (for example, an image equivalent to the intestinal wall image 41) obtained by photographing a part (for example, the inner wall of the duodenum) that may be the subject of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the position of the papilla N and the position of the treatment instrument can be cited.

[0256] The derivation unit 82C acquires the positional relationship information 116 from the image recognition unit 82B. The derivation unit 82C generates notification information 118 based on the positional relationship information 116. The notification information 118 is information for notifying the user of the positional relationship between the papilla N and the treatment instrument. The derivation unit 82C compares the position of the treatment instrument indicated by the positional relationship information 116 with the position of the papilla N. And, when the position of the treatment instrument is the same as the position of the papilla N, the derivation unit 82C generates notification information 118 with the content that the position of the treatment instrument is the same as the position of the papilla N. And, when the position of the treatment instrument is not the same as the position of the papilla N, the derivation unit 82C generates notification information 118 with the content that the position of the treatment instrument is not the same as the position of the papilla N.

[0257] In addition, here, the case where the position of the treatment instrument coincides with the position of the nipple N has been described, but this is merely an example. For example, it can be determined whether the position of the treatment instrument and the position of the nipple N are within a preset range (for example, within a preset range of distance and angle).

[0258] As an example, as Figure 29 shown, the display control unit 82D acquires the notification information 118 from the derivation unit 82C. The derivation unit 82C outputs the notification information 118 to the display control unit 82D. At this time, the display control unit 82D generates a display image 94 including content for notifying the user of the positional relationship between the treatment instrument represented by the notification information 118 and the nipple N. In Figure 29 the example shown, an example in which a message "The position of the treatment instrument coincides with the position of the nipple" is shown on the screen 37 in the display device 13 is shown.

[0259] As described above, in the duodenoscope system 10 according to the fourth embodiment, in the image recognition unit 82B of the processor 82, image recognition processing is performed on the intestinal wall image 41 to determine the positional relationship between the treatment instrument and the nipple. In the derivation unit 82C, determination related to the positional relationship between the treatment instrument and the nipple N is performed based on the positional relationship information 116 indicating the positional relationship between the treatment instrument and the nipple, and notification information 118 is generated based on the determination result. In the display control unit 82D, a display image 94 is generated based on the notification information 118 and output to the display device 13. The display image 94 includes a display related to the positional relationship between the treatment instrument represented by the notification information 118 and the nipple N. Thus, the user observing the intestinal wall image 41 can be made to perceive what kind of relationship the position of the treatment instrument and the position of the nipple N have.

[0260] (11th modification example)

[0261] In the above-described fourth embodiment, an example of the positional relationship between the treatment instrument and the nipple N is given by determining the relationship between the position of the nipple N and the position of the treatment instrument, but the technology of the present invention is not limited thereto. In the 11th modification example of the present invention, the relationship between the advancing direction of the treatment instrument and the nipple orientation ND is determined.

[0262] As an example, as Figure 30 shown, the image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the learned model 84J. Thereby, the learned model 84J outputs the positional relationship information 116A corresponding to the input time-series image group 89. Here, the positional relationship information 116A is information capable of determining the nipple orientation ND and the advancing direction of the treatment instrument (for example, the angle formed by the nipple orientation ND and the advancing direction of the treatment instrument).

[0263] The learned model 84J is obtained by optimizing a neural network through machine learning using training data on the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be the subject of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation that can determine the relationship between the nipple orientation ND and the advancing direction of the treatment instrument can be cited.

[0264] The derivation unit 82C acquires the position relationship information 116A from the image recognition unit 82B. The derivation unit 82C generates a notification information 118 based on the position relationship information 116A, and the notification information 118 is information for notifying the user of the position relationship between the nipple N and the treatment instrument. When the angle formed by the nipple orientation ND and the advancing direction of the treatment instrument is within a preset range, the derivation unit 82C generates a notification information 118 with content indicating their consistency. Further, when the angle formed by the nipple orientation ND and the advancing direction of the treatment instrument exceeds the preset range, the derivation unit 82C generates a notification information 118 with content indicating their inconsistency.

[0265] As described above, in the duodenoscope system 10 according to the 11th modification example, in the image recognition unit 82B, the relationship between the advancing direction of the treatment instrument and the nipple orientation ND is determined. In the derivation unit 82C, the notification information 118 is generated based on the position relationship information 116A indicating the relationship between the advancing direction of the treatment instrument and the nipple orientation ND. Thereby, the user observing the intestinal wall image 41 can be made aware of what kind of relationship exists between the advancing direction of the treatment instrument and the nipple orientation ND.

[0266] In addition, in the above 11th modification example, a method example of determining the relationship between the advancing direction of the treatment instrument and the nipple orientation ND in the image recognition unit 82B is given, but the technology of the present invention is not limited thereto. For example, in the image recognition unit 82B, the relationship between the advancing direction of the treatment instrument and the nipple orientation ND, and the relationship between the position of the nipple N and the position of the treatment instrument can be determined. At this time, the position relationship information 116A is information indicating the relationship between the advancing direction of the treatment instrument and the nipple orientation ND and the relationship between the position of the nipple N and the position of the treatment instrument, and the derivation unit 82C makes a determination related to the relationship between the advancing direction of the treatment instrument and the nipple orientation ND and a determination related to the relationship between the position of the nipple N and the position of the treatment instrument based on the position relationship information 116A. Moreover, the derivation unit 82C generates the notification information 118 based on these determination results.

[0267] (12th modification example)

[0268] In the above-described fourth embodiment, an example of a method for determining the relationship between the position of the nipple N and the position of the treatment instrument was described as the positional relationship between the treatment instrument and the nipple N. However, the technology of the present invention is not limited to this. In this eleventh modification example, the relationship between the advancing direction of the treatment instrument and the traveling direction TD of the bile duct is determined.

[0269] As an example, as Figure 31 shown, the image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 to the learned model 84K. Thereby, the learned model 84K outputs the position relationship information 116B corresponding to the input time-series image group 89. Here, the position relationship information 116B is information capable of determining the relationship between the traveling direction TD of the bile duct and the advancing direction of the treatment instrument (for example, the angle formed by the direction of the tangent line of the opening end portion on the traveling direction TD of the bile duct (hereinafter, simply referred to as "bile duct tangent direction") and the advancing direction of the treatment instrument).

[0270] The learned model 84K is obtained by optimizing the neural network by performing machine learning on the neural network using training data. The training data is a plurality of data (that is, multi-frame data) obtained by establishing a correspondence between example data and correct answer data. The example data is, for example, an image (for example, an image corresponding to the intestinal wall image 41) obtained by photographing a part (for example, the inner wall of the duodenum) that may be an object of ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the relationship between the traveling direction TD of the bile duct and the advancing direction of the treatment instrument can be cited.

[0271] The derivation unit 82C acquires the position relationship information 116B from the image recognition unit 82B. The derivation unit 82C generates notification information 118 based on the position relationship information 116B. The notification information 118 is information for notifying the user of the relationship between the traveling direction TD of the bile duct and the advancing direction of the treatment instrument. When the angle formed by the bile duct tangent direction and the advancing direction of the treatment instrument is within a preset range, the derivation unit 82C generates notification information 118 with content indicating that the two are consistent. Further, when the angle formed by the bile duct tangent direction and the advancing direction of the treatment instrument exceeds the preset range, the derivation unit 82C generates notification information 118 with content indicating that the two are inconsistent.

[0272] As described above, in the duodenoscope system 10 according to the 12th modified example, in the image recognition unit 82B, the relationship between the advancing direction of the treatment instrument and the traveling direction TD of the bile duct is determined. In the derivation unit 82C, the notification information 118 is generated based on the positional relationship information 116B indicating the relationship between the advancing direction of the treatment instrument and the traveling direction TD of the bile duct. Thereby, the user observing the intestinal wall image 41 can be made to perceive what kind of relationship the advancing direction of the treatment instrument has with the traveling direction TD of the bile duct.

[0273] (13th modified example)

[0274] In the above-described 4th embodiment, the relationship between the position of the papilla N and the position of the treatment instrument is cited as an example of the positional relationship between the treatment instrument and the papilla N, but the technique of the present invention is not limited thereto. In the 13th modified example, the relationship between the advancing direction of the treatment instrument and the orientation of the plane perpendicular to the rising direction RD with respect to the papilla elevation NA (hereinafter, also simply referred to as "vertical plane orientation") is determined.

[0275] As an example, as Figure 32 shown, the image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 to the learned model 84L. Thereby, the learned model 84L outputs the positional relationship information 116C corresponding to the input time-series image group 89. Here, the positional relationship information 116C is information capable of determining the relationship between the vertical plane orientation and the advancing direction of the treatment instrument (for example, the angle formed by the vertical plane orientation and the advancing direction of the treatment instrument).

[0276] The learned model 84L is obtained by optimizing the neural network by performing machine learning on the neural network using training data. The training data is a plurality of data (that is, multi-frame data) obtained by establishing a correspondence between example data and correct answer data. The example data is, for example, an image (for example, an image corresponding to the intestinal wall image 41) obtained by photographing a part (for example, the inner wall of the duodenum) that may be an object of ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the relationship between the vertical plane orientation and the advancing direction of the treatment instrument can be cited.

[0277] The derivation unit 82C obtains the positional relationship information 116C from the image recognition unit 82B. The derivation unit 82C generates a notification information 118 based on the positional relationship information 116C. The notification information 118 is information for notifying the user of the relationship between the vertical plane orientation and the advancing direction of the treatment instrument. When the angle formed by the vertical plane orientation and the advancing direction of the treatment instrument is within a preset range, the derivation unit 82C generates the notification information 118 with the content that the two are consistent. Further, when the angle formed by the vertical plane orientation and the advancing direction of the treatment instrument exceeds the preset range, the derivation unit 82C generates the notification information 118 with the content that the two are inconsistent.

[0278] As described above, in the duodenoscope system 10 according to the 13th modification example, in the image recognition unit 82B, the relationship between the vertical plane orientation and the advancing direction of the treatment instrument is determined. In the derivation unit 82C, the notification information 118 is generated based on the positional relationship information 116C indicating the relationship between the vertical plane orientation and the advancing direction of the treatment instrument. Thereby, the user observing the intestinal wall image 41 can be made aware of what kind of relationship exists between the vertical plane orientation and the advancing direction of the treatment instrument.

[0279] (14th modification example)

[0280] In the above-described 4th embodiment, an example of a method of determining the positional relationship between the treatment instrument and the papilla N by performing image recognition processing on the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In the 14th modification example of the present invention, by performing image recognition processing on the intestinal wall image 41, an evaluation value related to the positional relationship between the treatment instrument and the papilla N is obtained.

[0281] As an example, as Figure 33 shown, each time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0282] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the learned model 84M. Thereby, the learned model 84M outputs evaluation value information 120 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the evaluation value information 120 output from the learned model 84M. Here, the evaluation value information 120 is information capable of determining an evaluation value related to the appropriate configuration of the papilla N and the treatment instrument (for example, the success degree of the surgical procedure determined based on the configuration of the papilla N and the treatment instrument). The evaluation value information 120 is, for example, a plurality of scores of an activation function (for example, a softmax function or the like) input to the output layer of the learned model 84M (scores for each success or failure of the surgical procedure).

[0283] The learned model 84M is obtained by optimizing a neural network through machine learning using training data on the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be an object of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation that can determine an evaluation value related to the appropriate configuration of the papilla N and the treatment instrument (e.g., an annotation indicating whether the surgical procedure is successful or not) can be cited.

[0284] Moreover, the image recognition unit 82B inputs the time-series image group 89 into the learned model 84N. Thereby, the learned model 84N outputs contact presence / absence information 122 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the contact presence / absence information 122 output from the learned model 84M. Here, the contact presence / absence information 122 is information that can determine whether or not the papilla N is in contact with the treatment instrument.

[0285] The learned model 84N is obtained by optimizing a neural network through machine learning using training data on the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be an object of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation that can determine whether or not the papilla N is in contact with the treatment instrument can be cited.

[0286] The derivation unit 82C acquires the contact presence / absence information 122 from the image recognition unit 82B. The derivation unit 82C determines whether or not contact between the treatment instrument and the papilla N is detected based on the contact presence / absence information 122. When contact between the treatment instrument and the papilla N is detected, the derivation unit 82C generates notification information 124 based on the evaluation value information 120. The notification information 124 is information for notifying the user of the success probability of the surgical procedure (e.g., text indicating the success probability of the surgical procedure).

[0287] As an example, as Figure 34 shown, the display control unit 82D acquires the notification information 124 from the derivation unit 82C. The derivation unit 82C outputs the notification information 124 to the display control unit 82D. At this time, the display control unit 82D generates a display image 94 including content for notifying the user of the success probability of the surgical procedure indicated by the notification information 124. In Figure 34In the example shown, an example is shown in the display device 13 where a message "Cannula insertion success probability: 90%" is displayed on the screen 37.

[0288] As described above, in the duodenoscope system 10 according to the 14th modification example, in the image recognition unit 82B of the processor 82, image recognition processing is performed on the intestinal wall image 41, and an evaluation value related to the arrangement of the treatment instrument and the papilla N is calculated. In the derivation unit 82C, notification information 124 is generated based on the evaluation value information 120 indicating the evaluation value. In the display control unit 82D, a display image 94 is generated based on the notification information 124 and output to the display device 13. The display image 94 includes a display related to the success probability of the surgical procedure represented by the notification information 124. Thereby, it is possible to notify the user observing the intestinal wall image 41 of the success probability of the surgical procedure using the treatment instrument. After the user grasps the success probability of the surgical procedure, the user can study whether to continue or change the operation, so that the success of the surgical procedure using the treatment instrument can be supported.

[0289] Furthermore, in the duodenoscope system 10 according to the 14th modification example, in the image recognition unit 82B, image recognition processing is performed on the intestinal wall image 41 to determine whether the treatment instrument is in contact with the papilla N. And in the derivation unit 82C, based on the contact presence / absence information 122, when the treatment instrument is in contact with the papilla N, notification information 124 is generated based on the evaluation value information 120. Thereby, it is possible to notify the user observing the intestinal wall image 41 of the success probability of the surgical procedure using the treatment instrument only in the required scenario. In other words, it is possible to support the surgical procedure on the papilla N using the treatment instrument at an appropriate timing.

[0290] <Fifth Embodiment>

[0291] In the above-described fourth embodiment, an example of determining the positional relationship between the treatment instrument and the papilla N by performing image recognition processing on the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In the fifth embodiment, when the treatment instrument is a cutting instrument, the cutting direction is obtained based on the result of the image recognition processing of the intestinal wall image 41.

[0292] For example, in ERCP examination, a cutting instrument (for example, a papillotome) is sometimes used as the treatment instrument. This is because by using the cutting instrument to cut the papilla N, it is easy to insert the treatment instrument into the papilla N or to remove foreign substances in the bile duct T or the pancreatic duct S. At this time, if the direction of cutting the papilla N using the cutting instrument (that is, the cutting direction) is erroneously selected, the surgical procedure may sometimes be difficult to succeed due to inadvertent bleeding or the like. Therefore, in the fifth embodiment, by performing image recognition processing on the intestinal wall image 41, the direction recommended as the cutting direction (that is, the cutting recommended direction) is determined.

[0293] As an example, as Figure 35 shown, every time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, it updates the time-series image group 89 in a FIFO manner.

[0294] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 to the learned model 84E. As a result, the learned model 84E outputs the bulge direction information 104 corresponding to the input time-series image group 89.

[0295] The derivation unit 82C acquires the bulge direction information 104 from the image recognition unit 82B. Moreover, the derivation unit 82C derives the incision recommendation direction information 126 based on the bulge direction information 104. The incision recommendation direction information 126 is information capable of determining the incision recommendation direction (for example, a set of position coordinates of the starting point and the ending point of the incision recommendation direction). The derivation unit 82C derives the incision recommendation direction according to a prescribed azimuth relationship between the bulge direction RD and the incision recommendation direction. Specifically, when the bulge direction RD is set to the 12 o'clock direction, the derivation unit 82C derives the incision recommendation direction as the 11 o'clock direction.

[0296] As an example, as Figure 36 shown, the display control unit 82D acquires the incision recommendation direction information 126 from the derivation unit 82C. The display control unit 82D generates an incision direction image 93F representing the incision direction according to the incision direction represented by the incision recommendation direction information 126. Moreover, the display control unit 82D generates a display image 94 including the incision direction image 93F and the intestinal wall image 41, and outputs it to the display device 13. In Figure 36 the example shown, on the display device 13, the intestinal wall image 41 with the incision direction image 93F superimposed and displayed on the screen 36 is shown.

[0297] As described above, in the duodenoscope system 10 according to the present 5th embodiment, the incision recommendation direction information 126 is generated in the derivation unit 82C. In the display control unit 82D, the display image 94 is generated based on the incision recommendation direction information 126 and output to the display device 13. The display image 94 includes the incision direction image 93F representing the incision recommendation direction represented by the incision recommendation direction information 126. As a result, the user observing the intestinal wall image 41 can grasp the incision recommendation direction. As a result, the success of the incision of the papilla N can be supported.

[0298] (15th Modification Example)

[0299] In addition, in the above-described fifth embodiment, an example of a method for determining the recommended cutting direction was described, but the technology of the present invention is not limited thereto. In this 15th modification example, a direction that is not recommended as a cutting direction (i.e., a non-recommended cutting direction) can be determined.

[0300] As an example, as Figure 37 shown, the derivation unit 82C derives non-recommended cutting direction information 127. The non-recommended cutting direction information 127 is information capable of determining the non-recommended cutting direction (for example, an angle indicating a direction other than the recommended cutting direction). The derivation unit 82C derives the recommended cutting direction based on a prescribed azimuth relationship between the bulge direction RD and the recommended cutting direction. Specifically, when the bulge direction RD is set to the 12 o'clock direction, the derivation unit 82C derives the recommended cutting direction as the 11 o'clock direction. And the derivation unit 82C determines a range other than a preset angular range (for example, a range of ±5 degrees centered on the recommended cutting direction) including the recommended cutting direction as the non-recommended cutting direction.

[0301] The display control unit 82D acquires the non-recommended cutting direction information 127 from the derivation unit 82C. The display control unit 82D generates an image representing the non-recommended cutting direction, i.e., the non-recommended cutting direction image 93G, based on the non-recommended cutting direction indicated by the non-recommended cutting direction information 127. And the display control unit 82D generates a display image 94 including the non-recommended cutting direction image 93G and the intestinal wall image 41, and outputs it to the display device 13. In Figure 37 the example shown, on the display device 13, the intestinal wall image 41 with the non-recommended cutting direction image 93G superimposed and displayed on the screen 36 is shown.

[0302] As described above, in the duodenoscope system 10 according to this 15th modification example, the non-recommended cutting direction information 127 is generated in the derivation unit 82C. In the display control unit 82D, the display image 94 is generated based on the non-recommended cutting direction information 127 and output to the display device 13. The display image 94 includes the non-recommended cutting direction image 93G representing the non-recommended cutting direction indicated by the non-recommended cutting direction information 127. Thereby, the user observing the intestinal wall image 41 can grasp the non-recommended cutting direction. As a result, it is possible to support the success of cutting the papilla N.

[0303] In addition, in each of the above-described embodiments, as a method of displaying the operation direction to the user, an example of a method in which an image of an arrow indicating the operation direction is displayed on the screen 36 has been described. However, the technology of the present invention is not limited thereto. For example, the image for displaying the operation direction to the user may be a triangular image indicating the operation direction. Also, it may be a method of displaying a message indicating the operation direction instead of or together with the image indicating the operation direction. Moreover, the image indicating the operation direction may be displayed on another window or another display device instead of on the screen 36.

[0304] In addition, in each of the above-described embodiments, an example of a method of showing the bile duct direction TD has been described. However, the technology of the present invention is not limited thereto. It may also be a method of showing the traveling direction of the pancreatic duct S instead of or together with the bile duct direction TD.

[0305] In addition, in each of the above-described embodiments, an example of a method of outputting various information to the display device 13 has been described. However, the technology of the present invention is not limited thereto. For example, it may be output to a sound output device such as a speaker (not shown) instead of or together with the display device 13, or it may be output to a printing device such as a printer (not shown).

[0306] In each of the above-described embodiments, an example of a method in which various information is output to the display device 13 and these information are displayed on the screen 36 of the display device 13 has been described. However, the technology of the present invention is not limited thereto. The various information may also be output to an electronic medical record server. The electronic medical record server is a server for storing electronic medical record information indicating the diagnosis and treatment results for a patient. The electronic medical record information includes various information.

[0307] The electronic medical record server is connected to the duodenoscope system 10 via a network. The electronic medical record server acquires the intestinal wall image 41 and various information from the duodenoscope system 10. The electronic medical record server stores the intestinal wall image 41 and various information as part of the diagnosis and treatment results represented by the electronic medical record information.

[0308] The electronic medical record server is also connected to terminals other than the duodenoscope system 10 (for example, a personal computer installed in a medical facility) via a network. A user such as the doctor 14 can acquire the intestinal wall image 41 and various information stored in the electronic medical record server via the terminal. In this way, by storing the intestinal wall image 41 and various information in the electronic medical record server, the user can acquire the intestinal wall image 41 and various information.

[0309] In addition, in each of the above-described embodiments, an example of performing AI-based image recognition processing on the intestinal wall image 41 has been described. However, the technology of the present invention is not limited thereto. For example, pattern matching-based image recognition processing may be performed.

[0310] In the above-described embodiment, an example of a mode in which the medical support process is performed by the processor 82 of the computer 76 included in the image processing device 25 has been described, but the technology of the present invention is not limited thereto. For example, the medical support process may be performed by the processor 70 of the computer 64 included in the control device 22. Further, the device that performs the medical support process may be provided outside the duodenoscope 12. As a device provided outside the duodenoscope 12, for example, at least one server and / or at least one personal computer communicably connected to the duodenoscope 12 can be cited. Further, it may be configured such that the medical support process is distributedly performed by a plurality of devices.

[0311] In the above-described embodiment, an example of a mode in which the medical support program 84A is stored in the NVM 84 has been described, but the technology of the present invention is not limited thereto. For example, the medical support program 84A may be stored in a portable non-temporary storage medium such as an SSD or a USB memory. The medical support program 84A stored in the non-temporary storage medium is installed in the computer 76 of the duodenoscope 12. The processor 82 executes the medical support process in accordance with the medical support program 84A.

[0312] Further, the medical support program 84A is stored in a storage device such as another computer or server connected to the duodenoscope 12 via a network, and the medical support program 84A is downloaded according to a request from the duodenoscope 12 and installed in the computer 76.

[0313] In addition, it is not necessary to store all of the medical support program 84A in a storage device such as another computer or server device connected to the duodenoscope 12, or in the NVM 84, and a part of the medical support program 84A may be stored.

[0314] As the hardware resources for executing the medical support process, various processors shown below can be used. As the processor, for example, a general-purpose processor, i.e., a CPU, that functions as the hardware resources for executing the medical support process by executing software, i.e., a program, can be cited. Further, as the processor, for example, a processor having a circuit structure specifically designed for executing a specific process, i.e., a dedicated circuit, such as an FPGA, a PLD, or an ASIC, can be cited. A memory is built in or connected to any of the processors, and any of the processors executes the medical support process by using the memory.

[0315] The hardware resources for executing the medical support process may be constituted by one of these various processors, or may be constituted by a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs, or a combination of a CPU and an FPGA). Further, the hardware resources for executing the medical support process may be one processor.

[0316] As an example of a configuration including one processor, first, there is the following method: a processor is configured by a combination of one or more CPUs and software, and this processor functions as a hardware resource for executing medical support processing. Second, there is the following method: as represented by an SoC or the like, a processor is used that implements the functions of an entire system including multiple hardware resources for executing medical support processing with one IC chip. In this way, the medical support processing is implemented by using one or more of the above-described various processors as hardware resources.

[0317] In addition, as the hardware configuration of these various processors, more specifically, a circuit formed by combining circuit elements such as semiconductor elements can be used. And the above-described medical support processing is merely an example. Therefore, of course, unnecessary steps can be deleted, new steps can be added, or the processing order can be replaced within the scope of not departing from the gist.

[0318] The description and illustration content shown above are detailed descriptions of parts related to the technology of the present invention, and are merely examples of the technology of the present invention. For example, the description related to the above-described structure, function, operation, and effect is a description related to an example of the structure, function, operation, and effect of parts related to the technology of the present invention. Therefore, it goes without saying that parts that are not needed can be deleted from the description and illustration content shown above, new elements can be added, or replacements can be made within the scope of not departing from the gist of the technology of the present invention. And, in order to avoid complication and facilitate understanding of the parts related to the technology of the present invention, in the description and illustration content shown above, descriptions related to common general technical knowledge that does not require special explanation are omitted on the basis that the technology of the present invention can be implemented.

[0319] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. And, in this specification, when three or more matters are connected and expressed by "and / or", the same way of thinking as "A and / or B" is also applied.

[0320] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as when each is specifically and individually described.

[0321] The entire invention of Japanese Patent Application No. 2022-177613 filed on November 4, 2022 is incorporated herein by reference.

Claims

1. A medical support device, which includes a processor, The processor performs the following processes: Obtain intestinal direction-related information related to the intestinal direction of the duodenum according to geometric characteristic information capable of determining the geometric characteristics of the duodenum inserted into the endoscopic viewer; Output the intestinal direction-related information.

2. The medical support device according to claim 1, wherein The intestinal direction-related information includes offset information indicating the offset between the posture of the endoscopic viewer and the intestinal direction.

3. The medical support device according to claim 1, wherein The geometric characteristic information includes an intestinal wall image obtained by photographing the intestinal wall of the duodenum using a camera provided on the endoscopic viewer, The processor obtains the intestinal direction-related information by performing a first image recognition process on the intestinal wall image.

4. The medical support device according to claim 1, wherein Outputting the intestinal direction-related information includes displaying the intestinal direction-related information on a first screen.

5. The medical support device according to claim 1, wherein The intestinal direction-related information includes first direction information capable of determining a first direction that intersects the intestinal direction at a specified angle.

6. The medical support device according to claim 5, wherein The first direction information is obtained by performing a second image recognition process on an intestinal wall image obtained by photographing the intestinal wall of the duodenum using a camera provided on the endoscopic viewer.

7. The medical support device according to claim 6, wherein The first direction information is information obtained with a reliability above a threshold value through an AI-based image recognition process as the second image recognition process.

8. The medical support device according to claim 1, wherein The processor obtains posture information capable of determining the posture of the endoscopic viewer in a state where the endoscopic viewer is inserted into the duodenum, The intestinal direction-related information includes posture adjustment support information for supporting the adjustment of the posture, The posture adjustment support information is information set according to the offset between the intestinal direction and the posture determined by the posture information.

9. The medical support device according to claim 1, wherein The intestinal direction-related information includes condition information indicating a condition for making the optical axis direction of a camera provided on the endoscopic viewer coincide with a second direction that intersects the intestinal direction at a specified angle by changing the posture of the endoscopic viewer.

10. The medical support device according to claim 9, wherein The condition includes an operation condition related to the operation performed on the endoscopic viewer to make the optical axis direction coincide with the second direction.

11. The medical support device according to claim 1, wherein When the optical axis direction of a camera provided on the endoscopic viewer coincides with a third direction that intersects the intestinal direction at a specified angle, The intestinal direction-related information includes notification information notifying the content that the optical axis direction coincides with the third direction.

12. The medical support device according to claim 1, wherein, the processor performs the following processing: detecting the duodenal papilla region by performing a third image recognition process on the intestinal wall image, the intestinal wall image being obtained by photographing the intestinal wall of the duodenum using a camera provided in the endoscope viewer; displaying the duodenal papilla region on a second screen; displaying papilla orientation information indicating the orientation of the duodenal papilla region and obtained based on the intestinal tract direction-related information on the second screen.

13. The medical support device according to claim 1, wherein, the processor performs the following processing: detecting the duodenal papilla region by performing a fourth image recognition process on the intestinal wall image, the intestinal wall image being obtained by photographing the intestinal wall of the duodenum using a camera provided in the endoscope viewer; displaying the duodenal papilla region on a third screen; displaying traveling direction information indicating the traveling direction of the tube leading to the opening of the duodenal papilla region and obtained based on the intestinal tract direction-related information on the third screen.

14. The medical support device according to claim 13, wherein, the tube is a bile duct or a pancreatic duct.

15. The medical support device according to claim 1, wherein, the geometric characteristic information includes depth information indicating the depth of the duodenum, and the intestinal tract direction-related information is obtained based on the depth information.

16. An endoscope, comprising: the medical support device according to any one of claims 1 to 15; and the endoscope viewer.

17. A medical support method, comprising the following steps: obtaining intestinal tract direction-related information related to the intestinal tract direction of the duodenum based on geometric characteristic information capable of determining the geometric characteristics of the duodenum inserted into the endoscope viewer; and outputting the intestinal tract direction-related information.

18. A program for causing a computer to execute a process including the following steps: obtaining intestinal tract direction-related information related to the intestinal tract direction of the duodenum based on geometric characteristic information capable of determining the geometric characteristics of the duodenum inserted into the endoscope viewer; and outputting the intestinal tract direction-related information.

Images