Medical assistance device, endoscope, and medical assistance method

Abstract

A medical assistance device comprising a processor, wherein the processor acquires papilla orientation-related information relating to the orientation of the duodenal papilla on the basis of an intestinal wall image obtained by capturing an image of the intestinal wall including the duodenal papilla in the duodenum using a camera provided to an endoscope, displays the intestinal wall image on a screen, and displays the papilla orientation-related information on the screen.

Description

Medical support device, endoscope, and medical support method

[0001] The technology of the present disclosure relates to a medical support device, an endoscope, and a medical support method.

[0002] Japanese Patent Application Laid-Open Publication No. 2020-62218 discloses a learning device that includes an acquisition unit that acquires multiple pieces of information that associate images of the duodenal papilla of Vata in the bile duct with information indicating a cannulation method, which is a method of inserting a catheter into the bile duct; a learning unit that performs machine learning using information indicating the cannulation method as training data based on images of the duodenal papilla of Vata in the bile duct; and a memory unit that associates and stores the results of the machine learning by the learning unit with the information indicating the cannulation method.

[0003] One embodiment of the technology disclosed herein provides a medical support device, an endoscope, and a medical support method that enable a user observing an intestinal wall image to visually grasp information related to the orientation of the duodenal papilla.

[0004] A first aspect of the technology of the present disclosure is a medical support device that includes a processor, which acquires papilla orientation-related information relating to the orientation of the duodenal papilla based on an intestinal wall image obtained by capturing an image of the intestinal wall including the duodenal papilla in the duodenum using a camera attached to an endoscope, and displays the intestinal wall image on a screen and the papilla orientation-related information on the screen.

[0005] A second aspect of the technique of the present disclosure is the medical support device according to the first aspect, in which the papilla orientation-related information includes bulging direction information indicating a bulging direction of the duodenal papilla.

[0006] A third aspect of the technique of the present disclosure is the medical support device according to the second aspect, in which the nipple orientation information includes a protrusion direction image indicating the protrusion direction.

[0007] A fourth aspect of the technology of the present disclosure is a medical support device according to any one of the first to third aspects, in which the duodenal papilla has an opening, and the papilla orientation-related information includes surface orientation information indicating the direction of the surface on which the opening is located.

[0008] A fifth aspect of the technique of the present disclosure is the medical support device according to the fourth aspect, in which the nipple orientation-related information includes angle-related information relating to a relative angle between the plane and the orientation of the endoscope.

[0009] A sixth aspect of the technology of the present disclosure is a medical support device according to any one of the first to third aspects, in which the duodenal papilla has an opening, and the papilla orientation-related information includes face direction information indicating the direction of the face in which the opening is located, and angle-related information regarding the relative angle between the face and the posture of the endoscope.

[0010] A seventh aspect of the technology of the present disclosure is a medical support device according to any one of the first to sixth aspects, in which the papilla orientation-related information includes a surface image capable of identifying a surface that intersects with the protrusion direction of the duodenal papilla at a predetermined angle.

[0011] An eighth aspect of the technology of the present disclosure is a medical support device according to any one of the first to seventh aspects, in which the papilla orientation-related information includes coincidence information that can identify the degree of coincidence between the protrusion direction of the duodenal papilla and the optical axis direction of the endoscope.

[0012] A ninth aspect of the technology of the present disclosure is a medical support device relating to any one of the first to eighth aspects, in which the duodenal papilla includes a papillary protuberance and a headband fold covering the papillary protuberance, and the papilla orientation related information includes first direction information indicating a first direction from the top of the papillary protuberance toward the top of the headband fold.

[0013] A tenth aspect of the technology of the present disclosure is the medical support device according to the ninth aspect, in which the first direction information includes a first direction image indicating the first direction.

[0014] An eleventh aspect of the technology of the present disclosure is a medical support device according to the ninth or tenth aspect, in which the nipple prominence has an opening, the nipple orientation-related information includes running direction information indicating the running direction of a tube leading to the opening, and the running direction information is determined based on the first direction information.

[0015] A twelfth aspect of the technique of the present disclosure is the medical support device according to the eleventh aspect, in which the driving direction information includes a driving direction image indicating the driving direction.

[0016] A thirteenth aspect of the technology of the present disclosure is a medical support device according to any one of the first to twelfth aspects, in which the duodenal papilla has a papillary protuberance and a fold portion including a headband fold that covers the papillary protuberance, and the processor identifies the second direction based on the appearance of the fold portion shown in the intestinal image.

[0017] A fourteenth aspect of the technology of the present disclosure is a medical support device according to the thirteenth aspect, in which the processor identifies the second direction based on the appearance of an area including the papillary prominence and folds shown in the intestinal wall image.

[0018] A fifteenth aspect of the technology of the present disclosure is a medical support device according to any one of the first to fourteenth aspects, in which a processor acquires nipple orientation-related information by performing a first image recognition process on an intestinal wall image.

[0019] A sixteenth aspect of the technology of the present disclosure is a medical support device according to any one of the first to fifteenth aspects, in which a processor identifies the running direction of a tube leading to the opening of the duodenal papilla based on an intestinal wall image, and displays on a screen running direction information that allows the running direction to be identified within the intestinal wall image.

[0020] A seventeenth aspect of the technology of the present disclosure is a medical support device according to the sixteenth aspect, in which a processor acquires, based on an intestinal wall image, diverticulum area information capable of identifying a diverticulum area, which is an image area showing a diverticulum within the intestinal wall image, and changes the display mode of the travel direction information based on the diverticulum area information.

[0021] An eighteenth aspect of the technique of the present disclosure is the medical support device according to the seventeenth aspect, in which the display mode is a mode in which the running direction avoids the diverticulum region identified from the diverticulum region information.

[0022] A 19th aspect of the technology of the present disclosure is a medical support device according to any one of the 16th to 18th aspects, in which a processor acquires, based on an intestinal wall image, diverticulum area information capable of identifying a diverticulum area, which is an image area showing a diverticulum within the intestinal wall image, identifies a positional relationship between the diverticulum and the running direction based on the diverticulum area information and the running direction, and, if the positional relationship is such that the diverticulum intersects with the running direction, outputs notification information notifying that the diverticulum intersects with the running direction.

[0023] A twentieth aspect of the technology of the present disclosure is a medical support device according to any one of the first to nineteenth aspects, in which, when an endoscope having an endoscopic scope and a treatment tool is inserted into the duodenum, a processor determines a first relationship between the position of the treatment tool and the position of the duodenal papilla, and / or a second relationship between the direction of travel of the treatment tool and the orientation of the duodenal papilla, based on an image of the intestinal wall in which the treatment tool is visible, and executes a first notification process to issue a notification according to the first relationship and / or the second relationship.

[0024] A 21st aspect of the technology of the present disclosure is a medical support device according to any one of the first to 20th aspects, in which, when an endoscope having an endoscopic scope and a treatment tool is inserted into the duodenum, a processor determines a third relationship between the direction of travel of the treatment tool and a first direction related to the orientation of the duodenal papilla based on an image of the intestinal wall in which the treatment tool is visible, and executes a second notification process to issue a notification according to the third relationship.

[0025] A 22nd aspect of the technology of the present disclosure is a medical support device according to any one of the 1st to 21st aspects, in which a processor executes a third notification process that identifies the running direction of a tube leading to an opening of the duodenal papilla based on an intestinal wall image, and when an endoscope having an endoscope scope and a treatment tool is inserted into the duodenum, identifies the direction of advancement of the treatment tool based on the intestinal wall image in which the treatment tool is shown, and issues a notification according to a fourth relationship between the running direction and the advancement direction.

[0026] A 23rd aspect of the technology of the present disclosure is a medical support device relating to any one of the first to 22nd aspects, in which the papilla direction related information includes recommended incision direction information indicating a recommended incision direction for the duodenal papilla using an incision instrument that incises the duodenal papilla, or non-recommended incision direction information indicating a direction that is not recommended as an incision direction.

[0027] A 24th aspect of the technology of the present disclosure is a medical support device according to any one of the first to 23rd aspects, in which, when an endoscope having an endoscopic scope and a treatment tool is inserted into the duodenum, a processor obtains an evaluation value regarding the positional relationship between the duodenal papilla and the treatment tool based on an intestinal wall image in which the treatment tool is visible, and outputs information based on the evaluation value.

[0028] A 25th aspect of the technology of the present disclosure is a medical support device according to the 24th aspect, in which, when an endoscope having an endoscope scope and a treatment tool is inserted into the duodenum, the processor outputs information based on an evaluation value if it detects that the treatment tool is in contact with the duodenal papilla based on an intestinal wall image showing the treatment tool.

[0029] A 26th aspect of the technology of the present disclosure is a medical support device that includes a processor, and that identifies the running direction of a tube leading to the opening of the duodenal papilla based on an intestinal wall image obtained by capturing an image of the intestinal wall including the duodenal papilla in the duodenum with a camera attached to an endoscope, displays the intestinal wall image on a screen, and displays running direction information on the screen that allows the running direction to be identified within the intestinal wall image.

[0030] A twenty-seventh aspect of the technique of the present disclosure is an endoscope including a medical support device according to any one of the first to twenty-sixth aspects and an endoscope scope.

[0031] A 28th aspect of the technology of the present disclosure is a medical support method that includes acquiring papilla orientation related information related to the orientation of the duodenal papilla based on an intestinal wall image obtained by imaging the intestinal wall including the duodenal papilla in the duodenum with a camera provided in an endoscope, displaying the intestinal wall image on a screen, and displaying the papilla orientation related information on the screen.

[0032] A 29th aspect of the technology of the present disclosure is a medical support method that includes identifying the running direction of a tube leading to the opening of the duodenal papilla based on an intestinal wall image obtained by capturing an image of the intestinal wall including the duodenal papilla in the duodenum with a camera provided in an endoscope, displaying the intestinal wall image on a screen, and displaying running direction information on the screen that allows the running direction to be identified within the intestinal wall image.

[0033] 1 is a conceptual diagram showing an example of an aspect in which a duodenoscope system is used. FIG. 2 is a conceptual diagram showing an example of the overall configuration of a duodenoscope system. FIG. 3 is a block diagram showing an example of the hardware configuration of the electrical system of a duodenoscope system. FIG. 4 is a conceptual diagram showing an example of an aspect in which a duodenoscope is used. FIG. 5 is a block diagram showing an example of the hardware configuration of the electrical system of an image processing device. FIG. 6 is a conceptual diagram showing an example of the correlation between an endoscope, a duodenoscope main body, an image acquisition unit, an image recognition unit, and a derivation unit. FIG. 7 is a conceptual diagram showing an example of the correlation between a display device, an image acquisition unit, an image recognition unit, a derivation unit, and a display control unit. FIG. 8 is a flowchart showing an example of the flow of medical support processing. FIG. 9 is a conceptual diagram showing an example of the correlation between an endoscope, a duodenoscope main body, an image acquisition unit, an image recognition unit, and a derivation unit. FIG. 10 is a conceptual diagram showing an example of the correlation between an endoscope, a duodenoscope main body, an image acquisition unit, an image recognition unit, and a derivation unit. 1 is a conceptual diagram showing an example of the correlation between a display device, an image recognition unit, a derivation unit, and a display control unit. FIG. 1 is a conceptual diagram showing an example of the correlation between a display device, a derivation unit, and a display control unit. FIG. 2 is a conceptual diagram showing an example of the correlation between an endoscope, a duodenoscope main body, an image acquisition unit, an image recognition unit, and a derivation unit. FIG. 3 is a conceptual diagram showing an example of the correlation between a display device, an image recognition unit, a derivation unit, and a display control unit. FIG. 4 is a conceptual diagram showing an example of a manner in which an endoscope is faced directly to a papilla. FIG. 5 is a conceptual diagram showing an example of the correlation between an endoscope, a duodenoscope main body, an image acquisition unit, an image recognition unit, and a derivation unit. FIG. 6 is a conceptual diagram showing an example of the correlation between an endoscope, a duodenoscope main body, an image acquisition unit, an image recognition unit, and a derivation unit. FIG. 7 is a conceptual diagram showing an example of the correlation between a display device, an image recognition unit, a derivation unit, and a display control unit. FIG. 8 is a conceptual diagram showing an example of the correlation between an endoscope, a duodenoscope main body, an image acquisition unit, an image recognition unit, and a derivation unit. 1 is a conceptual diagram showing an example of the correlation between a display device, an image recognition unit, a derivation unit, and a display control unit; 2 is a conceptual diagram showing an example of the correlation between an endoscope, an image acquisition unit, an image recognition unit, and a derivation unit; 3 is a conceptual diagram showing an example of the correlation between a display device, a derivation unit, and a display control unit;1 is a flowchart showing an example of the flow of medical support processing. FIG. 1 is a conceptual diagram showing an example of the correlation between a display device, a derivation unit, and a display control unit. FIG. 2 is a conceptual diagram showing an example of the correlation between an endoscope, an image acquisition unit, an image recognition unit, and a derivation unit. FIG. 3 is a conceptual diagram showing an example of the correlation between an endoscope, an image acquisition unit, an image recognition unit, and a derivation unit. FIG. 4 is a conceptual diagram showing an example of the correlation between an endoscope, an image acquisition unit, an image recognition unit, and a derivation unit. FIG. 5 is a conceptual diagram showing an example of the correlation between an endoscope, an image acquisition unit, an image recognition unit, and a derivation unit. FIG. 6 is a conceptual diagram showing an example of the correlation between an endoscope, an image acquisition unit, an image recognition unit, and a derivation unit.

[0034] Hereinafter, exemplary embodiments of a medical support device, an endoscope, and a medical support method according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0035] First, the terms used in the following description will be explained.

[0036] CPU is an abbreviation for "Central Processing Unit." GPU is an abbreviation for "Graphics Processing Unit." RAM is an abbreviation for "Random Access Memory." NVM is an abbreviation for "Non-volatile memory." EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory." ASIC is an abbreviation for "Application Specific Integrated Circuit." PLD is an abbreviation for "Programmable Logic Device." FPGA is an abbreviation for "Field-Programmable Gate Array." SoC is an abbreviation for "System-on-a-chip." SSD is an abbreviation for "Solid State Drive." USB is an abbreviation for "Universal Serial Bus." HDD is an abbreviation for "Hard Disk Drive." EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". FIFO is an abbreviation for "First In First Out". ERCP is an abbreviation for "Endoscopic Retrograde Cholangio-Pancreatography". TOF is an abbreviation for "Time of Flight".

[0037] 1 , a duodenoscope system 10 includes a duodenoscope 12 and a display device 13. The duodenoscope 12 is used by a doctor 14 in an endoscopic examination. The duodenoscope 12 is communicatively connected to a communication device (not shown), and information obtained by the duodenoscope 12 is transmitted to the communication device. The communication device receives the information transmitted from the duodenoscope 12 and performs processing using the received information (for example, processing to record the information in an electronic medical record, etc.).

[0038] The duodenoscope 12 includes an endoscope 18. The duodenoscope 12 is a device for performing medical treatment on an observation target 21 (e.g., the duodenum) inside the body of a subject 20 (e.g., a patient) using the endoscope 18. The observation target 21 is an object to be observed by a doctor 14. The endoscope 18 is inserted into the body of the subject 20. The duodenoscope 12 causes the endoscope 18 inserted inside the body of the subject 20 to capture images of the observation target 21 inside the body of the subject 20, and performs various medical procedures on the observation target 21 as necessary. The duodenoscope 12 is an example of an "endoscope" according to the technology of the present disclosure.

[0039] The duodenoscope 12 acquires and outputs an image showing the state of the inside of the body by capturing an image of the inside of the body of the subject 20. In this embodiment, the duodenoscope 12 is an endoscope having an optical imaging function that captures an image of reflected light obtained by irradiating light inside the body and reflecting it off an observation target 21.

[0040] The duodenoscope 12 is equipped with a control device 22, a light source device 24, and an image processing device 25. The control device 22 and the light source device 24 are mounted on a wagon 34. The wagon 34 has a plurality of stands arranged vertically, with the image processing device 25, the control device 22, and the light source device 24 mounted from the lower stand to the upper stand. In addition, a display device 13 is mounted on the top stand of the wagon 34.

[0041] The control device 22 is a device that controls the entire duodenoscope 12. The image processing device 25 is a device that performs image processing on images captured by the duodenoscope 12 under the control of the control device 22.

[0042] The display device 13 displays various information including images (for example, images that have been subjected to image processing by the image processing device 25). Examples of the display device 13 include a liquid crystal display and an EL display. Alternatively, a tablet terminal with a display may be used instead of the display device 13 or together with the display device 13.

[0043] The display device 13 displays multiple screens side by side. In the example shown in FIG. 1 , screens 36, 37, and 38 are shown. The screen 36 displays an endoscopic image 40 acquired by the duodenoscope 12. The endoscopic image 40 shows an observation target 21. The endoscopic image 40 is an image acquired by capturing an image of the observation target 21 inside the body of the subject 20 using a camera 48 (see FIG. 2 ) provided on the endoscope 18. An example of the observation target 21 is the intestinal wall of the duodenum. For ease of explanation, the following description will use an intestinal wall image 41, which is an endoscopic image 40 capturing an image of the intestinal wall of the duodenum as the observation target 21. Note that the duodenum is merely an example, and any region that can be imaged by the duodenoscope 12 may be used. Examples of regions that can be imaged by the duodenoscope 12 include the esophagus and the stomach. The intestinal wall image 41 is an example of an "intestinal wall image" according to the technology of the present disclosure.

[0044] A moving image including a plurality of frames of intestinal wall images 41 is displayed on the screen 36. That is, the plurality of frames of intestinal wall images 41 are displayed on the screen 36 at a predetermined frame rate (for example, several tens of frames per second).

[0045] 2, the duodenoscope 12 includes an operating section 42 and an insertion section 44. The insertion section 44 is partially curved by operating the operating section 42. The insertion section 44 is inserted while being curved in accordance with the shape of the observation target 21 (for example, the shape of the duodenum) in accordance with the operation of the operating section 42 by the physician 14.

[0046] The distal end 46 of the insertion section 44 is provided with a camera 48, an illumination device 50, a treatment opening 51, and an erection mechanism 52. The camera 48 and the illumination device 50 are provided on the side of the distal end 46. In other words, the duodenoscope 12 is a side-viewing endoscope. This makes it easier to observe the intestinal wall of the duodenum.

[0047] The camera 48 is a device that captures an image of the inside of the subject 20 to obtain an intestinal wall image 41 as a medical image. An example of the camera 48 is a CMOS camera. However, this is merely an example, and other types of cameras such as a CCD camera may also be used. The camera 48 is an example of a "camera" according to the technology of the present disclosure.

[0048] The illumination device 50 has an illumination window 50A. The illumination device 50 emits light through the illumination window 50A. Types of light emitted from the illumination device 50 include, for example, visible light (e.g., white light) and invisible light (e.g., near-infrared light). The illumination device 50 also emits special light through the illumination window 50A. Examples of the special light include light for BLI and / or light for LCI. The camera 48 captures images of the inside of the subject 20 by an optical method while light is being emitted from the illumination device 50 inside the subject 20.

[0049] The treatment opening 51 is used as a treatment tool ejection port for ejecting a treatment tool 54 from the distal end portion 46, a suction port for sucking blood and internal waste, and a delivery port for delivering fluid.

[0050] A treatment tool 54 protrudes from the treatment opening 51 in accordance with the operation of the doctor 14. The treatment tool 54 is inserted into the insertion section 44 from a treatment tool insertion port 58. The treatment tool 54 passes through the insertion section 44 via the treatment tool insertion port 58 and protrudes from the treatment opening 51 into the body of the subject 20. In the example shown in FIG. 2 , a cannula protrudes from the treatment opening 51 as the treatment tool 54. The cannula is merely one example of the treatment tool 54, and other examples of the treatment tool 54 include a papillotomy knife, a snare, etc.

[0051] The standing mechanism 52 changes the protruding direction of the treatment tool 54 protruding from the treatment opening 51. The standing mechanism 52 includes a guide 52A, which rises relative to the protruding direction of the treatment tool 54, thereby changing the protruding direction of the treatment tool 54 along the guide 52A. This makes it easy to protrude the treatment tool 54 toward the intestinal wall. In the example shown in FIG. 2 , the standing mechanism 52 changes the protruding direction of the treatment tool 54 to a direction perpendicular to the traveling direction of the tip portion 46. The standing mechanism 52 is operated by the doctor 14 via the operation unit 42. This adjusts the degree of change in the protruding direction of the treatment tool 54.

[0052] The endoscope 18 is connected to the control device 22 and the light source device 24 via a universal cord 60. The control device 22 is connected to the display device 13 and the reception device 62. The reception device 62 receives instructions from a user (e.g., the doctor 14) and outputs the received instructions as an electrical signal. In the example shown in Fig. 2, a keyboard is given as an example of the reception device 62. However, this is merely an example, and the reception device 62 may also be a mouse, a touch panel, a foot switch, and / or a microphone, etc.

[0053] The control device 22 controls the entire duodenoscope 12. For example, the control device 22 controls the light source device 24 and exchanges various signals with the camera 48. The light source device 24 emits light under the control of the control device 22 and supplies the light to the illumination device 50. The illumination device 50 has a built-in light guide, and the light supplied from the light source device 24 passes through the light guide and is irradiated from illumination windows 50A and 50B. The control device 22 causes the camera 48 to capture an image, acquires an intestinal wall image 41 (see FIG. 1 ) from the camera 48, and outputs the image to a predetermined output destination (for example, the image processing device 25).

[0054] The image processing device 25 is communicably connected to the control device 22, and performs image processing on the intestinal wall image 41 output from the control device 22. Details of the image processing in the image processing device 25 will be described later. The image processing device 25 outputs the processed intestinal wall image 41 to a predetermined output destination (e.g., the display device 13). Note that, although an example in which the intestinal wall image 41 output from the control device 22 is output to the display device 13 via the image processing device 25 has been described here, this is merely one example. The control device 22 and the display device 13 may be connected, and the intestinal wall image 41 that has been subjected to image processing by the image processing device 25 may be displayed on the display device 13 via the control device 22.

[0055] 3 , the control device 22 includes a computer 64, a bus 66, and an external I / F 68. The computer 64 includes a processor 70, a RAM 72, and an NVM 74. The processor 70, the RAM 72, the NVM 74, and the external I / F 68 are connected to the bus 66.

[0056] For example, the processor 70 has a CPU and a GPU, and controls the entire control device 22. The GPU operates under the control of the CPU, and is responsible for executing various graphic processing and performing calculations using neural networks. The processor 70 may be one or more CPUs that have integrated GPU functionality, or one or more CPUs that do not have integrated GPU functionality.

[0057] The RAM 72 is a memory that temporarily stores information and is used as a work memory by the processor 70. The NVM 74 is a nonvolatile storage device that stores various programs, various parameters, and the like. An example of the NVM 74 is a flash memory (for example, an EEPROM and / or an SSD). Note that the flash memory is merely one example, and the NVM 74 may be another nonvolatile storage device such as an HDD, or may be a combination of two or more types of nonvolatile storage devices.

[0058] The external I / F 68 controls the exchange of various information between devices that exist outside the control device 22 (hereinafter also referred to as "external devices") and the processor 70. An example of the external I / F 68 is a USB interface.

[0059] The external I / F 68 is connected to a camera 48 as one of the external devices, and the external I / F 68 controls the exchange of various information between the camera 48 provided in the endoscope 18 and the processor 70. The processor 70 controls the camera 48 via the external I / F 68. The processor 70 also acquires, via the external I / F 68, intestinal wall images 41 (see FIG. 1 ) obtained by imaging the inside of the body of the subject 20 with the camera 48 provided in the endoscope 18.

[0060] The light source device 24 is connected to the external I / F 68 as one of the external devices, and the external I / F 68 controls the exchange of various information between the light source device 24 and the processor 70. The light source device 24 supplies light to the illumination device 50 under the control of the processor 70. The illumination device 50 irradiates the light supplied from the light source device 24.

[0061] The external I / F 68 is connected to a reception device 62 as one of the external devices, and the processor 70 acquires instructions accepted by the reception device 62 via the external I / F 68 and executes processing according to the acquired instructions.

[0062] The external I / F 68 is connected to the image processing device 25 as one of the external devices, and the processor 70 outputs the intestinal wall image 41 to the image processing device 25 via the external I / F 68 .

[0063] Incidentally, among the procedures for the duodenum using an endoscope, a procedure called ERCP (endoscopic retrograde cholangiopancreatography) may be performed. As an example, as shown in FIG. 4 , in an ERCP examination, for example, a duodenoscope 12 is first inserted into the duodenum J via the esophagus and stomach. In this case, the insertion state of the duodenoscope 12 may be confirmed by X-ray imaging. Then, a tip 46 of the duodenoscope 12 reaches the vicinity of the duodenal papilla N (hereinafter also simply referred to as "papilla N") present in the intestinal wall of the duodenum J.

[0064] In an ERCP examination, for example, a cannula 54A is inserted through the papilla N. The papilla N is a region that protrudes from the intestinal wall of the duodenum J, and the openings of the ends of the bile duct T (e.g., the common bile duct, intrahepatic bile duct, and cystic duct) and the pancreatic duct S are present at the papillary protuberance NA of the papilla N. X-ray imaging is performed in a state where a contrast agent is injected into the bile duct T, the pancreatic duct S, etc. through the opening of the papilla N via the cannula 54A. As described above, an ERCP examination includes various procedures, such as inserting the duodenoscope 12 into the duodenum J, confirming the position, orientation, and type of the papilla N, and inserting a treatment tool (e.g., a cannula) into the papilla N. Therefore, the physician 14 needs to operate the duodenoscope 12 and observe the condition of the target region according to each procedure.

[0065] For example, when inserting the duodenoscope 12 into the duodenum J, if the endoscope 18 of the duodenoscope 12 is tilted relative to the direction of the intestinal tract, the papilla N will be viewed in a tilted state, which may result in misidentifying the running direction of the bile duct T and pancreatic duct S from the papilla N. Therefore, it is necessary to understand the degree to which the orientation of the endoscope 18 is tilted relative to the direction of the intestinal tract within the duodenum J.

[0066] In view of the above circumstances, medical support processing is performed by the processor 82 of the image processing device 25 to support the implementation of medical treatment for the duodenum, including ERCP examination.

[0067] 5 , the image processing device 25 includes a computer 76, an external I / F 78, and a bus 80. The computer 76 includes a processor 82, an NVM 84, and a RAM 81. The processor 82, the NVM 84, the RAM 81, and the external I / F 78 are connected to the bus 80. The computer 76 is an example of a "medical support device" and a "computer" according to the technology of the present disclosure. The processor 82 is an example of a "processor" according to the technology of the present disclosure.

[0068] The hardware configuration of the computer 76 (i.e., the processor 82, the NVM 84, and the RAM 81) is basically the same as the hardware configuration of the computer 64 shown in Fig. 3, and therefore a description of the hardware configuration of the computer 76 will be omitted here. Also, the role of the external I / F 78 in the image processing device 25, namely, sending and receiving information to and from the outside, is basically the same as the role of the external I / F 68 in the control device 22 shown in Fig. 3, and therefore a description thereof will be omitted here.

[0069] A medical support processing program 84A is stored in the NVM 84. The medical support processing program 84A is an example of a "program" according to the technology of the present disclosure. The processor 82 reads the medical support processing program 84A from the NVM 84 and executes the read medical support processing program 84A on the RAM 81. The medical support processing according to this embodiment is realized by the processor 82 operating as an image acquisition unit 82A, an image recognition unit 82B, a derivation unit 82C, and a display control unit 82D in accordance with the medical support processing program 84A executed on the RAM 81.

[0070] A trained model 84B is stored in the NVM 84. In this embodiment, the image recognition unit 82B performs AI-based image recognition processing as the image recognition processing for object detection. The trained model 84B is optimized by performing machine learning on the neural network in advance.

[0071] As an example, as shown in FIG. 6, the image acquisition unit 82A acquires the intestinal wall image 41 generated by the camera 48 capturing images at an imaging frame rate (e.g., several tens of frames per second) from the camera 48 on a frame-by-frame basis.

[0072] The image acquisition unit 82A holds a time-series image group 89. The time-series image group 89 is a plurality of time-series intestinal wall images 41 that capture the object of observation 21. The time-series image group 89 includes, for example, a certain number of frames (e.g., a predetermined number of frames within a range of several tens to several hundreds of frames) of intestinal wall images 41. The image acquisition unit 82A updates the time-series image group 89 in a FIFO manner every time it acquires an intestinal wall image 41 from the camera 48.

[0073] Here, an example is given in which the time-series image group 89 is stored and updated by the image acquisition unit 82A, but this is merely one example. For example, the time-series image group 89 may be stored and updated in a memory connected to the processor 82, such as the RAM 81.

[0074] The image recognition unit 82B performs image recognition processing on the time-series image group 89 using the trained model 84B. By performing the image recognition processing, the intestinal tract direction CD included in the observation object 21 is detected. Here, the intestinal tract direction CD refers to the direction of the duodenum lumen. Here, detecting the intestinal tract direction refers to processing for storing in a memory in a state in which the intestinal tract direction information 90 (e.g., position coordinates indicating the extension direction of the duodenum), which is information that can identify the intestinal tract direction CD, and the intestinal wall image 41 are associated with each other.

[0075] The trained model 84B is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify the intestinal direction CD.

[0076] An example of an annotation in the correct answer data is an annotation of the intestinal tract direction CD based on the shape of the intestinal folds shown in the intestinal wall image 41 (for example, an annotation in which the line segment connecting the centers of the arcs of the fold shape is the intestinal tract direction CD). In addition, other annotations in the correct answer data include annotations based on depth information when the intestinal wall image 41 is a depth image (for example, an annotation in which the direction of increasing depth shown by the depth information is the intestinal tract direction CD).

[0077] Note that, although an example in which only one trained model 84B is used by the image recognition unit 82B is given here, this is merely one example. For example, a trained model 84B selected from multiple trained models 84B may be used by the image recognition unit 82B. In this case, each trained model 84B is created by performing machine learning specialized for a specific ERCP examination technique (e.g., the position of the duodenoscope 12 relative to the papilla N), and the trained model 84B corresponding to the currently performed ERCP examination technique is selected and used by the image recognition unit 82B.

[0078] The image recognition unit 82B inputs the intestinal wall image 41 acquired from the image acquisition unit 82A to the trained model 84B. As a result, the trained model 84B outputs intestinal tract direction information 90 corresponding to the input intestinal wall image 41. The image recognition unit 82B acquires the intestinal tract direction information 90 output from the trained model 84B.

[0079] The deriving unit 82C derives the amount of deviation of the endoscope 18 with respect to the intestinal tract direction CD (hereinafter simply referred to as the "deviation amount"). Here, the deviation amount refers to the degree of deviation between the posture of the endoscope 18 and the intestinal tract direction CD. Specifically, the deviation amount refers to the amount of deviation between the direction along the imaging surface of the imaging element of the camera 48 provided in the endoscope 18 (e.g., the vertical direction in the angle of view) and the intestinal tract direction CD. Furthermore, since the camera 48 is provided at the tip portion 46, the deviation amount can also be considered as the angle between the longitudinal direction SD of the tip portion 46 (e.g., the central axial direction when the tip portion 46 is cylindrical) and the intestinal tract direction CD.

[0080] The derivation unit 82C acquires intestinal tract direction information 90 from the image recognition unit 82B. The derivation unit 82C also acquires attitude information 91 from an optical fiber sensor 18A provided in the endoscope 18. The attitude information 91 is information indicating the attitude of the endoscope 18. The optical fiber sensor 18A is a sensor arranged along the longitudinal direction inside the endoscope 18 (e.g., the insertion section 44 and the tip end 46). Use of the optical fiber sensor 18A makes it possible to detect the attitude of the endoscope 18 (e.g., the inclination of the tip end 46 from a reference position (e.g., the straight state of the endoscope 18)). In this case, it is possible to appropriately utilize known endoscope attitude detection technology such as that described in Japanese Patent No. 6797834.

[0081] Although the attitude detection technology using the optical fiber sensor 18A has been described above, this is merely one example. For example, the tilt of the tip 46 of the endoscope 18 may be detected using a so-called electromagnetic navigation system. In this case, it is possible to appropriately use known endoscope attitude detection technology such as that disclosed in Japanese Patent No. 6534193.

[0082] The derivation unit 82C derives displacement amount information 93, which is information indicating the displacement amount, using the intestinal tract direction information 90 and the posture information 91. In the example shown in Fig. 6, an angle A is shown as the displacement amount information 93. The derivation unit 82C derives the displacement amount using, for example, a displacement amount calculation formula (not shown). The displacement amount calculation formula is a calculation formula in which the position coordinates of the intestinal tract direction CD indicated by the intestinal tract direction information 90 and the position coordinates of the longitudinal direction SD of the tip portion 46 indicated by the posture information 91 are independent variables, and the angle formed between the intestinal tract direction CD and the longitudinal direction SD of the tip portion 46 is a dependent variable.

[0083] As an example, as shown in FIG. 7 , the display control unit 82D acquires an intestinal wall image 41 from the image acquisition unit 82A. The display control unit 82D also acquires intestinal tract direction information 90 from the image recognition unit 82B. The display control unit 82D then acquires deviation amount information 93 from the derivation unit 82C. The display control unit 82D generates an operation instruction image 93A for aligning the longitudinal direction SD of the tip unit 46 with the intestinal tract direction CD according to the deviation amount indicated by the deviation amount information 93. The operation instruction image 93A is, for example, an arrow indicating the operation direction of the tip unit 46 that reduces the deviation amount. The display control unit 82D generates a display image 94 including the intestinal wall image 41, the intestinal tract direction CD indicated by the intestinal tract direction information 90, and the operation instruction image 93A, and outputs the display device 13. Specifically, the display control unit 82D controls a GUI (Graphical User Interface) for displaying the display image 94, thereby causing the display device 13 to display the screen 36. The screen 36 is an example of a "first screen" according to the technology of the present disclosure. The operation instruction image 93A is an example of "posture adjustment support information" according to the technology of the present disclosure.

[0084] Although the above description has been given of an example in which the operation instruction image 93A is displayed on the screen 36 to allow the user to understand the amount of misalignment, the technology of the present disclosure is not limited to this. For example, a message (not shown) indicating the operation to reduce the amount of misalignment may be displayed on the screen 36. One example of the message is "Tilt the tip of the duodenoscope 10 degrees toward the back." The user may be notified by an audio output device such as a speaker.

[0085] The user can understand the intestinal tract direction CD by visually checking the screen 36 of the display device 13. In addition, the user can understand the operation for reducing the misalignment between the tip 46 of the endoscope 18 and the intestinal tract direction CD by visually checking the operation instruction image 93A displayed on the screen 36.

[0086] Next, the operation of the portion of the duodenoscope system 10 related to the technology of the present disclosure will be described with reference to FIG.

[0087] FIG. 8 shows an example of the flow of medical support processing performed by the processor 82.

[0088] 8, first, in step ST10, the image acquisition unit 82A determines whether or not one frame of image data has been captured by the camera 48 provided on the endoscope 18. If one frame of image data has not been captured by the camera 48 in step ST10, the determination is negative, and the determination in step ST10 is made again. If one frame of image data has been captured by the camera 48 in step ST10, the determination is positive, and the medical support process proceeds to step ST12.

[0089] In step ST12, the image acquisition unit 82A acquires one frame of the intestinal wall image 41 from the camera 48 provided in the endoscope 18. After the processing of step ST12 is executed, the medical support processing proceeds to step ST14.

[0090] In step ST14, the image recognition unit 82B detects the intestinal direction CD by performing AI image recognition processing (i.e., image recognition processing using the trained model 84B) on the intestinal wall image 41 acquired in step ST12. After the processing of step ST14 is executed, the medical support processing proceeds to step ST16.

[0091] In step ST16, the derivation unit 82C acquires the posture information 91 from the optical fiber sensor 18A of the endoscope 18. After the processing of step ST16 is executed, the medical support processing proceeds to step ST18.

[0092] In step ST18, the derivation unit 82C derives the amount of deviation based on the intestinal tract direction CD obtained by the image recognition unit 82B in step ST14 and the posture information 91 acquired in step ST16. Specifically, the derivation unit 82C derives the angle between the intestinal tract direction CD and the longitudinal direction SD of the tip portion 46 indicated by the posture information 91. After the processing of step ST18 is executed, the medical support processing proceeds to step ST20.

[0093] In step ST20, the display control unit 82D generates a display image 94 in which the intestinal direction CD and an operation instruction image 93A corresponding to the deviation amount derived in step ST18 are superimposed on the intestinal wall image 41. After the processing of step ST20 is executed, the medical support processing proceeds to step ST22.

[0094] In step ST22, the display control unit 82D outputs the display image 94 generated in step ST20 to the display device 13. After the processing of step ST22 is executed, the medical support processing proceeds to step ST24.

[0095] In step ST24, the display control unit 82D determines whether a condition for terminating the medical support process is satisfied. One example of the condition for terminating the medical support process is that an instruction to terminate the medical support process has been given to the duodenoscope system 10 (for example, that an instruction to terminate the medical support process has been accepted by the acceptance device 62).

[0096] In step ST24, if the condition for terminating the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST 10. In step ST24, if the condition for terminating the medical support process is satisfied, the determination is positive, and the medical support process ends.

[0097] As described above, in the duodenoscope system 10 according to the first embodiment, the image recognition unit 82B of the processor 82 performs image recognition processing on the intestinal wall image 41, and as a result of the image recognition processing, the intestinal tract direction CD in the intestinal wall image 41 is detected. Then, intestinal tract direction information 90 indicating the intestinal tract direction CD is output to the display control unit 82D, and a display image 94 generated by the display control unit 82D is output to the display device 13. The display image 94 includes the intestinal tract direction CD superimposed on the intestinal wall image 41. This allows the user to recognize the intestinal tract direction CD, and this configuration makes it easier for the user to grasp the degree to which the posture of the endoscope 18 is deviated from the intestinal tract direction CD.

[0098] Furthermore, in the duodenoscope system 10 according to the first embodiment, the derivation unit 82C derives deviation amount information 93. The deviation amount information 93 indicates the amount of deviation between the posture of the endoscope 18 and the intestinal tract direction CD. The deviation amount information 93 is output to the display control unit 82D, and a display image 94 generated by the display control unit 82D is output to the display device 13. The display image 94 includes a display based on the deviation amount information 93. This allows the user to recognize the amount of deviation between the posture of the endoscope 18 and the intestinal tract direction CD, and this configuration makes it easier for the user to understand how much the posture of the endoscope 18 has deviated from the intestinal tract direction CD.

[0099] Furthermore, in the duodenoscope system 10 according to the first embodiment, the image recognition unit 82B performs image recognition processing on the intestinal wall image 41, thereby obtaining intestinal direction information 90 indicating the intestinal direction CD. This allows for more accurate intestinal direction information 90 to be obtained compared to when the user visually specifies the intestinal direction CD for the intestinal wall image 41.

[0100] Furthermore, in the duodenoscope system 10 according to the first embodiment, the display control unit 82D outputs intestinal tract direction information 90 to the display device 13, and the display device 13 displays the intestinal tract direction CD on the screen 36. This makes it easier for the user to visually understand how much the posture of the endoscope 18 has deviated from the intestinal tract direction CD.

[0101] In the duodenoscope system 10 according to the first embodiment, the output unit 82C acquires, from the optical fiber sensor 18A, posture information 91, which is information that can identify the posture of the endoscope 18. The output unit 82C generates deviation amount information 93 based on the posture information 91 and the intestinal tract direction information 90. The display control unit 82D generates an instruction image 93A indicating an operation direction for reducing the deviation based on the deviation amount information 93. The display control unit 82D outputs the instruction image 93A to the display device 13, and the instruction image 93A is superimposed on the intestinal wall image 41 on the display device 13. This makes it easy for the user to adjust the posture of the endoscope 18 relative to the intestinal tract direction CD to the posture intended by the user when the endoscope 18 is inserted into the duodenum. For example, the user can change the posture of the endoscope 18 in the direction indicated by the instruction image 93A to bring the posture of the endoscope 18 closer to the intestinal tract direction CD.

[0102] Although the first embodiment described above exemplifies a configuration in which the intestinal direction CD is detected by image recognition processing using an AI system, the technology of the present disclosure is not limited to this. For example, the intestinal direction CD may be detected by image recognition processing using a pattern matching system. In this case, for example, a region indicating intestinal folds (i.e., a fold region) included in the intestinal wall image 41 may be detected, and the intestinal direction may be estimated from the arc shape of the fold region (for example, the line connecting the centers of the arcs may be estimated as the intestinal direction).

[0103] (First Modification) In the first embodiment, an example was described in which the intestinal direction CD was detected using an intestinal wall image 41 that did not include depth information, but the technology of the present disclosure is not limited to this. In this first modification, the image recognition unit 82B derives the intestinal direction using the intestinal wall image 41, which is a depth image. As shown in FIG. 9 as an example, the intestinal wall image 41 is a depth image that has depth information 41A, which is information indicating the depth of the subject, i.e., the distance to the intestinal wall, as pixel values. The depth of the duodenum is obtained, for example, by a distance measurement sensor mounted on the tip portion 46 using a so-called time-of-flight (TOF) method. The image recognition unit 82B acquires the intestinal wall image 41 from the image acquisition unit 82A.

[0104] The image recognition unit 82B derives intestinal tract direction information 90 based on the depth information 41A indicated by the intestinal wall image 41. The image recognition unit 82B derives the intestinal tract direction information 90 using, for example, an intestinal tract direction calculation formula 82B1. The intestinal tract direction calculation formula 82B1 is, for example, a calculation formula in which the depth indicated by the depth information 41A is an independent variable and a group of position coordinates of axes indicating the intestinal direction CD is a dependent variable. In this way, the intestinal tract direction information 90 is obtained based on the depth information 41A of the intestinal wall image 41.

[0105] As described above, in the duodenoscope system 10 according to the first modification, the intestinal wall image 41 has depth information 41A indicating the depth of the duodenum, and the intestinal direction information 90 is acquired based on the depth information 41A. The intestinal direction CD is a direction along the depth direction of the duodenal lumen. The depth information 41A reflects the depth of the duodenal lumen. Therefore, since the intestinal direction CD is derived based on the depth information 41A, the intestinal direction information 90 indicating the intestinal direction CD with higher accuracy can be obtained compared to when the depth information 41A is not taken into consideration.

[0106] (Second Modification) In the above-described first embodiment, an example was given in which the intestinal tract direction CD was obtained by image recognition processing of the intestinal wall image 41. However, the technology of the present disclosure is not limited to this. In this second modification, a direction that intersects with the intestinal tract direction CD at a predetermined angle (hereinafter also simply referred to as the "predetermined direction") is obtained.

[0107] As an example, as shown in FIG. 10, the image acquisition unit 82A updates the time-series image group 89 in a FIFO manner every time an intestinal wall image 41 is acquired from the camera 48.

[0108] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84C. As a result, the trained model 84C outputs vertical direction information 97 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the vertical direction information 97 output from the trained model 84C. Here, the vertical direction information 97 is information (for example, a group of position coordinates indicating an axis perpendicular to the intestinal tract direction CD) that can identify a direction VD perpendicular to the intestinal tract direction CD (hereinafter also simply referred to as the "vertical direction VD").

[0109] In the image recognition process using the trained model 84C, the confidence level of the identification result is calculated according to the result of identifying the direction perpendicular to the intestinal direction CD. Here, the confidence level is a statistical measure indicating the certainty of the identification result. The confidence level is, for example, a score input to an activation function (e.g., a softmax function) of the output layer of the trained model 84C. The vertical direction information 97 output from the trained model 84C has a score equal to or greater than a threshold (e.g., 0.9 or greater).

[0110] In this embodiment, "perpendicular" refers to not only perfectly perpendicular, but also perpendicular in the sense of including an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure. Here, the predetermined angle with respect to the intestinal tract direction CD is the perpendicular direction with respect to the intestinal tract direction CD, but the technology of the present disclosure is not limited to this. For example, the predetermined angle may be 45 degrees, 60 degrees, or 80 degrees.

[0111] The trained model 84C is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify the vertical direction VD.

[0112] The derivation unit 82C derives the degree of agreement between the predetermined direction and the direction of the optical axis of the camera 48. The agreement between the predetermined direction and the direction of the optical axis means that the direction in which the camera 48 is facing is the same as the direction predetermined by the user. In other words, this means that the tip end 46 on which the camera 48 is provided is not in a direction not intended by the user (for example, a direction tilted with respect to the intestinal tract direction CD).

[0113] Therefore, the derivation unit 82C acquires vertical direction information 97. The derivation unit 82C also acquires optical axis information 48A from the camera 48 of the endoscope 18. The optical axis information 48A is information that can identify the optical axis of the optical system of the camera 48. The derivation unit 82C then generates coincidence information 99 by comparing the direction indicated by the vertical direction information 97 with the direction of the optical axis indicated by the optical axis information 48A. The coincidence information 99 is information that indicates the degree of coincidence between the direction of the optical axis and a predetermined direction (for example, the angle between the direction of the optical axis and the predetermined direction). Note that in the present embodiment, "coincidence" refers to not only perfect coincidence but also coincidence that includes an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not contradict the spirit of the technology of the present disclosure.

[0114] Furthermore, the derivation unit 82C determines whether the direction of the optical axis matches the predetermined direction. If the direction of the optical axis matches the predetermined direction, the derivation unit 82C generates notification information 100. The notification information 100 is information that notifies the user that the direction of the optical axis matches the predetermined direction (for example, text indicating that the direction of the optical axis matches the predetermined direction).

[0115] 11 , the display control unit 82D acquires vertical direction information 97 from the image recognition unit 82B. The display control unit 82D also acquires coincidence information 99 from the derivation unit 82C. The display control unit 82D generates an operation instruction image 93B (e.g., an arrow indicating the operation direction) for aligning the optical axis direction with the predetermined direction, depending on the degree of coincidence between the optical axis direction indicated by the coincidence information 99 and the predetermined direction. The display control unit 82D then generates a display image 94 including the vertical direction VD indicated by the vertical direction information 97, the operation instruction image 93B, and the intestinal wall image 41, and outputs the display image 94 to the display device 13. In the example shown in FIG. 11 , the display device 13 displays the intestinal wall image 41 with the vertical direction VD and the operation instruction image 93B superimposed on the screen 36.

[0116] Furthermore, when the direction of the optical axis and the predetermined direction match, the derivation unit 82C outputs notification information 100 to the display control unit 82D instead of the coincidence information 99. In this case, the display control unit 82D generates a display image 94, instead of an operation instruction image 93B, that includes content notifying the user that the direction of the optical axis indicated by the notification information 100 matches the predetermined direction. In the example shown in FIG. 11 , a message stating "The optical axis and the vertical direction match" is displayed on the screen 37 of the display device 13.

[0117] Although an example in which a message based on the notification information 100 is displayed on the display device 13 has been described above, this is merely an example. For example, a symbol such as a circle based on the notification information 100 may be displayed. Furthermore, instead of the display device 13, or together with the display device 13, the notification information 100 may be output to an audio output device such as a speaker.

[0118] As described above, in the duodenoscope system 10 according to the second modification, the deriving unit 82C derives vertical direction information 97, which is information that can identify a direction perpendicular to the intestinal direction CD. The vertical direction information 97 is output to the display control unit 82D, and the display image 94 generated by the display control unit 82D is output to the display device 13. The display image 94 includes the vertical direction VD indicated by the vertical direction information 97. This allows the user to recognize the direction that intersects with the intestinal direction CD at a predetermined angle.

[0119] Furthermore, in the duodenoscope system 10 according to the second modification, the image recognition unit 82B performs image recognition processing on the intestinal wall image 41, thereby obtaining vertical direction information 97 indicating the vertical direction VD. This allows for the obtaining of more accurate vertical direction information 97 than when the user visually specifies the vertical direction VD for the intestinal wall image 41.

[0120] Furthermore, in the duodenoscope system 10 according to the second modification, in the image recognition process using the trained model 84C in the image recognition unit 82B, the vertical direction information 97 is obtained with a certainty level equal to or higher than a threshold value. As a result, in the image recognition process using the trained model 84C in the image recognition unit 82B, the vertical direction information 97 can be obtained with higher accuracy than when no threshold value is set for the certainty level.

[0121] In the duodenoscope system 10 according to the second modification, the output unit 82C acquires optical axis information 48A from the camera 48. The output unit 82C generates coincidence information 99 based on the optical axis information 48A and vertical direction information 97. The display control unit 82D generates a display image 94 based on the coincidence information 99 and outputs it to the display device 13. The display image 94 includes an indication of the degree of coincidence between the optical axis direction indicated by the coincidence information 99 and a predetermined direction. This allows the user to grasp the degree of deviation between the optical axis of the camera 48 and the vertical direction VD. For example, if the optical axis and the vertical direction VD are coincident, the camera 48 is likely facing the duodenal wall. Maintaining the endoscope 18 in this position makes it easier to find the papilla N present in the duodenal wall and to orient the camera 48 directly toward the papilla N.

[0122] Furthermore, in the duodenoscope system 10 according to the second modification, the display control unit 82D generates an operation instruction image 93B for aligning the direction of the optical axis with a predetermined direction based on the coincidence information 99. The display control unit 82D outputs the operation instruction image 93B to the display device 13, and the display device 13 displays the operation instruction image 93B superimposed on the intestinal wall image 41. This allows the user to understand the operation required to align the optical axis direction of the camera 48 with the vertical direction VD.

[0123] Furthermore, in the duodenoscope system 10 according to the second modification, the derivation unit 82C determines whether the direction of the optical axis matches the predetermined direction, and if the direction of the optical axis matches the predetermined direction, the derivation unit 82C generates notification information 100. The display control unit 82D generates a display image 94 based on the notification information 100 and outputs it to the display device 13. The display image 94 includes an indication that the direction of the optical axis indicated by the notification information 100 matches the predetermined direction. This allows the user to perceive that the direction of the optical axis matches the predetermined direction.

[0124] In the first embodiment, the intestinal direction CD is obtained by performing image recognition processing on the intestinal wall image 41. However, the technology of the present disclosure is not limited to this. In this third modification, the running direction TD of the bile duct is obtained based on the intestinal direction CD.

[0125] As an example, as shown in FIG. 12, the image acquisition unit 82A updates the time-series image group 89 in a FIFO manner every time an intestinal wall image 41 is acquired from the camera 48.

[0126] The image recognition unit 82B performs papilla detection processing using the trained model 84D on the time-series image group 89. The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84D. As a result, the trained model 84D outputs papilla region information 95 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the papilla region information 95 output from the trained model 84D. Here, the papilla region information 95 includes information (e.g., coordinates and range within the image) that can identify the papilla region N1 in the intestinal wall image 41 in which the papilla N appears.

[0127] The trained model 84D is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The training data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify the papilla region N1.

[0128] The derivation unit 82C derives running direction information 96, which is information indicating the running direction TD of the bile duct. The running direction information 96 includes information that can identify the direction in which the bile duct extends (e.g., position coordinates indicating the direction in which the bile duct extends). The derivation unit 82C acquires papilla region information 95 from the image recognition unit 82B. The derivation unit 82C also acquires intestinal tract direction information 90, obtained by image recognition processing using the trained model 84B (see FIG. 6), from the image recognition unit 82B. The derivation unit 82C then derives the running direction information 96 based on the intestinal tract direction information 90 and the papilla region information 95. The derivation unit 82C derives the running direction TD, for example, from a predetermined orientation relationship between the intestinal tract direction CD and the running direction TD. Specifically, when the intestinal tract direction CD is set to the 6 o'clock direction, the derivation unit 82C derives the running direction TD as the 11 o'clock to 12 o'clock direction. Furthermore, the derivation unit 82C sets the nipple region N1 indicated by the nipple region information 95 as the starting point of the traveling direction TD.

[0129] 13 , the display control unit 82D acquires running direction information 96 from the derivation unit 82C. The display control unit 82D also acquires papilla region information 95 from the image recognition unit 82B. The display control unit 82D generates a display image 94 in which the running direction TD indicated by the running direction information 96 and the papilla region N1 indicated by the papilla region information 95 are superimposed on the intestinal wall image 41 acquired from the image acquisition unit 82A (see FIG. 6 ), and outputs the display image 94 to the display device 13. The display device 13 displays the intestinal wall image 41 with the running direction TD superimposed on the screen 36.

[0130] As described above, in the duodenoscope system 10 according to the third modified example, the image recognition unit 82B performs a papilla detection process using the trained model 84D. Papilla region information 95 is obtained by the papilla detection process. Furthermore, the image recognition unit 82B performs an image recognition process using the trained model 84A, thereby obtaining intestinal direction information 90. The derivation unit 82C derives running direction information 96 based on the intestinal direction information 90 and the papilla region information 95. The display control unit 82D then outputs a display image 94 to the display device 13. The display image 94 includes a papilla region N1 indicated by the papilla region information 95 and the running direction TD of the bile duct indicated by the running direction information 96. The papilla region N1 and the running direction TD of the bile duct are displayed on the screen 36 of the display device 13. This allows a user observing the papilla N through the screen 36 to easily visually grasp the running direction TD of the bile duct.

[0131] For example, in an ERCP examination, the camera 48 may be positioned directly facing the papilla N. In this case, by using the running direction of the bile duct or pancreatic duct, it becomes easier to grasp the posture of the endoscope 18. Furthermore, when inserting a treatment tool into the papilla N, by knowing the running direction of the bile duct or pancreatic duct, it becomes easier to perform the operation of intubating the bile duct or pancreatic duct within the papilla N.

[0132] (Fourth Modification) In the above-described first embodiment, an example was given in which the intestinal tract direction CD was obtained by image recognition processing of the intestinal wall image 41. However, the technology of the present disclosure is not limited to this. In this fourth modification, the orientation of the papilla prominence NA at the papilla N (hereinafter also simply referred to as "papilla orientation ND") is obtained based on the intestinal tract direction CD.

[0133] The image recognition unit 82B performs image recognition processing on the intestinal wall image 41, thereby obtaining intestinal tract direction information 90 and nipple region information 95 (see FIG. 12 ). As shown in FIG. 14 as an example, the derivation unit 82C generates nipple direction information 102 based on the intestinal tract direction information 90 and nipple region information 95. The nipple direction information 102 is information that can identify the nipple direction ND (e.g., the direction in which the nipple prominence NA faces the treatment tool). The nipple direction ND is obtained, for example, as a tangent to the nipple prominence NA in the running direction TD of the bile duct. Therefore, the derivation unit 82C derives the running direction TD of the bile duct from the intestinal direction CD indicated by the intestinal direction information 90, and further derives the direction of the tangent to the nipple prominence NA from the running direction TD as the nipple direction ND.

[0134] The display control unit 82D acquires the nipple orientation information 102 from the derivation unit 82C. The display control unit 82D generates a display image 94 in which the nipple orientation ND indicated by the nipple orientation information 102 and the nipple region N1 indicated by the nipple region information 95 are superimposed on the intestinal wall image 41 acquired from the image acquisition unit 82A (see FIG. 6 ), and outputs the display image 94 to the display device 13. The display device 13 displays the intestinal wall image 41 with the nipple orientation ND superimposed on the screen 36.

[0135] Although the nipple direction ND is displayed as an arrow in this example, this is merely an example. The nipple direction ND may be displayed as text.

[0136] As described above, in the duodenoscope system 10 according to the fourth modification, the image recognition unit 82B performs a nipple detection process (see FIG. 12 ) to obtain nipple region information 95. The image recognition unit 82B also performs an image recognition process using the trained model 84B (see FIG. 6 ), thereby obtaining intestinal direction information 90. The derivation unit 82C derives nipple orientation information 102 based on the intestinal direction information 90. The display control unit 82D then outputs a display image 94 to the display device 13. The display image 94 includes a nipple region N1 indicated by the nipple region information 95 and a nipple direction ND indicated by the nipple direction information 102. The nipple region N1 and the nipple direction ND are displayed on the screen 36 of the display device 13. This allows a user observing the nipple N through the screen 36 to easily visually grasp the nipple direction ND.

[0137] For example, in an ERCP examination, the camera 48 may be positioned directly facing the nipple N. In this case, by using the nipple direction ND, it becomes easier to grasp the posture of the endoscope 18. Furthermore, when inserting a treatment tool into the nipple N, by grasping the nipple direction ND, it becomes possible to orient the treatment tool directly toward the nipple N, making it easier to insert the treatment tool into the nipple N.

[0138] Second Embodiment In the first embodiment, an example was described in which the intestinal tract direction CD was obtained by image recognition processing on the intestinal wall image 41. However, the technology of the present disclosure is not limited to this. In the second embodiment, the intestinal wall image 41 is an image obtained by capturing an image of the intestinal wall including the papilla N, and the protrusion direction RD of the papilla N is obtained by image recognition processing on the intestinal wall image 41.

[0139] For example, in an ERCP examination, the camera 48 may be positioned directly facing the protrusion direction RD of the papilla N. This makes it easier to estimate the running directions of the bile duct T and pancreatic duct S extending from the papilla N and to insert a treatment tool (e.g., a cannula) into the papilla N. Therefore, in the second embodiment, the protrusion direction RD of the papilla N is obtained by image recognition processing of the intestinal wall image 41. The protrusion direction RD is an example of the "protrusion direction" and "first direction" according to the technology of the present disclosure.

[0140] As an example, as shown in FIG. 15, the image acquisition unit 82A updates the time-series image group 89 in a FIFO manner every time an intestinal wall image 41 is acquired from the camera 48.

[0141] The image recognition unit 82B acquires a time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84E. As a result, the trained model 84E outputs uplift direction information 104 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the uplift direction information 104 output from the trained model 84E. Here, the uplift direction information 104 is information that can identify the direction in which the nipple N uplifts (for example, a group of position coordinates of an axis indicating the uplift direction RD). The uplift direction information 104 is an example of "nipple orientation-related information" and "uplift direction information" according to the technology of the present disclosure.

[0142] The trained model 84E is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The training data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify the protrusion direction RD of the papilla N.

[0143] Here, the protrusion direction RD of the nipple N is identified as, for example, the direction from the apex of the nipple protrusion NA of the nipple N to the apex of the headband fold H1. This is because, according to medical findings, the protrusion direction RD of the nipple N often coincides with the direction from the apex of the nipple protrusion NA to the apex of the headband fold H1. Here, there are multiple folds (e.g., folds H1 to H3) around the protruding portion of the nipple N. The headband fold H1 is the fold closest to the nipple protrusion NA. Therefore, an example of an annotation in the correct answer data is an annotation in which the direction passing through the apex of the headband fold H1 is the protrusion direction RD.

[0144] The derivation unit 82C derives the degree of coincidence between the protrusion direction RD and the direction of the optical axis of the camera 48. The fact that the protrusion direction RD and the direction of the optical axis coincide means that the direction in which the camera 48 is facing is directly facing the nipple N. In other words, this means that the tip 46 on which the camera 48 is provided is not in a direction not intended by the user (for example, a direction tilted with respect to the protrusion direction RD of the nipple N).

[0145] Therefore, the derivation unit 82C acquires the protrusion direction information 104 from the image recognition unit 82B. The derivation unit 82C also acquires the optical axis information 48A from the camera 48 of the endoscope 18. The derivation unit 82C then generates the coincidence information 103 by comparing the direction indicated by the vertical direction information 97 with the direction of the optical axis indicated by the optical axis information 48A. The coincidence information 103 is information that can identify the degree of coincidence between the direction of the optical axis and the protrusion direction RD (for example, the angle formed between the direction of the optical axis and the protrusion direction RD). The coincidence information 103 is an example of the "coincidence information" according to the technology of the present disclosure.

[0146] As an example, as shown in FIG. 16 , the display control unit 82D acquires protrusion direction information 104 from the image recognition unit 82B. The display control unit 82D also acquires coincidence information 103 from the derivation unit 82C. The display control unit 82D generates an operation instruction image 93C (e.g., an arrow indicating an operation direction) for aligning the optical axis direction with the protrusion direction RD, depending on the degree of coincidence between the optical axis direction indicated by the coincidence information 103 and the protrusion direction RD. The display control unit 82D then generates a display image 94 including the protrusion direction RD indicated by the protrusion direction information 104, the operation instruction image 93C, and the intestinal wall image 41, and outputs the display image 94 to the display device 13. In the example shown in FIG. 16 , the display device 13 displays the intestinal wall image 41 with the protrusion direction RD and the operation instruction image 93C superimposed on the screen 36.

[0147] 17 , the doctor 14 operates the endoscope 18 to move the optical axis of the camera 48 closer to the protrusion direction RD, thereby obtaining an intestinal wall image 41 in which the papilla N and the camera 48 are directly facing each other, which makes it easier to estimate the running direction of the bile duct T and pancreatic duct S extending from the papilla N and to insert a treatment tool (e.g., a cannula) into the papilla N.

[0148] As described above, in the duodenoscope system 10 according to the second embodiment, the image recognition unit 82B of the processor 82 performs image recognition processing on the intestinal wall image 41, and as a result of the image recognition processing, the protrusion direction RD of the papilla N in the intestinal wall image 41 is detected. Then, protrusion direction information 104 indicating the protrusion direction RD is output to the display control unit 82D, and a display image 94 generated by the display control unit 82D is output to the display device 13. The display image 94 includes the protrusion direction RD superimposed on the intestinal wall image 41. In this way, the protrusion direction RD is displayed on the screen 36 of the display device 13. This allows the user observing the intestinal wall image 41 to visually grasp the protrusion direction RD of the papilla N.

[0149] Furthermore, in the duodenoscope system 10 according to the second embodiment, the image recognition unit 82B obtains bulging direction information 104 based on the intestinal wall image 41. The bulging direction information 104 is output to the display control unit 82D, and a display image 94 generated by the display control unit 82D is output to the display device 13. The display image 94 includes a display based on the bulging direction information 104. This allows the user observing the intestinal wall image 41 to visually grasp the bulging direction RD of the papilla N.

[0150] Furthermore, in the duodenoscope system 10 according to the second embodiment, the display controller 82D generates a display image 94. The display image 94 includes an image of an arrow indicating the protrusion direction RD. This allows the user observing the intestinal wall image 41 to visualize the protrusion direction RD of the papilla N and visually grasp it.

[0151] In the duodenoscope system 10 according to the second embodiment, the output unit 82C acquires optical axis information 48A from the camera 48. The output unit 82C generates coincidence information 103 based on the optical axis information 48A and the protrusion direction information 104. The display control unit 82D generates a display image 94 based on the coincidence information 103 and outputs it to the display device 13. The display image 94 includes an indication of the degree of coincidence between the optical axis direction indicated by the coincidence information 103 and the protrusion direction RD. This allows the user observing the intestinal wall image 41 to visually grasp the degree of coincidence between the protrusion direction RD of the papilla N and the optical axis direction. For example, if the optical axis and the protrusion direction RD are coincident, it is highly likely that the camera 48 is directly facing the papilla N. Maintaining the endoscope 18 in this position makes it easier to observe the papilla N and to insert a treatment tool into the papilla N.

[0152] Furthermore, in the duodenoscope system 10 according to the second embodiment, in the image recognition process performed by the image recognition unit 82B, the protuberance direction RD is identified as the direction from the apex of the papilla N's papillary protuberance NA toward the apex of the fold H1. A display image 94 generated by the display control unit 82D is then output to the display device 13. The display image 94 includes the protuberance direction RD. This allows the user observing the intestinal wall image 41 to visually grasp the direction from the opening of the papillary protuberance NA toward the apex of the fold H1. As a result, it is possible to easily identify the running direction TD of the bile duct leading to the opening of the papilla N.

[0153] Furthermore, in the duodenoscope system 10 according to the second embodiment, in the image recognition process performed by the image recognition unit 82B, the protuberance direction RD is identified as the direction from the apex of the papilla N's papillary protuberance NA toward the apex of the fold H1. A display image 94 generated by the display control unit 82D is then output to the display device 13. The display image 94 includes an image of an arrow indicating the protuberance direction RD. This allows the user observing the intestinal wall image 41 to visually grasp the direction from the opening of the papillary protuberance NA toward the apex of the fold H1. As a result, it is possible to easily identify the running direction TD of the bile duct leading to the opening of the papilla N.

[0154] Furthermore, in the duodenoscope system 10 according to the second embodiment, the image recognition unit 82B performs image recognition processing on the intestinal wall image 41 to obtain bulging direction information 104 indicating the bulging direction RD. This allows for the obtainment of bulging direction information 104 with higher accuracy than when the user visually specifies the bulging direction RD for the intestinal wall image 41.

[0155] (Fifth Modification) In the second embodiment, the protrusion direction RD is specified as the direction from the apex of the papillary protrusion NA to the apex of the headband fold H1. However, the technology of the present disclosure is not limited to this. The protrusion direction RD is specified based on the configuration of the multiple folds H1 to H3.

[0156] 18 , the image recognition unit 82B acquires a time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84E. As a result, the trained model 84E outputs uplift direction information 104 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the uplift direction information 104 output from the trained model 84E.

[0157] Here, the protruding direction RD of the nipple N is identified as, for example, a direction passing through the top of the headband fold H1. According to medical findings, the protruding direction RD of the nipple N may coincide with the direction passing through the top of the headband fold H1. Therefore, an example of an annotation in the correct answer data is an annotation that defines the direction passing through the top of the headband fold H1 as the protruding direction RD.

[0158] Although the above description has been given with reference to a case in which the rising direction RD is specified as a direction passing through the apex of the headband pleat H1, this is merely an example. The rising direction RD may also be specified as a direction passing through at least one of the apexes of the pleats H1 to H3.

[0159] As described above, in the duodenoscope system 10 according to the fifth modification, the image recognition process in the image recognition unit 82B identifies the protrusion direction RD based on the configuration of the multiple folds H1 to H3. The display control unit 82D then generates a display image 94, which is output to the display device 13. The display image 94 includes the protrusion direction RD. This allows the user observing the intestinal wall image 41 to visually grasp the protrusion direction RD as the direction passing through the apex of the headband fold H1 of the papillary prominence NA.

[0160] In the second embodiment, the protrusion direction RD is determined as the direction from the apex of the nipple protrusion NA to the apex of the headband pleat H1. However, the technology of the present disclosure is not limited to this. In the sixth modification, the protrusion direction RD is determined based on the nipple protrusion NA and the plurality of pleats H1 to H3.

[0161] 19 , the image recognition unit 82B acquires a time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84E. As a result, the trained model 84E outputs uplift direction information 104 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the uplift direction information 104 output from the trained model 84E.

[0162] Here, the protrusion direction RD of the nipple N is identified as, for example, the direction from the top of the nipple protrusion NA through the tops of the headband folds H1, H2, and H3. According to medical findings, the protrusion direction RD of the nipple N may coincide with the direction from the top of the nipple protrusion NA through the tops of the headband folds H1, H2, and H3. Therefore, an example of an annotation in the correct answer data is an annotation in which the direction from the top of the nipple protrusion NA through the tops of the headband folds H1, H2, and H3 is defined as the protrusion direction RD.

[0163] As described above, in the duodenoscope system 10 according to the sixth modification, the image recognition process in the image recognition unit 82B identifies the protuberance direction RD based on the papillary protuberance NA and the multiple folds H1 to H3. The display control unit 82D then generates a display image 94, which is output to the display device 13. The display image 94 includes the protuberance direction RD. This allows the user observing the intestinal wall image 41 to visually grasp the protuberance direction RD as the direction passing through the apex of the papillary protuberance NA and the apexes of the multiple folds H1 to H3.

[0164] In the second embodiment, the bulging direction RD is obtained by image recognition processing of the intestinal wall image 41. However, the technology of the present disclosure is not limited to this. In the seventh modification, the running direction TD of the bile duct is obtained based on the bulging direction RD.

[0165] The image recognition unit 82B performs image recognition processing on the intestinal wall image 41, thereby obtaining bulging direction information 104 and papilla region information 95 (see FIGS. 12 and 15). As an example, as shown in FIG. 20, the derivation unit 82C derives running direction information 96 based on the bulging direction information 104. The running direction TD of the bile duct has a predetermined orientation relationship with the bulging direction RD of the papilla N. Specifically, when the bulging direction RD is set to the 12 o'clock direction, the derivation unit 82C derives the running direction TD as the 11 o'clock direction.

[0166] The display control unit 82D acquires the traveling direction information 96 from the derivation unit 82C. The display control unit 82D generates a display image 94 in which the traveling direction TD indicated by the traveling direction information 96 is superimposed on the intestinal wall image 41 acquired from the image acquisition unit 82A (see FIG. 6 ), and outputs the display image 94 to the display device 13. The display device 13 displays the intestinal wall image 41 with the traveling direction TD superimposed on the screen 36.

[0167] As described above, in the duodenoscope system 10 according to the seventh modification, the running direction information 96 is obtained in the lead-out unit 82C based on the protrusion direction information 104. In this way, since the running direction information 96 is obtained from the protrusion direction information 104, it is easier to identify the running direction TD compared to when the running direction information 96 is obtained by image recognition processing.

[0168] Furthermore, in the duodenoscope system 10 according to the seventh modification, the display controller 82D generates a display image 94. The display image 94 includes an image indicating the running direction TD. This allows the user observing the intestinal wall image 41 to visually grasp the running direction TD of the bile duct.

[0169] (Eighth Modification) In the second embodiment, an example was described in which the protrusion direction RD of the papilla N was obtained by image recognition processing of the intestinal wall image 41. However, the technology of the present disclosure is not limited to this. In this eighth modification, the direction MD of the plane in which the opening of the papilla N exists (hereinafter also simply referred to as the "plane direction MD") is obtained by image recognition processing of the intestinal wall image 41.

[0170] As an example, as shown in FIG. 21, the image acquisition unit 82A updates the time-series image group 89 in a FIFO manner every time an intestinal wall image 41 is acquired from the camera 48.

[0171] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84F. As a result, the trained model 84F outputs face orientation information 106 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the face orientation information 106 output from the trained model 84F. Here, the face orientation information 106 is information that can identify the face orientation MD (for example, a group of position coordinates of axes that indicate the face orientation MD). The face orientation information 106 is an example of "nipple orientation-related information" and "face orientation information" according to the technology of the present disclosure.

[0172] The trained model 84F is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify the plane direction MD.

[0173] The derivation unit 82C derives the relative angle between the plane P on which the opening K of the papilla N is provided and the orientation of the endoscope 18. The relative angle between the plane P on which the opening K of the papilla N is provided and the orientation of the endoscope 18 approaching 0 means that the camera 48 is approaching a state in which it is directly facing the papilla N. Therefore, the derivation unit 82C acquires plane direction information 106 from the image recognition unit 82B. The derivation unit 82C also acquires orientation information 91 from the optical fiber sensor 18A of the endoscope 18. The derivation unit 82C then generates relative angle information 108 by comparing the plane P on which the opening K is located based on the orientation of the surface indicated by the plane direction information 106 with the orientation of the endoscope 18 indicated by the orientation information 91. The relative angle information 108 is information indicating the angle A formed between the plane P and the orientation of the endoscope 18 (e.g., the imaging plane of the camera 48). The relative angle information 108 is an example of “angle-related information” according to the technology of the present disclosure.

[0174] 22 , the display control unit 82D acquires surface direction information 106 from the image recognition unit 82B. The display control unit 82D also acquires relative angle information 108 from the derivation unit 82C. The display control unit 82D generates an operation instruction image 93D (e.g., an arrow indicating an operation direction) for orienting the camera 48 directly toward the papilla N, according to the angle indicated by the relative angle information 108. The display control unit 82D then generates a display image 94 including the surface direction MD indicated by the surface direction information 106, the operation instruction image 93D, and the intestinal wall image 41, and outputs the display image 94 to the display device 13. In the example shown in FIG. 22 , the display device 13 displays the intestinal wall image 41 with the surface direction MD and the operation instruction image 93D superimposed on the screen 36.

[0175] As described above, in the duodenoscope system 10 according to the eighth modification, the image recognition unit 82B of the processor 82 performs image recognition processing on the intestinal wall image 41, and as a result of the image recognition processing, the plane direction MD of the papilla N in the intestinal wall image 41 is detected. Then, plane direction information 106 indicating the plane direction MD is output to the display control unit 82D, and a display image 94 generated by the display control unit 82D is output to the display device 13. The display image 94 includes the plane direction MD superimposed on the intestinal wall image 41. In this way, the plane direction MD is displayed on the screen 36 of the display device 13. This allows the user observing the intestinal wall image 41 to visually grasp the plane direction MD of the papilla N.

[0176] Furthermore, in the duodenoscope system 10 according to the eighth modification, the derivation unit 82C acquires, from the optical fiber sensor 18A, posture information 91, which is information that can identify the posture of the endoscope 18. The derivation unit 82C also generates relative angle information 108 based on the posture information 91 and the planar direction information 106. The display control unit 82D then generates an operation instruction image 93D for orienting the camera 48 directly toward the papilla N based on the relative angle information 108. The display control unit 82D outputs the operation instruction image 93D to the display device 13, which then displays the operation instruction image 93D superimposed on the intestinal wall image 41. This makes it easy for the user to adjust the posture of the endoscope 18 relative to the planar direction MD of the papilla N to the posture intended by the user when the endoscope 18 is inserted into the duodenum.

[0177] In the second embodiment, the protrusion direction RD obtained by image recognition processing of the intestinal wall image 41 is displayed. However, the technology of the present disclosure is not limited to this. In this ninth modification, a nipple surface image 93E is displayed.

[0178] As an example, as shown in FIG. 23 , the display control unit 82D acquires uplift direction information 104 from the image recognition unit 82B. The display control unit 82D generates a nipple surface image 93E based on the uplift direction RD indicated by the uplift direction information 104. The nipple surface image 93E is an image that can identify a surface that intersects with the uplift direction RD at a predetermined angle (e.g., 90 degrees). The nipple surface image 93E is an example of "nipple orientation-related information" and "surface image" according to the technology disclosed herein. Furthermore, the display control unit 82D adjusts the nipple surface image 93E to a size and shape corresponding to the nipple region N1 based on the nipple region information 95 obtained by the image recognition unit 82B. The display control unit 82D also generates an operation instruction image 93C.

[0179] Then, the display control unit 82D generates a display image 94 including the nipple surface image 93E, the operation instruction image 93C, and the intestinal wall image 41, and outputs it to the display device 13. In the example shown in Fig. 23 , the display device 13 displays the intestinal wall image 41 on the screen 36, with the nipple surface image 93E and the operation instruction image 93C superimposed thereon.

[0180] As described above, in the duodenoscope system 10 according to the ninth modification, the display controller 82D generates a nipple surface image 93E based on the protrusion direction information 104. The display controller 82D outputs the nipple surface image 93E to the display device 13, which displays the nipple surface image 93E superimposed on the intestinal wall image 41. This allows the user observing the intestinal wall image 41 to easily visually estimate the position of the opening contained in the papilla N.

[0181] Third Embodiment In the first embodiment, the intestinal direction CD is obtained by image recognition processing of the intestinal wall image 41, and in the second embodiment, the protrusion direction RD is obtained by image recognition processing of the intestinal wall image 41. However, the technology of the present disclosure is not limited to this. In the third embodiment, the running direction TD of the bile duct T is obtained by image recognition processing of the intestinal wall image 41.

[0182] For example, in an ERCP examination, a treatment tool (e.g., a cannula) may be inserted into the papilla N, and the treatment tool may then be intubated into the bile duct T or pancreatic duct S inside the papilla N. In this case, it is difficult to determine the running direction of the bile duct T or pancreatic duct S present inside the papilla N from the intestinal wall image 41. Therefore, in the third embodiment, the running direction of the bile duct T or pancreatic duct S is obtained by image recognition processing of the intestinal wall image 41. Note that, for convenience of explanation, the following description will be given taking the case of the bile duct T as an example.

[0183] As an example, as shown in FIG. 24, the image acquisition unit 82A updates the time-series image group 89 in a FIFO manner every time an intestinal wall image 41 is acquired from the camera 48.

[0184] The image recognition unit 82B acquires a time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84G. As a result, the trained model 84G outputs driving direction information 96 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the driving direction information 96 output from the trained model 84E. The driving direction information 96 is an example of "driving direction information" according to the technology of the present disclosure.

[0185] The trained model 84G is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify the traveling direction TD.

[0186] Here, the running direction TD of the bile duct T is identified as, for example, a direction passing through the apexes of the multiple folds of the papilla N. This is because, according to medical findings, the running direction of the bile duct T may coincide with a line connecting the apexes of the folds. Therefore, an example of an annotation in the correct answer data is an annotation that defines the direction passing through the apexes of the folds of the papilla N as the running direction TD of the bile duct T.

[0187] The acquired time-series image group 89 is input to the trained model 84H. As a result, the trained model 84H outputs diverticulum region information 110 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the diverticulum region information 110 output from the trained model 84H. The diverticulum region information 110 is information (coordinates indicating the size and position of the diverticulum) that can identify an area indicating a diverticulum present in the papilla N. Here, a diverticulum is a region in which a part of the papilla N protrudes in a pouch-like manner outside the duodenum.

[0188] The trained model 84H is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify an area indicating a diverticulum.

[0189] The derivation unit 82C derives a display mode for the traveling direction TD. For example, the traveling direction TD is specified in a mode that avoids diverticula. This is because, according to medical findings, the traveling direction TD may be formed so as to avoid diverticula. Therefore, the derivation unit 82C changes the display mode of the traveling direction TD based on the diverticulum region information 110. Specifically, the derivation unit 82C changes the display mode of the traveling direction TD indicated by the traveling direction information 96 to a mode that avoids the diverticula, for a portion that intersects with the diverticula indicated by the diverticulum region information 110. In this way, the derivation unit 82C generates display mode information 112 that indicates the display mode of the changed traveling direction TD.

[0190] 25 as an example, the display control unit 82D acquires display mode information 112 from the derivation unit 82C. The display control unit 82D generates a display image 94 including the changed traveling direction TD indicated by the display mode information 112 and an intestinal wall image 41, and outputs the display image 94 to the display device 13. In the example shown in FIG. 25 , the display device 13 displays the intestinal wall image 41 with the changed traveling direction TD superimposed on the screen 36.

[0191] Next, the operation of the portion of the duodenoscope system 10 related to the technology of the present disclosure will be described with reference to FIG.

[0192] Fig. 26 shows an example of the flow of medical support processing performed by the processor 82. The flow of medical support processing shown in Fig. 26 is an example of a "medical support method" according to the technique of the present disclosure.

[0193] 26, first, in step ST110, the image acquisition unit 82A determines whether or not one frame of image data has been captured by the camera 48 provided on the endoscope 18. If one frame of image data has not been captured by the camera 48 in step ST10, the determination is negative, and the determination in step ST110 is made again. If one frame of image data has been captured by the camera 48 in step ST110, the determination is positive, and the medical support process proceeds to step ST112.

[0194] In step ST112, the image acquisition unit 82A acquires one frame of the intestinal wall image 41 from the camera 48 provided in the endoscope 18. After the processing of step ST112 is executed, the medical support processing proceeds to step ST114.

[0195] In step ST114, the image recognition unit 82B detects the traveling direction TD by performing AI image recognition processing (i.e., image recognition processing using the trained model 84G) on the intestinal wall image 41 acquired in step ST112. After the processing of step ST114 is executed, the medical support processing proceeds to step ST116.

[0196] In step ST116, the image recognition unit 82B detects the diverticulum region by performing AI image recognition processing (i.e., image recognition processing using the trained model 84H) on the intestinal wall image 41 acquired in step ST112. After the processing of step ST116 is executed, the medical support processing proceeds to step ST118.

[0197] In step ST118, the derivation unit 82C changes the display mode of the traveling direction TD based on the traveling direction TD obtained by the image recognition unit 82B in step ST114 and the diverticulum region obtained by the image recognition unit 82B in step ST116. Specifically, the derivation unit 82C changes the display mode of the traveling direction TD to avoid the diverticulum region. After the processing of step ST118 is executed, the medical support processing proceeds to step ST120.

[0198] In step ST120, the display control unit 82D generates a display image 94 in which the traveling direction TD, the display mode of which has been changed by the derivation unit 82C in step ST118, is superimposed on the intestinal wall image 41. After the processing of step ST120 is executed, the medical support processing proceeds to step ST122.

[0199] In step ST122, the display control unit 82D outputs the display image 94 generated in step ST120 to the display device 13. After the processing in step ST122 is executed, the medical support processing proceeds to step ST124.

[0200] In step ST124, the display control unit 82D determines whether a condition for terminating the medical support process is satisfied. One example of the condition for terminating the medical support process is that an instruction to terminate the medical support process has been given to the duodenoscope system 10 (for example, that an instruction to terminate the medical support process has been accepted by the acceptance device 62).

[0201] In step ST124, if the condition for terminating the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST110. In step ST124, if the condition for terminating the medical support process is satisfied, the determination is positive, and the medical support process ends.

[0202] As described above, in the duodenoscope system 10 according to the third embodiment, the image recognition unit 82B of the processor 82 performs image recognition processing on the intestinal wall image 41, and as a result of the image recognition processing, the running direction TD of the bile duct in the intestinal wall image 41 is detected. Then, running direction information 96 indicating the running direction TD is output to the display control unit 82D, and a display image 94 generated by the display control unit 82D is output to the display device 13. The display image 94 includes the running direction TD superimposed on the intestinal wall image 41. In this way, the running direction TD is displayed on the screen 36 of the display device 13. This allows the user observing the intestinal wall image 41 to visually grasp the running direction TD of the bile duct.

[0203] Furthermore, in the duodenoscope system 10 according to the third embodiment, the image recognition unit 82B performs image recognition processing on the intestinal wall image 41 to obtain diverticulum region information 110. The derivation unit 82C generates display mode information 112 based on the running direction information 96 and the diverticulum region information 110. The display mode information 112 indicating the changed running direction TD is then output to the display control unit 82D, and the display image 94 generated by the display control unit 82D is output to the display device 13. The display image 94 includes the changed running direction TD superimposed on the intestinal wall image 41. In this manner, the changed running direction TD is displayed on the screen 36 of the display device 13. This allows the user observing the intestinal wall image 41 to visually understand the bile duct running direction TD, which has been changed in response to the presence of a diverticulum. For example, this prevents the user observing the intestinal wall image 41 from visually misunderstanding the bile duct running direction TD leading to the opening of the papilla N due to the presence of a diverticulum.

[0204] Furthermore, in the duodenoscope system 10 according to the third embodiment, the display mode information 112 in the lead-out unit 82C indicates the traveling direction TD indicated by the traveling direction information 96, which has been changed to avoid the diverticulum. The changed traveling direction TD is then displayed on the screen 36 of the display device 13. This allows the user observing the intestinal wall image 41 to visually understand the traveling direction TD of the bile duct that has been changed to avoid the diverticulum.

[0205] In the third embodiment, the display mode of the bile duct running direction TD is changed to avoid the diverticulum, but the technology of the present disclosure is not limited to this. For example, the area of ​​the bile duct running direction TD that intersects with the diverticulum may be hidden, or the area may be displayed as a dashed line or semi-transparent.

[0206] In the third embodiment, an example is described in which a diverticulum is detected from the intestinal wall image 41 by image recognition processing and the display mode of the traveling direction TD is changed depending on the diverticulum, but the technology of the present disclosure is not limited to this. For example, a mode in which a diverticulum is not detected may also be used.

[0207] In the third embodiment, an example in which the running direction TD of the bile duct is displayed while avoiding a diverticulum has been described, but the technology of the present disclosure is not limited to this. In the tenth modification, when the running direction TD of the bile duct intersects with a diverticulum, the user is notified of this.

[0208] As an example, as shown in FIG. 27 , the derivation unit 82C acquires traveling direction information 96 and diverticulum region information 110 from the image recognition unit 82B. The derivation unit 82C determines the positional relationship between the diverticulum and the traveling direction TD based on the diverticulum region information 110 and the traveling direction information 96. Specifically, the derivation unit 82C compares the traveling direction TD indicated by the traveling direction information 96 with the position and size of the diverticulum indicated by the diverticulum region information 110 to determine whether the diverticulum intersects with the traveling direction TD. Then, when the derivation unit 82C determines that the traveling direction TD and the diverticulum intersect with each other, it generates notification information 114. The notification information 114 is an example of "notification information" according to the technology of the present disclosure.

[0209] The derivation unit 82C outputs the notification information 114 to the display control unit 82D. In this case, the display control unit 82D generates a display image 94 including content notifying the user that the diverticulum indicated by the notification information 114 intersects with the traveling direction TD. In the example shown in Fig. 27, a message stating "The diverticulum intersects with the traveling direction" is displayed on the screen 37 of the display device 13.

[0210] As described above, in the duodenoscope system 10 according to the tenth modification, the lead-out unit 82C identifies the positional relationship between the diverticulum and the traveling direction TD based on the diverticulum region information 110 and the traveling direction information 96, and generates notification information 114 based on the identification result. The display control unit 82D generates a display image 94 based on the notification information 114 and outputs it to the display device 13. The display image 94 includes an indication that the diverticulum indicated by the notification information 114 intersects with the traveling direction. This allows the user to perceive that the diverticulum intersects with the traveling direction. For example, it is possible to prevent a user observing the intestinal wall image 41 from visually misunderstanding the traveling direction TD of the bile duct leading to the opening of the papilla N due to the presence of a diverticulum.

[0211] Fourth Embodiment In the first to third embodiments, examples have been described in which information about biological tissue, such as the intestinal tract direction CD, the papilla N, and the running direction TD of the bile duct, is identified by image recognition processing on the intestinal wall image 41. However, the technology of the present disclosure is not limited to this. In this fourth embodiment, the relationship between the treatment tool and the biological tissue is identified by performing image recognition processing on the intestinal wall image 41.

[0212] For example, in an ERCP examination, various treatments using a treatment tool may be performed on the papilla N (e.g., inserting a cannula into the papilla N). In this case, the positional relationship between the papilla N and the treatment tool affects the success of the procedure. For example, if the direction of advancement of the treatment tool does not match the papilla direction ND, the treatment tool will not be able to properly approach the papilla N, making it difficult to successfully perform the procedure. Therefore, in the fourth embodiment, the positional relationship between the treatment tool and the papilla N is identified by image recognition processing of the intestinal wall image 41.

[0213] As an example, as shown in FIG. 28, the image acquisition unit 82A updates the time-series image group 89 in a FIFO manner every time an intestinal wall image 41 is acquired from the camera 48.

[0214] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84I. As a result, the trained model 84I outputs positional relationship information 116 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the positional relationship information 116 output from the trained model 84I. Here, the positional relationship information 116 is information that can identify the position of the papilla N and the position of the treatment tool (for example, the distance and angle between the position of the papilla N and the position of the tip of the treatment tool).

[0215] The trained model 84I is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify the position of the papilla N and the position of a treatment tool.

[0216] The derivation unit 82C acquires positional relationship information 116 from the image recognition unit 82B. Based on the positional relationship information 116, the derivation unit 82C generates notification information 118, which is information that notifies the user of the positional relationship between the nipple N and the treatment tool. The derivation unit 82C compares the position of the treatment tool indicated by the positional relationship information 116 with the position of the nipple N. If the position of the treatment tool and the position of the nipple N match, the derivation unit 82C generates notification information 118 indicating that the position of the treatment tool and the position of the nipple N match. Furthermore, if the position of the treatment tool and the position of the nipple N do not match, the derivation unit 82C generates notification information 118 indicating that the position of the treatment tool and the position of the nipple N do not match.

[0217] Although the case where the position of the treatment tool and the position of the papilla N coincide with each other has been described here as an example, this is merely an example. For example, it may be determined whether the position of the treatment tool and the position of the papilla N are within a predetermined range (for example, within a predetermined distance and angle range).

[0218] As an example, as shown in Fig. 29 , the display control unit 82D acquires notification information 118 from the derivation unit 82C. The derivation unit 82C outputs the notification information 118 to the display control unit 82D. In this case, the display control unit 82D generates a display image 94 including content notifying the user of the positional relationship between the treatment tool and the papilla N indicated by the notification information 118. In the example shown in Fig. 29 , a message stating "The position of the treatment tool and the position of the papilla match" is displayed on the screen 37 of the display device 13.

[0219] As described above, in the duodenoscope system 10 according to the fourth embodiment, the image recognition unit 82B of the processor 82 performs image recognition processing on the intestinal wall image 41 to identify the positional relationship between the treatment tool and the papilla. The derivation unit 82C determines the positional relationship between the treatment tool and the papilla N based on positional relationship information 116 indicating the positional relationship between the treatment tool and the papilla, and generates notification information 118 based on the determination result. The display control unit 82D generates a display image 94 based on the notification information 118 and outputs it to the display device 13. The display image 94 includes an indication of the positional relationship between the treatment tool and the papilla N indicated by the notification information 118. This allows the user observing the intestinal wall image 41 to perceive the relationship between the position of the treatment tool and the position of the papilla N.

[0220] In the fourth embodiment, the relationship between the position of the treatment tool and the nipple N is specified by the relationship between the position of the nipple N and the position of the treatment tool. However, the technology of the present disclosure is not limited to this. In this eleventh modification, the relationship between the direction of advancement of the treatment tool and the nipple direction ND is specified.

[0221] 30 , the image recognition unit 82B acquires a time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84J. As a result, the trained model 84J outputs positional relationship information 116A corresponding to the input time-series image group 89. Here, the positional relationship information 116A is information that can identify the nipple direction ND and the traveling direction of the treatment tool (for example, the angle formed between the nipple direction ND and the traveling direction of the treatment tool).

[0222] The trained model 84J is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify the relationship between the papilla direction (ND) and the direction of travel of the treatment tool.

[0223] The derivation unit 82C acquires positional relationship information 116A from the image recognition unit 82B. Based on the positional relationship information 116A, the derivation unit 82C generates notification information 118, which is information that notifies the user of the positional relationship between the nipple N and the treatment tool. If the angle formed by the nipple direction ND and the direction of travel of the treatment tool is within a predetermined range, the derivation unit 82C generates notification information 118 that the two match. Furthermore, if the angle formed by the nipple direction ND and the direction of travel of the treatment tool exceeds the predetermined range, the derivation unit 82C generates notification information 118 that the two do not match.

[0224] As described above, in the duodenoscope system 10 according to the eleventh modification, the image recognition unit 82B identifies the relationship between the treatment tool's traveling direction and the papilla direction ND. The output unit 82C generates notification information 118 based on positional relationship information 116A indicating the relationship between the treatment tool's traveling direction and the papilla direction ND. This allows the user observing the intestinal wall image 41 to perceive the relationship between the treatment tool's traveling direction and the papilla direction ND.

[0225] Although the eleventh modification example has been described above as an example in which the image recognition unit 82B determines the relationship between the treatment tool's traveling direction and the nipple direction ND, the technology of the present disclosure is not limited to this. For example, the image recognition unit 82B may determine the relationship between the position of the nipple N and the position of the treatment tool, in addition to the relationship between the treatment tool's traveling direction and the nipple direction ND. In this case, the positional relationship information 116A is information indicating the relationship between the treatment tool's traveling direction and the nipple direction ND and the relationship between the position of the nipple N and the position of the treatment tool. The derivation unit 82C determines the relationship between the treatment tool's traveling direction and the nipple direction ND and the relationship between the position of the nipple N and the position of the treatment tool based on the positional relationship information 116A. Furthermore, the derivation unit 82C generates notification information 118 based on these determination results.

[0226] In the fourth embodiment, the relationship between the position of the treatment tool and the papilla N is specified by the relationship between the position of the papilla N and the position of the treatment tool. However, the technology of the present disclosure is not limited to this. In this eleventh modification, the relationship between the direction of advancement of the treatment tool and the running direction TD of the bile duct is specified.

[0227] 31 , the image recognition unit 82B acquires a time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84K. As a result, the trained model 84K outputs positional relationship information 116B corresponding to the input time-series image group 89. Here, the positional relationship information 116B is information that can identify the relationship between the running direction TD of the bile duct and the traveling direction of the treatment tool (for example, the angle between the direction of the tangent to the opening end in the running direction TD of the bile duct (hereinafter simply referred to as the "bile duct tangent direction") and the traveling direction of the treatment tool).

[0228] The trained model 84K is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify the relationship between the running direction TD of the bile duct and the traveling direction of the treatment tool.

[0229] The derivation unit 82C acquires positional relationship information 116B from the image recognition unit 82B. Based on the positional relationship information 116B, the derivation unit 82C generates notification information 118, which is information that notifies the user of the relationship between the running direction TD of the bile duct and the traveling direction of the treatment tool. If the angle formed between the bile duct tangential direction and the traveling direction of the treatment tool is within a predetermined range, the derivation unit 82C generates notification information 118 that the two coincide. Furthermore, if the angle formed between the bile duct tangential direction and the traveling direction of the treatment tool exceeds the predetermined range, the derivation unit 82C generates notification information 118 that the two do not coincide.

[0230] As described above, in the duodenoscope system 10 according to the twelfth modification, the image recognition unit 82B identifies the relationship between the treatment tool's traveling direction and the bile duct's running direction TD. The output unit 82C generates notification information 118 based on positional relationship information 116B indicating the relationship between the treatment tool's traveling direction and the bile duct's running direction TD. This allows the user observing the intestinal wall image 41 to perceive the relationship between the treatment tool's traveling direction and the bile duct's running direction TD.

[0231] (Thirteenth Modification) In the fourth embodiment, an example was given in which the relationship between the position of the papilla N and the position of the treatment tool was specified as the positional relationship between the treatment tool and the papilla N, but the technology of the present disclosure is not limited to this. In this thirteenth modification, the relationship between the direction of advancement of the treatment tool and the orientation of a plane perpendicular to the protrusion direction RD of the papillary prominence NA (hereinafter also simply referred to as the "vertical plane orientation") is specified.

[0232] 32 , the image recognition unit 82B acquires a time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84L. As a result, the trained model 84L outputs positional relationship information 116C corresponding to the input time-series image group 89. Here, the positional relationship information 116C is information that can identify the relationship between the orientation of the vertical surface and the traveling direction of the treatment tool (for example, the angle between the orientation of the vertical surface and the traveling direction of the treatment tool).

[0233] The trained model 84L is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify the relationship between the vertical surface orientation and the direction of travel of the treatment tool.

[0234] The derivation unit 82C acquires positional relationship information 116C from the image recognition unit 82B. Based on the positional relationship information 116C, the derivation unit 82C generates notification information 118, which is information that notifies the user of the relationship between the vertical surface orientation and the treatment tool traveling direction. If the angle formed between the vertical surface orientation and the treatment tool traveling direction is within a predetermined range, the derivation unit 82C generates notification information 118 that the two match. Furthermore, if the angle formed between the vertical surface orientation and the treatment tool traveling direction exceeds the predetermined range, the derivation unit 82C generates notification information 118 that the two do not match.

[0235] As described above, in the duodenoscope system 10 according to the thirteenth modification, the image recognition unit 82B identifies the relationship between the vertical orientation and the direction of travel of the treatment tool. The output unit 82C generates notification information 118 based on positional relationship information 116B indicating the relationship between the vertical orientation and the direction of travel of the treatment tool. This allows the user observing the intestinal wall image 41 to perceive the relationship between the vertical orientation and the direction of travel of the treatment tool.

[0236] (14th Modification) In the above-described fourth embodiment, an example was given in which the positional relationship between the treatment tool and the papilla N was identified by performing image recognition processing on the intestinal wall image 41. However, the technology of the present disclosure is not limited to this. In this 14th Modification, an evaluation value regarding the positional relationship between the treatment tool and the papilla N is acquired by performing image recognition processing on the intestinal wall image 41.

[0237] As an example, as shown in FIG. 33, the image acquisition unit 82A updates the time-series image group 89 in a FIFO manner every time an intestinal wall image 41 is acquired from the camera 48.

[0238] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84M. As a result, the trained model 84M outputs evaluation value information 120 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the evaluation value information 120 output from the trained model 84M. Here, the evaluation value information 120 is information that can identify an evaluation value regarding the appropriate positioning of the papilla N and the treatment tool (e.g., the degree of success of the procedure determined depending on the positioning of the papilla N and the treatment tool). The evaluation value information 120 is, for example, a plurality of scores (scores for each success or failure of the procedure) that are input to an activation function (e.g., a softmax function) of the output layer of the trained model 84M.

[0239] The trained model 84M is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify an evaluation value regarding the appropriate positioning of the papilla N and the treatment tool (e.g., an annotation indicating the success or failure of the procedure).

[0240] The image recognition unit 82B also inputs the time-series image group 89 to the trained model 84N. As a result, the trained model 84N outputs contact presence / absence information 122 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the contact presence / absence information 122 output from the trained model 84N. Here, the contact presence / absence information 122 is information that can identify the presence or absence of contact between the papilla N and the treatment tool.

[0241] The trained model 84N is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify whether or not the papilla N is in contact with a treatment tool.

[0242] The derivation unit 82C acquires contact presence / absence information 122 from the image recognition unit 82B. Based on the contact presence / absence information 122, the derivation unit 82C determines whether or not contact between the treatment tool and the papilla N has been detected. When contact between the treatment tool and the papilla N has been detected, the derivation unit 82C generates notification information 124 based on the evaluation value information 120. The notification information 124 is information for notifying the user of the success probability of the procedure (for example, text indicating the success probability of the procedure).

[0243] As an example, as shown in Fig. 34 , the display control unit 82D acquires notification information 124 from the derivation unit 82C. The derivation unit 82C outputs the notification information 124 to the display control unit 82D. In this case, the display control unit 82D generates a display image 94 including content notifying the user of the success probability of the procedure indicated by the notification information 124. In the example shown in Fig. 34 , a message stating "Probability of successful cannulation: 90%" is displayed on the screen 37 of the display device 13.

[0244] As described above, in the duodenoscope system 10 according to the fourteenth modification, the image recognition unit 82B of the processor 82 performs image recognition processing on the intestinal wall image 41 to calculate an evaluation value related to the arrangement of the treatment tool and the papilla N. The derivation unit 82C generates notification information 124 based on evaluation value information 120 indicating the evaluation value. The display control unit 82D generates a display image 94 based on the notification information 124 and outputs it to the display device 13. The display image 94 includes an indication of the success probability of the procedure indicated by the notification information 124. This makes it possible to notify the user observing the intestinal wall image 41 of the success probability of the procedure using the treatment tool. After understanding the success probability of the procedure, the user can consider continuing or changing the operation, thereby supporting the success of the procedure using the treatment tool.

[0245] Furthermore, in the duodenoscope system 10 according to the fourteenth modification, the image recognition unit 82B performs image recognition processing on the intestinal wall image 41 to determine whether or not the treatment tool is in contact with the papilla N. Then, in the derivation unit 82C, if the treatment tool is in contact with the papilla N based on the contact presence / absence information 122, notification information 124 is generated based on the evaluation value information 120. This allows the user observing the intestinal wall image 41 to be notified of the success rate of the procedure using the treatment tool only when necessary. In other words, the procedure on the papilla N using the treatment tool can be supported at the appropriate time.

[0246] Fifth Embodiment In the above-described fourth embodiment, an example was given in which the positional relationship between the treatment tool and the papilla N was identified by performing image recognition processing on the intestinal wall image 41. However, the technology of the present disclosure is not limited to this. In the fifth embodiment, when the treatment tool is an incision tool, the incision direction is obtained based on the result of image recognition processing on the intestinal wall image 41.

[0247] For example, in an ERCP examination, an incision instrument (e.g., a papillotomy knife) may be used as a treatment tool. The incision instrument is used to cut open the papilla N, thereby facilitating the insertion of a treatment tool into the papilla N and the removal of foreign bodies from the bile duct T or the pancreatic duct S. In this case, selecting the wrong direction (i.e., incision direction) for cutting open the papilla N with the incision instrument may result in unintended bleeding, making it difficult to successfully complete the procedure. Therefore, in the fifth embodiment, an image recognition process is performed on the intestinal wall image 41 to identify a recommended incision direction (i.e., recommended incision direction).

[0248] As an example, as shown in FIG. 35, the image acquisition unit 82A updates the time-series image group 89 in a FIFO manner every time an intestinal wall image 41 is acquired from the camera 48.

[0249] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the trained model 84E. As a result, the trained model 84E outputs uplift direction information 104 corresponding to the input time-series image group 89.

[0250] The derivation unit 82C acquires the protuberance direction information 104 from the image recognition unit 82B. Then, the derivation unit 82C derives recommended incision direction information 126 based on the protuberance direction information 104. The recommended incision direction information 126 is information that can identify the recommended incision direction (for example, a group of position coordinates of the start point and end point of the recommended incision direction). The derivation unit 82C derives the recommended incision direction, for example, from a predetermined orientation relationship between the protuberance direction RD and the recommended incision direction. Specifically, when the protuberance direction RD is set to the 12 o'clock direction, the derivation unit 82C derives the recommended incision direction as the 11 o'clock direction. The recommended incision direction information 126 is an example of "recommended incision direction information" according to the technology of the present disclosure.

[0251] 36 as an example, the display control unit 82D acquires recommended incision direction information 126 from the derivation unit 82C. The display control unit 82D generates an incision direction image 93F, which is an image indicating the incision direction, based on the incision direction indicated by the recommended incision direction information 126. The display control unit 82D then generates a display image 94 including the incision direction image 93F and the intestinal wall image 41, and outputs it to the display device 13. In the example shown in FIG. 36 , the display device 13 displays the intestinal wall image 41 with the incision direction image 93F superimposed on the screen 36.

[0252] As described above, in the duodenoscope system 10 according to the fifth embodiment, the output unit 82C generates recommended incision direction information 126. The display control unit 82D generates a display image 94 based on the recommended incision direction information 126 and outputs it to the display device 13. The display image 94 includes an incision direction image 93F indicating the recommended incision direction indicated by the recommended incision direction information 126. This allows the user observing the intestinal wall image 41 to understand the recommended incision direction. As a result, successful incision of the papilla N can be supported.

[0253] (Fifteenth Modification) Note that, in the above fifth embodiment, an example in which a recommended incision direction is identified has been described, but the technology of the present disclosure is not limited to this. In this fifteenth modification, a direction that is not recommended as an incision direction (i.e., a non-recommended incision direction) may be identified.

[0254] As an example, as shown in FIG. 37 , the derivation unit 82C derives non-recommended incision direction information 127. The non-recommended incision direction information 127 is information that can identify a non-recommended incision direction (e.g., an angle indicating a direction other than the recommended incision direction). The derivation unit 82C derives the recommended incision direction, for example, from a predetermined orientation relationship between the protrusion direction RD and the recommended incision direction. Specifically, when the protrusion direction RD is set to the 12 o'clock direction, the derivation unit 82C derives the recommended incision direction as the 11 o'clock direction. Then, the derivation unit 82C identifies a range excluding a predetermined angle range including the recommended incision direction (e.g., a range of ±5 degrees around the recommended incision direction) as the non-recommended incision direction. The non-recommended incision direction information 127 is an example of "non-recommended incision direction information" according to the technology of the present disclosure.

[0255] The display control unit 82D acquires non-recommended incision direction information 127 from the derivation unit 82C. The display control unit 82D generates a non-recommended incision direction image 93G, which is an image indicating a non-recommended incision direction, based on the non-recommended incision direction indicated by the non-recommended incision direction information 127. The display control unit 82D then generates a display image 94 including the non-recommended incision direction image 93G and the intestinal wall image 41, and outputs it to the display device 13. In the example shown in Figure 37, the display device 13 displays the intestinal wall image 41 with the non-recommended incision direction image 93G superimposed on the screen 36.

[0256] As described above, in the duodenoscope system 10 according to the fifteenth modification, the output unit 82C generates non-recommended incision direction information 127. The display control unit 82D generates a display image 94 based on the non-recommended incision direction information 127 and outputs it to the display device 13. The display image 94 includes a non-recommended incision direction image 93G indicating the non-recommended incision direction indicated by the non-recommended incision direction information 127. This allows the user observing the intestinal wall image 41 to understand the non-recommended incision direction. As a result, successful incision of the papilla N can be supported.

[0257] In the above-described embodiments, an example has been described in which an image of an arrow indicating the operation direction is displayed on the screen 36 as a mode of indicating the operation direction to the user. However, the technology of the present disclosure is not limited to this. For example, the image indicating the operation direction to the user may be an image of a triangle indicating the operation direction. Alternatively, a message indicating the operation direction may be displayed instead of or together with the image indicating the operation direction. Furthermore, the image indicating the operation direction may not be displayed on the screen 36, but may be displayed in a separate window or a separate display device.

[0258] Furthermore, in each of the above embodiments, an example in which the bile duct direction TD is indicated has been described, but the technology of the present disclosure is not limited to this. Instead of the bile duct direction TD, or together with the bile duct direction TD, the running direction of the pancreatic duct S may be indicated.

[0259] Furthermore, in each of the above embodiments, various information is output to the display device 13. However, the technology of the present disclosure is not limited to this. For example, instead of the display device 13, or together with the display device 13, the information may be output to an audio output device such as a speaker (not shown), or may be output to a printing device such as a printer (not shown).

[0260] In the above embodiments, various information is output to the display device 13 and displayed on the screen 36 of the display device 13. However, the technology of the present disclosure is not limited to this. The various information may be output to an electronic medical record server. The electronic medical record server is a server for storing electronic medical record information indicating the results of medical treatment for a patient. The electronic medical record information includes various information.

[0261] The electronic medical record server is connected to the duodenoscope system 10 via a network. The electronic medical record server acquires the intestinal wall image 41 and various information from the duodenoscope system 10. The electronic medical record server stores the intestinal wall image 41 and various information as part of the medical treatment results indicated by the electronic medical record information.

[0262] The electronic medical record server is also connected via a network to terminals other than the duodenoscope system 10 (for example, personal computers installed in medical facilities). Users such as doctors 14 can access the intestinal wall images 41 and various information stored in the electronic medical record server via their terminals. In this way, since the intestinal wall images 41 and various information are stored in the electronic medical record server, users can access the intestinal wall images 41 and various information.

[0263] In addition, in the above-described embodiments, an example in which AI-based image recognition processing is performed on the intestinal wall image 41 has been described, but the technology of the present disclosure is not limited to this. For example, pattern matching-based image recognition processing may be performed.

[0264] In the above embodiment, an example in which medical support processing is performed by the processor 82 of the computer 76 included in the image processing device 25 has been described, but the technology of the present disclosure is not limited to this. For example, medical support processing may be performed by the processor 70 of the computer 64 included in the control device 22. Furthermore, the device that performs medical support processing may be provided external to the duodenoscope 12. Examples of devices that may be provided external to the duodenoscope 12 include at least one server and / or at least one personal computer that are communicatively connected to the duodenoscope 12. Furthermore, medical support processing may be performed in a distributed manner by multiple devices.

[0265] In the above embodiment, an example has been described in which the medical support processing program 84A is stored in the NVM 84, but the technology of the present disclosure is not limited to this. For example, the medical support processing program 84A may be stored in a portable non-transitory storage medium such as an SSD or USB memory. The medical support processing program 84A stored in the non-transitory storage medium is installed in the computer 76 of the duodenoscope 12. The processor 82 executes medical support processing in accordance with the medical support processing program 84A.

[0266] Alternatively, the medical support processing program 84A may be stored in a storage device such as another computer or server connected to the duodenoscope 12 via a network, and the medical support processing program 84A may be downloaded and installed in the computer 76 in response to a request from the duodenoscope 12.

[0267] It is not necessary to store the entire medical support processing program 84A in the storage device of another computer or server device connected to the duodenoscope 12, or in the NVM 84; only a portion of the medical support processing program 84A may be stored therein.

[0268] The hardware resources that execute the medical support processing can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes medical support processing by executing software, i.e., a program. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with a circuit configuration specifically designed to execute specific processing. Each processor has built-in or connected memory, and executes medical support processing by using the memory.

[0269] The hardware resource for executing the medical support processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the medical support processing may be a single processor.

[0270] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute medical support processing, on a single IC chip, as typified by SoC. In this way, medical support processing is realized using one or more of the various processors described above as hardware resources.

[0271] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0272] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0273] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0274] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0275] The disclosure of Japanese Patent Application No. 2022-177614, filed on November 4, 2022, is incorporated herein by reference in its entirety.

Claims

1. A medical support device comprising a processor, wherein the processor obtains nipple orientation-related information related to the orientation of the major duodenal papilla based on an intestinal wall image obtained by imaging the intestinal wall including the major duodenal papilla in the duodenum with a camera provided on an endoscope scope, displays the intestinal wall image on a screen, displays the nipple orientation-related information on the screen, and the nipple orientation-related information includes elevation direction information indicating the elevation direction of the major duodenal papilla. Medical support device.

2. The nipple orientation-related information includes an elevation direction image indicating the elevation direction. The medical support device according to claim 1.

3. The major duodenal papilla has an opening, and the nipple orientation-related information includes plane direction information indicating the direction of the plane in which the opening is present. The medical support device according to claim 1.

4. The nipple orientation-related information includes angle-related information regarding the relative angle between the plane and the posture of the endoscope scope. The medical support device according to claim 3.

5. The major duodenal papilla has an opening, and the nipple orientation-related information includes plane direction information indicating the direction of the plane in which the opening is present and angle-related information regarding the relative angle between the plane and the posture of the endoscope scope. The medical support device according to claim 1.

6. The nipple orientation-related information includes a plane image capable of specifying a plane that intersects the elevation direction of the major duodenal papilla at a predetermined angle. The medical support device according to claim 1.

7. The nipple orientation-related information includes degree-of-coincidence information capable of specifying the degree of coincidence between the elevation direction of the major duodenal papilla and the optical axis direction of the endoscope scope. The medical support device according to claim 1.

8. The major duodenal papilla includes a nipple elevation and an annular fold covering the nipple elevation, and the nipple orientation-related information includes first direction information indicating a first direction from the top of the nipple elevation to the top of the annular fold. The medical support device according to claim 1.

9. The first direction information includes a first direction image indicating the first direction. The medical support device according to claim 8.

10. The nipple elevation has an opening, and the nipple orientation-related information includes running direction information indicating the running direction of the bile duct or pancreatic duct leading to the opening, wherein the running direction information is determined based on the first direction information. The medical support device according to claim 8.

11. The running direction information includes a running direction image indicating the running direction. The medical support device according to claim 10.

12. The duodenal papilla has a papillary elevation and a fold portion including a circumferential fold covering the papillary elevation. The processor identifies the second direction based on the aspect of the fold portion shown in the intestinal wall image. The medical support device according to claim 1.

13. The processor identifies the second direction based on the aspect of the region including the papillary elevation and the fold portion shown in the intestinal wall image. The medical support device according to claim 12.

14. The processor obtains the papilla orientation-related information by performing a first image recognition process on the intestinal wall image. The medical support device according to claim 1.

15. The processor identifies the running direction of the tube leading to the opening of the duodenal papilla based on the intestinal wall image, and displays running direction information capable of identifying the running direction in the intestinal wall image on the screen. The medical support device according to claim 1.

16. When an endoscope having the endoscope scope and the treatment tool is inserted into the duodenum, the processor identifies a first relationship between the position of the treatment tool and the position of the duodenal papilla, and / or a second relationship between the advancing direction of the treatment tool and the orientation of the duodenal papilla, based on the intestinal wall image showing the treatment tool, and executes a first notification process for performing notification according to the first relationship and / or the second relationship. The medical support device according to claim 1.

17. When an endoscope having the endoscope scope and the treatment tool is inserted into the duodenum, the processor identifies a third relationship between the advancing direction of the treatment tool and a first orientation related to the orientation of the duodenal papilla, based on the intestinal wall image showing the treatment tool, and executes a second notification process for performing notification according to the third relationship. The medical support device according to claim 1.

18. The processor identifies the running direction of the tube leading to the opening of the duodenal papilla based on the intestinal wall image, and when an endoscope having the endoscope scope and the treatment tool is inserted into the duodenum, identifies the advancing direction of the treatment tool based on the intestinal wall image showing the treatment tool, and executes a third notification process for performing notification according to a fourth relationship between the running direction and the advancing direction. The medical support device according to claim 1.

19. The papilla orientation-related information includes incision recommended direction information indicating a direction recommended as an incision direction with respect to the duodenal papilla by an incision instrument for incising the duodenal papilla, or incision non-recommended direction information indicating a direction not recommended as the incision direction. The medical support device according to claim 1.

20. When an endoscope having the endoscope scope and the treatment tool is inserted into the duodenum, The processor, acquires an evaluation value regarding the positional relationship between the duodenal papilla and the treatment tool based on the intestinal wall image in which the treatment tool is shown, outputs information based on the evaluation value The medical support device according to claim 1.

21. When an endoscope having the endoscope scope and the treatment tool is inserted into the duodenum, when the processor detects a state in which the treatment tool is in contact with the duodenal papilla based on the intestinal wall image in which the treatment tool is shown, the processor outputs information based on the evaluation value. The medical support device according to claim 20.

22. Comprising a processor, The processor, identifies the running direction of the bile duct or pancreatic duct leading to the opening of the duodenal papilla based on an intestinal wall image obtained by imaging the intestinal wall including the duodenal papilla in the duodenum with a camera provided on the endoscope scope, displays the intestinal wall image on the screen, displays running direction information capable of identifying the running direction in the intestinal wall image on the screen Medical support device.

23. The medical support device according to any one of claims 1 to 22, and the endoscope scope. Endoscope.

24. acquiring papilla orientation-related information related to the orientation of the duodenal papilla based on an intestinal wall image obtained by imaging the intestinal wall including the duodenal papilla in the duodenum with a camera provided on the endoscope scope, displaying the intestinal wall image on the screen, and displaying the papilla orientation-related information on the screen, wherein the papilla orientation-related information includes elevation direction information indicating the elevation direction of the duodenal papilla Medical support method.

25. identifying the running direction of the bile duct or pancreatic duct leading to the opening of the duodenal papilla based on an intestinal wall image obtained by imaging the intestinal wall including the duodenal papilla in the duodenum with a camera provided on the endoscope scope, displaying the intestinal wall image on the screen, and displaying running direction information capable of identifying the running direction in the intestinal wall image on the screen Medical support method.

Images