Medical assistance device, endoscope, medical assistance method, and program

Abstract

This medical assistance device comprises a processor. The processor acquires first reference site information capable of three-dimensionally identifying a reference site including a duodenum on the basis of a captured image acquired by imaging an intestinal wall of the duodenum by using a scope of an endoscope, acquires second reference site information related to the reference site and direction information related to a travel direction of a tube leading to the duodenum from volume data, generates a three-dimensional travel direction image obtained by forming a combined image of the travel direction with the captured image on the basis of the direction information adjusted by reconciling the first reference site information and the second reference site information, displays the captured image on a screen, and displays the three-dimensional travel direction image within the captured image.

Description

Medical support device, endoscope, medical support method, and program

[0001] The technology of the present disclosure relates to a medical support device, an endoscope, a medical support method, and a program.

[0002] Japanese Patent Application Laid-Open No. 2022-105685 discloses an endoscopic processor including an endoscopic image acquisition unit, a virtual endoscopic image acquisition unit, a virtual endoscopic image reconstruction unit, and a diagnostic support information output unit.

[0003] In the endoscopic processor described in Japanese Patent Laid-Open Publication No. 2022-105685, an endoscopic image acquisition unit acquires an endoscopic image of a patient from an endoscope. The virtual endoscopic image acquisition unit acquires a virtual endoscopic image reconstructed based on a three-dimensional medical image of the patient captured in advance. The virtual endoscopic image reconstruction unit reconstructs a corrected virtual endoscopic image that most closely matches the endoscopic image acquired by the virtual endoscopic image acquisition unit based on the degree of match between the virtual endoscopic image acquired by the virtual endoscopic image acquisition unit and the endoscopic image acquired by the endoscopic image acquisition unit. The diagnostic support information output unit associates each pixel of the endoscopic image acquired by the endoscopic image acquisition unit with a distance image obtained from the corrected virtual endoscopic image reconstructed by the virtual endoscopic image reconstruction unit, and outputs diagnostic support information based on feature parameters corrected accordingly.

[0004] One embodiment of the technology disclosed herein provides a medical support device, an endoscope, a medical support method, and a program that enable a user observing an image to visually recognize the running direction of a tube leading to the duodenum through the image.

[0005] A first aspect of the technology of the present disclosure is a medical support device that includes a processor, and that acquires first reference site information that can identify a reference site included in the duodenum in three dimensions based on an image obtained by imaging the intestinal wall of the duodenum with an endoscope, acquires second reference site information related to the reference site and directional information related to the running direction of a tube leading to the duodenum from the volume data, generates a running direction image in which the running direction is visualized to match the captured image based on the directional information adjusted by aligning the first reference site information with the second reference site information, displays the captured image on a screen, and displays the running direction image within the captured image.

[0006] A second aspect of the technology of the present disclosure is a medical support device according to the first aspect, in which the first reference part information is a first image area in which the reference part is represented as a three-dimensional image, the second reference part information is a second image area in which the reference part is represented as a three-dimensional image, and the processor generates a driving direction image based on direction information adjusted by aligning the first image area and the second image area.

[0007] A third aspect of the technology of the present disclosure is a medical support device according to the first or second aspect, in which a processor generates a duodenal image in which the duodenum is represented as a three-dimensional image based on an acquired image, and acquires first reference site information from the duodenal image.

[0008] A fourth aspect of the technique of the present disclosure is the medical support device according to the third aspect, in which the duodenum image is a three-dimensional image generated based on a plurality of distance images.

[0009] A fifth aspect of the technology of the present disclosure is a medical support device according to the fourth aspect, in which, each time the captured image is updated by a specified number of frames, the processor generates a duodenum image based on the updated captured image, generates a traveling direction image based on the direction information each time a duodenum image is generated, displays the updated captured image on a screen, and displays the generated traveling direction image within the captured image each time a traveling direction image is generated.

[0010] A sixth aspect of the technology of the present disclosure is a medical support device according to any one of the first to fifth aspects, in which the directional information is a three-dimensional directional image in which the running direction is represented in a three-dimensional image, and the running direction image is an image based on a two-dimensional image obtained by projecting the three-dimensional directional image onto a captured image.

[0011] A seventh aspect of the technology of the present disclosure is a medical support device according to any one of the first to sixth aspects, in which the captured image includes a papilla image showing the duodenal papilla, and the travel direction image is displayed in association with the papilla image.

[0012] An eighth aspect of the technology of the present disclosure is a medical support device according to the seventh aspect, in which the tube leads to an opening in the duodenal papilla, and the travel direction image is an image showing a first direction along the tube with the opening as the base point.

[0013] A ninth aspect of the technique of the present disclosure is the medical support device according to the eighth aspect, in which the first direction corresponds to an insertion direction of a medical instrument to be inserted into a tube.

[0014] A tenth aspect of the technology of the present disclosure is a medical support device according to any one of the first to ninth aspects, in which the reference site is a duodenal papilla, a medical marker, and / or a fold.

[0015] An eleventh aspect of the technology of the present disclosure is a medical support device according to any one of the first to tenth aspects, in which the processor adjusts the scale of the directional information based on the distance from the endoscope to the intestinal wall and the volume data.

[0016] A twelfth aspect of the technique of the present disclosure is the medical support device according to any one of the first to tenth aspects, in which the duct is a bile duct and / or a pancreatic duct.

[0017] A thirteenth aspect of the technology of the present disclosure is a medical support device according to any one of the first to twelfth aspects, in which the driving direction image displayed within the captured image is updated in real time.

[0018] A fourteenth aspect of the technique of the present disclosure is an endoscope including a medical support device according to any one of the first to thirteenth aspects and an endoscope.

[0019] A fifteenth aspect of the technology of the present disclosure is a medical support method including: acquiring first reference site information capable of identifying a reference site included in the duodenum in three dimensions based on an image obtained by imaging the intestinal wall of the duodenum with an endoscope; acquiring second reference site information related to the reference site and directional information related to the running direction of a tube leading to the duodenum from volume data; generating a running direction image in which the running direction is aligned with the captured image based on the directional information adjusted by aligning the first reference site information with the second reference site information; displaying the captured image on a screen; and displaying the running direction image within the captured image.

[0020] A sixteenth aspect of the technology of the present disclosure is a program for causing a computer to execute processes including: acquiring first reference site information capable of identifying a reference site included in the duodenum in three dimensions based on an image obtained by imaging the intestinal wall of the duodenum with an endoscope; acquiring second reference site information related to the reference site and direction information related to the running direction of a tube leading to the duodenum from volume data; generating a running direction image in which the running direction is aligned with the captured image based on the direction information adjusted by aligning the first reference site information with the second reference site information; displaying the captured image on a screen; and displaying the running direction image within the captured image.

[0021] 15B is a conceptual diagram showing an example of an aspect in which a duodenoscope system is used. FIG. 15C is a conceptual diagram showing an example of the overall configuration of a duodenoscope system. FIG. 15D is a block diagram showing an example of the hardware configuration of an electrical system of a duodenoscope system. FIG. 15E is a conceptual diagram showing an example of an aspect of the duodenum, bile duct, and pancreatic duct. FIG. 15F is a block diagram showing an example of the main functions of a processor included in an endoscope, and an example of information stored in an NVM. FIG. 15G is a conceptual diagram showing an example of the processing content of a first acquisition unit. FIG. 15H is a conceptual diagram showing an example of the processing content of a second acquisition unit. FIG. 15I is a conceptual diagram showing an example of the first processing content of an adjustment unit. FIG. 15J is a conceptual diagram showing an example of the second processing content of an adjustment unit. FIG. 15J is a conceptual diagram showing an example of the processing content of a synthesis unit. FIG. 15I is a conceptual diagram showing an example of the first processing content of a third acquisition unit. FIG. 15I is a conceptual diagram showing an example of the second processing content of a third acquisition unit. FIG. 15I is a conceptual diagram showing an example of the first processing content of a control unit. FIG. 15I is a conceptual diagram showing an example of the second processing content of a control unit. FIG. 15I is a flowchart showing an example of the flow of medical support processing. FIG. 15B is a continuation of the flowchart shown in FIG. 15A is a conceptual diagram showing a modified example of the processing content of the first acquisition unit. FIG. 15B is a conceptual diagram showing a modified example of the processing content of the third acquisition unit. FIG. 10 is a conceptual diagram showing an example of a manner in which information is output from an image processing device or a control device to a server and / or a printer via a network.

[0022] Hereinafter, exemplary embodiments of a medical support device, an endoscope, a medical support method, and a program according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] CPU is an abbreviation for "Central Processing Unit." GPU is an abbreviation for "Graphics Processing Unit." RAM is an abbreviation for "Random Access Memory." NVM is an abbreviation for "Non-volatile memory." EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory." ASIC is an abbreviation for "Application Specific Integrated Circuit." PLD is an abbreviation for "Programmable Logic Device." FPGA is an abbreviation for "Field-Programmable Gate Array." SoC is an abbreviation for "System-on-a-chip." SSD is an abbreviation for "Solid State Drive." USB is an abbreviation for "Universal Serial Bus." HDD is an abbreviation for "Hard Disk Drive." EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". FIFO is an abbreviation for "First In First Out". TOF is an abbreviation for "Time Of Flight". RNN-SLAM is an abbreviation for "Reccurent Neural Network - Simultaneous Localization and Mapping". VAE is an abbreviation for "Variational Autoencoder". GAN is an abbreviation for "Generative Adversarial Networks".CT is an abbreviation for "Computed Tomography." MRI is an abbreviation for "Magnetic Resonance Imaging." 3D is an abbreviation for "Three Dimensions."

[0025] 1 , a duodenoscope system 10 includes a duodenoscope 12 and a display device 13. The duodenoscope 12 is used by a doctor 14 in an endoscopic examination. The duodenoscope 12 is an example of an "endoscope" according to the technology of the present disclosure.

[0026] The duodenoscope 12 is communicably connected to a communication device (not shown), and information obtained by the duodenoscope 12 is transmitted to the communication device. The communication device receives the information transmitted from the duodenoscope 12 and performs processing using the received information (for example, recording the information in an electronic medical record, etc.).

[0027] The duodenoscope 12 includes a duodenoscope body 18 (in other words, an endoscope). The duodenoscope 12 is a device for performing medical examinations on an observation target 21 (e.g., the duodenum) contained inside the body of a subject 20 (e.g., a patient) using the duodenoscope body 18. The observation target 21 is an object observed by a doctor 14.

[0028] The duodenoscope main body 18 is inserted into the body of the subject 20. The duodenoscope 12 causes the duodenoscope main body 18 inserted into the body of the subject 20 to capture an image of an observation target 21 inside the body of the subject 20, and performs various medical procedures on the observation target 21 as necessary.

[0029] The duodenoscope 12 acquires and outputs an image showing the state of the inside of the body by capturing an image of the inside of the body of the subject 20. In this embodiment, the duodenoscope 12 is an endoscope having an optical imaging function that captures an image of reflected light obtained by irradiating light inside the body and reflecting it off an observation target 21.

[0030] The duodenoscope 12 includes a control device 22, a light source device 24, and an image processing device 25. The control device 22, the light source device 24, and the image processing device 25 are mounted on a wagon 34. The wagon 34 has a plurality of stands arranged vertically, with the image processing device 25, the control device 22, and the light source device 24 mounted from the lower stand to the upper stand. In addition, a display device 13 is mounted on the top stand of the wagon 34.

[0031] The control device 22 controls the entire duodenoscope 12. The image processing device 25, under the control of the control device 22, performs various image processing on the image obtained by capturing an image of the observation target 21 by the duodenoscope main body 18.

[0032] The display device 13 displays various information including images. Examples of the display device 13 include a liquid crystal display and an EL display. Alternatively, a tablet terminal with a display may be used instead of the display device 13 or together with the display device 13.

[0033] A plurality of screens are displayed side by side on the display device 13. In the example shown in Fig. 1, screens 36A to 36C are shown as an example of the plurality of screens.

[0034] The screen 36A displays a captured image 40 obtained by the duodenoscope 12. The captured image 40 shows an observation target 21. The captured image 40 is an image obtained by imaging the observation target 21 inside the body of the subject 20 using the duodenoscope main body 18. An example of the observation target 21 is the intestinal wall of the duodenum. For ease of explanation, the following description will use an intestinal wall image as an example of the captured image 40. The intestinal wall image refers to an image obtained by imaging the intestinal wall of the duodenum as the observation target 21. Note that the duodenum is merely an example, and any region that can be imaged by the duodenoscope 12 may be used. Examples of regions that can be imaged by the duodenoscope 12 include the esophagus and the stomach. The captured image 40 is an example of a "captured image" according to the technology of the present disclosure.

[0035] The screen 36A displays a moving image including a plurality of frames of captured images 40 in chronological order. That is, the screen 36A displays a plurality of frames of captured images 40 in chronological order at a predetermined frame rate (e.g., several tens of frames per second). The screen 36A is an example of a "screen" according to the technology of the present disclosure.

[0036] Screen 36A is the main screen, while screens 36B and 36C are sub-screens, and various information to assist the doctor 14 in performing the procedure using the duodenoscope 12 is displayed on screens 36B and 36C.

[0037] 2, the duodenoscope main body 18 includes an operating section 42 and an insertion section 44. The insertion section 44 is partially curved by operating the operating section 42. The insertion section 44 is inserted while curving in accordance with the shape of the observation target 21 (for example, the shape of the duodenum) in accordance with the operation of the operating section 42 by the physician 14.

[0038] The distal end 46 of the insertion section 44 is provided with a camera 48, a distance measurement sensor 49, an illumination device 50, a treatment opening 51, and an erection mechanism 52. The camera 48 and the illumination device 50 are provided on the side of the distal end 46. In other words, the duodenoscope 12 is configured as a side-viewing endoscope, which makes it easy to observe the intestinal wall of the duodenum.

[0039] The camera 48 is a device that captures an image 40 as a medical image by capturing an image of the inside of the body of the subject 20. An example of the camera 48 is a CMOS camera. However, this is merely an example, and other types of cameras such as a CCD camera may also be used. The camera 48 is an example of an "endoscope" according to the technology of the present disclosure.

[0040] The ranging sensor 49 is an optical ranging sensor. For example, the ranging sensor 49 measures distance (i.e., distance) in synchronization with the timing of image capture by the camera 48. An example of the ranging sensor 49 is a device including a time-of-flight (TOF) camera. The TOF camera is a camera that measures three-dimensional information using a time-of-flight (TOF) method. Note that, although the camera 48 and the ranging sensor 49 are separate entities here, the camera 48 may also be equipped with a TOF camera function. In this case, the camera 48 can obtain depth information simultaneously with the captured image 40.

[0041] The illumination device 50 has an illumination window 50A. The illumination device 50 emits light through the illumination window 50A. Types of light emitted from the illumination device 50 include, for example, visible light (e.g., white light) and invisible light (e.g., near-infrared light). The illumination device 50 also emits special light through the illumination window 50A. Examples of the special light include light for BLI and / or light for LCI. The camera 48 captures images of the inside of the subject 20 by an optical method while light is being emitted from the illumination device 50 inside the subject 20.

[0042] The treatment opening 51 is used as a treatment tool ejection port for ejecting a treatment tool 54 from the distal end portion 46, a suction port for sucking blood and internal waste, and a delivery port for delivering fluid.

[0043] A treatment tool 54 protrudes from the treatment opening 51 in accordance with the operation of the physician 14. The treatment tool 54 is inserted into the insertion section 44 from a treatment tool insertion port 58. The treatment tool 54 passes through the insertion section 44 via the treatment tool insertion port 58 and protrudes from the treatment opening 51 into the body of the subject 20. In the example shown in FIG. 2 , a cannula 54A protrudes from the treatment opening 51 as the treatment tool 54. The cannula 54A is merely one example of the treatment tool 54, and other examples of the treatment tool 54 include a catheter, a guidewire, a papillotomy knife, and a snare. The treatment tool 54 is an example of a "medical instrument" according to the technology of the present disclosure.

[0044] The standing mechanism 52 changes the protruding direction of the treatment tool 54 protruding from the treatment opening 51. The standing mechanism 52 includes a guide 52A, which rises relative to the protruding direction of the treatment tool 54, thereby changing the protruding direction of the treatment tool 54 along the guide 52A. This makes it easier to protrude the treatment tool 54 toward the intestinal wall. In the example shown in FIG. 2 , the standing mechanism 52 changes the protruding direction of the treatment tool 54 to a direction perpendicular to the traveling direction of the tip portion 46. The standing mechanism 52 is operated by the doctor 14 via the operation unit 42. This adjusts the degree of change in the protruding direction of the treatment tool 54.

[0045] The duodenoscope main body 18 is connected to the control device 22 and the light source device 24 via a universal cord 60. A reception device 62 is connected to the control device 22. An image processing device 25 is also connected to the control device 22. A display device 13 is also connected to the image processing device 25. That is, the control device 22 is connected to the display device 13 via the image processing device 25.

[0046] Note that, because the image processing device 25 is exemplified here as an external device for expanding the functions performed by the control device 22, an example in which the control device 22 and the display device 13 are indirectly connected via the image processing device 25 is given, but this is merely one example. For example, the display device 13 may be directly connected to the control device 22. In this case, for example, the functions of the image processing device 25 may be incorporated into the control device 22, or the control device 22 may be equipped with a function for causing a server (not shown) to execute the same processing as that executed by the image processing device 25 (for example, the medical support processing described below) and for receiving and using the processing results from the server.

[0047] The reception device 62 receives instructions from a user (e.g., the doctor 14) and outputs the received instructions as an electrical signal to the control device 22. Examples of the reception device 62 include a keyboard, a mouse, a touch panel, a foot switch, and a microphone.

[0048] The control device 22 controls the light source device 24 , exchanges various signals with the camera 48 , and exchanges various signals with the image processing device 25 .

[0049] The light source device 24 emits light under the control of the control device 22 and supplies the light to the illumination device 50. A light guide is built into the illumination device 50, and the light supplied from the light source device 24 passes through the light guide and is irradiated from an illumination window 50A. The control device 22 causes the camera 48 to capture an image, acquires a captured image 40 (see FIG. 1 ) from the camera 48, and outputs the image to a predetermined output destination (for example, the image processing device 25).

[0050] The image processing device 25 performs various image processing on the captured image 40 input from the control device 22. The image processing device 25 outputs the captured image 40 that has been subjected to various image processing to a predetermined output destination (for example, the display device 13).

[0051] Although the embodiment in which the captured image 40 output from the control device 22 is output to the display device 13 via the image processing device 25 has been described above, this is merely one example. The control device 22 and the display device 13 may be connected, and the captured image 40 that has been subjected to image processing in the image processing device 25 may be displayed on the display device 13 via the control device 22.

[0052] 3 , the control device 22 includes a computer 64, a bus 66, and an external I / F 68. The computer 64 includes a processor 70, a RAM 72, and an NVM 74. The processor 70, the RAM 72, the NVM 74, and the external I / F 68 are connected to the bus 66.

[0053] For example, the processor 70 has a CPU and a GPU, and controls the entire control device 22. The GPU operates under the control of the CPU, and is responsible for executing various graphic processing and performing calculations using neural networks. The processor 70 may be one or more CPUs that have integrated GPU functionality, or one or more CPUs that do not have integrated GPU functionality.

[0054] The RAM 72 is a memory that temporarily stores information and is used as a work memory by the processor 70. The NVM 74 is a nonvolatile storage device that stores various programs, various parameters, and the like. An example of the NVM 74 is a flash memory (for example, an EEPROM and / or an SSD). Note that the flash memory is merely one example, and the NVM 74 may be another nonvolatile storage device such as an HDD, or may be a combination of two or more types of nonvolatile storage devices.

[0055] The external I / F 68 controls the exchange of various information between one or more devices (hereinafter also referred to as “first external devices”) existing outside the control device 22 and the processor 70. An example of the external I / F 68 is a USB interface.

[0056] The external I / F 68 is connected to the camera 48 as one of the first external devices, and the external I / F 68 controls the exchange of various information between the camera 48 and the processor 70. The processor 70 controls the camera 48 via the external I / F 68. The processor 70 also acquires, via the external I / F 68, captured images 40 (see FIG. 1 ) obtained by the camera 48 capturing an image of the inside of the body of the subject 20.

[0057] The external I / F 68 is connected to the distance measurement sensor 49 as one of the first external devices, and the external I / F 68 controls the exchange of various information between the distance measurement sensor 49 and the processor 70. The processor 70 controls the distance measurement sensor 49 and acquires the results of distance measurement performed by the distance measurement sensor 49 (for example, the distance or three-dimensional information measured by the distance measurement sensor 49).

[0058] The light source device 24 is connected to the external I / F 68 as one of the first external devices, and the external I / F 68 controls the exchange of various information between the light source device 24 and the processor 70. The light source device 24 supplies light to the illumination device 50 under the control of the processor 70. The illumination device 50 irradiates the light supplied from the light source device 24.

[0059] The external I / F 68 is connected to a reception device 62 as one of the first external devices, and the processor 70 acquires instructions accepted by the reception device 62 via the external I / F 68 and executes processing according to the acquired instructions.

[0060] The image processing device 25 includes a computer 76 and an external I / F 78. The computer 76 includes a processor 80, a RAM 82, and an NVM 84. The processor 80, the RAM 82, the NVM 84, and the external I / F 78 are connected to a bus 86. Here, the image processing device 25 is an example of a "medical support device" according to the technology of the present disclosure, the computer 76 is an example of a "computer" according to the technology of the present disclosure, and the processor 80 is an example of a "processor" according to the technology of the present disclosure.

[0061] The hardware configuration of the computer 76 (i.e., the processor 80, RAM 82, and NVM 84) is basically the same as the hardware configuration of the computer 64, so a description of the hardware configuration of the computer 76 will be omitted here.

[0062] The external I / F 78 controls the exchange of various information between the processor 80 and one or more devices (hereinafter also referred to as "second external devices") that exist outside the image processing device 25. An example of the external I / F 78 is a USB interface.

[0063] The control device 22 is connected to the external I / F 78 as one of the second external devices. In the example shown in Fig. 3, the external I / F 68 of the control device 22 is connected to the external I / F 78. The external I / F 78 controls the exchange of various information between the processor 80 of the image processing device 25 and the processor 70 of the control device 22. For example, the processor 80 acquires the captured image 40 (see Fig. 1 ) from the processor 70 of the control device 22 via the external I / Fs 68 and 78, and performs various image processing on the acquired captured image 40.

[0064] The display device 13 is connected as one of the second external devices to the external I / F 78. The processor 80 controls the display device 13 via the external I / F 78 to cause the display device 13 to display various information (e.g., the captured image 40 that has been subjected to various image processing).

[0065] One known procedure for the duodenum using a duodenoscope 12 is called ERCP (endoscopic retrograde cholangiopancreatography). As shown in FIG. 4 , in an ERCP examination, the duodenoscope 12 is first inserted into the duodenum 88 via the esophagus and stomach. The insertion state of the duodenoscope 12 may be confirmed using an X-ray image obtained by X-ray imaging. The distal end 46 of the duodenoscope 12 then reaches the vicinity of a duodenal papilla 90 (hereinafter simply referred to as the "papilla 90") located in the intestinal wall of the duodenum 88.

[0066] In an ERCP examination, for example, a cannula 54A, which is one type of treatment tool 54, is inserted into the papilla 90 from the duodenum 88 side. Here, the papilla 90 is a portion that protrudes from the intestinal wall of the duodenum 88. At a papillary prominence 90A, which is the tip of the papilla 90, there are present the ends of one or more ducts 92 that lead to internal organs (e.g., the gallbladder and pancreas), i.e., openings 90A1 that lead to the ducts 92. In other words, the ducts 92 lead to the openings 90A1 present in the papillary prominence 90A.

[0067] Examples of the one or more ducts 92 include a bile duct 92A and a pancreatic duct 92B. The opening 90A1 may exist individually for each of the bile duct 92A and the pancreatic duct 92B, or may exist in common for both the bile duct 92A and the pancreatic duct 92B. The bile duct 92A is an example of a "bile duct" according to the technology of the present disclosure, and the pancreatic duct 92B is an example of a "pancreatic duct" according to the technology of the present disclosure. Hereinafter, for convenience of explanation, when there is no need to distinguish between the bile duct 92A and the pancreatic duct 92B, they will be referred to as "ducts 92."

[0068] In an ERCP examination, X-ray imaging is performed with a contrast medium injected into the tube 92 through the opening 90A1. When inserting the cannula 54A into the tube 92, the physician 14 must accurately grasp the direction 94 of the tube 92. In particular, since the direction 94 in the vicinity of the opening 90A1 is substantially the same as the insertion direction of the cannula 54A into the opening 90A1, it is very important for the physician 14 to visually grasp the direction 94 in the vicinity of the opening 90A1.

[0069] Examples of the running direction 94 of the duct 92 include the running direction 94A of the bile duct 92A and the running direction 94B of the pancreatic duct 92B. When inserting the cannula 54A into the bile duct 92A, it is effective for the physician 14 to visually understand the running direction 94A. When inserting the cannula 54A into the pancreatic duct 92B, it is effective for the physician 14 to visually understand the running direction 94B. Here, an example is given in which the cannula 54A is inserted into the opening 90A1. However, even when a catheter or guidewire is inserted into the duct 92 as the treatment tool 54, or when a papillotomy knife is brought into contact with the opening 90A1 as the treatment tool 54, it is very effective for the physician 14 to visually understand the running direction 94.

[0070] In view of the above circumstances, in this embodiment, as an example shown in FIG. 5, medical support processing is performed by a processor 80 of the image processing device 25.

[0071] A medical support program 96 is stored in the NVM 84. The medical support program 96 is an example of a "program" according to the technology of the present disclosure. The processor 80 performs medical support processing by reading the medical support program 96 from the NVM 84 and executing the read medical support program 96 on the RAM 82. The medical support processing is realized by the processor 80 operating as a first acquisition unit 80A, a second acquisition unit 80B, a third acquisition unit 80C, an adjustment unit 80D, a synthesis unit 80E, and a control unit 80F in accordance with the medical support program 96 executed on the RAM 82.

[0072] The NVM 84 stores a 3D construction model 98, a nipple detection model 100, a type prediction model 102, and a support information table 104. As will be described in detail later, the 3D construction model 98 is used by the first acquisition unit 80A, the nipple detection model 100 is used by the adjustment unit 80D, and the type prediction model 102 and the support information table 104 are used by the control unit 80F.

[0073] 6 as an example, the first acquisition unit 80A acquires a first duodenum image 108 based on the time-series image group 106. The first duodenum image 108 is acquired by, for example, generating the first duodenum image 108 using a 3D construction model 98.

[0074] To generate the first duodenum image 108, the first acquisition unit 80A first acquires the captured image 40, which is generated by the camera 48 capturing images at an imaging frame rate (e.g., several tens of frames per second), from the camera 48 in frame units. A frame unit is an example of a "specified frame number unit" according to the technology of the present disclosure. Note that, although a frame unit is illustrated here, this is merely an example, and a specified frame number unit of two or more frames may also be used.

[0075] The first acquisition unit 80A holds a time-series image group 106. The time-series image group 106 is a plurality of frames of captured images 40 in chronological order, in which the object of observation 21 is captured. The time-series image group 106 includes, for example, a certain number of frames (e.g., a predetermined number of frames within a range of several tens to several hundreds of frames) of captured images 40. Furthermore, one or more frames of the captured images 40 of the certain number of frames show a nipple 90. In other words, one or more frames of the captured images 40 include a nipple image 110 showing the nipple 90.

[0076] The first acquisition unit 80A updates the time-series image group 106 in a FIFO manner every time it acquires a captured image 40 from the camera 48 .

[0077] Although the example in which the time-series image group 106 is stored and updated by the first acquisition unit 80A is given here, this is merely an example. For example, the time-series image group 106 may be stored and updated in a memory connected to the processor 80, such as the RAM 82.

[0078] The 3D construction model 98 is a generative model using a neural network, and generates a first duodenum image 108, which is a three-dimensional image, based on the time-series image group 106. The 3D construction model 98 is a trained model obtained by performing machine learning on a neural network (e.g., a recurrent neural network) using a plurality of images corresponding to the time-series image group 106 as training data. An example of the 3D construction model 98 is RNN-SLAM. Note that RNN-SLAM is merely an example, and the 3D construction model 98 may also be an autoencoder such as a VAE or GAN, as long as it is a trained generative network capable of generating the first duodenum image 108.

[0079] The first acquisition unit 80A inputs the time-series image group 106 to the 3D construction model 98. In response to this, the 3D construction model 98 generates and outputs a first duodenum image 108 based on the time-series image group 106. The first acquisition unit 80A acquires the first duodenum image 108 output from the 3D construction model 98.

[0080] The first duodenum image 108 includes a first papilla region 108A. The first papilla region 108A is information that enables the papilla 90 included in the duodenum 88 to be identified in three dimensions. In the example shown in Fig. 6, an image region in which the papilla 90 is expressed as a three-dimensional image is shown as an example of the first papilla region 108A. The first duodenum image 108 is generated each time the captured images 40 are acquired frame by frame by the first acquisition unit 80A and the time-series image group 106 is updated.

[0081] In this embodiment, the nipple 90 is an example of a "reference site" and a "duodenal papilla" according to the technology of the present disclosure. The first nipple region 108A is an example of a "first reference site information" and a "first image region" according to the technology of the present disclosure.

[0082] As an example, as shown in FIG. 7 , the NVM 84 stores volume data 112. The volume data 112 is an image obtained by stacking a plurality of two-dimensional slice images 114 obtained by imaging the subject 20 using a modality and dividing the images into voxels V. An example of a modality is a CT device. The CT device is merely one example, and other examples of modalities include an MRI device or an ultrasound diagnostic device. The position of each of all voxels V defining the three-dimensional image is specified by three-dimensional coordinates.

[0083] The second acquisition unit 80B acquires information about the observation target site from the volume data 112. In the example shown in Fig. 7, an observation target site image 116, which is a three-dimensional image showing the observation target site, is shown as the information about the observation target site. That is, the second acquisition unit 80B acquires the observation target site image 116 from the volume data 112 as information about the observation target site.

[0084] The observation target region is the duodenum 88 including the papilla 90 and the duct 92 (see FIG. 4 ). The observation target region (i.e., the observation target region image 116 acquired from the volume data 112 by the second acquisition unit 80B) is selected, for example, in accordance with an instruction received by the reception device 62.

[0085] The observation region image 116 acquired from the volume data 112 by the second acquisition unit 80B includes a second duodenum image 118 and a duct image 120. The second duodenum image 118 is a three-dimensional image showing the duodenum 88. The second duodenum image 118 includes a second papilla region 118A. The second papilla region 118A is an image region in which the papilla 90 is represented as a three-dimensional image. The duct image 120 is an image region in which the duct 92 is represented as a three-dimensional image. The duct image 120 includes a bile duct image 120A and a pancreatic duct image 120B. The bile duct image 120A is an image region in which the bile duct 92A is represented as a three-dimensional image, and the pancreatic duct image 120B is an image region in which the pancreatic duct 92B is represented as a three-dimensional image. In this embodiment, the second nipple region 118A is an example of the "second reference part information" and the "second image region" according to the technology of the present disclosure.

[0086] As an example, as shown in FIG. 8 , the adjustment unit 80D performs object detection processing on the captured image 40 using a nipple detection model 100, thereby detecting a nipple image 110 in the captured image 40. The nipple detection model 100 is a trained model for object detection using an AI method, and is optimized by performing machine learning on a neural network using first training data. The first training data is a plurality of data (i.e., data for a plurality of frames) in which first example data and first correct answer data are associated with each other. The first example data is an image corresponding to the captured image 40. The first correct answer data is correct answer data (i.e., annotation) for the first example data. An example of the first correct answer data is an annotation that can identify an image area in which the nipple 90 is captured.

[0087] Adjustment unit 80D acquires captured image 40 from camera 48 and inputs the acquired captured image 40 to nipple detection model 100. As a result, nipple detection model 100 detects nipple image 110 from the input captured image 40 and outputs identification information 122 with which the detected nipple image 110 can be identified (for example, a plurality of coordinates indicating the position of nipple image 110 within captured image 40). Adjustment unit 80D acquires the identification information 122 output from nipple detection model 100.

[0088] The adjustment unit 80D acquires a distance 125 measured by the distance measurement sensor 49 in synchronization with the image capture by the camera 48 to obtain the captured image 40 input to the nipple detection model 100. The distance 125 refers to, for example, the distance from a reference position (here, as an example, the position of the imaging plane of the camera 48 where the image capture was performed to obtain the captured image 40 input to the nipple detection model 100) to the intestinal wall 88A including the papilla 90 in the duodenum 88. In the example shown in Figure 8, the pylorus is cited as an example of the reference position. The distance measurement sensor 49 measures multiple distances 125 from the reference position to multiple points within the intestinal wall 88A.

[0089] The adjustment unit 80D calculates a first reference distance 126, which is the distance to the nipple 90 indicated by the nipple image 110 identified by the identification information 122. For example, if there are multiple measurement points within the nipple 90 indicated by the nipple image 110 identified by the identification information 122, the average value of the multiple distances 125 for the multiple measurement points is calculated as the first reference distance 126. Note that although the average value of the multiple distances 125 for the multiple measurement points within the nipple 90 is exemplified here, the first reference distance 126 may also be a statistical value such as the median, mode, or maximum value of the multiple distances 125 for the multiple measurement points within the nipple 90.

[0090] The adjustment unit 80D uses the second duodenum image 118 included in the observation target area image 116 acquired by the second acquisition unit 80B to measure the distance from the reference position (here, as an example, the pylorus) to the papilla 90 indicated by the second papilla region 118A as the second reference distance 128.

[0091] 9 , the adjustment unit 80D calculates a dissimilarity 129 between the first reference distance 126 and the second reference distance 128. Examples of the dissimilarity 129 include the ratio of the second reference distance 128 to the first reference distance 126, or the absolute value of the difference between the first reference distance 126 and the second reference distance 128.

[0092] The adjustment unit 80D adjusts the scale of the observation target portion image 116 acquired by the second acquisition unit 80B by referring to the dissimilarity 129. For example, the adjustment unit 80D adjusts the scale of the observation target portion image 116 by using the dissimilarity 129 itself as a magnification, or adjusts the scale of the observation target portion image 116 using a magnification calculated based on the dissimilarity 129. An example of a magnification calculated based on the dissimilarity 129 is a magnification derived from an arithmetic expression in which the dissimilarity 129 is an independent variable and the magnification is a dependent variable.

[0093] 10 , the composition unit 80E generates a composite image 123 based on the first duodenum image 108 acquired by the first acquisition unit 80A and the observation target region image 116 whose scale has been adjusted by the adjustment unit 80D. To generate the composite image 123, the composition unit 80E first adjusts the duct image 120 by aligning the first nipple region 108A included in the first duodenum image 108 acquired by the first acquisition unit 80A with the second nipple region 118A included in the observation target region image 116 whose scale has been adjusted by the adjustment unit 80D. For example, the composition unit 80E aligns the observation target region image 116 and the first duodenum image 108 based on the first nipple region 108A and the second nipple region 118A. That is, the observation site image 116 is aligned with the first duodenum image 108 so that the first papilla region 108A and the second papilla region 118A coincide with each other. Here, the observation site image 116 is aligned with the first duodenum image 108, and this is achieved, for example, by aligning a plurality of feature points that are common between the first papilla region 108A and the second papilla region 118A.

[0094] With the observation site image 116 thus aligned with the first duodenum image 108, the synthesis unit 80E synthesizes the duct image 120 with the first duodenum image 108 to generate a synthesized image 123. That is, the synthesis unit 80E synthesizes the duct image 120 with the first duodenum image 108 by adding the duct image 120 to a position corresponding to the second papilla region 118A within the first papilla region 108A (i.e., a position corresponding to the position of the duct image 120 connected to the second papilla region 118A).

[0095] 11 , the third acquisition unit 80C acquires a three-dimensional running direction image 124 as information relating to the running direction 94 of the tube 92 leading to the duodenum 88 (see FIG. 4 ) based on a composite image 123 generated by the synthesis unit 80E. The three-dimensional running direction image 124 is an image in which the running direction 94 of the tube 92 is expressed in a three-dimensional image. The three-dimensional running direction image 124 is an example of "directional information" and "three-dimensional direction image" according to the techniques of the present disclosure.

[0096] The third acquisition unit 80C performs thinning processing on the tube image 120 combined with the first duodenum image 108 to generate a three-dimensional running direction image 124 (e.g., a curve indicating the running direction 94) indicating the running direction 94 of the tube 92 indicated by the tube image 120. The running direction 94 of the tube 92 can also be said to be the axial direction of the tube image 120.

[0097] The three-dimensional running direction image 124 includes a three-dimensional bile duct direction image 124A and a three-dimensional pancreatic duct direction image 124B. The three-dimensional bile duct direction image 124A shows the running direction 94A (see FIG. 4) of the bile duct 92A and is obtained by thinning the bile duct image 120A. The three-dimensional pancreatic duct direction image 124B shows the running direction 94B (see FIG. 4) of the pancreatic duct 92B and is obtained by thinning the pancreatic duct image 120B.

[0098] The third acquisition unit 80C acquires a captured image 40 including a nipple image 110 from the camera 48. The third acquisition unit 80C identifies a three-dimensional region 108B corresponding to the captured image 40 acquired from the camera 48 within the composite image 123. The captured image 40 acquired from the camera 48 by the third acquisition unit 80C corresponds to a two-dimensional image in which the three-dimensional region 108B is projected onto a plane. The third acquisition unit 80C sets the captured image 40 as a projection plane at a position directly opposite the three-dimensional region 108B within the composite image 123. The third acquisition unit 80C generates a composite captured image 130 by rendering a three-dimensional travel direction image 124 on the captured image 40 set within the composite image 123.

[0099] The composite captured image 130 is an image including the captured image 40 and the two-dimensional traveling direction image 132 (for example, an image in which the two-dimensional traveling direction image 132 is superimposed on the captured image 40). The two-dimensional traveling direction image 132 is an image in which the three-dimensional traveling direction image 124 is rendered onto the captured image 40 (i.e., an image in which the three-dimensional traveling direction image 124 is projected onto the captured image 40). The two-dimensional traveling direction image 132 includes a two-dimensional bile duct direction image 132A and a two-dimensional pancreatic duct direction image 132B. The two-dimensional bile duct direction image 132A is an image in which the three-dimensional bile duct direction image 124A is rendered onto the captured image 40. The two-dimensional pancreatic duct direction image 132B is an image in which the three-dimensional pancreatic duct direction image 124B is rendered onto the captured image 40.

[0100] 12 as an example, the third acquisition unit 80C acquires a simplified driving direction image 136 that is a simplification of the two-dimensional driving direction image 132 in the composite captured image 130. The simplified driving direction image 136 is an example of a "driving direction image" according to the technology of the present disclosure.

[0101] To generate the simplified traveling direction image 136, the third acquisition unit 80C first extracts tangents 134 from the two-dimensional traveling direction image 132. The tangents 134 are tangents to the end of the two-dimensional traveling direction image 132 that faces the nipple image 110 (i.e., the position where the opening 90A1 (see FIG. 4) is shown). The tangents 134 are classified into bile duct tangents 134A and pancreatic duct tangents 134B. The bile duct tangent 134A is a tangent to the end of the two-dimensional bile duct direction image 132A that faces the nipple image 110 (i.e., the position where the opening 90A1 leading to the bile duct 92A (see FIG. 4) is shown). The pancreatic duct tangent 134B is a tangent to the end of the two-dimensional pancreatic duct direction image 132B that faces the nipple image 110 (i.e., the position where the opening 90A1 leading to the pancreatic duct 92B (see FIG. 4) is shown).

[0102] The third acquisition unit 80C generates a simplified traveling direction image 136 by imaging the traveling direction 94 of the tube 92 in accordance with the composite captured image 130 based on the two-dimensional traveling direction image 132. For example, the third acquisition unit 80C generates the simplified traveling direction image 136 by imaging the tangent 134. The simplified traveling direction image 136 is an image showing an arrow formed along the tangent 134, starting from the end of the tangent 134 on the nipple image 110 side (i.e., the position where the opening 90A1 leading to the tube 92 (see FIG. 4) is captured). The direction indicated by the arrow shown in the simplified traveling direction image 136 corresponds to the traveling direction 94 of the tube 92 at the position of the opening 90A1.

[0103] In this embodiment, the starting point of the arrow indicated by the simplified running direction image 136 is an example of a "base point" according to the technology of the present disclosure, the running direction 94 of the tube 92 is an example of a "first direction" and a "direction corresponding to the insertion direction" according to the technology of the present disclosure, and the simplified running direction image 136 is an example of an "image indicating the first direction" according to the technology of the present disclosure.

[0104] The simplified travel direction image 136 includes a simplified bile duct direction image 136A and a simplified pancreatic duct direction image 136B. The simplified bile duct direction image 136A is an image showing an arrow formed along the bile duct tangent 134A, starting from the end of the bile duct tangent 134A on the papilla image 110 side (i.e., the position where the opening 90A1 leading to the bile duct 92A (see FIG. 4) is shown). The direction indicated by the arrow shown in the simplified bile duct direction image 136A corresponds to the travel direction 94A (see FIG. 4) of the bile duct 92A at the position of the opening 90A1, and is referred to by the physician 14 as the insertion direction of the cannula 54A or the like into the bile duct 92A. The insertion direction of the cannula 54A or the like into the bile duct 92A is an example of the "insertion direction of a medical instrument to be inserted into a duct" according to the technology of the present disclosure.

[0105] The simplified pancreatic duct direction image 136B is an image showing an arrow formed along the pancreatic duct tangent 134B, starting from the end of the pancreatic duct tangent 134B on the papilla image 110 side (i.e., the position where the opening 90A1 leading to the pancreatic duct 92B (see FIG. 4) is shown). The direction indicated by the arrow shown in the simplified pancreatic duct direction image 136B corresponds to the running direction 94B (see FIG. 4) of the pancreatic duct 92B at the position of the opening 90A1, and is referred to by the physician 14 as the insertion direction of the cannula 54A or the like into the pancreatic duct 92B. The insertion direction of the cannula 54A or the like into the pancreatic duct 92B is an example of the "insertion direction of a medical instrument to be inserted into a duct" according to the technology of the present disclosure.

[0106] The simplified bile duct direction image 136A and the simplified pancreatic duct direction image 136B are generated in a manner that allows them to be distinguished from each other. For example, the third acquisition unit 80C generates the simplified bile duct direction image 136A and the simplified pancreatic duct direction image 136B in a manner that allows them to be distinguished from each other by changing the color, density, brightness, and / or pattern of the simplified bile duct direction image 136A and the simplified pancreatic duct direction image 136B.

[0107] 13 , the control unit 80F performs image recognition processing using the type prediction model 102 on the composite captured image 130 acquired by the third acquisition unit 80C, thereby predicting the type of nipple 90 appearing in the composite captured image 130. Then, the control unit 80F acquires type information 138 indicating the predicted type (e.g., the name of the type of nipple 90).

[0108] The type prediction model 102 is obtained by optimizing the neural network through machine learning using second training data. The second training data is a plurality of data (i.e., a plurality of frames of data) in which second example data and second correct answer data are associated with each other. The second example data is, for example, an image obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum) and adding an image corresponding to the simplified traveling direction image 136 (e.g., an image corresponding to the composite captured image 130 to which the simplified traveling direction image 136 has been added). The second correct answer data is correct answer data (i.e., an annotation) corresponding to the second example data. An example of the second correct answer data is an annotation indicating the type of the papilla 90.

[0109] Although the example data included in the second teacher data is an image obtained by imaging a region that may be the subject of an ERCP examination and adding an image corresponding to the simplified traveling direction image 136 to the image, this is merely an example. For example, the example data included in the second teacher data may be the image itself obtained by imaging a region that may be the subject of an ERCP examination (i.e., an image corresponding to the captured image 40 without the simplified traveling direction image 136 added).

[0110] Although an example in which only one type prediction model 102 is used by the control unit 80F has been given here, this is merely one example. For example, the control unit 80F may use a type prediction model 102 selected from a plurality of type prediction models 102. In this case, each type prediction model 102 may be created by performing machine learning specialized for each ERCP examination technique (e.g., the position of the duodenoscope 12 relative to the papilla 90), and the type prediction model 102 corresponding to the currently performed ERCP examination technique may be selected and used by the control unit 80F.

[0111] The control unit 80F inputs the composite captured image 130 acquired by the third acquisition unit 80C to the type prediction model 102. As a result, the type prediction model 102 predicts the type of nipple 90 appearing in the input composite captured image 130 and outputs type information 138 indicating the predicted type. The control unit 80F acquires the type information 138 output from the type prediction model 102.

[0112] The control unit 80F derives support information 140 corresponding to the type information 138 acquired from the type prediction model 102 from the support information table 104. The support information table 104 is a table in which papilla type information 104A, junction type information 104B, and a schema 104C, which correspond to one another, are associated with one another. The papilla type information 104A is information that can identify the type of papilla 90 (e.g., the name of the type of papilla 90). The junction type information 104B is information that is defined for each type of papilla 90 and can identify the junction type at which the bile duct and pancreatic duct join. The schema 104C is a schematic diagram that schematically illustrates the manner in which the bile duct and pancreatic duct join.

[0113] The support information 140 is information including nipple type information 104A, merging format information 104B, and a schema 104C corresponding to the type information 138 acquired from the type prediction model 102 by the control unit 80F.

[0114] 14 , as an example, the control unit 80F displays the composite captured image 130 acquired by the third acquisition unit 80C on a screen 36A, displays support information 140 on a screen 36B, and displays the composite image 123 generated by the compositing unit 80E on a screen 36C. The composite captured image 130 displayed on the screen 36A is updated in real time. In this case, for example, each time the first acquisition unit 80A acquires an image 40 and updates the time-series image group 106, the composite captured image 123 is updated. Each time the composite image 123 is updated, the composite captured image 130 including the two-dimensional driving direction image 132 is updated, and accordingly, the simplified driving direction image 136 is also updated.

[0115] The composite captured image 130 displayed on the screen 36A may be updated each time the captured image 40 is acquired by the first acquisition unit 80A in units of one frame, or each time the captured image 40 is acquired by the first acquisition unit 80A in units of multiple frames.

[0116] Next, the operation of the portion of the duodenoscope system 10 according to the technology of the present disclosure will be described with reference to FIGS. 15A and 15B.

[0117] 15A and 15B show an example of the flow of medical support processing performed by the processor 80. The flow of medical support processing shown in Fig. 15A and 15B is an example of a "medical support method" according to the technology of the present disclosure.

[0118] 15A , first, in step ST10, the first acquisition unit 80A determines whether or not one frame of image data has been captured by the camera 48 for the observation target 21. If one frame of image data has not been captured by the camera 48 for the observation target 21 in step ST10, the determination is negative, and the determination in step ST10 is made again. If one frame of image data has been captured by the camera 48 for the observation target 21 in step ST10, the determination is positive, and the medical support process proceeds to step ST12.

[0119] In step ST12, the first acquisition unit 80A acquires one frame of the captured image 40 obtained by capturing an image of the observation target 21 with the camera 48 (see FIG. 6 ). After the processing of step ST12 is executed, the medical support processing proceeds to step ST14.

[0120] In step ST14, the first acquisition unit 80A determines whether or not a certain number of frames of captured images 40 are held. If the certain number of frames of captured images 40 are not held in step ST14, the determination is negative, and the medical support processing proceeds to step ST10. If the certain number of frames of captured images 40 are held in step ST14, the determination is positive, and the medical support processing proceeds to step ST16.

[0121] In step ST16, the first acquisition unit 80A updates the time-series image group 106 by adding the captured images 40 acquired in step ST12 to the time-series image group 106 in a FIFO manner (see FIG. 6 ). After the processing of step ST16 is executed, the medical support processing proceeds to step ST18.

[0122] In step ST18, the first acquisition unit 80A acquires a first duodenum image 108 by inputting the time-series image group 106 into the 3D constructed model 98. After the processing of step ST18 is executed, the medical support processing proceeds to step ST20.

[0123] In step ST20, the second acquisition unit 80B acquires the observation target region image 116 from the volume data 112 (see FIG. 7). After the processing of step ST20 is executed, the medical support processing proceeds to step ST22.

[0124] In step ST22, the adjustment unit 80D acquires a first reference distance 126 (see FIG. 8 ) based on the captured image 40 used to acquire the first duodenum image 108 and the distance measurement result by the distance measurement sensor 49, and also acquires a second reference distance 128 (see FIG. 8 ) using the second duodenum image 118 included in the observation target region image 116 acquired by the second acquisition unit. After the processing of step ST22 is executed, the medical support processing proceeds to step ST24.

[0125] In step ST24, the adjustment unit 80D adjusts the scale of the observation region image 116 acquired in step ST20 based on the degree of difference 129 between the first reference distance 126 and the second reference distance 128 acquired in step ST22 (see FIG. 9 ). After the processing of step ST24 is executed, the medical support processing proceeds to step ST26.

[0126] In step ST26, the synthesis unit 80E adjusts the duct image 120 by aligning the first nipple region 108A included in the first duodenum image 108 acquired in step ST18 with the second nipple region 118A included in the observation site image 116, the scale of which has been adjusted in step ST24. The synthesis unit 80E synthesizes the duct image 120 with the first duodenum image 108 to generate a synthesized image 123 (see FIG. 10 ). After the processing of step ST26 is executed, the medical support processing proceeds to step ST28.

[0127] In step ST28, thinning processing is performed on the tube image 120 to generate a three-dimensional running direction image 124 (see FIG. 11 ) that indicates the running direction 94 of the tube 92 shown in the tube image 120. After the processing of step ST28 is executed, the medical support processing proceeds to step ST30 shown in FIG. 15B.

[0128] 15B, the third acquisition unit 80C acquires the captured image 40 including the nipple image 110 from the camera 48 (see FIG. 11). The third acquisition unit 80C then identifies the three-dimensional region 108B corresponding to the captured image 40 acquired from the camera 48 within the composite image 123 (see FIG. 11). After the processing of step ST30 is executed, the medical support processing proceeds to step ST32.

[0129] In step ST32, the third acquisition unit 80C sets the captured image 40 as a projection plane at a position directly opposite the three-dimensional area 108B in the composite image 123. Then, the third acquisition unit 80C renders the three-dimensional traveling direction image 124 on the captured image 40 set in the composite image 123.

[0130] By performing the above-described processing, a composite captured image 130 is generated (see FIG. 11). After the processing in step ST32 is executed, the medical support processing proceeds to step ST34.

[0131] In step ST34, the third acquisition unit 80C extracts a tangent line 134 from the two-dimensional travel direction image 132 included in the composite captured image 130 (see FIG. 12). After the processing of step ST34 is executed, the medical support processing proceeds to step ST36.

[0132] In step ST36, the third acquisition unit 80C generates a simplified travel direction image 136 along the tangent line 134 based on the tangent line 134 (see FIG. 12 ). After the processing of step ST36 is executed, the medical support processing proceeds to step ST38.

[0133] In step ST38, the control unit 80F derives support information 140 based on the composite captured image 130 (see FIG. 13). After the process of step ST38 is executed, the medical support process proceeds to step ST40.

[0134] In step ST40, the control unit 80F displays the composite captured image 130 generated in step ST32 on the screen 36A, displays the support information 140 derived in step ST38 on the screen 36B, and displays the composite image 123 generated in step ST26 on the screen 36C. After the processing of step ST40 is executed, the medical support processing proceeds to step ST42.

[0135] In step ST42, the control unit 80F determines whether a condition for terminating the medical support process is satisfied. One example of the condition for terminating the medical support process is that an instruction to terminate the medical support process has been given to the duodenoscope system 10 (for example, that an instruction to terminate the medical support process has been accepted by the acceptance device 62).

[0136] In step ST42, if the condition for terminating the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST10 shown in Fig. 15A. In step ST42, if the condition for terminating the medical support process is satisfied, the determination is positive, and the medical support process ends.

[0137] As described above, in the duodenoscope system 10, the first duodenum image 108 is generated based on the captured image 40, and the first papilla region 108A is acquired from the first duodenum image 108. The second papilla region 118A and the three-dimensional traveling direction image 124 are acquired from the volume data 112. The three-dimensional traveling direction image 124 is adjusted by aligning the first papilla region 108A with the second papilla region 118A, and is then visualized to match the captured image 40. This generates a simplified traveling direction image 136 (see FIG. 12 ). The composite captured image 130 is then displayed on the screen 36A. The simplified traveling direction image 136 is displayed within the composite captured image 130 ( FIG. 14 ). The direction indicated by the arrow in the simplified traveling direction image 136 corresponds to the traveling direction 94 of the tube 92 at the position of the opening 90A1. Therefore, the doctor 14 observing the composite image 130 can visually recognize the running direction 94 of the tube 92 leading to the duodenum 88 through the composite image 130 .

[0138] In the duodenoscope system 10, the duct 92 includes a bile duct 92A and a pancreatic duct 92B. Therefore, the doctor 14 observing the composite captured image 130 can visually recognize, through the composite captured image 130, the running direction 94A of the bile duct 92A leading to the duodenum 88 and the running direction 94B of the pancreatic duct 92B leading to the duodenum 88.

[0139] Furthermore, in the duodenoscope system 10, the position of the three-dimensional traveling direction image 124 is adjusted by aligning the first nipple region 108A with the second nipple region 118A. The first nipple region 108A and the second nipple region 118A are both image regions that show the nipple 90. Therefore, the first nipple region 108A and the second nipple region 118A can be easily aligned, and as a result, the position of the three-dimensional traveling direction image 124 can be easily adjusted.

[0140] Furthermore, in the duodenoscope system 10, a simplified traveling direction image 136 is generated based on the three-dimensional traveling direction image 124 adjusted by aligning the first papilla region 108A and the second papilla region 118A. The first papilla region 108A is an image region obtained from the first duodenum image 108 generated based on the captured image 40, and the second papilla region 118A is an image region obtained from the observation target region image 116 acquired from the volume data 112. Therefore, the first papilla region 108A and the second papilla region 118A used to generate the simplified traveling direction image 136 can be easily obtained. As a result, the simplified traveling direction image 136 can be easily generated.

[0141] Furthermore, in the duodenoscope system 10, each time the captured image 40 is updated frame by frame, a first duodenum image 108 is generated based on the updated captured image 40. Furthermore, each time the first duodenum image 108 is generated, a simplified traveling direction image 136 is generated based on the three-dimensional traveling direction image 124. A composite captured image 130 based on the updated captured image 40 is displayed on the screen 36A, and each time a simplified traveling direction image 136 is generated, the generated simplified traveling direction image 136 is displayed within the composite captured image 130 (see FIG. 14 ). Therefore, each time the captured image 40 is updated, the physician 14 observing the composite captured image 130 generated based on the captured image 40 can visually recognize the traveling direction 94 of the tube 92 leading to the duodenum 88 through the composite captured image 130.

[0142] Furthermore, in the duodenoscope system 10, the simplified travel direction image 136 displayed within the composite image 130 is updated in real time. Therefore, the doctor 14 observing the composite image 130 can visually recognize the current travel direction 94 of the tube 92 corresponding to the composite image 130 displayed on the screen 36A through the composite image 130.

[0143] Furthermore, in the duodenoscope system 10, an image in which the three-dimensional traveling direction image 124 is rendered on the captured image 40 set as a projection plane at a position directly opposite the three-dimensional region 108B in the composite image 123 is used as the simplified traveling direction image 136. Therefore, the traveling direction 94 of the tube 92 can be displayed as a two-dimensional image in the captured image 40.

[0144] Furthermore, in the duodenoscope system 10, a simplified traveling direction image 136 is displayed on the screen 36A along the tangent line 134. The tangent line 134 is a tangent line to the duct 92 at the opening 90A1 of the papilla 90, and is associated with the nipple image 110 showing the papilla 90. Therefore, the simplified traveling direction image 136 is displayed on the screen 36A in association with the nipple image 110. This allows the doctor 14 observing the composite captured image 130 to visually recognize the relationship (e.g., positional relationship) between the traveling direction 94 of the duct 92 and the papilla 90.

[0145] Furthermore, in the duodenoscope system 10, an image showing a traveling direction 94 along the tube 92 starting from the opening 90A1 of the papilla 90 is displayed as a simplified traveling direction image 136 within the composite captured image 130. This allows the physician 14 observing the composite captured image 130 to visually recognize the traveling direction 94 along the tube 92 starting from the opening 90A1 of the papilla 90. Furthermore, the traveling direction 94 along the tube 92 starting from the opening 90A1 of the papilla 90 corresponds to the insertion direction of the cannula 54A or the like, and therefore can assist the physician 14 observing the composite captured image 130 in smoothly inserting the cannula 54A or the like into the tube 92.

[0146] Furthermore, in the duodenoscope system 10, the scale of the three-dimensional traveling direction image 124 is adjusted based on the distance 125 measured by the distance sensor 49 and the observation target region image 116 acquired from the volume data 112. Therefore, a two-dimensional traveling direction image 132 can be generated based on the highly accurate three-dimensional traveling direction image 124.

[0147] In the above embodiment, an example in which the simplified traveling direction image 136, which is an example of a "traveling direction image" according to the technology of the present disclosure, is displayed on the screen 36A, is given, but this is merely one example. For example, instead of the simplified traveling direction image 136, or together with the simplified traveling direction image 136, all or part of the two-dimensional traveling direction image 132 (for example, an image of a part continuing from the opening 90A1 to the back side of the pipe 92) may be displayed on the screen 36A.

[0148] In the above embodiment, an example has been described in which the composition unit 80E adjusts the position of the duct image 120 by comparing and aligning the observation target site image 116 and the first duodenum image 108. However, the technology of the present disclosure is not limited to this. For example, the composition unit 80E may adjust the position of the duct image 120 by aligning the observation target site image 116 and the first duodenum image 108 using three-dimensional coordinates defining the pixels of the observation target site image 116 and three-dimensional coordinates defining the pixels of the first duodenum image 108. In this case, the composition unit 80E may align the observation target site image 116 and the first duodenum image 108 based on, for example, distances from a reference position (e.g., the position of the pylorus) to multiple feature points of the papilla 90 (i.e., multiple distances 125 measured by the distance measurement sensor 49), the three-dimensional coordinates of the multiple feature points in the first duodenum image 108, and the three-dimensional coordinates of the multiple feature points in the observation target site image 116. The three-dimensional coordinates of multiple feature points in the first duodenum image 108 are an example of "first reference site information" according to the technology of the present disclosure, and the three-dimensional coordinates of multiple feature points in the observation target site image 116 are an example of "second reference site information" according to the technology of the present disclosure.

[0149] In the above embodiment, an example has been described in which the synthesis unit 80E aligns the first duodenum image 108 with the observation site image 116 using the first papilla region 108A and the second papilla region 118A indicating the papilla 90, but the technology of the present disclosure is not limited to this. For example, the synthesis unit 80E may align the first duodenum image 108 with the observation site image 116 using information (e.g., three-dimensional coordinates or a three-dimensional image) capable of three-dimensionally identifying a medical marker (e.g., a hemostatic clip) included in the duodenum 88 and / or information (e.g., three-dimensional coordinates or a three-dimensional image) capable of three-dimensionally identifying the folds of the duodenum 88. In this case, the same effect as in the above embodiment can be obtained.

[0150] Although information capable of three-dimensionally identifying the nipple 90, information capable of three-dimensionally identifying a medical marker, and / or information capable of three-dimensionally identifying a fold has been exemplified herein as an example of the "first reference site information" according to the technology of the present disclosure, information capable of three-dimensionally identifying a reference site used for alignment may be determined in accordance with instructions given by the doctor 20 observing the captured image 40 displayed on the screen 36A (e.g., instructions accepted by the reception device 62). Furthermore, when the treatment tool 54 reaches the opening 90A1, the tip position of the treatment tool 54 (i.e., the position at which the treatment tool 54 is in contact with the opening 90A1) may be detected by AI or non-AI image recognition processing in accordance with instructions given by the doctor 20 (e.g., instructions accepted by the reception device 62), and the information capable of three-dimensionally identifying the detected tip position may be used as information capable of three-dimensionally identifying the nipple 90.

[0151] In the above embodiment, the second reference distance 128 is given as an example of the "distance from the endoscope to the intestinal wall" according to the technology of the present disclosure, and an example in which the distance from the pylorus to the papilla 90 is measured using the second duodenal image 118 is given. However, this is merely an example. For example, the second reference distance 128 may be measured using position information (i.e., information indicating the current position of the camera 48) and attitude information (i.e., information indicating the attitude of the camera 48) obtained using an optical fiber sensor provided along the longitudinal direction inside the duodenoscope main body 18 (e.g., the insertion section 44 and the distal end section 46). Furthermore, the second reference distance 128 is not limited to such an optical fiber method, and may also be measured using a conventionally known electromagnetic navigation method.

[0152] In the above embodiment, the second reference distance 128 is given as an example of the "distance from the endoscope to the intestinal wall" according to the technology of the present disclosure, and the distance from the pylorus to the papilla 90 is given as an example of the second reference distance 128, but this is merely an example. Other examples of the distance from the camera 48 to the intestinal wall 88A include the distance from the tip of the tip portion 16 to the intestinal wall 88A, or the distance from the imaging position of the camera 48 (i.e., the current position of the camera 48) to the intestinal wall 88A.

[0153] In the above embodiment, an example (see FIG. 6 ) has been given in which the first duodenum image 108 is generated based on the time-series image group 106, but this is merely one example. For example, as shown in FIG. 16 , the first acquisition unit 80A may generate the first duodenum image 108 based on a plurality of distance images 146. In this case, for example, the first acquisition unit 80A acquires the plurality of distance images 146 using a distance image generation model 144. The acquisition of the plurality of distance images 146 is realized, for example, by generating the plurality of distance images 146 using a 3D construction model 98.

[0154] The distance image generation model 144 is a generation model using a neural network, and generates a plurality of distance images 146 based on the time-series image group 106 and the first reference distance 126. The distance image generation model 144 is a trained model obtained by performing machine learning on a neural network using, as training data, a plurality of images corresponding to the time-series image group 106 and a distance corresponding to the first reference distance 126. An example of the distance image generation model 144 is an autoencoder such as a VAE or a GAN.

[0155] The first acquisition unit 80A inputs the time-series image group 106 and the first reference distance 126 to the distance image generation model 144. As a result, the distance image generation model 144 generates and outputs a plurality of distance images 146 based on the time-series image group 106 and the first reference distance 126. The first acquisition unit 80A acquires the plurality of distance images 146 output from the distance image generation model 144.

[0156] The first acquisition unit 80A generates the first duodenum image 108 using the 3D construction model 148. The 3D construction model 148 is a generation model using a neural network, and generates the first duodenum image 108 based on the plurality of distance images 146. The 3D construction model 148 differs from the 3D construction model 98 shown in Fig. 6 in that it is a trained model obtained by performing machine learning on a neural network using a plurality of distance images corresponding to the plurality of distance images 146 as training data, instead of a plurality of images corresponding to the time-series image group 106.

[0157] The first acquisition unit 80A inputs the plurality of distance images 146 into the 3D construction model 148. In response, the 3D construction model 148 generates and outputs the first duodenum image 108 based on the plurality of distance images 146. The first acquisition unit 80A acquires the first duodenum image 108 output from the 3D construction model 148. In the duodenoscope system 10, the first duodenum image 108 acquired by the first acquisition unit 80A is used in the same manner as in the above embodiment, thereby achieving the same effects as in the above embodiment.

[0158] In the above embodiment, the third acquisition unit 80C forms the simplified traveling direction image 136 along the tangent line 134, starting from the end of the tangent line 134 on the nipple image 110 side (i.e., the position where the opening 90A1 leading to the tube 92 (see FIG. 4) is shown), but the technology of the present disclosure is not limited to this. For example, as shown in FIG. 17 , the third acquisition unit 80C may form the simplified traveling direction image 136 along an extension of the tangent line 134.

[0159] In the example shown in FIG. 17 , the simplified traveling direction image 136 is arranged so that the tip of the arrow indicated by the simplified traveling direction image 136 is located at the end of the tangent 134 on the papilla image 110 side (i.e., the position where the opening 90A1 leading to the duct 92 (see FIG. 4) is shown). Specifically, the simplified bile duct direction image 136A is formed along an extension of the bile duct tangent 134A so that the arrow indicated by the simplified bile duct direction image 136A points to the end of the bile duct tangent 134A on the papilla image 110 side (i.e., the position where the opening 90A1 leading to the bile duct 92A (see FIG. 4) is shown). Furthermore, the simplified pancreatic duct direction image 136B is formed along an extension of the pancreatic duct tangent 134B so that the arrow indicated by the simplified pancreatic duct direction image 136B points to the end of the pancreatic duct tangent 134B on the papilla image 110 side (i.e., the position where the opening 90A1 leading to the pancreatic duct 92B (see FIG. 4) is shown).

[0160] In this way, the position indicated by the arrow shown in the simplified bile duct direction image 136A (i.e., the position of the tip of the arrow shown in the simplified bile duct direction image 136A) corresponds to the position of the opening 90A1 of the bile duct 92A, allowing the physician 14 observing the composite image 130 to visually grasp the position of the opening 90A1 of the bile duct 92A and the insertion direction of the cannula 54 or the like into the opening 90A1 of the bile duct 92A. Furthermore, the position indicated by the arrow shown in the pancreatic duct / bile duct direction image 136B (i.e., the position of the tip of the arrow shown in the simplified pancreatic duct direction image 136B) corresponds to the position of the opening 90A1 of the pancreatic duct 92B, allowing the physician 14 observing the composite image 130 to visually grasp the position of the opening 90A1 of the pancreatic duct 92B and the insertion direction of the cannula 54 or the like into the opening 90A1 of the pancreatic duct 92B.

[0161] In the above embodiment, an example in which the composite image 123, the composite captured image 130, and the support information 140 are displayed on the display device 13 has been described, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 18 , when the image processing device 25 or the control device 22 is connected to a server 152 (e.g., a server that manages electronic medical records) via a network 150, the composite image 123, the composite captured image 130, and / or the support information 140 may be output from the image processing device 25 or the control device 22 to the server 152 and stored in the server 152. Furthermore, the composite image 123, the composite captured image 130, and / or the support information 140 may be stored in the electronic medical record.

[0162] Furthermore, if a printer 154 is connected to the network 150, the composite image 123, the composite captured image 130, and / or the support information 140 may be output from the image processing device 25 or the control device 22 to the printer 154, and the printer 154 may print the composite image 123, the composite captured image 130, and the support information 140 on a recording medium (e.g., paper, etc.).

[0163] In the above embodiment, an example has been described in which the nipple image 110 is detected by executing AI processing (i.e., processing using the nipple detection model 100), but the technology of the present disclosure is not limited to this, and the nipple image 110 may be detected by executing non-AI processing (e.g., template matching, etc.). The same can be said for processing using the type prediction model 102 shown in Figure 13.

[0164] In the above embodiment, an example was described in which medical support processing was performed by the processor 70 of the computer 76 included in the duodenoscope 12, but the technology of the present disclosure is not limited to this, and the device that performs medical support processing may be provided external to the duodenoscope 12. Examples of devices that may be provided external to the duodenoscope 12 include at least one server and / or at least one personal computer that are communicatively connected to the duodenoscope 12. Furthermore, medical support processing may be performed in a distributed manner by multiple devices.

[0165] In the above embodiment, an example in which the medical support program 96 is stored in the NVM 74 has been described, but the technology of the present disclosure is not limited to this. For example, the medical support program 96 may be stored in a computer-readable non-transitory storage medium such as an SSD or USB memory. The non-transitory storage medium may be a stationary non-transitory storage medium or a portable non-transitory storage medium. The medical support program 96 stored in the non-transitory storage medium is installed in the computer 76 of the duodenoscope 12. The processor 70 executes medical support processing in accordance with the medical support program 96.

[0166] In addition, the medical support program 96 may be stored in a storage device such as another computer or server connected to the duodenoscope 12 via a network, and the medical support program 96 may be downloaded and installed on the computer 76 in response to a request from the duodenoscope 12.

[0167] It is not necessary to store the entire medical support program 96 in a storage device such as another computer or server device connected to the duodenoscope 12, or in the NVM 74; only a portion of the medical support program 96 may be stored therein.

[0168] The hardware resources that execute the medical support processing can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes medical support processing by executing software, i.e., a program. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with a circuit configuration specifically designed to execute specific processing. Each processor has built-in or connected memory, and executes medical support processing by using the memory.

[0169] The hardware resource for executing the medical support processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the medical support processing may be a single processor.

[0170] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute medical support processing, on a single IC chip, as typified by SoC. In this way, medical support processing is realized using one or more of the various processors described above as hardware resources.

[0171] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0172] The above-described contents and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described contents and illustrations within the scope of the gist of the technology of the present disclosure.

[0173] Furthermore, in order to avoid confusion and to facilitate understanding of the parts relating to the technology of the present disclosure, the above description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable implementation of the technology of the present disclosure.

[0174] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0175] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. Comprising a processor, The processor is configured to: Based on an imaging image obtained by imaging the intestinal wall of the duodenum with an endoscope scope, acquire first reference site information capable of identifying a reference site included in the duodenum in three dimensions, From the volume data, acquire second reference site information regarding the reference site and direction information regarding the running direction of a tube communicating with the duodenum, Based on the direction information adjusted by aligning the first reference site information and the second reference site information, generate a running direction image in which the running direction is imaged in accordance with the imaging image, Display the imaging image on a screen, Display the running direction image within the imaging image A medical support device.

2. The first reference site information is a first image area in which the reference site is represented as a three-dimensional image, The second reference site information is a second image area in which the reference site is represented as a three-dimensional image, The processor generates the running direction image based on the direction information adjusted by aligning the first image area and the second image area The medical support device according to claim 1.

3. The processor is configured to: Based on the imaging image, generate a duodenum image in which the duodenum is represented as a three-dimensional image, Acquire the first reference site information from the duodenum image The medical support device according to claim 1.

4. The duodenum image is a three-dimensional image generated based on a plurality of distance images The medical support device according to claim 3.

5. Each time the imaging image is updated in units of a specified number of frames, The processor is configured to: Based on the updated imaging image, generate the duodenum image, Each time the duodenum image is generated, generate the running direction image based on the direction information, Display the updated imaging image on the screen, Each time the running direction image is generated, display the generated running direction image within the imaging image The medical support device according to claim 4.

6. The direction information is a three-dimensional direction image in which the running direction is represented as a three-dimensional image, The running direction image is an image based on a two-dimensional image obtained by projecting the three-dimensional direction image onto the imaging image The medical support device according to claim 1.

7. The imaging image includes a papilla image showing the duodenal papilla, The running direction image is displayed in a state associated with the papilla image The medical support device according to claim 1.

8. The tube communicates with an opening within the major duodenal papilla, and the running direction image is an image showing a first direction along the tube with the opening as a reference point. The medical support device according to claim 7.

9. The first direction corresponds to the insertion direction of a medical instrument inserted into the tube. The medical support device according to claim 8.

10. The reference site is the major duodenal papilla, a medical marker, and / or a fold. The medical support device according to claim 1.

11. The processor adjusts the scale of the direction information based on the distance from the endoscope scope to the intestinal wall and the volume data. The medical support device according to claim 1.

12. The tube is a bile duct and / or a pancreatic duct. The medical support device according to claim 1.

13. The running direction image displayed in the captured image is updated in real time. The medical support device according to claim 1.

14. A medical support device according to any one of claims 1 to 13, and the endoscope scope, are provided. Endoscope.

15. Obtaining first reference site information capable of three-dimensionally specifying a reference site included in the duodenum based on a captured image obtained by imaging the intestinal wall of the duodenum with an endoscope scope; Obtaining second reference site information regarding the reference site and direction information regarding the running direction of a tube communicating with the duodenum from volume data; Generating a running direction image in which the running direction is imaged in accordance with the captured image based on the direction information adjusted by aligning the first reference site information and the second reference site information; Displaying the captured image on a screen, and displaying the running direction image within the captured image. Medical support method.

16. Causing a computer to obtain first reference site information capable of three-dimensionally specifying a reference site included in the duodenum based on a captured image obtained by imaging the intestinal wall of the duodenum with an endoscope scope; obtain second reference site information regarding the reference site and direction information regarding the running direction of a tube communicating with the duodenum from volume data; generate a running direction image in which the running direction is imaged in accordance with the captured image based on the direction information adjusted by aligning the first reference site information and the second reference site information; display the captured image on a screen, and A program for executing a process including displaying the traveling direction image in the captured image.

Images