Medical assistance device, endoscope, and medical assistance method

Abstract

Provided is a medical assistance device provided with a processor that performs: a process for acquiring nipple orientation-related information relating to the orientation of a duodenal nipple on the basis of an intestinal wall image; the intestinal wall image is obtained by photographing an intestinal wall including a duodenal papilla in the duodenum by using a camera provided in an endoscope observer; displaying the intestinal wall image on a picture; and displaying the nipple orientation-related information on a screen.

Description

Technical Field

[0001] The technology of the present invention relates to a medical support device, an endoscope, and a medical support method. Background Art

[0002] In Japanese Patent Application Laid-Open No. 2020-62218, a learning device is disclosed, which includes: an acquisition unit that acquires a plurality of pieces of information obtained by associating an image of the major duodenal papilla of the bile duct with information indicating a cannula insertion method as a method of inserting a catheter into the bile duct; a learning unit that machine-learns information indicating the cannula insertion method as training data based on the image of the major duodenal papilla of the bile duct; and a storage unit that associates and stores the result of machine learning by the learning unit with information indicating the cannula insertion method. Summary of the Invention

[0003] An embodiment of the technology of the present invention provides a medical support device, an endoscope, and a medical support method that can enable a user observing an intestinal wall image to visually grasp information related to the orientation of the major duodenal papilla.

[0004] Means for Solving the Technical Problem

[0005] A first aspect of the technology of the present invention is a medical support device, which includes a processor that performs the following processing: obtaining papilla orientation-related information related to the orientation of the major duodenal papilla based on an intestinal wall image, the intestinal wall image being obtained by photographing an intestinal wall including the major duodenal papilla in the duodenum using a camera provided in an endoscope viewer; displaying the intestinal wall image on a screen; and displaying the papilla orientation-related information on the screen.

[0006] In the medical support device according to the second aspect of the technology of the present invention, the papilla orientation-related information includes elevation direction information indicating the elevation direction of the major duodenal papilla.

[0007] In the medical support device according to the third aspect of the technology of the present invention, the papilla orientation information includes an elevation direction image indicating the elevation direction.

[0008] In the medical support device according to the fourth aspect of the technology of the present invention, among the first to third aspects, the major duodenal papilla has an opening, and the papilla orientation-related information includes surface direction information indicating the direction of the surface where the opening is located.

[0009] In the medical support device according to the fifth aspect of the technology of the present invention, the papilla orientation-related information includes angle-related information related to the relative angle between the surface and the posture of the endoscope viewer.

[0010] In the sixth aspect related to the technology of the present invention, in the medical support device related to any one of the first to third aspects, the duodenal papilla has an opening, and the papilla orientation-related information includes: surface direction information indicating the direction of the surface where the opening is located; and angle-related information related to the relative angle between the surface and the posture of the endoscope viewer.

[0011] In the seventh aspect related to the technology of the present invention, in the medical support device related to any one of the first to sixth aspects, the papilla orientation-related information includes a surface image capable of determining a surface that intersects the elevation direction of the duodenal papilla at a specified angle.

[0012] In the eighth aspect related to the technology of the present invention, in the medical support device related to any one of the first to seventh aspects, the papilla orientation-related information includes consistency information capable of determining the consistency between the elevation direction of the duodenal papilla and the optical axis direction of the endoscope viewer.

[0013] In the ninth aspect related to the technology of the present invention, in the medical support device related to any one of the first to eighth aspects, the duodenal papilla includes a papilla elevation and a circumferential fold covering the papilla elevation, and the papilla orientation-related information includes first direction information indicating a first direction from the top of the papilla elevation to the top of the circumferential fold.

[0014] In the tenth aspect related to the technology of the present invention, in the medical support device related to the ninth aspect, the first direction information includes a first direction image indicating the first direction.

[0015] In the eleventh aspect related to the technology of the present invention, in the medical support device related to the ninth or tenth aspect, the papilla elevation has an opening, and the papilla orientation-related information includes traveling direction information indicating the traveling direction of the tube leading to the opening, and the traveling direction information is set according to the first direction information.

[0016] In the twelfth aspect related to the technology of the present invention, in the medical support device related to the eleventh aspect, the traveling direction information includes a traveling direction image indicating the traveling direction.

[0017] In the thirteenth aspect related to the technology of the present invention, in the medical support device related to any one of the first to twelfth aspects, the duodenal papilla has a papilla elevation and a fold portion including a circumferential fold covering the papilla elevation, and the processor determines a second direction according to the way the fold portion is reflected in the intestinal image.

[0018] In the fourteenth aspect related to the technology of the present invention, in the medical support device related to the thirteenth aspect, the processor determines the second direction according to the way of the area including the papilla elevation and the fold portion reflected in the intestinal wall image.

[0019] A fifteenth aspect according to the technology of the present invention is the medical support device according to any one of the first to fourteenth aspects, wherein the processor acquires the information related to the nipple orientation by executing the first image recognition processing on the intestinal wall image.

[0020] In the 16th method involved in the technology of the present invention, in the medical support device involved in any one of the 1st to 15th methods, the processor performs the following processing: determining the travel direction of the tube leading to the opening of the duodenal papilla based on the intestinal wall image; and displaying the travel direction information that can determine the travel direction within the intestinal wall image on the screen.

[0021] In the 17th method involved in the technology of the present invention, in the medical support device involved in the 16th method, the processor performs the following processing: obtaining diverticulum area information, i.e., the diverticulum area, which is an image area representing the diverticulum in the intestinal wall image and can determine the diverticulum, based on the intestinal wall image; and changing the display method of the travel direction information based on the diverticulum area information.

[0022] An eighteenth aspect according to the technology of the present invention is the medical support device according to the seventeenth aspect, wherein the display aspect is an aspect in which the traveling direction avoids the diverticulum region specified based on the diverticulum region information.

[0023] In the 19th mode involved in the technology of the present invention, in the medical support device involved in any one of the 16th to 18th modes, the processor performs the following processing: obtaining diverticulum area information that can determine the image area representing the diverticulum in the intestinal wall image, that is, the diverticulum area, based on the intestinal wall image; determining the positional relationship between the diverticulum and the traveling direction based on the diverticulum area information and the traveling direction; when the positional relationship is a positional relationship in which the diverticulum and the traveling direction intersect, outputting notification information that notifies the content that the diverticulum is in a positional relationship in which the diverticulum and the traveling direction are in an intersecting positional relationship.

[0024] The 20th method involved in the technology of the present invention is a medical support device involved in any one of the 1st to 19th methods. When an endoscope having an endoscopic observer and a treatment instrument is inserted into the duodenum, the processor performs the following processing: determining a first relationship between the position of the treatment instrument and the position of the duodenal papilla and / or a second relationship between the forward direction of the treatment instrument and the orientation of the duodenal papilla based on an intestinal wall image in which the treatment instrument is reflected; and executing a first notification processing for performing a notification corresponding to the first relationship and / or the second relationship.

[0025] In the 21st mode related to the technology of the present invention, in the medical support device related to any one of the 1st to 20th modes, when an endoscope having an endoscope viewer and a treatment instrument is inserted into the duodenum, the processor performs the following processing: determining a third relationship between the advancing direction of the treatment instrument and a first orientation related to the orientation of the duodenal papilla based on the intestinal wall image showing the treatment instrument; and executing a second notification process for performing a notification corresponding to the third relationship.

[0026] In the 22nd mode related to the technology of the present invention, in the medical support device related to any one of the 1st to 21st modes, the processor performs the following processing: determining the traveling direction of a tube leading to the opening of the duodenal papilla based on the intestinal wall image; when an endoscope having an endoscope viewer and a treatment instrument is inserted into the duodenum, determining the advancing direction of the treatment instrument based on the intestinal wall image showing the treatment instrument; and executing a third notification process for performing a notification corresponding to a fourth relationship between the traveling direction and the advancing direction.

[0027] In the 23rd mode related to the technology of the present invention, in the medical support device related to any one of the 1st to 22nd modes, the nipple orientation related information includes incision recommendation direction information or incision non-recommendation direction information. The above-mentioned incision recommendation direction information represents a direction recommended as the incision direction of the duodenal papilla using an incision instrument for incising the duodenal papilla, and the above-mentioned incision non-recommendation direction information represents a direction not recommended as the incision direction.

[0028] In the 24th mode related to the technology of the present invention, in the medical support device related to any one of the 1st to 23rd modes, when an endoscope having an endoscope viewer and a treatment instrument is inserted into the duodenum, the processor performs the following processing: obtaining an evaluation value related to the positional relationship between the duodenal papilla and the treatment instrument based on the intestinal wall image showing the treatment instrument; and outputting information based on the evaluation value.

[0029] In the 25th mode related to the technology of the present invention, in the medical support device related to the 24th mode, when an endoscope having an endoscope viewer and a treatment instrument is inserted into the duodenum, the processor outputs information based on the evaluation value when it detects a state in which the treatment instrument contacts the duodenal papilla based on the intestinal wall image showing the treatment instrument.

[0030] The 26th mode related to the technology of the present invention is a medical support device, which includes a processor

[0031] The processor performs the following processing: determining the traveling direction of the tube leading to the opening of the duodenal papilla based on an intestinal wall image obtained by photographing the intestinal wall containing the duodenal papilla in the duodenum using a camera provided in an endoscope viewer; displaying the intestinal wall image on a screen; and displaying traveling direction information capable of determining the traveling direction within the intestinal wall image on the screen.

[0032] The 27th aspect of the technology of the present invention is an endoscope, which includes: a medical support device according to any one of the 1st to 26th aspects; and an endoscope viewer.

[0033] The 28th aspect of the technology of the present invention is a medical support method, which includes the following steps: obtaining nipple orientation-related information related to the orientation of the duodenal papilla based on an intestinal wall image obtained by photographing the intestinal wall containing the duodenal papilla in the duodenum using a camera provided in an endoscope viewer; displaying the intestinal wall image on a screen; and displaying the nipple orientation-related information on the screen.

[0034] The 29th aspect of the technology of the present invention is a medical support method, which includes the following steps: determining the traveling direction of the tube leading to the opening of the duodenal papilla based on an intestinal wall image obtained by photographing the intestinal wall containing the duodenal papilla in the duodenum using a camera provided in an endoscope viewer; displaying the intestinal wall image on a screen; and displaying traveling direction information capable of determining the traveling direction within the intestinal wall image on the screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a conceptual diagram showing an example of a mode using a duodenoscope system.

[0036] Figure 2 is a conceptual diagram showing an example of the overall structure of a duodenoscope system.

[0037] Figure 3 is a block diagram showing an example of the hardware structure of the electrical system of a duodenoscope system.

[0038] Figure 4 is a conceptual diagram showing an example of a mode using a duodenoscope.

[0039] Figure 5 is a block diagram showing an example of the hardware structure of the electrical system of an image processing device.

[0040] Figure 6 is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope main body, an image acquisition unit, an image recognition unit, and an export unit.

[0041] Figure 7 It is a conceptual diagram showing an example of the correlation among a display device, an image acquisition unit, an image recognition unit, an export unit, and a display control unit.

[0042] Figure 8 It is a flowchart showing an example of the process of medical support processing.

[0043] Figure 9 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope body, an image acquisition unit, an image recognition unit, and an export unit.

[0044] Figure 10 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope body, an image acquisition unit, an image recognition unit, and an export unit.

[0045] Figure 11 It is a conceptual diagram showing an example of the correlation among a display device, an image recognition unit, an export unit, and a display control unit.

[0046] Figure 12 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an export unit.

[0047] Figure 13 It is a conceptual diagram showing an example of the correlation among a display device, an image recognition unit, an export unit, and a display control unit.

[0048] Figure 14 It is a conceptual diagram showing an example of the correlation among a display device, an export unit, and a display control unit.

[0049] Figure 15 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope body, an image acquisition unit, an image recognition unit, and an export unit.

[0050] Figure 16 It is a conceptual diagram showing an example of the correlation among a display device, an image recognition unit, an export unit, and a display control unit.

[0051] Figure 17 It is a conceptual diagram showing an example of the method of aligning the endoscope viewer with the papilla.

[0052] Figure 18 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope body, an image acquisition unit, an image recognition unit, and an export unit.

[0053] Figure 19 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope body, an image acquisition unit, an image recognition unit, and an export unit.

[0054] Figure 20It is a conceptual diagram showing an example of the correlation among a display device, an image recognition unit, an output unit, and a display control unit.

[0055] Figure 21 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, a duodenoscope body, an image acquisition unit, an image recognition unit, and an output unit.

[0056] Figure 22 It is a conceptual diagram showing an example of the correlation among a display device, an image recognition unit, an output unit, and a display control unit.

[0057] Figure 23 It is a conceptual diagram showing an example of the correlation among a display device, an image recognition unit, an output unit, and a display control unit.

[0058] Figure 24 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an output unit.

[0059] Figure 25 It is a conceptual diagram showing an example of the correlation among a display device, an output unit, and a display control unit.

[0060] Figure 26 It is a flowchart showing an example of the process of medical support processing.

[0061] Figure 27 It is a conceptual diagram showing an example of the correlation among a display device, an output unit, and a display control unit.

[0062] Figure 28 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an output unit.

[0063] Figure 29 It is a conceptual diagram showing an example of the correlation among a display device, an output unit, and a display control unit.

[0064] Figure 30 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an output unit.

[0065] Figure 31 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an output unit.

[0066] Figure 32 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an output unit.

[0067] Figure 33 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an output unit.

[0068] Figure 34 It is a conceptual diagram showing an example of the correlation among a display device, an output unit, and a display control unit.

[0069] Figure 35 It is a conceptual diagram showing an example of the correlation among an endoscope viewer, an image acquisition unit, an image recognition unit, and an output unit.

[0070] Figure 36 It is a conceptual diagram showing an example of the correlation among a display device, an output unit, and a display control unit.

[0071] Figure 37 It is a conceptual diagram showing an example of the correlation among a display device, an output unit, and a display control unit. Detailed implementation manners

[0072] Hereinafter, an example of an embodiment of a medical support device, an endoscope, and a medical support method according to the technology of the present invention will be described with reference to the accompanying drawings.

[0073] First, the terms used in the following description will be described.

[0074] CPU refers to the abbreviation of "Central Processing Unit: Central Processor". GPU refers to the abbreviation of "Graphics Processing Unit: Graphics Processor". RAM refers to the abbreviation of "Random Access Memory: Random Access Memory". NVM refers to the abbreviation of "Non-volatile memory: Non-volatile Memory". EEPROM refers to the abbreviation of "Electrically Erasable Programmable Read-Only Memory: Electrically Erasable Programmable Read-Only Memory". ASIC refers to the abbreviation of "Application Specific Integrated Circuit: Application Specific Integrated Circuit". PLD refers to the abbreviation of "Programmable Logic Device: Programmable Logic Device". FPGA refers to the abbreviation of "Field-Programmable Gate Array: Field Programmable Gate Array". SoC refers to the abbreviation of "System-on-a-chip: System-on-a-chip". SSD refers to the abbreviation of "Solid State Drive: Solid State Drive". USB refers to the abbreviation of "Universal Serial Bus: Universal Serial Bus". HDD refers to the abbreviation of "Hard Disk Drive: Hard Disk Drive". EL refers to the abbreviation of "Electro-Luminescence: Electro-Luminescence". CMOS refers to the abbreviation of "Complementary Metal Oxide Semiconductor: Complementary Metal Oxide Semiconductor". CCD refers to the abbreviation of "Charge Coupled Device: Charge Coupled Device". AI refers to the abbreviation of "Artificial Intelligence: Artificial Intelligence". BLI refers to the abbreviation of "Blue Light Imaging: Blue Light Imaging". LCI refers to the abbreviation of "Linked Color Imaging: Linked Color Imaging". I / F refers to the abbreviation of "Interface: Interface". FIFO refers to the abbreviation of "First In First Out: First In First Out". ERCP refers to the abbreviation of "Endoscopic Retrograde Cholangio-Pancreatography: Endoscopic Retrograde Cholangio-Pancreatography". TOF refers to the abbreviation of "Time of Flight: Time of Flight".

[0075] <The First Embodiment>

[0076] As an example, such as Figure 1As shown, the duodenoscope system 10 includes a duodenoscope 12 and a display device 13. The duodenoscope 12 is used by a doctor 14 during an endoscopic examination. The duodenoscope 12 is communicably connected to a communication device (not shown), and the information obtained by the duodenoscope 12 is sent to the communication device. The communication device receives the information sent from the duodenoscope 12 and performs processing using the received information (for example, processing such as recording in an electronic medical record, etc.).

[0077] The duodenoscope 12 includes an endoscope viewer 18. The duodenoscope 12 is a device for diagnosing and treating an observation object 21 (for example, the duodenum) contained in the body of a subject 20 (for example, a patient) using the endoscope viewer 18. The observation object 21 is the object observed by the doctor 14. The endoscope viewer 18 is inserted into the body of the subject 20. The duodenoscope 12 causes the endoscope viewer 18 inserted into the body of the subject 20 to photograph the observation object 21 in the body of the subject 20, and performs various medical treatments on the observation object 21 as needed. The duodenoscope 12 is an example of the "endoscope" related to the technology of the present invention.

[0078] The duodenoscope 12 obtains and outputs an image representing the internal morphology by photographing the inside of the body of the subject 20. In the present embodiment, the duodenoscope 12 is an endoscope having an optical imaging function, and this optical imaging function photographs the reflected light obtained by irradiating light in the body and reflected by the observation object 21.

[0079] The duodenoscope 12 includes a control device 22, a light source device 24, and an image processing device 25. The control device 22 and the light source device 24 are provided on a trolley 34. A plurality of them are arranged in the vertical direction on the trolley 34, and the image processing device 25, the control device 22, and the light source device 24 are arranged from the lower-stage trolley to the upper-stage trolley. And, the display device 13 is provided on the topmost stage of the trolley 34.

[0080] The control device 22 is a device that controls the entire duodenoscope 12. And, the image processing device 25 is a device that performs image processing on the images photographed by the duodenoscope 12 under the control of the control device 22.

[0081] The display device 13 displays various information including images (for example, images that have undergone image processing by the image processing device 25). As an example of the display device 13, a liquid crystal display or an EL display, etc. can be cited. And, instead of or together with the display device 13, a tablet terminal with a display can be used.

[0082] A plurality of screens are arranged and displayed on the display device 13. In Figure 1In the example shown, screens 36, 37, and 38 are shown. An endoscopic image 40 obtained by the duodenoscope 12 is displayed on screen 36. An observation object 21 is reflected in the endoscopic image 40. The endoscopic image 40 is an image obtained by photographing the observation object 21 with a camera 48 (refer to Figure 2 ) provided in the endoscopic viewer 18 inside the body of the subject 20. As the observation object 21, the intestinal wall of the duodenum can be cited. Hereinafter, for the sake of convenience of explanation, an endoscopic image 40 obtained by photographing the intestinal wall of the duodenum as the observation object 21, that is, an intestinal wall image 41, will be described. In addition, the duodenum is merely an example, and any area that can be photographed by the duodenoscope 12 is acceptable. As an area that can be photographed by the duodenoscope 12, for example, the esophagus or the stomach can be cited. The intestinal wall image 41 is an example of the "intestinal wall image" related to the technology of the present invention.

[0083] A moving image composed of multiple frames of intestinal wall images 41 is displayed on screen 36. That is, on screen 36, multiple frames of intestinal wall images 41 are displayed at a predetermined frame rate (for example, dozens of frames per second).

[0084] As an example, as shown in Figure 2 , the duodenoscope 12 includes an operation unit 42 and an insertion unit 44. The insertion unit 44 is locally bent by operating the operation unit 42. The insertion unit 44 is inserted while bending according to the shape of the observation object 21 (for example, the shape of the duodenum) in accordance with the operation of the operation unit 42 by the doctor 14.

[0085] A camera 48, a lighting device 50, a treatment opening 51, and an erecting mechanism 52 are provided at the front end 46 of the insertion unit 44. The camera 48 and the lighting device 50 are provided on the side of the front end 46. That is, the duodenoscope 12 becomes a side-viewing endoscope. Thus, it is easy to observe the intestinal wall of the duodenum.

[0086] The camera 48 is a device that obtains the intestinal wall image 41 as a medical image by photographing inside the body of the subject 20. As an example of the camera 48, a CMOS camera can be cited. However, this is merely an example, and other types of cameras such as a CCD camera can also be used. The camera 48 is an example of the "camera" related to the technology of the present invention.

[0087] The illumination device 50 has an illumination window 50A. The illumination device 50 irradiates light through the illumination window 50A. As the types of light irradiated from the illumination device 50, for example, visible light (e.g., white light, etc.) and non-visible light (e.g., near-infrared light, etc.) can be cited. Also, the illumination device 50 irradiates special light through the illumination window 50A. As special light, for example, light for BLI and / or light for LCI can be cited. The camera 48 optically photographs the inside of the subject 20 in a state where the inside of the subject 20 is irradiated with light from the illumination device 50.

[0088] The treatment opening 51 serves as an instrument protrusion opening for allowing the treatment instrument 54 to protrude from the front end portion 46, a suction port for sucking blood and internal dirt, etc., and a fluid delivery port for delivering fluid.

[0089] According to the operation of the doctor 14, the treatment instrument 54 protrudes from the treatment opening 51. The treatment instrument 54 is inserted into the insertion portion 44 from the treatment instrument insertion opening 58. The treatment instrument 54 passes through the inside of the insertion portion 44 via the treatment instrument insertion opening 58 and protrudes into the body of the subject 20 from the treatment opening 51. In Figure 2 In the example shown, as the treatment instrument 54, a cannula protrudes from the treatment opening 51. The cannula is only an example of the treatment instrument 54. As another example of the treatment instrument 54, a papillotome or a snare, etc. can be cited.

[0090] The erecting mechanism 52 changes the protruding direction of the treatment instrument 54 protruding from the treatment opening 51. The erecting mechanism 52 includes a guide member 52A. By raising the guide member 52A with respect to the protruding direction of the treatment instrument 54, the protruding direction of the treatment instrument 54 changes along the guide member 52A. Thereby, it is easy to make the treatment instrument 54 protrude toward the intestinal wall. In Figure 2 In the example shown, by the erecting mechanism 52, the protruding direction of the treatment instrument 54 is changed to a direction orthogonal to the advancing direction of the front end portion 46. The erecting mechanism 52 is operated by the doctor 14 via the operation portion 42. Thereby, the degree of change in the protruding direction of the treatment instrument 54 can be adjusted.

[0091] The endoscope viewer 18 is connected to the control device 22 and the light source device 24 via a general-purpose cord 60. A display device 13 and a receiving device 62 are connected to the control device 22. The receiving device 62 receives an instruction from a user (e.g., the doctor 14) and outputs the received instruction as an electrical signal. In Figure 2 In the example shown, as an example of the receiving device 62, a keyboard can be cited. However, this is only an example, and the receiving device 62 can also be a mouse, a touch panel, a foot switch, and / or a microphone, etc.

[0092] The control device 22 controls the entire duodenoscope 12. For example, the control device 22 controls the light source device 24 or performs transmission and reception of various signals with the camera 48. The light source device 24 emits light under the control of the control device 22 and supplies light to the illumination device 50. A light guide is built into the illumination device 50, and the light supplied from the light source device 24 is irradiated from the illumination windows 50A and 50B through the light guide. The control device 22 causes the camera 48 to perform imaging, obtains the intestinal wall image 41 (refer to Figure 1 ) from the camera 48 and outputs it to a specified output destination (for example, the image processing device 25).

[0093] The image processing device 25 is communicably connected to the control device 22, and the image processing device 25 performs image processing on the intestinal wall image 41 output from the control device 22. Details of the image processing in the image processing device 25 will be described later. The image processing device 25 outputs the intestinal wall image 41 on which image processing has been performed to a specified output destination (for example, the display device 13). In addition, here, an example of the method in which the intestinal wall image 41 output from the control device 22 is output to the display device 13 via the image processing device 25 has been described, but this is merely an example. It may also be a method in which the control device 22 is connected to the display device 13, and the intestinal wall image 41 on which image processing has been performed by the image processing device 25 is displayed on the display device 13 via the control device 22.

[0094] As an example, as Figure 3 shown, the control device 22 includes a computer 64, a bus 66, and an external I / F 68. The computer 64 includes a processor 70, a RAM 72, and an NVM 74. The processor 70, the RAM 72, the NVM 74, and the external I / F 68 are connected to the bus 66.

[0095] For example, the processor 70 has a CPU and a GPU and controls the entire control device 22. The GPU operates under the control of the CPU and undertakes various processes of the graphics system and operations using neural networks, etc. In addition, the processor 70 may be one or more CPUs integrated with GPU functions, or may be one or more CPUs without integrated GPU functions.

[0096] The RAM 72 is a memory that temporarily stores information and is used as a working memory by the processor 70. The NVM 74 is a non-volatile storage device that stores various programs and various parameters, etc. As an example of the NVM 74, a flash memory (for example, EEPROM and / or SSD) can be cited. In addition, the flash memory is merely an example, and it may be other non-volatile storage devices such as an HDD, or a combination of two or more non-volatile storage devices.

[0097] The external I / F 68 is responsible for the transmission and reception of various information between the devices existing outside the control device 22 (hereinafter, also referred to as "external devices") and the processor 70. As an example of the external I / F 68, a USB interface can be cited.

[0098] Connected to the external I / F 68 is a camera 48 as one of the external devices. The external I / F 68 is responsible for the transmission and reception of various information between the camera 48 provided in the endoscope viewer 18 and the processor 70. The processor 70 controls the camera 48 via the external I / F 68. Further, the processor 70 acquires, via the external I / F 68, the intestinal wall image 41 obtained by photographing the inside of the subject 20 with the camera 48 provided in the endoscope viewer 18 (refer to Figure 1 ).

[0099] Connected to the external I / F 68 is a light source device 24 as one of the external devices. The external I / F 68 is responsible for the transmission and reception of various information between the light source device 24 and the processor 70. The light source device 24 supplies light to the illumination device 50 under the control of the processor 70. The illumination device 50 irradiates the light supplied from the light source device 24.

[0100] Connected to the external I / F 68 is a receiving device 62 as one of the external devices. The processor 70 acquires, via the external I / F 68, the instruction received by the receiving device 62 and executes the process corresponding to the acquired instruction.

[0101] Connected to the external I / F 68 is an image processing device 25 as one of the external devices. The processor 70 outputs the intestinal wall image 41 to the image processing device 25 via the external I / F 68.

[0102] In the treatment of the duodenum using an endoscope, a treatment called ERCP (endoscopic retrograde cholangiopancreatography) examination is sometimes performed. As an example, as Figure 4 shown, in the ERCP examination, for example, first the duodenoscope 12 is inserted into the duodenum J via the esophagus and the stomach. At this time, the insertion state of the duodenoscope 12 can be confirmed by X-ray imaging. Further, the distal end portion 46 of the duodenoscope 12 reaches near the duodenal papilla N (hereinafter, also simply referred to as "papilla N") existing on the intestinal wall of the duodenum J.

[0103] In an ERCP examination, for example, a cannula 54A is inserted from the papilla N. Here, the papilla N is a part that protrudes from the intestinal wall of the duodenum J, and the openings of the ends of the bile duct T (for example, the common bile duct, intrahepatic bile duct, cystic duct) and the pancreatic duct S are present in the papilla elevation NA of the papilla N. An X-ray imaging is performed in a state where a contrast agent is injected into the bile duct T and the pancreatic duct S etc. through the cannula 54A from the opening of the papilla N. Thus, in an ERCP examination, various surgical procedures are included, such as inserting a duodenoscope 12 into the duodenum J, confirming the position, orientation, and type of the papilla N, and further inserting a treatment instrument (for example, a cannula) into the papilla N. Therefore, the doctor 14 needs to operate the duodenoscope 12 and observe the state of the target site according to each surgical procedure.

[0104] For example, in the case of inserting the duodenoscope 12 into the duodenum J, if the endoscope viewer 18 of the duodenoscope 12 is in a state of being inclined relative to the intestinal tract direction, the papilla N will be visually recognized in an inclined state, so it is possible to misidentify the traveling directions of the bile duct T and the pancreatic duct S from the papilla N. Therefore, it is necessary to know to what extent the posture of the endoscope viewer 18 is inclined relative to the intestinal tract direction within the duodenum J.

[0105] Therefore, in view of this situation, in order to support the implementation of medical treatment for the duodenum including the ERCP examination, a medical support process is performed by the processor 82 of the image processing device 25.

[0106] As an example, as Figure 5 shown, the image processing device 25 includes a computer 76, an external I / F 78, and a bus 80. The computer 76 includes a processor 82, an NVM 84, and a RAM 81. The processor 82, the NVM 84, the RAM 81, and the external I / F 78 are connected to the bus 80. The computer 76 is an example of the "medical support device" and the "computer" related to the technology of the present invention. The processor 82 is an example of the "processor" related to the technology of the present invention.

[0107] In addition, the hardware structure of the computer 76 (that is, the processor 82, the NVM 84, and the RAM 81) is basically the same as the hardware structure of the computer 64 shown in Figure 3 shown, so the description related to the hardware structure of the computer 76 is omitted here. And the role of the external I / F 78 in the image processing device 25 for receiving and transmitting information with the outside is basically the same as the role of the external I / F 68 in the control device 22 shown in Figure 3 shown, so the description is omitted here.

[0108] A medical support program 84A is stored in the NVM 84. The medical support program 84A is an example of the "program" related to the technology of the present invention. The processor 82 reads the medical support program 84A from the NVM 84 and executes the read medical support program 84A on the RAM 81. The medical support process according to the present embodiment is implemented by the processor 82 operating as an image acquisition unit 82A, an image recognition unit 82B, a derivation unit 82C, and a display control unit 82D according to the medical support program 84A executed on the RAM 81.

[0109] A learned model 84B is stored in the NVM 84. In the present embodiment, the image recognition unit 82B performs image recognition processing in an AI manner as object detection image recognition processing. The learned model 84B is optimized by performing machine learning on a neural network in advance.

[0110] As an example, as Figure 6 shown, the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48 in units of one frame, and the intestinal wall image 41 is generated by the camera 48 shooting at a shooting frame rate (for example, dozens of frames per second).

[0111] The image acquisition unit 82A holds the time-series image group 89. The time-series image group 89 is a plurality of intestinal wall images 41 showing the time series of the observation object 21. In the time-series image group 89, for example, the intestinal wall images 41 of a specified number of frames (for example, the number of frames preset in the range of dozens to hundreds of frames) are included. Each time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0112] Here, an example of the method of holding and updating the time-series image group 89 by the image acquisition unit 82A is given, but this is only an example. For example, the time-series image group 89 may also be held and updated in a memory connected to the processor 82, such as the RAM 81.

[0113] The image recognition unit 82B performs image recognition processing on the time-series image group 89 using the learned model 84B. By performing the image recognition processing, the intestinal tract direction CD included in the observation object 21 is detected. Here, the intestinal tract direction CD refers to the lumen direction of the duodenum. Here, the detection of the intestinal tract direction refers to the process of storing in the memory in a state where the intestinal tract direction information 90 (for example, position coordinates indicating the direction in which the duodenum extends), which is information capable of determining the intestinal tract direction CD, is associated with the intestinal wall image 41.

[0114] The learned model 84B is obtained by optimizing a neural network through machine learning using training data on the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image equivalent to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be the subject of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, it is an annotation that can determine the intestinal tract direction CD.

[0115] Here, as an example of the annotation in the correct answer data, an annotation of the intestinal tract direction CD based on the fold shape of the intestinal tract shown in the intestinal wall image 41 can be cited (e.g., a line segment connecting the centers of the arcs of the fold shape is used as the annotation of the intestinal tract direction CD). And, as another annotation in the correct answer data, when the intestinal wall image 41 is a depth image, an annotation based on depth information can be cited (e.g., an annotation that sets the direction in which the depth in the depth direction represented by the depth information increases as the intestinal tract direction CD).

[0116] In addition, here, an example of the method in which only one learned model 84B is used by the image recognition unit 82B is given, but this is just one example. For example, it can also be set that the learned model 84B selected from a plurality of learned models 84B is used by the image recognition unit 82B. At this time, each learned model 84B is created by performing specific machine learning according to the surgical procedure of the ERCP examination (e.g., the position of the duodenoscope 12 relative to the papilla N, etc.), and it is only necessary to select the learned model 84B corresponding to the current surgical procedure of the ERCP examination and use it by the image recognition unit 82B.

[0117] The image recognition unit 82B inputs the intestinal wall image 41 obtained from the image acquisition unit 82A into the learned model 84B. Thereby, the learned model 84B outputs intestinal tract direction information 90 corresponding to the input intestinal wall image 41. The image recognition unit 82B acquires the intestinal tract direction information 90 output from the learned model 84B.

[0118] The derivation unit 82C derives the offset amount of the endoscope viewer 18 with respect to the intestinal tract direction CD (hereinafter, simply referred to as "offset amount"). Here, the offset amount refers to the degree of deviation between the posture of the endoscope viewer 18 and the intestinal tract direction CD. Specifically, the offset amount refers to the offset amount between the direction along the imaging surface of the imaging element of the camera 48 provided on the endoscope viewer 18 (e.g., the up and down directions in the viewing angle) and the intestinal tract direction CD. And, since the camera 48 is provided at the front end portion 46, the offset amount can also be said to be the angle between the longitudinal direction SD of the front end portion 46 (e.g., the central axis direction when the front end portion 46 is cylindrical) and the intestinal tract direction CD.

[0119] The derivation unit 82C acquires the intestinal tract direction information 90 from the image recognition unit 82B. Further, the derivation unit 82C acquires the posture information 91 from the optical fiber sensor 18A provided in the endoscope viewer 18. The posture information 91 is information indicating the posture of the endoscope viewer 18. The optical fiber sensor 18A is a sensor arranged along the longitudinal direction inside the endoscope viewer 18 (for example, the insertion section 44 and the distal end section 46). By using the optical fiber sensor 18A, the posture of the endoscope viewer 18 (for example, the inclination of the distal end section 46 with respect to a reference position (for example, the straight state of the endoscope viewer 18)) can be detected. At this time, for example, a known posture detection technique of an endoscope such as Japanese Patent Publication No. 6797834 can be appropriately used.

[0120] Further, here, a posture detection technique using the optical fiber sensor 18A is cited, but this is merely an example. For example, a so-called electromagnetic navigation method may be used to detect the inclination of the distal end section 46 of the endoscope viewer 18. At this time, for example, a known posture detection technique of an endoscope such as Japanese Patent Publication No. 6534193 can be appropriately used.

[0121] The derivation unit 82C uses the intestinal tract direction information 90 and the posture information 91 to derive information indicating an offset amount, that is, offset amount information 93. In Figure 6 the example shown, the angle A is shown as the offset amount information 93. The derivation unit 82C derives the offset amount using, for example, an offset calculation formula (not shown). The offset calculation formula is a calculation formula in which the position coordinates of the intestinal tract direction CD represented by the intestinal tract direction information 90 and the position coordinates of the longitudinal direction SD of the distal end section 46 represented by the posture information 91 are set as independent variables, and the angle formed by the intestinal tract direction CD and the longitudinal direction SD of the distal end section 46 is set as the dependent variable.

[0122] As an example, as Figure 7As shown, the display control unit 82D acquires the intestinal wall image 41 from the image acquisition unit 82A. Also, the display control unit 82D acquires the intestinal tract direction information 90 from the image recognition unit 82B. Further, the display control unit 82D acquires the offset information 93 from the derivation unit 82C. The display control unit 82D generates an operation instruction image 93A for aligning the longitudinal direction SD of the distal end portion 46 with the intestinal tract direction CD according to the offset represented by the offset information 93. The operation instruction image 93A is, for example, an arrow indicating the operation direction of the distal end portion 46 where the offset becomes smaller. The display control unit 82D generates a display image 94 including the intestinal wall image 41, the intestinal tract direction CD represented by the intestinal tract direction information 90, and the operation instruction image 93A, and outputs it to the display device 13. Specifically, the display control unit 82D controls the GUI (Graphical User Interface) for displaying the display image 94, causing the display device 13 to display the screen 36. The screen 36 is an example of the "first screen" related to the technology of the present invention. The operation instruction image 93A is an example of the "posture adjustment support information" related to the technology of the present invention.

[0123] In addition, here, an example of a method of enabling the user to grasp the offset by displaying the operation instruction image 93A on the screen 36 has been described, but the technology of the present invention is not limited thereto. For example, a message (not shown) indicating the operation content for reducing the offset can be displayed on the screen 36. As an example of the message, "Please tilt the distal end portion of the duodenoscope 10 degrees toward the back side" etc. can be cited. It can also be notified to the user through a sound output device such as a speaker.

[0124] The user can grasp the intestinal tract direction CD by visually recognizing the screen 36 of the display device 13. And by visually recognizing the operation instruction image 93A displayed on the screen 36, the user can grasp the operation for reducing the offset between the distal end portion 46 of the endoscope viewer 18 and the intestinal tract direction CD.

[0125] Next, with reference to Figure 8 , the operation related to the part of the duodenoscope system 10 related to the technology of the present invention will be described.

[0126] In Figure 8 , an example of the process of the medical support process performed by the processor 82 is shown.

[0127] In Figure 8In the medical support process shown, first, in step ST10, the image acquisition unit 82A determines whether one frame of shooting has been performed by the camera 48 provided in the endoscope viewer 18. In step ST10, if one frame of shooting has not been performed by the camera 48, it is determined as negative, and the determination in step ST10 is performed again. In step ST10, if one frame of shooting has been performed by the camera 48, it is determined as positive, and the medical support process proceeds to step ST12.

[0128] In step ST12, the image acquisition unit 82A acquires one frame of intestinal wall image 41 from the camera 48 provided in the endoscope viewer 18. After performing the process of step ST12, the medical support process proceeds to step ST14.

[0129] In step ST14, the image recognition unit 82B detects the intestinal tract direction CD by performing AI-based image recognition processing (i.e., image recognition processing using the learned model 84B) on the intestinal wall image 41 acquired in step ST12. After performing the process of step ST14, the medical support process proceeds to step ST16.

[0130] In step ST16, the derivation unit 82C acquires the posture information 91 from the optical fiber sensor 18A of the endoscope viewer 18. After performing the process of step ST16, the medical support process proceeds to step ST18.

[0131] In step ST18, the derivation unit 82C derives an offset based on the intestinal tract direction CD obtained by the image recognition unit 82B in step ST14 and the posture information 91 acquired in step ST16. Specifically, the derivation unit 82C derives the angle between the intestinal tract direction CD and the longitudinal direction SD of the distal end portion 46 represented by the posture information 91. After performing the process of step ST18, the medical support process proceeds to step ST20.

[0132] In step ST20, the display control unit 82D generates a display image 94 in which the intestinal wall image 41 is superimposed with the intestinal tract direction CD and the operation instruction image 93A corresponding to the offset derived in step ST18. After performing the process of step ST20, the medical support process proceeds to step ST22.

[0133] In step ST22, the display control unit 82D outputs the display image 94 generated in step ST20 to the display device 13. After performing the process of step ST22, the medical support process proceeds to step ST24.

[0134] In step ST24, the display control unit 82D determines whether the conditions for ending the medical support process are satisfied. As an example of the conditions for ending the medical support process, there may be mentioned the condition of giving an instruction to end the medical support process to the duodenoscope system 10 (for example, the condition that the instruction to end the medical support process is received by the receiving device 62).

[0135] In step ST24, when the conditions for ending the medical support process are not satisfied, the determination is negative, and the medical support process proceeds to step ST10. In step ST24, when the conditions for ending the medical support process are satisfied, the determination is positive, and the medical support process ends.

[0136] As described above, in the duodenoscope system 10 according to the first embodiment, in the image recognition unit 82B of the processor 82, an image recognition process is performed on the intestinal wall image 41. As a result of the image recognition process, the intestinal tract direction CD in the intestinal wall image 41 is detected. And the intestinal tract direction information 90 indicating the intestinal tract direction CD is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes the intestinal tract direction CD superimposed on the intestinal wall image 41. Thus, the user can recognize the intestinal tract direction CD, and according to this configuration, it is possible to make it easy for the user to grasp how much the posture of the endoscope viewer 18 is deviated from the intestinal tract direction CD.

[0137] And, in the duodenoscope system 10 according to the first embodiment, the deviation amount information 93 is derived in the derivation unit 82C. The deviation amount information 93 indicates the deviation amount between the posture of the endoscope viewer 18 and the intestinal tract direction CD. The deviation amount information 93 is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes a display based on the deviation amount information 93. Thus, the user can recognize the deviation amount between the posture of the endoscope viewer 18 and the intestinal tract direction CD, and according to this configuration, it is possible to make it easy for the user to grasp how much the posture of the endoscope viewer 18 is deviated from the intestinal tract direction CD.

[0138] And, in the duodenoscope system 10 according to the first embodiment, in the image recognition unit 82B, by performing an image recognition process on the intestinal wall image 41, the intestinal tract direction information 90 indicating the intestinal tract direction CD can be obtained. Thus, compared with the case where the user designates the intestinal tract direction CD for the intestinal wall image 41 by visual observation, the intestinal tract direction information 90 with higher accuracy can be obtained.

[0139] Further, in the duodenoscope system 10 according to the first embodiment, the intestinal tract direction information 90 is output to the display device 13 by the display control unit 82D. In the display device 13, the intestinal tract direction CD is displayed on the screen 36. Thereby, it is possible for the user to easily visually grasp how much the posture of the endoscope viewer 18 is deviated from the intestinal tract direction CD.

[0140] Further, in the duodenoscope system 10 according to the first embodiment, the posture information 91 that can determine the posture of the endoscope viewer 18 is obtained from the optical fiber sensor 18A by the derivation unit 82C. In the derivation unit 82C, the offset amount information 93 is generated based on the posture information 91 and the intestinal tract direction information 90. Further, in the display control unit 82D, an operation instruction image 93A indicating the operation direction for reducing the offset amount is generated based on the offset amount information 93. The display control unit 82D outputs the operation instruction image 93A to the display device 13. In the display device 13, the operation instruction image 93A is superimposed and displayed on the intestinal wall image 41. Thereby, in a state where the endoscope viewer 18 is inserted into the duodenum, it is easy to set the posture of the endoscope viewer 18 relative to the intestinal tract direction CD to the posture desired by the user. For example, the user can bring the intestinal tract direction CD closer to the posture of the endoscope viewer 18 by performing an operation of changing the posture of the endoscope viewer 18 in the direction shown in the operation instruction image 93A.

[0141] In addition, in the above first embodiment, a method example of detecting the intestinal tract direction CD by image recognition processing based on the AI method is given, but the technology of the present invention is not limited thereto. For example, the intestinal tract direction CD can be detected by image recognition processing based on the pattern matching method. At this time, for example, it can be the following method: detecting a region representing the folds of the intestinal tract (i.e., the fold region) included in the intestinal wall image 41, and inferring the intestinal tract direction based on the arc shape of the fold region (for example, inferring the line connecting the centers of the arcs as the intestinal tract direction).

[0142] (First Modified Example)

[0143] In the above first embodiment, a method example of detecting the intestinal tract direction CD using the intestinal wall image 41 that does not include depth information has been described, but the technology of the present invention is not limited thereto. In this first modified example, in the image recognition unit 82B, the derivation of the intestinal tract direction using the intestinal wall image 41 as a depth image is performed. As an example, as Figure 9As shown, the intestinal wall image 41 is a depth image having depth information 41A as pixel values. The depth information 41A is information representing the depth (i.e., the distance to the intestinal wall) of the duodenum as the subject. Regarding the depth of the duodenum, for example, it is obtained by distance measurement in a so-called TOF method using a distance measurement sensor mounted on the distal end portion 46. The image recognition unit 82B acquires the intestinal wall image 41 from the image acquisition unit 82A.

[0144] The image recognition unit 82B derives intestinal tract direction information 90 based on the depth information 41A represented by the intestinal wall image 41. The image recognition unit 82B, for example, uses an intestinal tract direction calculation formula 82B1 to derive the intestinal tract direction information 90. The intestinal tract direction calculation formula 82B1 is, for example, a formula that takes the depth of the depth represented by the depth information 41A as an independent variable and the position coordinate group representing the axis of the intestinal tract direction CD as a dependent variable. Thus, the intestinal tract direction information 90 is obtained based on the depth information 41A of the intestinal wall image 41.

[0145] As described above, in the duodenoscope system 10 according to the first modification example, the intestinal wall image 41 has depth information 41A representing the depth of the duodenum, and the intestinal tract direction information 90 is obtained based on the depth information 41A. The intestinal tract direction CD is the direction along the depth direction in the lumen of the duodenum. And, the depth information 41A reflects the depth of the lumen of the duodenum. Therefore, the intestinal tract direction CD is derived based on the depth information 41A, and thus, compared with the case where the depth information 41A is not considered, the intestinal tract direction information 90 representing the intestinal tract direction CD with higher representation accuracy can be obtained.

[0146] (Second Modification Example)

[0147] In the above first embodiment, an example of a method of obtaining the intestinal tract direction CD by image recognition processing of the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In this second modification example, a direction (hereinafter, also simply referred to as "predetermined direction") that intersects the intestinal tract direction CD at a predetermined angle can be obtained.

[0148] As an example, as Figure 10 shown, every time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, it updates the time-series image group 89 in a FIFO manner.

[0149] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 into the learned model 84C. As a result, the learned model 84C outputs the vertical direction information 97 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the vertical direction information 97 output from the learned model 84C. Here, the vertical direction information 97 is information (for example, a set of position coordinates indicating the axis orthogonal to the intestinal tract direction CD) capable of determining the direction VD (hereinafter, also simply referred to as "vertical direction VD") orthogonal to the intestinal tract direction CD.

[0150] In the image recognition process using the learned model 84C, the reliability of a specific result is calculated based on the result of determining the direction orthogonal to the intestinal tract direction CD. Here, the reliability is a statistical measure indicating the reliability of the specific result. The reliability is, for example, the score of the activation function (for example, softmax function, etc.) input to the output layer of the learned model 84C. The vertical direction information 97 output from the learned model 84C has a score equal to or higher than a threshold value (for example, 0.9 or higher).

[0151] In addition, in the present embodiment, "vertical" not only means completely vertical, but also means vertical in the sense including the errors generally allowed in the technical field to which the technology of the present invention belongs and not violating the gist of the technology of the present invention. And here, as the specified angle with respect to the intestinal tract direction CD, the vertical direction with respect to the intestinal tract direction CD is cited, but the technology of the present invention is not limited thereto. For example, the specified angle can be 45 degrees, 60 degrees, or 80 degrees.

[0152] The learned model 84C is obtained by optimizing a neural network through machine learning using training data. The training data is a plurality of data (that is, multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (for example, an image corresponding to the intestinal wall image 41) obtained by photographing a part (for example, the inner wall of the duodenum) that may be an object of ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, it is an annotation capable of determining the vertical direction VD.

[0153] The derivation unit 82C derives the degree of coincidence between the specified direction and the direction of the optical axis of the camera 48. The coincidence between the specified direction and the direction of the optical axis means that the direction in which the camera 48 is oriented is the same as the direction preset by the user. That is, it means a state in which the front end portion 46 provided with the camera 48 is not in a direction not desired by the user (for example, a direction inclined with respect to the intestinal tract direction CD).

[0154] Therefore, the derivation unit 82C acquires the vertical direction information 97. Further, the derivation unit 82C acquires the optical axis information 48A from the camera 48 of the endoscope viewer 18. The optical axis information 48A is information capable of determining the optical axis of the optical system of the camera 48. Further, the derivation unit 82C generates the consistency information 99 by comparing the direction indicated by the vertical direction information 97 with the direction of the optical axis indicated by the optical axis information 48A. The consistency information 99 is information indicating the degree of coincidence between the direction of the optical axis and a specified direction (for example, the angle formed by the direction of the optical axis and the specified direction). In addition, in the present embodiment, "coincidence" means not only complete coincidence but also coincidence in the sense including an error that is generally allowed in the technical field to which the technology of the present invention belongs and does not violate the gist of the technology of the present invention.

[0155] Moreover, the derivation unit 82C determines whether the direction of the optical axis coincides with the specified direction. When the direction of the optical axis coincides with the specified direction, the derivation unit 82C generates the notification information 100. The notification information 100 is information for notifying the user of the content that the direction of the optical axis coincides with the specified direction (for example, text indicating that the direction of the optical axis coincides with the specified direction).

[0156] As an example, as Figure 11 shown, the display control unit 82D acquires the vertical direction information 97 from the image recognition unit 82B. Further, the display control unit 82D acquires the consistency information 99 from the derivation unit 82C. The display control unit 82D generates an operation instruction image 93B (for example, an arrow indicating an operation direction) for making the direction of the optical axis coincide with the specified direction according to the degree of coincidence between the direction of the optical axis indicated by the consistency information 99 and the specified direction. Further, the display control unit 82D generates a display image 94 including the vertical direction VD indicated by the vertical direction information 97, the operation instruction image 93B, and the intestinal wall image 41, and outputs it to the display device 13. In Figure 11 the example shown, in the display device 13, an intestinal wall image 41 on which the vertical direction VD and the operation instruction image 93B are superimposed and displayed is shown on the screen 36.

[0157] Moreover, when the direction of the optical axis coincides with the specified direction, the derivation unit 82C outputs the notification information 100 to the display control unit 82D instead of the consistency information 99. At this time, the display control unit 82D generates a display image 94 including the content for notifying the user that the direction of the optical axis indicated by the notification information 100 coincides with the specified direction instead of the operation instruction image 93B. In Figure 11 the example shown, in the display device 13, an example in which a message "The optical axis coincides with the vertical direction" is displayed on the screen 37 is shown.

[0158] In addition, here, an example of a method of displaying a message based on the notification information 100 in the display device 13 has been described, but this is merely an example. For example, a mark such as a circular mark based on the notification information 100 may be displayed. Also, instead of or together with the display device 13, a method of outputting the notification information 100 to a sound output device such as a speaker may be used.

[0159] As described above, in the duodenoscope system 10 according to the second modification example, vertical direction information 97, which is information capable of determining a direction orthogonal to the intestinal tract direction CD, is derived in the derivation unit 82C. The vertical direction information 97 is output to the display control unit 82D, and a display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes a vertical direction VD represented by the vertical direction information 97. Thus, the user can recognize a direction that intersects the intestinal tract direction CD at a specified angle.

[0160] Also, in the duodenoscope system 10 according to the second modification example, in the image recognition unit 82B, by performing an image recognition process on the intestinal wall image 41, vertical direction information 97 representing the vertical direction VD can be obtained. Thus, compared with the case where the user visually observes the intestinal wall image 41 and designates the vertical direction VD, higher-precision vertical direction information 97 can be obtained.

[0161] Also, in the duodenoscope system 10 according to the second modification example, in the image recognition process using the learned model 84C in the image recognition unit 82B, the vertical direction information 97 is obtained with a reliability equal to or higher than a threshold value. Thus, in the image recognition process using the learned model 84C in the image recognition unit 82B, compared with the case where no threshold value is set for the reliability, higher-precision vertical direction information 97 can be obtained.

[0162] Also, in the duodenoscope system 10 according to the second modification example, the optical axis information 48A is acquired from the camera 48 by the derivation unit 82C. And in the derivation unit 82C, consistency information 99 is generated based on the optical axis information 48A and the vertical direction information 97. Further, in the display control unit 82D, a display image 94 is generated based on the consistency information 99 and output to the display device 13. The display image 94 includes a display related to the degree to which the direction of the optical axis represented by the consistency information 99 coincides with a specified direction. Thus, the user can grasp the degree to which the optical axis of the camera 48 is offset from the vertical direction VD. For example, when the optical axis coincides with the vertical direction VD, there is a high possibility that the camera 48 is facing the intestinal wall of the duodenum. By maintaining the posture of the endoscope viewer 18 in this state, it is easy to detect the papilla N existing on the intestinal wall of the duodenum, and further, it is also easy to make the camera 48 face the papilla N.

[0163] Moreover, in the duodenoscope system 10 according to the second modification example, in the display control unit 82D, an operation instruction image 93B for aligning the direction of the optical axis with a specified direction is generated based on the degree of coincidence information 99. The display control unit 82D outputs the operation instruction image 93B to the display device 13, and in the display device 13, the operation instruction image 93B is superimposed on the intestinal wall image 41. Thereby, the user can grasp the operation required to align the optical axis direction of the camera 48 with the vertical direction VD.

[0164] Moreover, in the duodenoscope system 10 according to the second modification example, in the derivation unit 82C, it is determined whether the direction of the optical axis coincides with the specified direction. When the direction of the optical axis coincides with the specified direction, the derivation unit 82C generates a notification message 100. In the display control unit 82D, a display image 94 is generated based on the notification message 100 and output to the display device 13. The display image 94 includes a display of the content indicating that the direction of the optical axis coincides with the specified direction represented by the notification message 100. Thereby, the user can perceive that the direction of the optical axis coincides with the specified direction.

[0165] (Third modification example)

[0166] In the above first embodiment, an example of obtaining the intestinal tract direction CD through image recognition processing of the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In this third modification example, the traveling direction TD of the bile duct is obtained based on the intestinal tract direction CD.

[0167] As an example, as Figure 12 shown, every time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0168] The image recognition unit 82B performs nipple detection processing on the time-series image group 89 using the learned model 84D. The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 into the learned model 84D. Thereby, the learned model 84D outputs nipple region information 95 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the nipple region information 95 output from the learned model 84D. Here, the nipple region information 95 includes information (for example, coordinates and range within the image) that can determine the nipple region N1 in the intestinal wall image 41 in which the nipple N is reflected.

[0169] The learned model 84D is obtained by optimizing a neural network through machine learning using training data. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The training data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be the subject of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the papilla region N1 can be cited.

[0170] The derivation unit 82C derives the traveling direction information 96 as information representing the traveling direction TD of the bile duct. The traveling direction information 96 includes information capable of determining the direction in which the bile duct extends (e.g., position coordinates indicating the direction in which the bile duct extends). The derivation unit 82C acquires the papilla region information 95 from the image recognition unit 82B. And, the derivation unit 82C acquires the intestinal tract direction information 90 obtained through image recognition processing using the learned model 84B (refer to Figure 6 ). And, the derivation unit 82C derives the traveling direction information 96 based on the intestinal tract direction information 90 and the papilla region information 95. The derivation unit 82C derives the traveling direction TD, for example, based on a prescribed azimuth relationship between the intestinal tract direction CD and the traveling direction TD. Specifically, when the intestinal tract direction CD is set to the 6 o'clock direction, the derivation unit 82C derives the traveling direction TD as the 11 o'clock to 12 o'clock direction. Moreover, the derivation unit 82C uses the papilla region N1 represented by the papilla region information 95 as the starting point of the traveling direction TD.

[0171] As an example, as Figure 13 shown, the display control unit 82D acquires the traveling direction information 96 from the derivation unit 82C. And, the display control unit 82D acquires the papilla region information 95 from the image recognition unit 82B. The display control unit 82D generates a display image 94 in which the traveling direction TD represented by the traveling direction information 96 and the papilla region N1 represented by the papilla region information 95 are superimposed and displayed on the intestinal wall image 41 acquired from the image acquisition unit 82A (refer to Figure 6 ), and outputs it to the display device 13. In the display device 13, the intestinal wall image 41 with the traveling direction TD superimposed and displayed is displayed on the screen 36.

[0172] As described above, in the duodenoscope system 10 according to the third modification example, in the image recognition unit 82B, the nipple detection process is performed using the learned model 84D. The nipple region information 95 is obtained through the nipple detection process. Also, in the image recognition unit 82B, the intestinal tract direction information 90 can be obtained by performing the image recognition process using the learned model 84A. The derivation unit 82C derives the traveling direction information 96 based on the intestinal tract direction information 90 and the nipple region information 95. Further, the display control unit 82D outputs the display image 94 to the display device 13. The display image 94 includes the nipple region N1 represented by the nipple region information 95 and the traveling direction TD of the bile duct represented by the traveling direction information 96. In the display device 13, the nipple region N1 and the traveling direction TD of the bile duct are displayed on the screen 36. Thereby, it is possible to enable a user who observes the nipple N through the screen 36 to easily visually recognize the traveling direction TD of the bile duct.

[0173] For example, in an ERCP examination, the camera 48 is sometimes directed at the nipple N. At this time, by utilizing the traveling direction of the bile duct or pancreatic duct, it is easy to grasp the posture of the endoscope viewer 18. Also, when inserting a treatment instrument into the nipple N, by grasping the traveling direction of the bile duct or pancreatic duct, it is easy to perform the intubation operation on the bile duct or pancreatic duct within the nipple N.

[0174] (Fourth Modification Example)

[0175] In the above-described first embodiment, an example of obtaining the intestinal tract direction CD through the image recognition process of the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In this fourth modification example, the orientation of the nipple bulge NA in the nipple N (hereinafter, also simply referred to as "nipple orientation ND") is obtained based on the intestinal tract direction CD.

[0176] In the image recognition unit 82B, the intestinal tract direction information 90 and the nipple region information 95 can be obtained by performing the image recognition process on the intestinal wall image 41 (see Figure 12 ). As an example, as Figure 14 shown, the derivation unit 82C generates the nipple orientation information 102 based on the intestinal tract direction information 90 and the nipple region information 95. The nipple orientation information 102 is information that can determine the nipple orientation ND (for example, the orientation in which the nipple bulge NA faces the treatment instrument). The nipple orientation ND is obtained, for example, as a tangent line at the nipple bulge NA in the traveling direction TD of the bile duct. Therefore, the derivation unit 82C derives the traveling direction TD of the bile duct based on the intestinal tract direction CD represented by the intestinal tract direction information 90, and further derives the direction of the tangent line at the nipple bulge NA as the nipple orientation ND based on the traveling direction TD.

[0177] The display control unit 82D acquires the nipple orientation information 102 from the derivation unit 82C. The display control unit 82D generates a display image 94 in which the nipple orientation ND represented by the nipple orientation information 102 and the nipple region N1 represented by the nipple region information 95 are superimposed and displayed on the intestinal wall image 41 acquired from the image acquisition unit 82A (refer to Figure 6 ), and outputs it to the display device 13. In the display device 13, the intestinal wall image 41 with the nipple orientation ND superimposed and displayed is displayed on the screen 36.

[0178] In addition, here, an example in which the nipple orientation ND is displayed as an arrow is given, but this is merely an example. The nipple orientation ND may be a method of representing the direction by text.

[0179] As described above, in the duodenoscope system 10 according to the fourth modification, in the image recognition unit 82B, a nipple detection process (refer to Figure 12 ) is performed to obtain the nipple region information 95. And, in the image recognition unit 82B, by using the learned model 84B (refer to Figure 6 ) for image recognition processing, the intestinal tract direction information 90 can be obtained. The derivation unit 82C derives the nipple orientation information 102 based on the intestinal tract direction information 90. And, the display image 94 is output to the display device 13 by the display control unit 82D. The display image 94 includes the nipple region N1 represented by the nipple region information 95 and the nipple orientation ND represented by the nipple orientation information 102. In the display device 13, the nipple region N1 and the nipple orientation ND are displayed on the screen 36. Thereby, it is possible to make it easy for the user observing the nipple N through the screen 36 to visually grasp the nipple orientation ND.

[0180] For example, in an ERCP examination, the camera 48 is sometimes directed at the nipple N. At this time, by using the nipple orientation ND, it is easy to grasp the posture of the endoscope viewer 18. And, when inserting a treatment instrument into the nipple N, by grasping the nipple orientation ND, the treatment instrument can be directed at the nipple N, and it is easy to insert the treatment instrument into the nipple N.

[0181] <Second Embodiment>

[0182] In the above-described first embodiment, an example of obtaining the intestinal tract direction CD by image recognition processing of the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In the second embodiment, the intestinal wall image 41 is an image obtained by photographing the intestinal wall including the nipple N, and the protruding direction RD of the nipple N is obtained by image recognition processing of the intestinal wall image 41.

[0183] For example, during ERCP examination, sometimes the camera 48 is directed exactly at the rising direction RD of the papilla N. Thereby, it is easy to infer the traveling directions of the bile duct T and the pancreatic duct S extending from the papilla N, or it is easy to insert a treatment instrument (for example, a cannula) into the papilla N. Therefore, in the second embodiment, the rising direction RD of the papilla N is obtained through image recognition processing of the intestinal wall image 41. The rising direction RD is an example of the "rising direction" and the "first direction" involved in the technology of the present invention.

[0184] As an example, as Figure 15 shown, every time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, it updates the time-series image group 89 in a FIFO manner.

[0185] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 into the learned model 84E. Thereby, the learned model 84E outputs the rising direction information 104 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the rising direction information 104 output from the learned model 84E. Here, the rising direction information 104 is information capable of determining the direction in which the papilla N rises (for example, a set of position coordinates of the axis representing the rising direction RD). The rising direction information 104 is an example of the "papilla orientation-related information" and the "rising direction information" involved in the technology of the present invention.

[0186] The learned model 84E is obtained by optimizing a neural network through machine learning of the neural network using training data. The training data is a plurality of data (that is, multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The training data is, for example, an image (for example, an image equivalent to the intestinal wall image 41) obtained by photographing a part (for example, the inner wall of the duodenum) that may be the subject of ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the rising direction RD of the papilla N can be cited.

[0187] Here, the rising direction RD of the papilla N is determined, for example, as the direction from the top of the papilla elevation NA of the papilla N towards the top of the encircling fold H1. This is because, according to medical diagnostic opinions, the rising direction RD of the papilla N generally coincides with the direction from the top of the papilla elevation NA towards the top of the encircling fold H1. Here, in the papilla N, there are a plurality of folds (for example, folds H1 to H3) around the elevated part. The encircling fold H1 is the fold closest to the papilla elevation NA. Therefore, as an example of the annotation in the correct answer data, an annotation setting the direction passing through the top of the encircling fold H1 as the rising direction RD can be cited.

[0188] The derivation unit 82C derives the degree of coincidence between the protruding direction RD and the direction of the optical axis of the camera 48. The coincidence between the protruding direction RD and the direction of the optical axis means that the direction toward which the camera 48 faces is directly opposite to the nipple N. That is, it means a state in which the front end portion 46 provided with the camera 48 is not in a direction that the user does not expect (for example, a direction in which the nipple N is inclined with respect to the protruding direction RD).

[0189] Therefore, the derivation unit 82C obtains the protruding direction information 104 from the image recognition unit 82B. And the derivation unit 82C obtains the optical axis information 48A from the camera 48 of the endoscope viewer 18. And the derivation unit 82C generates the coincidence information 103 by comparing the direction represented by the vertical direction information 97 with the direction of the optical axis represented by the optical axis information 48A. The coincidence information 103 is information capable of determining the degree of coincidence between the direction of the optical axis and the protruding direction RD (for example, the angle formed by the direction of the optical axis and the protruding direction RD). The coincidence information 103 is an example of the "coincidence information" related to the technology of the present invention.

[0190] As an example, as Figure 16 shown, the display control unit 82D obtains the protruding direction information 104 from the image recognition unit 82B. And the display control unit 82D obtains the coincidence information 103 from the derivation unit 82C. The display control unit 82D generates an operation instruction image 93C (for example, an arrow indicating the operation direction) for making the direction of the optical axis coincide with the protruding direction RD according to the degree of coincidence between the direction of the optical axis represented by the coincidence information 103 and the protruding direction RD. And the display control unit 82D generates a display image 94 including the protruding direction RD represented by the protruding direction information 104, the operation instruction image 93C, and the intestinal wall image 41, and outputs it to the display device 13. In Figure 16 the example shown, on the display device 13, an intestinal wall image 41 on which the protruding direction RD and the operation instruction image 93C are superimposed and displayed is shown on the screen 36.

[0191] As an example, as Figure 17 shown, the doctor 14 operates the endoscope viewer 18 to make the optical axis of the camera 48 approach the protruding direction RD. Thereby, an intestinal wall image 41 when the nipple N faces the camera 48 can be obtained, so it is easy to infer the traveling directions of the bile duct T and the pancreatic duct S extending from the nipple N, or it is easy to insert a treatment instrument (for example, a cannula) into the nipple N.

[0192] As described above, in the duodenoscope system 10 according to the second embodiment, in the image recognition unit 82B of the processor 82, image recognition processing is performed on the intestinal wall image 41. As a result of the image recognition processing, the rising direction RD of the papilla N in the intestinal wall image 41 is detected. And the rising direction information 104 indicating the rising direction RD is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes the rising direction RD superimposed on the intestinal wall image 41. In this way, the rising direction RD is displayed on the screen 36 in the display device 13. Thereby, the user observing the intestinal wall image 41 can visually grasp the rising direction RD of the papilla N.

[0193] And, in the duodenoscope system 10 according to the second embodiment, in the image recognition unit 82B, the rising direction information 104 is obtained based on the intestinal wall image 41. The rising direction information 104 is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes a display based on the rising direction information 104. Thereby, the user observing the intestinal wall image 41 can visually grasp the rising direction RD of the papilla N.

[0194] And, in the duodenoscope system 10 according to the second embodiment, in the display control unit 82D, the display image 94 is generated. The display image 94 includes an image of an arrow indicating the rising direction RD. Thereby, the rising direction RD of the papilla N can be visualized and the user observing the intestinal wall image 41 can visually grasp it.

[0195] And, in the duodenoscope system 10 according to the second embodiment, in the derivation unit 82C, the optical axis information 48A is obtained from the camera 48. And, in the derivation unit 82C, the consistency information 103 is generated based on the optical axis information 48A and the rising direction information 104. In the display control unit 82D, the display image 94 is generated based on the consistency information 103 and output to the display device 13. The display image 94 includes a display related to the degree of coincidence between the direction of the optical axis represented by the consistency information 103 and the rising direction RD. Thereby, the user observing the intestinal wall image 41 can visually grasp the degree of coincidence between the rising direction RD of the papilla N and the optical axis direction. For example, when the optical axis coincides with the rising direction RD, it is highly likely that the camera 48 is facing the papilla N directly. By maintaining the posture of the endoscope viewer 18 in this state, it is easy to observe the papilla N, and further, it is easy to insert the treatment instrument into the papilla N.

[0196] Further, in the duodenoscope system 10 according to the second embodiment, in the image recognition process in the image recognition unit 82B, the elevation direction RD is determined as the direction from the top of the nipple elevation NA of the nipple N toward the circumferential fold H1. And the display image 94 generated in the display control unit 82D is output to the display device 13. The elevation direction RD is included in the display image 94. Thereby, the user observing the intestinal wall image 41 can visually grasp the direction from the opening of the nipple elevation NA toward the top of the circumferential fold H1. As a result, the traveling direction TD of the bile duct leading to the opening of the nipple N can be easily determined.

[0197] Further, in the duodenoscope system 10 according to the second embodiment, in the image recognition process in the image recognition unit 82B, the elevation direction RD is determined as the direction from the top of the nipple elevation NA of the nipple N toward the circumferential fold H1. And the display image 94 generated in the display control unit 82D is output to the display device 13. An image of an arrow indicating the elevation direction RD is included in the display image 94. Thereby, the user observing the intestinal wall image 41 can visually grasp the direction from the opening of the nipple elevation NA toward the top of the circumferential fold H1. As a result, the traveling direction TD of the bile duct leading to the opening of the nipple N can be easily determined.

[0198] Further, in the duodenoscope system 10 according to the second embodiment, in the image recognition unit 82B, by performing an image recognition process on the intestinal wall image 41, elevation direction information 104 indicating the elevation direction RD can be obtained. Thus, compared with the case where the user designates the elevation direction RD for the intestinal wall image 41 by visual observation, higher-precision elevation direction information 104 can be obtained.

[0199] (Fifth Modification Example)

[0200] In the above-described second embodiment, an example in which the elevation direction RD is determined as the direction from the top of the nipple elevation NA toward the top of the circumferential fold H1 has been described, but the technology of the present invention is not limited thereto. The elevation direction RD is determined according to the patterns of the plurality of folds H1 to H3.

[0201] As an example, as Figure 18 shown, the image recognition unit 82B acquires the time-sequence image group 89 from the image acquisition unit 82A, and inputs the acquired time-sequence image group 89 to the learned model 84E. Thereby, the learned model 84E outputs the elevation direction information 104 corresponding to the input time-sequence image group 89. The image recognition unit 82B acquires the elevation direction information 104 output from the learned model 84E.

[0202] Here, the raising direction RD of the nipple N is determined, for example, as the direction passing around the top of the fold H1. According to medical diagnostic opinions, the raising direction RD of the nipple N sometimes coincides with the direction passing around the top of the fold H1. Therefore, as an example of the annotation in the correct solution data, an annotation that sets the direction passing around the top of the fold H1 as the raising direction RD can be cited.

[0203] In addition, here, a method of determining the raising direction RD as the direction passing around the top of the fold H1 has been described, but this is merely an example. The raising direction RD can be determined as the direction passing through at least one of the tops of the multiple folds H1 to H3.

[0204] As described above, in the duodenoscope system 10 according to the fifth modification example, in the image recognition process in the image recognition unit 82B, the raising direction RD is determined according to the forms of the multiple folds H1 to H3. And, the display image 94 generated in the display control unit 82D is output to the display device 13. The raising direction RD is included in the display image 94. Thereby, the user observing the intestinal wall image 41 can visually recognize the direction passing around the top of the fold H1 of the nipple bulge NA as the raising direction RD.

[0205] (Sixth Modification Example)

[0206] In the above-described second embodiment, an example of a method in which the raising direction RD is determined as the direction from the top of the nipple bulge NA toward the top of the fold H1 has been described, but the technology of the present invention is not limited thereto. In the sixth modification example of the present invention, the raising direction RD is determined according to the nipple bulge NA and the multiple folds H1 to H3.

[0207] As an example, as Figure 19 shown, the image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 to the learned model 84E. Thereby, the learned model 84E outputs the raising direction information 104 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the raising direction information 104 output from the learned model 84E.

[0208] Here, the raising direction RD of the nipple N is determined, for example, as the direction from the top of the nipple bulge NA passing through the tops of the fold H1, the fold H2, and the fold H3. According to medical diagnostic opinions, the raising direction RD of the nipple N sometimes coincides with the direction from the top of the nipple bulge NA passing through the tops of the fold H1, the fold H2, and the fold H3. Therefore, as an example of the annotation in the correct solution data, an annotation that sets the direction from the top of the nipple bulge NA passing through the tops of the fold H1, the fold H2, and the fold H3 as the raising direction RD can be cited.

[0209] As described above, in the duodenoscope system 10 according to the sixth modification, in the image recognition process in the image recognition unit 82B, the bulge direction RD is determined based on the papillary bulge NA and the plurality of folds H1 to H3. Further, the display image 94 generated in the display control unit 82D is output to the display device 13. The bulge direction RD is included in the display image 94. Thus, the user observing the intestinal wall image 41 can visually recognize the direction passing through the top of the papillary bulge NA and the tops of the plurality of folds H1 to H3 as the bulge direction RD.

[0210] (Seventh Modification)

[0211] In the above-described second embodiment, an example of obtaining the bulge direction RD by performing image recognition processing on the intestinal wall image 41 has been described, but the technique of the present invention is not limited thereto. In the seventh modification, the traveling direction TD of the bile duct is obtained based on the bulge direction RD.

[0212] In the image recognition unit 82B, by performing image recognition processing on the intestinal wall image 41, the bulge direction information 104 and the papilla region information 95 can be obtained (see Figure 12 and Figure 15 ). As an example, as Figure 20 shown, the derivation unit 82C derives the traveling direction information 96 based on the bulge direction information 104. The traveling direction TD of the bile duct has a predetermined azimuth relationship with the bulge direction RD of the papilla N. Specifically, when the bulge direction RD is set to the 12 o'clock direction, the derivation unit 82C derives the traveling direction TD as the 11 o'clock direction.

[0213] The display control unit 82D acquires the traveling direction information 96 from the derivation unit 82C. The display control unit 82D generates a display image 94 in which the traveling direction TD indicated by the traveling direction information 96 is superimposed on the intestinal wall image 41 acquired from the image acquisition unit 82A (see Figure 6 ), and outputs the display image 94 to the display device 13. In the display device 13, the intestinal wall image 41 with the traveling direction TD superimposed thereon is displayed on the screen 36.

[0214] As described above, in the duodenoscope system 10 according to the seventh modification, in the derivation unit 82C, the traveling direction information 96 is obtained based on the bulge direction information 104. Thus, since the traveling direction information 96 is obtained from the bulge direction information 104, it is easier to determine the traveling direction TD than in the case where the traveling direction information 96 is obtained by image recognition processing.

[0215] Moreover, in the duodenoscope system 10 according to the seventh modification, in the display control unit 82D, a display image 94 is generated. The display image 94 includes an image indicating the traveling direction TD. Thus, the user observing the intestinal wall image 41 can visually grasp the traveling direction TD of the bile duct.

[0216] (Eighth Modification)

[0217] In the above-described second embodiment, an example of obtaining the elevation direction RD of the papilla N by image recognition processing of the intestinal wall image 41 has been described. However, the technology of the present invention is not limited thereto. In the eighth modification, by image recognition processing of the intestinal wall image 41, the direction MD of the surface where the opening exists in the papilla N (hereinafter, also simply referred to as "surface direction MD") can be obtained.

[0218] As an example, as Figure 21 shown, every time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0219] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the learned model 84F. Thus, the learned model 84F outputs surface direction information 106 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the surface direction information 106 output from the learned model 84F. Here, the surface direction information 106 is information capable of determining the surface direction MD (for example, a set of position coordinates of an axis indicating the surface direction MD). The surface direction information 106 is an example of the "papilla orientation-related information" and "surface direction information" related to the technology of the present invention.

[0220] The learned model 84F is obtained by optimizing a neural network through machine learning using training data for the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a correspondence between example data and correct answer data. The example data is, for example, an image (for example, an image corresponding to the intestinal wall image 41) obtained by photographing a part (for example, the inner wall of the duodenum) that may be an object of ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the surface direction MD can be cited.

[0221] The lead-out unit 82C leads out the relative angle between the surface P having the opening K of the nipple N and the posture of the endoscope viewer 18. That the relative angle between the surface P having the opening K of the nipple N and the posture of the endoscope viewer 18 is close to 0 means that it is close to the state where the camera 48 faces the nipple N directly. Therefore, the lead-out unit 82C acquires the surface direction information 106 from the image recognition unit 82B. And the lead-out unit 82C acquires the posture information 91 from the optical fiber sensor 18A of the endoscope viewer 18. And the lead-out unit 82C generates the relative angle information 108 by comparing the surface orientation of the surface having the opening K represented by the surface direction information 106 with the posture of the endoscope viewer 18 represented by the posture information 91. The relative angle information 108 is information representing the angle A formed by the surface P and the posture of the endoscope viewer 18 (for example, the imaging surface of the camera 48). The relative angle information 108 is an example of the "angle-related information" related to the technology of the present invention.

[0222] As an example, as Figure 22 shown, the display control unit 82D acquires the surface direction information 106 from the image recognition unit 82B. And the display control unit 82D acquires the relative angle information 108 from the lead-out unit 82C. The display control unit 82D generates an operation instruction image 93D (for example, an arrow indicating the operation direction) for making the camera 48 face the nipple N directly according to the angle represented by the relative angle information 108. And the display control unit 82D generates a display image 94 including the surface direction MD represented by the surface direction information 106, the operation instruction image 93D, and the intestinal wall image 41, and outputs it to the display device 13. In Figure 22 the example shown, on the display device 13, the intestinal wall image 41 with the surface direction MD and the operation instruction image 93D superimposed and displayed is shown on the screen 36.

[0223] As described above, in the duodenoscope system 10 according to the eighth modification, in the image recognition unit 82B of the processor 82, an image recognition process is performed on the intestinal wall image 41, and as a result of the image recognition process, the surface direction MD of the nipple N in the intestinal wall image 41 is detected. And the surface direction information 106 representing the surface direction MD is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes the surface direction MD superimposed on the intestinal wall image 41. In this way, the surface direction MD is displayed on the screen 36 of the display device 13. Thereby, the user observing the intestinal wall image 41 can visually grasp the surface direction MD of the nipple N.

[0224] Moreover, in the duodenoscope system 10 according to this eighth modification example, the posture information 91 is obtained from the fiber optic sensor 18A by the derivation unit 82C, and the posture information 91 is information capable of determining the posture of the endoscope viewer 18. Further, in the derivation unit 82C, the relative angle information 108 is generated based on the posture information 91 and the surface direction information 106. Moreover, in the display control unit 82D, the operation instruction image 93D for causing the camera 48 to face the nipple N is generated based on the relative angle information 108. The display control unit 82D outputs the operation instruction image 93D to the display device 13, and in the display device 13, the operation instruction image 93D is superimposed and displayed on the intestinal wall image 41. Thus, in a state where the endoscope viewer 18 is inserted into the duodenum, it is easy to set the posture of the surface direction MD of the endoscope viewer 18 with respect to the nipple N to a posture desired by the user.

[0225] (Ninth Modification Example)

[0226] In the above-described second embodiment, an example of a method of displaying the raised direction RD obtained by the image recognition process of the intestinal wall image 41 has been described, but the technique of the present invention is not limited thereto. In this ninth modification example, the nipple surface image 93E is displayed.

[0227] As an example, as Figure 23 shown, the display control unit 82D acquires the raised direction information 104 from the image recognition unit 82B. The display control unit 82D generates the nipple surface image 93E based on the raised direction RD indicated by the raised direction information 104. The nipple surface image 93E is an image capable of determining a plane that intersects the raised direction RD at a specified angle (for example, 90 degrees). The nipple surface image 93E is an example of the "nipple orientation-related information" and the "plane image" related to the technique of the present invention. Moreover, the display control unit 82D adjusts the nipple surface image 93E to a size and shape corresponding to the nipple region N1 based on the nipple region information 95 obtained in the image recognition unit 82B. And the display control unit 82D generates the operation instruction image 93C.

[0228] Moreover, the display control unit 82D generates a display image 94 including the nipple surface image 93E, the operation instruction image 93C, and the intestinal wall image 41, and outputs it to the display device 13. In Figure 23 the example shown, in the display device 13, the intestinal wall image 41 on which the nipple surface image 93E and the operation instruction image 93C are superimposed and displayed is shown on the screen 36.

[0229] As described above, in the duodenoscope system 10 according to the ninth modified example, in the display control unit 82D, the papillary surface image 93E is generated based on the bulge direction information 104. The display control unit 82D outputs the papillary surface image 93E to the display device 13. In the display device 13, the papillary surface image 93E is superimposed and displayed on the intestinal wall image 41. Thus, it is easy for a user observing the intestinal wall image 41 to visually predict the position of the opening included in the papilla N.

[0230] <Third Embodiment>

[0231] In the above-described first embodiment, an example of obtaining the intestinal tract direction CD through image recognition processing of the intestinal wall image 41 was given, and in the above-described second embodiment, an example of obtaining the bulge direction RD through image recognition processing of the intestinal wall image 41 was given and described. However, the technology of the present invention is not limited thereto. In the third embodiment of the present invention, the traveling direction TD of the bile duct T is obtained through image recognition processing of the intestinal wall image 41.

[0232] For example, in an ERCP examination, a treatment instrument (for example, a cannula) is sometimes inserted into the papilla N, and further, the treatment instrument is inserted into the bile duct T or the pancreatic duct S inside the papilla N. At this time, in the intestinal wall image 41, it is difficult to grasp the traveling direction of the bile duct T or the pancreatic duct S existing inside the papilla N. Therefore, in the third embodiment of the present invention, the traveling direction of the bile duct T or the pancreatic duct S is obtained through image recognition processing of the intestinal wall image 41. In addition, hereinafter, for the sake of convenience of explanation, the case of the bile duct T will be described as an example.

[0233] As an example, as Figure 24 shown, every time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0234] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 to the learned model 84G. Thus, the learned model 84G outputs the traveling direction information 96 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the traveling direction information 96 output from the learned model 84E. The traveling direction information 96 is an example of the "traveling direction information" related to the technology of the present invention.

[0235] The learned model 84G is obtained by optimizing a neural network through machine learning using training data on the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be an object of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the traveling direction TD can be cited.

[0236] Here, the traveling direction TD of the bile duct T is determined, for example, as the direction passing through the tops of a plurality of folds of the papilla N. This is because, according to medical diagnostic opinions, the traveling direction of the bile duct T sometimes coincides with the line connecting the tops of the folds. Therefore, as an example of the annotation in the correct answer data, an annotation that sets the direction passing through the tops of the folds of the papilla N as the traveling direction TD of the bile duct T can be cited.

[0237] Then, the acquired time-series image group 89 is input into the learned model 84H. As a result, the learned model 84H outputs diverticulum region information 110 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the diverticulum region information 110 output from the learned model 84H. The diverticulum region information 110 is information (coordinates indicating the size and position of the diverticulum) capable of determining the region representing the diverticulum existing in the papilla N. Here, the diverticulum is a region where a part of the papilla N protrudes outward in a sac shape into the duodenum.

[0238] The learned model 84H is obtained by optimizing a neural network through machine learning using training data on the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be an object of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the region representing the diverticulum can be cited

[0239] The derivation unit 82C derives a manner for displaying the traveling direction TD. For example, the traveling direction TD is determined in a manner that avoids the diverticulum. This is because, according to medical diagnostic opinions, the traveling direction TD sometimes forms while avoiding the diverticulum. Therefore, the derivation unit 82C changes the display manner of the traveling direction TD according to the diverticulum region information 110. Specifically, the derivation unit 82C changes the part that intersects the diverticulum represented by the diverticulum region information 110 to a manner that avoids the diverticulum on the traveling direction TD represented by the traveling direction information 96. In this way, the derivation unit 82C generates display manner information 112 representing the changed display manner of the traveling direction TD.

[0240] As an example, as Figure 25 shown, the display control unit 82D acquires the display mode information 112 from the derivation unit 82C. The display control unit 82D generates a display image 94 including the changed traveling direction TD and the intestinal wall image 41 represented by the display mode information 112, and outputs it to the display device 13. In Figure 25 the example shown, in the display device 13, the intestinal wall image 41 with the changed traveling direction TD superimposed and displayed on the screen 36 is shown.

[0241] Next, with reference to Figure 26 , the operation related to the part of the duodenoscope system 10 related to the technology of the present invention will be described.

[0242] In Figure 26 , an example of the process of the medical support process performed by the processor 82 is shown. Figure 26 The process of the medical support process shown is an example of the "medical support method" related to the technology of the present invention.

[0243] In Figure 26 the medical support process shown, first, in step ST110, the image acquisition unit 82A determines whether one frame of shooting has been performed by the camera 48 provided in the endoscope viewer 18. In step ST10, if one frame of shooting has not been performed by the camera 48, it is determined as negative, and the determination of step ST110 is performed again. In step ST110, if one frame of shooting has been performed by the camera 48, it is determined as positive, and the medical support process proceeds to step ST112.

[0244] In step ST112, the image acquisition unit 82A acquires one frame of the intestinal wall image 41 from the camera 48 provided in the endoscope viewer 18. After performing the process of step ST112, the medical support process proceeds to step ST114.

[0245] In step ST114, the image recognition unit 82B detects the traveling direction TD by performing AI-based image recognition processing (that is, image recognition processing using the learned model 84G) on the intestinal wall image 41 acquired in step ST112. After performing the process of step ST114, the medical support process proceeds to step ST116.

[0246] In step ST116, the image recognition unit 82B detects the diverticulum region by performing AI-based image recognition processing (that is, image recognition processing using the learned model 84H) on the intestinal wall image 41 acquired in step ST112. After performing the process of step ST116, the medical support process proceeds to step ST118.

[0247] In step ST118, the derivation unit 82C changes the display mode of the traveling direction TD based on the traveling direction TD obtained by the image recognition unit 82B in step ST114 and the diverticulum region obtained by the image recognition unit 82B in step ST116. Specifically, the derivation unit 82C changes the display mode of the traveling direction TD so as to avoid the diverticulum region. After the process of step ST118 is executed, the medical support process proceeds to step ST120.

[0248] In step ST120, the display control unit 82D generates a display image 94 in which the traveling direction TD whose display mode has been changed by the derivation unit 82C in step ST118 is superimposed on the intestinal wall image 41. After the process of step ST120 is executed, the medical support process proceeds to step ST122.

[0249] In step ST122, the display control unit 82D outputs the display image 94 generated in step ST120 to the display device 13. After the process of step ST122 is executed, the medical support process proceeds to step ST124.

[0250] In step ST124, the display control unit 82D determines whether the condition for ending the medical support process is satisfied. As an example of the condition for ending the medical support process, a condition for giving an instruction to end the medical support process to the duodenoscope system 10 (for example, a condition in which the instruction to end the medical support process is received by the receiving device 62) can be cited.

[0251] In step ST124, when the condition for ending the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST110. In step ST124, when the condition for ending the medical support process is satisfied, the determination is affirmative, and the medical support process ends.

[0252] As described above, in the duodenoscope system 10 according to the present third embodiment, in the image recognition unit 82B of the processor 82, an image recognition process is performed on the intestinal wall image 41, and as a result of the image recognition process, the traveling direction TD of the bile duct in the intestinal wall image 41 is detected. Then, the traveling direction information 96 indicating the traveling direction TD is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes the traveling direction TD superimposed on the intestinal wall image 41. Thus, the traveling direction TD is displayed on the screen 36 in the display device 13. Thereby, a user observing the intestinal wall image 41 can visually grasp the traveling direction TD of the bile duct.

[0253] Moreover, in the duodenoscope system 10 according to the third embodiment, in the image recognition unit 82B, image recognition processing is performed on the intestinal wall image 41 to obtain diverticulum region information 110. In the derivation unit 82C, display mode information 112 is generated based on the traveling direction information 96 and the diverticulum region information 110. Then, the display mode information 112 indicating the changed traveling direction TD is output to the display control unit 82D, and the display image 94 generated in the display control unit 82D is output to the display device 13. The display image 94 includes the changed traveling direction TD superimposed on the intestinal wall image 41. In this way, the changed traveling direction TD is displayed on the screen 36 of the display device 13. Thereby, the user observing the intestinal wall image 41 can visually grasp the traveling direction TD of the bile duct changed according to the presence of the diverticulum. For example, it is possible to suppress the occurrence of a situation where the user observing the intestinal wall image 41 visually misgrasps the traveling direction TD of the bile duct leading to the opening of the papilla N due to the presence of the diverticulum.

[0254] Moreover, in the duodenoscope system 10 according to the third embodiment, in the derivation unit 82C, the display mode information 112 indicates the traveling direction TD changed in a manner that avoids the diverticulum in the traveling direction TD indicated by the traveling direction information 96. Then, the changed traveling direction TD is displayed on the screen 36 of the display device 13. Thereby, the user observing the intestinal wall image 41 can visually grasp the traveling direction TD of the bile duct changed in a manner that avoids the diverticulum.

[0255] In addition, in the above third embodiment, as an example of the method of changing the display mode of the traveling direction TD of the bile duct, the method of avoiding the diverticulum is given, but the technology of the present invention is not limited thereto. For example, in the traveling direction TD of the bile duct, the area intersecting with the diverticulum can be set not to be displayed, or the area intersecting with the diverticulum can be set as a dotted line or semi-transparent.

[0256] Moreover, in the above third embodiment, an example of the method of detecting the diverticulum in the intestinal wall image 41 through image recognition processing and changing the display mode of the traveling direction TD according to the diverticulum is given, but the technology of the present invention is not limited thereto. For example, it can also be a method without detecting the diverticulum.

[0257] (The tenth modification example)

[0258] In the above third embodiment, an example of the method of displaying the traveling direction TD of the bile duct avoiding the diverticulum is given, but the technology of the present invention is not limited thereto. In this tenth modification example, when the traveling direction TD of the bile duct intersects with the diverticulum, the user is notified of this content.

[0259] As an example, as Figure 27As shown, the derivation unit 82C obtains the traveling direction information 96 and the diverticulum region information 110 from the image recognition unit 82B. The derivation unit 82C determines the positional relationship between the diverticulum and the traveling direction TD based on the diverticulum region information 110 and the traveling direction information 96. Specifically, the derivation unit 82C compares the traveling direction TD represented by the traveling direction information 96 with the position and size of the diverticulum represented by the diverticulum region information 110 to determine whether the diverticulum and the traveling direction TD intersect. Further, when the derivation unit 82C determines that the traveling direction TD and the diverticulum have an intersecting positional relationship, it generates a notification information 114. The notification information 114 is an example of the "notification information" related to the technology of the present invention.

[0260] The derivation unit 82C outputs the notification information 114 to the display control unit 82D. At this time, the display control unit 82D generates a display image 94 including content for notifying the user that the diverticulum represented by the notification information 114 intersects the traveling direction TD. In Figure 27 the example shown, an example is shown in which the display device 13 shows a message "Diverticulum intersects the traveling direction" on the screen 37.

[0261] As described above, in the duodenoscope system 10 according to the present 10th modification example, in the derivation unit 82C, the positional relationship between the diverticulum and the traveling direction TD is determined based on the diverticulum region information 110 and the traveling direction information 96, and the notification information 114 is generated based on the determination result. In the display control unit 82D, the display image 94 is generated based on the notification information 114 and output to the display device 13. The display image 94 includes a display of the content that the diverticulum represented by the notification information 114 intersects the traveling direction. Thereby, the user can be made aware that the diverticulum intersects the traveling direction. For example, it is possible to suppress a situation where a user observing the intestinal wall image 41 visually misgrasps the traveling direction TD of the bile duct leading to the opening of the papilla N due to the presence of the diverticulum.

[0262] <Fourth Embodiment>

[0263] In the above-described First Embodiment to the above-described Third Embodiment, examples of ways of determining information related to a living tissue such as the intestinal tract direction CD, the papilla N, and the traveling direction TD of the bile duct by performing image recognition processing on the intestinal wall image 41 have been described, but the technology of the present invention is not limited thereto. In the present Fourth Embodiment, the relationship between the treatment instrument and the living tissue is determined by performing image recognition processing on the intestinal wall image 41.

[0264] For example, in an ERCP examination, various treatments using a treatment instrument are sometimes performed on the papilla N (for example, inserting a cannula into the papilla N). At this time, the positional relationship between the papilla N and the treatment instrument affects the success or failure of the surgical procedure. For example, when the advancing direction of the treatment instrument is not consistent with the orientation ND of the papilla, the treatment instrument cannot properly enter the papilla N, making it difficult to succeed in the surgical procedure. Therefore, in the fourth embodiment, the positional relationship between the treatment instrument and the papilla N is determined by image recognition processing of the intestinal wall image 41.

[0265] As an example, as Figure 28 shown, every time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0266] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 into the learned model 84I. As a result, the learned model 84I outputs the positional relationship information 116 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the positional relationship information 116 output from the learned model 84I. Here, the positional relationship information 116 is information capable of determining the position of the papilla N and the position of the treatment instrument (for example, the distance and angle between the position of the papilla N and the position of the tip of the treatment instrument).

[0267] The learned model 84I is obtained by optimizing a neural network through machine learning using training data. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image obtained by photographing a part that may be the object of an ERCP examination (for example, the inner wall of the duodenum) (for example, an image equivalent to the intestinal wall image 41). The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the position of the papilla N and the position of the treatment instrument can be cited.

[0268] The derivation unit 82C acquires the positional relationship information 116 from the image recognition unit 82B. The derivation unit 82C generates notification information 118 based on the positional relationship information 116, and the notification information 118 is information for notifying the user of the positional relationship between the papilla N and the treatment instrument. The derivation unit 82C compares the position of the treatment instrument indicated by the positional relationship information 116 with the position of the papilla N. And, when the position of the treatment instrument is the same as the position of the papilla N, the derivation unit 82C generates notification information 118 with the content that the position of the treatment instrument is the same as the position of the papilla N. And, when the position of the treatment instrument is not the same as the position of the papilla N, the derivation unit 82C generates notification information 118 with the content that the position of the treatment instrument is not the same as the position of the papilla N.

[0269] In addition, here, the case where the position of the treatment instrument coincides with the position of the nipple N has been described, but this is merely an example. For example, it is possible to determine whether the position of the treatment instrument and the position of the nipple N are within a preset range (for example, within a preset range of distance and angle).

[0270] As an example, as Figure 29 shown, the display control unit 82D acquires the notification information 118 from the derivation unit 82C. The derivation unit 82C outputs the notification information 118 to the display control unit 82D. At this time, the display control unit 82D generates a display image 94 including content for notifying the user of the positional relationship between the treatment instrument and the nipple N represented by the notification information 118. In Figure 29 the example shown, an example is shown in which a message "The position of the treatment instrument coincides with the position of the nipple" is displayed on the screen 37 in the display device 13.

[0271] As described above, in the duodenoscope system 10 according to the fourth embodiment, in the image recognition unit 82B of the processor 82, image recognition processing is performed on the intestinal wall image 41 to determine the positional relationship between the treatment instrument and the nipple. In the derivation unit 82C, determination related to the positional relationship between the treatment instrument and the nipple N is performed based on the positional relationship information 116 indicating the positional relationship between the treatment instrument and the nipple, and notification information 118 is generated based on the determination result. In the display control unit 82D, a display image 94 is generated based on the notification information 118 and output to the display device 13. The display image 94 includes a display related to the positional relationship between the treatment instrument and the nipple N represented by the notification information 118. Thus, the user observing the intestinal wall image 41 can be made aware of what kind of relationship the position of the treatment instrument has with the position of the nipple N.

[0272] (11th Modification Example)

[0273] In the above-described fourth embodiment, the relationship between the position of the nipple N and the position of the treatment instrument is given as an example of the positional relationship between the treatment instrument and the nipple N, but the technique of the present invention is not limited thereto. In the 11th modification example of the present invention, the relationship between the advancing direction of the treatment instrument and the nipple orientation ND is determined.

[0274] As an example, as Figure 30 shown, the image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 to the learned model 84J. Thus, the learned model 84J outputs positional relationship information 116A corresponding to the input time-series image group 89. Here, the positional relationship information 116A is information capable of determining the nipple orientation ND and the advancing direction of the treatment instrument (for example, the angle formed by the nipple orientation ND and the advancing direction of the treatment instrument).

[0275] The learned model 84J is obtained by optimizing a neural network through machine learning using training data on the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be the subject of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation that can determine the relationship between the nipple orientation ND and the advancing direction of the treatment instrument can be cited.

[0276] The derivation unit 82C acquires the position relationship information 116A from the image recognition unit 82B. The derivation unit 82C generates a notification information 118 based on the position relationship information 116A, and the notification information 118 is information for notifying the user of the position relationship between the nipple N and the treatment instrument. When the angle formed by the nipple orientation ND and the advancing direction of the treatment instrument is within a preset range, the derivation unit 82C generates the notification information 118 with the content that the two are consistent. Further, when the angle formed by the nipple orientation ND and the advancing direction of the treatment instrument exceeds the preset range, the derivation unit 82C generates the notification information 118 with the content that the two are inconsistent.

[0277] As described above, in the duodenoscope system 10 according to the present 11th modification example, in the image recognition unit 82B, the relationship between the advancing direction of the treatment instrument and the nipple orientation ND is determined. In the derivation unit 82C, the notification information 118 is generated based on the position relationship information 116A indicating the relationship between the advancing direction of the treatment instrument and the nipple orientation ND. Thereby, the user observing the intestinal wall image 41 can be made aware of what kind of relationship exists between the advancing direction of the treatment instrument and the nipple orientation NI).

[0278] In addition, in the above 11th modification example, an example of the method of determining the relationship between the advancing direction of the treatment instrument and the nipple orientation ND in the image recognition unit 82B is given, but the technology of the present invention is not limited thereto. For example, in the image recognition unit 82B, the relationship between the advancing direction of the treatment instrument and the nipple orientation ND, and the relationship between the position of the nipple N and the position of the treatment instrument can be determined. At this time, the position relationship information 116A is information indicating the relationship between the advancing direction of the treatment instrument and the nipple orientation ND and the relationship between the position of the nipple N and the position of the treatment instrument, and the derivation unit 82C makes a determination related to the relationship between the advancing direction of the treatment instrument and the nipple orientation ND and a determination related to the relationship between the position of the nipple N and the position of the treatment instrument based on the position relationship information 116A. Moreover, the derivation unit 82C generates the notification information 118 based on these determination results.

[0279] (12th modification example)

[0280] In the above-described fourth embodiment, an example of a method for determining the relationship between the position of the nipple N and the position of the treatment instrument was described as the positional relationship between the treatment instrument and the nipple N. However, the technology of the present invention is not limited thereto. In this eleventh modification example, the relationship between the advancing direction of the treatment instrument and the traveling direction TD of the bile duct is determined.

[0281] As an example, as Figure 31 shown, the image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 into the learned model 84K. Thereby, the learned model 84K outputs the position relationship information 116B corresponding to the input time-series image group 89. Here, the position relationship information 116B is information capable of determining the relationship between the traveling direction TD of the bile duct and the advancing direction of the treatment instrument (for example, the angle formed by the direction of the tangent line of the opening end portion in the traveling direction TD of the bile duct (hereinafter, simply referred to as "bile duct tangent direction") and the advancing direction of the treatment instrument).

[0282] The learned model 84K is obtained by optimizing a neural network through machine learning using training data for the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a correspondence between example data and correct answer data. The example data is, for example, an image (for example, an image corresponding to the intestinal wall image 41) obtained by photographing a part (for example, the inner wall of the duodenum) that may be an object of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the relationship between the traveling direction TD of the bile duct and the advancing direction of the treatment instrument can be cited.

[0283] The derivation unit 82C acquires the position relationship information 116B from the image recognition unit 82B. The derivation unit 82C generates notification information 118 based on the position relationship information 116B. The notification information 118 is information for notifying the user of the relationship between the traveling direction TD of the bile duct and the advancing direction of the treatment instrument. The derivation unit 82C generates the notification information 118 with the content that the two are consistent when the angle formed by the bile duct tangent direction and the advancing direction of the treatment instrument is within a preset range. And, the derivation unit 82C generates the notification information 118 with the content that the two are inconsistent when the angle formed by the bile duct tangent direction and the advancing direction of the treatment instrument exceeds the preset range.

[0284] As described above, in the duodenoscope system 10 according to the 12th modified example, in the image recognition unit 82B, the relationship between the advancing direction of the treatment instrument and the traveling direction TD of the bile duct is determined. In the derivation unit 82C, the notification information 118 is generated based on the positional relationship information 116B indicating the relationship between the advancing direction of the treatment instrument and the traveling direction TD of the bile duct. Thus, the user observing the intestinal wall image 41 can be made aware of what kind of relationship exists between the advancing direction of the treatment instrument and the traveling direction TD of the bile duct.

[0285] (13th modified example)

[0286] In the above-described 4th embodiment, the relationship between the position of the papilla N and the position of the treatment instrument is given as an example of the positional relationship between the treatment instrument and the papilla N, but the technique of the present invention is not limited thereto. In the 13th modified example, the relationship between the advancing direction of the treatment instrument and the orientation of the plane perpendicular to the rising direction RD with respect to the papilla elevation NA (hereinafter, also simply referred to as "vertical plane orientation") is determined.

[0287] As an example, as Figure 32 shown, the image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 into the learned model 84L. Thus, the learned model 84L outputs the positional relationship information 116C corresponding to the input time-series image group 89. Here, the positional relationship information 116C is information capable of determining the relationship between the vertical plane orientation and the advancing direction of the treatment instrument (for example, the angle formed by the vertical plane orientation and the advancing direction of the treatment instrument).

[0288] The learned model 84L is obtained by optimizing a neural network through machine learning of the neural network using training data. The training data is a plurality of data (i.e., multi-frame data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (for example, an image corresponding to the intestinal wall image 41) obtained by photographing a part (for example, the inner wall of the duodenum) that may be the subject of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation capable of determining the relationship between the vertical plane orientation and the advancing direction of the treatment instrument can be cited.

[0289] The derivation unit 82C obtains the positional relationship information 116C from the image recognition unit 82B. The derivation unit 82C generates a notification information 118 based on the positional relationship information 116C. The notification information 118 is information for notifying the user of the relationship between the vertical plane orientation and the advancing direction of the treatment instrument. When the angle formed by the vertical plane orientation and the advancing direction of the treatment instrument is within a preset range, the derivation unit 82C generates the notification information 118 with the content that the two are consistent. Further, when the angle formed by the vertical plane orientation and the advancing direction of the treatment instrument exceeds the preset range, the derivation unit 82C generates the notification information 118 with the content that the two are inconsistent.

[0290] As described above, in the duodenoscope system 10 according to the 13th modification example, in the image recognition unit 82B, the relationship between the vertical plane orientation and the advancing direction of the treatment instrument is determined. In the derivation unit 82C, the notification information 118 is generated based on the positional relationship information 116B indicating the relationship between the vertical plane orientation and the advancing direction of the treatment instrument. Thereby, the user observing the intestinal wall image 41 can be made aware of what kind of relationship exists between the vertical plane orientation and the advancing direction of the treatment instrument.

[0291] (14th modification example)

[0292] In the above-described 4th embodiment, an example of a method of determining the positional relationship between the treatment instrument and the papilla N by performing image recognition processing on the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In the 14th modification example of the present invention, by performing image recognition processing on the intestinal wall image 41, an evaluation value related to the positional relationship between the treatment instrument and the papilla N is obtained.

[0293] As an example, as Figure 33 shown, each time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, the time-series image group 89 is updated in a FIFO manner.

[0294] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A and inputs the acquired time-series image group 89 into the learned model 84M. As a result, the learned model 84M outputs evaluation value information 120 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the evaluation value information 120 output from the learned model 84M. Here, the evaluation value information 120 is information capable of determining an evaluation value related to the appropriate configuration of the papilla N and the treatment instrument (for example, the success degree of the surgical procedure determined based on the configuration of the papilla N and the treatment instrument). The evaluation value information 120 is, for example, a plurality of scores of an activation function (for example, a softmax function, etc.) input to the output layer of the learned model 84M (scores for each success or failure of the surgical procedure).

[0295] The learned model 84M is obtained by optimizing a neural network through machine learning using training data on the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be an object of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation that can determine an evaluation value related to the appropriate configuration of the papilla N and the treatment instrument (e.g., an annotation indicating whether the surgical procedure is successful or not) can be cited.

[0296] Furthermore, the image recognition unit 82B inputs the time-series image group 89 into the learned model 84N. As a result, the learned model 84N outputs contact presence / absence information 122 corresponding to the input time-series image group 89. The image recognition unit 82B acquires the contact presence / absence information 122 output from the learned model 84N. Here, the contact presence / absence information 122 is information that can determine whether or not the papilla N is in contact with the treatment instrument.

[0297] The learned model 84N is obtained by optimizing a neural network through machine learning using training data on the neural network. The training data is a plurality of data (i.e., multi-frame data) obtained by establishing a corresponding association between example data and correct answer data. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by photographing a part (e.g., the inner wall of the duodenum) that may be an object of an ERCP examination. The correct answer data is an annotation corresponding to the example data. As an example of the correct answer data, an annotation that can determine whether or not the papilla N is in contact with the treatment instrument can be cited.

[0298] The derivation unit 82C acquires the contact presence / absence information 122 from the image recognition unit 82B. The derivation unit 82C determines whether or not contact between the treatment instrument and the papilla N is detected based on the contact presence / absence information 122. When contact between the treatment instrument and the papilla N is detected, the derivation unit 82C generates notification information 124 based on the evaluation value information 120. The notification information 124 is information for notifying the user of the success probability of the surgical procedure (e.g., text indicating the success probability of the surgical procedure).

[0299] As an example, as Figure 34 shown, the display control unit 82D acquires the notification information 124 from the derivation unit 82C. The derivation unit 82C outputs the notification information 124 to the display control unit 82D. At this time, the display control unit 82D generates a display image 94 including content for notifying the user of the success probability of the surgical procedure indicated by the notification information 124. In Figure 34In the example shown, an example is shown in the display device 13 where a message "Cannula insertion success probability: 90%" is displayed on the screen 37.

[0300] As described above, in the duodenoscope system 10 according to the 14th modified example, in the image recognition unit 82B of the processor 82, image recognition processing is performed on the intestinal wall image 41, and an evaluation value related to the configuration of the treatment instrument and the papilla N is calculated. In the derivation unit 82C, notification information 124 is generated based on the evaluation value information 120 indicating the evaluation value. In the display control unit 82D, a display image 94 is generated based on the notification information 124 and output to the display device 13. The display image 94 includes a display related to the success probability of the surgical procedure indicated by the notification information 124. Thus, it is possible to notify the user observing the intestinal wall image 41 of the success probability of the surgical procedure using the treatment instrument. After the user grasps the success probability of the surgical procedure, the user can study whether to continue or change the operation, so that the surgical procedure using the treatment instrument can be supported.

[0301] Moreover, in the duodenoscope system 10 according to the 14th modified example, in the image recognition unit 82B, image recognition processing is performed on the intestinal wall image 41 to determine whether the treatment instrument is in contact with the papilla N. And in the derivation unit 82C, based on the contact presence / absence information 122, when the treatment instrument is in contact with the papilla N, notification information 124 is generated based on the evaluation value information 120. Thus, it is possible to notify the user observing the intestinal wall image 41 of the success probability of the surgical procedure using the treatment instrument only in the required scenario. In other words, it is possible to support the surgical procedure on the papilla N using the treatment instrument at an appropriate time.

[0302] <Fifth Embodiment>

[0303] In the above-described fourth embodiment, an example of a method of determining the positional relationship between the treatment instrument and the papilla N by performing image recognition processing on the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. In the fifth embodiment, when the treatment instrument is a cutting instrument, the cutting direction is obtained based on the result of the image recognition processing of the intestinal wall image 41.

[0304] For example, in an ERCP examination, a cutting instrument (for example, a papillotome) is sometimes used as the treatment instrument. This is because by using the cutting instrument to cut the papilla N, it is easy to insert the treatment instrument into the papilla N or to remove foreign substances in the bile duct T or the pancreatic duct S. At this time, if the direction (i.e., the cutting direction) of cutting the papilla N using the cutting instrument is erroneously selected, the surgical procedure may sometimes be difficult to succeed due to accidental bleeding or the like. Therefore, in the fifth embodiment, the direction recommended as the cutting direction (i.e., the cutting recommended direction) is determined by performing image recognition processing on the intestinal wall image 41.

[0305] As an example, as Figure 35 shown, every time the image acquisition unit 82A acquires the intestinal wall image 41 from the camera 48, it updates the time-series image group 89 in a FIFO manner.

[0306] The image recognition unit 82B acquires the time-series image group 89 from the image acquisition unit 82A, and inputs the acquired time-series image group 89 to the learned model 84E. As a result, the learned model 84E outputs the bulge direction information 104 corresponding to the input time-series image group 89.

[0307] The derivation unit 82C acquires the bulge direction information 104 from the image recognition unit 82B. And, the derivation unit 82C derives the incision recommendation direction information 126 based on the bulge direction information 104. The incision recommendation direction information 126 is information capable of determining the incision recommendation direction (for example, a set of position coordinates of the start point and the end point of the incision recommendation direction). The derivation unit 82C derives the incision recommendation direction according to a specified azimuth relationship between the bulge direction RD and the incision recommendation direction. Specifically, when the bulge direction RD is set to the 12 o'clock direction, the derivation unit 82C derives the incision recommendation direction as the 11 o'clock direction. The incision recommendation direction information 126 is an example of the "incision recommendation direction information" related to the technology of the present invention.

[0308] As an example, as Figure 36 shown, the display control unit 82D acquires the incision recommendation direction information 126 from the derivation unit 82C. The display control unit 82D generates an incision direction image 93F representing the incision direction according to the incision direction represented by the incision recommendation direction information 126. And, the display control unit 82D generates a display image 94 including the incision direction image 93F and the intestinal wall image 41, and outputs it to the display device 13. In Figure 36 the example shown, in the display device 13, the intestinal wall image 41 with the incision direction image 93F superimposed and displayed on the screen 36 is shown.

[0309] As described above, in the duodenoscope system 10 according to the present 5th embodiment, the incision recommendation direction information 126 is generated in the derivation unit 82C. In the display control unit 82D, the display image 94 is generated based on the incision recommendation direction information 126 and output to the display device 13. The display image 94 includes the incision direction image 93F representing the incision recommendation direction represented by the incision recommendation direction information 126. As a result, the user observing the intestinal wall image 41 can grasp the incision recommendation direction. As a result, the success of the incision of the papilla N can be supported.

[0310] (15th Modification Example)

[0311] In addition, in the above-described fifth embodiment, an example of a method for determining the recommended cutting direction was described, but the technology of the present invention is not limited thereto. In this 15th modification example, a direction that is not recommended as a cutting direction (i.e., a non-recommended cutting direction) can be determined.

[0312] As an example, as Figure 37 shown, the derivation unit 82C derives non-recommended cutting direction information 127. The non-recommended cutting direction information 127 is information capable of determining the non-recommended cutting direction (for example, an angle indicating a direction other than the recommended cutting direction). The derivation unit 82C derives the recommended cutting direction, for example, based on a predetermined azimuth relationship between the bulge direction RD and the recommended cutting direction. Specifically, when the bulge direction RD is set to the 12 o'clock direction, the derivation unit 82C derives the recommended cutting direction as the 11 o'clock direction. And the derivation unit 82C determines a range other than a preset angular range including the recommended cutting direction (for example, a range of ±5 degrees centered on the recommended cutting direction) as the non-recommended cutting direction. The non-recommended cutting direction information 127 is an example of the "non-recommended cutting direction information" related to the technology of the present invention.

[0313] The display control unit 82D acquires the non-recommended cutting direction information 127 from the derivation unit 82C. The display control unit 82D generates an image representing the non-recommended cutting direction, i.e., a non-recommended cutting direction image 93G, based on the non-recommended cutting direction indicated by the non-recommended cutting direction information 127. And the display control unit 82D generates a display image 94 including the non-recommended cutting direction image 93G and the intestinal wall image 41, and outputs it to the display device 13. In Figure 37 the example shown, on the display device 13, an intestinal wall image 41 with the non-recommended cutting direction image 93G superimposed and displayed on the screen 36 is shown.

[0314] As described above, in the duodenoscope system 10 according to this 15th modification example, the non-recommended cutting direction information 127 is generated in the derivation unit 82C. In the display control unit 82D, a display image 94 is generated based on the non-recommended cutting direction information 127 and output to the display device 13. The display image 94 includes a non-recommended cutting direction image 93G representing the non-recommended cutting direction indicated by the non-recommended cutting direction information 127. Thereby, the user observing the intestinal wall image 41 can grasp the non-recommended cutting direction. As a result, it is possible to support the success of the cutting of the papilla N.

[0315] In addition, in each of the above-described embodiments, as a method of displaying the operation direction to the user, an example of a method in which an image of an arrow indicating the operation direction is displayed on the screen 36 has been described, but the technology of the present invention is not limited thereto. For example, the image for displaying the operation direction to the user may be a triangular image indicating the operation direction. Also, it may be a method of displaying a message indicating the operation direction instead of or together with the image indicating the operation direction. Moreover, the image indicating the operation direction may be displayed on another window or another display device instead of being displayed on the screen 36.

[0316] In addition, in each of the above-described embodiments, an example of a method of showing the bile duct direction TD has been described, but the technology of the present invention is not limited thereto. It may also be a method of showing the traveling direction of the pancreatic duct S instead of or together with the bile duct direction TD.

[0317] In addition, in each of the above-described embodiments, an example of a method of outputting various information to the display device 13 has been described, but the technology of the present invention is not limited thereto. For example, it may be output to a sound output device such as a speaker (not shown) instead of or together with the display device 13, or it may be output to a printing device such as a printer (not shown).

[0318] In each of the above-described embodiments, an example of a method in which various information is output to the display device 13 and these information are displayed on the screen 36 of the display device 13 has been described, but the technology of the present invention is not limited thereto. The various information may also be output to an electronic medical record server. The electronic medical record server is a server for storing electronic medical record information representing the diagnosis and treatment results for a patient. The electronic medical record information includes various information.

[0319] The electronic medical record server is connected to the duodenoscope system 10 via a network. The electronic medical record server acquires the intestinal wall image 41 and various information from the duodenoscope system 10. The electronic medical record server stores the intestinal wall image 41 and various information as part of the diagnosis and treatment results represented by the electronic medical record information.

[0320] The electronic medical record server is also connected to terminals other than the duodenoscope system 10 (for example, personal computers installed in medical treatment facilities) via a network. Users such as the doctor 14 can acquire the intestinal wall image 41 and various information stored in the electronic medical record server via the terminal. In this way, by storing the intestinal wall image 41 and various information in the electronic medical record server, users can acquire the intestinal wall image 41 and various information.

[0321] In addition, in each of the above-described embodiments, an example of performing AI-based image recognition processing on the intestinal wall image 41 has been described, but the technology of the present invention is not limited thereto. For example, image recognition processing using a pattern matching method may be performed.

[0322] In the above-mentioned embodiment, an example of a method in which the processor 82 of the computer 76 included in the image processing device 25 performs medical support processing is given for explanation, but the technology of the present invention is not limited to this. For example, the medical support processing can be performed by the processor 70 of the computer 64 included in the control device 22. In addition, the device for performing the medical support processing can be provided outside the duodenoscope 12. As the device provided outside the duodenoscope 12, for example, at least one server and / or at least one personal computer that can be communicatively connected to the duodenoscope 12 can be cited. In addition, it can also be set that the medical support processing is distributed by multiple devices.

[0323] In the above embodiment, the example of storing the medical support processing program 84A in the NVM 84 is given for explanation, but the technology of the present invention is not limited to this. For example, the medical support processing program 84A can be stored in a portable non-temporary storage medium such as an SSD or a USB memory. The medical support processing program 84A stored in the non-temporary storage medium is installed in the computer 76 of the duodenoscope 12. The processor 82 executes the medical support processing according to the medical support processing program 84A.

[0324] Furthermore, the medical support processing program 84A is stored in a storage device such as another computer or server connected to the duodenoscope 12 via a network, and the medical support processing program 84A is downloaded in response to a request from the duodenoscope 12 and installed in the computer 76 .

[0325] Furthermore, it is not necessary to store all of the medical support processing program 84A in a storage device such as another computer or server device connected to the duodenoscope 12 or the NVM 84 , and a part of the medical support processing program 84A may be stored.

[0326] As hardware resources for executing medical support processing, various processors shown below can be used. As a processor, for example, a general-purpose processor, i.e., a CPU, which functions as a hardware resource for executing medical support processing by executing software, i.e., a program, can be cited. Also, as a processor, for example, a processor, i.e., a dedicated circuit, such as an FPGA, a PLD, or an ASIC, which has a circuit structure specially designed for executing specific processing can be cited. Any processor has a memory built in or connected, and any processor executes medical support processing by using the memory.

[0327] The hardware resource for executing the medical support processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resource for executing the medical support processing may be a single processor.

[0328] As an example of a configuration including one processor, first, there is the following method: a processor is configured by a combination of one or more CPUs and software, and this processor functions as a hardware resource for executing medical support processing. Second, there is the following method: as represented by an SoC or the like, a processor is used that implements the functions of an entire system including multiple hardware resources for executing medical support processing with one IC chip. In this way, the medical support processing is implemented by using one or more of the above various processors as hardware resources.

[0329] In addition, as the hardware configuration of these various processors, more specifically, a circuit formed by combining circuit elements such as semiconductor elements can be used. And the above medical support processing is only an example. Therefore, of course, unnecessary steps can be deleted, new steps can be added, or the processing order can be replaced within the scope not departing from the gist.

[0330] The description and illustration content shown above are detailed descriptions of parts related to the technology of the present invention, and are only examples of the technology of the present invention. For example, the description related to the above structure, function, operation, and effect is a description related to an example of the structure, function, operation, and effect of parts related to the technology of the present invention. Therefore, it goes without saying that within the scope not departing from the gist of the technology of the present invention, unnecessary parts can be deleted from the description and illustration content shown above, new elements can be added, or replacements can be made. And in order to avoid complication and facilitate understanding of the parts related to the technology of the present invention, in the description and illustration content shown above, on the basis of being able to implement the technology of the present invention, descriptions related to common technical knowledge that does not require special explanation are omitted.

[0331] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. And in this specification, when three or more matters are connected and expressed by "and / or", the same way of thinking as "A and / or B" is also applied.

[0332] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as when each is specifically and individually described.

[0333] The entire invention of Japanese Patent Application No. 2022-177614 filed on November 4, 2022 is incorporated herein by reference.

Claims

1. A medical support device, which includes a processor, The processor performs the following processes: Obtain papilla orientation-related information related to the orientation of the major duodenal papilla based on an intestinal wall image, where the intestinal wall image is obtained by photographing the intestinal wall including the major duodenal papilla within the duodenum using a camera provided in an endoscope viewer; Display the intestinal wall image on a screen; and Display the papilla orientation-related information on the screen.

2. The medical support device according to claim 1, wherein The papilla orientation-related information includes elevation direction information indicating the elevation direction of the major duodenal papilla.

3. The medical support device according to claim 2, wherein The papilla orientation information includes an elevation direction image indicating the elevation direction.

4. The medical support device according to claim 1, wherein The major duodenal papilla has an opening, The papilla orientation-related information includes plane direction information indicating the direction of the plane where the opening is located.

5. The medical support device according to claim 4, wherein The papilla orientation-related information includes angle-related information related to the relative angle between the plane and the posture of the endoscope viewer.

6. The medical support device according to claim 1, wherein The major duodenal papilla has an opening, The papilla orientation-related information includes: plane direction information indicating the direction of the plane where the opening is located; and angle-related information related to the relative angle between the plane and the posture of the endoscope viewer.

7. The medical support device according to claim 1, wherein The papilla orientation-related information includes a plane image capable of determining a plane that intersects the elevation direction of the major duodenal papilla at a specified angle.

8. The medical support device according to claim 1, wherein The papilla orientation-related information includes consistency information capable of determining the consistency between the elevation direction of the major duodenal papilla and the optical axis direction of the endoscope viewer.

9. The medical support device according to claim 1, wherein The major duodenal papilla includes a papilla elevation and a surrounding fold covering the papilla elevation, The papilla orientation-related information includes first direction information indicating a first direction from the top of the papilla elevation to the top of the surrounding fold.

10. The medical support device according to claim 9, wherein The first direction information includes a first direction image indicating the first direction.

11. The medical support device according to claim 9, wherein The papilla elevation has an opening, The papilla orientation-related information includes traveling direction information indicating the traveling direction of a tube leading to the opening, The traveling direction information is set based on the first direction information.

12. The medical support device according to claim 11, wherein The traveling direction information includes a traveling direction image indicating the traveling direction.

13. The medical support device according to claim 1, wherein The major duodenal papilla has a papilla elevation and a fold portion including a surrounding fold covering the papilla elevation, The processor determines the second direction based on the manner in which the fold appears in the intestinal wall image.

14. The medical support device according to claim 13, in, The processor determines the second direction in a manner that includes a region of the papillary protuberance and the fold portion projected on the intestinal wall image.

15. The medical support device according to claim 1, in, The processor acquires the nipple orientation related information by performing a first image recognition process on the intestinal wall image.

16. The medical support device according to claim 1, in, The processor performs the following processing: determining the travel direction of the tube leading to the opening of the duodenal papilla based on the intestinal wall image; and The traveling direction information capable of determining the traveling direction in the intestinal wall image is displayed on the screen.

17. The medical support device according to claim 16, in, The processor performs the following processing: Acquiring diverticulum region information capable of determining an image region representing a diverticulum, that is, a diverticulum region, within the intestinal wall image according to the intestinal wall image; and The display manner of the traveling direction information is changed according to the diverticulum region information.

18. The medical support device according to claim 17, in, The display manner is a manner in which the traveling direction avoids the diverticulum area determined according to the diverticulum area information.

19. The medical support device according to claim 16, in, The processor performs the following processing: Acquiring diverticulum region information capable of determining an image region representing a diverticulum, namely, a diverticulum region, within the intestinal wall image according to the intestinal wall image; determining a positional relationship between the diverticulum and the traveling direction according to the diverticulum area information and the traveling direction; and When the positional relationship is a positional relationship in which the diverticulum intersects the traveling direction, notification information is output to notify that the diverticulum is in a positional relationship in which the diverticulum intersects the traveling direction.

20. The medical support device according to claim 1, in, When an endoscope having the endoscope observation device and the treatment tool is inserted into the duodenum, The processor performs the following processing: determining a first relationship between the position of the treatment instrument and the position of the duodenal papilla and / or a second relationship between the advancing direction of the treatment instrument and the orientation of the duodenal papilla based on the intestinal wall image in which the treatment instrument is shown; and A first notification process is executed for performing a notification corresponding to the first relationship and / or the second relationship.

21. The medical support device according to claim 1, in, When an endoscope having the endoscope and the treatment tool is inserted into the duodenum, The processor performs the following processing: determining a third relationship between the advancing direction of the treatment instrument and the first orientation related to the orientation of the duodenal papilla based on the intestinal wall image in which the treatment instrument is shown; and A second notification process for performing notification corresponding to the third relationship is executed.

22. The medical support device according to claim 1, in, The processor performs the following processing: Determine the traveling direction of the tube leading to the opening of the duodenal papilla based on the intestinal wall image; When inserting an endoscope having the endoscopic viewer and the treatment instrument into the duodenum, determine the advancing direction of the treatment instrument based on the intestinal wall image showing the treatment instrument; and Execute a third notification process for performing a notification corresponding to a fourth relationship between the traveling direction and the advancing direction.

23. The medical support device according to claim 1, wherein the papilla orientation-related information includes incision recommended direction information or incision non-recommended direction information, the incision recommended direction information indicates a direction recommended as an incision direction for incising the duodenal papilla using an incision instrument for the duodenal papilla, and the incision non-recommended direction information indicates a direction not recommended as the incision direction.

24. The medical support device according to claim 1, wherein when inserting an endoscope having the endoscopic viewer and the treatment instrument into the duodenum, the processor performs the following processing: Obtain an evaluation value related to the positional relationship between the duodenal papilla and the treatment instrument based on the intestinal wall image showing the treatment instrument; and Output information based on the evaluation value.

25. The medical support device according to claim 24, wherein when inserting an endoscope having the endoscopic viewer and the treatment instrument into the duodenum, the processor outputs information based on the evaluation value when detecting a state in which the treatment instrument contacts the duodenal papilla based on the intestinal wall image showing the treatment instrument.

26. A medical support device, comprising a processor, the processor performs the following processing: Determine the traveling direction of the tube leading to the opening of the duodenal papilla based on the intestinal wall image obtained by photographing the intestinal wall including the duodenal papilla in the duodenum using a camera provided in the endoscopic viewer; Display the intestinal wall image on a screen; and Display traveling direction information capable of determining the traveling direction within the intestinal wall image on the screen.

27. An endoscope, comprising: The medical support device according to any one of claims 1 to 26; and The endoscopic viewer.

28. A medical support method, comprising the following steps: Obtain papilla orientation-related information related to the orientation of the duodenal papilla based on the intestinal wall image obtained by photographing the intestinal wall including the duodenal papilla in the duodenum using a camera provided in the endoscopic viewer; Display the intestinal wall image on a screen; and Display the papilla orientation-related information on the screen.

29. A medical support method, comprising the following steps: Determine the traveling direction of the tube leading to the opening of the duodenal papilla based on the intestinal wall image obtained by photographing the intestinal wall including the duodenal papilla in the duodenum using a camera provided in the endoscopic viewer; Display the intestinal wall image on a screen; and Display the travel direction information that can determine the travel direction within the intestinal wall image on the screen.

Images