Medical assistance device, endoscope, medical assistance method, and program

Abstract

This medical assistance device comprises a processor. The processor executes an image recognition process on an intestinal wall image obtained by imaging an intestinal wall including duodenal papillae in the duodenum using a camera provided to an endoscope to identify papilla types which are the types of duodenal papillae, and outputs relevant information related to the papilla type.

Description

Medical support device, endoscope, medical support method, and program

[0001] The technology of the present disclosure relates to a medical support device, an endoscope, a medical support method, and a program.

[0002] Japanese Patent Application Laid-Open Publication No. 2020-62218 discloses a learning device that includes an acquisition unit that acquires multiple pieces of information that associate images of the duodenal papilla of Vata in the bile duct with information indicating a cannulation method, which is a method of inserting a catheter into the bile duct; a learning unit that performs machine learning using information indicating the cannulation method as training data based on images of the duodenal papilla of Vata in the bile duct; and a memory unit that associates and stores the results of the machine learning by the learning unit with the information indicating the cannulation method.

[0003] One embodiment of the technique of the present disclosure provides a medical support device, an endoscope, a medical support method, and a program that can support the implementation of medical care according to the type of duodenal papilla.

[0004] A first aspect of the technology of the present disclosure is a medical support device that includes a processor, which identifies the type of duodenal papilla by performing image recognition processing on an intestinal wall image obtained by capturing an image of the intestinal wall, including the duodenal papilla, in the duodenum using a camera attached to an endoscope, and outputs relevant information related to the papilla type.

[0005] A second aspect of the technology of the present disclosure is the medical support device according to the first aspect, in which outputting the related information is displaying the related information on a screen.

[0006] A third aspect of the technique of the present disclosure is the medical support device according to the first or second aspect, in which the related information includes a schema determined according to the type of nipple.

[0007] A fourth aspect of the technology of the present disclosure is a medical support device according to any one of the first to third aspects, in which the related information includes confluence type information, which is determined according to the type of papilla and is information that can identify the confluence type at which the bile duct and pancreatic duct join.

[0008] A fifth aspect of the technology of the present disclosure is a medical support device according to any one of the first to fourth aspects, in which the image recognition process includes a classification process for classifying nipple types, and the related information includes certainty information indicating the certainty of each nipple type classified by the classification process.

[0009] A sixth aspect of the technology of the present disclosure is a medical support device according to any one of the first to fifth aspects, in which the frequency of occurrence of the confluence of the bile duct and pancreatic duct is determined for each type of papilla, and the processor outputs, as related information, information including frequency of occurrence information indicating the frequency of occurrence according to the identified type of papilla.

[0010] A seventh aspect of the technology of the present disclosure is a medical support device according to any one of the first to fifth aspects, wherein the papilla types include a first papilla type, which has any of a plurality of confluence types where the bile duct and the pancreatic duct converge, and when the processor identifies the first papilla type as the papilla type, it outputs, as related information, information including occurrence frequency information indicating the occurrence frequency of each confluence type.

[0011] An eighth aspect of the technology of the present disclosure is a medical support device according to the seventh aspect, in which the first nipple type is a villous type or a flat type, and the multiple confluence types are a partition type and a common duct type.

[0012] A ninth aspect of the technology of the present disclosure is a medical support device according to any one of the first to eighth aspects, in which the related information includes auxiliary information, which is a confluence pattern where the bile duct and the pancreatic duct join, and which is information that assists medical procedures performed for the confluence pattern determined according to the type of papilla.

[0013] A tenth aspect of the technology of the present disclosure is the medical support device according to the ninth aspect, in which the processor outputs auxiliary information when a plurality of merging types exist for the identified nipple type.

[0014] An eleventh aspect of the technology of the present disclosure is a medical support device according to any one of the first to tenth aspects, in which a processor identifies the type of papilla by performing image recognition processing on an intestinal wall image on a frame-by-frame basis.

[0015] A twelfth aspect of the technology of the present disclosure is a medical support device according to any one of the first to tenth aspects, in which the image recognition processing includes a first image recognition processing and a second image recognition processing, and the processor detects a duodenal papilla region by executing the first image recognition processing on an intestinal wall image, and identifies the papilla type by executing the second image recognition processing on the detected duodenal papilla region.

[0016] A thirteenth aspect of the technique of the present disclosure is the medical support device according to any one of the first to twelfth aspects, wherein the related information is stored in an external device and / or a medical record.

[0017] A fourteenth aspect of the technique of the present disclosure is an endoscope including a medical support device according to any one of the first to thirteenth aspects and an endoscope.

[0018] A fifteenth aspect of the technology of the present disclosure is a medical support method that includes identifying the type of duodenal papilla by performing image recognition processing on an intestinal wall image obtained by capturing an image of the intestinal wall, including the duodenal papilla, in the duodenum using a camera provided in an endoscope, and outputting related information related to the papilla type.

[0019] A sixteenth aspect of the technology of the present disclosure is a program for causing a computer to execute processing including identifying the papilla type, which is the type of duodenal papilla, by performing image recognition processing on an intestinal wall image obtained by capturing an image of the intestinal wall including the duodenal papilla in the duodenum using a camera provided on an endoscope, and outputting related information related to the papilla type.

[0020] 1 is a conceptual diagram showing an example of an aspect in which a duodenoscope system is used. FIG. 2 is a conceptual diagram showing an example of the overall configuration of a duodenoscope system. FIG. 3 is a block diagram showing an example of the hardware configuration of the electrical system of a duodenoscope system. FIG. 4 is a conceptual diagram showing an example of an aspect in which a duodenoscope is used. FIG. 5 is a block diagram showing an example of the hardware configuration of the electrical system of an image processing device. FIG. 6 is a conceptual diagram showing an example of the correlation between an endoscope, an NVM, an image acquisition unit, an image recognition unit, and a support information acquisition unit. FIG. 7 is a conceptual diagram showing an example of the correlation between a display device, an image acquisition unit, an image recognition unit, a support information acquisition unit, and a display control unit. FIG. 8 is a flowchart showing an example of the flow of medical support processing. FIG. 9 is a conceptual diagram showing an example of the correlation between an endoscope, an NVM, an image acquisition unit, an image recognition unit, and a support information acquisition unit. FIG. 10 is a conceptual diagram showing an example of the correlation between a display device, an image acquisition unit, an image recognition unit, a support information acquisition unit, and a display control unit. 1 is a conceptual diagram showing an example of the correlation between a display device, an image acquisition unit, an image recognition unit, a support information acquisition unit, and a display control unit. 2 is a conceptual diagram showing an example of the correlation between an endoscope, an NVM, an image acquisition unit, an image recognition unit, and a support information acquisition unit. 3 is a conceptual diagram showing an example of how intestinal wall images, support information, and papilla type information generated by a duodenoscope system are stored in an electronic medical record server.

[0021] Hereinafter, exemplary embodiments of a medical support device, an endoscope, a medical support method, and a program according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] CPU is an abbreviation for "Central Processing Unit." GPU is an abbreviation for "Graphics Processing Unit." RAM is an abbreviation for "Random Access Memory." NVM is an abbreviation for "Non-volatile memory." EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory." ASIC is an abbreviation for "Application Specific Integrated Circuit." PLD is an abbreviation for "Programmable Logic Device." FPGA is an abbreviation for "Field-Programmable Gate Array." SoC is an abbreviation for "System-on-a-chip." SSD is an abbreviation for "Solid State Drive." USB is an abbreviation for "Universal Serial Bus." HDD is an abbreviation for "Hard Disk Drive." EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". FIFO is an abbreviation for "First In First Out". ERCP is an abbreviation for "Endoscopic Retrograde Cholangio-Pancreatography".

[0024] 1 , a duodenoscope system 10 includes a duodenoscope 12 and a display device 13. The duodenoscope 12 is used by a doctor 14 in an endoscopic examination. The duodenoscope 12 is communicatively connected to a communication device (not shown), and information obtained by the duodenoscope 12 is transmitted to the communication device. The communication device receives the information transmitted from the duodenoscope 12 and performs processing using the received information (for example, processing to record the information in an electronic medical record, etc.).

[0025] The duodenoscope 12 includes an endoscope 18. The duodenoscope 12 is a device for performing medical treatment on an observation target 21 (e.g., the duodenum) inside the body of a subject 20 (e.g., a patient) using the endoscope 18. The observation target 21 is an object to be observed by a doctor 14. The endoscope 18 is inserted into the body of the subject 20. The duodenoscope 12 causes the endoscope 18 inserted inside the body of the subject 20 to capture images of the observation target 21 inside the body of the subject 20, and performs various medical procedures on the observation target 21 as necessary. The duodenoscope 12 is an example of an "endoscope" according to the technology of the present disclosure.

[0026] The duodenoscope 12 acquires and outputs an image showing the state of the inside of the body by capturing an image of the inside of the body of the subject 20. In this embodiment, the duodenoscope 12 is an endoscope having an optical imaging function that captures an image of reflected light obtained by irradiating light inside the body and reflecting it off an observation target 21.

[0027] The duodenoscope 12 is equipped with a control device 22, a light source device 24, and an image processing device 25. The control device 22 and the light source device 24 are mounted on a wagon 34. The wagon 34 has a plurality of stands arranged vertically, with the image processing device 25, the control device 22, and the light source device 24 mounted from the lower stand to the upper stand. In addition, a display device 13 is mounted on the top stand of the wagon 34.

[0028] The control device 22 is a device that controls the entire duodenoscope 12. The image processing device 25 is a device that performs image processing on images captured by the duodenoscope 12 under the control of the control device 22.

[0029] The display device 13 displays various information including images (for example, images that have been subjected to image processing by the image processing device 25). Examples of the display device 13 include a liquid crystal display and an EL display. Alternatively, a tablet terminal with a display may be used instead of the display device 13 or together with the display device 13.

[0030] The display device 13 displays multiple screens side by side. In the example shown in FIG. 1 , screens 36, 37, and 38 are shown. The screen 36 displays an endoscopic image 40 acquired by the duodenoscope 12. The endoscopic image 40 shows an observation target 21. The endoscopic image 40 is an image acquired by capturing an image of the observation target 21 inside the body of the subject 20 using a camera 48 (see FIG. 2 ) provided on the endoscope 18. An example of the observation target 21 is the intestinal wall of the duodenum. For ease of explanation, the following description will use an intestinal wall image 41, which is an endoscopic image 40 capturing an image of the intestinal wall of the duodenum as the observation target 21. Note that the duodenum is merely an example, and any region that can be imaged by the duodenoscope 12 may be used. Examples of regions that can be imaged by the duodenoscope 12 include the esophagus and the stomach. The intestinal wall image 41 is an example of an "intestinal wall image" according to the technology of the present disclosure.

[0031] A moving image including a plurality of frames of intestinal wall images 41 is displayed on the screen 36. That is, the plurality of frames of intestinal wall images 41 are displayed on the screen 36 at a predetermined frame rate (for example, several tens of frames per second).

[0032] 2, the duodenoscope 12 includes an operating section 42 and an insertion section 44. The insertion section 44 is partially curved by operating the operating section 42. The insertion section 44 is inserted while being curved in accordance with the shape of the observation target 21 (for example, the shape of the duodenum) in accordance with the operation of the operating section 42 by the physician 14.

[0033] The distal end 46 of the insertion section 44 is provided with a camera 48, an illumination device 50, a treatment opening 51, and an erection mechanism 52. The camera 48 and the illumination device 50 are provided on the side of the distal end 46. In other words, the duodenoscope 12 is a side-viewing endoscope. This makes it easier to observe the intestinal wall of the duodenum.

[0034] The camera 48 is a device that captures an image of the inside of the subject 20 to obtain an intestinal wall image 41 as a medical image. An example of the camera 48 is a CMOS camera. However, this is merely an example, and other types of cameras such as a CCD camera may also be used. The camera 48 is an example of a "camera" according to the technology of the present disclosure.

[0035] The illumination device 50 has an illumination window 50A. The illumination device 50 emits light through the illumination window 50A. Types of light emitted from the illumination device 50 include, for example, visible light (e.g., white light) and invisible light (e.g., near-infrared light). The illumination device 50 also emits special light through the illumination window 50A. Examples of the special light include light for BLI and / or light for LCI. The camera 48 captures images of the inside of the subject 20 by an optical method while light is being emitted from the illumination device 50 inside the subject 20.

[0036] The treatment opening 51 is used as a treatment tool ejection port for ejecting a treatment tool 54 from the distal end portion 46, a suction port for sucking blood and internal waste, and a delivery port for delivering fluid.

[0037] A treatment tool 54 protrudes from the treatment opening 51 in accordance with the operation of the doctor 14. The treatment tool 54 is inserted into the insertion section 44 from a treatment tool insertion port 58. The treatment tool 54 passes through the insertion section 44 via the treatment tool insertion port 58 and protrudes from the treatment opening 51 into the body of the subject 20. In the example shown in FIG. 2 , a cannula protrudes from the treatment opening 51 as the treatment tool 54. The cannula is merely one example of the treatment tool 54, and other examples of the treatment tool 54 include a papillotomy knife, a snare, etc.

[0038] The standing mechanism 52 changes the protruding direction of the treatment tool 54 protruding from the treatment opening 51. The standing mechanism 52 includes a guide 52A, which rises relative to the protruding direction of the treatment tool 54, thereby changing the protruding direction of the treatment tool 54 along the guide 52A. This makes it easy to protrude the treatment tool 54 toward the intestinal wall. In the example shown in FIG. 2 , the standing mechanism 52 changes the protruding direction of the treatment tool 54 to a direction perpendicular to the traveling direction of the tip portion 46. The standing mechanism 52 is operated by the doctor 14 via the operation unit 42. This adjusts the degree of change in the protruding direction of the treatment tool 54.

[0039] The endoscope 18 is connected to the control device 22 and the light source device 24 via a universal cord 60. The control device 22 is connected to the display device 13 and the reception device 62. The reception device 62 receives instructions from a user (e.g., the doctor 14) and outputs the received instructions as an electrical signal. In the example shown in Fig. 2, a keyboard is given as an example of the reception device 62. However, this is merely an example, and the reception device 62 may also be a mouse, a touch panel, a foot switch, and / or a microphone, etc.

[0040] The control device 22 controls the entire duodenoscope 12. For example, the control device 22 controls the light source device 24 and exchanges various signals with the camera 48. The light source device 24 emits light under the control of the control device 22 and supplies the light to the illumination device 50. The illumination device 50 has a built-in light guide, and the light supplied from the light source device 24 passes through the light guide and is irradiated from illumination windows 50A and 50B. The control device 22 causes the camera 48 to capture an image, acquires an intestinal wall image 41 (see FIG. 1 ) from the camera 48, and outputs the image to a predetermined output destination (for example, the image processing device 25).

[0041] The image processing device 25 is communicably connected to the control device 22, and performs image processing on the intestinal wall image 41 output from the control device 22. Details of the image processing in the image processing device 25 will be described later. The image processing device 25 outputs the processed intestinal wall image 41 to a predetermined output destination (e.g., the display device 13). Note that, although an example in which the intestinal wall image 41 output from the control device 22 is output to the display device 13 via the image processing device 25 has been described here, this is merely one example. The control device 22 and the display device 13 may be connected, and the intestinal wall image 41 that has been subjected to image processing by the image processing device 25 may be displayed on the display device 13 via the control device 22.

[0042] 3 , the control device 22 includes a computer 64, a bus 66, and an external I / F 68. The computer 64 includes a processor 70, a RAM 72, and an NVM 74. The processor 70, the RAM 72, the NVM 74, and the external I / F 68 are connected to the bus 66.

[0043] For example, the processor 70 has a CPU and a GPU, and controls the entire control device 22. The GPU operates under the control of the CPU, and is responsible for executing various graphic processing and performing calculations using neural networks. The processor 70 may be one or more CPUs that have integrated GPU functionality, or one or more CPUs that do not have integrated GPU functionality.

[0044] The RAM 72 is a memory that temporarily stores information and is used as a work memory by the processor 70. The NVM 74 is a nonvolatile storage device that stores various programs, various parameters, and the like. An example of the NVM 74 is a flash memory (for example, an EEPROM and / or an SSD). Note that the flash memory is merely one example, and the NVM 74 may be another nonvolatile storage device such as an HDD, or may be a combination of two or more types of nonvolatile storage devices.

[0045] The external I / F 68 controls the exchange of various information between devices that exist outside the control device 22 (hereinafter also referred to as "external devices") and the processor 70. An example of the external I / F 68 is a USB interface.

[0046] The external I / F 68 is connected to a camera 48 as one of the external devices, and the external I / F 68 controls the exchange of various information between the camera 48 provided in the endoscope 18 and the processor 70. The processor 70 controls the camera 48 via the external I / F 68. The processor 70 also acquires, via the external I / F 68, intestinal wall images 41 (see FIG. 1 ) obtained by imaging the inside of the body of the subject 20 with the camera 48 provided in the endoscope 18.

[0047] The light source device 24 is connected to the external I / F 68 as one of the external devices, and the external I / F 68 controls the exchange of various information between the light source device 24 and the processor 70. The light source device 24 supplies light to the illumination device 50 under the control of the processor 70. The illumination device 50 irradiates the light supplied from the light source device 24.

[0048] The external I / F 68 is connected to a reception device 62 as one of the external devices, and the processor 70 acquires instructions accepted by the reception device 62 via the external I / F 68 and executes processing according to the acquired instructions.

[0049] The external I / F 68 is connected to the image processing device 25 as one of the external devices, and the processor 70 outputs the intestinal wall image 41 to the image processing device 25 via the external I / F 68 .

[0050] Incidentally, among the procedures for the duodenum using an endoscope, a procedure called ERCP (endoscopic retrograde cholangiopancreatography) may be performed. As an example, as shown in FIG. 4 , in an ERCP examination, for example, a duodenoscope 12 is first inserted into the duodenum J via the esophagus and stomach. In this case, the insertion state of the duodenoscope 12 may be confirmed by X-ray imaging. Then, a tip 46 of the duodenoscope 12 reaches the vicinity of the duodenal papilla N (hereinafter also simply referred to as "papilla N") present in the intestinal wall of the duodenum J.

[0051] In an ERCP examination, for example, a cannula 54A is inserted through the papilla N. The papilla N is a protruding portion of the intestinal wall of the duodenum J, and the openings of the ends of the bile duct T (e.g., common bile duct, intrahepatic bile duct, cystic duct) and pancreatic duct S are present at the papillary protuberance NA of the papilla N. X-ray imaging is performed after a contrast agent is injected into the bile duct T, pancreatic duct S, etc. through the opening of the papilla N via the cannula 54A. In this ERCP examination, it is important to determine the type of papilla N before performing the procedure. This is because the type of papilla N affects the success or failure of insertion when inserting the cannula 54A, and the condition of the bile duct T and pancreatic duct S (e.g., duct shape, etc.) corresponding to the type of papilla N also affects the success or failure of intubation after insertion. However, for example, it is difficult for the physician 14 to constantly keep track of the type of papilla N while operating the duodenoscope 12.

[0052] Furthermore, for example, a doctor 14 with little experience in ERCP examinations may refer to information related to the procedure, including the type of papilla N, but in this case too, because the doctor is concentrating on operating the duodenoscope 12, it is difficult for the doctor 14 to refer to text or notes and confirm information related to the procedure.

[0053] In view of the above circumstances, in this embodiment, medical support processing is performed by the processor 82 of the image processing device 25 in order to support the implementation of medical care according to the type of duodenal papilla.

[0054] 5 , the image processing device 25 includes a computer 76, an external I / F 78, and a bus 80. The computer 76 includes a processor 82, an NVM 84, and a RAM 81. The processor 82, the NVM 84, the RAM 81, and the external I / F 78 are connected to the bus 80. The computer 76 is an example of a "medical support device" and a "computer" according to the technology of the present disclosure. The processor 82 is an example of a "processor" according to the technology of the present disclosure.

[0055] The hardware configuration of the computer 76 (i.e., the processor 82, the NVM 84, and the RAM 81) is basically the same as the hardware configuration of the computer 64 shown in Fig. 3, and therefore a description of the hardware configuration of the computer 76 will be omitted here. Also, the role of the external I / F 78 in the image processing device 25, namely, sending and receiving information to and from the outside, is basically the same as the role of the external I / F 68 in the control device 22 shown in Fig. 3, and therefore a description thereof will be omitted here.

[0056] A medical support processing program 84A is stored in the NVM 84. The medical support processing program 84A is an example of a "program" according to the technology of the present disclosure. The processor 82 reads the medical support processing program 84A from the NVM 84 and executes the read medical support processing program 84A on the RAM 81. The medical support processing is realized by the processor 82 operating as an image acquisition unit 82A, an image recognition unit 82B, a support information acquisition unit 82C, and a display control unit 82D in accordance with the medical support processing program 84A executed on the RAM 81.

[0057] A trained model 84B is stored in the NVM 84. In this embodiment, the image recognition unit 82B performs AI-based image recognition processing as the image recognition processing for object detection. The trained model 84B is optimized by performing machine learning on the neural network in advance.

[0058] The NVM 84 stores a support information table 83. The support information table 83 will be described in detail later.

[0059] As an example, as shown in FIG. 6 , the image acquisition unit 82A acquires an intestinal wall image 41 from a camera 48 provided on an endoscope 18, the intestinal wall image 41 being generated by the camera 48 capturing images at an imaging frame rate (e.g., several tens of frames per second) on a frame-by-frame basis.

[0060] The image acquisition unit 82A holds a time-series image group 89. The time-series image group 89 is a plurality of time-series intestinal wall images 41 that capture the object of observation 21. The time-series image group 89 includes, for example, a certain number of frames (e.g., a predetermined number of frames within a range of several tens to several hundreds of frames) of intestinal wall images 41. The image acquisition unit 82A updates the time-series image group 89 in a FIFO manner every time it acquires an intestinal wall image 41 from the camera 48.

[0061] Here, an example is given in which the time-series image group 89 is stored and updated by the image acquisition unit 82A, but this is merely one example. For example, the time-series image group 89 may be stored and updated in a memory connected to the processor 82, such as the RAM 81.

[0062] The image recognition unit 82B acquires the intestinal wall image 41 of a frame designated by the user from the time-series image group 89 held by the image acquisition unit 82A. The designated frame is, for example, a frame at a time point designated by the user operating the operation unit 42. The image recognition unit 82B performs image recognition processing on the intestinal wall image 41 using the trained model 84B. By performing the image recognition processing, the type of papilla N included in the observation target 21 is identified. In this embodiment, identifying the type of papilla N refers to a process of storing in memory in association with papilla type information 90 (e.g., the name of the type of papilla N shown in the intestinal wall image 41) that can identify the type of papilla N. The papilla type information 90 is an example of "related information" according to the technology of the present disclosure.

[0063] The trained model 84B is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify the type of papilla N.

[0064] Note that, although an example in which only one trained model 84B is used by the image recognition unit 82B is given here, this is merely one example. For example, a trained model 84B selected from multiple trained models 84B may be used by the image recognition unit 82B. In this case, each trained model 84B is created by performing machine learning specialized for a specific ERCP examination technique (e.g., the position of the duodenoscope 12 relative to the papilla N), and the trained model 84B corresponding to the currently performed ERCP examination technique is selected and used by the image recognition unit 82B.

[0065] The image recognition unit 82B inputs the intestinal wall image 41 acquired from the image acquisition unit 82A to the trained model 84B. As a result, the trained model 84B outputs papilla type information 90 corresponding to the input intestinal wall image 41. The image recognition unit 82B acquires the papilla type information 90 output from the trained model 84B.

[0066] The support information acquisition unit 82C acquires support information 86 according to the type of papilla N. The support information 86 is information provided to the user to support the procedure in the ERCP examination. The support information 86 includes junction type information 86A and a schema 86B. The junction type information 86A is determined according to the type of papilla N and is information that can identify the junction type of the bile duct and the pancreatic duct. The schema 86B is an image showing the junction of the bile duct and the pancreatic duct. The support information 86, the junction type information 86A, and the schema 86B are examples of "related information" according to the technology of the present disclosure. The junction type information 86A is an example of "junction type information" according to the technology of the present disclosure, and the schema 86B is an example of "schema" according to the technology of the present disclosure.

[0067] The support information acquisition unit 82C acquires nipple type information 90 from the image recognition unit 82B. The support information acquisition unit 82C also acquires a support information table 83 from the NVM 84. The support information acquisition unit 82C uses the support information table 83 to acquire support information 86 corresponding to the nipple type information 90. Here, the support information table 83 is information in which nipple type information 90, merging format information 86A, and a schema 86B, which correspond to one another, are associated according to the corresponding relationships. The support information table 83 is a table that, for example, uses nipple type information 90 as input information and uses merging format information 86A and a schema 86B corresponding to the type of nipple N as output information.

[0068] 6 , the support information table 83 shows an example of an image in which, when the type of papilla N is a separate opening type, the confluence type is a separated type, and the schema 86B shows an example of an image in which the bile duct and pancreatic duct are separated within the papilla N. Also, the support information table 83 shows an example of an image in which, when the type of papilla N is an onion type, the confluence type is a separated type, and the schema 86B shows an example of an image in which the bile duct and pancreatic duct are separated within the papilla N and the pancreatic duct branches within the papilla N. Furthermore, the support information table 83 shows an example of an image in which, when the type of papilla N is a nodular type, the confluence type is a partition type, and the schema 86B shows an example of an image in which the bile duct and pancreatic duct are adjacent to each other at the tip side of the protrusion of the papilla N.

[0069] Note that although examples of the separate opening type, onion type, and nodule type are given here as input information for the support information table 83, these are merely examples. The contents of the input information and output information for the support information table 83 are determined appropriately based on medical knowledge regarding the type of nipple N and the merging type. The output information of the support information table 83 may also be merging type information 86A alone. In this case, the schema 86B has the merging type information 86A as incidental information. Then, based on the merging type information 86A acquired using the support information table 83, the support information acquisition unit 82C acquires a schema 86B having incidental information corresponding to the merging type information 86A.

[0070] Furthermore, in deriving the support information 86, a support information calculation formula (not shown) may be used instead of the support information table 83. The support information calculation formula is a calculation formula in which a value indicating the type of nipple N is an independent variable, and a value indicating the joining type and a value indicating the schema 86B are dependent variables.

[0071] 7 as an example, the display control unit 82D acquires an intestinal wall image 41 from the image acquisition unit 82A. The display control unit 82D also acquires papilla type information 90 from the image recognition unit 82B. The display control unit 82D also acquires support information 86 from the support information acquisition unit 82C. The display control unit 82D generates a display image 94 including the intestinal wall image 41, the type of papilla N indicated by the papilla type information 90, and the merging format and schema indicated by the support information 86, and outputs the display image 94 to the display device 13. Specifically, the display control unit 82D controls a GUI (Graphical User Interface) to display the display image 94, thereby causing the display device 13 to display screens 36 to 38. The screens 36 to 38 are examples of "screens" according to the technology of the present disclosure.

[0072] 7, an intestinal wall image 41 is displayed on the screen 36. A schema 86B is displayed on the screen 37. A message indicating the type of papilla N and a message indicating the merging format are displayed on the screen 38. For example, the doctor 14 visually checks the intestinal wall image 41 displayed on the screen 36, and also checks the schema 86B displayed on the screen 37 and the message displayed on the screen 38. This allows the doctor 14 to use information on the type of papilla N and the merging format when inserting a cannula into the papilla N.

[0073] Although the intestinal wall image 41, papilla type information 90, and support information 86 are displayed on the screens 36 to 38 of the display device 13 in the above example, this is merely an example. The intestinal wall image 41, papilla type information 90, and support information 86 may be displayed on a single screen. Alternatively, the intestinal wall image 41, papilla type information 90, and support information 86 may be displayed on separate display devices 13.

[0074] Next, the operation of the portion of the duodenoscope system 10 related to the technology of the present disclosure will be described with reference to FIG.

[0075] Fig. 8 shows an example of the flow of medical support processing performed by the processor 82. The flow of medical support processing shown in Fig. 8 is an example of a "medical support method" according to the technique of the present disclosure.

[0076] 8 , first, in step ST10, the image acquisition unit 82A determines whether or not the user has designated a frame from the time-series image group 89 captured by the camera 48 provided on the endoscope 18. If no frame has been designated in step ST10, the determination is negative, and the determination in step ST10 is made again. If a frame has been designated in step ST10, the determination is positive, and the medical support process proceeds to step ST12.

[0077] In step ST12, the image acquisition unit 82A acquires the intestinal wall image 41 of a specified frame from the camera 48 provided in the endoscope 18. After the processing of step ST12 is executed, the medical support processing proceeds to step ST14.

[0078] In step ST14, the image recognition unit 82B performs AI image recognition processing (i.e., image recognition processing using the trained model 84B) on the intestinal wall image 41 acquired in step ST12 to detect the type of papilla N. After the processing of step ST14 is executed, the medical support processing proceeds to step ST16.

[0079] In step ST16, the support information acquisition unit 82C acquires the support information table 83 from the NVM 84. After the processing of step ST16 is executed, the medical support processing proceeds to step ST18.

[0080] In step ST18, the support information acquisition unit 82C uses the support information table 83 to acquire support information 86 corresponding to the type of nipple N. Specifically, the support information acquisition unit 82C acquires junction format information 86A and a schema 86B as the support information 86 from the support information table 83. After the processing of step ST18 is executed, the medical support processing proceeds to step ST20.

[0081] In step ST20, the display control unit 82D generates a display image 94 that displays the intestinal wall image 41, the type of papilla N indicated by the papilla type information 90, the junction type indicated by the junction type information 86A, and the schema 86B. After the processing of step ST20 is executed, the medical support processing proceeds to step ST22.

[0082] In step ST22, the display control unit 82D outputs the display image 94 generated in step ST20 to the display device 13. After the processing of step ST22 is executed, the medical support processing proceeds to step ST24.

[0083] In step ST24, the display control unit 82D determines whether a condition for terminating the medical support process is satisfied. One example of the condition for terminating the medical support process is that an instruction to terminate the medical support process has been given to the duodenoscope system 10 (for example, that an instruction to terminate the medical support process has been accepted by the acceptance device 62).

[0084] In step ST24, if the condition for terminating the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST 10. In step ST24, if the condition for terminating the medical support process is satisfied, the determination is positive, and the medical support process ends.

[0085] As described above, in the duodenoscope system 10 according to the first embodiment, the processor 82 performs image recognition processing on the intestinal wall image 41 using the image recognition unit 82B, thereby identifying the type of papilla N. The support information acquisition unit 82C then acquires support information 86 based on the papilla type information 90. The display control unit 82D outputs the papilla type information 90 and the support information 86 to an external device (e.g., the display device 13). The type of papilla N indicated by the papilla type information 90 is displayed, for example, on the display device 13 together with the intestinal wall image 41, allowing the user to understand the type of papilla N while operating the duodenoscope 12. This configuration thereby enables support for the implementation of medical care according to the type of papilla N.

[0086] Furthermore, in the duodenoscope system 10 according to the first embodiment, under the control of the display controller 82D, the type of papilla N indicated by papilla type information 90, the confluence type of the bile duct and pancreatic duct indicated by confluence type information 86A, and a schema 86B are displayed on the display device 13. The user can visually confirm the various pieces of information displayed on the display device 13 while operating the duodenoscope 12. This configuration provides visual support for the performance of medical care according to the type of papilla N.

[0087] Furthermore, in the duodenoscope system 10 according to the first embodiment, the support information 86 includes a schema 86B that is determined according to the type of papilla N. For example, the schema 86B is an image that schematically shows the confluence of the bile duct and the pancreatic duct. This allows visual support using the schema 86B to be provided as support for the performance of medical care according to the type of papilla N. Furthermore, since the support information 86 includes the schema 86B, it is possible to easily grasp information that can be used in the performance of medical care, compared to when the support information 86 is displayed only as text, for example.

[0088] Furthermore, in the duodenoscope system 10 according to the first embodiment, the support information 86 includes junction type information 86A indicating the junction type of the bile duct and the pancreatic duct. The junction type information 86A is determined according to the type of papilla N and is information capable of identifying the junction type of the bile duct and the pancreatic duct. This allows the user to recognize the junction type of the bile duct and the pancreatic duct. In an ERCP examination, a treatment instrument such as a cannula may be inserted into the bile duct or the pancreatic duct. In this case, the junction type of the bile duct and the pancreatic duct (e.g., whether they are independent ducts or a common duct) affects the success or failure of the intubation. Therefore, by allowing the user to recognize the junction type of the bile duct and the pancreatic duct, medical assistance can be realized.

[0089] Furthermore, in the duodenoscope system 10 according to the first embodiment, the image recognition unit 82B of the processor 82 performs image recognition processing on a frame-by-frame basis to identify the type of papilla N contained in the intestinal wall image 41. This allows for identification of the type of papilla N with a simpler configuration than when a portion of the intestinal wall image 41 is extracted and image recognition processing is performed on the extracted image region basis.

[0090] Second Embodiment In the first embodiment, an example was described in which the type of nipple N was identified by the image recognition processing in the image recognition unit 82B, but the technology of the present disclosure is not limited to this. In the second embodiment, as a result of the image recognition processing in the image recognition unit 82B, the type of nipple N is classified, and a confidence level for each classified type of nipple N is obtained.

[0091] As an example, as shown in FIG. 9 , the image acquisition unit 82A acquires an intestinal wall image 41 from the camera 48 provided on the endoscope 18. The image recognition unit 82B acquires the intestinal wall image 41 of a frame specified by the user. The image recognition unit 82B performs image recognition processing on the intestinal wall image 41 using a trained model 84C. By performing the image recognition processing, the type of papilla N included in the observation target 21 is classified, and a confidence level for each classified type of papilla N is output. That is, the image recognition processing includes a classification processing for classifying the type of papilla N. As described above, there are multiple types of papilla N defined based on medical findings, and the classification processing determines which of these types of papilla N the papilla N corresponds to. Then, in the classification processing, a confidence level for each type of papilla N is calculated according to the classification result of the papilla N. Here, the confidence level is a statistical measure indicating the reliability of the classification result. The confidence level is, for example, a plurality of scores (scores for each type of nipple N) input to an activation function (e.g., a softmax function) of the output layer of the trained model 84C.

[0092] The trained model 84C is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is the classification result of the papilla N (e.g., data in which the type of papilla N is annotated as multi-label).

[0093] The image recognition unit 82B inputs the intestinal wall image 41 acquired from the image acquisition unit 82A to the trained model 84C. As a result, the trained model 84C outputs certainty information 92 corresponding to the input intestinal wall image 41. The image recognition unit 82B acquires the certainty information 92 output from the trained model 84B. The certainty information 92 includes the certainty of each type of papilla N in the intestinal wall image 41 in which the papilla N appears. The certainty information 92 is an example of "certainty information" according to the technology of the present disclosure.

[0094] The support information acquisition unit 82C acquires certainty level information 92 from the image recognition unit 82B. The support information acquisition unit 82C acquires support information 86 corresponding to the type of nipple N that indicates the highest certainty level among the certainty levels indicated by the certainty level information 92. Specifically, the support information acquisition unit 82C uses the support information table 83 to acquire junction format information 86A and a schema 86B corresponding to the type of nipple N that indicates the highest certainty level.

[0095] 10 , the display control unit 82D acquires an intestinal wall image 41 from the image acquisition unit 82A. The display control unit 82D also acquires certainty factor information 92 from the image recognition unit 82B. The display control unit 82D also acquires support information 86 from the support information acquisition unit 82C. The display control unit 82D generates a display image 94 including the intestinal wall image 41, the certainty factor for each type of papilla N indicated by the certainty factor information 92, and the merging format and schema indicated by the support information 86, and causes the display device 13 to display the images 36 to 38.

[0096] In the example shown in Fig. 10, an intestinal wall image 41 is displayed on screen 36, and a schema 86B is displayed on screen 37. A message indicating the certainty of papilla N and a message indicating the merging type are displayed on screen 38. In the example shown in Fig. 10, the types and certainty of papilla N are shown as separate opening type: 70%, onion type: 20%, nodular type: 5%, and villous type: 5%. The message for separate opening type, which is the type of papilla N with the highest certainty, is displayed in a frame to distinguish it from the others.

[0097] Although the above description has been given with reference to an example in which all classification results are displayed, this is merely an example. For example, a configuration may be adopted in which only classification results with a predetermined confidence level (e.g., 30%) or higher are displayed. Furthermore, the message with the highest confidence level may be displayed in any other distinguishable manner, such as by changing the color or font. Furthermore, the message with the highest confidence level may not be displayed in a manner that is distinguishable from the others.

[0098] For example, the doctor 14 visually checks the intestinal wall image 41 displayed on the screen 36, and also checks the schema 86B displayed on the screen 37 and the message displayed on the screen 38. This allows the doctor 14 to use information on the type of papilla N and the confluence type when inserting a cannula into the papilla N.

[0099] As described above, in the duodenoscope system 10 according to the second embodiment, the image recognition unit 82B of the processor 82 performs image recognition processing. The image recognition processing includes a classification process for classifying the type of papilla N. The image recognition unit 82B then performs image recognition processing on the intestinal wall image 41 to classify the type of papilla N, and outputs certainty information 92 indicating the certainty of each classified type of papilla N. The display device 13 displays the certainty of each type of papilla N indicated by the certainty information 92. The user can grasp the type of papilla N and the certainty while operating the duodenoscope 12. This reduces the possibility of an error when the user determines the type of papilla N. In other words, compared to when only the result of identifying the type of papilla N is displayed, the user can grasp the certainty of the identified result and the possibility of other types of papilla N. This configuration thus supports the implementation of medical care according to the type of papilla N.

[0100] In the second embodiment, an example was described in which the certainty factor of the type of papilla N was displayed, but the technology of the present disclosure is not limited to this. In the third embodiment, the frequency of occurrence of the confluence type of the bile duct and the pancreatic duct is displayed together with the certainty factor.

[0101] 11 as an example, the image acquisition unit 82A acquires an intestinal wall image 41 from a camera 48 provided on the endoscope 18. The image recognition unit 82B performs image recognition processing on the intestinal wall image 41 using a trained model 84C. The image recognition unit 82B inputs the intestinal wall image 41 acquired from the image acquisition unit 82A to the trained model 84C. As a result, the trained model 84C outputs certainty factor information 92 corresponding to the input intestinal wall image 41. The image recognition unit 82B acquires the certainty factor information 92 output from the trained model 84B.

[0102] The support information acquisition unit 82C acquires certainty information 92 from the image recognition unit 82B. Using the support information table 85, the support information acquisition unit 82C acquires appearance frequency information 86C and a schema 86B corresponding to the type of nipple N with the highest certainty.

[0103] The support information table 85 is a table in which nipple type information 90, appearance frequency information 86C, and schema 86B, which correspond to one another, are associated according to the corresponding relationship. The support information table 85 is a table in which the type of nipple N indicated by the nipple type information 90 is used as input information, and the appearance frequency information 86C and schema 86B corresponding to the type of nipple N are used as output information.

[0104] In the example shown in Figure 11, the support information table 85 shows an example of an image in which the frequency of occurrence of the confluence type is 2 / 3 septate type and 1 / 3 common duct type when the type of papilla N is villous, and the schema 86B shows an example of an image in which the confluence type and the common duct type are indicated. The support information table 85 also shows an example of an image in which the frequency of occurrence of the confluence type is 2 / 3 septate type and 1 / 3 common duct type when the type of papilla N is flat, and the schema 86B shows an example of an image in which the confluence type and the common duct type are indicated. Furthermore, the support information table 85 also shows an example in which the frequency of occurrence of the confluence type is mostly septate type when the type of papilla N is nodular. Here, the villous type and the flat type are examples of the "first papilla type" according to the technology of the present disclosure.

[0105] Note that while examples of villous, flat, and nodular types are given here as input information to the support information table 85, these are merely examples. The contents of the input information and output information of the support information table 85 are determined appropriately based on medical knowledge regarding the frequency of occurrence of the type of papilla N and the confluence form. Furthermore, the output information of the support information table 85 may be only the occurrence frequency information 86C. In this case, the schema 86B has the occurrence frequency information 86C as incidental information. Then, based on the occurrence frequency information 86C acquired using the support information table 85, the support information acquisition unit 82C acquires a schema 86B having incidental information corresponding to the occurrence frequency information 86C.

[0106] Furthermore, in deriving the support information 86, a support information calculation formula (not shown) may be used instead of the support information table 85. The support information calculation formula is a calculation formula in which the type of nipple N is an independent variable and the appearance frequency information 86C and the schema 86B are dependent variables.

[0107] 12 , the display control unit 82D acquires an intestinal wall image 41 from the image acquisition unit 82A. The display control unit 82D also acquires certainty factor information 92 from the image recognition unit 82B. The display control unit 82D also acquires support information 86 from the support information acquisition unit 82C. The display control unit 82D generates a display image 94 including the intestinal wall image 41, the certainty factor for each type of papilla N indicated by the certainty factor information 92, and the appearance frequency and schema 86B of the confluence type indicated by the support information 86, and causes the display device 13 to display the images 36 to 38.

[0108] In the example shown in Fig. 12, an intestinal wall image 41 is displayed on the screen 36, and a schema 86B is displayed on the screen 37. In the example shown in Fig. 12, the schema 86B includes an image showing a septum type and an image showing a common duct type. In addition, the image showing the septum type has an appearance frequency of 2 / 3 displayed in the upper left corner, and the image showing the common duct type has an appearance frequency of 1 / 3 displayed in the upper left corner. A message indicating the certainty of papilla N and a message indicating the merging type are displayed on the screen 38. The message indicating the merging type indicates that the merging type is either the septum type or the common duct type.

[0109] For example, the doctor 14 visually checks the intestinal wall image 41 displayed on the screen 36, and also visually checks the schema 86B displayed on the screen 37 and the message displayed on the screen 38. This allows the doctor 14 to use information on the type of papilla N and the frequency of occurrence of the confluence type when inserting a cannula into the papilla N.

[0110] As described above, in the duodenoscope system 10 according to the third embodiment, the support information acquisition unit 82C of the processor 82 acquires, using the support information table 85, occurrence frequency information 86C and a schema 86B indicating the occurrence frequency of the confluence type of the bile duct and the pancreatic duct. The support information acquisition unit 82C then outputs the occurrence frequency information 86C and the schema 86B as support information 86. The occurrence frequency and the schema 86B indicated by the occurrence frequency information 86C are displayed on the display device 13. The user can grasp the type of papilla N and the occurrence frequency of the confluence type while operating the duodenoscope 12. This contributes to realizing a highly accurate judgment by the user when visually determining the type of papilla N.

[0111] Furthermore, in the duodenoscope system 10 according to the third embodiment, the occurrence frequency information 86C includes information indicating the occurrence frequency of each merging type (e.g., septum type 2 / 3 and common duct type 1 / 3). The schema 86B also shows the occurrence frequency along with an image indicating the merging type. The support information acquisition unit 82C outputs the occurrence frequency information 86C and the schema 86B, and the display device 13 displays a message indicating the occurrence frequency of the merging type of the bile duct and the pancreatic duct, along with the schema 86B. The user can grasp the type of papilla N and the occurrence frequency of the merging type while operating the duodenoscope 12. This contributes to the realization of a highly accurate judgment by the user when visually determining which of multiple merging types the papilla N is.

[0112] Furthermore, in the duodenoscope system 10 according to the third embodiment, when the type of papilla N is villous or flat, the multiple confluence types are either septum or common duct. For example, in the support information table 85, when the type of papilla N is villous, the occurrence frequency of the confluence type is 2 / 3 septum and 1 / 3 common duct, and an example of an image in which the schema 86B indicates the septum and common duct types is shown. In addition, in the support information table 85, when the type of papilla N is flat, the occurrence frequency of the confluence type is 2 / 3 septum and 1 / 3 common duct, and an example of an image in which the schema 86B indicates the septum and common duct types is shown. This contributes to realizing a highly accurate judgment by the user when visually determining whether the confluence type of a villous or flat papilla is septum or common duct.

[0113] In the third embodiment, the frequency of occurrence of the confluence of the bile duct and the pancreatic duct is displayed. However, the technology of the present disclosure is not limited to this. In this first modification, a message to assist medical treatment is displayed.

[0114] 13 , the display control unit 82D acquires an intestinal wall image 41 from the image acquisition unit 82A. The display control unit 82D also acquires confidence level information 92 from the image recognition unit 82B. The display control unit 82D also acquires support information 86 from the support information acquisition unit 82C. The support information 86 includes auxiliary information 86D. The auxiliary information 86D is information that assists in medical treatment, and the medical treatment here refers to treatment performed on the confluence of the bile duct and the pancreatic duct, which is determined depending on the type of papilla N.

[0115] For example, when the type of papilla N is villous, the occurrence frequency of the bile duct and pancreatic duct confluence type is 2 / 3 septate type and 1 / 3 common duct type. In other words, multiple confluence types of the bile duct and pancreatic duct are conceivable, making it difficult to uniquely identify them. When the confluence type differs, the procedure, such as the method of cannula insertion, must also be appropriately modified. Therefore, when multiple confluence types are identified, auxiliary information 86D is provided to assist in the medical procedure, making it easier for the user to perform the medical procedure. The auxiliary information 86D is an example of "auxiliary information" according to the technology of the present disclosure.

[0116] The auxiliary information 86D may be set as an output value of the support information table 85 (see FIG. 11 ), or may be input in advance by the user. The content of the assistance indicated by the auxiliary information 86D may include, for example, information about the insertion amount when inserting a cannula, information about the insertion method, etc.

[0117] The display control unit 82D generates a display image 94 including the intestinal wall image 41, the certainty indicated by the certainty information 92, and the auxiliary content indicated by the auxiliary information 86D, and causes the display device 13 to display the screens 36 to 38. The screen 37 displays a message indicating the auxiliary content together with the schema 86B. In the example shown in Fig. 13, the message "Please start with shallow intubation" is displayed as the auxiliary content.

[0118] As described above, in the duodenoscope system 10 according to the first modification, the support information 86 includes auxiliary information 86D, which is information for assisting in medical procedures performed for a merging type determined according to the type of papilla N. The auxiliary information 86D is output from the support information acquisition unit 82C. A message of the assistance content indicated by the support information 86 is displayed on the display device 13. While operating the duodenoscope 12, the user can grasp the type of papilla N and the assistance content available for medical procedures for the merging type. This contributes to the accurate implementation of medical procedures for the merging type determined according to the type of papilla N.

[0119] Furthermore, in the duodenoscope system 10 according to the first modification, the support information acquisition unit 82C of the processor 82 outputs auxiliary information 86D when there are multiple confluence patterns of the bile duct and the pancreatic duct corresponding to the type of papilla N. This contributes to the accurate implementation of medical procedures according to the confluence patterns, even when there are multiple confluence patterns for the type of papilla N.

[0120] (Second Modification) In each of the above embodiments, an example was given in which image recognition processing was performed on the entire intestinal wall image 41 to identify the type of papilla N, but the technology of the present disclosure is not limited to this. In this second modification, papilla detection processing is performed on the intestinal wall image 41, and then type identification processing is performed.

[0121] 14 as an example, the image acquisition unit 82A acquires an intestinal wall image 41 from a camera 48 provided on the endoscope 18. The image recognition unit 82B performs image recognition processing on the intestinal wall image 41. The image recognition processing includes a nipple detection processing that detects an area indicating a nipple N in the intestinal wall image 41, and a type identification processing that identifies the type of the nipple N. The nipple detection processing is an example of a "first image recognition processing" according to the technology of the present disclosure, and the type identification processing is an example of a "second image recognition processing" according to the technology of the present disclosure.

[0122] First, the image recognition unit 82B performs a papilla detection process on the intestinal wall image 41. The image recognition unit 82B inputs the intestinal wall image 41 acquired from the image acquisition unit 82A to the trained model for papilla detection 84D. As a result, the trained model for papilla detection 84D outputs papilla region information 93 corresponding to the input intestinal wall image 41. The papilla region information 93 is information that can identify the region indicating the papilla N in the intestinal wall image 41 (for example, the position coordinates within the image of the region indicating the papilla N). The image recognition unit 82B acquires the papilla region information 93 output from the trained model for papilla detection 84D.

[0123] The trained model for papilla detection 84D is obtained by optimizing the neural network through machine learning using training data. The training data may be, for example, training data in which multiple images (e.g., multiple images corresponding to multiple intestinal wall images 41 in time series) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum) are used as example data, and papilla region information 93 is used as correct answer data.

[0124] The image recognition unit 82B performs type identification processing on the region of the nipple N indicated by the nipple region information 93. The image recognition unit 82B inputs an image showing the nipple N identified by the nipple detection processing to the trained model for type identification 84E. As a result, the trained model for type identification 84E outputs nipple type information 90 based on the input image showing the nipple N. The image recognition unit 82B acquires the nipple type information 90 output from the trained model for type identification 84E.

[0125] The trained model 84E for type identification is obtained by optimizing the neural network through machine learning using training data. The training data is a plurality of data (i.e., a plurality of frames of data) in which example data and correct answer data are associated with each other. The example data is, for example, an image (e.g., an image corresponding to the intestinal wall image 41) obtained by imaging a region that may be the subject of an ERCP examination (e.g., the inner wall of the duodenum). The correct answer data is an annotation corresponding to the example data. An example of the correct answer data is an annotation that can identify the type of papilla N.

[0126] Although the embodiment has been described above with reference to an example in which the papilla N is detected using the trained model for papilla detection 84D and the type of papilla N is identified using the trained model for type identification 84E, the technology of the present disclosure is not limited to this. For example, a single trained model may be used to detect the papilla N and identify the type of papilla N in the intestinal wall image 41.

[0127] The support information acquisition unit 82C acquires support information 86 according to the type of papilla N. The display control unit 82D (see FIG. 7 ) generates a display image 94 including the intestinal wall image 41, the type of papilla N indicated by the papilla type information 90, and the merging format and schema 86B indicated by the support information 86, and outputs the display image 94 to the display device 13.

[0128] As described above, in the duodenoscope system 10 according to the second modification, the image recognition unit 82B of the processor 82 performs image recognition processing. The image recognition processing includes a nipple detection processing and a type identification processing. This allows the type of nipple N identified by the nipple detection processing to be identified, thereby improving the accuracy of identifying the type of nipple N compared to when the type identification processing is performed on the entire intestinal wall image 41.

[0129] In the above embodiments, examples have been described in which the papilla type information 90, support information 86, intestinal wall image 41, etc. are output to the display device 13 and displayed on the screens 36 to 38 of the display device 13, but the technology of the present disclosure is not limited to this. As an example, as shown in FIG. 15 , the papilla type information 90, support information 86, intestinal wall image 41, etc. may be output to an electronic medical record server 100. The electronic medical record server 100 is a server for storing electronic medical record information 102 that indicates the results of medical treatment for a patient. The electronic medical record information 102 includes the papilla type information 90, support information 86, intestinal wall image 41, etc.

[0130] The electronic medical record server 100 is connected to the duodenoscope system 10 via a network 104. The electronic medical record server 100 acquires intestinal wall images 41 from the duodenoscope system 10. The electronic medical record server 100 stores papilla type information 90, support information 86, intestinal wall images 41, etc. as part of the medical treatment results indicated by electronic medical record information 102. The electronic medical record server 100 is an example of an "external device" according to the technology of the present disclosure, and the electronic medical record information 102 is an example of a "medical record" according to the technology of the present disclosure.

[0131] The electronic medical record server 100 is also connected to terminals other than the duodenoscope system 10 (for example, personal computers installed in medical facilities) via a network 104. A user such as a doctor 14 can obtain the papilla type information 90, support information 86, intestinal wall images 41, etc. stored in the electronic medical record server 100 via the terminal. In this way, since the papilla type information 90, support information 86, intestinal wall images 41, etc. are stored in the electronic medical record server 100, the user can obtain the papilla type information 90, support information 86, intestinal wall images 41, etc.

[0132] In addition, in each of the above embodiments, examples have been given in which the papilla type information 90, the support information 86, the intestinal wall image 41, etc. are output to the display device 13, but the technology of the present disclosure is not limited to this. For example, the papilla type information 90, the support information 86, the intestinal wall image 41, etc. may be output to an audio output device such as a speaker (not shown), or may be output to a printing device such as a printer (not shown).

[0133] In addition, in the above-described embodiments, an example in which AI-based image recognition processing is performed on the intestinal wall image 41 has been described, but the technology of the present disclosure is not limited to this. For example, pattern matching-based image recognition processing may be performed.

[0134] In the above embodiment, an example in which medical support processing is performed by the processor 82 of the computer 76 included in the image processing device 25 has been described, but the technology of the present disclosure is not limited to this. For example, medical support processing may be performed by the processor 70 of the computer 64 included in the control device 22. Furthermore, the device that performs medical support processing may be provided external to the duodenoscope 12. Examples of devices that may be provided external to the duodenoscope 12 include at least one server and / or at least one personal computer that are communicatively connected to the duodenoscope 12. Furthermore, medical support processing may be performed in a distributed manner by multiple devices.

[0135] In the above embodiment, an example has been described in which the medical support processing program 84A is stored in the NVM 84, but the technology of the present disclosure is not limited to this. For example, the medical support processing program 84A may be stored in a portable non-transitory storage medium such as an SSD or USB memory. The medical support processing program 84A stored in the non-transitory storage medium is installed in the computer 76 of the duodenoscope 12. The processor 82 executes medical support processing in accordance with the medical support processing program 84A.

[0136] Alternatively, the medical support processing program 84A may be stored in a storage device such as another computer or server connected to the duodenoscope 12 via a network, and the medical support processing program 84A may be downloaded and installed in the computer 76 in response to a request from the duodenoscope 12.

[0137] It is not necessary to store the entire medical support processing program 84A in the storage device of another computer or server device connected to the duodenoscope 12, or in the NVM 84; only a portion of the medical support processing program 84A may be stored therein.

[0138] The hardware resources that execute the medical support processing can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes medical support processing by executing software, i.e., a program. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with a circuit configuration specifically designed to execute specific processing. Each processor has built-in or connected memory, and executes medical support processing by using the memory.

[0139] The hardware resource for executing the medical support processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the medical support processing may be a single processor.

[0140] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute medical support processing, on a single IC chip, as typified by SoC. In this way, medical support processing is realized using one or more of the various processors described above as hardware resources.

[0141] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0142] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0143] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0144] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0145] The disclosure of Japanese Patent Application No. 2022-177612, filed on November 4, 2022, is incorporated herein by reference in its entirety.

Claims

1. A medical support device comprising a processor, wherein the processor performs image recognition processing on an intestinal wall image obtained by imaging an intestinal wall including the major duodenal papilla in the duodenum by a camera provided in an endoscope scope, to identify the papilla type, which is the type of the major duodenal papilla, and outputs related information associated with the papilla type, the related information being information corresponding to the identified papilla type.

2. Outputting the related information is displaying the related information on a screen. The medical support device according to Claim 1.

3. The related information includes a schema defined according to the papilla type. The medical support device according to Claim 1.

4. The related information includes confluence format information, wherein the confluence format information is information that is determined according to the papilla type and that can identify a confluence format in which the bile duct and the pancreatic duct merge. The medical support device according to Claim 1.

5. The image recognition processing includes classification processing for classifying the papilla type, and the related information includes confidence information indicating a confidence level for each papilla type classified by the classification processing. The medical support device according to Claim 1.

6. For each papilla type, an appearance frequency of a confluence format in which the bile duct and the pancreatic duct merge is defined, and the processor outputs, as the related information, information including appearance frequency information indicating the appearance frequency corresponding to the identified papilla type. The medical support device according to Claim 1.

7. The papilla type includes a first papilla type, wherein the first papilla type has any one of a plurality of confluence formats in which the bile duct and the pancreatic duct merge, and when the processor identifies the first papilla type as the papilla type, the processor outputs, as the related information, information including appearance frequency information indicating the appearance frequency for each confluence format. The medical support device according to Claim 1.

8. The first papilla type is a villous type or a flat type, and the plurality of confluence formats are a septate type and a common channel type. The medical support device according to Claim 7.

9. The related information includes auxiliary information, wherein the auxiliary information is information that assists in a medical treatment performed on a confluence format in which the bile duct and the pancreatic duct merge and that is determined according to the papilla type. The medical support device according to Claim 1.

10. When there are a plurality of the confluence formats for the identified papilla type, the processor outputs the auxiliary information. The medical support device according to Claim 9.

11. The processor identifies the papilla type by performing the image recognition process on the intestinal wall image in units of frames. The medical support device according to claim 1.

12. The image recognition process includes a first image recognition process and a second image recognition process. The processor detects the duodenal papilla region by performing the first image recognition process on the intestinal wall image, and identifies the papilla type by performing the second image recognition process on the detected duodenal papilla region. The medical support device according to claim 1.

13. The related information is stored in an external device and / or a medical record. The medical support device according to claim 1.

14. A medical support device according to any one of claims 1 to 13, and the endoscope scope. Endoscope.

15. Identifying the papilla type, which is the type of the duodenal papilla, by performing an image recognition process on an intestinal wall image obtained by imaging an intestinal wall including the duodenal papilla in the duodenum with a camera provided in the endoscope scope, and outputting related information related to the papilla type, which is related information corresponding to the identified papilla type. Medical support method.

16. On a computer, identifying the papilla type, which is the type of the duodenal papilla, by performing an image recognition process on an intestinal wall image obtained by imaging an intestinal wall including the duodenal papilla in the duodenum with a camera provided in the endoscope scope, and a program for executing a process including outputting related information related to the papilla type, which is related information corresponding to the identified papilla type.

Images