Processing system, endoscope system, and operation method of image processing system

Abstract

To provide a processing system, etc. capable of improving convenience for treatment of a biliary tract.SOLUTION: A processing system 3 includes a processor 10 including hardware. The processor 10 acquires an endoscope image capturing the duodenal papilla including an oral protrusion, estimates a depth map of a region including the duodenal papilla from the endoscope image, estimates an apex line of the oral protrusion from the estimated depth map, and executes processing to display a guide display GM based on the estimated apex line by superimposing it on the endoscope image. A user can grasp the apex line of the oral protrusion by viewing the guide display GM.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a processing system, an endoscope system, and a processing method. [Background technology]

[0002] There is a known technique for using an endoscope to treat bile duct cancer, etc. Patent Document 1 discloses a technique for estimating the three-dimensional shape of the duodenal papilla using an endoscope capable of stereoscopic vision. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2001-167272 A Summary of the Invention [Problem to be solved by the invention]

[0004] Regarding treatment of the bile duct, there is a demand for further improvement in convenience, such as support for determining in which direction an incision should be made, etc. Patent Document 1 does not disclose a method for displaying an incision guide. [Means for solving the problem]

[0005] One aspect of the present disclosure relates to a processing system that includes a processor including hardware, which acquires an endoscopic image of the duodenal papilla including an oral ridge, estimates a depth map of a region including the duodenal papilla from the endoscopic image, estimates a ridge line of the oral ridge from the estimated depth map, and displays a guide display based on the estimated ridge line superimposed on the endoscopic image.

[0006] Another aspect of the present disclosure relates to an endoscope system including the above-described processing system and an endoscope.

[0007] Another aspect of the present disclosure relates to a processing method that performs processing of acquiring an endoscopic image of the duodenal papilla including an oral ridge, estimating a depth map of a region including the duodenal papilla from the endoscopic image, estimating a ridge line of the oral ridge from the estimated depth map, and displaying a guide display based on the estimated ridge line superimposed on the endoscopic image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram for explaining an example of the configuration of a processing system, etc. [Diagram 2] FIG. 1 shows organs and tissues involved in EST and other procedures. [Diagram 3] Schematic diagram of the duodenal papilla. [Figure 4] FIG. 2 is another diagram illustrating the duodenal papilla. [Diagram 5] FIG. 1 is a diagram for explaining the duodenal papilla and other parts when performing EST. [Figure 6] A conceptual diagram explaining the tip of the EST knife. [Figure 7] A diagram explaining the direction of incision in EST. [Figure 8] 4 is a flowchart illustrating a processing example of the present embodiment. [Figure 9] 11 is a flowchart illustrating an example of a top line estimation process. [Figure 10] FIG. 4 is a diagram for explaining an example of the arrangement of a processing unit. [Figure 11] FIG. 13 is a diagram illustrating another example of the configuration of the processing system. [Figure 12] FIG. 13 is a diagram for explaining estimation of a summit line. [Figure 13] 5A to 5C are diagrams for explaining the effects of the method according to the present embodiment. [Figure 14] FIG. 13 is a diagram for explaining another example of the configuration of a processing system, etc. [Figure 15] 10 is a flowchart illustrating an example of processing performed by a pressure determination unit. [Figure 16] FIG. 13 is a diagram for explaining another example of the configuration of the processing unit. [Figure 17]10 is a flowchart for explaining a processing example of an endoscopic image acquisition process. [Figure 18] 10 is a flowchart illustrating another example of the top line estimation process. [Figure 19] 4A and 4B are diagrams for explaining a first depth map and a second depth map. [Figure 20] FIG. 13 is a diagram for explaining another example of the configuration of the processing unit. [Figure 21] 10 is a flowchart illustrating another example of the top line estimation process. [Figure 22] FIG. 13 is a diagram for explaining another example of the configuration of the processing unit. [Figure 23] 10 is a flowchart illustrating another example of the top line estimation process. [Figure 24] 5A to 5C are diagrams for explaining generation of point cloud data. [Diagram 25] 5A to 5C are views for explaining first point cloud data and second point cloud data. [Figure 26] FIG. 13 is a diagram for explaining another example of the configuration of the processing unit. [Figure 27] 10 is a flowchart illustrating another example of the top line estimation process. [Figure 28] FIG. 1 is a diagram explaining the apex line starting from the nipple opening. [Figure 29] FIG. 13 is a diagram for explaining another example of the configuration of the processing unit. [Diagram 30] 10 is a flowchart illustrating another example of the top line estimation process. [Diagram 31] 13A and 13B are diagrams illustrating a vertex line starting from one end of the treatment tool. [Diagram 32] FIG. 13 is a diagram for explaining another example of the configuration of the processing unit. [Diagram 33] 10 is a flowchart illustrating another example of the top line estimation process. [Diagram 34] FIG. [Diagram 35] FIG. 13 is a diagram for explaining an example in which the structure of an organ is changed. [Diagram 36] 1A and 1B are diagrams for explaining the relationship between the structure of an organ and the display of an incision area. [Figure 37] FIG. 13 is a diagram for explaining another example of the configuration of the processing unit. [Figure 38] 11 is a flowchart illustrating an example of processing performed by an incision length estimation unit. [Figure 39] 13A to 13C are diagrams for explaining the effects of the processing performed by the incision length estimation unit. [Diagram 40] FIG. 13 is a diagram illustrating another example of the configuration of the incision length estimation unit. [Diagram 41] 13 is a flowchart illustrating another example of processing performed by the incision length estimation unit. [Diagram 42] FIG. 13 is a diagram illustrating another example of the configuration of the incision length estimation unit. [Diagram 43] 13 is a flowchart illustrating another example of processing performed by the incision length estimation unit. [Diagram 44] FIG. 13 is a diagram for explaining another example of the configuration of the processing unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A preferred embodiment of the present disclosure will be described in detail below. Note that the embodiment described below does not unduly limit the contents described in the claims, and all of the configurations described in the embodiment are not necessarily essential configurations.

[0010] FIG. 1 is a block diagram for explaining a configuration example of an endoscope system 1 of the present embodiment and a processing system 3 included in the endoscope system 1. The processing system 3 of the present embodiment includes a processor 10. The processor 10 of the present embodiment is configured with the following hardware. The hardware may include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware may be configured with one or more circuit devices and one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, ICs, etc. The one or more circuit elements are, for example, resistors, capacitors, etc.

[0011] For example, the processing system 3 of the present embodiment may include a memory 20 (not shown in FIG. 1) and a processor 10 that operates based on information stored in the memory 20. This allows the processor 10 to function as a display control unit 50, a processing unit 100, or the like. The information is, for example, a program and various data. The program may include, for example, a trained model 22 described later in FIG. 11. The processor 10 may use a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or the like. The memory 20 may be a semiconductor memory such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), a register, a magnetic storage device such as a hard disk device, or an optical storage device such as an optical disk device. For example, the memory 20 stores computer-readable instructions, and the functions of each unit are realized as processing by the processor 10 executing the instructions. The instructions here may be instructions of an instruction set constituting a program, or instructions that instruct a hardware circuit of the processor 10 to operate. The memory 20 is also called a storage device. For convenience of explanation, the subject of the processing related to the method of the present embodiment is unified as the processor 10 unless otherwise specified, but it can be appropriately read as the processing unit 100 as software. In addition, in FIG. 1, the processing system 3 is illustrated as including one processor 10, but this does not prevent the method of the present embodiment from being realized by multiple processors 10. That is, the display control unit 50 and the processing unit 100 may be configured by different processors 10. The same applies to each unit included in the processing unit 100. For example, the depth estimation unit 110 and the top line estimation unit 120 described later in FIG. 10 may be realized by different processors 10. The same applies to FIG. 14, FIG. 16, FIG. 20, FIG. 22, FIG. 26, FIG. 29, FIG. 32, FIG. 37, FIG. 40, FIG. 42, and FIG. 44.

[0012] The display control unit 50 receives an image signal from an imager (not shown) provided at the tip of the endoscope 5, generates a display image from the image signal, and displays it on a display device 9 (not shown in FIG. 1). In this embodiment, an image captured by an imager (not shown) provided at the tip of the endoscope 5 is called an endoscopic image. The endoscopic image may be a still image of a video captured by the camera. In this embodiment, as described later in FIG. 2, the imaging device captures an image of the duodenal papilla including the oral ridge. That is, the processor 10 of this embodiment functions as the display control unit 50 and acquires an endoscopic image of the duodenal papilla including the oral ridge. The duodenal papilla is a part of the duodenum that includes the papilla opening and its surrounding structures, as will be described later in detail. The surrounding structures are, for example, an oral ridge, a headband fold, an annular fold, a frenulum, etc. In this embodiment, an endoscopic image of the duodenal papilla including the oral ridge is an endoscopic image in which at least the oral ridge of the above-mentioned surrounding structures is shown.

[0013] The processing unit 100 controls each unit of the processing system 3. Although the details will be described later, the processing unit 100 functions as each unit that executes the process related to the method of this embodiment. For example, the processing unit 100 receives display image data generated by the display control unit 50, generates various data by each unit included in the processing unit 100, and transmits the generated various data to the display control unit 50. As a result, the display control unit 50 controls the display device 9 based on the image data captured by the imager and the various data received from the processing unit 100. As a result, the endoscopic image and the guide display GM described later are superimposed and displayed on the display device 9. The units included in the processing unit 100 are, for example, a vertex line estimation unit 120, an incision area generation unit 160, an incision length estimation unit 170, etc., which will be described later.

[0014] The endoscope 5 is, for example, a medical flexible endoscope. As described below, this embodiment relates to a method of displaying a guide display GM when performing EST after ERCP. Although not shown in detail, the endoscope 5 in this case is mainly a side-viewing endoscope in which an objective lens of an imager, an illumination lens, and an opening of a treatment tool channel are provided on the side of the tip of the endoscope. Note that ERCP is an abbreviation for Endoscopic Retrograde Cholangio Pancreatography, and EST is an abbreviation for Endoscopic Sphincterotomy.

[0015] FIG. 2 shows the organs and tissues related to the above-mentioned procedure. An organ is a unique structure formed by a collection of multiple types of tissues and has a specific function. For example, in FIG. 2, the liver, gallbladder, pancreas, esophagus, stomach, and duodenum correspond to the organs. A tissue is formed by the combination of related cells, such as blood vessels, muscles, and skin. For example, in FIG. 2, the bile duct and pancreatic duct correspond to the tissues. The example of the organ structure shown in FIG. 2 shows a general example that has not been changed by surgical treatment, etc., which will be described later in FIG. 35. The structure of an organ can also be said to be the anatomical structure of the digestive tract. The digestive tract is the stomach and intestines. The stomach here includes the remnant stomach when a gastrectomy is performed. The intestines include the small intestine, large intestine, etc. The small intestine includes the duodenum, jejunum, and ileum. In the case of the organ structure shown in FIG. 2, the tip of the endoscope is inserted from the stomach side toward the duodenum in the ERCP procedure.

[0016] Then, the insertion section is inserted to a position where the papilla is roughly captured on the imager of the endoscope 5, and the endoscope 5 is aligned with respect to the duodenal papilla. Specifically, the position of the endoscope tip is adjusted so that, for example, the objective lens of the imager of the endoscope 5 faces the duodenal papilla directly and the duodenal papilla is located in the center of the imaging area of ​​the imager.

[0017] FIG. 3 is a diagram showing a typical shape of the duodenal papilla when viewed from the front. As shown in FIG. 3, structures called oral protuberance, bandage fold, annular fold, and frenulum exist around the papilla opening. The oral protuberance is a protuberance extending from the papilla opening in a ridge-like shape. In FIG. 3, directions DR11, DR12, and DR13 are shown as mutually orthogonal directions for convenience of explanation. In this embodiment, the side toward which the direction DR12 faces may be called the "upper side". In addition, the right side of the duodenal papilla shown in FIG. 3 is shown in FIG. 4 as a schematic diagram. In addition, in the following explanation, the side toward which the direction DR13 faces may be called the "rear side". On the other hand, the side opposite to the side toward which the direction DR13 faces, that is, the objective lens side of the imager of the endoscope 5, may be called the "near side".

[0018] In ERCP, a cannula is inserted into a treatment tool channel (not shown) of the endoscope 5, the cannula is projected from the channel opening at the endoscope tip, and the tip of the cannula is inserted into the bile duct through the papilla opening. A cannula is a medical tube that is inserted into the body and used for medical purposes. A contrast agent is injected into the cannula, and the contrast agent flows into the bile duct from the cannula tip. In this state, an X-ray image or a CT (Computed Tomography) image showing the bile duct, the gallbladder, and the pancreatic duct is obtained by performing X-ray photography or CT photography. After that, a guide wire is inserted into the cannula, the guide wire is projected from the cannula tip, and the guide wire is inserted into the bile duct. Then, the cannula is removed while the guide wire is left in the bile duct. As a result, as shown in FIG. 5, only the guide wire is projected from the endoscope tip (not shown) and left in the bile duct. Although not shown, the guide wire passes through the treatment tool channel of the endoscope 5 and extends to the outside of the treatment tool insertion port. This allows various endoscopic treatment tools and the like to be inserted through the treatment tool insertion port and passed through the endoscopic treatment tools to the bile duct.

[0019] In the EST procedure, for example, an EST knife is inserted into the bile duct along the indwelling guide wire. FIG. 6 is a conceptual diagram of the tip of the EST knife of this embodiment. In this embodiment, a wide variety of techniques related to known EST knives can be applied, and FIG. 6 is merely an example. The EST knife is configured, for example, to have multiple lumens in an insulating tube. For example, the lumen shown in A1 is provided for passing the above-mentioned guide wire. A knife wire passes through the lumen shown in A2, and when tension is not applied to the knife wire, the knife wire is stored in the insulating tube. A handle (not shown) is provided on the treatment tool insertion port side of the EST knife, and tension is applied to the knife wire by pulling the handle toward the treatment tool insertion port, and the knife wire is exposed from the opening of the insulating tube as shown in A3. Therefore, for example, when it is desired to incise and expand the nipple opening, the state shown in A3 is created, and high-frequency current is passed through the knife wire, so that the tissue around the nipple opening is burned and incised. Note that the lumen shown in A4 can be used, for example, for injecting a contrast agent. Furthermore, the tip surface shown in A5 is merely a conceptual representation of the fact that the EST knife has multiple lumens and that the knife wire does not protrude from the tip of the EST knife, and is not necessarily the same as the tip surface of an actual EST knife.

[0020] An example of a procedure using an EST knife is the removal of gallstones. For example, although not shown, when removing gallstones in the bile duct, a basket treatment tool passed through a guide wire is inserted into the bile duct, and the basket treatment tool is removed while the gallstone is captured in the basket. If the gallstone is large, there is a method of crushing the gallstone, but the method of incising and expanding the papilla opening and removing the gallstone is preferable in terms of shortening the procedure time, so a procedure using an EST knife is adopted in many cases. This is because the shorter the procedure time, the lower the possibility of complications occurring.

[0021] When incising the papilla opening with an EST knife, it is extremely important to make the incision in a direction along the bile duct. Incising in a direction other than along the bile duct may cause complications such as arterial damage and retroperitoneal perforation, and these must be avoided. However, as is clear from FIG. 5, the bile duct is located on the back side of the papilla opening, so the user cannot directly recognize the path of the bile duct from the endoscopic image. More specifically, for example, if the direction indicated by the dotted line area A10 in FIG. 7 is along the bile duct, the area is the direction to be incised with EST, but it is difficult to recognize the area indicated by A10 using only the endoscopic image as a clue. Therefore, in this embodiment, the direction to be incised with the EST knife is estimated by the method described below, and a guide display GM that guides the estimated direction is displayed.

[0022] An example of processing according to the method of this embodiment will be described with reference to the flow chart of FIG. 8. First, a process of optimizing the intraluminal pressure (step S100) is performed. For example, a state in which the duodenal lumen is contracted prevents clear imaging of the duodenal papilla by the imager of the endoscope 5 and incision treatment by EST, and so the intraluminal pressure is increased to expand the duodenal lumen. On the other hand, since the intestinal wall of the duodenum is thin, it is also necessary to prevent the duodenal lumen from being excessively expanded. Step S100 may be performed, for example, by a user manually operating the gas supply / degassing device 7 described later while observing an endoscopic image, or may be performed automatically by the processing system 3 as described later. Although the flow chart is omitted, even after the intraluminal pressure is optimized by step S100, the user periodically checks from the endoscopic image whether the intraluminal pressure is within a desired range.

[0023] Then, the processor 10 performs an endoscopic image acquisition process (step S200). For example, as described above, the processor 10 functions as the display control unit 50 and acquires image data of an endoscopic image captured by an imager at the tip of the endoscope 5. Then, the processor 10 performs a crest line estimation process (step S300) described later in FIG. 9. Then, the processor 10 performs a process of displaying a guide display GM based on the crest line estimated in step S300 superimposed on the endoscopic image (step S400). The crest line refers to a linear area including the part located at the most anterior side of the oral ridge, and the crest line may have a certain width. Alternatively, the linear area corresponding to the ridge of the oral ridge may be considered as the crest line. It is empirically known that the crest line corresponds to the path of the bile duct, but it is difficult to directly grasp the crest line from an endoscopic image. As described above, the duodenal papilla has a hachimaki fold and a ring fold, but these structures vary greatly from person to person and are considered to have no correlation with the crest line.

[0024] 9 is a more detailed flowchart of the apex line estimation process (step S300). The processor 10 performs a process (step S302) of estimating a depth map of the oral ridge and its surroundings from the endoscopic image. The depth map refers to information in which the depth of the subject at each point on the map is assigned to each point. In the depth map of this embodiment, for example, the depth of the subject at each pixel is assigned to each pixel of the endoscopic image.

[0025] The processor 10 then performs a process (step S304) of estimating the apex line of the oral ridge from the depth map. In the following description or illustrations, estimating the depth map of the oral ridge and its surroundings from the endoscopic image may be simply referred to as estimating the depth map from the endoscopic image. In the following description or illustrations, estimating the apex line of the oral ridge from the depth map may be simply referred to as estimating the apex line from the depth map.

[0026] The process shown in the flowchart of Fig. 9 can be realized, for example, by configuring the processing unit 100 as shown in Fig. 10. In Fig. 10, the processing unit 100 includes a depth estimation unit 110 and a top line estimation unit 120. That is, the processor 10 executes step S302 in Fig. 9 by functioning as the depth estimation unit 110 in Fig. 10, and executes step S304 in Fig. 9 by functioning as the top line estimation unit 120 in Fig. 10.

[0027] More specifically, for example, by configuring the processing system 3 as in the configuration example shown in Fig. 11, the processor 10 can function as a depth estimation unit 110. In Fig. 11, the processing system 3 further includes a memory 20, an input unit 30, and an output unit 40. A trained model 22 is stored in the memory 20.

[0028] The input unit 30 is an interface that receives data from the outside. The output unit 40 is an interface that transmits data inferred from an inference stage described later to the outside. The specific hardware of the input unit 30 and the output unit 40 is appropriately determined according to the function performed by the processor 10. In addition, when one processor 10 performs different types of inference, data output from the output unit 40 by one inference may be input again to the input unit 30, and the processor 10 may perform another inference based on the re-input data. For example, the depth estimation unit 110 and the apex line estimation unit 120 described later in FIG. 10 may function by the same processor 10. In this case, the depth map output from the output unit 40 by the processor 10 functioning as the depth estimation unit 110 is input again to the input unit 30, the processor 10 functions as the apex line estimation unit 120, and the apex line annotation data is output from the output unit 40. Note that in the following description of the process using the trained model 22, illustration and description of the input unit 30 and the output unit 40 are omitted for convenience.

[0029] The trained model 22 is a program module generated by performing machine learning as supervised learning. The trained model 22 is generated by supervised learning based on a data set in which input data and correct labels are associated. More specifically, for example, the trained model 22 is generated by a learning device (not shown) in the learning stage. The learning device stores an untrained model in which a weighting factor described below is set as an initial value in a storage device (not shown). Then, training data as a data set in which input data and correct labels are associated with each other is input to the untrained model, and the weighting factor is optimized by providing feedback to the untrained model based on the inference result, and the trained model 22 is generated.

[0030] 11 illustrates one trained model 22 stored in the memory 20, the memory 20 may store a plurality of trained models 22. For example, when the processor 10 functions as the depth estimation unit 110 and also functions as a pressure determination unit 60 described later, trained models 22 corresponding to each function are stored in the memory 20, and a trained model 22 corresponding to the processing to be executed is appropriately selected.

[0031] In the trained model 22 of this embodiment, at least a part of the model includes a neural network. Although not shown, the neural network has an input layer to which data is input, an intermediate layer that performs calculations based on the output from the input layer, and an output layer that outputs data based on the output from the intermediate layer. The number of intermediate layers is not particularly limited. The number of nodes included in each intermediate layer is also not particularly limited. A node included in a given layer in the intermediate layer is connected to a node in an adjacent layer. A weighting factor is set for each connection. Each node multiplies the output of the previous node by the weighting factor to obtain a sum of the multiplication results. Furthermore, each node adds a bias to the sum and applies an activation function to the sum to obtain the output of the node. This process is performed sequentially from the input layer to the output layer to obtain the output of the neural network. Note that various functions such as a sigmoid function and a ReLU function are known as activation functions, and these can be widely applied in this embodiment.

[0032] The trained model 22 may be generated by performing additional learning on an existing trained model. For the additional learning, a method such as fine tuning is used. Specifically, for example, the weight coefficient is optimized by reconstructing the existing trained model, such as by adding an output layer, and performing machine learning using the training data described above.

[0033] The trained model 22 read out when functioning as the depth estimation unit 110 is trained as a data set in which, for example, an endoscopic image is used as input data and a depth map is used as a correct label. In the inference stage, an endoscopic image is input to the input unit 30, and the processor 10 reads out the corresponding trained model 22 from the memory 20 and inputs the endoscopic image to the trained model 22. As a result, the inferred depth map is output from the trained model 22 via the output unit 40.

[0034] A monocular depth estimation model that is publicly available as an open source may be used as the trained model 22 for functioning as the depth estimation unit 110. In this way, since there is no need to provide the above-mentioned learning stage, the burden of preparing training data can be reduced. The burden of preparing training data is, for example, the burden of preparing an endoscope 5 equipped with a depth sensor or the like in order to create a depth map of the duodenal papilla as a correct label. The monocular depth estimation model described above may be modified, such as by performing the above-mentioned additional learning.

[0035] In this way, the processor 10 functions as the depth estimation unit 110 to perform step S302 in FIG. 9. For example, when an endoscopic image of the area shown in A21 in FIG. 12 is input, a depth map shown in A22 is output. A depth value is included for each pixel constituting the depth map shown in A22. Note that the depth map of this embodiment is displayed in gradations consisting of 4 to 5 gradations for convenience, but may be displayed in more gradations. Also, in order to make the gist of the invention easier to understand, the depth map shown in A22 is expressed excluding headband folds, annular folds, and unevenness due to the frenulum. The same applies to the depth map described later. Also, for convenience of explanation, A21 shows the duodenal papilla without a treatment tool inserted therein, but it may be an endoscopic image of the duodenal papilla at a stage where the above-mentioned guide wire is placed in the bile duct, or an endoscopic image of the duodenal papilla at a stage where an EST treatment tool is further inserted, and is determined appropriately.

[0036] 9 is performed by the processor 10 reading out the corresponding trained model 22, and functions as the apex line estimation unit 120. In this case, the trained model 22 is machine-learned as a data set in which the depth map output from the depth estimation unit 110 is used as input data and the apex line annotation data is used as a correct answer label. Note that, in order to improve accuracy in the inference stage, an endoscopic image corresponding to the depth map may be further added as input data.

[0037] 9 can be realized by a method that does not use machine learning. For example, the depth map output from the depth estimation unit 110 is used as input data, the average depth value is calculated for every predetermined number of pixels, and the apex line is estimated based on the calculated value.

[0038] In this way, the processor 10 functions as the summit line estimation unit 120 to estimate the area indicated by A24 as the summit line based on the area indicated by A23 in Fig. 12, for example. The area indicated by A23 is an area consisting of gradations indicating the near side in the depth map. The processor 10 then generates image data of the guide display GM based on the estimated summit line, and outputs the image data of the generated guide display GM.

[0039] By performing such a top line estimation process (step S300), an endoscopic image as shown in A31 of Fig. 13 is displayed on the display device 9 described below in step S400. A guide display GM is superimposed on the duodenal papilla in the endoscopic image shown in A31.

[0040] From the above, the processing system 3 of this embodiment includes a processor 10 including hardware. The processor 10 acquires an endoscopic image of the duodenal papilla including the oral ridge, estimates a depth map of the area including the duodenal papilla from the endoscopic image, estimates the apex line of the oral ridge from the estimated depth map, and performs processing to display a guide display GM based on the estimated apex line superimposed on the endoscopic image. The path of the bile duct cannot be directly grasped from an endoscopic image in which the duodenal papilla is captured. In addition, although it is known that the path of the bile duct is correlated with the shape of the oral ridge, since the endoscopic image is a two-dimensional image that does not have information about the depth direction, it is difficult to directly grasp the shape of the oral ridge from an endoscopic image in which the duodenal papilla is captured. In this regard, the processing system 3 of this embodiment estimates a depth map from an endoscopic image, and therefore can estimate the shape of the oral ridge based on the depth map. In addition, the apex line of the oral ridge is estimated from the depth map, and the guide display GM based on the estimated apex line is displayed superimposed on the endoscopic image, so that the user can grasp the apex line of the oral ridge by looking at the guide display GM. This improves the convenience of treatment for the bile duct.

[0041] Also, for example, a method of mounting an imaging device for stereoscopic vision on the tip of an endoscope has been proposed, but actually mounting such an imaging device on the tip of an endoscope is difficult due to the problem of increasing the size of the tip of the endoscope, etc. In this regard, by applying the method of the present embodiment, a depth map is estimated from a normal endoscopic image, and the apex line of the oral protuberance is estimated, so that the path of the bile duct in the duodenal papilla can be estimated without mounting an imaging device for stereoscopic vision.

[0042] Moreover, the method of the present embodiment may be realized as an endoscope system 1. That is, the endoscope system 1 of the present embodiment includes the above-mentioned processing system 3 and endoscope 5. By doing so, it is possible to obtain the same effects as those described above.

[0043] The technique of this embodiment may be realized as a processing method. That is, the processing method of this embodiment performs a process of acquiring an endoscopic image of the duodenal papilla including the oral bulge (step S200), a process of estimating a depth map of a region including the duodenal papilla from the endoscopic image (step S302), and a process of estimating a crest line of the oral bulge from the estimated depth map (step S304). The processing method of this embodiment further performs a process of displaying a guide based on the estimated crest line superimposed on the endoscopic image (step S400). In this way, the same effect as above can be obtained.

[0044] Furthermore, in the processing system 3 of this embodiment, the guide display GM may be an incision guide display in EST. The same applies when the technique of this embodiment is realized as a processing method. That is, in the processing method of this embodiment, the guide display GM may be an incision guide display in EST. In this way, the user can perform EST while estimating the apex line of the oral protuberance. Specifically, for example, the user can perform an incision in a desired direction as shown in A32 of FIG. 13 by moving the knife wire along the guide display GM while energizing the knife wire of the EST knife.

[0045] Furthermore, the processing system 3 of this embodiment may include a memory 20 that stores a trained model 22 that has been trained to estimate a depth map from an input image. Furthermore, the processor 10 may estimate a depth map by inputting an endoscopic image to the trained model 22. In this manner, a processing system 3 that estimates a depth map using the trained model 22 that has been machine-learned can be constructed.

[0046] The method of this embodiment is not limited to the above, and various modifications such as adding features are possible. For example, the processing system 3 of this embodiment may automatically perform step S100 of FIG. 8. Specifically, for example, the processing system 3 may be configured as shown in the configuration example of FIG. 14, and the processing example shown in the flowchart of FIG. 15 may be incorporated into step S100 of FIG. 8, thereby realizing automation of the processing corresponding to step S100. In addition, after step S200 and subsequent steps of FIG. 8 are performed, the processing of FIG. 15 may be periodically performed, for example, by timer interrupt processing. In this way, an intraluminal pressure suitable for performing the method of this embodiment can be maintained.

[0047] 14 is a block diagram showing the endoscope system 1 of the present embodiment in more detail. In addition to the processing system 3 and endoscope 5 described above, the endoscope system 1 further includes an air supply / deaeration device 7 and a display device 9. The processor 10 included in the processing system 3 further includes a pressure determination unit 60 in addition to the display control unit 50 and processing unit 100 described above.

[0048] The pressure determination unit 60 determines the pressure in the duodenal lumen based on the endoscopic image received from the endoscope 5 via the display control unit 50, and controls the gas supply / degassing device 7 based on the result of the determination. The gas supply / degassing device 7 is connected to the endoscope tip via an internal channel of the insertion part (not shown). This allows gas to be supplied to or evacuated from the duodenal lumen via the endoscope tip. The gas supply / degassing device 7 is a known device, and detailed description thereof will be omitted.

[0049] A processing example of the flowchart of FIG. 15 will be described. The processor 10 estimates the pressure in the duodenal lumen from the endoscopic image (step S102), and performs a process of determining whether the estimated pressure in the duodenal lumen is higher than a predetermined range (step S110). A means for implementing step S102 will be described later. The predetermined range is appropriately determined by the user based on past cases, etc. If the estimated pressure in the duodenal lumen is higher than the predetermined range (YES in step S110), the processor 10 performs a process of degassing the duodenal lumen (step S112) and ends the flow. This makes it possible to further reduce the pressure in the duodenal lumen. On the other hand, if the processor 10 determines that the estimated pressure in the duodenal lumen is not higher than the predetermined range (NO in step S110), it performs a process of determining whether the estimated pressure is lower than the predetermined range (step S120). If the processor 10 determines that the estimated pressure is lower than the predetermined range (YES in step S120), it performs a process of supplying air to the duodenal lumen (step S122) and ends the flow. This allows the pressure in the duodenal lumen to be increased. On the other hand, if the processor 10 determines that the estimated pressure is not lower than the predetermined range (NO in step S120), it ends the flow. In other words, if NO in step S110 and NO in step S120, it is determined that the estimated pressure is within the predetermined range and there is no need to supply air or evacuate.

[0050] The trained model 22 read out when the processor 10 is made to function as the pressure determination unit 60 is machine-trained using a data set in which, for example, an endoscopic image of the inside of the duodenal lumen, feature quantities such as the number of wrinkles in the duodenal lumen and the depth of the wrinkles in the duodenal lumen for the endoscopic image, and a pressure determination result in the duodenal lumen are associated with each other. The pressure determination result in the duodenal lumen is a judgment result classified, for example, as "the pressure in the duodenal lumen is within a predetermined range," "the pressure in the duodenal lumen is higher than the predetermined range," "the pressure in the duodenal lumen is lower than the predetermined range," and the like, and serves as a correct answer label.

[0051] Then, in the inference stage, an endoscopic image of the inside of the duodenal lumen is input to the input unit 30 as input data. Then, the processor 10 reads out the trained model 22 from the memory 20. As a result, the processor 10 functions as a pressure determination unit 60, and performs a process of extracting feature quantities such as the number of wrinkles and the depth of the wrinkles from the input endoscopic image, a process of selecting a correct label corresponding to the extracted feature quantities, and a process of outputting the selected correct label as output data via the output unit 40. This realizes step S102 in FIG. 15. Then, the processes from step S102 onwards are performed according to the contents of the selected correct label.

[0052] Thus, in the processing system 3 of this embodiment, the processor 10 estimates whether the pressure in the duodenal lumen is within the appropriate pressure range based on the endoscopic image (step S102), and if the pressure in the duodenal lumen is outside the appropriate pressure range, performs a process of supplying or degassing the pressure in the duodenal lumen to within the appropriate pressure range (steps S112, S122). The same applies when the technique of this embodiment is realized as a processing method. That is, the processing method of this embodiment further performs a process of estimating whether the pressure in the duodenal lumen is within the appropriate pressure range based on the endoscopic image, and if the pressure in the duodenal lumen is outside the appropriate pressure range, a process of supplying or degassing the pressure in the duodenal lumen to within the appropriate pressure range. In this way, the inside of the duodenal lumen can be made suitable for estimating the apex line, incising the EST, and the like.

[0053] For example, among the above-mentioned correct labels, the label "The pressure in the duodenal lumen is higher than a predetermined range" may be further ranked into multiple stages. Furthermore, the processor 10 may select a command to be output from the pressure determination unit 60 to the gas supply / degassing device 7 so as to correspond to the output rank, and output the selected command to the gas supply / degassing device 7. The same applies to the label "The pressure in the duodenal lumen is lower than a predetermined range". This allows the gas supply / degassing device 7 to be controlled more accurately.

[0054] Although not shown, the processing units 100 in FIGS. 20, 22, 26, 29, 32, 37, and 44 described later may also be configured to control the pressure determination unit 60.

[0055] Also, for example, this embodiment may be a method of acquiring a plurality of endoscopic images in which the shape of the duodenal papilla may be different, and estimating the apex line based on a depth map estimated from each endoscopic image. For example, the above-mentioned method can be realized by configuring the processing unit 100 as shown in the configuration example of Fig. 16, configuring step S200 of Fig. 8 as shown in the processing example of the flowchart of Fig. 17, and configuring step S300 of Fig. 8 as shown in the processing example of the flowchart of Fig. 18. The processing unit 100 shown in Fig. 16 differs from the processing unit 100 shown in Fig. 10 in that the apex line estimation unit 120 further includes a depth estimation result subtraction unit 122.

[0056] The endoscopic image acquisition process (step S200) shown in Fig. 17 will be described. The processor 10 acquires a first endoscopic image (step S202), and then acquires a second endoscopic image (step S204). After that, the process returns to the flowchart of Fig. 8, and the apex line estimation process (step S300) and the process of step S400 are performed.

[0057] The combination of the first endoscopic image and the second endoscopic image is not particularly limited as long as the endoscopic images are endoscopic images in which the shape of the duodenal papilla may be different. For example, the first endoscopic image may be an endoscopic image of the duodenal papilla in a state in which the treatment tool is not inserted and includes an oral bulge, as shown in A41 of FIG. 19. The second endoscopic image may be an endoscopic image of the duodenal papilla in a state in which the treatment tool is inserted, as shown in A42 of FIG. 19. The state in which the second endoscopic image shown in A42 is captured is, for example, a state in which a guide wire as a treatment tool is inserted, but it may also be a state in which the above-mentioned cannula is inserted, or a state immediately before EST is started by inserting an EST knife. In this way, in the processing system 3 of this embodiment, the processor 10 estimates the apex line based on the first endoscopic image in a state in which the treatment tool is not inserted and the second endoscopic image in a state in which the treatment tool is inserted. The same applies when the method of this embodiment is realized as a processing method. That is, in the processing method of this embodiment, a process of estimating the apex line is performed based on a first endoscope image in a state where the treatment tool is not inserted and a second endoscope image in a state where the treatment tool is inserted (steps S202, S204, and S300). In this way, the path of the bile duct can be estimated with higher accuracy. Since the treatment tool is inserted into the bile duct, it is considered that the change in the apex line corresponds to the change in the path of the bile duct.

[0058] In addition, step S202 is not limited to a process of acquiring a first endoscopic image by imaging by the imager of the endoscope 5 in real time, and may be a process of selecting a first endoscopic image already stored in the memory 20 or the like. The same applies to step S204. For example, as described above, when the first endoscopic image is an endoscopic image of the duodenal papilla in a state where a treatment tool is not inserted, for example, an endoscopic image captured when the imager at the tip of the endoscope is aligned with the duodenal papilla when performing ERCP may be stored separately in the memory 20 or the like. In addition, when an endoscopic image of the duodenal papilla at the timing when the guidewire is first inserted into the bile duct is used as the second endoscopic image, an endoscopic image captured at that timing may be stored separately in the memory 20 or the like. Alternatively, when an endoscopic image of the duodenal papilla immediately before the start of EST is used as the second endoscopic image, a process of acquiring an endoscopic image captured by the imager of the endoscope 5 in real time may be used as step S204.

[0059] The crest line estimation process (step S300) shown in FIG. 18 will be described. The processor 10 estimates a first depth map from a first endoscope image (step S310), estimates a second depth map from a second endoscope image (step S320), and estimates a crest line based on the difference between the first depth map and the second depth map (step S350). Steps S310 and S320 in FIG. 18 correspond to step S302 in FIG. 9. For example, the first depth map shown in A61 is generated based on the first endoscope image shown in A41 in FIG. 19 by step S310. Similarly, the second depth map shown in A62 is generated based on the second endoscope image shown in A42 in FIG. 19 by step S320. For convenience of explanation, the first depth map shown in A61 in FIG. 19 displays a portion corresponding to the endoscope image within the dotted frame of the first endoscope image shown in A41. Similarly, the second depth map indicated by A62 displays a portion of the second endoscopic image indicated by A42 that corresponds to the endoscopic image enclosed by the dotted line frame.

[0060] Thereafter, the processor 10 functions as a depth estimation result subtraction unit 122, and obtains the difference between the second depth map shown in A62 and the first depth map shown in A61. That is, the difference in gradation value is obtained for each pixel of each depth map, and a group of pixels with a relatively high difference value is estimated to be a region where the depth has changed due to the insertion of the treatment tool. When the treatment tool is a guidewire, the guidewire passes through the bile duct as described above in FIG. 5, and therefore the above-mentioned region where the depth has changed can be estimated to be a region where the bile duct passes.

[0061] In this way, in the processing system 3 of this embodiment, the processor 10 estimates the apex line based on the difference between the first depth map estimated from the first endoscope image and the second depth map estimated from the second endoscope image. In this way, a processing system 3 that estimates the apex line based on the difference between depth maps in different states can be constructed. The same applies when the technique of this embodiment is realized as a processing method. That is, in the processing method of this embodiment, a process (steps S310, S320, and S350) is performed to estimate the apex line based on the difference between the first depth map estimated from the first endoscope image and the second depth map estimated from the second endoscope image. In this way, a processing method that estimates the apex line based on the difference between depth maps in different states can be constructed.

[0062] As described above in FIG. 11 and the like, step S302 in FIG. 9 is performed using the machine-learned trained model 22, and therefore steps S310 and S320 in FIG. 18 are also performed using the trained model 22. That is, the processing system 3 of this embodiment includes a memory 20 that stores the trained model 22 trained to estimate a depth map from an input image. The processor 10 estimates a first depth map by inputting a first endoscope image to the trained model 22, estimates a second depth map by inputting a second endoscope image to the trained model 22, and estimates a vertex line based on the first depth map and the second depth map. In this way, a processing system 3 that estimates a vertex line based on two depth maps can be constructed using the machine-learned trained model 22.

[0063] Also, for example, this embodiment may be a method of estimating the apex line after performing alignment. For example, the above-mentioned method can be realized by configuring the processing unit 100 as shown in the configuration example of FIG. 20, configuring step S200 of FIG. 8 as shown in the processing example of the flowchart of FIG. 17, and configuring step S300 of FIG. 8 as shown in the processing example of the flowchart of FIG. 21. The configuration example of the processing unit 100 in FIG. 20 differs from the configuration example of the processing unit 100 in FIG. 16 in that it further includes a key point detection unit 130 and a first depth estimation result alignment unit 141. Note that the description of the processing already described will be omitted.

[0064] The apex line estimation process (step S300) of Fig. 21 will be described. The processor 10 performs a process (step S310) of estimating a first depth map from a first endoscope image similar to that of Fig. 18, and also performs a process (step S312) of detecting key points from the first endoscope image by functioning as the key point detection unit 130. The key points here refer to position information that serves as a reference for aligning the first depth map with the second depth map, and specifically, are nipple openings, headband folds, frenulum, etc. The key points can also be called feature points or landmarks. The process of making the processor 10 function as the key point detection unit 130 corresponds to step S312.

[0065] Step S312 is performed using the trained model 22 that has been machine-learned. For example, the trained model 22 that is read out when the processor 10 is made to function as the keypoint detection unit 130 is trained to, for example, when an endoscopic image in which the duodenal papilla is captured is input, perform segmentation or the like on the endoscopic image to output the papilla opening, which is a keypoint. The trained model 22 related to step S312 may use at least a part of a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), or a model further developed from these, which are used in the image recognition field. Examples of models further developed from CNN include SegNet (Segmentation Network), FCN (Fully Convolutional Network), U-Net (U-Shaped Network), PSPNet (Pyramid Scene Parsing Network), YOLO (You Only Look Once), SSD (Single Shot Multi-Box Detector), and the like.

[0066] Thereafter, the processor 10 performs a process (step S320) of estimating a second depth map from the second endoscopic image similar to that of Fig. 18, and also performs a process (step S322) of detecting key points from the second endoscopic image. Step S322 is performed using the machine-learned trained model 22, similar to the above-mentioned step S312.

[0067] Thereafter, the processor 10 functions as the first depth estimation result alignment unit 141, and performs a process (step S330) of aligning the first depth map and the second depth map based on the detected key point. For example, as shown in A51 of Fig. 19 described above, the nipple opening is detected as a key point from the first endoscope image in step S312. Similarly, as shown in A52 of Fig. 19, the nipple opening is detected as a key point from the second endoscope image in step S322.

[0068] Then, the processor 10 functions as the first depth estimation result alignment unit 141, and specifies, for example, the position shown in A71 in FIG. 19 as the position of the nipple opening in the first depth map based on the coordinate information of the key point detected in A51. If the key point shown in A51 is detected as an area, the position of the center of gravity of the indicated area may be specified as the position shown in A71. Similarly, the processor 10 specifies, for example, the position shown in A72 in FIG. 19 as the position of the nipple opening in the second depth map based on the coordinate information of the key point detected in A52. Then, the processor 10 converts the position coordinates of either the first depth map or the second depth map so that the position coordinates shown in A71 and the position coordinates shown in A72 are the same coordinates. After that, the processor 10 performs the above-mentioned step S350 to estimate the apex line.

[0069] Note that step S330 can also be performed using the trained model 22. For example, the trained model 22 may be trained using a data set that associates an endoscopic image, key points in the endoscopic image, and information on the correspondence between key points in different endoscopic images.

[0070] Also, the above-mentioned technique can be similarly realized by configuring the processing unit 100 as in the configuration example shown in Fig. 22, configuring step S200 in Fig. 8 as in the processing example shown in the flowchart of Fig. 17, and configuring step S300 in Fig. 8 as in the processing example shown in the flowchart of Fig. 23. The configuration example of the processing unit 100 in Fig. 22 differs from the configuration example of the processing unit 100 in Fig. 16 in that it further includes a point cloud data generation unit 132 and a second depth estimation result alignment unit 142.

[0071] The apex line estimation process (step S300) of Fig. 23 will be described. After performing a process (step S310) of estimating a first depth map from a first endoscopic image similar to that of Fig. 18, the processor 10 performs a process (step S314) of generating first point cloud data from the first depth map. For example, the processor 10 functions as the point cloud data generating unit 132, acquires a depth map as shown in A63 of Fig. 24 from the depth estimation unit 110 as the first depth map, and generates point cloud data as shown in A83 based on a predetermined algorithm. The predetermined algorithm may be a known method of generating point cloud data from a depth map.

[0072] Thereafter, the processor 10 performs a process (step S320) of estimating a second depth map from the second endoscopic image similar to that of Fig. 18, and then performs a process (step S324) of generating second point cloud data from the second depth map. Step S324 is a process similar to the above-mentioned step S314.

[0073] The first and second endoscopic images in the processing example of Fig. 23 are the same as those in the processing example of Fig. 21. That is, the first endoscopic image in the processing of Fig. 23 is an endoscopic image of the duodenal papilla in a state where a treatment tool is not inserted as shown in A44 of Fig. 25, for example, and the second endoscopic image in the processing of Fig. 23 is an endoscopic image of the duodenal papilla in a state where a treatment tool is inserted as shown in A45 of Fig. 25, for example. As a result, for the first endoscopic image shown in A44, a first depth map not shown in Fig. 25 is generated in step S310, and first point cloud data shown in A84 is generated based on the first depth map in step S314. Similarly, for the second endoscopic image shown in A45, a second depth map not shown in Fig. 25 is generated in step S320, and second point cloud data shown in A85 is generated based on the second depth map in step S324.

[0074] Thereafter, the processor 10 functions as the second depth estimation result alignment unit 142, and performs a process (step S340) of aligning the first depth map and the second depth map based on the first point cloud data and the second point cloud data. Although a detailed description of the ICP algorithm is omitted as it is a known method, for each point of one point cloud data, the closest point is searched for in the other point cloud data, and the points are associated with each other, and the position and orientation of the coordinate system of the first point cloud data and the coordinate system of the second point cloud data are adjusted so as to minimize the difference between the associated points. As a result, for example, the difference between each point constituting the contour part shown in A94 and each point constituting the contour part shown in A95 is minimized, and the first point cloud data and the second point cloud data are matched. Then, the processor 10 adjusts the positional relationship between the first depth map and the second depth map so as to correspond to the adjustment of the coordinate system of the point cloud data.

[0075] Note that step S340 can also be performed using the trained model 22. For example, a combination of point cloud data with different corresponding position coordinates can be used as input data, and the trained model 22 can be trained using the coordinate transformation data used when adjusting the positional relationship as a correct answer label and a data set. After that, the processor 10 estimates the apex line based on the difference between the first depth map and the second depth map after adjusting the positional relationship (step S350).

[0076] From the above, in the processing system 3 of this embodiment, the processor 10 estimates the apex line based on the difference between the first depth map and the second depth map in a state where the position of the duodenal papilla shown in the first endoscope image and the position of the duodenal papilla shown in the second endoscope image are aligned. The same applies when the method of this embodiment is realized as a processing method. That is, in the processing method of this embodiment, in a state where the position of the duodenal papilla shown in the first endoscope image and the position of the duodenal papilla shown in the second endoscope image are aligned, a process of estimating the apex line based on the difference between the first depth map and the second depth map (steps S330, S340, and S350) is performed. In this way, the accuracy of the estimation of the apex line can be further improved. In the process of inserting the treatment tool, the tip of the endoscope moves, and the position of the duodenal papilla in the first endoscope image and the position of the duodenal papilla in the second endoscope image may not match. In this regard, by applying the method of this embodiment, the position of the duodenal papilla can be matched before and after insertion of the treatment tool, so that the difference between the first depth map and the second depth map can be determined more accurately.

[0077] Also, for example, this embodiment may be a method of setting the position of one end of the guide display GM. For example, the processing unit 100 may be configured as shown in the configuration example of Fig. 26, and step S300 in Fig. 8 may be configured as shown in the flowchart of Fig. 27. The configuration example of the processing unit 100 in Fig. 26 differs from the configuration example of the processing unit 100 in Fig. 16 in that it further includes a nipple opening detection unit 150.

[0078] The apex line estimation process (step S300) in Fig. 27 will be described. The processor 10 performs the process of estimating a depth map from an endoscopic image (step S302) as described above, and also functions as the nipple opening detection unit 150 to perform the process of detecting the nipple opening on the endoscopic image (step S360).

[0079] Step S360 is performed using the machine-learned trained model 22. That is, the trained model 22 that is read out when the processor 10 is made to function as the papilla opening detection unit 150 is trained by machine learning so that, for example, when an endoscopic image in which the duodenal papilla is captured is input, the trained model 22 outputs the papilla opening by segmenting the endoscopic image.

[0080] Then, the processor 10 performs a process of estimating a vertex line based on the detected nipple opening (step S362). For example, in FIG. 28, the area shown in B1 is the area of ​​the nipple opening detected in step S360, and the area shown in B2 is the area estimated to be the foremost side in the depth map generated in step S302. In this case, the processor 10 calculates the barycentric coordinate of the area B1, for example, and sets the barycentric coordinate point as shown in B3. The processor 10 then performs a process of setting the barycentric coordinate of B3 and an imaginary line passing through the center of the area B2 as the vertex line, as shown in B4. For convenience of explanation, the direction of the line shown in B4 is assumed to match the vertical direction of the paper. The same applies to FIG. 34 described later. Although not shown, the processor 10 estimates that the line shown in B4, which starts from the point shown in B3 and includes the area B2, is the vertex line, and displays a guide display GM corresponding to the estimated vertex line.

[0081] Thus, in the processing system 3 of this embodiment, the processor 10 detects the nipple opening based on the endoscopic image, and estimates the apex line with the nipple opening as the starting point. This is the same when the technique of this embodiment is realized as a processing method. That is, in the processing method of this embodiment, a process of detecting the nipple opening based on the endoscopic image (step S360) is further performed, and a process of estimating the apex line with the nipple opening as the starting point (step S362) is performed. In this way, a guide display GM with the nipple opening as the starting point can be displayed. Although the apex line does not necessarily include the nipple opening, since EST is a procedure for incising and expanding the nipple opening, it is convenient for the user to display the guide display GM with the nipple opening as the starting point.

[0082] Also, a similar method can be realized by configuring the processing unit 100 as shown in the configuration example of Fig. 29 and configuring step S300 of Fig. 8 as shown in the processing example of the flowchart of Fig. 30. The configuration example of the processing unit 100 in Fig. 29 differs from the configuration example of the processing unit 100 in Fig. 16 in that it further includes a treatment tool detection unit 152.

[0083] The apex line estimation process (step S300) of Fig. 30 will be described. The processor 10 functions as the depth estimation unit 110 and performs the above-mentioned step S302, and also functions as the treatment tool detection unit 152 and performs a process of detecting the treatment tool on the endoscopic image (step S370).

[0084] Like step S360, step S370 is performed using the machine-learned trained model 22. That is, the trained model 22 read out when the processor 10 is made to function as the treatment tool detection unit 152 is trained by machine learning so that, when an endoscopic image of the duodenal papilla into which a treatment tool is inserted is input, the trained model 22 performs machine learning to output the region of the imaged treatment tool by performing segmentation or the like.

[0085] Then, the processor 10 performs a process of estimating the apex line based on the detected treatment tool (step S372). For example, in FIG. 31, B11 is the area of ​​the tip of the treatment tool detected in step S370 that is shown in the endoscopic image, and the area shown in B12 is the area estimated to be the closest in the depth map generated in step S302. In this case, the processor 10 performs a process of setting an imaginary line that passes through the center part of the area shown in B11 and the center part of the area shown in B12, as shown in B13. Then, although not shown, the processor 10 estimates that the line shown in B13, which starts from the area shown in B11 and includes the area B12, is the apex line, and displays a guide display GM corresponding to the estimated apex line.

[0086] In this way, in the processing system 3 of this embodiment, the processor 10 detects the part of the treatment tool that appears in the endoscopic image based on the endoscopic image, and estimates the apex line with the tip of the detected part as the starting point. This is the same when the method of this embodiment is realized as a processing method. That is, in the processing method of this embodiment, a process of detecting the part of the treatment tool that appears in the endoscopic image based on the endoscopic image (step S370) is further performed, and a process of estimating the apex line with the tip of the detected part as the starting point (step S372) is performed. In this way, a guide display GM with one end of the treatment tool as the starting point can be displayed. When performing EST, it is convenient for the user to display the guide display GM with the EST knife as the treatment tool as the starting point. Although the treatment tool may not be able to clearly see the nipple opening itself from the endoscopic image, the position of the tip of the treatment tool that appears in the endoscopic image corresponds to the position of the nipple opening, so the method described in Figures 29 to 31 can be expected to have the same effect as the method described in Figures 26 to 28.

[0087] In addition, this embodiment may be a method of displaying the guide display GM with a certain width, for example. Specifically, the above method can be realized by configuring the processing unit 100 as shown in the configuration example of FIG. 32 and configuring step S300 of FIG. 8 as shown in the processing example of the flowchart of FIG. 33. The configuration example of the processing unit 100 in FIG. 32 differs from the configuration example of the processing unit 100 in FIG. 26 in that it further includes an incision region generating unit 160. Although not shown, the processing unit 100 in FIG. 32 may be configured by adding the incision region generating unit 160 to the processing unit 100 in FIG. 29. In this case, a process equivalent to step S364 described later may be added to the flowchart of FIG. 30.

[0088] The apex line estimation process (step S300) in Fig. 33 will be described. Note that the description of the same processes as those in Fig. 27 will be omitted. After performing steps S302, S360, and S362 in Fig. 27, the processor 10 functions as the incision area generating unit 160 and performs a process (step S364) of generating an incision area that indicates an area between 11 o'clock and 12 o'clock when the estimated apex line is in the 12 o'clock direction. Note that the 12 o'clock direction here is the 12 o'clock direction when one end of the estimated apex line on the nipple opening side is the center.

[0089] For example, the apex line is estimated by step S302. Then, as a result of steps S360 and S362, the guide display GM is displayed starting from the center of gravity of the nipple opening and in a direction that matches the vertical direction of the paper, as shown in Fig. 34. Then, the processor 10 functions as the incision area generating unit 160, and performs a process of generating a virtual line shown in B21 and a process of generating the area shown in B22 as the incision area.

[0090] The processing in Fig. 33 etc. can be applied even when the structure of an organ has been changed by a certain surgical treatment. In other words, when the structure of an organ is changed, the appearance of the endoscopic image also changes, but the processing in Fig. 33 etc. can be applied regardless of the appearance of the endoscopic image.

[0091] A specific description will be given with reference to Figures 35 and 36. In the case where the structure of an organ has been changed by the Billroth II procedure as shown in Figure 35 as a prescribed surgical treatment, the tip of the endoscope is inserted from the jejunum side toward the duodenal papilla when performing EST or the like, which is different from the case shown in Figure 2. In the Billroth II procedure, as shown in C1 of Figure 35, after the pylorus side of the stomach is resected for the purpose of obesity treatment, gastric cancer resection, etc., the stump of the remaining stomach is anastomosed to the jejunum to bypass the food passage. Also, as shown in C2, the end of the duodenum is sutured closed. Note that obesity refers to a state in which excess body fat has accumulated to such an extent that it may cause a shortened life expectancy, health problems, etc. Obesity surgery is performed, for example, when the BMI exceeds a certain value, or when the BMI exceeds a certain value and there is a comorbid disease. BMI is an abbreviation for Body Mass Index. Note that other prescribed surgical treatments include Roux-en-Y gastric bypass surgery, etc.

[0092] For example, the duodenal papilla is present on the side wall of the lumen shown in C11 of Fig. 36, the duodenal papilla has the structure shown in C12, and the papilla opening is located at the position shown in C22. In Fig. 36, the direction DR1 is the direction from the stomach side to the duodenum side, and the direction DR2 is the opposite direction to the direction DR1, that is, the direction from the jejunum side to the duodenum side. The directions DR3 and DR4 are perpendicular to the direction DR1 and are along the wall surface of the lumen, and the direction DR3 is upward with respect to the paper.

[0093] When the structure of the organ has not been changed as shown in Fig. 2, the endoscope tip is inserted so as to face in the direction DR1. The endoscopic image captured by the imager at the endoscope tip, since the direction DR1 faces upward, is the image shown in C13. On the other hand, when the structure of the organ has been changed as shown in Fig. 35, the endoscope tip is inserted so as to face in the direction DR2. The endoscopic image captured by the imager at the endoscope tip inserted so as to face in the direction DR2 is the image shown in C14, since the direction DR2 faces upward. In other words, the endoscopic image shown in C13 and the endoscopic image shown in C14 are rotated 180 degrees around an axis perpendicular to the endoscopic image.

[0094] When the structure of the organ has not been changed as in Figure 2, the 11 o'clock direction when the guide display GM faces in the direction of 12 o'clock with one end of the guide display GM as the center is the direction shown by the dotted line in C23, and the area between the guide display GM and the dotted line in C23 is the incision area. On the other hand, when the structure of the organ has been changed as in Figure 35, the 11 o'clock direction when the guide display GM faces in the direction of 12 o'clock with one end of the guide display GM as the center is the direction shown by the dotted line in C24, and the area between the guide display GM and the dotted line in C24 is the incision area.

[0095] Thus, when only the endoscopic image of C14 is viewed, the direction of the guide display GM and the direction of the dotted line of C24 do not seem to be in a relationship of 12 o'clock and 11 o'clock. However, in this embodiment, since one end of the apex line on the nipple opening side is defined as the center of 12 o'clock, the incision area can be displayed accurately. From the above, in the processing system 3 of this embodiment, the processor 10 superimposes a guide display indicating the range between the 12 o'clock direction and the 11 o'clock direction when the estimated apex line is in the 12 o'clock direction with one end of the estimated apex line on the nipple opening side as the center on the endoscopic image (steps S364, S400). The same is true when the technique of this embodiment is realized as a processing method. That is, in the processing method of this embodiment, a process is performed in which a guide display indicating the range between the 12 o'clock direction and the 11 o'clock direction when the estimated apex line is in the 12 o'clock direction with one end of the estimated apex line on the nipple opening side as the center is superimposed on the endoscopic image (steps S364, S400). In this way, it is possible to promote safer incision procedures when performing incision procedures using an EST. For example, since the incision procedure using an EST is a manual procedure, errors may occur. However, deviation of the incision direction from the 1 o'clock direction to the 12 o'clock direction described above may damage the artery. In this regard, by applying the method of this embodiment, the incision direction can be guided to the 11 o'clock side, so that the possibility of damaging the artery can be reduced even if an error occurs in the incision procedure using the EST.

[0096] In addition, this embodiment may further include a method for further defining the length of the guide display GM. Specifically, for example, the processing unit 100 is configured as shown in the configuration example of FIG. 37, and the processing example shown in the flowchart of FIG. 38 is added to the processing example of FIG. 27, thereby realizing the above-mentioned method. The configuration example of the processing unit 100 in FIG. 37 is different from the configuration example of the processing unit 100 in FIG. 26 in that it further includes an incision length estimation unit 170. In addition, the incision length estimation unit 170 further includes a headband pleat estimation unit 180. Although not shown, the processing unit 100 in FIG. 37 may be configured by adding the incision length estimation unit 170 and the like to the processing unit 100 in FIG. 29. In this case, the processing example shown in the flowchart of FIG. 38 may be added to the processing example of FIG. 30.

[0097] A description will be given of an example of processing shown in the flowchart of Fig. 38. The processor 10 functions as the headband pleat estimation unit 180 and performs processing for estimating a headband pleat (step S380).

[0098] Step S380 is performed using the trained model 22 that has been machine-learned. That is, the trained model 22 that is read out when the processor 10 is made to function as the headband pleat estimation unit 180 is trained by machine learning to use, for example, a depth map generated by the depth estimation unit 110 as input data and output a region in the depth map that corresponds to the headband pleat as a correct answer label. Although not shown, for example, the incision length estimation unit 170 may receive an endoscopic image from the display control unit 50, and the trained model 22 may use the endoscopic image as input data and output a region in the endoscopic image that corresponds to the headband pleat as a correct answer label.

[0099] Then, the processor 10 functions as the incision length estimation unit 170, and performs a process (step S390) of estimating a guideline for the incision length from the position information of the headband fold. The processor 10 then performs step S400 in Fig. 8. As a result, an endoscopic image such as that shown in B31 in Fig. 39 can be obtained. In the endoscopic image shown in B31, a region estimated to be a headband fold is segmented as shown in B32 in step S380. Furthermore, a guide display GM is displayed in the length from the nipple opening shown in B33 to the headband fold region shown in B32 in step S390.

[0100] In this manner, in the processing system 3 of this embodiment, the processor 10 estimates the position information of the headband pleat (step S380), estimates the incision length information based on the estimated position information of the headband pleat (step S390), and displays the guide display GM of the length based on the estimated incision length superimposed on the endoscopic image. The same applies when the method of this embodiment is realized as a processing method. That is, in the processing method of this embodiment, a process of estimating the position information of the headband pleat (step S380) and a process of estimating the incision length information based on the estimated position information of the headband pleat (step S390) are further performed, and a process of displaying the guide display GM of the length based on the estimated incision length superimposed on the endoscopic image (step S400) is performed. In this manner, the user can perform EST by referring to the estimated position information of the headband pleat. This allows the user to perform EST so that the incision range of the EST is limited to a small incision or a medium incision, avoiding a large incision. This can prevent excessive bleeding, perforation, etc. A small incision refers to an incision that does not extend beyond the fold of the headband, a large incision refers to an incision up to the upper edge of the oral protuberance, and a medium incision refers to an incision that is intermediate between a small incision and a large incision.

[0101] 40, the incision length estimation unit 170 may further include a headband pleat estimation possibility determination unit 182 and an alarm information generation unit 184. In this case, the processing example of FIG. 38 may be modified to a processing example shown in the flowchart of FIG. 41.

[0102] A processing example shown in the flowchart of Fig. 41 will be described. Note that a description of the same processing as in Fig. 38 will be omitted as appropriate. After performing the above-mentioned step S380, the processor 10 functions as the headband pleats estimation feasibility determination unit 182, and performs a process (step S382) of determining whether the estimated probability of a headband pleat is equal to or greater than a predetermined value.

[0103] If the estimated probability of the headband pleat is less than a predetermined value (NO in step S382), the processor 10 functions as the alarm information generating unit 184 and generates alarm information (step S384). For example, the processor 10 may display the estimated probability of the headband pleat as shown in B34 of FIG. 39, and may further perform a display process indicating that the estimated probability is less than a predetermined value, or may perform a process of emitting a predetermined alarm sound, and a wide range of known methods can be applied. If the probability of the headband pleat being estimated is not equal to or greater than a predetermined value, the area different from the area shown in B32 is likely to be the actual headband pleat. Therefore, even if EST is performed only for the incision length of the displayed guide display GM, there is a possibility that an incision will be made in a range larger than the desired range. In this regard, by applying the method of this embodiment, it is possible to prevent an incision from being made in an unintended range. On the other hand, if the estimated probability of the headband pleat is equal to or greater than a predetermined value (YES in step S382), the processor 10 performs the above-mentioned step S390.

[0104] Also, the incision length estimation unit 170 may include an incision length shortening unit 186 instead of the alarm information generation unit 184 as shown in Fig. 42. In this case, the processing example of Fig. 38 may be modified to a processing example shown in the flowchart of Fig. 43.

[0105] A processing example shown in the flowchart of FIG. 43 will be described. Note that the description of the same processing as in FIG. 38 and FIG. 41 will be omitted as appropriate. The processor 10 performs the above-mentioned steps S380 and S382. If the estimated probability of the headband pleat is less than a predetermined value (NO in step S382), the processor 10 functions as the incision length estimation unit 170 including the incision length shortening unit 186, estimates an approximate incision length from the position information of the headband pleat, and performs processing to shorten the estimated approximate incision length (step S386). Then, the processor 10 performs step S400 in FIG. 8. As described above in FIG. 41, if NO in step S382, there is a high possibility that the area wider than the estimated headband pleat area shown in B32 in FIG. 39 is the actual headband pleat area. Therefore, although not shown in the figures, for example, the processor 10 calculates the length from the nipple opening to the headband fold, and displays a guide display GM having a length that is shortened by a certain percentage from the calculated length. In this way, it is possible to reduce the possibility that an area larger than the desired area will be incised when the estimation accuracy of the headband fold area is insufficient.

[0106] Although the methods by the processing unit 100 in Fig. 26, Fig. 32, and Fig. 37 are described individually, these methods can be appropriately combined. For example, as shown in Fig. 44, the processing unit 100 can be configured to include a nipple opening detection unit 150, an incision area generation unit 160, and an incision length estimation unit 170 in addition to a depth estimation unit 110 and a ridge line estimation unit 120. Similarly, although not shown, the processing unit 100 can be configured to include a treatment tool detection unit 152, an incision area generation unit 160, and an incision length estimation unit 170 in addition to a depth estimation unit 110 and a ridge line estimation unit 120.

[0107] Although the embodiments to which the present disclosure is applied and their modifications have been described above, the present disclosure is not limited to each embodiment or its modifications as it is, and in the implementation stage, the components can be modified and embodied within the scope of the gist of the disclosure. In addition, various disclosures can be formed by appropriately combining multiple components disclosed in each of the above-mentioned embodiments and modifications. For example, some components may be deleted from all the components described in each embodiment or modification. Furthermore, components described in different embodiments or modifications may be appropriately combined. In this way, various modifications and applications are possible within the scope of the gist of the disclosure. In addition, a term described at least once in the specification or drawings together with a different term having a broader meaning or the same meaning can be replaced with that different term in any part of the specification or drawings. [Explanation of symbols]

[0108] 1...endoscope system, 3...processing system, 5...endoscope, 7...deaeration device, 9...display device, 10...processor, 20...memory, 22...trained model, 30...input unit, 40...output unit, 50...display control unit, 60...pressure determination unit, 100...processing unit, 110...depth estimation unit, 120...top line estimation unit, 122...depth estimation result subtraction unit, 130...key point detection unit, 132...point cloud data generation unit, 1 41...first depth estimation result alignment unit, 142...second depth estimation result alignment unit, 150...nipple opening detection unit, 152...treatment tool detection unit, 160...incision area generation unit, 170...incision length estimation unit, 180...estimation unit, 182...estimation feasibility determination unit, 184...alarm information generation unit, 186...incision length shortening unit, DR1, DR2, DR3, DR4, DR11, DR12, DR13...direction, GM...guide display

Claims

1. a processor including hardware; The processor, An endoscopic image of the duodenal papilla including the oral protuberance is obtained. estimating a depth map of a region including the duodenal papilla from the endoscopic images; estimating a crest line of the oral ridge from the estimated depth map; A processing system that performs processing to superimpose a guide display based on the estimated apex line on the endoscopic image.

2. 2. The processing system of claim 1, The guide display is A processing system characterized by displaying an incision guide in EST.

3. 2. The processing system of claim 1, A memory that stores a trained model that is trained to estimate the depth map from an input image; The processor, A processing system characterized by estimating the depth map by inputting the endoscopic image into the learned model.

4. 2. The processing system of claim 1, The processor, A processing system characterized by estimating the apex line based on a first endoscopic image in a state where a treatment tool is not inserted and a second endoscopic image in a state where the treatment tool is inserted.

5. 5. The processing system of claim 4, The processor, A processing system characterized by estimating the apex line based on the difference between a first depth map estimated from the first endoscopic image and a second depth map estimated from the second endoscopic image.

6. 5. The processing system of claim 4, A memory that stores a trained model that is trained to estimate the depth map from an input image; The processor, Estimating a first depth map by inputting the first endoscopic image into the trained model; A second depth map is estimated by inputting the second endoscopic image into the trained model; A processing system for estimating the apex line based on the first depth map and the second depth map.

7. 7. The processing system of claim 6, The processor, A processing system characterized by estimating the apex line based on the difference between the first depth map and the second depth map when the position of the duodenal papilla in the first endoscopic image and the position of the duodenal papilla in the second endoscopic image are aligned.

8. 2. The processing system of claim 1, The processor, Detecting a nipple opening based on the endoscopic image; A processing system for estimating the apex line starting from the nipple opening.

9. 2. The processing system of claim 1, The processor, Detecting a portion of the treatment tool that is shown in the endoscopic image based on the endoscopic image; The processing system estimates the apex line starting from a tip of the detected portion.

10. 2. The processing system of claim 1, The processor, A processing system characterized by superimposing a guide display indicating the range from the 12 o'clock direction to the 11 o'clock direction when the estimated apex line is centered on one end of the estimated apex line on the nipple opening side, on the endoscopic image.

11. 2. The processing system of claim 1, The processor, The position information of the headband folds is estimated, Estimating information on an incision length based on the estimated position information of the headband fold; A processing system comprising: a guide display having a length based on the estimated incision length, the guide display being superimposed on the endoscopic image.

12. 2. The processing system of claim 1, The processor, estimating whether the pressure in the duodenal lumen is within an appropriate pressure range based on the endoscopic image; A treatment system characterized by performing a process of supplying or degassing air to bring the pressure in the duodenal lumen into the range of the appropriate pressure when the pressure in the duodenal lumen is outside the range of the appropriate pressure.

13. A processing system according to any one of claims 1 to 12; An endoscope and An endoscope system comprising:

14. acquiring an endoscopic image of the duodenal papilla including the oral protuberance; estimating a depth map of a region including the duodenal papilla from the endoscopic image; estimating a top line of the oral ridge from the estimated depth map; a process of superimposing a guide display based on the estimated apex line on the endoscopic image; A processing method comprising the steps of:

15. 15. The method of claim 14, The guide display is A processing method characterized by displaying an incision guide in EST.

16. 15. The method of claim 14, A processing method characterized by performing processing to estimate the apex line based on a first endoscopic image in a state where a treatment tool is not inserted and a second endoscopic image in a state where the treatment tool is inserted.

17. 17. The method of claim 16, A processing method characterized by performing a process of estimating the apex line based on the difference between a first depth map estimated from the first endoscopic image and a second depth map estimated from the second endoscopic image.

18. 18. The method of claim 17, A processing method characterized by performing a process of estimating the apex line based on the difference between the first depth map and the second depth map when the position of the duodenal papilla in the first endoscopic image and the position of the duodenal papilla in the second endoscopic image are aligned.

19. 15. The method of claim 14, Further performing a process of detecting a nipple opening based on the endoscopic image; A processing method comprising the steps of: estimating the apex line using the nipple opening as a starting point.

20. 15. The method of claim 14, A process of estimating position information of the headband pleats; and further performing a process of estimating information on an incision length based on the estimated position information of the headband fold. A processing method comprising the steps of: performing processing to superimpose the guide display having a length based on the estimated incision length on the endoscopic image.

Images