Surgical systems and devices and methods of configuring a surgical system and performing an endoscopy procedure, including an ERCP procedure

Abstract

Implementations relate to surgical systems and methods. The system includes a main assembly having an IMU subsystem, a camera, a catheter assembly, and a processor. The processor processes the image and IMU information, including determining whether the image includes a distal end of the catheter assembly and an intubation target. In response to determining that the image includes the distal end of the catheter assembly, the processor generates a 3-dimensional position of the distal end of the catheter assembly. When the image is determined to include the intubation target, the processor generates a 3-dimensional position of the intubation target. The processor also generates a prediction of one or more real-time trajectory paths of the distal end of the catheter assembly intubating the intubation target.

Description

Technical Field

[0001] This disclosure generally relates to surgical systems and apparatuses, and methods for configuring surgical systems and performing surgical actions, and more specifically to surgical systems, subsystems, processors, apparatuses, logic, methods, and processes for performing biliary and pancreatic endoscopy and other forms of endoscopy. Background Technology

[0002] Endoscopic retrograde cholangiopancreatography (ERCP) remains the gold standard for pathological diagnosis and therapeutic intervention in the biliary and pancreatic systems. During an ERCP procedure, a flexible, side-viewing duodenoscope is inserted through the patient's mouth and into the second part of the duodenum. The endoscopist (also referred to as the "surgeon" or "user" in this text) then inserts a cannula into the large (or small) papilla (or papillary orifice). Summary of the Invention

[0003] Generally, routine endoscopic procedures are prone to complications. Challenges or difficulties include, but are not limited to: reaching the papilla or biliary / pancreaticojejunostomy; performing cannulation of the biliary and pancreatic system; and preventing or minimizing postoperative complications. Surgeons have recognized that successful cannulation means more than just accessing the desired catheter; it requires performing cannulation in the most efficient and safest manner while minimizing complications.

[0004] The various exemplary embodiments of the present invention generally relate to and / or include systems, subsystems, processors, devices, logic, methods, and processes for solving routine problems, including those systems, subsystems, processors, devices, logic, methods, and processes described above and in this disclosure, and more specifically, the various exemplary embodiments relate to systems, subsystems, processors, devices, logic, methods, and processes for performing surgical procedures, including biliary and pancreatic endoscopy and other forms of endoscopy.

[0005] In an example implementation, a method of configuring a surgical system is described. The method includes providing a surgical system. The surgical system includes a main component. The main component includes a body for insertion into a patient's cavity. The main component also includes an inertial measurement unit (IMU) subsystem housed within the body. The main component also includes an image capture subsystem housed within the body. The main component also includes a catheter assembly housed within the body. The catheter assembly includes a proximal end and a distal end. The catheter assembly is configured to selectively extend the distal end of the catheter assembly outward from the body. At least a portion of the catheter assembly is configured to selectively bend in multiple directions. The surgical system also includes a processor. The processor is configured to receive and process communications from the IMU subsystem and the image capture subsystem. The method also includes capturing real-time images via the image capture subsystem. The method also includes acquiring real-time IMU information via the IMU subsystem. The real-time IMU information includes real-time 3D position information. The method also includes processing the acquired images and IMU information via the processor. The processing includes determining whether the acquired images include the distal end of the catheter assembly. The process further includes, in response to determining that the acquired image includes the distal end of the catheter assembly, identifying the distal end of the catheter assembly in the acquired image and generating a real-time 3D position of the distal end of the catheter assembly based on IMU information. The process also includes determining whether the acquired image includes an insertion target. When the processor determines that the acquired image includes an insertion target, the process further includes identifying the insertion target in the acquired image and generating a real-time 3D position of the insertion target based on IMU information and the acquired image. When the processor determines that the acquired image includes both the distal end of the catheter assembly and the insertion target, the process further includes predicting one or more real-time trajectory paths for the identified distal end of the catheter assembly to insert cannulate the identified insertion target based on the 3D position of the distal end of the catheter assembly and the 3D position of the insertion target.

[0006] In another exemplary embodiment, a method of configuring a surgical system is described. The method includes providing a surgical system. The surgical system includes a main component. The main component includes a body for insertion into a patient's cavity. The main component also includes an inertial measurement unit (IMU) subsystem housed within the body. The main component also includes an image capture subsystem housed within the body. The main component further includes a catheter assembly housed within the body. The catheter assembly includes a proximal end and a distal end. The catheter assembly is configured to selectively extend the distal end of the catheter assembly outward from the body. At least a portion of the catheter assembly is configured to selectively bend in multiple directions. The surgical system also includes a processor. The processor is configured to receive and process communications from the IMU subsystem and the image capture subsystem. The method includes generating a plurality of information groups by the processor, the plurality of information groups including a first information group and a second information group. Each of the plurality of information groups includes real-time information. The first information group includes: a first set of images captured by the image capture subsystem at a first moment; and a first set of IMU information obtained by the processor from the IMU subsystem at the first moment. The first set of IMU information includes 3D position information at the first moment. The second information group includes: a second set of images captured by the image capture subsystem at a second time point, which is a time after the first time point; and a second set of IMU information obtained by the processor from the IMU subsystem at the second time point. The second set of IMU information includes 3D position information at the second time point. The method also includes processing multiple information groups, including the first and second information groups, by the processor. This processing includes processing the first information group. Processing the first information group includes determining whether the first set of images includes the distal end of the catheter assembly. Processing the first information group further includes: in response to determining that the first set of images includes the distal end of the catheter assembly, identifying the distal end of the catheter assembly in the first set of images for the first time point; and generating a 3D position of the identified distal end of the catheter assembly for the first time point based on the first set of IMU information. Processing the first information group further includes determining whether the first set of images includes the cannulation target. When the processor determines that the first set of images includes an intubation target, the processing of the first information set includes identifying the intubation target in the first set of images at a first time; and generating a 3D position of the identified intubation target at the first time based on the first set of IMU information and the first set of images. When it is determined that the first set of images includes the distal end of the identified catheter assembly and the identified intubation target, the processing of the first information set includes performing a first prediction. The first prediction is a prediction of one or more trajectory paths for intubation of the identified intubation target at the distal end of the identified catheter assembly at the first time, based on the 3D position of the distal end of the identified catheter assembly at the first time and the 3D position of the identified intubation target at the first time. The processing also includes processing of the second information set.Processing the second set of information includes determining whether the second set of images includes the distal end of a catheter assembly. Processing the second set of information includes: in response to determining that the second set of images includes the distal end of a catheter assembly, identifying the distal end of the catheter assembly in the second set of images at a second time; and generating a 3D position of the identified distal end of the catheter assembly at the second time based on the second set of IMU information. Processing the second set of information includes determining whether the second set of images includes an insertion target. When the processor determines that the second set of images includes an insertion target, processing the second set of information includes identifying the insertion target in the second set of images at a second time; and generating a 3D position of the identified insertion target at the second time based on the second set of IMU information and the second set of images. When the processor determines that the second set of images includes both the identified distal end of the catheter assembly and the identified insertion target, processing the second set of information includes performing a second prediction. The second prediction is based on the 3D position of the distal end of the identified catheter assembly at the second time and the 3D position of the identified cannulation target at the second time, predicting one or more trajectory paths for cannulation of the distal end of the identified catheter assembly to the identified cannulation target at the second time.

[0007] In another exemplary embodiment, a surgical system is described. The surgical system includes a main component. The main component includes a body for insertion into a patient's cavity. The main component also includes an inertial measurement unit (IMU) subsystem housed within the body. The IMU subsystem is configured to provide real-time IMU information, including real-time 3D position information. The main component also includes an image capture subsystem housed within the body. The image capture subsystem is configured to capture real-time images. The main component also includes a catheter assembly housed within the body. The catheter assembly includes a proximal end and a distal end. The catheter assembly is configured to selectively extend the distal end of the catheter assembly outward from the body. At least a portion of the catheter assembly is configured to selectively bend in multiple directions. The surgical system also includes a processor. The processor is configured to receive real-time IMU information from the IMU subsystem. The processor is also configured to receive real-time images from the image capture subsystem. The processor is further configured to determine whether the received image includes the distal end of the catheter assembly. When the processor determines that the received image includes the distal end of a catheter assembly, the processor is configured to: identify the distal end of the catheter assembly in the acquired image; and generate a real-time 3D position of the distal end of the catheter assembly based on the received IMU information. The processor is also configured to determine whether the received image includes an insertion target. When the processor determines that the received image includes an insertion target, the processor is configured to: identify the insertion target in the received image; and generate a real-time 3D position of the insertion target based on the IMU information and the received image. When the processor determines that the received image includes both the distal end of a catheter assembly and an insertion target, the processor is configured to predict one or more real-time trajectory paths for the identified distal end of the catheter assembly to insert cannulate the identified insertion target based on the 3D position of the distal end of the catheter assembly and the 3D position of the insertion target. Attached Figure Description

[0008] To gain a more complete understanding of this disclosure, exemplary embodiments, and their advantages, reference is now made to the following description in conjunction with the accompanying drawings, wherein like reference numerals denote like features, and:

[0009] Figure 1A This is an illustration of an example implementation of the surgical system;

[0010] Figure 1B This is an illustration of an example implementation of a surgical system configured to pass through the patient's mouth and enter the patient's duodenum;

[0011] Figure 2A This is a perspective view illustrating an example implementation of the main components of a surgical system;

[0012] Figure 2BAnother illustration of a perspective view of an example implementation of the main component, wherein the conduit assembly extends outward from the body of the main component;

[0013] Figure 2C This is a transparent perspective view illustrating an example implementation of the main body;

[0014] Figure 2D This is another illustration of a transparent perspective view of an example implementation of the main component, wherein the conduit assembly extends outward from the body of the main component;

[0015] Figure 3A This is a cross-sectional view of an example implementation of the main components, illustrating the image capture component and the IMU component;

[0016] Figure 3B This is another cross-sectional view of an example implementation of the main component, illustrating the conduit assembly;

[0017] Figure 3C This is an illustration of another cross-sectional view of an example embodiment of the main component, which illustrates a conduit assembly extending outward from the body of the main component.

[0018] Figure 4A This is an illustration of a cross-sectional view of an example implementation of a main component having two image capture components;

[0019] Figure 4B This is an illustration of a cross-sectional view of an example implementation of a main component having two IMU components and an image capture component;

[0020] Figure 5A This is an illustration of an example implementation of the processor of a surgical system;

[0021] Figure 5B This is an illustration of an example implementation of a conduit assembly processor;

[0022] Figure 5C This is an illustration of an example implementation of the cannulated target processor;

[0023] Figure 6A This is an illustration of the camera view of the image capture component, depicting the intubation target, catheter assembly, predicted trajectory path, and post-intubation predicted trajectory path.

[0024] Figure 6B This is another illustration of the camera view of the image capture component, which depicts the intubation target, the catheter assembly, the predicted trajectory path, and the predicted trajectory path after intubation.

[0025] Figure 6CThis is another illustration of the camera view of the image capture component, which depicts the cannulation target, the catheter assembly positioned in an ideal location and / or orientation, the post-cannulation predicted trajectory path, and the predicted trajectory path aligned with the post-cannulation predicted trajectory path.

[0026] Figure 7A The illustration shows an example implementation of a method for configuring a surgical system; and

[0027] Figure 7B This is an illustration of an example implementation of a method for processing acquired images and IMU information.

[0028] Although similar reference numerals may be used to refer to similar elements in the figures for convenience, it is understood that each example implementation in the various example embodiments can be considered a different variation.

[0029] Example embodiments will now be described with reference to the accompanying drawings, which form part of this disclosure and illustrate example embodiments that can be practiced. As used in this disclosure and the appended claims, the terms “implementation,” “example implementation,” “exemplary implementation,” and “this implementation” do not necessarily refer to a single implementation, although they may refer to a single implementation, and various example embodiments may be readily combined and / or interchanged without departing from the scope or spirit of the example embodiments. Furthermore, the terminology used in this disclosure and the appended claims is for the purpose of describing example embodiments only and is not restrictive. In this respect, as used in this disclosure and the appended claims, the term “in” can include “in” and “on”, and the terms “a,” “an,” and “the” can include singular and plural references. Furthermore, as used in this disclosure and the appended claims, the term “in a manner” may also mean “by” depending on the context. Furthermore, as used in this disclosure and the appended claims, the term “if” may also mean “when” or “at” depending on the context. Furthermore, as used in this disclosure and the appended claims, the word “and / or” may refer to and cover any combination and all possible combinations of one or more of the associated listed items. Furthermore, as used in this disclosure and the appended claims, the terms "real-time," "instant," etc., may refer to receiving, accessing, transmitting, providing, processing, and / or storing in real time or near real time (each of which applies). Detailed Implementation

[0030] Routine endoscopic procedures, including ERCP, are often prone to complications, even when performed by experienced surgeons. For example, challenges in performing ERCP include, but are not limited to: difficulties in properly positioning and orienting the duodenoscope anterior to the papilla or in choledochojejunostomy / pancreaticojejunostomy; difficulties in cannulating the biliary-pancreatic system; and difficulties in preventing postoperative complications. Surgeons have recognized that successful cannulation not only means accessing the required catheter but also performing it in the most efficient and safe manner while minimizing postoperative complications, especially post-ERCP pancreatitis (or PEP). A major drawback of routine ERCP techniques is its reliance on indirect visualization of the bile ducts. Furthermore, the diagnosis of biliary lesions (e.g., stones, strictures) and their anatomical location relies on imprecise targeted sampling and / or two-dimensional imaging, which provides poor sensitivity in diagnosing malignancies.

[0031] This example implementation generally relates to and / or includes systems, subsystems, processors, devices, logic, methods, and processes for solving routine problems, including those systems, subsystems, processors, devices, logic, methods, and processes described above and in this disclosure. As will be further described in this disclosure, this example implementation relates to surgical systems, subsystems, processors, devices, logic, methods, and processes for performing surgical actions, including but not limited to one or more of the following: capturing real-time images (including video images and / or still images) via image capture subsystem 210 and processing the real-time images captured by image capture subsystem 210; measuring real-time measurement results (also referred to herein as “IMU information”) via IMU subsystem 220 and processing the real-time measurement results measured by IMU subsystem 220 (i.e., real-time processing); performing real-time processing on images to determine whether an image includes Capturing an image of the catheter assembly 230 (e.g., determining whether the catheter assembly 230 is within the image capture view of the image capture subsystem 210); performing real-time image processing to determine whether the image includes the distal end 230a of the catheter assembly 230; real-time identification of the distal end 230a of the catheter assembly 230 in the image; generating (and / or drawing, adding, overlaying, and / or covering) a visible indicator of the distal end 230a of the catheter assembly 230 in the image to be displayed on the graphics display 304, etc.; when it is determined that the image includes the distal end 230a of the catheter assembly 230, generating (i.e., generating in real-time) the distal end of the catheter assembly 230. Real-time three-dimensional position (e.g., Cartesian coordinates) of end portion 230a; generation (i.e., real-time generation) of real-time depth information of the distal end portion 230a of catheter assembly 230 (e.g., depth or distance information between the distal end portion 230a of catheter assembly 230 and a reference point on main assembly 200); real-time image processing to determine whether the image includes a cannulation target (e.g., in the case of performing ERCP procedures, the cannulation target may be a papilla or papillary foramen leading to the common bile duct (CBD) and pancreaticobiliary duct); it should be understood that other procedures and other cannulation targets are also contemplated without departing from the teachings of this disclosure; real-time identification of cannulation targets in the image. ; Generate (i.e., generate in real time) (and / or draw, add, overlay and / or cover) visible indicators of the cannulation target in the image to be displayed on the graphics display 304; when it is determined that the image includes the cannulation target, generate (i.e., generate in real time) the real-time three-dimensional position (e.g., Cartesian coordinates) of the cannulation target; generate (i.e., generate in real time) the real-time depth information of the cannulation target (e.g., depth or distance information between the cannulation target and a reference point on the main component 200); when it is determined that the image includes at least the distal end 230a of the catheter assembly 230, generate (i.e., generate in real time) the predicted real-time trajectory path 20 of the distal end 230a of the catheter assembly 230;When the image is determined to include at least the cannulation target, a predicted real-time post-cannulation trajectory path 10 for the cannulation target is generated (i.e., generated in real time); when the image is determined to include both the distal end 230a of the catheter assembly 230 and the cannulation target, a predicted real-time trajectory path 20 for cannulation of the distal end 230a of the catheter assembly 230 to the cannulation target and / or a predicted real-time post-cannulation trajectory path 10 is generated (i.e., generated in real time); when the image is determined to include at least the distal end 230a of the catheter assembly 230, the image for the distal end 230a of the catheter assembly 230 is displayed (i.e., displayed in real time) on the graphic display. Predicting a real-time trajectory path 20; when the image is determined to include at least the cannulation target, displaying (i.e., in real-time) the predicted real-time post-cannulation trajectory path 10 for the cannulation target on a graphic display; when the image is determined to include both the distal end 230a of the catheter assembly 230 and the cannulation target, displaying (i.e., in real-time) the predicted real-time trajectory path 20 and / or the predicted real-time post-cannulation trajectory path 10 for cannulation of the cannulation target by the distal end 230a of the catheter assembly 230 on a graphic display; and / or generating real-time depth information between the cannulation target and the distal end 230a of the catheter assembly 230.

[0032] Although the exemplary embodiments described in this disclosure may be primarily directed to biliary and pancreatic endoscopy, it should be understood that the exemplary embodiments may also be directed to and / or applied to other forms of endoscopy, including but not limited to bronchoscopy, colonoscopy, colposcopy, cystoscopy, esophagogastric and duodenal endoscopy (EGD), laparoscopy, laryngoscopy, proctoscopy, thoracoscopy, and bronchoscopy.

[0033] Example embodiments will now be described with reference to the accompanying drawings, which form part of this disclosure.

[0034] Example implementation of a surgical system (e.g., surgical system 100) for performing endoscopic examinations.

[0035] Figure 1A An example implementation of a surgical system (e.g., surgical system 100) for performing surgical procedures such as endoscopic procedures is illustrated. For example, as... Figure 1B As illustrated, the surgical system 100 can be configured to perform endoscopic retrograde cholangiopancreatography (ERCP) and pancreatography. It should be understood in this disclosure that exemplary embodiments of the surgical system 100 can also be configured to perform other surgical procedures, including but not limited to bronchoscopy, colonoscopy, colposcopy, cystoscopy, endoscopic esophago-gastroduodenoscopy (EGD), laparoscopy, laryngoscopy, proctoscopy, thoracoscopy, and bronchoscopy.

[0036] To perform the actions, functions, processes, and / or methods described above and in this disclosure, example embodiments of the surgical system 100 include one or more elements. For example, the surgical system 100 includes a main assembly (e.g., main assembly 200) disposed at a distal end of the surgical system 100 (e.g., distally relative to the surgeon's console 300). The main assembly 200 includes a proximal end 200b and a distal end 200a. The main assembly 200 can be configured or constructed to be inserted into a patient's cavity (e.g., through the patient's mouth and inserted into a second portion of the patient's duodenum).

[0037] The surgical system 100 also includes an elongated tubular member (e.g., an elongated tubular member 50) or the like, having a proximal end 50b and a distal end 50a. The distal end 50a of the elongated tubular member 50 can be configured or configured to be bent in one or more of a plurality of directions. This bending of the distal end 50a of the elongated tubular member 50 can be controlled by a surgeon via a surgical console or controller (e.g., surgical console 300). The distal end 50a of the elongated tubular member 50 can be secured or attached to the proximal end 200b of the main assembly 200.

[0038] The surgical system 100 also includes a surgeon's console (e.g., surgeon's console 300) connected to the proximal end 50b of the elongated tubular member 50. The surgeon's console 300 may include, be part of, and / or communicate with the following: one or more processors (e.g., processor 310), one or more surgeon controls (e.g., surgeon control 302), one or more graphic displays (e.g., graphic display 304), one or more communication channels or networks (e.g., communication channel 320 or network 320), and / or one or more databases (e.g., database 330).

[0039] As used in this disclosure, references to surgical system 100 (and / or one or more of its components), surgeon console 300 (and / or one or more of its components), and / or processor 310 (and / or one or more of its components) may also refer to, apply to, and / or include: one or more computing devices, processors, servers, systems, cloud-based computing, etc., and / or the functionality of one or more processors, computing devices, servers, systems, cloud-based computing, etc. The surgical system 100 (and / or one or more of its components), the surgeon's console 300 (and / or one or more of its components), and / or the processor 310 (and / or one or more of its components) may be or have any processor, server, system, device, computing device, controller, microprocessor, microcontroller, microchip, semiconductor device, etc., capable of being configured or configured to perform, in particular, the following: 3D positioning determination and control of one or more components of the surgical system 100; 3D orientation determination and control of one or more components of the surgical system 100; speed, velocity... Rate, vector and / or acceleration determination and control; determination and control of motion direction of one or more components of the surgical system 100; image capture (including video image and still image capture); image processing (including automatic image processing of video images and still images); feature extraction (including feature extraction from video images and still images); 3D location determination of the patient's nipple, opening or other site; depth estimation, determination and control (including 3D depth estimation, determination and control); trajectory path prediction, generation and control (including 3D trajectory path prediction, generation and control); and / or any one or more other actions, functions, methods and / or processes described above and in this disclosure. Alternatively or additionally, the surgical system 100 (and / or one or more of its components), the surgeon's console 300 (and / or one or more of its components), and / or the processor 310 (and / or one or more of its components) may include and / or be a part of the following: virtual machines, processors, computers, nodes, instances, hosts, or machines, including those virtual machines, processors, computers, nodes, instances, hosts, or machines in a networked computing environment.

[0040] As used in this disclosure, communication channel 320, network 320, cloud 320, etc., can be a collection of devices and / or virtual machines connected by a communication channel that facilitates communication between devices and allows devices to share resources, or include such a collection. Such resources can include any type of resources used to run instances, including hardware (e.g., servers, clients, mainframes, networks, network storage, data sources, memory, central processing unit time, scientific instruments, and other computing devices), and software, software licenses, available network services, and other non-hardware resources, or combinations thereof. Communication channel 320, network 320, cloud 320, etc., can include, but are not limited to, computing grid systems, peer-to-peer systems, network systems, distributed computing environments, cloud computing environments, video / image communication channels, etc. These communication channels 320, network 320, cloud 320, etc., can include hardware and / or software infrastructure configured to form a virtual organization composed of multiple resources, which may or may not be geographically dispersed. Communication channel 320, network 320, cloud 320, etc., can also refer to communication media between processes on the same device. As noted herein, processor 310, network element, node, or server may be a device deployed to execute a program as a socket listener and may include software instances.

[0041] These and other components of the surgical system 100 will now be described further with reference to the accompanying drawings.

[0042] Elongated tubular members (e.g., elongated tubular member 50).

[0043] At least as Figure 1AAs illustrated, the surgical system 100 includes an elongated tubular component (e.g., an elongated tubular component 50). The elongated tubular component 50 includes a proximal end 50b and a distal end 50a. The elongated tubular component 50 can be configured or constructed to cooperate with, communicate with, and / or be controlled and / or managed by the following to perform the various actions and / or functions described in this disclosure: a surgeon and / or one or more elements of the surgical system 100, including a surgeon's console 300, a surgeon's control unit 302, a graphic display 304, a processor 310, a network 320, and / or a database 330. For example, the elongated tubular component 50 can be configured or constructed to advance the main component 200 through the patient's cavity (e.g., through the patient's mouth and into a second part of the patient's duodenum). The elongated tubular member 50 can also be configured or arranged to allow the main assembly 200 (attached to the distal end 50a of the elongated tubular member 50) to advance around the curved, looped and / or bent sections of the patient's cavity by selectively controlling the bending position, bending angle, bending direction and / or bending location of at least a portion of the distal end 50a of the elongated tubular member 50.

[0044] To perform the actions, functions, processes, and / or methods described above and in this disclosure, example embodiments of the elongated tubular member 50 include one or more elements. For example, the elongated tubular member 50 includes one or more internal channels. These internal channels may be configured for the passage and / or reception of communication / data cables, including: cables for transmitting video images and / or still images from the image capture subsystem 210 to the processor 310 of the surgical system 100 and / or one or more other elements; and / or cables for transmitting IMU measurement results from the IMU subsystem 220 to the processor 310 of the surgical system 100 and / or one or more other elements. Alternatively or additionally, these internal channels may be configured for the passage and / or reception of power cables (e.g., for providing power to the image capture subsystem 210, a lighting source (not shown), and / or the IMU subsystem 220). Alternatively or additionally, these internal channels may be configured for the passage and / or reception of the proximal end of the conduit assembly 230 (not shown) and / or other control members (not shown) that can be configured or arranged to control the movement and / or orientation of the conduit assembly 230 (e.g., forward movement or extension; backward movement or retraction; bending position, bending angle and / or bending direction of at least a portion of the distal end 230a of the conduit assembly 230, etc.). Alternatively or additionally, these internal channels may be configured for the passage and / or reception of cables, wires, etc., for controlling the bending position, bending angle and / or bending direction of at least a portion of the distal end 50a of the elongated tubular member 50. The distal end 50a of the elongated tubular member 50 may include a plurality of bending members 50c or segments 50c that receive and / or secure to one or more cables, wires, etc., which control the bending position, bending angle, and / or bending direction of at least a portion of the distal end 50a of the elongated tubular member 50. In some example embodiments, the distal end 50a of the elongated tubular member 50 may also include one or more inflatable members (or balloons) (not shown) and / or one or more negative pressure openings (not shown) that can be used to anchor the distal end 50a of the elongated tubular member 50 to the inner wall of a patient's cavity (e.g., when the main assembly 200 has achieved an ideal target and / or preferred 3D position and orientation, such as anterior to the nipple, and the surgical system 100 is ready to extend the catheter assembly 230 to perform nipple cannulation).

[0045] Although the actions, functions, processes, and / or methods performed by the elongated tubular member 50 may be described in this disclosure as being performed by one or more specific elements of the elongated tubular member 50, without departing from the teachings of this disclosure, the actions, functions, processes, and / or methods performed by specific elements of the elongated tubular member 50 may also be performed by one or more other elements and / or by more than one element of the elongated tubular member 50 (and / or other elements of the surgical system 100) in concert. It should also be understood in this disclosure that, although the actions, functions, processes, and / or methods performed by the elongated tubular member 50 may be described in this disclosure as being performed by specific elements of the elongated tubular member 50, without departing from the teachings of this disclosure, the actions, functions, processes, and / or methods performed by two or more specific elements of the elongated tubular member 50 may also be combined and performed by one element of the elongated tubular member 50.

[0046] Main component (e.g., main component 200).

[0047] As illustrated in at least Figures 1 to 4, a surgical system 100 capable of being configured or configured to perform surgical actions including but not limited to endoscopic retrograde cholangiopancreatography (ERCP) cannulation includes one or more main components (e.g., main component 200). The main component 200 includes a proximal end portion 200b and a distal end portion 200a. The proximal end portion 200b of the main component 200 is attached to the distal end portion 50a of an elongated tubular member 50. The main component 200 can be configured or configured to cooperate with, communicate with, and / or be controlled and / or managed by one or more elements of the surgical system 100 to perform various actions and / or functions, wherein one or more elements of the surgical system 100 include a surgeon's console 300, a surgeon's control unit 302, a graphic display 304, a processor 310, a network 320, and / or a database 330.

[0048] For example, the main component 200 can be configured to perform 3D positioning determination and control of one or more elements of the surgical system 100 (e.g., determination and control of 3D positioning (e.g., in a Cartesian coordinate system, etc.) of the catheter assembly 230 and / or the distal end 50a of the elongated tubular member 50). As another example, the main component 200 can be configured to perform 3D orientation determination and control of one or more elements of the surgical system 100 (e.g., determination and control of 3D orientation of the catheter assembly 230 and the distal end 50a of the elongated tubular member 50). As another example, the main component 200 can be configured to perform velocity, rate, and / or acceleration determination and control of one or more elements of the surgical system 100 (e.g., determination and control of velocity, rate, and / or acceleration of the catheter assembly 230 and the distal end 50a of the elongated tubular member 50). As another example, the main component 200 can be configured to perform motion direction determination and control of one or more components of the surgical system 100 (e.g., determination and control of motion direction of the catheter assembly 230 and the distal end 50a of the elongated tubular member 50). As another example, the main component 200 can be configured to perform image capture (including capturing video images and still images via the image capture subsystem 210). As another example, the main component 200 can be configured to perform image processing (including automatic image processing of video images and still images via the processor 310 and / or network 320). As another example, the main component 200 can be configured or configured to perform feature recognition, extraction, classification, and / or size estimation (including, particularly, feature recognition, extraction, classification, and / or size estimation via processor 310 and / or network 320 based on historical and / or real-time video images, still images, and / or measurements from the IMU subsystem 220, such features as nipples, cavity openings, cavity walls, lesions, distal ends 230a of catheter assembly 230, etc.). As another example, the main component 200 can be configured or configured to perform 3D position estimation and / or determination of features (e.g., in Cartesian coordinates, etc.) (including, particularly, 3D position estimation and / or determination of features via processor 310 and / or network 320 based on historical and / or real-time video images, still images, and / or measurements from the IMU subsystem 220, such features as a patient's nipples, cavity openings, cavity walls, lesions, and / or other sites). As another example, the main component 200 can be configured or configured to perform 3D depth estimation, determination and control (including, particularly based on historical and / or real-time video images, still images and / or IMU subsystem 220 measurements via processor 310 and / or network 320, features such as nipples, lumen openings, lumen walls, lesions, distal end 230a of catheter assembly 230, etc.).As another example, the main component 200 can be configured or configured to perform: trajectory path prediction, determination, generation and control (including, in particular, 3D trajectory path prediction, generation and control of the duct assembly 230 via processor 310 and / or network 320 based on historical and / or real-time video images, still images and / or IMU subsystem 220 measurements); and / or any one or more other actions, functions, methods and / or processes described above and in this disclosure.

[0049] To perform the actions, functions, processes, and / or methods described above and in this disclosure, example embodiments of the main component 200 include one or more elements. For example, such as Figure 2A , Figure 2B , Figure 2C and Figure 2D 3D diagram and Figure 3A and Figure 3B As illustrated in the cross-sectional side view, the main component 200 includes a body 200', a proximal end portion 200b, and a distal end portion 200a. The main component 200 also includes one or more image capture subsystems 210. For example, such as at least Figures 2A to 2D as well as Figures 3A to 3B As shown, the main component 200 may include a single image capture subsystem 210 (e.g., a monocular duodenoscope, etc.) housed within the body 200'. As another example, Figure 4A As illustrated, the main component 200 may include two (or more) image acquisition subsystems 210 housed within the main body 200'. The main component 200 also includes one or more inertial measurement unit (IMU) subsystems 220. For example, such as at least Figures 2C to 2D as well as Figures 3A to 3B As shown, the main component 200 may include a single IMU subsystem 220 housed within the main body 200'. As another example, Figure 4B As illustrated, the main component 200 may include two (or more) inertial measurement unit (IMU) subsystems 210 housed within the main body 200'. The main component 200 also includes one or more conduit assemblies 230. For example, such as at least Figures 2A to 2D as well as Figures 3A to 3B As illustrated, the main component 200 may include a single catheter assembly 230 extending outward from the body 200' (such as at least...). Figure 2B , Figure 2D and Figure 3C (As shown in the illustration).

[0050] Although the actions, functions, processes, and / or methods performed by the main component 200 may be described in this disclosure as being performed by one or more specific elements of the main component 200, without departing from the teachings of this disclosure, the actions, functions, processes, and / or methods performed by specific elements of the main component 200 may also be performed by one or more other elements and / or by more than one element of the main component 200 (and / or other elements of the surgical system 100) in coordination. It should also be understood in this disclosure that, although the actions, functions, processes, and / or methods performed by the main component 200 are described in this disclosure as being performed by specific elements of the main component 200, without departing from the teachings of this disclosure, the actions, functions, processes, and / or methods performed by two or more specific elements of the main component 200 may also be combined and performed by one element of the main component 200.

[0051] These elements of the main component 200 will now be described further with reference to the accompanying drawings.

[0052] Image capture subsystem (e.g., image capture component 210 or image capture subsystem 210).

[0053] In an example implementation, the main component 200 includes one or more image capture subsystems (e.g., image capture assembly 210 or image capture subsystem 210). As illustrated in at least Figures 1 through 4, the image capture subsystem 210 is housed within the body 200' of the main component 200, and the body 200' includes an opening for each camera 210 of the image capture subsystem 210. The image capture subsystem 210 can be any image capture device or system capable of being configured or configured to capture video images and / or still images. For example, the image capture subsystem 210 may include a single camera 210 for capturing a single view of video images and / or still images (e.g., as in a monocular duodenoscope).

[0054] The image capture subsystem 210 can be configured to communicate wired and / or wirelessly with one or more other components of the surgeon's console 300, surgeon's control unit 302, graphic display 304, processor 310, network 320, database 330, and / or surgical system 100. For example, in operation, the image capture subsystem 210 is configured to transmit real-time video images and / or still images (also referred to as "images" in this disclosure) to the processor 310, which processes the images in real time (this may include processing data received from the IMU subsystem 220, actions detected by the processor 310 performed by the surgeon via the surgeon's control unit 302 and / or console 300, and / or historical information from one or more databases 330) and displays the images on the graphic display 304 for the surgeon to view. As will be further described in this disclosure, the processing performed by processor 310 can be executed independently or in conjunction with one or more networks 320, databases 330, and / or other processors (not shown), for example, in such an example embodiment, one or more actions or functions of surgical system 100 (e.g., including but not limited to one or more of the following: image preprocessing, processing, and display; feature recognition, extraction, classification, and / or size estimation; 3D position determination; 3D orientation determination; velocity, rate, and / or acceleration determination; motion direction determination; 3D depth estimation; trajectory path prediction, determination, and control) are performed by one or more elements of surgical system 100 based on the following: such processing performed by processor 310 and / or calculations, estimates, results, inferences, predictions, etc., generated and / or derived directly or indirectly, partially, collaboratively, or holistically via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these AI algorithms, engines, systems, processors, and / or processes may be provided locally via processor 310 (and / or processor 310 and / or one or more components of surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0055] Although exemplary embodiments of the image capture subsystem 210 are primarily described in this disclosure as having one camera 210 (see, for example, Figures 1 to 3), it should be understood that exemplary embodiments of the image capture subsystem 210 may include two (or more) cameras 210 (see, for example...). Figure 4A Thus, the example implementation having two or more cameras 210 is considered in this disclosure to improve the ability to generate 3D views and depth estimates on the graphics display 304, because two (or more) cameras 210 can capture two (or more) slightly different views of video images and / or still images. However, it is recognized in this disclosure that the example implementation having a single camera 210 for the main component 200 also offers advantages over the example implementation having two (or more) cameras 210, including but not limited to reductions in overall hardware size, weight, cost, and the size of one or more main components 200.

[0056] Inertial measurement unit (IMU) subsystem (e.g., IMU component 220 or IMU subsystem 220).

[0057] In an example implementation, the main component 200 includes one or more inertial measurement unit (IMU) components (e.g., IMU component 220 or IMU subsystem 220). Such as at least Figures 3A to 3B As illustrated, the IMU subsystem 220 is housed within the body 200' of the main component 200. The IMU subsystem 220 can be any inertial measurement device or system capable of being configured or configured to perform measurements and / or readings of, in particular, specific force, angular rate, position, and / or orientation using one or more accelerometers, gyroscopes, and / or magnetometers, EM trackers (not shown). For example, the IMU subsystem 220 may comprise a single IMU device 220 having one or more accelerometers, gyroscopes, magnetometers, and / or EM trackers (not shown).

[0058] The IMU subsystem 220 can be configured to communicate wired and / or wirelessly with one or more other components of the surgeon's console 300, surgeon's control unit 302, graphic display 304, processor 310, network 320, database 330, and / or surgical system 100. For example, in operation, the IMU subsystem 220 is configured to transmit real-time measurement results and / or readings to the processor 310, which processes the measurement results and / or readings in real time (this may include processing images received from the image capture subsystem 210, actions detected by the processor 310 performed by the surgeon via the surgeon's control unit 302 and / or console 300, and / or historical information from one or more databases 330). The IMU subsystem 220 can also provide these IMU measurement results to the graphic display 304 for display by the surgeon. As will be further described in this disclosure, the processing performed by processor 310 can be executed independently or in conjunction with one or more networks 320, databases 330, and / or other processors (not shown), for example, in such an example embodiment, one or more actions or functions of surgical system 100 (e.g., including but not limited to one or more of the following: image preprocessing, processing, and display; feature recognition, extraction, classification, and / or size estimation; 3D position determination; 3D orientation determination; velocity, rate, and / or acceleration determination; motion direction determination; 3D depth estimation; trajectory path prediction, determination, and control) are performed by one or more elements of surgical system 100 based on the following: such processing performed by processor 310 and / or calculations, estimates, results, inferences, predictions, etc., generated and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these AI algorithms, engines, systems, processors, and / or processes may be provided locally via processor 310 (and / or processor 310 and / or one or more components of surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0059] Although exemplary embodiments of the IMU subsystem 220 are primarily described in this disclosure as having an IMU device 220 (see, for example) Figures 3A to 3BHowever, it should be understood that exemplary implementations of the IMU subsystem 220 may include two (or more) IMU devices 220 (see, for example...). Figure 4B Such example embodiments having two or more IMU devices 220 are considered in this disclosure to improve the reliability of generating accurate and reliable measurement results and readings, including the ability to provide backup measurement results and readings in the event that one IMU device 220 is not functioning properly (or not working at all). However, it is recognized in this disclosure that example embodiments of the IMU subsystem 220 having a single IMU device 220 for the main component 200 also offer advantages over example embodiments of the IMU subsystem 220 having two (or more) IMU devices 220, including, but not limited to, reductions in overall size, weight, and one or more dimensions of the main component 200.

[0060] Surgeon console (e.g., Surgeon console 300).

[0061] Such as at least Figure 1A The illustrated surgical system 100, configurable or configured to perform surgical actions including but not limited to endoscopic retrograde cholangiopancreatography (ERCP) cannulation, includes one or more surgeon consoles (e.g., surgeon console 300). Surgeon console 300 can be configured or configured to specifically provide an interface to one or more surgeons for viewing, controlling, and managing the surgical system 100. Surgeon console 300 can be configured or configured to cooperate with, communicate with, be attached to, and / or control and / or manage one or more components of the surgical system 100, including an elongated tubular member 50 and a main assembly 200. The surgeon's console 300 may include and / or communicate with one or more other components of the surgical system 100, including one or more surgeon control units 302, one or more graphic displays 304, and / or one or more processors 310. The surgeon's console 300 may also include one or more networks 320 and / or one or more databases 330 and / or communicate with one or more networks 320 and / or one or more databases 330.

[0062] Although these diagrams may illustrate a surgeon control unit 302, a graphics display 304, a processor 310, a network 320, and a database 330, it should be understood that, without departing from the teachings of this disclosure, the surgical system 100 may include more than one or fewer surgeon control units 302, more than one or fewer graphics displays 304, more than one or fewer processors 310, more than one or fewer networks 320, and more than one or fewer databases 330.

[0063] These elements of the surgeon console 300 and / or those communicating with the surgeon console 300 will now be described further with reference to the accompanying drawings.

[0064] Surgical control unit (e.g., surgical control unit 302).

[0065] As illustrated in Figure 1, an example embodiment of the surgeon's console 300 includes one or more surgeon controls (e.g., surgeon control 302) and / or communication with one or more surgeon controls. Surgeon control 302 can be configured or configured to enable the surgeon to control one or more elements, aspects, and / or actions of the elongated tubular member 50 and / or the main assembly 200.

[0066] For example, the surgeon's control unit 302 may include a joystick, a bendable member, a touchscreen, a button, a scroll wheel, a trackball, a knob, a lever, and / or any other actuable controller to allow the surgeon to selectively control the bending position, bending angle, and / or bending direction of at least a portion of the distal end 50a of the elongated tubular member 50. As another example, the surgeon's control unit 302 may include a joystick, a bendable member, a touchscreen, a button, a scroll wheel, a trackball, a knob, a lever, and / or any other actuable controller to allow the surgeon to selectively control the length (e.g., the body 200' extending outward away from the main assembly 200, the body 200' retracting inward toward the main assembly 200, fully retracted and accommodated within the body 200' of the main assembly 200, etc.), bending position, bending angle, and / or bending direction of the catheter assembly 230. The surgeon control unit 302 may also include a joystick, touchscreen, buttons, scroll wheel, trackball, knob, joystick, and / or any other controller for the surgeon to selectively control the operation of the image capture assembly 210 (and / or each camera 210 of the image capture assembly 210 having two or more cameras 210 in the example embodiment), including capturing images and not capturing images, as well as other functions / features, such as controlling the focus, zoom, illumination, exposure, etc. of the camera 210 of the image capture assembly 210. The surgeon control unit 302 may also include buttons, toggle switches, touchscreen, and / or any other controller for the surgeon to selectively control the operation of a lighting source (not shown), including the brightness, color, direction, etc. of the lighting source. The surgeon control unit 302 may also include buttons, toggle switches, touchscreen, and / or any other controller for the surgeon to selectively control the operation of the IMU subsystem 220 (and / or each IMU device 220 of the IMU subsystem 220 having two or more IMU devices 220 in the example embodiment).

[0067] In an example implementation, surgeon actions performed on surgeon control unit 302 may be captured and transmitted by processor 310, network 320, database 330, and / or one or more other elements of surgical procedure 100, and / or surgeon actions performed on surgeon control unit 302 may be transmitted to (if applicable) processor 310, network 320, database 330, and / or one or more other elements of surgical procedure 100. For example, processor 310 (and / or network 320 and / or database 330) may use such information when performing processes such as image processing and display; feature recognition, extraction, classification, and / or size estimation; 3D position determination; 3D orientation determination; velocity, rate, and / or acceleration determination; motion direction determination; 3D depth estimation; trajectory path prediction, determination, and control. As another example, processor 310 (and / or network 320) may store such information in database 330 for future processing use (e.g., as historical information in future processing).

[0068] Graphics display (e.g., graphics display 304).

[0069] like Figure 1A As illustrated, an example implementation of the surgeon's console 300 includes one or more graphic displays (e.g., graphic display 304) and / or communicates with one or more graphic displays. Graphic display 304 can be configured to display real-time images captured by image capture subsystem 210, which may be provided directly from image capture subsystem 210 and / or via processor 310. Graphic display 304 can also be configured to display real-time 3D constructed or reconstructed images, data, and / or depth information of views / images captured by image capture subsystem 210 based on the processor 310's processing of images captured by image capture subsystem 210 and measurement results / readings of IMU subsystem 220 (and may also include historical information, including historically captured images and / or historical measurement results / readings).

[0070] The real-time image displayed on the graphic display 304 may also include real-time visual indicators (e.g., bounding boxes, etc.) of one or more features (e.g., nipple, cavity opening, cavity wall, lesion, distal end 230a of catheter assembly 230, etc.) identified, extracted, classified, and / or sized based on feature extraction processing. The real-time image displayed on the graphic display 304 may also include real-time 3D position indications of one or more components of the surgical system 100. The real-time image displayed on the graphic display 304 may also include real-time 3D orientation indications of one or more components of the surgical system 100 (e.g., orientation of the main assembly 200, including the orientation of catheter assembly 230). The real-time image displayed on the graphic display 304 may also include indications of velocity, rate, and / or acceleration of one or more components of the surgical system 100 (e.g., velocity, rate, and / or acceleration of the main assembly 200, including the velocity, rate, and / or acceleration of catheter assembly 230). The real-time image displayed on the graphic display 304 may also include indications of the direction of motion of one or more components of the surgical system 100 (e.g., the direction of motion of the main assembly 200, including the direction of motion of the catheter assembly 230). The real-time image displayed on the graphic display 304 may also include indications of the real-time 3D position of one or more features (e.g., nipple, cavity opening, cavity wall, lesion, distal end 230a of catheter assembly 230, etc.) identified, extracted, classified, and / or sized based on feature extraction processing. The real-time image displayed on the graphic display 304 may also include indications of 3D depth estimation based on one or more features (e.g., nipple, cavity opening, cavity wall, lesion, distal end 230a of catheter assembly 230, etc.) identified, extracted, classified, and / or sized based on feature extraction processing. As will be further described in this disclosure, the real-time image displayed on the graphic display 304 may include a trajectory path 20 of the distal end 230a of the catheter assembly 230, predicted, inferred, and / or estimated in real-time in 3D based on the current position and / or orientation of the distal end 230a of the catheter assembly 230. The real-time image displayed on the graphic display 304 may also include a post-cannulation trajectory path 10, predicted, inferred, and / or estimated in real-time in 3D for the cannulation target. The real-time image displayed on the graphic display 304 may also include a trajectory path 20 and a post-cannulation trajectory path 10, predicted, inferred, and / or estimated in real-time in 3D to ensure successful arrival of the distal end 230a of the catheter assembly 230 at the identified target or feature (e.g., successful cannulation of the nipple during ERCP cannulation).

[0071] Processor (e.g., processor 310).

[0072] Such as at least Figure 1AAs illustrated, the surgical system 100 includes one or more processors (e.g., processor 310). Processor 310 can be configured to perform, in particular, information processing, communication between components of the surgical system 100, and control and management of the components of the surgical system 100. For example, processor 310 can be configured to receive real-time images captured by image capture subsystem 210. Processor 310 can also be configured to receive real-time measurements (also referred to herein as IMU information) from IMU subsystem 220.

[0073] Upon receiving an image and / or IMU information, the processor 310 can be configured to process the image to determine whether the image includes the conduit assembly 230 (e.g., determining whether the conduit assembly 310 is within the image capture view of the image capture subsystem 210). The processor 310 can also be configured to process the image to determine whether the image includes the distal end 230a of the conduit assembly 230. Such image processing may include, in conjunction with, and / or apply automated image processing and / or artificial intelligence algorithms (e.g., machine learning and / or deep learning algorithms), which receive the image from the image capture subsystem 210 as input and provide feature detection (i.e., identification of the distal end 230a of the conduit assembly 230 in the image) as output. Therefore, the processor 310 can be configured to identify the distal end 230a of the conduit assembly 230 in the image. In some example implementations, processor 310 may also be configured to generate a visible indicator of the distal end 230a of catheter assembly 230 in an image to be displayed on graphics display 304. When processor 310 determines that the image includes the distal end 230a of catheter assembly 230, processor 310 may also be configured to generate a real-time 3D position (e.g., Cartesian coordinates) of the distal end 230a of catheter assembly 230. Processor 310 may also be configured to generate real-time depth information of the distal end 230a of catheter assembly 230 (e.g., depth information between the distal end 230a of catheter assembly 230 and a reference point on main assembly 200, such as the central axis of image capture assembly 210).

[0074] In addition to processing the image to identify the distal end 230a of the catheter assembly 230, the processor 310 can also be configured to process the image to determine whether the image includes a cannulation target. For example, during an ERCP cannulation procedure, the cannulation target may be a papilla (or a papillary orifice viewed from the duodenum) leading to the common bile duct (CBD) and pancreatic duct. It should be understood that other procedures and other cannulation targets are also contemplated without departing from the teachings of this disclosure. Such processing of the image may include, in conjunction with, and / or using automated image processing and / or artificial intelligence algorithms (e.g., machine learning and / or deep learning algorithms) that receive the image as input from the image capture subsystem 210 and provide feature detection (i.e., identification of the cannulation target in the image) as output. Therefore, the processor 310 can also be configured to identify the cannulation target in the image. In some example implementations, processor 310 can also be configured to generate visible indicators (e.g., bounding boxes) of the intubation target in an image to be displayed on graphics display 304. When processor 310 determines that the image includes the intubation target, processor 310 can also be configured to generate the real-time 3D position (e.g., Cartesian coordinates) of the intubation target. Processor 310 can also be configured to generate real-time depth information of the intubation target (e.g., depth information between the intubation target and a reference point on main component 200, such as the central axis of image capture component 210).

[0075] When processor 310 determines that the image includes at least the distal end 230a of catheter assembly 230, processor 310 can be configured or configured to generate (i.e., generate in real time) a predicted real-time trajectory path 20 of the distal end 230a of catheter assembly 230. When processor 310 determines that the image includes at least the cannulation target, processor 310 can also be configured or configured to generate (i.e., generate in real time) a predicted real-time post-cannulation trajectory path 10 for the cannulation target. When processor 310 determines that the image includes both the distal end 230a of catheter assembly 230 and the cannulation target, processor 310 can also be configured or configured to generate (i.e., generate in real time) a predicted real-time trajectory path 20 for cannulation of the distal end 230a of catheter assembly 230 to the cannulation target and / or a predicted real-time post-cannulation trajectory path 10. The generation of predicted trajectory paths 10 and 20 may include, in conjunction with, and / or apply the following: automatic image processing, automatic IMU information processing, and / or artificial intelligence algorithms (e.g., machine learning and / or deep learning algorithms). The automatic image processing, automatic IMU information processing, and / or artificial intelligence algorithms receive real-time images from the image capture subsystem 210, real-time IMU information from the IMU component 220, historical images from the database 330 and / or the network 320, and / or historical IMU information from the database 330 and / or the network 320 as inputs, and provide predicted trajectory path 20 and / or predicted post-cannula trajectory path 10 as outputs.

[0076] When processor 310 determines that an image includes at least the distal end 230a of catheter assembly 230, once predicted trajectory path 20 and / or predicted post-cannulation trajectory path 10 are generated, processor 310 can be configured or configured to display (i.e., in real-time) the predicted real-time trajectory path 20 of the distal end 230a of catheter assembly 230 on a graphical display. When processor 310 determines that an image includes at least a cannulation target, processor 310 can also be configured or configured to display (i.e., in real-time) the predicted real-time post-cannulation trajectory path 10 for the cannulation target on a graphical display. When processor 310 determines that an image includes both the distal end 230a of catheter assembly 230 and the cannulation target, processor 310 can also be configured or configured to display (i.e., in real-time) the predicted real-time trajectory path 20 and / or the predicted real-time post-cannulation trajectory path 10 for cannulation of the cannulation target by the distal end 230a of catheter assembly 230 on a graphical display. The processor 310 can also be configured to generate real-time depth information between the distal end 230a of the catheter assembly 230 and the distal end 230a of the catheter assembly 230, particularly based on the 3D position of the distal end 230a of the catheter assembly 230 and the cannulation target.

[0077] To perform the actions, functions, processes, and / or methods described above and in this disclosure, example embodiments of processor 310 include one or more elements. For example, such as Figure 5A As illustrated, processor 310 includes one or more image capture subsystem interfaces 311. Processor 310 also includes one or more IMU subsystem interfaces 312. Processor 310 also includes one or more catheter assembly processors 314. Processor 310 also includes one or more cannulation target processors 316. Processor 310 also includes one or more trajectory path processors 318.

[0078] Although the actions, functions, processes, and / or methods performed by processor 310 may be described in this disclosure as being performed by one or more specific elements of processor 310, without departing from the teachings of this disclosure, the actions, functions, processes, and / or methods performed by specific elements of processor 310 may also be performed by one or more other elements and / or by more than one element of processor 310 (and / or other elements of surgical system 100) in coordination. It should also be understood in this disclosure that, although the actions, functions, processes, and / or methods performed by processor 310 may be described in this disclosure as being performed by specific elements of processor 310, without departing from the teachings of this disclosure, the actions, functions, processes, and / or methods performed by two or more specific elements of processor 310 may be combined and performed by one element of processor 310.

[0079] These components of the processor 310 will now be described further with reference to the accompanying drawings.

[0080] Image capture subsystem interface (e.g., image capture subsystem interface 311).

[0081] Such as at least Figure 5A As illustrated, processor 310 includes one or more image capture subsystem interfaces (e.g., image capture subsystem interface 311). Image capture subsystem interface 311 can be configured or configured to perform communication with image capture subsystem 210. For example, image capture subsystem interface 311 receives real-time images captured by image capture subsystem 210. Image capture subsystem interface 311 can also receive historical images captured by image capture subsystem 210 (and / or other surgical system 100), such as historical images of successful procedures (e.g., successful ERCP cannulation procedures) and / or historical images of unsuccessful procedures. These historical images can be received from database 330 and / or network 320.

[0082] Images received by the image capture subsystem interface 311 (including real-time images received from the image capture subsystem 210 and / or historical images received from the database 330 and / or the network 320) are then selectively provided or sent to the catheter assembly processor 314 and the cannulation target processor 316 for further processing.

[0083] In an example implementation, real-time images captured by the image capture subsystem 210 may also be provided in real-time to the graphics display 304 (and network 320 and / or database 330) (i.e., in addition to providing real-time images to the image capture subsystem interface 311 (or in parallel with providing real-time images to the image capture subsystem interface 311)). Alternatively or additionally, real-time images captured by the image capture subsystem 210 may be provided to the image capture subsystem interface 311, which then selectively sends the real-time images to the graphics display 304, network 320, database 330, catheter assembly processor 314, and / or cannulation target processor 316.

[0084] IMU subsystem interface (e.g., IMU subsystem interface 312).

[0085] Such as at least Figure 5A As illustrated, processor 310 includes one or more IMU subsystem interfaces (e.g., IMU subsystem interface 312). IMU subsystem interface 312 can be configured or configured to perform communication with IMU subsystem 220. For example, IMU subsystem interface 312 receives real-time measurement results obtained by IMU subsystem 220. IMU subsystem interface 312 can also receive historical measurement results obtained by IMU subsystem 220 (and / or other surgical system 100), such as historical measurement results of successful procedures (e.g., successful ERCP cannulation procedures) and / or historical measurement results of unsuccessful procedures. These historical measurement results can be received from database 330 and / or network 320.

[0086] Measurement results received by the IMU subsystem interface 312 (including real-time measurement results received from the IMU subsystem 220 and / or historical measurement results received from the database 330 and / or the network 320) are then selectively provided or sent to the catheter assembly processor 314 and the cannulation target processor 316 for further processing.

[0087] In an example implementation, real-time measurement results obtained by IMU subsystem 220 may also be provided in real-time to network 320 and / or database 330 (i.e., in addition to providing real-time measurement results to IMU subsystem interface 311 (or in parallel with providing real-time measurement results to IMU subsystem interface 311)). Alternatively or additionally, real-time measurement results obtained by IMU subsystem 220 may be provided to IMU subsystem interface 312, which then selectively sends the real-time measurement results to network 320, database 330, catheter assembly processor 314, and / or cannulation target processor 316.

[0088] It should be understood that, without departing from the teachings of this disclosure, the image capture subsystem interface 311 and the IMU subsystem interface 312 can be combined as a single element.

[0089] Catheter assembly processor (e.g., catheter assembly processor 314).

[0090] Such as at least Figure 5A As illustrated, processor 310 includes one or more conduit assembly processors (e.g., conduit assembly processor 314). Conduit assembly processor 314 can be configured to perform processing, in particular, of information relating to conduit assembly 230. For example, conduit assembly processor 314 can be configured to receive real-time images captured by image capture subsystem 210 from image capture subsystem interface 311. Conduit assembly processor 314 can also be configured to receive real-time IMU information measured by IMU subsystem 220 from IMU subsystem interface 312. Conduit assembly processor 314 can also be configured to process the images to determine whether the images include conduit assembly 230. In some example embodiments, conduit assembly processor 314 can also be configured to generate a visible indicator of the distal end 230a of conduit assembly 230 in the image to be displayed on graphics display 304. When processor 310 determines that the image includes the distal end 230a of catheter assembly 230, catheter assembly processor 314 can also be configured to generate a real-time 3D position (e.g., Cartesian coordinates) of the distal end 230a of catheter assembly 230. Catheter assembly processor 314 can also be configured to generate real-time depth information of the distal end 230a of catheter assembly 230 (e.g., depth information between the distal end 230a of catheter assembly 230 and a reference point on main assembly 200, such as the central axis of image capture assembly 210).

[0091] To perform the actions, functions, processes, and / or methods described above and in this disclosure, example embodiments of the catheter assembly processor 314 include one or more elements. For example, such as Figure 5BAs illustrated, the catheter assembly processor 314 includes one or more catheter assembly detectors 314a. The catheter assembly processor 314 also includes one or more catheter assembly position generators 314b. The catheter assembly processor 314 also includes one or more catheter assembly visual indicator generators 314c.

[0092] Although the actions, functions, processes, and / or methods performed by the catheter assembly processor 314 may be described in this disclosure as being performed by one or more specific elements of the catheter assembly processor 314, without departing from the teachings of this disclosure, the actions, functions, processes, and / or methods performed by specific elements of the catheter assembly processor 314 may also be performed by one or more other elements and / or by more than one element of the catheter assembly processor 314 (and / or other elements of the processor 310 and / or surgical system 100) in concert. It should also be understood in this disclosure that, although the actions, functions, processes, and / or methods performed by the catheter assembly processor 314 may be described in this disclosure as being performed by specific elements of the catheter assembly processor 314, without departing from the teachings of the invention, the actions, functions, processes, and / or methods performed by two or more specific elements of the catheter assembly processor 314 may also be combined and performed by one element of the catheter assembly processor 314.

[0093] These elements of the catheter assembly processor 314 will now be described further with reference to the accompanying drawings.

[0094] (i) Catheter assembly detector (e.g., catheter assembly detector 314a).

[0095] Such as at least Figure 5B As illustrated, the catheter assembly processor 314 includes one or more catheter assembly detectors (e.g., catheter assembly detector 314a). Catheter assembly detector 314a can be configured to receive real-time images captured by image capture subsystem 210 from image capture subsystem interface 311. Catheter assembly detector 314a can also receive historical images captured by image capture subsystem 210 (and / or other surgical systems 100), such as historical images of successful procedures (e.g., successful ERCP cannulation procedures) and / or historical images of unsuccessful procedures. These historical images can be received from database 330 and / or network 320.

[0096] The conduit assembly detector 314a then processes the image (i.e., processes the real-time image, and may also process historical images) to determine whether the image includes the conduit assembly 230. More specifically, the conduit assembly detector 314a can be configured to process the image to determine whether the image includes the distal end 230a of the conduit assembly 230, and if it includes the distal end 230a of the conduit assembly 230, determines the position of the distal end 230a of the conduit assembly 230 in the image. In other words, the conduit assembly detector 314a determines whether the distal end 230a of the conduit assembly 310 is within the image capture view of the image capture subsystem 210, and identifies the position of the distal end 230a of the conduit assembly 310 within the image. These processes performed by the conduit assembly detector 314a—including calculations, estimations, results, inferences, predictions, etc.—can be generated and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these AI algorithms, engines, systems, processors, and / or processes may be provided locally via catheter component detector 314a (and / or processor 310 and / or one or more components of surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0097] Then, the catheter assembly detector 314a provides the identified / located distal end 230a of the catheter assembly 310 in the image to the catheter assembly location generator 314b for further processing.

[0098] (ii) Catheter assembly location generator (e.g., catheter assembly location generator 314b).

[0099] Such as at least Figure 5BAs illustrated, the catheter assembly processor 314 includes one or more catheter assembly location generators (e.g., catheter assembly location generator 314b). The catheter assembly location generator 314b is configurable to receive real-time images captured by the image capture subsystem 210 from the image capture subsystem interface 311. The catheter assembly location generator 314b also receives processing results from the catheter assembly detector 314a, including the identification or location of the distal end 230a of the catheter assembly 310 in the received images. The catheter assembly location generator 314b also receives real-time IMU information measured by the IMU subsystem 220 from the IMU subsystem interface 312.

[0100] Using the received information, the catheter assembly position generator 314b then generates the 3D position (e.g., Cartesian coordinates) of the distal end 230a of the catheter assembly 230 in real time. In the example implementation, the catheter assembly position generator 314b generates the 3D position (e.g., Cartesian coordinates) of a single point of the distal end 230a of the catheter assembly 230 in real time, such as the 3D position (e.g., Cartesian coordinates) of the farthest point of the distal end 230a of the catheter assembly 230. Each such 3D position of a single point of the distal end 230a of the catheter assembly 230 can be generated in a timely manner for each instance, and each such 3D position of a single point of the distal end 230a of the catheter assembly 230 can be generated continuously and periodically (e.g., every 1 ms, or higher or lower frequency) when an event occurs (e.g., when motion is detected via the IMU subsystem 220; when motion is detected via a changing image; when motion is detected via a change in the position of one or more points / features between images, etc.). Alternatively, the catheter assembly position generator 314b can generate, in real-time and on a per-instance basis, the 3D positions (e.g., Cartesian coordinates) of multiple points of the distal end 230a of the catheter assembly 230. These multiple 3D positions of the distal end 230a of the catheter assembly 230 can be points or portions constituting or defining the distal edge, corner, side, etc., of the distal end 230a of the catheter assembly 230. Each such 3D position of the distal end 230a of the catheter assembly 230 can be generated on a per-instance basis, and can be generated continuously and periodically (e.g., every 1 ms, or higher or lower frequency) when an event occurs (e.g., when motion is detected via the IMU subsystem 220; when motion is detected via a changing image; when motion is detected via a change in the position of one or more points / features between images, etc.).

[0101] Using the received information, the catheter assembly position generator 314b can also generate depth information of the distal end 230a of the catheter assembly 230 in real time. For example, the depth information of the distal end 230a of the catheter assembly 230 can be depth information that identifies the depth (or distance) between the distal end 230a of the catheter assembly 230 and a reference point on the main assembly 200 (e.g., the central axis of the image capture assembly 210, the central axis of the opening in the body 200' through which the catheter assembly 230 extends outward, etc.).

[0102] The generation of 3D position (and / or depth information) of the distal end 230a of the catheter assembly 230 by the catheter assembly position generator 314b—including calculation, estimation, result, inference, prediction, etc.—can be performed and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These AI algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these AI algorithms, engines, systems, processors, and / or processes can be provided locally via the catheter assembly position generator 314b (and / or processor 310 and / or one or more components of the surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0103] Then, the catheter assembly position generator 314b provides the 3D position (and / or depth information) of the distal end 230a of the catheter assembly 310 to the catheter assembly visual indicator generator 314c and / or trajectory path processor 318 for further processing. The catheter assembly position generator 314b may also provide the 3D position (and / or depth information) of the distal end 230a of the catheter assembly 310 to the graphic display 304 for display to the surgeon.

[0104] (iii) Catheter assembly visual indicator generator (e.g., catheter assembly visual indicator generator 314c).

[0105] Such as at least Figure 5BAs illustrated, the catheter assembly processor 314 includes one or more catheter assembly visual indicator generators (e.g., catheter assembly visual indicator generator 314c). The catheter assembly visual indicator generator 314c can be configured to receive real-time images captured by the image capture subsystem 210 from the image capture subsystem interface 311. The catheter assembly visual indicator generator 314c also receives processing results from the catheter assembly detector 314a, including the identification or localization of the distal end 230a of the catheter assembly 310 in the received images (if the distal end 230a is detected and identified). The catheter assembly visual indicator generator 314c can also receive real-time IMU information measured by the IMU subsystem 220 from the IMU subsystem interface 312.

[0106] Using the received information, the catheter assembly visual indicator generator 314c then generates, in real time, a visible indicator of the distal end 230a of the catheter assembly 230 for the image to be displayed on the graphics display 304. The visible indicator can be any visible indicator, including but not limited to highlights, outlines, symbols, bounding boxes, and / or similar objects. This visible indicator allows the surgeon to easily identify the distal end 230a of the catheter assembly 230 while viewing the graphics display 304.

[0107] The generation of visual indicators for the distal end 230a of the catheter assembly 230 by the catheter assembly visual indicator generator 314c—including calculation, estimation, result, inference, prediction, etc.—can be performed and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These AI algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these AI algorithms, engines, systems, processors, and / or processes can be provided locally via the catheter assembly visual indicator generator 314c (and / or processor 310 and / or one or more components of the surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0108] Then, the catheter assembly visual indicator generator 314c provides a visual indicator of the distal end 230a of the catheter assembly 310 to the trajectory path processor 318 for further processing. The catheter assembly visual indicator generator 314c can also provide a visual indicator of the distal end 230a of the catheter assembly 310 to a graphic display 304 for display to the surgeon.

[0109] Cannulated target processor (e.g., cannulated target processor 316).

[0110] Such as at least Figure 5A As illustrated, processor 310 includes one or more cannulation target processors (e.g., cannulation target processor 316). Cannulation target processor 316 can be configured to perform processing, in particular, of information relating to the cannulation target (e.g., a nipple (or nipple orifice)). For example, cannulation target processor 316 can be configured to receive real-time images captured by image capture subsystem 210 from image capture subsystem interface 311. Cannulation target processor 316 can also be configured to receive real-time IMU information measured by IMU subsystem 220 from IMU subsystem interface 312. Cannulation target processor 316 can also be configured to process the images to determine whether the images include the cannulation target. In some example embodiments, cannulation target processor 316 can also be configured to generate a visible indicator of the cannulation target in the image to be displayed on graphics display 304. When processor 310 determines that the image includes the cannulation target, cannulation target processor 316 can also be configured to generate the real-time 3D position (e.g., Cartesian coordinates) of the cannulation target. The cannulation target processor 316 can also be configured to generate real-time depth information of the cannulation target (e.g., depth information between the cannulation target and a reference point on the main component 200, such as the central axis of the image capture component 210).

[0111] To perform the actions, functions, processes, and / or methods described above and in this disclosure, example embodiments of the cannulation target processor 316 include one or more elements. For example, such as Figure 5C As illustrated, the intubation target processor 316 includes one or more intubation target detectors 316a. The intubation target processor 316 also includes one or more intubation target position generators 316b. The intubation target processor 316 also includes one or more intubation target visual indicator generators 316c.

[0112] Although the actions, functions, processes, and / or methods performed by the intubation target processor 316 may be described in this disclosure as being performed by one or more specific elements of the intubation target processor 316, without departing from the teachings of this disclosure, the actions, functions, processes, and / or methods performed by a specific element of the intubation target processor 316 may also be performed by one or more other elements and / or by more than one element of the intubation target processor 316 (and / or other elements of the processor 310 and / or surgical system 100) in concert. It should also be understood in this disclosure that, although the actions, functions, processes, and / or methods performed by the intubation target processor 316 may be described in this disclosure as being performed by a specific element of the intubation target processor 316, without departing from the teachings of this disclosure, the actions, functions, processes, and / or methods performed by two or more specific elements of the intubation target processor 316 may be combined and performed by one element of the intubation target processor 316. It should also be understood that, without departing from the teachings of this disclosure, the catheter assembly processor 314 (or one or more elements of the catheter assembly processor 314) and the cannulation target processor 316 (or one or more elements of the cannulation target processor 316) may be combined as a single processor.

[0113] These components of the cannulated target processor 316 will now be described further with reference to the accompanying drawings.

[0114] (i) Cannulation target detector (e.g., cannulation target detector 316a).

[0115] Such as at least Figure 5C As illustrated, the cannulation target processor 316 includes one or more cannulation target detectors (e.g., cannulation target detector 316a). Cannulation target detector 316a can be configured to receive real-time images captured by image capture subsystem 210 from image capture subsystem interface 311. Cannulation target detector 316a can also receive historical images captured by image capture subsystem 210 (and / or other surgical systems 100), such as historical images of successful procedures (e.g., successful ERCP cannulation procedures) and / or historical images of unsuccessful procedures. These historical images can be received from database 330 and / or network 320.

[0116] Then, the cannulation target detector 316a processes the image (i.e., processes the real-time image, and may also process historical images) to determine whether the image includes the cannulation target, and if so, determines the position of the cannulation target in the image. In other words, the cannulation target detector 316a determines whether the cannulation target is within the image capture view of the image capture subsystem 210 and identifies the position of the cannulation target within the image. These processes performed by the cannulation target detector 316a—including computation, estimation, results, inference, prediction, etc.—can be generated and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These AI algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes can be provided locally via the cannulation target detector 316a (and / or processor 310 and / or one or more components of the surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0117] Then, the cannula target detector 316a provides the identified / located cannula target in the image to the cannula target location generator 316b for further processing.

[0118] (ii) Intubation target location generator (e.g., intubation target location generator 316b).

[0119] Such as at least Figure 5C As illustrated, the intubation target processor 316 includes one or more intubation target location generators (e.g., intubation target location generator 316b). The intubation target location generator 316b is configurable to receive real-time images captured by the image capture subsystem 210 from the image capture subsystem interface 311. The intubation target location generator 316b also receives processing results from the intubation target detector 316a, including the identification or localization of intubation targets in the received images (if any intubation targets are detected and identified). The intubation target location generator 316b also receives real-time IMU information measured by the IMU subsystem 220 from the IMU subsystem interface 312.

[0120] Using the received information, the intubation target location generator 316b then generates the 3D location (e.g., Cartesian coordinates) of the intubation target in real time. In an example implementation, the intubation target location generator 316b generates the 3D location (e.g., Cartesian coordinates) of a single point of the intubation target, such as the center point of the intubation target (e.g., the center point of the nipple foramen when the intubation target is the nipple foramen). Each such 3D location of a single point of the intubation target can be generated in a timely manner for each instance, and each such 3D location of a single point of the intubation target can be generated continuously and periodically (e.g., every 1 ms, or higher or lower frequency) when an event occurs (e.g., when motion is detected via IMU subsystem 220; when motion is detected via a changing image; when motion is detected via a change in the position of one or more points / features between images, etc.). Alternatively, the intubation target location generator 316b can generate the 3D locations (e.g., Cartesian coordinates) of multiple points of the intubation target in real time and for each instance. Such multiple 3D positions of multiple points of the intubation target can be points or parts that constitute or define the edges, perimeters, sides, etc. of the intubation target. Each such 3D position of multiple points of the intubation target can be generated in real time for each instance, and each such 3D position of multiple points of the intubation target can be generated continuously and periodically (e.g., every 1 ms, or higher or lower frequency) when an event occurs (e.g., when motion is detected by the IMU subsystem 220; when motion is detected by a changing image; when motion is detected by a change in the position of one or more points / features between images, etc.).

[0121] Using the received information, the cannulation target location generator 316b can also generate depth information of the cannulation target in real time. For example, the depth information of the cannulation target can be depth information that identifies the depth (or distance) between the cannulation target and a reference point on the main component 200 (e.g., the central axis of the image capture component 210, the central axis of the opening in the body 200' through which the catheter component 230 extends outward, etc.).

[0122] The generation of 3D position (and / or depth information) of the intubation target by the intubation target position generator 316b—including calculation, estimation, result, inference, prediction, etc.—can be performed and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These AI algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these AI algorithms, engines, systems, processors, and / or processes can be provided locally via the intubation target position generator 316b (and / or processor 310 and / or one or more components of the surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0123] Then, the cannulation target location generator 316b provides the 3D location (and / or depth information) of the cannulation target to the cannulation target visual indicator generator 316c and / or trajectory path processor 318 for further processing. The cannulation target location generator 316b can also provide the 3D location (and / or depth information) of the cannulation target to the graphic display 304 for display to the surgeon.

[0124] (iii) Intubation target visual indicator generator (e.g., intubation target visual indicator generator 316c).

[0125] Such as at least Figure 5C As illustrated, the intubation target processor 316 includes one or more intubation target visual indicator generators (e.g., intubation target visual indicator generator 316c). The intubation target visual indicator generator 316c can be configured to receive real-time images captured by the image capture subsystem 210 from the image capture subsystem interface 311. The intubation target visual indicator generator 316c also receives processing results from the intubation target detector 316a, including the identification or location of intubation targets in the received images (if any intubation target is detected and identified). The intubation target visual indicator generator 316c can also receive real-time IMU information measured by the IMU subsystem 220 from the IMU subsystem interface 312.

[0126] Using the received information, the cannulation target visual indicator generator 316c then generates a visible indicator of the cannulation target in real time for the image to be displayed on the graphics display 304. This visible indicator can be any visible indicator, including but not limited to highlights, outlines, symbols, bounding boxes, and / or similar objects. Such a visible indicator allows the surgeon to easily identify the cannulation target while viewing the graphics display 304.

[0127] The generation of visual indicators for the intubation target by the visual indicator generator 316c—including calculation, estimation, result, inference, prediction, etc.—can be performed and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These AI algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these AI algorithms, engines, systems, processors, and / or processes can be provided locally via the visual indicator generator 316c (and / or processor 310 and / or one or more components of the surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0128] Then, the intubation target visual indicator generator 316c provides the visual indicator of the intubation target to the trajectory path processor 318 for further processing. The intubation target visual indicator generator 316c can also provide the visual indicator of the intubation target to the graphic display 304 for display to the surgeon.

[0129] Trajectory path processor (e.g., trajectory path processor 318).

[0130] Such as at least Figure 5AAs illustrated, processor 310 includes one or more trajectory path processors (e.g., trajectory path processor 318). Trajectory path processor 318 can be configured to receive real-time images captured by image capture subsystem 210 from image capture subsystem interface 311. Trajectory path processor 318 can also receive historical images captured by image capture subsystem 210 (and / or other surgical systems 100), such as historical images of successful procedures (e.g., successful bile duct cannulation procedures) and / or historical images of unsuccessful procedures. These historical images may be received from database 330 and / or network 320. Trajectory path processor 318 can also receive real-time IMU information measured by IMU subsystem 220 from IMU subsystem interface 312. The trajectory path processor 318 can also receive historical measurement results obtained from the IMU subsystem 220 (and / or other surgical systems 100), such as historical measurement results of successful procedures (e.g., successful ERCP cannulation procedures) (including 3D position, orientation, acceleration, etc. of the distal end 230a of the catheter assembly 230 and / or the cannulation target) and / or historical 3D position of unsuccessful procedures (including 3D position, orientation, acceleration, etc. of the distal end 230a of the catheter assembly 230 and / or the cannulation target). These historical measurement results can be received from the database 330 and / or the network 320. The trajectory path processor 318 can also receive the processing results of the catheter assembly detector 314a. The trajectory path processor 318 can also receive the processing results of the catheter assembly position generator 314b. The trajectory path processor 318 can also receive the processing results of the catheter assembly visual indicator generator 314c. The trajectory path processor 318 can also receive the processing results of the cannulation target detector 316a. The trajectory path processor 318 can also receive the processing results from the cannulation target position generator 316b. The trajectory path processor 318 can also receive the processing results from the cannulation target visual indicator generator 316c.

[0131] When the trajectory path processor 318 determines that the received image includes at least the intubation target, the trajectory path processor 318 generates one or more post-intubation predicted trajectory paths 10, 20 for the intubation target in real time. For example, Figure 6A , Figure 6B and Figure 6C As illustrated, when performing an ERCP cannulation procedure (for the common bile duct or CBD), the post-cannulation predicted trajectory path 10 (or bile duct orientation 10 or papillary plane 10) is a predicted trajectory path used to ensure that the catheter assembly 230 successfully reaches the CBD after passing through the papillary orifice.

[0132] When the trajectory path processor 318 determines that the received image includes at least the distal end 230a of the catheter assembly 230, the trajectory path processor 318 generates one or more predicted trajectory paths for the distal end 230a of the catheter assembly 230 in real time. For example, as Figures 6A to 6C As illustrated, when performing an ERCP cannulation procedure (for the common bile duct or CBD), the predicted trajectory path 20 is the predicted trajectory path of the catheter assembly 230, which is based in particular on the current position and orientation of the main assembly 200 (and the catheter assembly 230).

[0133] When the trajectory path processor 318 determines that the received image includes both the distal end 230a of the catheter assembly 230 and the insertion target, the trajectory path processor 318 generates one or more predicted trajectory paths between the distal end 230a of the catheter assembly 230 and the insertion target in real time (i.e., one or more predicted real-time trajectory paths for the distal end 230a of the catheter assembly 230 to insert cannulate the insertion target). For example, as Figures 6A to 6C As illustrated, when performing a cannulation procedure (for the common bile duct or CBD), the predicted trajectory path 20 is a predicted trajectory path of the catheter assembly 230, which is based in particular on the current position and orientation of the main assembly 200 (and the catheter assembly 230).

[0134] When the trajectory path processor 318 determines that the received image includes both the distal end 230a of the catheter assembly 230 and the cannulation target, the trajectory path processor 318 can also generate real-time depth information between the cannulation target and the distal end 230a of the catheter assembly 230.

[0135] As an illustrative example, Figure 6AThe illustration shows an example where the trajectory path processor 318 determines that both the distal end 230a of the catheter assembly 230 and the cannulation target are in the camera view of the image capture assembly 210 (i.e., both are identified in the received live image). The trajectory path processor 318 then generates a predicted trajectory path 20 for the distal end 230a of the catheter assembly 230 in real time, as described above and in this disclosure, based on information received in real time by the trajectory path processor 318. The trajectory path processor 318 also generates a post-cannulation predicted trajectory path 10 for the cannulation target in real time, as described above and in this disclosure, based on information received in real time by the trajectory path processor 318. The trajectory path processor 318 then processes the predicted trajectory path 20 and the post-cannulation predicted trajectory path 10, including comparing paths 10 and 20. In this particular example, the trajectory path processor 318 determines that the main assembly 200 (and the catheter assembly 230) is not in the ideal position and / or orientation for a successful ERCP cannulation procedure. In the example implementation, the trajectory path processor 318 may therefore perform one or more actions, including but not limited to: providing visual indication on the graphics display 304 that the main component 200 (and catheter assembly 230) is not in the ideal position and / or orientation; providing the surgeon with suggestions, guidance and / or visual orientation to reposition or move the main component 200 (and catheter assembly 230) toward the ideal position and / or orientation; preventing or locking the catheter assembly 230 to prevent the catheter assembly 230 from extending outward for ERCP cannulation, etc. Figure 6B The illustration shows an example of a surgeon moving the main assembly (and catheter assembly 230) toward a desired position and / or orientation (this can also be performed by attempting to align the predicted trajectory path 20 of the distal end 230a of catheter assembly 230 with the post-cannulation predicted trajectory path 10 for the cannulation target). Figure 6C As illustrated, the main assembly (and catheter assembly 230) has been moved to the ideal position and orientation, which can be easily identified by the surgeon based on the alignment and / or overlap of the predicted trajectory path 20 of the distal end 230a of the catheter assembly 230 with the post-intubation predicted trajectory path 10 for the intubation target.

[0136] The generation of the predicted trajectory path 20 and the post-intubation trajectory path 10 by the trajectory path processor 318—including calculation, estimation, result, inference, prediction, etc.—can be performed and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These AI algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these AI algorithms, engines, systems, processors, and / or processes can be provided locally via the trajectory path processor 318 (and / or processor 310 and / or one or more components of the surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0137] The trajectory path processor 318 then provides the predicted trajectory path 20 and / or the post-intubation trajectory path 10 to the graphic display 304 for display to the surgeon. The trajectory path processor 318 may also provide the predicted trajectory path 20 and / or the post-intubation trajectory path 10 to the network 320 and / or the database 330.

[0138] An example implementation of a method for configuring a surgical system (e.g., method 700).

[0139] Figure 7A The diagram illustrates an example implementation of a method for configuring a surgical system (e.g., method 700). Method 700 includes providing a surgical system (e.g., surgical system 100, such as at least...). Figure 1A (as illustrated in the figure) (e.g., action 710). Method 700 also includes capturing real-time images (e.g., action 720). Method 700 also includes acquiring real-time IMU information (e.g., action 730). Method 700 also includes processing the acquired images and IMU information (e.g., action 740).

[0140] These and other processes and / or actions of method 700 will now be described further with reference to the accompanying drawings.

[0141] Provide surgical systems (e.g., Action 710).

[0142] In an example implementation, method 700 includes providing a surgical system (e.g., action 710). Surgical system 100 includes a body (e.g., body 200', such as at least...). Figure 2A(As illustrated in the figure and described in this disclosure). Surgical system 100 also includes one or more inertial measurement unit (IMU) subsystems (e.g., IMU subsystem 220, such as at least...). Figure 3A (As illustrated in the figure and described in this disclosure). The IMU subsystem 220 is housed within the body 200' of the main component 200. The surgical system 100 also includes one or more image capture subsystems 210 (e.g., image capture subsystems 210, such as at least...). Figure 2A (As illustrated in the figure and described in this disclosure). The image capture subsystem 210 is housed within the body 200' of the main component 200. The surgical system 100 also includes one or more catheter assemblies (e.g., catheter assembly 230, such as at least...). Figure 2B (Illustrated and described in this disclosure). The catheter assembly 230 includes a proximal end and a distal end 230a. The catheter assembly 230 can be configured or configured to selectively extend the distal end 230a of the catheter assembly 230 outward from the body 200' (e.g., as shown). Figure 2B (As illustrated in the figure). The catheter assembly 230 (i.e., the portion extending away from the body 200') can be selectively varied (along the catheter assembly 230) in terms of length (distance), bending angle, bending direction, and bending position, so that the surgeon can follow any predicted trajectory path 20 (as described in this disclosure) and / or a post-intubation predicted trajectory path 10 (as described in this disclosure).

[0143] Capture real-time images (e.g., Action 720).

[0144] like Figure 7A As illustrated, method 700 includes capturing real-time images (e.g., action 720). The captured images can be video images and / or still images. Such images can be captured by an example implementation of image capture component 210 and provided in real-time to a processor (e.g., processor 310). More specifically, the images can be provided to an image capture subsystem interface (e.g., image capture subsystem interface 311, such as at least...). Figure 5A (As illustrated). Images may also be provided in real time, directly or indirectly, to a graphics display (e.g., graphics display 304) for display to the surgeon. Images may also be provided in real time, directly or indirectly, to a network (e.g., network 320) and / or a database (e.g., database 330).

[0145] In an example implementation, the captured image may be provided to processor 310 (i.e., image capture subsystem interface 311), which provides the captured image to a graphics display 304 for display to the surgeon; and provides the captured image to database 330 for storage. Processor 310 then performs image processing (e.g., action 740). In an example implementation where image processing (e.g., action 740) is not entirely performed by the local processor 310 (e.g., partially or entirely performed by cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.), processor 310 provides the captured image to network 320 (network 320 includes cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc., and / or communicates with cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.) for further processing.

[0146] Obtain real-time IMU information (e.g., Action 730).

[0147] like Figure 7A As illustrated, method 700 includes acquiring real-time IMU information (e.g., action 730). The acquired IMU information may include specific force, angular velocity, position, and / or orientation. Such IMU information may be measured or captured by an example implementation of IMU component 220 (which may include an accelerometer, gyroscope, and / or magnetometer) and provided in real-time to processor 310. More specifically, the IMU information may be provided to an IMU subsystem interface (e.g., IMU subsystem interface 312, such as at least...). Figure 5A (As illustrated in the diagram). IMU information can also be provided in real time, directly or indirectly, to the graphic display 304 for display to the surgeon. IMU information can also be provided in real time, directly or indirectly, to the network 320 and / or the database 330.

[0148] In an example implementation, IMU information may be provided to processor 310 (i.e., IMU subsystem interface 312), which provides the IMU information to a graphics display 304 for display to the surgeon and to a database 330 for storage. Processor 310 then performs processing on the IMU information (e.g., action 740). In an example implementation where the processing of the IMU information (e.g., action 740) is not entirely performed by the local processor 310 (e.g., partially or entirely performed by cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.), processor 310 provides the IMU information to network 320 (network 320 includes cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc., and / or communicates with cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.) for further processing.

[0149] The acquired images and IMU information are processed (e.g., action 740).

[0150] like Figure 7A and Figure 7BAs illustrated, method 700 includes processing image and IMU information (e.g., action 740). Processing 740 includes determining whether the image includes catheter assembly 230 (e.g., action 741). More specifically, processing 740 includes determining whether the image includes the distal end 230a of catheter assembly 230 (e.g., action 741). Determining whether the image includes the distal end 230a of catheter assembly 230 may include processing historical images captured by image capture subsystem 210 (and / or other surgical system 100), such as historical images of successful procedures (e.g., successful bile duct cannulation procedures) and / or historical images of unsuccessful procedures. These historical images may be received from database 330 and / or network 320. This processing for determining whether the image includes the distal end 230a of catheter assembly 230—including calculation, estimation, result, inference, prediction, etc.—may be performed and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these AI algorithms, engines, systems, processors, and / or processes may be provided locally (e.g., via the catheter assembly detector 314a and / or processor 310 and / or one or more components of the surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0151] When process 740 determines that the image includes the distal end 230a of catheter assembly 230, process 740 includes identifying the distal end 230a of catheter assembly 230 in the image. Furthermore, when process 740 determines that the image includes the distal end 230a of catheter assembly 230, process 740 includes generating a real-time 3D position of the distal end 230a of catheter assembly 230 (e.g., action 742). The generation of the real-time 3D position of the distal end 230a of catheter assembly 230 can be performed at least based on acquired IMU information. The generation of the real-time 3D position of the distal end 230a of catheter assembly 230 can also be based on the acquired image. This generation of the 3D position of the distal end 230a of catheter assembly 230—including calculation, estimation, result, inference, prediction, etc.—can be performed and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these AI algorithms, engines, systems, processors, and / or processes may be provided locally (e.g., via the catheter assembly location generator 314b and / or processor 310 and / or one or more components of the surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0152] Processing 740 also includes determining whether the image includes the insertion target (e.g., nipple or nipple hole) (e.g., action 743). Determining whether the image includes the insertion target may include processing historical images captured by the image capture subsystem 210 (and / or other surgical system 100), such as historical images of successful procedures (e.g., successful bile duct cannulation procedures) and / or historical images of unsuccessful procedures. These historical images may be received from database 330 and / or network 320. These processes used to determine whether the image includes the insertion target—including calculations, estimations, results, inferences, predictions, etc.—may be performed and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes can be provided locally (e.g., via cannulation target detector 316a and / or processor 310 and / or one or more components of surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0153] When process 740 determines that the image includes an intubation target, process 740 includes identifying the intubation target in the image. Furthermore, when process 740 determines that the image includes an intubation target, process 740 includes generating a real-time 3D position of the intubation target (e.g., action 744). The generation of the real-time 3D position of the intubation target can be performed at least based on acquired IMU information. The generation of the real-time 3D position of the intubation target can also be based on the acquired image. Such generation of the 3D position of the intubation target—including computation, estimation, result, inference, prediction, etc.—can be performed and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Furthermore, these artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes can be provided locally (e.g., via the cannulation target location generator 316b and / or processor 310 and / or one or more components of the surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0154] When process 740 determines that the image includes at least the distal end 230a of catheter assembly 230, process 740 includes predicting one or more trajectory paths 20 of the distal end 230a of catheter assembly 230. When process 740 determines that the image includes at least an insertion target, process 740 includes predicting one or more post-insertion trajectory paths 10 (as described in this disclosure). When process 740 determines that the image includes the distal end 230a of catheter assembly 230 and an insertion target, process 740 includes predicting one or more trajectory paths of the distal end 230a of catheter assembly 230 to the insertion target (e.g., action 745), which includes predicting trajectory paths 20 and post-insertion trajectory paths 10. Such prediction may be performed based at least on the 3D position of the distal end 230a of catheter assembly 230 and the 3D position of the insertion target. Furthermore, such predictions of trajectory path 20 and post-intubation trajectory path 10—including calculations, estimations, results, inferences, and predictions—can be performed and / or derived directly or indirectly, partially or wholly, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These AI algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), simultaneous localization and mapping (SLAM), etc. Moreover, these AI algorithms, engines, systems, processors, and / or processes can be provided locally (e.g., via trajectory path processor 318 and / or processor 310 and / or one or more elements of surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0155] After performing the predictions described above and in this disclosure, process 740 includes generating a predicted trajectory path 20 of the distal end 230a of the catheter assembly 230 on an image displayed on the graphics display 304 (e.g., based on the position and / or orientation of the distal end 230a and / or other portions of the catheter assembly 230). Process 740 also includes generating a post-intubation predicted trajectory path 10 on an image displayed on the graphics display 304 when the image includes an intubation target. Process 740 further includes generating a predicted trajectory path 20 for intubation of the distal end 230a of the catheter assembly 230 to the identified intubation target on an image displayed on the graphics display 304 when the image includes the distal end 230a of the catheter assembly 230 and the intubation target. This generation of the trajectory path 20 and the post-intubation trajectory path 10—including calculations, estimations, results, inferences, predictions, etc.—can be performed and / or derived directly or indirectly, in part or in whole, via artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes. These artificial intelligence (AI) algorithms, engines, systems, processors, and / or processes may include, but are not limited to, machine learning algorithms, deep learning algorithms, deep neural networks (DNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), simultaneous localization and mapping (SLAM), convolutional neural networks (CNNs), and region convolutional neural networks (R-CNNs). Furthermore, these AI algorithms, engines, systems, processors, and / or processes may be provided locally (e.g., via trajectory path processor 318 and / or processor 310 and / or one or more elements of surgical system 100) and / or via cloud computing 310, distributed computing 310, and / or non-local or decentralized artificial intelligence (AI) 310, etc.

[0156] Although various embodiments based on the disclosed principles have been described above, it should be understood that they are presented by way of example only and are not restrictive. Therefore, the breadth and scope of the exemplary embodiments described in this disclosure should not be limited to any of the exemplary embodiments described above, but should be defined only by the claims published from this disclosure and their equivalents. Furthermore, although the aforementioned advantages and features are provided in the described embodiments, the application of these published claims should not be limited to processes and structures for achieving any or all of the aforementioned advantages.

[0157] For example, the terms “communication,” “communicating,” “connection,” “connecting,” or other similar terms should generally be interpreted broadly as wired, wireless, and / or other forms of connection, whereby enabling voice and / or data to be sent, transmitted, broadcast, received, intercepted, acquired, and / or transferred (if each applies), between elements, devices, computing devices, telephones, processors, controllers, servers, networks, telephone networks, the cloud, and / or the like.

[0158] As another example, the terms “user,” “operator,” “surgeon,” or similar terms should generally be interpreted broadly as referring to a user who operates, controls, manages, etc., one or more components of a surgical system (e.g., surgical system 100) and / or a processor (e.g., processor 310).

[0159] Furthermore, as mentioned herein, processors, devices, computing devices, telephones, mobile phones, servers, gateway servers, communication gateway servers, and / or controllers can be any processor, computing device, and / or communication device, and can include virtual machines, computers, nodes, instances, hosts, or machines in a network computing environment. Also as mentioned herein, a network or cloud can be a collection of machines connected by communication channels that facilitate communication between machines and allow machines to share resources, or a collection including such machines. A network can also refer to a communication medium between processes on the same machine. Also as mentioned herein, network elements, nodes, or servers can be machines deployed to execute programs operating as socket listeners and can include software instances.

[0160] A database (or memory or storage device) may include any collection and / or arrangement of volatile and / or non-volatile components suitable for storing data. For example, a memory may include random access memory (RAM) devices, read-only memory (ROM) devices, magnetic storage devices, optical storage devices, solid-state devices, and / or any other suitable data storage device. In certain embodiments, a database may partially represent a computer-readable storage medium on which computer instructions and / or logic are encoded. A database may represent any number of memory components within a processor and / or computing device, local to the processor and / or computing device, and / or accessible by the processor and / or computing device.

[0161] The various terms used herein have specific meanings within the art. Whether a particular term should be interpreted as such a “technical term” depends on the context in which it is used. Such terms will be interpreted in accordance with the context in which they are used in this disclosure, and those terms in the disclosed context will be understood by one of ordinary skill in the art. The foregoing definitions do not exclude other meanings that may be assigned to these terms based on the disclosed context.

[0162] Words related to comparison, measurement, and time, such as “at that time,” “equivalent,” “during,” and “completely,” should be understood as meaning “basically at that time,” “basically equivalent,” “basically during,” and “basically completely,” where “basically” means that such comparisons, measurements, and times are feasible for achieving the expected results implied or explicitly stated.

[0163] Furthermore, the section headings and subject headings provided herein are intended to provide consistency with recommendations under various patent laws and practices, or otherwise to offer organizational clues. These headings should not limit or characterize any embodiments that can be set forth in any of the claims of this disclosure. Specifically, the description of the technology in the “Background Art” section should not be construed as an admission that the technology is prior art to any embodiment of this disclosure. Moreover, any reference to the word “invention” in the singular form in this disclosure should not be used to argue that there is only one novel point in this disclosure. Multiple inventions may be set forth according to the limitations of the claims of this disclosure, and these claims accordingly define the invention protected by them and its equivalents. In all cases, the scope of these claims should be considered in light of the content of this disclosure and its own merits, but should not be limited by the headings herein.

Claims

1. A surgical system for performing endoscopic retrograde cholangiopancreatography (ERCP) cannulation, the surgical system comprising: The main component includes: - A main body for insertion into a patient's cavity; - An inertial measurement unit subsystem, i.e., an IMU subsystem, is housed within the main body and is configured to provide real-time IMU information, including real-time 3D position information; - An image capture subsystem, housed within the main body, configured to capture real-time images; and - A catheter assembly, housed within the body, the catheter assembly having a proximal end and a distal end, the catheter assembly configured to selectively extend the distal end of the catheter assembly outwardly from the body, wherein at least a portion of the catheter assembly is configured to selectively bend in multiple directions; and Processor, the processor being configured to: - Receive real-time IMU information from the IMU subsystem; - Receive real-time images from the image capture subsystem; - Determine whether the received image includes the distal end of the catheter assembly; - In response to determining that the received image includes the distal end of the catheter assembly: -- Identifying the distal end of the catheter assembly in the acquired images; and -- Generate the real-time 3D position of the distal end of the catheter assembly based on the received IMU information; - Determine whether the received image includes the intubation target; - In response to determining that the received image includes the cannulation target: -- Identify the cannulation target in the received image; and -- Generate the real-time 3D position of the intubation target based on the IMU information and the received image; - In response to determining that the received image includes the distal end of the catheter assembly and the cannulation target: -- Based on the 3D position of the distal end of the catheter assembly and the 3D position of the cannulation target, predict one or more real-time trajectory paths for cannulation of the identified distal end of the catheter assembly to the identified cannulation target.

2. The surgical system according to claim 1, in, The real-time images received from the image capture subsystem include real-time video images.

3. The surgical system according to claim 1, in, The real-time IMU information received from the IMU subsystem includes real-time orientation and / or acceleration information.

4. The surgical system according to claim 1, in, The main component includes a duodenoscope.

5. The surgical system according to claim 1, in, The conduit assembly includes a device guided by a wire, and / or a conduit.

6. The surgical system according to claim 1, in, The processor is also configured to: Real-time depth information between the identified cannulation target and the identified distal end of the catheter assembly is generated based on the real-time 3D position of the identified distal end of the catheter assembly and the real-time 3D position of the identified cannulation target.

7. The surgical system according to claim 1, further comprising: A graphics display that communicates with the processor and the main component; The processor is further configured to: The real-time image captured by the image capture subsystem is displayed on the graphics display; A visible indicator of the identified intubation target is generated in the real-time image displayed on the graphics display.

8. The surgical system according to claim 7, in, The processor is also configured to: A visible indicator of the distal end of the identified catheter assembly is generated in the real-time image displayed on the graphic display.

9. The surgical system according to claim 7, further comprising: A visible indicator is generated in the real-time image displayed on the graphic display of the predicted one or more real-time trajectory paths between the distal end of the identified catheter assembly and the identified cannulation target.

10. The surgical system according to claim 1, in, The processor is also configured to: The initial position, orientation, and movement of the distal end of the catheter assembly are adjusted based on the predicted trajectory path.

Images