Robotic systems and methods for accompanying endoscopic and percutaneous medical procedures

Abstract

Certain aspects relate to systems and techniques for endoscopic assisted percutaneous medical procedures. A method may include inserting a first medical device having an elongate shaft and a first orientation sensor through a natural aperture of a patient into a treatment region of the patient. A first orientation of a first medical device within a treatment region may be determined using a first orientation sensor. A target location within the treatment region away from the determined first orientation may be defined. A second medical device may be percutaneously guided toward the target location. The method may be performed on a robot-enabled medical system.

Description

[0001] Priority application

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 738,706, filed on September 28, 2019, which is incorporated herein by reference. Technical Field

[0003] The systems and methods disclosed herein relate to medical systems and medical procedures, and more particularly to robotic systems and methods for accompanying endoscopic and percutaneous medical procedures, such as endoscopically assisted percutaneous medical procedures and laparoscopically assisted endoscopic procedures. Background Art

[0004] Medical procedures such as endoscopy and laparoscopy may involve accessing and visualizing the interior of a patient's anatomy for diagnostic and / or therapeutic purposes. For example, ureteroscopy is a medical procedure commonly used to treat kidney stones. During the procedure, a thin, flexible, tubular tool or instrument called a ureteroscope may be inserted into the urethra, through the bladder and ureters, and into the kidney. In some cases, percutaneous access to the kidney may also be required.

[0005] In certain medical procedures, surgical robotic systems may be used to control the insertion and / or manipulation of surgical tools. A surgical robotic system may include at least one robotic arm or other instrument positioning device including a manipulator assembly for controlling the positioning of surgical tools during a procedure. Summary of the invention

[0006] The systems, methods and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for the desirable attributes disclosed herein.

[0007] In a first aspect, a method for performing a medical procedure includes: inserting a first medical device comprising a slender shaft and a first position sensor through a natural orifice of the patient into a treatment area of ​​the patient; determining a first position of the first medical device within the treatment area using the first position sensor; defining a target position within the treatment area away from the determined first position; and percutaneously guiding a second medical device through the patient toward the target position.

[0008] The method may also include any combination of one or more of the following features: (a) aligning the output of the first orientation sensor with the coordinate frame of the preoperative model, and wherein determining the first orientation of the instrument in the treatment area using the first orientation sensor includes determining the first orientation with reference to the preoperative model; (b) wherein the preoperative model includes a three-dimensional reconstruction of the anatomical structure; (c) wherein defining the target position in the treatment area includes determining the target position with reference to the preoperative model; (d) wherein determining the target position with reference to the preoperative model includes: displaying the preoperative model to a user and receiving a selection of the target position with reference to the preoperative model; (e) wherein defining the target position includes determining the difference between the determined first orientation and the target position with reference to the preoperative model; (f) displaying a representation of the target position to the user; (g) wherein defining the target position in the treatment area includes: capturing one or more intraoperative medical images of the treatment area, and defining the target position with reference to the one or more intraoperative medical images; (h) wherein the one or more intraoperative medical images include one or more fluoroscopic images; (i) aligning the output of the first orientation sensor with the coordinate frame of the preoperative model, and wherein determining the first orientation of the instrument in the treatment area using the first orientation sensor includes determining the first orientation with reference to the preoperative model; (ii) wherein the one or more intraoperative medical images include one or more fluoroscopic images; (j) wherein defining a target position within the treatment area away from the determined first position comprises determining the target position along a first axis extending from a distal end of the elongated shaft of the first medical device; (k) wherein percutaneously guiding a second medical device through the patient toward the target position comprises: aligning a second axis of the second medical device with the target position, and advancing the second medical device toward the target position; (l) wherein: the second medical device is attached to a robotic arm; and the robotic arm constrains movement of the second medical device to movement along the second axis; (m) wherein the first medical device comprises an endoscope; (n) wherein the first medical device is robotically controlled; (o) wherein the second medical device is robotically controlled; (p) wherein the treatment area comprises a kidney, a bladder, a lung, or a gastrointestinal tract; (q) patient motion is determined using a first position sensor of the first medical device, and wherein percutaneously guiding the second medical device through the patient toward the target position is based in part on the determined patient motion; and / or (r) wherein the patient motion is due to breathing.

[0009] On the other hand, a method for performing a medical procedure includes: inserting a first medical device comprising a slender shaft and a first orientation sensor through a natural orifice of the patient into a treatment area of ​​the patient; aligning the output of the first orientation sensor with a coordinate frame of a preoperative model; displaying the preoperative model to a user; defining the orientation of one or more virtual references to create a boundary with reference to the preoperative model, the orientation of the one or more virtual references being determined based on the aligned output of the first orientation sensor; positioning the first medical device so that the first orientation sensor is away from the boundary; and guiding a second device through a percutaneous opening to or within the treatment area based on the one or more virtual references.

[0010] The method may also include any combination of one or more of the following features: (a) wherein determining the orientation of one or more virtual references includes, for each of the orientations: navigating a first medical device to a location within the treatment region where the virtual reference is to be placed; and defining the location as the orientation of the virtual reference based on the aligned output of the first orientation sensor; (b) wherein determining the orientation of one or more virtual references includes, for each of the orientations: receiving a user selection of a location where one of the one or more virtual references is to be placed; and determining a virtual reference orientation corresponding to the location with reference to at least a first orientation determined based on the first orientation sensor, wherein the virtual reference orientation is away from the first orientation; (c) wherein a boundary defines a resection volume; (d) wherein the first medical device includes an endoscope; (e) wherein the first medical device is robotically controlled; (f) wherein the treatment region includes a kidney, bladder, lung, or gastrointestinal tract; and / or (g) wherein the preoperative model is determined based on a CT scan.

[0011] In another aspect, a computer-readable medium includes instructions configured to cause at least one processor to: insert a first medical device including an elongated shaft and a first position sensor through a natural orifice of the patient into a treatment area of ​​the patient; determine a first position of the first medical device within the treatment area using the first position sensor; define a target position within the treatment area away from the determined first position; and percutaneously guide a second medical device through the patient toward the target position.

[0012] The computer-readable medium may also include any combination of one or more of the following features: (a) wherein the instructions further cause at least one processor to align the output of the first orientation sensor with the coordinate frame of the preoperative model, and wherein determining the first orientation of the instrument within the treatment area using the first orientation sensor includes determining the first orientation with reference to the preoperative model; (b) wherein the preoperative model includes a three-dimensional reconstruction of the anatomical structure; (c) wherein defining a target position within the treatment area includes determining the target position with reference to the preoperative model; (d) wherein determining the target position with reference to the preoperative model includes: displaying the preoperative model to a user and receiving a selection of the target position with reference to the preoperative model; (e) wherein defining the target position includes determining the difference between the determined first orientation and the target position with reference to the preoperative model; (f) wherein the instructions further cause at least one processor to display a representation of the target position to the user; (g) defining the target position within the treatment area includes: capturing one or more intraoperative medical images of the treatment area, and defining the target position with reference to the one or more intraoperative medical images; (h) wherein the one or more intraoperative medical images include one or more fluoroscopic images; (i) wherein the instructions The instructions also cause at least one processor to align the output of the first position sensor with one or more intraoperative medical images; (j) wherein defining a target position within the treatment area away from the determined first position includes determining the target position along a first axis extending from a distal end of the slender shaft of the first medical device; (k) wherein percutaneously guiding a second medical device through the patient toward the target position includes: aligning a second axis of the second medical device with the target position; and advancing the second medical device toward the target position; (l) wherein: the second medical device is attached to a robotic arm, and the robotic arm restricts movement of the second medical device to movement along the second axis; (m) wherein the first medical device includes an endoscope; (n) wherein the first medical device is robotically controlled; (o) the second medical device is robotically controlled; (p) wherein the treatment area includes a kidney, a bladder, a lung, or a gastrointestinal tract; (q) wherein the instructions also cause the at least one processor to: determine patient motion using the first position sensor of the first medical device, and wherein percutaneously guiding the second medical device through the patient toward the target position is based in part on the determined patient motion; and / or (r) wherein the patient motion is due to breathing.

[0013] In another aspect, a medical system includes a first medical device configured to be inserted through a natural orifice of the patient into a treatment area of ​​the patient, the first medical device comprising an elongated shaft and a first position sensor; a second medical device configured to be inserted through a percutaneous opening of the patient into the treatment area; at least one computer-readable memory having executable instructions stored thereon; and one or more processors communicating with the at least one computer-readable memory and configured to execute instructions so that the system at least: determines a first position of the first medical device within the treatment area based on an output of the first position sensor; defines a target position within the treatment area that is away from the determined first position; aligns the target position with the determined first position; and percutaneously guides the second medical device through the percutaneous opening toward the target position.

[0014] The system may also include any combination of one or more of the following features: (a) wherein the instructions further cause the system to align the output of the first orientation sensor with the coordinate frame of the preoperative model, and wherein determining the first orientation of the instrument within the treatment area using the first orientation sensor includes determining the first orientation with reference to the preoperative model; (b) wherein limiting the target position within the treatment area includes determining the target position with reference to the preoperative model; (c) wherein determining the target position with reference to the preoperative model includes: displaying the preoperative model to a user and receiving a selection of the target position with reference to the preoperative model; (d) wherein limiting the target position includes determining the difference between the determined first orientation and the target position with reference to the preoperative model; (e) a display configured to display a representation of the target position to the user; (f) wherein limiting the target position within the treatment area includes: capturing one or more intraoperative medical images of the treatment area, and limiting the target position with reference to the one or more intraoperative medical images. (g) wherein the one or more intraoperative medical images include one or more fluoroscopic images; (h) wherein the instructions further cause the system to align the output of the first orientation sensor with the one or more intraoperative medical images; (i) wherein defining a target position within the treatment area away from the determined first orientation includes determining the target position along a first axis extending from a distal end of a slender shaft of the first medical device; (j) wherein guiding a second medical device through the patient toward the target position includes: aligning a second axis of the second medical device with the target position, and advancing the second medical device toward the target position; (k) a robotic arm, wherein the second medical device is attached to the robotic arm, and the robotic arm constrains movement of the second medical device to movement along the second axis; (l) wherein the first medical device includes an endoscope; (m) wherein the first medical device is robotically controlled; and / or (n) wherein the second medical device is robotically controlled.

[0015] On the other hand, a computer-readable medium includes instructions configured to cause at least one processor to: insert a first medical device including an elongated shaft and a first orientation sensor through a natural orifice of the patient into a treatment area of ​​the patient; align an output of the first orientation sensor with a coordinate frame of a preoperative model; display the preoperative model to a user; define an orientation of one or more virtual references to create a boundary with reference to the preoperative model, the orientation of the one or more virtual references being determined based on the aligned output of the first orientation sensor; position the first medical device so that the first orientation sensor is away from the boundary; and guide a second device through a percutaneous opening to or within the treatment area based on the one or more virtual references.

[0016] The computer-readable medium may also include any combination of one or more of the following features: (a) wherein determining the orientation of one or more virtual references includes, for each of the orientations: navigating a first medical device to a location within a treatment region where a virtual reference is to be placed; and defining the location as the orientation of the virtual reference based on an output of a registration of a first orientation sensor; (b) wherein determining the orientation of one or more virtual references includes, for each of the orientations: receiving a user selection of a location where one of the one or more virtual references is to be placed; and determining a virtual reference orientation corresponding to the location with reference to at least a first orientation determined based on the first orientation sensor, wherein the virtual reference orientation is away from the first orientation; (c) wherein a boundary defines a resection volume; (d) wherein the first medical device includes an endoscope; (e) wherein the first medical device is robotically controlled; (f) wherein the treatment region includes a kidney, a bladder, a lung, or a gastrointestinal tract; and / or (g) wherein the preoperative model is determined based on a CT scan.

[0017] On the other hand, a medical system includes: a first medical device, which is configured to be inserted into a treatment area of ​​the patient through a natural orifice of the patient, the first medical device comprising a slender shaft and a first position sensor; a second medical device, which is configured to be inserted into the treatment area through a percutaneous opening of the patient; at least one computer-readable memory, the at least one computer-readable memory having executable instructions stored thereon; and one or more processors, which communicate with the at least one computer-readable memory and are configured to execute instructions so that the system at least: aligns the output of the first position sensor with the coordinate frame of a preoperative model; displays the preoperative model to a user; defines the orientation of one or more virtual references to create a boundary with reference to the preoperative model, the orientation of the one or more virtual references being determined based on the aligned output of the first position sensor; and guides the second medical device through the percutaneous opening to or within the treatment area based on the one or more virtual references.

[0018] The system may also include any combination of one or more of the following features: (a) wherein determining the orientation of one or more virtual references includes, for each of the orientations: navigating a first medical device to a location within the treatment region where the virtual reference is to be placed; and defining the location as the orientation of the virtual reference based on the aligned output of the first orientation sensor; (b) wherein determining the orientation of one or more virtual references includes, for each of the orientations: receiving a user selection of a location where one of the one or more virtual references is to be placed; and determining a virtual reference orientation corresponding to the location with reference to at least a first orientation determined based on the first orientation sensor, wherein the virtual reference orientation is away from the first orientation; (c) wherein a boundary defines a resection volume; (d) wherein the first medical device includes an endoscope; (e) wherein the first medical device is robotically controlled; (f) wherein the treatment region includes a kidney, bladder, lung, or gastrointestinal tract; and / or (g) wherein the preoperative model is determined based on a CT scan. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The disclosed aspects will hereinafter be described in conjunction with the accompanying drawings, which are provided to illustrate and not to limit the disclosed aspects, wherein like reference numerals represent like elements.

[0020] Figure 1 An embodiment of a cart-based robotic system arranged for diagnostic and / or therapeutic bronchoscopy is shown.

[0021] Figure 2 Depicted Figure 1 Other aspects of the robotic system.

[0022] Figure 3 Shown is a ureteroscopy arrangement. Figure 1 Implementation scheme of a robotic system.

[0023] Figure 4 Shown is a vascular procedure arranged Figure 1 Implementation scheme of a robotic system.

[0024] Figure 5 An embodiment of a table-based robotic system arranged for use in a bronchoscopy procedure is shown.

[0025] Figure 6 Provided Figure 5 Alternative views of the robotic system.

[0026] Figure 7 An exemplary system configured to stow a robotic arm is shown.

[0027] Figure 8 An embodiment of a table-based robotic system configured for use in a ureteroscopy procedure is shown.

[0028] Fig. 9 An embodiment of a table-based robotic system configured for laparoscopic procedures is shown.

[0029] Fig.10 Shown with pitch or tilt adjustment Figures 5 to 9 Implementation of a stage-based robotic system.

[0030] Fig.11 Provided Figures 5 to 10 Detailed illustration of the interface between the stage and column of a stage-based robotic system.

[0031] Fig.12 An alternative embodiment of a table-based robotic system is shown.

[0032] Fig.13 Shows Fig.12 End view of a stage-based robotic system.

[0033] Fig.14 An end view of a table-based robotic system with a robotic arm attached thereto is shown.

[0034] Fig.15 An exemplary instrument driver is shown.

[0035] Fig.16 An exemplary medical instrument having a pair of instrument drivers is shown.

[0036] Fig.17 An alternative design of an instrument drive and an instrument is shown, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument.

[0037] Fig.18 An instrument having an instrument-based insertion architecture is shown.

[0038] Fig.19 An exemplary controller is shown.

[0039] Fig. 20 Depicted is a block diagram showing a positioning system that estimates a Figures 1 to 10 The position of one or more elements of a robotic system, such as Figures 16 to 18 The location of the device.

[0040] Figures 21A to 21D Various steps in an embodiment of an endoscope-assisted percutaneous medical procedure in the kidney are shown.

[0041] Fig.21A An example of a first medical device being introduced into a kidney is shown.

[0042] Fig. 21BAn example of determination of a target position remote from a first medical device is shown.

[0043] Fig. 21C An example of alignment of a second medical device with a target location is shown.

[0044] Fig.21D An example of percutaneous insertion of a second medical device to reach a target location is shown.

[0045] Fig.22A and 22B Various steps in an embodiment of an alignment method for aligning a medical device with a target location in some concomitant endoscopic and percutaneous medical procedures are shown.

[0046] Fig.22A An example of a general alignment step is shown during which the distal end of the medical device approaches a target location.

[0047] Fig. 22B An example of a precision alignment step is shown during which the longitudinal axis of the medical device is aligned with the target location.

[0048] Fig.23A and 23B A method for assisting Fig. 22B An exemplary alignment interface for the precision alignment step.

[0049] Fig.23A An example of an alignment interface is shown when the medical device is not aligned with the target location.

[0050] Fig. 23B An example of an alignment interface is shown when a medical device is aligned with a target location.

[0051] Fig.24 Shown include Fig.23A and 23B An exemplary user interface for an alignment interface.

[0052] Fig.25A and 25B Various steps in another embodiment of an endoscope-assisted percutaneous medical procedure in the lung are shown.

[0053] Fig.25A An example of placing or creating a virtual fiducial during a procedure is shown.

[0054] Fig.25B An example of creating a boundary for one or more percutaneous devices based on a virtual fiducial is shown.

[0055] Fig.26 Exemplary embodiments accompanying endoscopic and percutaneous medical procedures in the gastrointestinal tract are shown.

[0056] Fig.27A is a flow chart illustrating an embodiment of a method for performing a medical procedure that includes rendezvousing a medical device with a target location.

[0057] Fig.27B is a flow chart illustrating an embodiment of a method for performing a medical procedure that includes placing a virtual fiducial to define a boundary. DETAILED DESCRIPTION

[0058] 1. Overview .

[0059] Aspects of the present disclosure may be integrated into a robotically enabled medical system that is capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy, and non-invasive procedures such as endoscopy. Among endoscopy procedures, the system may be able to perform bronchoscopy, ureteroscopy, gastroscopy, etc.

[0060] In addition to performing a wide range of procedures, the system may provide additional benefits such as enhanced imaging and guidance to assist the physician. Additionally, the system may provide the physician with the ability to perform procedures from an ergonomic position without the need for awkward arm motions and positions. Additionally, the system may provide the physician with the ability to perform procedures with improved ease of use, such that one or more of the system's instruments may be controlled by a single user.

[0061] For illustrative purposes, various embodiments will be described below in conjunction with the accompanying drawings. It should be understood that many other embodiments of the disclosed concepts are possible, and various advantages can be achieved using the disclosed embodiments. Titles are included herein for reference and to help locate the various sections. These titles are not intended to limit the scope of the concepts described therein. Such concepts may have applicability throughout the specification.

[0062] A. Robotic System – Cart .

[0063] Robot-enabled medical systems can be configured in a variety of ways, depending on the specific procedure. Figure 1An embodiment of a cart-based, robotically enabled system 10 arranged for diagnostic and / or therapeutic bronchoscopy is shown. During bronchoscopy, the system 10 may include a cart 11 having one or more robotic arms 12 to deliver medical instruments such as a steerable endoscope 13 (which may be a procedure-specific bronchoscope for bronchoscopy) to a natural orifice entry point (i.e., the mouth of a patient positioned on a table in this example) to deliver the diagnostic and / or therapeutic tool. As shown, the cart 11 may be positioned near the patient's upper torso to provide access to the entry point. Similarly, the robotic arms 12 may be actuated to position the bronchoscope relative to the entry point. The robotic arms 12 may also be utilized when performing gastrointestinal (GI) procedures with a gastroscope (a specialized endoscope used for GI procedures). Figure 1 The arrangement in . Figure 2 An exemplary embodiment of a cart is depicted in greater detail.

[0064] Continue to refer Figure 1 , once the cart 11 is properly positioned, the robotic arm 12 can insert the steerable endoscope 13 into the patient's body robotically, manually, or in combination. As shown, the steerable endoscope 13 may include at least two telescopic parts, such as an inner guide portion and an outer sheath portion, each of which is connected to a separate instrument driver from a set of instrument drivers 28, each of which is connected to the distal end of a separate robotic arm. This linear arrangement of the instrument drivers 28 that facilitates coaxial alignment of the guide portion with the sheath portion produces a "virtual track" 29, which can be repositioned in space by manipulating one or more robotic arms 12 to different angles and / or orientations. The virtual tracks described herein are depicted using dashed lines in the accompanying drawings, and therefore the dashed lines do not depict any physical structure of the system. The translation of the instrument driver 28 along the virtual track 29 causes the inner guide portion to telescope relative to the outer sheath portion, or to advance or retract the endoscope 13 from the patient. The angle of the virtual track 29 can be adjusted, translated, and pivoted based on clinical applications or physician preferences. For example, in bronchoscopy, the angle and orientation of the virtual track 29 as shown represents a compromise between providing the physician with access to the endoscope 13 while minimizing the friction caused by bending the endoscope 13 into the patient's mouth.

[0065] After insertion, the endoscope 13 can be guided down the patient's trachea and lungs using precise commands from the robotic system until the target destination or surgical site is reached. To enhance navigation through the patient's pulmonary network and / or to reach the desired target, the endoscope 13 can be manipulated to telescopically extend the inner guide portion from the outer sheath portion to obtain enhanced articulation and a larger bending radius. The use of a separate instrument driver 28 also allows the guide portion and the sheath portion to be driven independently of each other.

[0066] For example, the endoscope 13 can be guided to deliver a biopsy needle to a target, such as a lesion or nodule in a patient's lung. The needle can be deployed downward along a working channel that extends the length of the endoscope to obtain a tissue sample to be analyzed by a pathologist. Depending on the pathological results, additional tools can be deployed downward along the working channel of the endoscope for additional biopsies. After identifying that the nodule is malignant, the endoscope 13 can deliver tools through the endoscope to remove potential cancerous tissue. In some cases, diagnostic and therapeutic treatments can be delivered in separate procedures. In these cases, the endoscope 13 can also be used to deliver a benchmark to "mark" the location of the target nodule. In other cases, diagnostic and therapeutic treatments can be delivered during the same procedure.

[0067] The system 10 may also include a movable tower 30 that may be connected to the cart 11 via a support cable to provide control, electronic, fluid, optical, sensor and / or power support to the cart 11. Placing such functionality in the tower 30 allows for a smaller form factor cart 11 that can be more easily adjusted and / or repositioned by the operating physician and his / her staff. Additionally, dividing functionality between the cart / table and the support tower 30 reduces operating room clutter and facilitates improved clinical workflow. While the cart 11 may be positioned close to the patient, the tower 30 may be stowed in a remote location to be out of the way during the procedure.

[0068] To support the robotic system described above, the tower 30 may include components of a computer-based control system that stores computer program instructions, for example, in a non-transitory computer-readable storage medium such as a permanent magnetic storage drive, a solid-state drive, or the like. Whether the execution occurs in the tower 30 or in the cart 11, the execution of these instructions can control the entire system or its subsystems. For example, when executed by a processor of a computer system, the instructions can cause the components of the robotic system to actuate the associated brackets and arm mounts, actuate the robotic arm, and control the medical device. For example, in response to receiving a control signal, a motor in a joint of the robotic arm can position the arm into a specific posture.

[0069] The tower 30 may also include pumps, flow meters, valve controllers, and / or fluid pathways to provide controlled irrigation and aspiration capabilities to systems that may be deployed through the endoscope 13. These components may also be controlled using the computer system of the tower 30. In some embodiments, irrigation and aspiration capabilities may be delivered directly to the endoscope 13 via separate cables.

[0070] The tower 30 may include voltage and surge protectors designed to provide filtered and protected power to the cart 11, thereby avoiding the need to place power transformers and other auxiliary power components in the cart 11, resulting in a smaller, more mobile cart 11.

[0071] The tower 30 may also include support equipment for sensors deployed throughout the robotic system 10. For example, the tower 30 may include optoelectronic equipment for detecting, receiving, and processing data received from optical sensors or cameras throughout the robotic system 10. In conjunction with the control system, such optoelectronic equipment may be used to generate real-time images for display in any number of consoles deployed throughout the system (including display in the tower 30). Similarly, the tower 30 may also include electronic subsystems for receiving and processing signals received from deployed electromagnetic (EM) sensors. The tower 30 may also be used to house and position EM field generators for detection by EM sensors in or on medical devices.

[0072] The tower 30 may also include a console 31 in addition to other consoles available in the rest of the system (e.g., a console mounted on top of a cart). The console 31 may include a user interface and display screen, such as a touch screen, for a physician operator. The console in the system 10 is typically designed to provide both preoperative information and real-time information for robotic control and procedures, such as navigation and positioning information for the endoscope 13. When the console 31 is not the only console available to the physician, it may be used by a second operator (such as a nurse) to monitor the patient's health or vital signs and the operation of the system 10, as well as to provide procedure-specific data, such as navigation and positioning information. In other embodiments, the console 30 is housed in a body separate from the tower 30.

[0073] The tower 30 may be coupled to the cart 11 and the endoscope 13 via one or more cables or connections (not shown). In some embodiments, support functions from the tower 30 may be provided to the cart 11 via a single cable, thereby simplifying and eliminating clutter in the operating room. In other embodiments, specific functions may be coupled in separate wiring and connections. For example, while power may be provided to the cart 11 via a single cable, support for controls, optics, fluidics, and / or navigation may also be provided via separate cables.

[0074] Figure 2 Provided by Figure 1 Detailed illustration of an embodiment of a cart 11 of a cart-based robotic-enabled system is shown. The cart 11 generally includes an elongated support structure 14 (commonly referred to as a "column"), a cart base 15, and a console 16 at the top of the column 14. The column 14 may include one or more brackets, such as for supporting one or more robotic arms 12 ( Figure 2The bracket 17 (alternatively "arm support") may include a separately configurable arm mount that rotates along a vertical axis to adjust the base of the robotic arm 12 to better position it relative to the patient. The bracket 17 also includes a bracket interface 19 that allows the bracket 17 to translate vertically along the column 14.

[0075] The bracket interface 19 is connected to the column 14 through slots, such as slot 20, which are positioned on opposite sides of the column 14 to guide the vertical translation of the bracket 17. The slot 20 contains a vertical translation interface to position and maintain the bracket 17 at various vertical heights relative to the cart base 15. The vertical translation of the bracket 17 allows the cart 11 to adjust the reach of the robotic arm 12 to meet various table heights, patient sizes, and physician preferences. Similarly, the separately configurable arm mounts on the bracket 17 allow the robotic arm base 21 of the robotic arm 12 to be angled in a variety of configurations.

[0076] In some embodiments, the slot 20 may be supplemented with a slot cover that is flush and parallel to the slot surface to prevent dust and fluid from entering the internal cavity of the column 14 and the vertical translation interface when the carriage 17 is vertically translated. The slot cover can be deployed by a pair of spring reels positioned near the vertical top and bottom of the slot 20. The cover is coiled within the reel until it is deployed to extend and retract from the coiled state of the cover when the carriage 17 is vertically translated up and down. When the carriage 17 is translated toward the reel, the spring loading of the reel provides a force to retract the cover into the reel while also maintaining a tight seal when the carriage 17 is translated away from the reel. The cover can be connected to the carriage 17 using, for example, a bracket in the carriage interface 19 to ensure proper extension and retraction of the cover when the carriage 17 is translated.

[0077] Column 14 may internally include mechanisms such as gears and motors designed to mechanized translate carriage 17 using a vertically aligned lead screw in response to control signals generated in response to user input (eg, input from console 16 ).

[0078] The robotic arm 12 may generally include a robotic arm base 21 and an end effector 22 separated by a series of links 23, the series of links being connected by a series of joints 24, each joint including an independent actuator, each actuator including an independently controllable motor. Each independently controllable joint represents an independent degree of freedom available to the robotic arm 12. Each of the robotic arms 12 may have seven joints and therefore provide seven degrees of freedom. Multiple joints result in multiple degrees of freedom, thereby allowing "redundant" degrees of freedom. Having redundant degrees of freedom allows the robotic arm 12 to position its corresponding end effector 22 at a specific position, orientation, and trajectory in space using different connector positions and joint angles. This allows the system to position and guide the medical device from a desired point in space, while allowing the physician to move the arm joint to a clinically advantageous position away from the patient to create greater access while avoiding arm collisions.

[0079] The cart base 15 balances the weight of the column 14, bracket 17, and robotic arm 12 on the floor. Thus, the cart base 15 houses heavier components such as electronics, motors, power supplies, and components that enable movement and / or immobilization of the cart 11. For example, the cart base 15 includes rollable wheel-shaped casters 25 that allow the cart 11 to be easily moved around the room prior to a procedure. After reaching the proper orientation, the casters 25 can be secured using wheel locks to maintain the cart 11 in the proper orientation during the procedure.

[0080] The console 16 positioned at the vertical end of the column 14 allows both a user interface and a display screen (or dual-purpose device, such as, for example, a touch screen 26) for receiving user input to provide both preoperative and intraoperative data to the physician user. Potential preoperative data on the touch screen 26 may include preoperative planning, navigation and mapping data derived from a preoperative computerized tomography (CT) scan, and / or a record from a preoperative patient interview. The intraoperative data on the display may include optical information provided from tools, sensors, and coordinate information from sensors, as well as important patient statistics, such as respiration, heart rate, and / or pulse. The console 16 may be positioned and tilted to allow the physician to approach the console 16 from the side of the column 14 opposite to the bracket 17. From this position, the physician can observe the console 16, the robotic arm 12, and the patient while operating the console 16 from behind the cart 11. As shown, the console 16 also includes a handle 27 to help manipulate and stabilize the cart 11.

[0081] Figure 3An embodiment of a robotically enabled system 10 arranged for ureteroscopy is shown. In a ureteroscopy procedure, a cart 11 can be positioned to deliver a ureteroscope 32 (a procedure-specific endoscope designed to traverse a patient's urethra and ureters) to the patient's lower abdominal region. In a ureteroscopy, it may be desirable to align the ureteroscope 32 directly with the patient's urethra to reduce friction and forces on sensitive anatomical structures in this area. As shown, the cart 11 can be aligned at the foot of the table to allow the robotic arm 12 to position the ureteroscope 32 for direct linear access to the patient's urethra. The robotic arm 12 can insert the ureteroscope 32 directly into the patient's lower abdomen through the urethra along a virtual track 33 from the foot of the table.

[0082] After insertion into the urethra, the ureteroscope 32 can be navigated into the bladder, ureters, and / or kidneys for diagnostic and / or therapeutic applications using similar control techniques as in bronchoscopy. For example, the ureteroscope 32 can be guided into the ureters and kidneys to break up accumulated kidney stones using a laser or ultrasonic lithotripsy device deployed down the working channel of the ureteroscope 32. After lithotripsy is complete, the resulting stone fragments can be removed using a basket deployed down the ureteroscope 32.

[0083] Figure 4 An embodiment of a robotically enabled system 10 similarly arranged for vascular procedures is shown. In a vascular procedure, the system 10 can be configured so that the cart 11 can deliver a medical device 34 (such as a steerable catheter) to an access point in the femoral artery of the patient's leg. The femoral artery presents both a larger diameter for navigation and a relatively less circuitous and tortuous path to the patient's heart, which simplifies navigation. As in a ureteroscopy procedure, the cart 11 can be positioned toward the patient's legs and lower abdomen to allow the robotic arm 12 to provide a virtual track 35 for direct linear access to the femoral artery access point in the patient's thigh / hip region. After insertion into the artery, the medical device 34 can be guided and inserted by translating the instrument drive 28. Alternatively, the cart can be positioned around the patient's upper abdomen to reach alternative vascular access points, such as the carotid and brachial arteries near the shoulder and wrist.

[0084] B. Robotic system – unit .

[0085] Embodiments of the robotic-enabled medical system may also incorporate a patient table. Incorporating a table reduces the amount of capital equipment in the operating room by removing a cart, which allows for greater access to the patient. Figure 5An embodiment of such a robotically enabled system is shown arranged for use in a bronchoscopy procedure. System 36 includes a support structure or column 37 for supporting a platform 38 (shown as a "table" or "bed") on a floor. Much like a cart-based system, the end effector of a robotic arm 39 of system 36 includes an instrument drive 42 designed to manipulate an elongated medical instrument, such as a slender medical instrument, through or along a virtual track 41 formed by the linear alignment of the instrument drive 42. Figure 5 In practice, a C-arm for providing fluoroscopic imaging can be positioned over the upper abdominal region of the patient by placing the emitter and detector around the table 38.

[0086] Figure 6 An alternative view of the system 36 without the patient and medical device for discussion purposes is provided. As shown, the column 37 may include one or more brackets 43 shown as annular in the system 36, on which one or more robotic arms 39 may be based. The bracket 43 may translate along a vertical column interface 44 extending along the length of the column 37 to provide different vantage points from which the robotic arm 39 may be positioned to reach the patient. The bracket 43 may rotate around the column 37 using a mechanical motor positioned within the column 37 to allow the robotic arm 39 to access multiple sides of the table 38, such as both sides of the patient. In an embodiment with multiple brackets, the brackets may be individually positioned on the column and may translate and / or rotate independently of the other brackets. Although the bracket 43 does not need to be around the column 37 or even circular, the annular shape as shown facilitates the rotation of the bracket 43 around the column 37 while maintaining structural balance. The rotation and translation of the bracket 43 allows the system 36 to align medical devices such as endoscopes and laparoscopes to different entry points on the patient. In other embodiments (not shown), the system 36 may include a patient table or bed with an adjustable arm support in the form of a rod or rail extending alongside the patient table or bed. One or more robotic arms 39 may be attached to the adjustable arm support (e.g., via a shoulder with an elbow joint) that may be vertically adjusted. By providing vertical adjustment, the robotic arms 39 advantageously can be compactly stored beneath the patient table or bed and subsequently raised during a procedure.

[0087] The robotic arm 39 may be mounted on the carriage 43 via a set of arm mounts 45 including a series of joints that may be individually rotated and / or telescopically extended to provide additional configurability to the robotic arm 39. Additionally, the arm mounts 45 may be positioned on the carriage 43 so that when the carriage 43 is appropriately rotated, the arm mounts 45 may be positioned on the same side of the table 38 (e.g., Figure 6 As shown), on the opposite side of the platform 38 (as shown Fig. 9 shown) or on an adjacent side of stage 38 (not shown).

[0088] Column 37 provides structural support for stage 38 and provides a path for vertical translation of carriage 43. Internally, column 37 may be equipped with a lead screw for guiding the vertical translation of the carriage, and a motor for mechanizing the lead screw-based translation of carriage 43. Column 37 may also transmit power and control signals to carriage 43 and robotic arm 39 mounted thereon.

[0089] The base 46 has Figure 2 The cart base 15 in the illustrated cart 11 functions similarly, accommodating heavier components to balance the table / bed 38, column 37, bracket 43, and robotic arm 39. The table base 46 may also incorporate rigid casters to provide stability during the procedure. Casters deployed from the bottom of the table base 46 may extend in opposite directions on both sides of the base 46 and retract when the system 36 requires movement.

[0090] continue Figure 6 , the system 36 may also include a tower (not shown) that divides the functions of the system 36 between the table and the tower to reduce the form factor and volume of the table. As in the previously disclosed embodiments, the tower can provide a variety of support functions to the table, such as processing, computing and control capabilities, power, fluid and / or optical and sensor processing. The tower can also be movable to be positioned away from the patient, thereby improving the physician's access and eliminating clutter in the operating room. In addition, placing the components in the tower allows more storage space in the table base 46 for potential stowage of the robotic arm 39. The tower may also include a main controller or console that provides both a user interface (such as a keyboard and / or pendant) for user input and a display screen (or touch screen) for preoperative and intraoperative information (such as real-time imaging, navigation and tracking information). In some embodiments, the tower may also include a holder for a gas cylinder to be used for insufflation.

[0091] In some embodiments, the table base can fold up and store the robotic arm when not in use. Figure 7 A system 47 for stowing the robotic arm in an embodiment of a table-based system is shown. In the system 47, a carriage 48 can be translated vertically into a base 49 to stow a robotic arm 50, an arm mount 51, and the carriage 48 within the base 49. A base cover 52 can be translated and retracted open to deploy the carriage 48, the arm mount 51, and the robotic arm 50 around a column 53, and closed to stow the carriage, the arm mount, and the robotic arm to protect them when not in use. The base cover 52 can be sealed with a membrane 54 along the edges of its opening to prevent dust and fluids from entering when closed.

[0092] Figure 8An embodiment of a robotically enabled table-based system configured for a ureteroscopy procedure is shown. During ureteroscopy, the table 38 may include a rotating portion 55 for positioning the patient at an angle to the column 37 and the table base 46. The rotating portion 55 may rotate or pivot about a pivot point (e.g., located below the patient's head) so as to position the bottom portion of the rotating portion 55 away from the column 37. For example, the pivoting of the rotating portion 55 allows a C-arm (not shown) to be positioned above the patient's lower abdomen without competing for space with the column (not shown) below the table 38. By rotating the bracket 35 (not shown) about the column 37, the robotic arm 39 can insert the ureteroscope 56 directly into the patient's groin area along the virtual track 57 to reach the urethra. During ureteroscopy, stirrups 58 may also be fixed to the rotating portion 55 of the table 38 to support the orientation of the patient's legs during the procedure and allow full access to the patient's groin area.

[0093] In a laparoscopic procedure, minimally invasive instruments may be inserted into the patient's anatomical structure through a small incision in the patient's abdominal wall. In some embodiments, the minimally invasive instrument includes an elongated rigid member, such as a shaft, for accessing the patient's anatomical structure. After the patient's abdominal cavity is inflated, the instrument may be guided to perform a surgical or medical task, such as grasping, cutting, ablating, suturing, etc. In some embodiments, the instrument may include a scope, such as a laparoscope. Fig. 9 An embodiment of a robotically enabled table-based system configured for laparoscopic procedures is shown. Fig. 9 As shown, the carriage 43 of the system 36 can be rotated and vertically adjusted to position the pair of robotic arms 39 on opposite sides of the table 38 so that the arm mounts 45 can be used to position instruments 59 through minimal incisions on both sides of the patient to reach his / her abdominal cavity.

[0094] To accommodate laparoscopic procedures, the robotic-enabled table system can also tilt the platform to a desired angle. Fig.10 An embodiment of a robotic-enabled medical system with pitch or tilt adjustment is shown. Fig.10 As shown, the system 36 can accommodate the tilt of the table 38 to position one portion of the table at a greater distance from the floor than another portion. Additionally, the arm mount 45 can be rotated to match the tilt so that the robotic arm 39 maintains the same planar relationship with the table 38. To accommodate steeper angles, the column 37 can also include a telescoping portion 60 that allows vertical extension of the column 37 to prevent the table 38 from contacting the floor or colliding with the table base 46.

[0095] Fig.11A detailed illustration of the interface between the table 38 and the column 37 is provided. The pitch rotation mechanism 61 can be configured to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom. The pitch rotation mechanism 61 can be implemented by positioning orthogonal axes 1, 2 at the column-to-table interface, each axis being actuated by a separate motor 3, 4 in response to an electrical pitch angle command. Rotation along one screw 5 will enable tilt adjustment in one axis 1, while rotation along another screw 6 will enable tilt adjustment along another axis 2. In some embodiments, a spherical joint can be used to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom.

[0096] For example, pitch adjustment is particularly useful when attempting to position the table in a Trendelenburg position (i.e., positioning the patient's lower abdomen at a higher orientation from the floor than the patient's upper abdomen) for lower abdominal surgery. The Trendelenburg position causes the patient's internal organs to glide toward his / her upper abdomen by gravity, thereby clearing the abdominal cavity to allow minimally invasive tools to enter and perform a lower abdominal surgical or medical procedure, such as a laparoscopic prostatectomy.

[0097] Fig.12 and Fig.13 An isometric view and an end view of an alternative embodiment of a table-based surgical robotic system 100 are shown. The surgical robotic system 100 includes a table 101 that can be configured to support one or more robotic arms (see, e.g., Fig.14 ) is provided with one or more adjustable arm supports 105 attached to the table 101. In the illustrated embodiment, a single adjustable arm support 105 is shown, but additional arm supports may be provided on opposite sides of the table 101. The adjustable arm support 105 may be configured so that it can move relative to the table 101 to adjust and / or change the orientation of the adjustable arm support 105 and / or any robotic arm mounted thereto relative to the table 101. For example, the adjustable arm support 105 may be adjusted in one or more degrees of freedom relative to the table 101. The adjustable arm support 105 provides high flexibility to the system 100, including the ability to easily stow one or more adjustable arm supports 105 and any robotic arm attached thereto below the table 101. The adjustable arm support 105 may be raised from a stowed orientation to a position below the upper surface of the table 101. In other embodiments, the adjustable arm support 105 can be raised from the stowed position to a position above the upper surface of the table 101 .

[0098] The adjustable arm support 105 may provide several degrees of freedom, including lift, lateral translation, tilt, etc. Fig.12 and Fig.13 In the exemplary embodiment of the present invention, the arm support 105 is configured to have four degrees of freedom. Fig.12105 are shown by arrows in . A first degree of freedom allows adjustment of the adjustable arm support 105 in the z-direction ("Z lift"). For example, the adjustable arm support 105 may include a bracket 109 that is configured to move upward or downward along or relative to the column 102 of the support table 101. A second degree of freedom allows the adjustable arm support 105 to tilt. For example, the adjustable arm support 105 may include a swivel joint that allows the adjustable arm support 105 to be aligned with the bed in the Trendelenburg position. A third degree of freedom allows the adjustable arm support 105 to "pivot up", which can be used to adjust the distance between one side of the table 101 and the adjustable arm support 105. A fourth degree of freedom allows the adjustable arm support 105 to translate along the longitudinal length of the table.

[0099] Fig.12 and Fig.13 The surgical robot system 100 in the embodiment of the present invention may include a table supported by a column 102 mounted to a base 103. The base 103 and the column 102 support the table 101 relative to the support surface. The floor axis 131 and the support axis 133 are located at Fig.13 Shown in.

[0100] Adjustable arm support 105 can be mounted to column 102. In other embodiments, arm support 105 can be mounted to table 101 or base 103. Adjustable arm support 105 can include bracket 109, rod or rail connector 111, and rod or rail 107. In some embodiments, one or more robotic arms mounted to rail 107 can translate and move relative to each other.

[0101] The bracket 109 may be attached to the column 102 via a first joint 113 that allows the bracket 109 to move relative to the column 102 (e.g., such as up and down along a first or vertical axis 123). The first joint 113 may provide a first degree of freedom ("Z lift") to the adjustable arm support 105. The adjustable arm support 105 may include a second joint 115 that provides a second degree of freedom (tilt) to the adjustable arm support 105. The adjustable arm support 105 may include a third joint 117 that provides a third degree of freedom ("upward pivot") to the adjustable arm support 105. Additional joints 119 may be provided (in Fig.13 ), the additional joint mechanically constrains the third joint 117 to maintain the orientation of the rail 107 when the rail connector 111 rotates about the third axis 127. The adjustable arm support 105 may include a fourth joint 121, which may provide the adjustable arm support 105 with a fourth degree of freedom (translation) along a fourth axis 129.

[0102] Fig.14An end view of a surgical robotic system 140A having two adjustable arm supports 105A, 105B mounted on opposite sides of a table 101 is shown according to one embodiment. A first robotic arm 142A is attached to a rod or rail 107A of the first adjustable arm support 105B. The first robotic arm 142A includes a base 144A attached to the rail 107A. The distal end of the first robotic arm 142A includes an instrument drive mechanism 146A that can be attached to one or more robotic medical instruments or tools. Similarly, the second robotic arm 142B includes a base 144B attached to the rail 107B. The distal end of the second robotic arm 142B includes an instrument drive mechanism 146B. The instrument drive mechanism 146B can be configured to be attached to one or more robotic medical instruments or tools.

[0103] In some embodiments, one or more of the robotic arms 142A, 142B include an arm with seven or more degrees of freedom. In some embodiments, one or more of the robotic arms 142A, 142B may include eight degrees of freedom, including an insertion axis (including 1 degree of freedom for insertion), a wrist (including 3 degrees of freedom for wrist pitch, yaw, and roll), an elbow (including 1 degree of freedom for elbow pitch), a shoulder (including 2 degrees of freedom for shoulder pitch and yaw), and a base 144A, 144B (including 1 degree of freedom for translation). In some embodiments, the insertion degrees of freedom may be provided by the robotic arms 142A, 142B, while in other embodiments, the instrument itself provides insertion via an instrument-based insertion architecture.

[0104] C. Device Drivers and Interfaces .

[0105] The end effector of the system's robotic arm may include: (i) an instrument driver (alternatively referred to as an "instrument drive mechanism" or "instrument device manipulator") that incorporates an electromechanical device for actuating a medical device; and (ii) a removable or detachable medical device that may be free of any electromechanical components, such as a motor. This dichotomy may be driven by the need to sterilize medical instruments used in medical procedures; and the inability to adequately sterilize expensive capital equipment due to its complex mechanical assemblies and sensitive electronics. Therefore, the medical device may be designed to be disassembled, removed, and interchanged from the instrument driver (and therefore from the system) for separate sterilization or disposal by a physician or physician's staff. In contrast, the instrument driver does not need to be changed or sterilized, and can be covered for protection.

[0106] Fig.15An example instrument driver is shown. The instrument driver 62, positioned at the distal end of the robotic arm, includes one or more drive units 63 arranged in parallel axes to provide controlled torque to a medical device via a drive shaft 64. Each drive unit 63 includes a separate drive shaft 64 for interacting with the instrument, a gear head 65 for converting motor shaft rotation into a desired torque, a motor 66 for generating the drive torque, an encoder 67 for measuring the speed of the motor shaft and providing feedback to the control circuit, and a control circuit 68 for receiving control signals and actuating the drive unit. Each drive unit 63 is independently controlled and motorized, and the instrument driver 62 can provide multiple (e.g., Fig.15 In operation, the control circuit 68 will receive the control signal, transmit the motor signal to the motor 66, compare the resulting motor speed measured by the encoder 67 with the desired speed, and modulate the motor signal to generate the desired torque.

[0107] For the procedures requiring a sterile environment, the robotic system can be combined with a drive interface, such as a sterile adapter connected to a sterile cover, which is located between the instrument driver and the medical device. The main purpose of the sterile adapter is to transfer angular motion from the drive shaft of the instrument driver to the drive input of the instrument, while maintaining the physical separation between the drive shaft and the drive input and thus maintaining sterility. Therefore, an exemplary sterile adapter may include a series of rotational inputs and rotational outputs intended to cooperate with the drive shaft of the instrument driver and the drive input on the instrument. The sterile cover composed of a thin flexible material (such as transparent or translucent plastic) connected to the sterile adapter is designed to cover capital equipment, such as instrument drivers, robot arms and carts (in cart-based systems) or tables (in tables-based systems). The use of the cover will allow capital equipment to be positioned near the patient while still being located in an area that does not require sterilization (i.e., non-sterile area). On the other side of the sterile cover, medical devices can dock with patients in areas that require sterilization (i.e., sterile areas).

[0108] D.Medical devices .

[0109] Fig.16An example medical device with a paired instrument driver is shown. Similar to other instruments designed for use with a robotic system, the medical device 70 includes an elongated shaft 71 (or elongated body) and an instrument base 72. The instrument base 72, also referred to as an "instrument handle" due to its intended design for manual interaction by a physician, can typically include a rotatable drive input 73 (e.g., a socket, pulley, or reel) that is designed to mate with a drive output 74 of a drive interface on an instrument driver 75 extending through a distal end of a robotic arm 76. When physically connected, latched, and / or coupled, the mating drive input 73 of the instrument base 72 can share an axis of rotation with the drive output 74 in the instrument driver 75 to allow torque to be transferred from the drive output 74 to the drive input 73. In some embodiments, the drive output 74 can include a spline that is designed to mate with a socket on the drive input 73.

[0110] The elongated shaft 71 is designed to be delivered through an anatomical opening or lumen (e.g., as in endoscopy) or through a minimally invasive incision (e.g., as in laparoscopy). The elongated shaft 71 can be flexible (e.g., having properties similar to an endoscope) or rigid (e.g., having properties similar to a laparoscope), or a customized combination of both flexible and rigid portions. When designed for laparoscopy, the distal end of the rigid elongated shaft can be connected to an end effector extending from an articulated wrist formed by a connecting fork having at least one degree of freedom and a surgical tool or medical instrument (e.g., a grasper or scissors) that can be actuated based on the force from the tendon when the drive input rotates in response to a torque received from the drive output 74 of the instrument driver 75. When designed for endoscopy, the distal end of the flexible elongated shaft can include a manipulable or controllable curved segment that articulates and bends based on the torque received from the drive output 74 of the instrument driver 75.

[0111] Tendons along the elongated shaft 71 are used to transmit torque from the instrument driver 75 along the elongated shaft 71. These individual tendons (e.g., pull wires) can be individually anchored to individual drive inputs 73 within the instrument handle 72. From the handle 72, the tendons are guided downward along one or more traction cavities of the elongated shaft 71 and anchored at the distal portion of the elongated shaft 71, or in the wrist at the distal portion of the elongated shaft. During surgical procedures such as laparoscopic, endoscopic, or hybrid surgery, these tendons can be connected to a distally mounted end effector, such as a wrist, a grasper, or scissors. In such an arrangement, the torque applied to the drive input 73 transmits tension to the tendons, causing the end effector to actuate in some manner. In some embodiments, during surgical procedures, the tendons can cause the joint to rotate around an axis, causing the end effector to move in one direction or another. Alternatively, the tendons can be connected to one or more jaws of a grasper at the distal end of the elongated shaft 71, wherein the tension from the tendons causes the grasper to close.

[0112] In endoscopy, the tendon can be coupled to a bending or articulation segment located along the elongated shaft 71 (e.g., at the distal end) via an adhesive, a control ring, or other mechanical fastener. When fixedly attached to the distal end of the bending segment, the torque applied to the drive input 73 will be transmitted downward along the tendon, causing the softer bending segment (sometimes referred to as the articulation segment or region) to bend or articulate. Along the unbending segment, it may be advantageous to spiral or swirl a separate traction cavity that guides a separate tendon along the wall (or inside) of the endoscope shaft to balance the radial forces caused by the tension in the traction wire. For specific purposes, the angle of the spiral and / or the spacing therebetween can be varied or designed, wherein a tighter spiral exhibits less axial compression under load force, while a lower spiral amount causes greater axial compression under load force, but limits bending. In another case, the traction cavity can be guided parallel to the longitudinal axis of the elongated shaft 71 to allow controlled articulation in the desired bending or articulation segment.

[0113] In endoscopy, the elongated shaft 71 houses a plurality of components to assist in robotic procedures. The shaft 71 may include a working channel at the distal end of the shaft 71 for deploying surgical tools (or medical devices), flushing and / or aspiration of the surgical area. The shaft 71 may also house wires and / or optical fibers to transmit signals to / from an optical assembly at the distal end, which may include an optical camera. The shaft 71 may also house an optical fiber to carry light from a proximally located light source (e.g., a light emitting diode) to the distal end of the shaft 71.

[0114] At the distal end of the instrument 70, the distal tip may also include an opening for a working channel for delivering tools for diagnosis and / or treatment, flushing and suctioning the surgical site. The distal tip may also include a port for a camera (such as a fiber optic scope or a digital camera) to capture images of the internal anatomical space. Relatedly, the distal tip may also include a port for a light source, which is used to illuminate the anatomical space when the camera is used.

[0115] exist Fig.16 In the example of , the drive shaft axis, and therefore the drive input axis, is orthogonal to the axis of the elongated shaft 71. However, this arrangement complicates the ability of the elongated shaft 71 to roll. Rolling the elongated shaft 71 along its axis while keeping the drive input 73 stationary can cause undesirable tangling of the tendon as it extends out of the drive input 73 and into the retraction cavity within the elongated shaft 71. Such resulting tangling of the tendon can disrupt any control algorithm designed to predict the motion of the flexible elongated shaft 71 during an endoscopic procedure.

[0116] Fig.17 An alternative design of an instrument driver and an instrument is shown, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument. As shown, a circular instrument driver 80 includes four drive units, and its drive output 81 is aligned in parallel at the end of a robot arm 82. The drive units and their respective drive outputs 81 are contained in a rotating assembly 83 of an instrument driver 80 driven by a drive unit in the drive unit in the assembly 83. In response to the torque provided by the rotating drive unit, the rotating assembly 83 rotates along a circular bearing, which connects the rotating assembly 83 to the non-rotating portion 84 of the instrument driver 80. Electric power and control signals can be transmitted to the rotating assembly 83 from the non-rotating portion 84 of the instrument driver 80 by electrical contact, and the electrical contact can be maintained by the rotation of the brush slip ring connection (not shown). In other embodiments, the rotating assembly 83 can be responsive to a separate drive unit integrated into the non-rotating portion 84, and is therefore not parallel to other drive units. The rotating mechanism 83 allows the instrument driver 80 to allow the drive unit and its corresponding drive output 81 to rotate around the instrument driver axis 85 as a single unit.

[0117] Similar to previously disclosed embodiments, the instrument 86 may include an elongated shaft portion 88 and an instrument base 87 (shown with a transparent outer skin for discussion purposes) that includes a plurality of drive inputs 89 (such as sockets, pulleys, and spools) configured to receive drive outputs 81 in the instrument driver 80. Unlike previously disclosed embodiments, the instrument shaft 88 extends from the center of the instrument base 87, with the axis of the instrument base being substantially parallel to the axis of the drive inputs 89, rather than being substantially parallel to the axis of the drive inputs 89. Fig.16 as orthogonal in the design.

[0118] When coupled to the rotation assembly 83 of the instrument driver 80, the medical device 86, including the instrument base 87 and the instrument shaft 88, rotates in combination with the rotation assembly 83 about the instrument driver axis 85. Since the instrument shaft 88 is located at the center of the instrument base 87, the instrument shaft 88 is coaxial with the instrument driver axis 85 when attached. Therefore, the rotation of the rotation assembly 83 causes the instrument shaft 88 to rotate about its own longitudinal axis. In addition, when the instrument base 87 rotates with the instrument shaft 88, any tendons connected to the drive input 89 in the instrument base 87 do not tangle during rotation. Therefore, the parallelism of the axes of the drive output 81, the drive input 89, and the instrument shaft 88 allows the shaft to rotate without tangling any control tendons.

[0119] Fig.18 An instrument with an instrument-based insertion architecture according to some embodiments is shown. The instrument 150 can be coupled to any of the instrument drivers described above. The instrument 150 includes an elongated shaft 152, an end effector 162 connected to the shaft 152, and a handle 170 coupled to the shaft 152. The elongated shaft 152 includes a tubular member having a proximal portion 154 and a distal portion 156. The elongated shaft 152 includes one or more channels or grooves 158 along its outer surface. The groove 158 is configured to receive one or more wires or cables 180 passing through the groove. Therefore, one or more cables 180 extend along the outer surface of the elongated shaft 152. In other embodiments, the cable 180 may also pass through the elongated shaft 152. The manipulation of the one or more cables 180 (e.g., via the instrument driver) causes the actuation of the end effector 162.

[0120] The instrument handle 170 (also referred to as the instrument base) may generally include an attachment interface 172 having one or more mechanical input members 174, such as a socket, pulley, or spool, which are designed to reciprocate with one or more torque connectors on the attachment surface of the instrument driver.

[0121] In some embodiments, the instrument 150 includes a series of pulleys or cables that enable the elongated shaft 152 to translate relative to the handle 170. In other words, the instrument 150 itself includes an instrument-based insertion architecture that accommodates instrument insertion, thereby minimizing reliance on a robotic arm to provide insertion of the instrument 150. In other embodiments, a robotic arm may be largely responsible for instrument insertion.

[0122] E. Controller .

[0123] Any of the robotic systems described herein may include an input device or controller for manipulating an instrument attached to the robotic arm. In some embodiments, the controller may be coupled to the instrument (e.g., communicatively, electronically, electrically, wirelessly, and / or mechanically) such that manipulation of the controller causes corresponding manipulation of the instrument, e.g., via master-slave control.

[0124] Fig.19 1 is a perspective view of an embodiment of the controller 182. In this embodiment, the controller 182 comprises a hybrid controller that can have both impedance and admittance control. In other embodiments, the controller 182 can utilize only impedance or passive control. In other embodiments, the controller 182 can utilize only admittance control. By being a hybrid controller, the controller 182 can advantageously have lower perceived inertia when in use.

[0125] In the illustrated embodiment, the controller 182 is configured to allow manipulation of two medical devices and includes two handles 184. Each of the handles 184 is connected to a gimbal 186. Each gimbal 186 is connected to a positioning platform 188.

[0126] like Fig.19 As shown, each positioning platform 188 includes a SCARA arm (selective compliance assembly robot arm) 198 coupled to the column 194 via a prismatic joint 196. The prismatic joint 196 is configured to translate along the column 194 (e.g., along the rail 197) to allow each of the handles 184 to translate in the z direction, thereby providing a first degree of freedom. The SCARA arm 198 is configured to allow the handles 184 to move in the xy plane, thereby providing two additional degrees of freedom.

[0127] In some embodiments, one or more load sensors are positioned in the controller. For example, in some embodiments, a load sensor (not shown) is positioned in the body of each gimbal in the gimbal 186. By providing a load sensor, a portion of the controller 182 can be operated under admittance control, thereby advantageously reducing the perceived inertia of the controller when in use. In some embodiments, the positioning platform 188 is constructed for admittance control, and the gimbal 186 is constructed for impedance control. In other embodiments, the gimbal 186 is constructed for admittance control, and the positioning platform 188 is constructed for impedance control. Therefore, for some embodiments, the translational degree of freedom or the azimuth degree of freedom of the positioning platform 188 can rely on admittance control, while the rotational degree of freedom of the gimbal 186 relies on impedance control.

[0128] F. Navigation and Control .

[0129] Conventional endoscopy may involve the use of fluoroscopy (e.g., as may be delivered by a C-arm) and other forms of radiation-based imaging modalities to provide intracavitary guidance to the operating physician. In contrast, the robotic system contemplated by the present disclosure may provide non-radiation-based navigation and positioning devices to reduce physician exposure to radiation and reduce the amount of equipment in the operating room. As used herein, the term "positioning" may refer to determining and / or monitoring the orientation of an object in a reference coordinate system. Technologies such as preoperative mapping, computer vision, real-time EM tracking, and robotic command data may be used alone or in combination to achieve a radiation-free operating environment. In other cases where radiation-based imaging modalities are still used, preoperative mapping, computer vision, real-time EM tracking, and robotic command data may be used alone or in combination to improve information obtained only by radiation-based imaging modalities.

[0130] Fig. 20 is a block diagram illustrating a positioning system 90 for estimating the position of one or more elements of a robotic system, such as the position of an instrument, according to an example embodiment. The positioning system 90 may be a set of one or more computer devices configured to execute one or more instructions. The computer device may be embodied by a processor (or multiple processors) and a computer readable memory in one or more of the components discussed above. By way of example and not limitation, the computer device may be located at Figure 1 The tower 30 shown, Figures 1 to 4 The cart 11 shown, Figures 5 to 14 The bed shown, etc.

[0131] like Fig. 20 As shown, the positioning system 90 may include a positioning module 95 that processes the input data 91-94 to generate position data 96 for the distal tip of the medical device. The position data 96 may be data or logic that represents the position and / or orientation of the distal end of the device relative to a reference frame. The reference frame may be a reference frame relative to the patient's anatomy or a known object such as an EM field generator (see below for a discussion of EM field generators).

[0132] The various input data 91-94 are now described in more detail. Preoperative mapping can be accomplished using a collection of low-dose CT scans. The preoperative CT scans are reconstructed into three-dimensional images that are visualized, for example, as "slices" of cross-sectional views of the patient's internal anatomical structures. When analyzed as a whole, image-based models of anatomical cavities, spaces, and structures for the patient's anatomical structures (such as the patient's lung network) can be generated. Techniques such as centerline geometry can be determined and approximated from CT images to form a three-dimensional volume of the patient's anatomical structure, which is referred to as model data 91 (also referred to as "preoperative model data" when generated using only preoperative CT scans). The use of centerline geometry is discussed in U.S. Patent Application No. 14 / 523,760, the contents of which are incorporated herein in their entirety. Network topology models can also be derived from CT images and are particularly suitable for bronchoscopy.

[0133] In some embodiments, the instrument can be equipped with a camera to provide visual data (or image data) 92. The positioning module 95 can process the visual data 92 to implement one or more vision-based (or image-based) position tracking modules or features. For example, the preoperative model data 91 can be used in conjunction with the visual data 92 to implement computer vision-based tracking of a medical instrument (e.g., an endoscope or an instrument advanced through a working channel of an endoscope). For example, using the preoperative model data 91, the robotic system can generate a library of expected endoscopic images based on the model based on the expected path of travel of the endoscope, each image being connected to a location within the model. During a surgical procedure, the robotic system can reference the library to compare real-time images captured at a camera (e.g., a camera at the distal end of an endoscope) with those in the image library to assist in positioning.

[0134] Other computer vision based tracking techniques use feature tracking to determine the motion of the camera, and therefore the motion of the endoscope. Some features of the localization module 95 may identify circular geometric structures in the preoperative model data 91 that correspond to anatomical cavities and track changes in those geometric structures to determine which anatomical cavity is selected, as well as track relative rotational and / or translational motion of the camera. The use of topological maps may further enhance vision based algorithms or techniques.

[0135] Optical flow (another computer vision-based technique) can analyze the displacement and translation of image pixels in a video sequence in the visual data 92 to infer camera motion. Examples of optical flow techniques can include motion detection, object segmentation calculations, brightness, motion compensated encoding, stereo disparity measurements, etc. Through multiple iterations of multi-frame comparisons, the motion and position of the camera (and therefore the endoscope) can be determined.

[0136] The positioning module 95 can use real-time EM tracking to generate the real-time position of the endoscope in a global coordinate system that can be aligned to the patient's anatomical structure represented by the preoperative model. In EM tracking, an EM sensor (or tracker) comprising one or more sensor coils in one or more positions and orientations embedded in a medical device (e.g., an endoscopic tool) measures changes in the EM field generated by one or more static EM field generators positioned at known locations. The position information detected by the EM sensor is stored as EM data 93. The EM field generator (or transmitter) can be placed close to the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coil of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be "aligned" to the patient's anatomical structure (e.g., a preoperative model) during the surgical procedure to determine a geometric transformation that aligns a single position in the coordinate system with a position in the preoperative model of the patient's anatomical structure. Once registered, an embedded EM tracker in one or more locations of a medical device (eg, the distal tip of an endoscope) can provide a real-time indication of the progress of the medical device through the patient's anatomy.

[0137] The robotic commands and kinematic data 94 may also be used by a positioning module 95 to provide orientation data 96 for the robotic system. The pitch and yaw of the device derived from the joint motion commands may be determined during preoperative calibration. These calibration measurements may be used in conjunction with known insertion depth information to estimate the orientation of the instrument during the surgical procedure. Alternatively, these calculations may be analyzed in conjunction with EM, vision, and / or topological modeling to estimate the orientation of the medical device within the network.

[0138] like Fig. 20 As shown, the positioning module 95 can use a variety of other input data. For example, although Fig. 20 Not shown, but an instrument utilizing shape sensing fibers may provide shape data that positioning module 95 may use to determine the position and shape of the instrument.

[0139] The localization module 95 may use the input data 91-94 in combination. In some cases, such a combination may use a probabilistic approach, where the localization module 95 assigns a confidence weight to a position determined from each of the input data 91-94. Thus, in situations where the EM data may be unreliable (such as where EM interference may be present), the confidence in the position determined by the EM data 93 may be reduced, and the localization module 95 may rely more heavily on the visual data 92 and / or the robot command and kinematics data 94.

[0140] As discussed above, the robotic systems discussed herein may be designed to incorporate a combination of one or more of the above techniques. A computer-based control system for a robotic system located in a tower, bed, and / or cart may store computer program instructions, for example, in a non-transitory computer-readable storage medium (such as a permanent magnetic storage drive, a solid-state drive, etc.) that, when executed, causes the system to receive and analyze sensor data and user commands, generate control signals for the entire system, and display navigation and positioning data, such as the position of an instrument in a global coordinate system, an anatomical map, etc.

[0141] 2. Robotic systems and methods for accompanying endoscopic and percutaneous medical procedures .

[0142] Embodiments of the present disclosure relate to robotic systems and methods for accompanying endoscopic and percutaneous medical procedures, such as endoscopically assisted percutaneous (or laparoscopic) medical procedures and laparoscopically assisted endoscopic procedures. The systems and methods may be embodied or employed in a robotically enabled medical system, such as the one described above with reference to Figures 1 to 20 Those described and those described in further detail below.

[0143] Many medical procedures involve guiding a medical device to a target location within a patient's treatment area. In some cases, these medical procedures may involve percutaneously guiding a medical device through an opening (such as a percutaneous access port) to reach the target area. Because the target location is typically internal, it may be difficult to accurately guide the medical device to the target location. Typically, physicians, particularly highly trained radiologists, rely on real-time fluoroscopic images to guide the medical device to the target location. However, such procedures are difficult. The radiologist must derive a three-dimensional path to the target location based on the two-dimensional fluoroscopic images, which often results in inaccurate guidance and placement of the medical device. In addition, the use of fluoroscopic fluoroscopy can expose patients and medical staff to radiation for an undesirable amount of time.

[0144] For example, percutaneous nephrolithotomy (PCNL) is a medical procedure involving percutaneous access to the kidney to remove kidney stones. Typically, PCNL is performed in two steps. First, a radiologist guides an interventional sheath percutaneously into the kidney to enter the treatment area. The radiologist relies on two-dimensional fluoroscopic images to guide and place the interventional sheath. Second, with the interventional sheath remaining in place, the urologist then enters the treatment area through the interventional sheath to remove the kidney stones. Radiologists usually need to perform the first part of the procedure, which increases the cost, complexity and operation scheduling time delay of the procedure, while in an ideal situation, the procedure only requires urologists and their staff to perform. In addition, because radiologists have not been trained in urology, radiologists usually place the interventional sheath in a suboptimal position. However, urologists must rely on radiologists to place the interventional sheath because urologists have not been trained in radiology and therefore cannot guide the interventional sheath themselves. Because radiologists rely on fluoroscopic imaging technology, another disadvantage of PCNL is the radiation exposure mentioned above.

[0145] The methods and systems described in this patent application for accompanying endoscopic and percutaneous medical procedures can provide improved guidance and placement of percutaneous and / or endoscopic medical devices. As will be described in more detail with reference to the examples illustrated below, the methods and systems can be advantageously used to define a rendezvous point for medical devices within a treatment area and / or to define boundaries within a treatment area for use during a procedure.

[0146] FIG. 21A to FIG. 21D Various steps in an embodiment of an endoscope-assisted percutaneous medical procedure in a kidney 202 are shown. In the illustrated example, the medical procedure is an endoscope-assisted percutaneous nephrolithotomy (PCNL) procedure for removing a kidney stone 212, but the principles illustrated by the example are applicable to other types of medical procedures, all of which are intended to be within the scope of the present disclosure. As will be described in detail below, in the illustrated example, a first medical device 204 is inserted through a natural orifice of a patient into the kidney 202 ( Fig.21A The first medical device 204 includes a position sensor 206. The position sensor 206 is referenced to determine a target position 208 for guiding the percutaneous insertion of the second medical device 214 ( Fig. 21B ). The target location 208 may be remote from the orientation sensor 206, as shown. In some embodiments, the target location 208 may be a location (such as a point in space) or a trajectory (such as a line in space). In some embodiments, the target location 208 may be displayed as a point or a line on a graphical user interface. Align the second medical device 214 with the target location 208 ( Fig. 21C Finally, a second medical device 214 is percutaneously inserted toward the target location 208 ( Fig.21D). In this orientation, PCNL may be performed by the second medical device 214. In some embodiments, PCNL may be assisted by the first medical device 204.

[0147] As will be described in more detail below, FIG. 21A to FIG. 21D The illustrated endoscope-assisted PCNL procedure may provide several advantages. For example, the procedure may allow for improved accuracy of percutaneous placement of the second medical device 214, because the insertion of the second medical device 214 may be guided toward the target location 208. In addition, in some embodiments, the procedure allows for the guidance of the second medical device 214 without fluoroscopic visualization. This may advantageously reduce or eliminate radiation exposure to the patient and medical staff during the procedure. This may also simplify the procedure, because it may allow the procedure to be performed entirely by a urologist, rather than by a radiologist and a urologist at the same time. In addition, in some embodiments, because the target location 208 may be determined with reference to the position sensor 206, but at a position away from the position sensor 206, the target location 208 may be determined at a position that the first medical device 204 cannot directly approach. For example, in some embodiments, the procedure may allow for the placement of the portal in a single attempt. In contrast, other techniques (such as fluoroscopic fluoroscopic guidance techniques) sometimes require the physician to withdraw and replace tools multiple times until the correct position is finally achieved. These and other advantages will be described in more detail below with more particular reference to the accompanying drawings.

[0148] As mentioned above, FIG. 21A to FIG. 21D The various steps of an endoscopically assisted percutaneous nephrolithotomy (PCNL) procedure for removing a kidney stone 212 are shown. Fig.21A As shown, in the illustrated example, the kidney stone 212 is located at the entrance of the renal calyx. In some cases, the urologist may wish to insert an introducing sheath into the renal calyx in the kidney 202 and behind the kidney stone 212. PCNL can then be performed through the introducing sheath to break up and remove the kidney stone 212. Accurate placement of the introducing sheath may be important to maximize the efficiency of the procedure and minimize stress and impact on the patient.

[0149] In the illustrated example, in order to accurately place the introduction sheath, first, the first medical device 204 can be introduced into the kidney 202. In some embodiments, the first medical device 204 is inserted into the patient through a natural orifice. For example, the first medical device 204 can be inserted into the kidney 202 through the urethra, bladder, and ureter. In some embodiments or in other procedures, other natural patient orifices can be used. In some embodiments, the first medical device 204 can be inserted percutaneously.

[0150] The first medical device 204 can be an endoscope. In some embodiments, the first medical device 204 can be robotically controlled. For example, the first medical device can be the above reference Figures 1 to 20 In some embodiments, the first medical device 204 may be manually controlled. Fig.21A As shown, the first medical device 204 may include an elongated shaft 205. The elongated shaft 205 may be articulated and controllable so that the first medical device 204 can be navigated through the patient's anatomy into the kidney 202. Figures 16 to 18 Several embodiments of such medical devices are described. In some embodiments, the medical device 204 may also include various other features, such as an optical system (such as a camera) that may allow an operator to visualize the treatment area from the perspective of the first medical device 204, and a working channel that may allow additional medical tools or instruments to be delivered to the treatment area through the elongated shaft 205. In some embodiments, the working channel may be used to deliver fluids or to aspirate fluids or debris.

[0151] Likewise Fig.21A As shown, the first medical device 204 may include a position sensor 206. In the illustrated embodiment, the position sensor 206 is positioned at or near the distal end of the elongated shaft 205 of the first medical device 204. In other embodiments, the position sensor 206 may be positioned in other locations on the elongated shaft 205. In some embodiments, the first medical device 204 includes a plurality of position sensors 206. The position sensor 206 is configured to provide an output from which the position or position of the position sensor 206 (and the first medical device 204) can be determined. In some embodiments, the position sensor 206 includes an electromagnetic (EM) sensor configured to generate a detectable signal within an EM field from which the position of the EM sensor within the EM field can be determined. In some embodiments, the position sensor 206 includes a shape sensing fiber from which the pose or shape of the elongated shaft 205 can be determined and used to determine the position of the medical device 204. In other embodiments, other types of position sensors can be used.

[0152] like Fig.21A As shown, during the procedure, a first medical device 204 is guided into the kidney 202. In some embodiments, the guidance of the first medical device 204 is facilitated by a navigation or positioning system, such as the positioning system 90 described above. The operator can visualize the kidney stone 212 using an optical system on the first medical device 204.

[0153] like Fig. 21B As shown, with the first medical device 204 positioned within the kidney 202, the operator (or system) can determine a target location 208. The target location 208 can represent a location where a second medical device 214 is desired to be placed (see Fig. 21C and 21D). As will be referred to below Fig. 21C and 21D As described, the target location 208 can provide a beacon for guiding the insertion of the second medical device 214. For example, the second medical device 214 can be guided so as to meet the target location 208.

[0154] In some embodiments, the target location 208 can be determined with reference to the position sensor 206, but at a location away from the position sensor 206. That is, the target location 208 need not coincide with the position sensor 206 (or any other point on the first medical device 204).

[0155] For example, in some embodiments, the output of the position sensor 206 can be registered with a preoperative model of the kidney 202. The registration can involve, for example, using the position sensor 206 to map the path of the first medical device 204 through the anatomical structure and matching the mapped path with the preoperative model. As another example, the registration can involve navigating the medical device 204 to one or more anatomical landmarks and using the output of the position sensor 206 at the one or more anatomical landmarks to align the preoperative model with the output of the position sensor 206. The above reference to the positioning system 90 and Fig. 20 Additional details of registration are provided. Once the preoperative model is registered, in some embodiments, an operator (or system) can determine a target location 208 by selecting the target location 208 within the preoperative model. For example, the preoperative model can be displayed to the operator, and the operator can select a location from within the preoperative model as the target location 208. As described above, the target location 208 can be remote from the position sensor 206. In some embodiments, using the preoperative model and data from the position sensor 206, the distance and direction between the position of the position sensor 206 and the selected target location 208 can be determined by the system. For example, the position of the position sensor 206 can be represented as x, y, z coordinates, the target location 208 can be represented by x', y', z' coordinates, and the distance and direction between the x, y, z coordinates and the x', y', z' coordinates can be determined.

[0156] As another example, the target position 208 can be determined with reference to intraoperative medical imaging (such as one or more fluoroscopic images). For example, the operator can select a position on the fluoroscopic image as the target position 208. Similarly, the target position 208 can be far away from the position sensor 206. In some embodiments, the output of the position sensor can be registered with the fluoroscopic image so that the relationship between the target position 208 and the position sensor 206 can be determined. For example, the position sensor 206 can be an electromagnetic (EM) sensor. The EM base frame can be registered to the preoperative CT image by moving an instrument including the EM sensor back and forth in a known bronchial branch. This can generate corresponding point pairs in two coordinate frames (i.e., the EM base frame and the fluoroscopic frame). Using the corresponding point pairs, an algorithm can be used to determine the transformation between the coordinate frames. This process is generally or collectively referred to as registration. Once the transformation is determined, the position sensor 206 can be shown in the fluoroscopic image frame, and the relationship between the target position 208 defined in the image frame and the sensor can be determined.

[0157] In some embodiments, the target location 208 is selected along an axis 216 extending outwardly from the distal end of the elongated shaft 205 of the first medical device 204 (e.g., Fig. 21B ). For example, the target position 208 can be a projection of the position of the position sensor 206 along the axis 216. In other embodiments, the target position 208 need not be located on the axis 216. That is, the target position 208 can be a projection of the position sensor 206 in any direction.

[0158] Selecting a target location 208 that is away from the orientation sensor 206 can advantageously allow the use of a target location 208 that is not directly accessible to the first medical instrument 204. For example, in the illustrated example, a kidney stone 212 is located in the entrance of the renal calyx. In this orientation, the kidney stone 212 may prevent the medical instrument 204 from navigating into the renal calyx. Advantageously, the target location 208 can be projected into the renal calyx even if the first medical instrument 204 cannot physically navigate into the renal calyx. Similarly, the target location 208 can be determined at an orientation that cannot even be visualized with the first medical instrument 204. In the illustrated example, the kidney stone 212 may even block visual access to the renal calyx. Regardless, the target location 208 can advantageously be projected into the renal calyx.

[0159] like Fig. 21C As shown, the target location 208 can be used as a beacon to align the second medical device 214. In the illustrated embodiment, the second medical device 214 can be a percutaneous interventional sheath. In other embodiments, the second medical device can be, for example, other types of medical devices, such as endoscopes or laparoscopic tools. In the illustrated embodiment, the second medical device 214 can be manipulated until the axis 218 of the second medical device 214 is aligned with the target location 208.

[0160] In some embodiments, the second medical device 214 can be positioned on an instrument positioning device such as a robotic arm. The instrument positioning device can be robotically controlled to automatically align the second medical device 214 with the target location. For example, the second medical device 214 can be robotically aligned with the target location 208 based on the orientation of the instrument positioning device or the output of an orientation sensor on the second medical device 214.

[0161] In some embodiments, the second medical device 214 can be manually aligned with the target location 208. In such embodiments, the second medical device 214 can include one or more position sensors for determining the position and orientation of the second medical device 214. Fig.22A and 22B An exemplary process for aligning the second medical device 214 is described in further detail.

[0162] like Fig.21D As shown, with the second medical device 214 aligned with the target location 208, the second medical device 214 can be guided or inserted to meet the target location 208. In the illustrated embodiment, the second medical device 214 is percutaneously inserted into the kidney 202, for example, through a percutaneous opening 220. In some embodiments, the percutaneous opening 220 is formed by the second medical device 214. In some embodiments, the percutaneous opening 220 is separately formed, for example, by a separate medical tool, before the second medical device 214 is inserted.

[0163] Insertion of the second medical device 214 can be performed in a variety of ways. For example, the insertion can be manual or robotic. In some manual embodiments, the physician manually inserts the second medical device 214. The physician can guide the second medical device 214 toward the target location 208. In some embodiments, an alignment interface can be provided that helps the physician maintain alignment of the medical device 214 with the target location 208 during insertion. Fig.23A and 23B An exemplary alignment interface is shown.

[0164] In some embodiments, the insertion of the second medical device 214 may be robotic. The second medical device 214 may be positioned on an instrument positioning device such as a robot arm. In some embodiments, when the physician inserts the second medical device 214 by physically manipulating (e.g., pushing) the second medical device 214 (or the instrument positioning device), the instrument positioning device remains aligned during insertion. In these embodiments, the alignment can be maintained robotically while the actual insertion of the second medical device 214 is performed manually. For example, the physician may hold the handle on the second medical device 214 or the instrument positioning device to push the second medical device 214 into the patient's body. The instrument positioning device may limit or restrict the movement of the second medical device 214 to movement along the insertion axis. For example, the instrument positioning device may limit or prevent movement that would cause the second medical device 214 to be out of alignment with the target position 208. In some embodiments, the movement is limited to only along the insertion axis. In some embodiments, the movement is limited to a range around the insertion axis. For example, the movement may be limited to a conical or cylindrical boundary around the insertion axis. In some embodiments, the movement is limited by a tactile boundary. Haptic boundaries can physically limit movement outside of the allowed range or provide tactile feedback to the physician that movement is off track.

[0165] In some embodiments, the physician commands the insertion using a controller, and both alignment and insertion are performed robotically. As with the previous example, the robotic system can restrict or prevent motion so that the second medical device 214 remains aligned with the target location 208.

[0166] In some embodiments, during alignment of the second medical device 214 ( Fig. 21C ) and insert( Fig.21D ), the first medical device 204 remains in the treatment area (e.g., in the kidney 202). The output signal of the position sensor 206 can be used to track patient motion during the procedure. Patient motion can include breathing. For example, when the patient breathes, the kidney 202 can move slightly. This motion can be tracked using the position sensor 206. This patient motion can then be compensated during alignment and insertion of the second medical device 214, so that the accuracy of the second medical device 214 meeting the target location 208 is improved. In some embodiments, in order to accurately track and compensate for patient motion, the first medical device 204 should remain positioned during alignment and insertion so that the position sensor 206 remains close to the target location 208.

[0167] In some embodiments, when the second medical device 214 reaches Fig.21D When the second medical device 214 is at the target position 208 shown, the insertion of the second medical device 214 stops. Fig.21D In the orientation shown, a physician may perform PCNL to remove a kidney stone 212 .

[0168] As mentioned above, Fig.22A and 22B Various steps in an embodiment of an alignment process that may be used to align the second medical device 214 with the target location 208 are shown. Fig.22A An example of a general alignment step is shown during which the distal end of the second medical instrument 214 approaches the target location 208, and Fig. 22B An example of a precision alignment step is shown during which the axis 218 of the second medical device 214 is aligned with the target location 208. In some embodiments, Fig.22A and 22B The alignment process described is for manual alignment and insertion of the second medical device 214. In some embodiments, a similar process may be employed for robotic alignment and insertion.

[0169] like Fig.22A As shown, the alignment process can involve first generally positioning the second medical device 214 relative to the target location 208. As shown, this can involve moving the second medical device 214 along the surface of the patient's skin 203 or near the surface of the patient's skin until the distal tip of the second medical device 214 is positioned within a zone 226 (visually represented by the area between two dashed lines) proximate the target location 208. The orientation of the zone 226 can be determined as an anatomical region proximate the target location 208 or an anatomical region where percutaneous insertion is desired. The orientation of the distal tip of the second medical device 214 can be tracked using an orientation sensor 222, which can be similar to any orientation sensor previously described. As shown, the physician can, for example, move the second medical device 214 back and forth in the direction of arrow 224 until the orientation sensor 222 is within the zone 226. When the orientation sensor 222 is within the zone 226, the system can provide an alert to the physician. The alert can, for example, be audible or visual. As Fig.22A As shown, during this step, the axis 218 of the second medical instrument 214 does not necessarily need to be aligned with the target location 208. Instead, this step can be directed to merely bringing the second medical instrument 214 into the zone 226 proximate to the target location 208.

[0170] Once the second medical device 214 is positioned within the zone 226 (to provide general alignment), the physician may then work to align the axis 218 of the second medical device 214 with the target location 208. An example of this step is shown in FIG. Fig. 22B. During this step, the physician may hold the distal tip of the second medical instrument 214 in place and rotate or pivot the second medical instrument 214 about that point until the axis 218 is aligned with the target location 208. For example, the physician may rotate or pivot the second medical instrument 214 in the direction of arrow 228 until the axis 218 is aligned with the target location 208. This may provide a precise alignment for the second medical instrument 214 and the target location 208.

[0171] An alignment interface may be used to facilitate alignment of the axis 218 with the target location 208. In some embodiments, the alignment interface is a graphical user interface that provides a visual representation of the alignment. Fig.23A and 23B An example of such an alignment interface is shown. In other embodiments, the alignment interface may, for example, use other methods to indicate alignment, such as auditory cues.

[0172] Fig.23A and 23B A method for assisting Fig. 22B An exemplary alignment interface 230 of a precision alignment step. Fig.23A and 23B , the upper portion of the drawing shows an example of the alignment interface 230, while the lower portion of the drawing shows a corresponding example of how the second medical device 214 and the target location 208 are aligned.

[0173] like Fig.23A As shown, the alignment interface 230 may include a graphical or visual representation of the alignment. In the illustrated embodiment, the alignment interface 230 includes an outer circle 232, a target indicator 234, and an instrument indicator 236. The target indicator 234 may indicate the orientation of the target location 208. In some embodiments, the target indicator 234 remains at the center of the circle 232. The instrument indicator 236 may indicate the current alignment of the second medical instrument 214 relative to the target location 208. As the second medical instrument 214 moves, the instrument indicator 236 moves around the circle 232 when the relative alignment between the second medical instrument 214 and the target location 208 changes.

[0174] In some embodiments, the alignment interface 230 represents an alignment view along the axis of the second medical device 214. For example, Fig.23A As shown, the instrument indicator 236 is positioned above and to the left of the target indicator 234. This may indicate that the second medical instrument 214 is offset in a direction above and to the left of the target location 208. The physician may interpret the alignment interface 230 to know that the second medical instrument 214 should be pivoted to the lower right in order to align with the target location 208.

[0175] Fig. 23BThe alignment interface 230 is shown when the second medical device 214 is aligned with the target location 208. As shown, when aligned, the instrument indicator 236 can overlap the target indicator 234. The physician can first use the alignment interface 230 to properly align the second medical device 214 with the target location 208. The physician can then continue to use the alignment interface 230 during insertion to maintain alignment during insertion. As the physician inserts the second medical device 214, the physician can focus on maintaining alignment by keeping the instrument indicator 236 overlapped with the target indicator 234. The user can then continue to insert the second medical device 214 until the desired depth is reached.

[0176] The alignment interface 230 may be displayed to a user, for example, as part of a graphical user interface of the robotic system. Fig.24 An example of such a graphical user interface 235 including an alignment interface 230 is shown. The graphical user interface 235 may include various screen portions for displaying information to a user. For example, in the illustrated embodiment, the graphical user interface 235 includes an alignment indicator 230, an endoscopic view 238, and a model or fluoroscopic view 240. The endoscopic view 238 can display a real-time view from the optical system on the first medical device 204. The model or fluoroscopic view 240 can display a preoperative view of a real-time fluoroscopic view of the treatment area. In some embodiments, the target position 208 can be displayed on the model or fluoroscopic view 240. In some embodiments, the physician can use the model or fluoroscopic view 240 to select the target position 208. Fig.24 The graphical user interface 235 shown in FIG. 1 is provided by way of example only. Fig.24 Other graphical user interfaces 235 showing more, less, or other types of information.

[0177] Fig.25A and 25B Various steps in another embodiment of an endoscope-assisted percutaneous medical procedure in lung 302 are shown. In the illustrated example, the medical procedure is an endoscope-assisted resection procedure for removing a portion of lung 302, but the principles illustrated by the example are also applicable to other types of medical procedures, all of which are intended to be within the scope of the present disclosure. As will be described in detail below, in the illustrated example, a first medical device 304 is inserted into lung 302 through a natural orifice of a patient ( Fig.25A The first medical device 304 includes a position sensor 306. The position of one or more virtual references 308 used to define the resection boundary can be determined with reference to the position sensor 306 ( Fig.25A Boundary 309 may be determined based on virtual reference 308 ( Fig.25BOne or more second medical devices 314 may perform percutaneous resection of a portion of lung 302 based on virtual reference 308 and boundary 309 ( Fig.25B ).

[0178] As will be described in more detail below, Fig.25A and 25B The endoscopically assisted resection procedure shown can provide several advantages. For example, the procedure can allow for improved accuracy in defining the boundary 309. In addition, in some embodiments, the movement of the second medical device 314 can be limited so that the second medical device 314 cannot go beyond the boundary. This can minimize the overall portion of the lung 302 that is resected and reduce the possibility that healthy tissue will be damaged. In addition, in some embodiments, the procedure allows for guiding the second medical device 314 without fluoroscopic visualization. Previously, such resection procedures involved placing physical radiopaque markers (e.g., metal markers) in the lungs, and then observing the markers through fluoroscopic fluoroscopy during the procedure to visualize the boundaries. Alternatively, the lungs can be physically marked with dyes, which can be visualized with bronchoscopes to guide resection. In some embodiments, the virtual benchmarks 308 and boundaries 309 described in the present patent application can be observed with reference to preoperative models that have been registered with anatomical structures, and can therefore be observed without fluoroscopic fluoroscopy. These and other advantages will be described in more detail below with more specific reference to the accompanying drawings.

[0179] Fig.25A An example of placing or creating a virtual reference 308 during a procedure is shown. In the illustrated example, a physician may wish to remove a portion of a lung 302. To guide the removal, the physician may place a virtual reference 308, which may be used to define a boundary around the portion of the lung 302 to be removed. The removal may then be performed percutaneously based on the boundary. In the illustrated example, to accurately place the virtual reference 308, first, a first medical instrument 304 is guided into the lung 302. In some embodiments, the first medical instrument 304 is inserted into the patient through a natural orifice. For example, the first medical instrument 304 may be inserted into the lung 302 through the patient's mouth and trachea. In some embodiments or in other procedures, other natural patient orifices may be used. In some embodiments, the first medical instrument 304 may be inserted percutaneously.

[0180] The first medical device 304 can be an endoscope, such as a bronchoscope. In some embodiments, the first medical device 304 can be robotically controlled. For example, the first medical device 304 can be the above reference Figures 1 to 20 In some embodiments, the first medical device 304 may be manually controlled. Fig.25AAs shown, the first medical device 304 may include an elongated shaft. The elongated shaft may be articulated and controllable as previously described, so that the first medical device 304 can be navigated through the patient's airway into the lung 302. The first medical device 304 may also include a position sensor 306 as described in the previous example. The position sensor 306 is configured to provide an output signal, from which the position of the position sensor 306 can be determined. Guidance of the first medical device 304 may be facilitated by a navigation or positioning system, such as the positioning system 90 described above.

[0181] like Fig.25A As shown, with the first medical device 304 positioned within the lung 302, the physician (or system) can determine the position of one or more virtual fiducials 308. The virtual fiducials 308 can be used to mark the boundaries of the resection volume, similar to the way physical fiducials or dyes were previously used. However, the virtual fiducials 308 can be virtually placed, for example, with reference to a preoperative model that has been registered with the anatomical structure. For example, in some embodiments, the output of the position sensor 306 can be registered with the preoperative model of the lung 302. Registration can be completed as described above, for example, by using the position sensor 306 to map the path of the first medical device 304 through the anatomical structure, and matching the mapped path with the preoperative model, or navigating the medical device 304 to one or more anatomical landmarks, and using the output of the position sensor 306 at one or more anatomical landmarks to register the preoperative model with the output of the position sensor 306.

[0182] In some embodiments, the virtual fiducial 308 is placed by navigating the first medical instrument 304 to the position in the lung 302 where the virtual fiducial 308 is desired to be placed, determining the position using the position sensor 306, and virtually placing the virtual fiducial at the determined position in the preoperative model. The physician may then navigate the first medical instrument 304 to the next position where the virtual fiducial 308 is desired to be placed, and repeat the process until all desired virtual fiducials 308 are placed.

[0183] In some embodiments, with the preoperative model registered with the patient's anatomy and the output of the position sensor 306, the position for placing the virtual fiducial 308 may be selected with reference to the preoperative model. For example, in some embodiments, the position of the virtual fiducial 308 may be determined with reference to the position sensor 306, but at a position away from the position sensor 306. That is, in some embodiments, the position of the virtual fiducial 308 need not coincide with the position sensor 306 (or any other part of the medical device 314). For example, the preoperative model may be displayed to the physician, and the physician may select the position of the virtual fiducial 308 on the displayed preoperative model.

[0184] In some embodiments, the position of the virtual reference 308 may be determined before surgery with reference to the preoperative model. The first medical instrument 304 may then be navigated into the lung 302 to register the preoperative model with the anatomical structure. In some embodiments, the first medical instrument 304 may be used to verify the placement of the preoperatively selected position of the virtual reference 308 by, for example, navigating to a position corresponding to the position of the virtual reference 308. In some embodiments, the physician may then adjust the placement position of the virtual reference 308 intraoperatively if necessary.

[0185] As another example, the position of virtual fiducial 308 may be determined with reference to intraoperative medical imaging (such as a fluoroscopic image).For example, an operator may select the position of virtual fiducial 308 on the fluoroscopic image.

[0186] The position of the virtual reference 308 can be, for example, Fig.24 The graphical user interface shown is displayed to the user.

[0187] like Fig.25B As shown, the orientation of virtual reference 308 can be used to define boundary 309. Boundary 309 can define the resection volume. In some embodiments, the system is configured to fit a line or surface passing through virtual reference 308 to define boundary 309. In some embodiments, boundary 309 can be, for example, Fig.24 308 is displayed to the user on the graphical user interface shown. In some embodiments, the border 309 is omitted and the virtual reference 308 is used as the border.

[0188] Boundary 309 (or virtual reference 308 itself) can be used to guide one or more second medical instruments 314 during resection. Resection can be performed percutaneously, but this is not required in all embodiments. In the illustrated embodiment, two second medical instruments 314 are shown. The second medical instrument can be, for example, a laparoscopic medical instrument, such as the one described above with reference to Figures 1 to 20 In some embodiments, the second medical device 314 can be positioned on an instrument positioning device such as a robotic arm. The instrument positioning device can be robotically controlled. In some embodiments, the robotic control can limit or prevent the second medical device 314 from breaking or crossing the boundary 309. This can limit or prevent the removal of unintended tissue.

[0189] In some embodiments, the second medical device 314 may be manually controlled. The system may provide an indication of when the second medical device 314 is approaching the boundary 309 so that the physician is alerted. The indication may be a visual, audible, or tactile signal.

[0190] In some embodiments, the first medical device 304 remains in the treatment area (e.g., in the lung 302) during ablation with the second medical device 314. As described above, the output signal of the position sensor 306 can be used to track patient motion, such as breathing, during the procedure. This motion can be tracked using the position sensor 306. This patient motion can then be compensated for during ablation.

[0191] Fig.26 An exemplary embodiment of an endoscope-assisted percutaneous medical procedure in the gastrointestinal tract is shown. In the illustrated example, a physician wishes to percutaneously obtain a biopsy of the pancreas. However, it is often difficult to ensure that a percutaneously inserted medical instrument intersects the pancreas. In this example, the percutaneously inserted instrument can be aligned with a target position determined in part based on an orientation sensor of the endoscopically inserted instrument.

[0192] In the illustrated example, the first medical instrument 504 is guided through the patient's mouth, esophagus 532, stomach 534, and duodenum 536 to the mastoid 538. The mastoid 538 can be visually identified, for example, using an optical system on the first medical instrument 504. The first medical instrument 504 includes the above-mentioned position sensor 506. A target position 508 in the pancreas can be determined relative to the position determined by the position sensor 506. This is done because the physician knows that the pancreas is located opposite the mastoid 538. Therefore, the target position 508 can be determined at a position away from the position sensor 506. The target position 508 can then be used as a beacon for aligning and guiding the percutaneous insertion of the second medical instrument 514 as described above. For example, the axis 518 of the second medical instrument can be aligned with the target position 508, and then the second medical instrument can be percutaneously inserted through the opening 520 so as to intersect with the target position 508 and obtain a biopsy. As previously described, the alignment of the second medical instrument 514 can be maintained robotically.

[0193] Fig.27A 6 is a flow chart illustrating an embodiment of a method 600 for performing a medical procedure that includes bringing a medical device into contact with a target location. The method 600 begins at block 602 where a first medical device is inserted into a treatment area. The treatment area may include, for example, a kidney, a bladder, a lung, a stomach, a gastrointestinal tract, etc. The first medical device may be an endoscope. In some embodiments, the first medical device is inserted into the treatment area through a natural patient orifice. The first medical device may be a laparoscope. In some embodiments, the first medical device is inserted into the treatment area percutaneously or through a percutaneous opening. The first medical device may be robotically controlled, for example using the method described above with reference to Figures 1 to 20 The robot-enabled medical system. The first medical device may be manually controlled.

[0194] Next, method 600 moves to block 604, where a first position sensor on a first medical device is used to determine a first position of the first medical device. The first medical device may include a first position sensor. The first position sensor may be an EM sensor, a shape sensing fiber, or any other type of sensor for determining position. In some embodiments, the output of the first position sensor may be registered with the preoperative model so that the position of the first position sensor is determined with reference to the preoperative model. The preoperative model may be developed, for example, based on a CT scan or other method as described above.

[0195] The method 600 then moves to block 606, where a target position is determined within the treatment area away from the first orientation. The target position may represent a meeting point for a second medical device. The target position may be displayed to a user. In some embodiments, defining the target position within the treatment area includes determining the target position with reference to a preoperative model. For example, determining the target position with reference to a preoperative model may be accomplished by displaying the preoperative model to a user and receiving a selection of the target position with reference to the preoperative model.

[0196] In some embodiments, the distance and direction between the first position as determined by the first position sensor and the target position may be determined. The distance and direction may be calculated, for example, based on a registered pre-operative model.

[0197] In some embodiments, defining the target location within the treatment area may include capturing one or more intraoperative medical images of the treatment area, and defining the target location with reference to the one or more intraoperative medical images. The intraoperative medical images may be one or more fluoroscopic images. The one or more medical images may be registered with the output of the position sensor.

[0198] Finally, the method moves to block 608 where the second medical device is directed to the target location. The second medical device may be a laparoscope. In some embodiments, the second medical device is inserted into the treatment area percutaneously or through a percutaneous opening. The second medical device may be an endoscope. In some embodiments, the second medical device is inserted into the treatment area through a natural patient orifice. The second medical device may be robotically controlled, for example using the method described above with reference to Figures 1 to 20 The robotic-enabled medical system described herein. The second medical device may be manually controlled. In some embodiments, initial access is obtained by percutaneously inserting a first medical device, which may include a built-in position sensor. The first medical device may be a needle, which includes a thin-walled cannula that is left behind to create a small access channel into the patient's body. Then, a second medical device (such as a guide wire) can be inserted through the cannula and into the patient's body, at which time the cannula that has been passed through cannula can be removed while ensuring that the wire remains in place. The wire can then be used as a guide to deliver the dilation tool and ultimately deliver a larger port.

[0199] In some embodiments, guiding the second medical device toward the target location may include aligning a second axis of the second medical device with the target location, and advancing the second medical device toward the target location. In some embodiments, the second medical device is attached to a robotic arm or other device positioning device. The robotic arm may constrain the movement of the second medical device to movement along or around the second axis to maintain alignment with the target location. The robotic arm may provide a tactile boundary that maintains the alignment of the second medical device.

[0200] In some embodiments, method 600 also includes determining patient motion using a first position sensor of a first medical device. For example, during a procedure, the first medical device may remain in a treatment area and the first position sensor may monitor patient motion. In some embodiments, guidance of a second device may compensate for the measured patient motion. Patient motion that may be compensated may include, for example, motion due to breathing.

[0201] Fig.27B 610 is a flow chart illustrating an embodiment of a method 610 for performing a medical procedure that includes placing a virtual reference to define a boundary. The boundary may be, for example, a resection boundary. The method 610 begins at box 612, where a first medical device is inserted into a treatment area. The treatment area may include, for example, a kidney, a bladder, a lung, a stomach, a gastrointestinal tract, etc. The first medical device may be an endoscope. In some embodiments, the first medical device is inserted into the treatment area through a natural patient orifice. The first medical device may be a laparoscope. In some embodiments, the first medical device is inserted into the treatment area percutaneously or through a percutaneous opening. The first medical device may be robotically controlled, for example using the above reference Figures 1 to 20 The robot-enabled medical system. The first medical device may be manually controlled.

[0202] Next, method 610 moves to block 614, where the first position sensor is registered with the preoperative model. The first medical device may include a first position sensor. The first position sensor may be an EM sensor, a shape sensing fiber, or any other type of sensor for determining position. In some embodiments, the output of the first position sensor may be registered with the preoperative model so that the position of the first position sensor is determined with reference to the preoperative model. The preoperative model may be developed, for example, based on a CT scan or other method as described above.

[0203] Method 610 then moves to box 616, where the orientation of one or more virtual references is defined to create a boundary with reference to the preoperative model. In some embodiments, determining the orientation of one or more virtual references includes navigating the first medical device to the location in the treatment area where the virtual reference is to be placed, and defining the location as the orientation of the virtual reference based on the aligned output of the first orientation sensor. In some embodiments, determining the orientation of one or more virtual references includes receiving a user selection of a location where one of the one or more virtual references is to be placed, and determining a virtual reference orientation corresponding to the location with reference to at least a first orientation determined based on the first orientation sensor, wherein the virtual reference orientation is away from the first orientation. In some embodiments, a virtual curve or surface is fitted to the virtual reference to define the boundary. The boundary and / or virtual reference may be displayed to the physician.

[0204] Next, method 610 moves to block 618, where the first medical device is positioned away from the boundary. In some embodiments, in this orientation, the first orientation sensor of the first medical device can be utilized as described above to monitor patient motion during the procedure. In some embodiments, block 618 can be omitted, and the first medical device can remain at the boundary during the procedure. Whether the first device remains at the boundary can depend on, for example, the use and timing of the boundary. For example, in Fig.25B In the example, the boundary represents the area to be removed. In this case, when the surgeon removes the distal part of the lung, the first instrument can be positioned within the boundary to provide better visualization. However, when the surgeon needs to remove the area where the first instrument is located within the boundary, the first instrument should be pulled away to give room for operation.

[0205] Finally, method 600 moves to block 620 where a second medical device is guided within the treatment region based on one or more virtual fiducials. The second medical device may be a laparoscope. In some embodiments, the second medical device is inserted into the treatment region percutaneously or through a percutaneous opening. The second medical device may be an endoscope. In some embodiments, the second medical device is inserted into the treatment region through a natural patient orifice. The second medical device may be robotically controlled, for example using the method described above with reference to Figures 1 to 20 The robotic-enabled medical system. The second medical instrument may be manually controlled. In some embodiments, the movement of the second medical instrument is limited or restricted so that the second medical instrument cannot exceed the boundary. In some embodiments, when the second medical instrument is attached to a robotic arm or other instrument positioning device, a tactile boundary is used to create a limit or restriction on the movement of the second medical instrument.

[0206] In some embodiments, methods 600, 610 can be performed, for example, using the method described above with reference to Figures 1 to 20 The robot-enabled medical system is used to perform the above.

[0207] The above-described concomitant endoscopic and percutaneous (eg, laparoscopic) systems and methods may provide a number of advantages, including providing improved placement accuracy for endoscopic or laparoscopic tools, and providing clearly defined or otherwise improved resection margins.

[0208] The above-described concomitant endoscopic and percutaneous (e.g., laparoscopic) systems and methods may also be used during combined endoscopic and laparoscopic surgery (CELS), which may be performed robotically using the above-described systems. An example of a procedure that may be advantageously performed using CELS is a colon polypectomy, but other examples exist. Polyps may be assessed endoscopically based on their size, type, and location to determine whether they may be removed. When polyps cannot be removed endoscopically, they may be removed via segmental colectomy, which is associated with a relatively high complication rate and increased recovery time. During multi-segment resections, CELS may enable extraluminal mobilization of the colon (using laparoscopic instruments) to enable easier intraluminal resection of the polyps (using endoscopic instruments).

[0209] When performed manually, CELS typically requires at least two physicians (one to control the laparoscopic instrument and one to control the endoscopic instrument) and two assistants (to hold the laparoscope and the colonoscope, respectively). While one physician is moving the instrument, the remaining providers may keep their instruments stationary, which may be physically demanding over an extended period of time. There may be additional staff in the room to help with instrument changes, handing sutures or gauze, handling specimens after removal, and controlling laparoscopic instruments, etc. In addition, communication between the two operating physicians may be slow and difficult. For example, it may be difficult for one physician to communicate the location of the instrument he or she is controlling to another physician, or vice versa. The above-described systems and methods can reduce or eliminate these difficulties that may occur when CELS is performed manually by allowing a unit physician to control two instruments and / or providing improved convergence between the two instruments.

[0210] In addition to the above examples of endoscopic diagnosis and surgical resection of cancerous tumors, other exemplary medical procedures can benefit from the systems and methods described herein, including bronchoscopic positioning and simultaneous thoracoscopic resection of lung cancer, endoscopic positioning and laparoscopic resection of gastrointestinal cancer, laparoscopically assisted endoscopic positioning and resection of gastrointestinal cancer, endoscopic imaging or visualization of gastrointestinal reconstruction procedures, such as gastrectomy, gastric bypass (roux-en-y-gastric bypass), etc., ureteroscopic stone / tumor positioning and percutaneous removal / resection. In some embodiments, such procedures can be performed in a single treatment stage. In some embodiments, such procedures can be performed by a minimum number of clinicians, and in some cases, can be performed by a single physician. In addition, in some embodiments, a single type of console can be used to control simultaneous procedures to perform simultaneous procedures.

[0211] In some cases, one instrument (e.g., an endoscopically inserted instrument) may be able to provide better visualization of a treatment site (e.g., a lesion), while another instrument (e.g., a laparoscopically inserted instrument) may be better suited to treat (e.g., biopsy or resect) the treatment site. Vice versa. In some cases, a laparoscopically inserted instrument may provide better visualization, while an endoscopically inserted instrument may provide better treatment. The above-described methods and systems may advantageously allow each instrument to be used in a more appropriate manner while delivering information that can be used to guide the other instrument. This may advantageously allow for greater precision and reduce the overall time of a procedure, thereby providing improved patient outcomes.

[0212] 3. Implement systems and terminology .

[0213] Embodiments disclosed herein provide systems, methods, and devices for endoscope-assisted percutaneous medical procedures.

[0214] It should be noted that as used herein, the terms "couple," "coupling," "coupled," or other variations of the word coupled may indicate an indirect connection or a direct connection. For example, if a first component is "coupled" to a second component, the first component may be indirectly connected to the second component via another component or directly connected to the second component.

[0215] Phrases referring to the processes and functions of a specific computer implementation described herein may be stored as one or more instructions on a processor-readable or computer-readable medium. The term "computer-readable medium" refers to any available medium that can be accessed by a computer or processor. By way of example and not limitation, such a medium may include a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a compact disk read-only memory (CD-ROM) or other optical disk storage device, a magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. It should be noted that a computer-readable medium may be tangible and non-transient. As used herein, the term "code" may refer to software, instructions, code or data that can be executed by a computing device or processor.

[0216] The method disclosed herein includes one or more steps or actions for implementing the method. The method steps and / or actions may be interchangeable with each other without departing from the scope of the claims. In other words, unless the correct operation of the method being described requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0217] As used herein, the term "plurality" means two or more. For example, a plurality of components indicates two or more components. The term "determine" encompasses a variety of actions, and thus, "determine" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, etc. Additionally, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Additionally, "determine" may include parsing, selecting, choosing, establishing, etc.

[0218] Unless explicitly stated otherwise, the phrase “based on” does not mean “based only on.” In other words, the phrase “based on” describes both “based only on” and “based at least on.”

[0219] The foregoing specific embodiments of the disclosed specific implementations are provided to enable any person skilled in the art to make or use the present invention. Various modifications to these specific implementations will be apparent to those skilled in the art, and the general principles defined herein may be applied to other specific implementations without departing from the scope of the present invention. For example, it should be understood that one of ordinary skill in the art will be able to adopt a plurality of corresponding alternatives and equivalent structural details, such as equivalent ways of fastening, mounting, coupling or engaging tool parts, equivalent mechanisms for producing specific actuating motions, and equivalent mechanisms for delivering electrical energy. Therefore, the present invention is not intended to be limited to the specific implementations shown herein, but is given the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A system comprising: a first instrument configured to be inserted into an anatomical structure, the first instrument comprising a position sensor; and A control system, the control system being configured to: receiving an output of the position sensor indicating a position of the first device relative to a first coordinate frame; determining a position of the first instrument relative to a second coordinate frame associated with a model of the anatomical structure; placing a first virtual fiducial on the model of the anatomical structure based at least in part on the orientation of the first instrument relative to the second coordinate frame; as well as Insertion of a second instrument into the anatomical structure is guided based on placement of the first virtual fiducial on the model of the anatomical structure.

2. The system according to claim 1, wherein: The first instrument is configured to be inserted through a natural orifice of the anatomical structure, and the second instrument is configured to be inserted percutaneously into the anatomical structure.

3. The system according to claim 1, wherein: Determining the position of the first device relative to the second coordinate frame includes: navigating the first instrument to an anatomical landmark within the anatomical structure; identifying the location of the anatomical landmarks on the model of the anatomical structure; and In response to navigating the first instrument to the anatomical landmark, matching the position of the first instrument in the first coordinate frame with the position of the anatomical landmark on the model of the anatomical structure.

4. The system according to claim 1, wherein: Determining the position of the first device relative to the second coordinate frame includes: navigating the first instrument along a path through the anatomical structure; mapping the path of the first instrument onto a model of the anatomical structure; and In response to navigating the first instrument along the path, a position of the first instrument in the first coordinate frame is matched to the path mapped onto the model of the anatomy.

5. The system according to claim 1, wherein: Placing the first virtual reference on the model of the anatomical structure comprises: In response to navigating the first instrument to a desired location within the anatomical structure, the first virtual fiducial is placed at the position of the first instrument relative to the second coordinate frame.

6. The system according to claim 1, wherein: The control system is also configured to: A second virtual fiducial is placed on the model of the anatomical structure based at least in part on the orientation of the first instrument relative to the second coordinate frame, the first virtual fiducial and the second virtual fiducial defining a boundary associated with the model of the anatomical structure.

7. The system according to claim 6, wherein: Guiding the insertion of the second instrument into the anatomical structure comprises: The movement of the second instrument is controlled so that the second instrument does not cross the boundary.

8. The system according to claim 6, wherein: Guiding the insertion of the second instrument into the anatomical structure further comprises: When the second instrument approaches the boundary, at least one of a visual signal, an auditory signal, or a tactile signal is generated.

9. The system according to claim 1, wherein: Guiding the insertion of the second instrument into the anatomical structure comprises: A graphical interface is displayed, the graphical interface including a model of the anatomical structure with the first virtual fiducial placed thereon.

10. A method performed by a system, the method comprising: receiving an output of a position sensor indicating a position of a first instrument relative to a first coordinate frame, the first instrument being configured to be inserted into an anatomical structure; determining a position of the first instrument relative to a second coordinate frame associated with a model of the anatomical structure; placing a first virtual fiducial on the model of the anatomical structure based at least in part on the orientation of the first instrument relative to the second coordinate frame; as well as Insertion of a second instrument into the anatomical structure is guided based on placement of the first virtual fiducial on the model of the anatomical structure.

Images