System, method, and workflow for concomitant treatment

Abstract

To provide a system, a method, and a workflow that can be used in a robot capable of performing various medical treatments including both low invasive treatment using an abdominoscope or the like and non-invasive treatment using an endoscope or the like.SOLUTION: A method for concomitant treatment includes: operating a flexible tool 13 by using a first robot arm 12 of a robot-controllable system 10 of a cart base; operating a rigid tool by using a second robot arm of the robot system; displaying a feedback from the flexible tool; and displaying a feedback from the rigid tool.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Application No. 62 / 831,064, filed April 8, 2019, which is incorporated by reference in its entirety.

[0002] FIELD OF THEINVENTION The systems and methods disclosed herein relate to systems and methods for performing medical procedures, and more particularly, ancillary procedures. [Background technology]

[0003] A variety of medical procedures can be performed using a robotic medical system to control the insertion and / or manipulation of one or more medical instruments. For certain medical conditions, more than one medical procedure may be performed to completely treat the medical condition. A robotically controllable medical system can include one or more robotic arms or any other instrument positioning device. The robotic medical system can also include a controller that is used to control the positioning of the instruments during each of the procedures via the manipulation of the robotic arms and / or instrument positioning devices. Summary of the Invention [Means for solving the problem]

[0004] The systems, methods, and devices of the present disclosure each have several innovative aspects, no single one of which is solely responsible for the desirable attributes disclosed herein.

[0005] In one aspect, a surgical method is provided that includes manipulating a flexible instrument using a first robotic arm of a robotic system, manipulating a rigid instrument using a second robotic arm of the robotic system, displaying feedback from the flexible instrument, and displaying feedback from the rigid instrument.

[0006] In another aspect, a surgical method is provided that includes manipulating a flexible instrument using a first robotic arm of a robotic system; manipulating a rigid instrument using a second robotic arm of the robotic system; displaying a first feedback from the flexible instrument on a viewscreen as a primary view; and displaying a second feedback from the rigid instrument on the same viewscreen as a secondary view.

[0007] In yet another aspect, a surgical method is provided, the surgical method including manipulating a flexible instrument using a first robotic arm of a robotic system through a natural orifice of a patient, manipulating a rigid instrument using a second robotic arm of the robotic system through an incision formed in the patient, and displaying feedback information from the flexible instrument and the rigid instrument.

[0008] In yet still another aspect, a surgical method is provided that includes introducing a flexible instrument into a patient via a natural orifice of the patient, manipulating the flexible instrument using a first robotic arm of a robotic system through the natural orifice, deploying a second robotic arm of the robotic system from a stowed position to a setup position in response to receiving an input signal via a user input device, manipulating a rigid instrument using the second robotic arm of the robotic system through an incision made in the patient, and displaying feedback from at least one of the flexible instrument and the rigid instrument.

[0009] In yet another aspect, a robotic system is provided that includes a flexible instrument, a rigid instrument, a first robotic arm configured to manipulate the flexible instrument, a second robotic arm configured to manipulate the rigid instrument, and a display configured to display field of view data from the flexible instrument and the rigid instrument.

[0010] In still yet another aspect, a non-transitory computer readable storage medium is provided having instructions stored thereon that, when executed, cause at least one computing device to manipulate a flexible instrument using a first robotic arm of a robotic system, manipulate a rigid instrument using a second robotic arm of the robotic system, display feedback from the flexible instrument, and display feedback from the rigid instrument. [Brief description of the drawings]

[0011] The disclosed aspects are hereinafter described in conjunction with the accompanying drawings, which illustrate, by way of example only, and not by way of limitation, the disclosed aspects, and in which like designations refer to like elements. [Figure 1] 1 illustrates an embodiment of a cart-based robotic system deployed for a diagnostic and / or therapeutic bronchoscopy procedure. [Diagram 2] 2 is a further embodiment of the robotic system of FIG. 1. [Diagram 3] 2 illustrates an embodiment of the robotic system of FIG. 1 positioned for ureteroscopy. [Figure 4] 2 illustrates an embodiment of the robotic system of FIG. 1 positioned for a vascular procedure. [Diagram 5] 1 illustrates an embodiment of a table-based robotic system positioned for a bronchoscopy procedure. [Figure 6] FIG. 6 is an alternative view of the robotic system of FIG. [Figure 7] 1 is an exemplary system configured to store a robotic arm. [Figure 8] 1 is an embodiment of a table-based robotic system configured for a ureteroscopy procedure. [Figure 9] 1 is an embodiment of a table-based robotic system configured for laparoscopic procedures. [Figure 10] 10 illustrates an embodiment of the table-based robotic system of FIGS. 5-9 with pitch or tilt adjustment. [Figure 11]FIG. 11 provides a detailed illustration of the interface between the table and column of the table-based robotic system of FIGS. [Figure 12] 1 illustrates an alternative embodiment of a table-based robotic system. [Figure 13] FIG. 13 illustrates an end view of the table-based robotic system of FIG. [Figure 14] FIG. 1 shows an end view of a table-based robotic system with a robotic arm attached. [Figure 15] 1 illustrates an exemplary instrument driver. [Figure 16] 1 illustrates an exemplary medical instrument with paired instrument drivers. [Figure 17] 13 shows an alternative design of the instrument driver and instrument, where the axis of the drive unit is parallel to the axis of the elongate shaft of the instrument. [Figure 18] 1 illustrates an instrument having an instrument-based insertion architecture. [Figure 19] 1 illustrates an exemplary controller. [Figure 20] FIG. 17 shows a block diagram illustrating a localization system for estimating the position of one or more elements of the robotic system of FIGS. 1-10, such as the position of the instrument of FIGS. 16-18, according to an exemplary embodiment. [Figure 21] 1 illustrates an embodiment of a bed-based robotic system configured to perform a concomitant procedure in accordance with aspects of the present disclosure. [Figure 22] 1 illustrates another embodiment of a bed-based robotic system configured to perform an associated procedure in accordance with aspects of the present disclosure. [Diagram 23] 1 illustrates yet another embodiment of a robotic system configured to perform a concomitant procedure in accordance with aspects of the present disclosure. [Figure 24] 13A-13C illustrate two configurations of another embodiment of a bed-based robotic system configured to perform an associated procedure in accordance with aspects of the present disclosure. [Diagram 25] 13A-13C illustrate two configurations of another embodiment of a bed-based robotic system configured to perform an associated procedure in accordance with aspects of the present disclosure. [Figure 26] 1 is a flow chart illustrating an exemplary method operable by a robotic system or components thereof for performing an associated medical procedure in accordance with aspects of the present disclosure. [Figure 27A] A flow chart illustrating another exemplary method operable by a robotic system or components thereof for performing endoscopic and thoracic procedures, along with associated therewith, in accordance with aspects of the present disclosure, is provided. [Figure 27B] A flow chart illustrating another exemplary method operable by a robotic system or components thereof for performing endoscopic and thoracic procedures, along with associated therewith, in accordance with aspects of the present disclosure, is provided. [Figure 28] 1 is a flow chart illustrating an exemplary method operable by a robotic system or components thereof for performing an associated medical procedure, including treatment escalation, in accordance with an aspect of the present disclosure. [Figure 29] 1 illustrates an exemplary console including one or more types of interfaces for controlling a robotic arm, according to an embodiment of the present disclosure. [Diagram 30] 30 illustrates an expanded view of the controller shown in FIG. 29 according to an embodiment of the present disclosure. [Diagram 31] FIG. 31 illustrates an enlarged view of one of the handles shown in FIGS. 29 and 30 according to an embodiment of the present disclosure. [Diagram 32] FIG. 30 illustrates an enlarged view of the pendant shown in FIG. 29 according to an embodiment of the present disclosure. [Diagram 33] 1 is a flowchart illustrating an exemplary method operable by a robotic system or components thereof for performing a concomitant medical procedure via a single user interface in accordance with an aspect of the present disclosure. [Diagram 34] 1A-1C are exemplary diagrams that may be displayed by a viewer during an accompanying medical procedure in accordance with aspects of the present disclosure. [Diagram 35] 1A-1C are exemplary diagrams that may be displayed by a viewer during an accompanying medical procedure in accordance with aspects of the present disclosure. [Diagram 36]FIG. 11 is another exemplary diagram that may be displayed by a viewer during an accompanying medical procedure in accordance with an aspect of the present disclosure. [Figure 37] 1 is a flow chart illustrating an exemplary method operable by a robotic system or components thereof for switching between displayed images while performing an associated medical procedure in accordance with aspects of the present disclosure. [Figure 38A] 1 is a flowchart illustrating an exemplary workflow for performing combined endoscopic and laparoscopic surgery (CELS) in accordance with an aspect of the present disclosure. [Figure 38B] 1 is a flowchart illustrating an exemplary workflow for performing combined endoscopic and laparoscopic surgery (CELS) in accordance with an aspect of the present disclosure. [Figure 39] 1 illustrates one embodiment of a bed-based robotic system configured to perform an associated procedure in accordance with aspects of the present disclosure. [Diagram 40] 1 is a flowchart illustrating another exemplary workflow for performing CELS, according to an embodiment of the present disclosure. [Diagram 41] 1 illustrates another embodiment of a robotic system configured to perform an associated procedure in accordance with aspects of the present disclosure. [Diagram 42] 1 is an exemplary still image captured from a video screen of an accompanying system during a colorectal intervention, according to an aspect of the present disclosure. [Diagram 43] 1 is a flow chart illustrating an exemplary patient preparation procedure for performing CELS, according to an embodiment of the present disclosure. [Diagram 44] 1 is a flow chart illustrating an exemplary insufflation procedure for performing CELS, according to an embodiment of the present disclosure. [Diagram 45] 1 is a flowchart illustrating an exemplary imaging procedure for performing CELS, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] 1.Overview Aspects of the present disclosure may be integrated into a robotic-enabled medical system capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy, and non-invasive procedures such as endoscopic procedures. Among endoscopic procedures, the system may perform bronchoscopy, ureteroscopy, gastroscopy, etc.

[0013] In addition to performing a wide range of procedures, the system can provide additional benefits such as enhanced imaging and guidance to assist the physician. Additionally, the system can provide the physician with the ability to perform procedures from an ergonomic position without requiring awkward arm movements and postures. Still further, the system can provide the physician with the ability to perform procedures with improved ease of use, such that one or more of the instruments of the system can be controlled by a single user.

[0014] Various embodiments are described below in conjunction with the drawings for purposes of explanation. It should be understood that many other implementations of the disclosed concepts are possible, and that various advantages may be achieved in the disclosed implementations. Headings are included herein for reference and to aid in locating the various sections. These headings are not intended to limit the scope of the concepts described therein. Such concepts may have applicability throughout this specification.

[0015] A. Robot System - Cart Robot-enabled medical systems can be configured in a variety of ways depending on the particular procedure. FIG. 1 illustrates an embodiment of a cart-based robot-controllable system 10 arranged for a diagnostic and / or therapeutic bronchoscopy procedure. During a bronchoscopy procedure, the system 10 can include a cart 11 having one or more robotic arms 12 for delivering a medical instrument, such as a steerable endoscope 13, which can be a procedure-specific bronchoscope for bronchoscopy, to a natural orifice access point (i.e., the patient's mouth, which in this example is positioned on a table) for delivering the diagnostic and / or therapeutic tools. As shown, the cart 11 can be positioned adjacent to the patient's upper torso to provide access to the access point. Similarly, the robotic arms 12 can be actuated to position the bronchoscope relative to the access point. The arrangement of FIG. 1 can also be utilized when performing a gastrointestinal (GI) procedure with a gastroscope (a specialized endoscope for GI procedures). FIG. 2 depicts an exemplary embodiment of the cart in more detail.

[0016] With continued reference to FIG. 1, once the cart 11 is properly positioned, the robotic arm 12 can insert the steerable endoscope 13 into the patient robotically, manually, or a combination thereof. As shown, the steerable endoscope 13 may include at least two nested parts, such as an inner leader portion and an outer sheath portion, each portion coupled to a separate instrument driver (also referred to as an instrument drive mechanism (IDM)) from a set of instrument drivers 28, with each instrument driver coupled to the distal end of an individual robotic arm. This linear arrangement of the instrument drivers 28, which facilitates coaxial alignment of the leader portion with the sheath portion, creates a "virtual rail" 29 that can be repositioned in space by manipulating one or more robotic arms 12 to different angles and / or positions. The virtual rails described herein are shown in the figures using dashed lines, and thus the dashed lines do not indicate the physical structure of the system. Translation of the instrument driver 28 along the virtual rail 29 nests the inner leader portion relative to the outer sheath portion or advances or retracts the endoscope 13 from the patient. The angle of the virtual rail 29 may be adjusted, translated, and pivoted based on clinical application or physician preference. For example, in bronchoscopy, the angle and position of the virtual rail 29 as shown represents a compromise that provides the physician access to the endoscope 13 while minimizing friction from bending the endoscope 13 into the patient's mouth.

[0017] The endoscope 13 may be directed down the patient's trachea and lungs after insertion using precise commands from the robotic system until the target destination or surgical site is reached. To enhance navigation through the patient's pulmonary network and / or to reach the desired target, the endoscope 13 may be steered to telescope the inner leader portion out from the outer sheath portion for enhanced articulation and a larger bend radius. The use of a separate instrument driver 28 also allows the leader portion and sheath portion to be driven independently of one another.

[0018] For example, the endoscope 13 may be directed to deliver a biopsy needle to a target, such as a lesion or nodule in the patient's lung. The needle may be deployed down a working channel that spans the length of the endoscope to obtain a tissue sample that is analyzed by a pathologist. Depending on the results of the pathology, additional tools may be deployed down the working channel of the endoscope for additional biopsies. After identifying the nodule as malignant, the endoscope 13 may endoscopically deliver an instrument to resect the potential cancerous tissue. In some cases, diagnostic and therapeutic procedures may be provided in separate procedures. In these situations, the endoscope 13 may also be used to deliver fiducials to "mark" the location of the target nodule. In other examples, diagnostic and therapeutic procedures may be delivered during the same procedure.

[0019] The system 10 may also include a movable tower 30 that may be connected to the cart 11 via a support cable to provide support for control, electronics, fluidics, optics, sensors, and / or power to the cart 11. Locating such functionality within the tower 30 allows for a smaller form factor cart 11 that may be more easily adjusted and / or repositioned by the operating physician and his / her staff. Additionally, the division of functionality between the cart / table and the support tower 30 reduces clutter in the operating room and promotes improved clinical workflow. The cart 11 may be located in close proximity to the patient, while the tower 30 may be stored in a remote location to keep it out of the way during the procedure.

[0020] In support of the robotic system described above, the tower 30 may include computer-based control system components that store computer program instructions in a non-transitory computer-readable storage medium, such as, for example, a persistent magnetic storage drive, a solid-state drive, or the like. Execution of these instructions, whether execution occurs in the tower 30 or the cart 11, may control the entire system or subsystems thereof. For example, when executed by a processor in the computer system, the instructions may cause the robotic system components to actuate associated carriages and arm mounts, operate a robotic arm, and control a medical instrument. For example, in response to receiving control signals, motors in the joints of the robotic arm may position the arm in a particular pose.

[0021] The tower 30 may also include pumps, flow meters, valve controls, and / or fluid access to provide controlled irrigation and aspiration capabilities to a system that may be deployed through the endoscope 13. These components may also be controlled using the computer system of the tower 30. In some embodiments, irrigation and aspiration capabilities may be delivered directly to the endoscope 13 through separate cables.

[0022] The tower 30 may include voltage and surge protection designed to provide filtered and protected power to the cart 11, thereby avoiding the need to place power transformers and other auxiliary power components within the cart 11 and making the cart 11 smaller and more mobile.

[0023] The tower 30 may also include support equipment for sensors deployed throughout the robotic system 10. For example, the tower 30 may include optoelectronic equipment for detecting, receiving, and processing data received from optical sensors or cameras throughout the robotic system 10. In combination with a control system, such optoelectronic equipment may be used to generate real-time images for display in any number of consoles located throughout the system, including in the tower 30. Similarly, the tower 30 may also include electronic subsystems for receiving signals from deployed electromagnetic (EM) sensors and processing the received signals. The tower 30 may also be used to house and position EM field generators for detection by EM sensors in or on the medical instrument.

[0024] The tower 30 may also include a console 31 in addition to other consoles available to the rest of the system, for example a console mounted on top of a cart. The console 31 may include a user interface for the operator physician and a display screen, such as a touch screen. The consoles in the system 10 are generally designed to provide both pre-operative and real-time information of the procedure, such as robotic control and navigation and localization information for the endoscope 13. If the console 31 is not the only console available to the physician, the console 31 may be used by a second operator, such as a nurse, to monitor the health or vital signs of the patient and the operation of the system, as well as provide procedure-specific data, such as navigation and localization information. In other embodiments, the console 30 is housed in a body separate from the tower 30.

[0025] The tower 30 may be coupled to the cart 11 and endoscope 13 via one or more cables or connections (not shown). In some embodiments, support functions from the tower 30 may be provided to the cart 11 through a single cable, simplifying and decluttering the operating room. In other embodiments, certain functions may be coupled with separate wiring and connections. For example, power may be provided to the cart through a single power cable, while support for control, optics, fluidics, and / or navigation may be provided through separate cables.

[0026] FIG. 2 provides a detailed view of an embodiment of a cart from the cart-based robotic controllable system shown in FIG. The cart 11 generally includes an elongated support structure 14 (often referred to as a "column"), a cart base 15, and a console 16 at the top of the column 14. The column 14 may include one or more carriages, such as a carriage 17 (alternatively an "arm support") for supporting the deployment of one or more robotic arms 12 (three are shown in FIG. 2). The carriage 17 may include an individually configurable arm mount that rotates along a vertical axis to adjust the base of the robotic arm 12 for better positioning relative to the patient. The carriage 17 also includes a carriage interface 19 that allows the carriage 17 to translate vertically along the column 14.

[0027] The carriage interface 19 is connected to the column 14 through slots, such as slots 20 positioned on either side of the column 14 to guide the vertical translation of the carriage 17. The slots 20 include vertical translation interfaces for positioning and holding the carriage at various vertical heights relative to the cart base 15. The vertical translation of the carriage 17 allows the cart 11 to adjust the reach of the robotic arm 12 to meet various table heights, patient sizes, and physician preferences. Similarly, an individually configurable arm mount on the carriage 17 allows the robotic arm base 21 of the robotic arm 12 to be angled in various configurations.

[0028] In some embodiments, a slot cover may be added to the slot 20 that is flush and parallel with the slot surface to prevent dirt and fluids from entering the interior chamber of the column 14 and the vertical translation interface as the carriage 17 translates vertically. The slot cover may be deployed through a pair of spring spools positioned near the vertical top and bottom of the slot 20. The cover is coiled within the spools until it is deployed to expand and contract as the carriage 17 translates vertically up and down. The spring loading of the spools provides the force to retract the cover into the spools as the carriage 17 translates towards the spools while also maintaining a seal as the carriage 17 translates away from the spools. The cover may be connected to the carriage 17 using, for example, a bracket within the carriage interface 19 to facilitate proper extension and retraction of the cover as the carriage 17 translates.

[0029] Column 14 may include mechanisms therein, such as gears and motors, designed to use a vertically aligned leadscrew to mechanically translate carriage 17 in response to control signals generated in response to user input, such as input from console 16.

[0030] The robotic arm 12 may generally include a robotic arm base 21 and an end effector 22 separated by a series of links 23 connected by a series of joints 24, each joint including an independent actuator, each actuator including an independently controllable motor. Each independently controllable joint represents an independent degree of freedom available to the robotic arm. Each of the arms 12 has seven joints, thus providing seven degrees of freedom. A large number of joints results in a large number of degrees of freedom, allowing for "redundant" degrees of freedom. The redundant degrees of freedom allow the robotic arms 12 to position their respective end effectors 22 at specific positions, orientations, and trajectories in space using different articulation positions and joint angles. This allows the system to position and orient the medical instrument from a desired point in space, while at the same time allowing the physician to move the arm joints to a clinically advantageous position away from the patient to create better access while avoiding arm collisions.

[0031] The cart base 15 balances the weight of the column 14, carriage 17, and arm 12 on the floor. Thus, the cart base 15 houses the heavier parts such as electronics, motors, power supplies, and components that allow the cart to be either mobile and / or immobilized. For example, the cart base 15 includes casters 25 in the form of rollable wheels that allow the cart to be easily moved around the room before a procedure. After reaching the proper position, the casters 25 may be immobilized using wheel locks to hold the cart 11 in place during a procedure.

[0032] The console 16 positioned at the vertical end of the column 14 allows for both a user interface and a display screen (or dual-purpose device, e.g., touch screen 26) for receiving user input to provide both pre-operative and intra-operative data to the physician user. Potential pre-operative data on the touch screen 26 may include pre-operative planning, navigation and mapping data derived from a pre-operative computed tomography (CT) scan, and / or notes from a pre-operative patient interview. Intra-operative data on the display may include optical information provided by tools, sensor and coordinate information from sensors, and vital patient statistics such as respiration, heart rate, and / or pulse. The console 16 may be positioned and angled to allow a physician to access the console from the side of the column 14 opposite the carriage 17. From this position, the physician can view the console 16, the robotic arm 12, and the patient while operating the console 16 from behind the cart 11. As shown, the console 16 also includes a handle 27 to aid in maneuvering and stabilizing the cart 11.

[0033] FIG. 3 illustrates an embodiment of the robotically controllable system 10 arranged for ureteroscopy. In a ureteroscopy procedure, the cart 11 may be positioned to deliver a ureteroscope 32, a procedure-specific endoscope designed to traverse the patient's urethra and ureters, to the patient's lower abdominal region. In a ureteroscopy, it may be desirable for the ureteroscope 32 to be directly aligned with the patient's urethra to reduce friction and forces on sensitive anatomical structures in the region. As shown, the cart 11 may be aligned to the table leg to allow the robotic arm 12 to position the ureteroscope 32 for direct linear access to the patient's urethra. From the table leg, the robotic arm 12 can insert the ureteroscope 32 along a virtual rail 33 directly through the urethra and into the patient's lower abdominal region.

[0034] After insertion into the urethra using control techniques similar to those in bronchoscopy, the ureteroscope 32 may be navigated to the bladder, ureters, and / or kidneys for diagnostic and / or therapeutic applications. For example, the ureteroscope 32 may be directed to the ureters and kidneys to fragment kidney stone buildup using a laser or ultrasonic lithotripsy device deployed down the working channel of the ureteroscope 32. After lithotripsy is completed, the resulting stone fragments may be removed using a basket deployed down the ureteroscope 32.

[0035] FIG. 4 shows an embodiment of a similarly arranged robotic controllable system for a vascular procedure. In a vascular procedure, the system 10 may be configured such that the cart 11 can deliver a medical instrument 34, such as a steerable catheter, to an access point in the femoral artery in the patient's leg. The femoral artery presents both a larger diameter for navigation and a less circuitous and tortuous path to the patient's heart, which simplifies navigation. As in a ureteroscopy procedure, the cart 11 may be positioned toward the patient's leg and lower abdomen, allowing the robotic arm 12 to provide a virtual rail 35 with direct linear access to the femoral artery access point in the patient's thigh / hip region. After insertion into the artery, the medical instrument 34 may be oriented and inserted by translating the instrument driver 28. Alternatively, the cart may be positioned around the patient's upper abdomen to reach alternative vascular access points, such as the carotid and brachial arteries near the shoulder and wrist.

[0036] B. Robot System - Table An embodiment of a robot-enabled medical system may also incorporate a patient table. The incorporation of a table reduces the amount of capital equipment in the operating room by eliminating carts and allows greater access to the patient. FIG. 5 shows one embodiment of such a robot-controllable system arranged for a bronchoscopy procedure. The system 36 includes a support structure or column 37 for supporting a platform 38 (shown as a "table" or "bed") above the floor. As with cart-based systems, the end effector of the robotic arm 39 of the system 36 includes an instrument driver 42 designed to manipulate an elongated medical instrument, such as the bronchoscope 40 of FIG. 5, through or along a virtual rail 41 formed from a linear alignment of the instrument driver 42. In practice, a C-arm for providing fluoroscopic imaging may be positioned above the upper abdominal region of the patient by placing emitters and detectors around the table 38.

[0037] FIG. 6 provides an alternative view of the system 36 without the patient and medical instruments for discussion purposes. As shown, the column 37 may include one or more carriages 43, illustrated as a ring shape in the system 36, which may be the base for one or more robotic arms 39. The carriages 43 may translate along a vertical column interface 44 that spans the length of the column 37 to provide different vantage points where the robotic arms 39 may be positioned to reach the patient. The carriages 43 may rotate around the column 37 using mechanical motors positioned within the column 37 to allow the robotic arms 39 to have access to multiple sides of the table 38, such as both sides of the patient. In embodiments with multiple carriages, the carriages may be individually positioned on the column and may translate and / or rotate independently of the other carriages. The carriages 43 do not have to surround the column 37 or even be circular, but the ring shape as illustrated facilitates the carriages 43 to rotate around the column 37 while maintaining structural balance. Rotation and translation of carriage 43 allows the system to align medical instruments such as endoscopes and laparoscopes to different access points on the patient. In other embodiments (not shown), system 36 can include a patient table or bed with adjustable arm supports in the form of parallel extending bars or rails. One or more robotic arms 39 can be mounted on adjustable arm supports that can be adjusted vertically (e.g., via a shoulder with an elbow joint). By providing vertical adjustment, robotic arm 39 can advantageously be stored compactly under a patient table or bed and then raised during a procedure.

[0038] The arm 39 may be attached to the carriage through a set of arm mounts 45 that include a series of joints that may rotate and / or nest independently to provide additional configurability to the robotic arm 39. Additionally, the arm mounts 45 may be positioned on the carriage 43 such that, when the carriage 43 is appropriately rotated, the arm mounts 45 may be positioned either to one side of the table 38 (as shown in FIG. 6), to both sides of the table 38 (as shown in FIG. 9), or to adjacent sides of the table 38 (not shown).

[0039] The column 37 structurally provides support for the table 38 and a path for the vertical translation of the carriage. Internally, the column 37 may be equipped with a lead screw for guiding the vertical translation of the carriage, and a motor for mechanizing the translation of said carriage based on the lead screw. The column 37 may also transmit power and control signals to the carriage 43 and the robotic arm 39 mounted thereon.

[0040] The table base 46 serves a similar function as the cart base 15 of the cart 11 shown in FIG. 2, housing the heavier components to counterbalance the table / bed 38, column 37, carriage 43, and robotic arm 39. The table base 46 may also incorporate rigid casters to provide stability during a procedure. The casters, which deploy from the bottom of the table base 46, extend in opposite directions on either side of the base 46 and may be retracted when the system 36 needs to be moved.

[0041] Continuing with FIG. 6, the system 36 may also include a tower (not shown) that divides the functions of the system 36 between the table and the tower to reduce the form factor and bulk of the table. As with the previously disclosed embodiments, the tower may provide various support functions to the table, such as processing, computing, and control capabilities, power, fluidics, and / or optical and sensor processing. The tower may also be movable to be positioned away from the patient to improve physician access and reduce clutter in the operating room. Additionally, locating components in the tower allows for more storage space in the table base for potential storage of a robotic arm. The tower may also include a master controller or console that provides both a user interface for user input, such as a keyboard and / or pendant, and a display screen (or touch screen) for pre-operative and intra-operative information, such as real-time imaging, navigation, and tracking information. In some embodiments, the tower may also include a holder for a gas tank used for insufflation.

[0042] In some embodiments, the table base may house and store the robotic arm when not in use. FIG. 7 shows a system 47 for housing the robotic arm in one embodiment of a table-based system. In the system 47, the carriage 48 may be translated vertically into the base 49 to store the robotic arm 50, arm mount 51, and carriage 48 therein. The base cover 52 may be translated and retracted to open to position the carriage 48, arm mount 51, and arm 50 about the column 53, and closed to house and protect them when not in use. The base cover 52 may be sealed with a membrane 54 along the edge of its opening to prevent dirt and fluid ingress when closed.

[0043] FIG. 8 illustrates one embodiment of a robotically controllable table-based system configured for a ureteroscopy procedure. For ureteroscopy, the table 38 may include a swivel portion 55 for positioning the patient off-angle from the column 37 and table base 46. The swivel portion 55 may rotate or pivot about a pivot point (e.g., positioned below the patient's head) to position the bottom of the swivel portion 55 away from the column 37. For example, pivoting the swivel portion 55 allows a C-arm (not shown) to be positioned above the patient's lower abdomen without competing for space with the column (not shown) below the table 38. By rotating the carriage 35 (not shown) about the column 37, the robotic arm 39 may insert the ureteroscope 56 directly into the patient's groin area along the virtual rail 57 to reach the urethra. In ureteroscopy, stirrups 58 may be secured to a swivel portion 55 of table 38 to support the position of the patient's legs during the procedure and allow clear access to the patient's groin area.

[0044] In a laparoscopic procedure, minimally invasive instruments may be inserted into a patient's anatomy through small incisions in the patient's abdominal wall. In some embodiments, the minimally invasive instruments include an elongated rigid member, such as a shaft, that is used to access anatomy within the patient. After distension of the patient's abdominal cavity, the instruments may be oriented to perform a surgical or medical task, such as grasping, cutting, ablation, suturing, etc. In some embodiments, the instruments may include a scope, such as a laparoscope. FIG. 9 shows an embodiment of a robotically controlled table-based system configured for a laparoscopic procedure. As illustrated in FIG. 9, the carriage 43 of the system 36 may be rotated and vertically adjusted to position a pair of robotic arms 39 on either side of the table 38 such that the instruments 59 may be positioned using arm mounts 45 to pass through minimal incisions on either side of the patient to reach the patient's abdominal cavity.

[0045] To accommodate laparoscopic procedures, the robotic-enabled table system may also tilt the platform to a desired angle. FIG. 10 shows an embodiment of a robotically controllable medical system with pitch or tilt adjustment. As shown in FIG. 10, the system 36 can accommodate the tilt of the table 38 to position one side of the table at a greater distance from the floor than the other side. Additionally, the arm mount 45 may rotate to match the tilt so that the arm 39 maintains the same planar relationship with the table 38. To accommodate steeper angles, the column 37 may also include a nesting portion 60 that allows the column 37 to extend vertically to prevent the table 38 from contacting the floor or colliding with the base 46.

[0046] FIG. 11 provides a detailed illustration of the interface between the table 38 and the column 37. The pitch rotation mechanism 61 may be configured to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom. The pitch rotation mechanism 61 may be enabled by the positioning of orthogonal axes 1, 2 at the column-table interface, each actuated by a separate motor 3, 4 in response to an electrical pitch angle command. Rotation along one screw 5 allows tilt adjustment in one axis 1, and rotation along the other screw 6 allows tilt adjustment along the other axis 2. In some embodiments, a ball joint may be used to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom.

[0047] For example, the pitch adjustment is particularly useful when attempting to position the table in the Trendelenburg position, i.e., positioning the patient's lower abdomen higher off the floor than the patient's lower abdomen for lower abdominal surgery. The Trendelenburg position allows gravity to slide the patient's internal organs toward the patient's upper abdomen, emptying the abdominal cavity for entry of minimally invasive tools to perform lower abdominal surgical or medical procedures, such as laparoscopic prostatectomy.

[0048] 12 and 13 show isometric and end views of another embodiment of a table-based surgical robotic system 100. The surgical robotic system 100 includes one or more adjustable arm supports 105 (see, e.g., FIG. 14) that can be configured to support one or more robotic arms relative to the table 101. In the illustrated embodiment, a single adjustable arm support 105 is shown, but additional arm supports can be provided on the opposite side of the table 101. The adjustable arm support 105 can be configured to move it relative to the table 101 to adjust and / or change the position of the adjustable arm support 105 and / or any robotic arms attached thereto relative to the table 101. For example, the adjustable arm support 105 can be adjusted with one or more degrees of freedom relative to the table 101. The adjustable arm support 105 provides great versatility to the system 100, including the ability to easily accommodate one or more adjustable arm supports 105 and any robotic arms attached thereto under the table 101. The adjustable arm support 105 can be raised from a stowed position to a position below the top surface of the table 101. In other embodiments, the adjustable arm support 105 can be raised from a stowed position to a position above the top surface of the table 101.

[0049] The adjustable arm support 105 can provide several degrees of freedom, including lift (e.g., vertical translation), lateral translation, tilt, etc. In the illustrated embodiment of Figs. 12 and 13, the arm support 105 is configured with four degrees of freedom, indicated by the arrows in Fig. 12. The first degree of freedom allows for adjustment of the adjustable arm support 105 in the z-direction ("Z-lift"). For example, the adjustable arm support 105 can include a carriage 109 configured to move up and down along or relative to the column 102 that supports the table 101. The second degree of freedom allows the adjustable arm support 105 to tilt. For example, the adjustable arm support 105 can include a rotational joint, which can allow the adjustable arm support 105 to align with the bed in a Trendelenburg position. The third degree of freedom allows the adjustable arm support 105 to "pivot up", which can be used to adjust the distance between the side of the table 101 and the adjustable arm support 105. A fourth degree of freedom allows translation of the adjustable arm support 105 along the longitudinal length of the table.

[0050] The surgical robotic system 100 of Figures 12 and 13 can include a table supported by a column 102 attached to a base 103. The base 103 and the column 102 support the table 101 against a support surface. A bed axis 131 and a support axis 133 are shown in Figure 13.

[0051] The adjustable arm support 105 can be mounted to the column 102. In other embodiments, the arm support 105 can be mounted to the table 101 or the base 103. The adjustable arm support 105 can include a carriage 109, a bar or rail connector 111, and a bar or rail 107. In some embodiments, one or more robotic arms mounted to the rail 107 can translate and move relative to each other.

[0052] The carriage 109 may be attached to the column 102 by a first joint 113, which allows the carriage 109 to move relative to the column 102 (e.g., up and down a first or vertical axis 123). The first joint 113 may provide a first degree of freedom ("Z lift") to the adjustable arm support 105. The adjustable arm support 105 may include a second joint 115 that provides a second degree of freedom (tilt) to the adjustable arm support 105. The adjustable arm support 105 may include a third joint 117 that may provide a third degree of freedom ("upward pivot") to the adjustable arm support 105. An additional joint 119 (shown in FIG. 13) may be provided that mechanically constrains the third joint 117 to maintain the orientation of the rail 107 when the rail connector 111 is rotated about a third axis 127. The adjustable arm support 105 may include a fourth joint 121 that may provide a fourth degree of freedom (translation) of the adjustable arm support 105 along a fourth axis 129 .

[0053] 14 shows an end view of a surgical robotic system 140A having two adjustable arm supports 105A, 105B mounted on the face side of a table 101. A first robotic arm 142A is mounted to a bar or rail 107A of the first adjustable arm support 105B. The first robotic arm 142A includes a base 144A mounted to the rail 107A. The distal end of the first robotic arm 142A includes an instrument drive mechanism 146A that can be attached to one or more robotic medical instruments or tools. Similarly, the second robotic arm 142B includes a base 144B mounted to the rail 107B. The distal end of the second robotic arm 142B includes an instrument drive mechanism 146B. The instrument drive mechanism 146B can be configured to attach to one or more robotic medical instruments or tools.

[0054] In some embodiments, one or more of the robotic arms 142A, 142B include arms with seven or more degrees of freedom. In some embodiments, one or more of the robotic arms 142A, 142B can include eight degrees of freedom, including an insertion axis (one degree of freedom including insertion), wrist (three degrees of freedom including wrist pitch, yaw, and roll), elbow (one degree of freedom including elbow pitch), shoulder (two degrees of freedom including shoulder pitch and yaw), and base 144A, 144B (one degree of freedom including translation). In some embodiments, the insertion degree of freedom can be provided by the robotic arms 142A, 142B, while in other embodiments, the instrument itself provides the insertion via an instrument-based insertion architecture.

[0055] C. Instrument Drivers and Interfaces The end effector of the robotic arm of the system includes (i) an instrument driver (alternatively called an "instrument drive mechanism" or "instrument device manipulator") that incorporates electromechanical means for actuating the medical instrument, and (ii) a removable or detachable medical instrument that may lack any electromechanical components such as a motor. This dichotomy may be based on the need to sterilize medical instruments used in medical procedures, their complex mechanical assemblies and delicate electronics making it impossible to adequately sterilize expensive capital equipment. Thus, medical instruments can be designed to be detached, removed, and replaced from the instrument driver (and thus the system) for individual sterilization or disposal by a physician or physician's staff. In contrast, instrument drivers do not need to be replaced or sterilized and can be draped for protection.

[0056] FIG. 15 illustrates an exemplary instrument driver. The instrument driver 62, located at the distal end of the robotic arm, includes one or more drive units 63 arranged with parallel axes to provide a controlled torque to the medical instrument via a drive shaft 64. Each drive unit 63 includes an individual drive shaft 64 for interacting with the instrument, a gear head 65 for converting the rotation of the motor shaft to a desired torque, a motor 66 for generating the drive torque, an encoder 67 for measuring the speed of the motor shaft and providing feedback to the control circuit, and a control circuit 68 for receiving the control signal and operating the drive unit. Each drive unit 63 is independently controlled and motorized, and the instrument driver 62 can provide multiple (four as shown in FIG. 15) independent drive outputs to the medical instrument. In operation, the control circuit 68 receives the control signal, sends a motor signal to the motor 66, compares the resulting motor speed measured by the encoder 67 to a desired speed, and modulates the motor signal to generate the desired torque.

[0057] For procedures requiring a sterile environment, the robotic system may incorporate a drive interface, such as a sterile adapter connected to a sterile drape, located between the instrument driver and the medical instrument. The primary purpose of the sterile adapter is to transmit angular motion from the drive shaft of the instrument driver to the drive input of the instrument while maintaining physical separation, and therefore sterility, between the drive shaft and the drive input. Thus, an exemplary sterile adapter may be comprised of a set of rotational inputs and outputs intended to mate with the drive shaft of the instrument driver and a drive input to the instrument. The sterile drape, which is connected to the sterile adapter, is comprised of a thin flexible material, such as a transparent or translucent plastic, and is designed to cover the instrument driver, the robotic arm, and capital equipment, such as a cart (in a cart-based system) or a table (in a table-based system). The use of the drape allows the capital equipment to be placed in close proximity to the patient while still being positioned in an area that does not require sterility (i.e., the non-sterile field). On the other side of the sterile drape, the medical instrument may interface with the patient in an area that requires sterility (i.e., the sterile field).

[0058] D. Medical equipment FIG. 16 illustrates an exemplary medical instrument with a paired instrument driver. Similar to other instruments designed for use with a robotic system, the medical instrument 70 includes an elongated shaft 71 (or elongated body) and an instrument base 72. The instrument base 72, also referred to as an "instrument handle" due to its design intended for manual interaction by a physician, may generally include a rotatable drive input 73, e.g., a receptacle, pulley, or spool, designed to mate with a drive output 74 that extends through a drive interface on an instrument driver 75 at the distal end of a robotic arm 76. When physically connected, latched, and / or coupled, the mated drive input 73 of the instrument base 72 may share a rotational axis with the drive output 74 in the instrument driver 75 to enable the transfer of torque from the drive output 74 to the drive input 73. In some embodiments, the drive output 74 may include a spline designed to mate with a receptacle on the drive input 73.

[0059] The elongated shaft 71 is designed to be delivered through either an anatomical opening or lumen, such as in an endoscope, or a minimally invasive incision, such as in a laparoscopy. The elongated shaft 71 may be either flexible (e.g., with properties similar to an endoscope) or rigid (e.g., with properties similar to a laparoscope), or may include a customized combination of both flexible and rigid portions. When designed for laparoscopy, the distal end of the rigid elongated shaft may be connected to an end effector extending from a jointed wrist formed from a clevis having at least one degree of freedom, and a surgical tool or medical instrument, such as a grasper or scissors, that may be actuated based on forces from a tendon when the drive input rotates in response to torque received from the drive output 74 of the instrument driver 75. When designed for endoscopy, the distal end of the flexible elongated shaft may include a steerable or controllable bend that may articulate and bend based on torque received from the drive output 74 of the instrument driver 75.

[0060] Torque from the instrument driver 75 is transferred downstream of the elongated shaft 71 using tendons along the shaft 71. These individual tendons, such as pull wires, may be individually secured to individual drive inputs 73 in the instrument handle 72. From the handle 72, the tendons are directed down one or more pull lumens along the elongated shaft 71 and secured to a distal portion of the elongated shaft 71, or to a wrist at the distal portion of the elongated shaft. During a surgical procedure, such as a laparoscopic, endoscopic, or hybrid procedure, these tendons may be coupled to a distally attached end effector, such as a wrist, grasper, or scissors. Under such a configuration, torque exerted on the drive input 73 transfers tension to the tendons, thereby actuating the end effector in some way. In some embodiments, during a surgical procedure, the tendons can rotate the joint about an axis, thereby moving the end effector in one direction or another. Alternatively, the tendon may be connected to one or more jaws of a grasper at the distal end of the elongate shaft 71, with tension from the tendon causing the grasper to close.

[0061] In endoscopy, the tendons may be coupled to a flex or articulating section positioned along the elongated shaft 71 (e.g., at the distal end) via adhesive, control ring, or other mechanical fixation. If fixedly attached to the distal end of the flex, torque exerted on the drive input 73 is transferred downstream of the tendons, causing the softer flex (sometimes referred to as the articulatable section or region) to flex or articulate. Along the non-flexing portions, it may be advantageous to spiral or convolute the individual pull lumens that point along (or inwardly) the wall of the endoscope shaft to counterbalance radial forces resulting from tension in the pull wires. The angle of the helix and / or spacing between them may be altered or designed for a particular purpose, with a narrower helix exhibiting less shaft compression under load forces, while a lesser amount of helix results in more shaft compression under load forces, but also exhibits flexion limitations. At the other end of the spectrum, the pull lumen may be oriented parallel to the longitudinal axis of the elongate shaft 71 to allow for controlled articulation at a desired bend or articulatable section.

[0062] In endoscopy, the elongated shaft 71 houses several components that aid in robotic procedures. The shaft may define a working channel for positioning surgical tools (or medical instruments), irrigation, and / or aspiration to the surgical field at the distal end of the shaft 71. The shaft 71 may also house wires and / or optical fibers for transmitting signals to and from an optical assembly at the distal tip, which may include an optical camera. The shaft 71 may also house optical fibers for carrying light from a proximally located light source, such as a light emitting diode, to the distal end of the shaft.

[0063] At the distal end of the instrument 70, the distal tip may include a working channel opening for delivering tools to the surgical site for diagnosis and / or treatment, irrigation, and aspiration. The distal tip may also include a port for a camera, such as a fiberscope or digital camera, to capture images of the internal anatomical space. Relatedly, the distal tip may also include a port for a light source to illuminate the anatomical space when a camera is used.

[0064] 16, the drive shaft axis, and therefore the drive input axis, is orthogonal to the axis of the elongated shaft. However, this arrangement complicates the ability of the elongated shaft 71 to roll. Allowing the elongated shaft 71 to roll along its axis while holding the drive input 73 stationary can result in undesirable entanglement of the tendons as they extend from the drive input 73 and enter the pull lumen within the elongated shaft 71. Such resulting entanglement of tendons can interfere with any control algorithm intended to predict the movement of a flexible elongated shaft during an endoscopic procedure.

[0065] FIG. 17 shows an alternative design of the instrument driver and instrument, where the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument. As shown, a circular instrument driver 80 includes four drive units with parallel aligned drive outputs 81 at the end of a robotic arm 82. The drive units and their respective drive outputs 81 are housed in a rotating assembly 83 of the instrument driver 80, which is driven by one of the drive units in the assembly 83. In response to torque provided by the rotating drive unit, the rotating assembly 83 rotates along a circular bearing connecting the rotating assembly 83 to a non-rotating portion 84 of the instrument driver. Power and control signals may be transferred from the non-rotating portion 84 of the instrument driver 80 to the rotating assembly 83 through electrical contacts and may be maintained through rotation by a brushed slip ring connection (not shown). In other embodiments, the rotating assembly 83 may be responsive to a separate drive unit that is integrated into the non-rotatable portion 84 and thus not parallel to the other drive units. The rotation mechanism 83 enables the instrument driver 80 to rotate the drive units and their respective drive outputs 81 as a single unit about the instrument driver axis 85 .

[0066] Similar to the previously disclosed embodiments, the instrument 86 may include an elongated shaft portion 88 and an instrument base 87 (shown with a clear exterior skin for purposes of illustration) that includes a number of drive inputs 89 (such as receptacles, pulleys, and spools) configured to receive the drive outputs 81 in the instrument driver 80. Unlike the previously disclosed embodiments, the instrument shaft 88 extends from the center of the instrument base 87 with an axis that is substantially parallel to the axis of the drive inputs 89, rather than orthogonal as in the design of FIG.

[0067] When coupled to the rotation assembly 83 of the instrument driver 80, the medical instrument 86, including the instrument base 87 and the instrument shaft 88, rotates with the rotation assembly 83 about the instrument driver axis 85. Because the instrument shaft 88 is positioned at the center of the instrument base 87, the instrument shaft 88 is coaxial with the instrument driver axis 85 when attached. Thus, rotation of the rotation assembly 83 causes the instrument shaft 88 to rotate about its own longitudinal axis. Furthermore, as the instrument base 87 rotates with the instrument shaft 88, any tendons connected to the drive input 89 in the instrument base 87 do not become entangled during rotation. Thus, the parallelism of the axes of the drive output 81, drive input 89, and instrument shaft 88 allows for shaft rotation without entangling the control tendons.

[0068] 18 illustrates an instrument having an instrument-based insertion architecture, according to some embodiments. The instrument 150 can be coupled to any of the instrument drivers described above. The instrument 150 includes an elongated shaft 152, an end effector 162 connected to the shaft 152, and a handle 170 coupled to the shaft 152. The elongated shaft 152 includes a tubular member having a proximal portion 154 and a distal portion 156. The elongated shaft 152 includes one or more channels or grooves 158 along its outer surface. The grooves 158 are configured to receive one or more wires or cables 180 therethrough. Thus, the one or more cables 180 extend along the outer surface of the elongated shaft 152. In other embodiments, the cables 180 can also extend through the elongated shaft 152. Manipulation of one or more of the cables 180 (e.g., via an instrument driver) results in actuation of the end effector 162.

[0069] The instrument handle 170, which may also be referred to as an instrument base, may generally include a mounting interface 172 having one or more mechanical inputs 174, such as receptacles, pulleys or spools, designed to reciprocally mate with one or more torque couplers on a mounting surface of the instrument driver.

[0070] In some embodiments, instrument 150 includes a series of pulleys or cables that allow elongate shaft 152 to translate relative to handle 170. In other words, instrument 150 itself includes an instrument-based insertion architecture that accommodates the insertion of the instrument, thereby minimizing reliance on a robotic arm to provide insertion of instrument 150. In other embodiments, the robotic arm can be significantly involved in the insertion of the instrument.

[0071] E. Controller Any of the robotic systems described herein can include an input device or controller for manipulating an instrument attached to the robotic arm. In some embodiments, the controller can be coupled (e.g., communicatively, electronically, electrically, wirelessly, and / or mechanically) to the instrument such that manipulation of the controller causes a corresponding manipulation of the instrument, e.g., via master-slave control.

[0072] 19 is a perspective view of an embodiment of controller 182. In this embodiment, controller 182 includes a hybrid controller that can have both impedance control and admittance control. In other embodiments, controller 182 can utilize only impedance or passive control. In other embodiments, controller 182 can utilize only admittance control. By being a hybrid controller, controller 182 can advantageously have a lower perceived inertia during use.

[0073] In the illustrated embodiment, the controller 182 is configured to enable operation of two medical instruments and includes two handles 184. Each of the handles 184 is connected to a gimbal 186. Each gimbal 186 is connected to a positioning platform 188.

[0074] 19, each positioning platform 188 includes a selective compliance assembly robot arm (SCARA arm) 198 coupled to a column 194 by a prismatic joint 196. The prismatic joint 196 is configured to translate along the column 194 (e.g., along a rail 197) to allow each of the handles 184 to be translated in the z-direction, providing a first degree of freedom. The SCARA arm 198 is configured to allow movement of the handles 184 in the xy-plane, providing two additional degrees of freedom.

[0075] In some embodiments, one or more load cells are positioned within the controller. For example, in some embodiments, a load cell (not shown) is positioned within the body of each of the gimbals 186. By providing load cells, a portion of the controller 182 can operate under admittance control, thereby advantageously reducing the perceived inertia of the controller during use. In some embodiments, the positioning platform 188 is configured for admittance control while the gimbal 186 is configured for impedance control. In other embodiments, the gimbal 186 is configured for admittance control and the positioning platform 188 is configured for impedance control. Thus, in some embodiments, the translational or positional degree of freedom of the positioning platform 188 can rely on admittance control while the rotational degree of freedom of the gimbal 186 relies on impedance control.

[0076] F. Navigation and Control Traditional endoscopy may involve the use of fluoroscopy (e.g., as may be delivered through a C-arm) and other forms of radiation-based imaging modalities to provide intraluminal guidance to the operator-physician. In contrast, the robotic system contemplated by the present disclosure may provide non-radiation-based navigation and localization means to reduce the physician's exposure to radiation and reduce the amount of equipment in the operating room. As used herein, the term "localization" may refer to determining and / or monitoring the position of an object in a reference coordinate system. Techniques such as pre-operative mapping, computer vision, real-time EM tracking, and robot command data may be used individually or in combination to achieve a radiation-free operating environment. In other cases where radiation-based imaging modalities are still used, pre-operative mapping, computer vision, real-time EM tracking, and robot command data may be used individually or in combination to improve the information obtained solely by radiation-based imaging modalities.

[0077] 20 is a block diagram illustrating a localization system 90 for estimating the position of one or more elements of a robotic system, such as the position of an instrument, according to an exemplary embodiment. The localization system 90 may be a set of one or more computing devices configured to execute one or more instructions. The computing devices may be embodied by a processor (or processors) and computer readable memory in one or more of the components discussed above. By way of example, and not by way of limitation, the computing devices may be in the tower 30 shown in FIG. 1, the cart shown in FIGS. 1-4, the bed shown in FIGS. 5-14, etc.

[0078] 20, the localization system 90 may include a localization module 95 that processes the input data 91-94 to generate location data 96 of the distal tip of the medical instrument. The location data 96 may be data or logic that represents the location and / or orientation of the distal end of the instrument relative to a frame of reference. The frame of reference may be relative to a known object, such as the patient's anatomy or an EM field generator (see discussion of EM field generators below).

[0079] The various input data 91-94 will now be described in more detail. Pre-operative mapping can be accomplished utilizing the acquisition of low-dose CT scans. The pre-operative CT scans are reconstructed into three-dimensional images that are visualized, for example, as "slices" of a notch diagram of the patient's internal anatomy. When analyzed as a whole, an image-based model can be generated for the anatomical cavities, spaces, and structures of the patient's anatomy, such as the patient's pulmonary network. Techniques such as center-line geometry can be determined and approximated from the CT images to create a three-dimensional volume of the patient's anatomy, referred to as model data 91 (also referred to as "pre-operative model data" when generated using only pre-operative CT scans). The use of center-line geometry is discussed in U.S. Patent Application Serial No. 14 / 523,760, the contents of which are incorporated herein in their entirety. Network topological models may also be derived from CT images and are particularly suited for bronchoscopy.

[0080] In some embodiments, the instrument may be equipped with a camera to provide visual data 92. The localization module 95 may process the visual data to enable one or more vision-based location tracking. For example, pre-operative model data may be used along with the visual data 92 to enable computer vision-based tracking of a medical instrument (e.g., an endoscope, or an instrument advancing through the working channel of the endoscope). For example, using the pre-operative model data 91, the robotic system may generate a library of predicted endoscopic images from the model based on the expected path of travel of the endoscope, with each image linked to a location in the model. During surgery, this library may be referenced by the robotic system to compare real-time images captured by a camera (e.g., a camera at the distal end of the endoscope) to those in the image library to assist with localization.

[0081] Other computer vision based tracking techniques use tracking functions to determine the motion of the camera and therefore the endoscope. Some features of the localization module 95 may identify circular geometric shapes in the pre-operative model data 91 that correspond to anatomical lumens and track changes in those geometries to determine which anatomical lumens have been selected as well as the relative rotational and / or translational motion of the camera. The use of a phase map may further enhance vision based algorithms or techniques.

[0082] Optical flow, another computer vision based technique, may analyze the displacement and translation of image pixels in a video sequence in the visual data 92 to infer camera movement. Examples of optical flow techniques may include motion detection, object segmentation calculations, luminance, motion compensated coding, stereo disparity measurements, etc. By comparing multiple frames over multiple iterations, the movement and position of the camera (and therefore the endoscope) can be determined.

[0083] The localization module 95 can use real-time EM tracking to generate a real-time location of the endoscope in a global coordinate system that can be registered to the patient's anatomy represented by the pre-operative model. In EM tracking, an EM sensor (or tracker), consisting of one or more sensor coils embedded in a medical instrument (e.g., an endoscopic instrument) at one or more positions and orientations, measures the variations in an EM field generated by one or more static EM field generators positioned at known locations. The position information detected by the EM sensor is stored as EM data 93. The EM field generator (or transmitter) can be placed in close proximity to the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coil of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be "registered" intraoperatively to the patient's anatomy (e.g., the pre-operative model) to determine a geometric transformation that aligns a single position in the coordinate system with a position in the pre-operative model of the patient's anatomy. Once registered, an EM tracker embedded at one or more locations on the medical instrument (e.g., the distal tip of an endoscope) can provide a real-time indication of the medical instrument's progression through the patient's anatomy.

[0084] The robotic commands and kinematic data 94 may also be used by a localization module 95 to provide localization data 96 for the robotic system. The device pitch and yaw resulting from the articulation commands can be determined during pre-operative calibration. Intraoperatively, these calibration measurements can be used in combination with known insertion depth information to estimate the position of the instrument. Alternatively, these calculations may be analyzed in combination with EM, vision, and / or phase modeling to estimate the position of the medical instrument within the network.

[0085] As Fig. 20 illustrates, several other input data can be used by the localization module 95. For example, although not shown in Fig. 20, an instrument that utilizes shape sensing fibers can provide shape data that the localization module 95 can use to determine the position and shape of the instrument.

[0086] The localization module 95 may use a combination of the input data 91-94. In some cases, such a combination may use a probabilistic approach in which the localization module 95 assigns confidence weights to the location determined from each of the input data 91-94. Thus, if the EM data may be unreliable (such as when EM interference is present), causing the location determined by the EM data 93 to be less reliable, the localization module 95 may rely more heavily on the vision data 92 and / or the robot command and kinematics data 94.

[0087] As discussed above, the robotic systems discussed herein can be designed to incorporate one or a combination of two or more of the above-mentioned technologies. The computer-based control system of the robotic system, whether based on the tower, bed, and / or cart, may store, for example, in a non-transitory computer-readable storage medium, such as a persistent magnetic storage drive, solid-state drive, etc., computer program instructions that, when executed, cause the system to receive and analyze sensor data and user commands, generate system-wide control signals, and display navigation and localization data, such as instrument position within a global coordinate system, anatomical maps, etc.

[0088] 2. Introduction to robotic-assisted ancillary procedures. Treatment of certain medical conditions may include performing more than one medical procedure to completely treat the medical condition. For example, diagnosis and management of a lung lesion may include multiple treatment episodes to perform medical procedures including flexible endoscopy and thoracoscopy. After a lesion is found from a radiological study, such as through analysis of a CT scan, the physician may perform an endoscopic diagnosis and subsequent treatment over the course of multiple treatment episodes. In one example, if the physician suspects early-stage cancer in the patient, the physician may order the patient to first undergo an endoscopic procedure for the diagnosis of cancer. During the endoscopic procedure, the nodule may be biopsied, and if the physician determines that removal of the nodule is necessary, the physician may have the patient undergo a second treatment episode for surgical removal of the nodule.

[0089] There are drawbacks to performing multiple treatment episodes. Clinical costs and time demands from both caregivers and patients to diagnose and treat a patient's condition are increased due to such a multi-episode approach. In addition, during a surgical resection procedure, procedures (e.g., endoscopy) may need to be performed repeatedly to help precisely locate the tumor and provide an operative target for surgical resection. Furthermore, when staging a medical procedure over multiple treatment episodes, the patient may have to undergo multiple anesthesia episodes, which may increase risk and inconvenience to the patient. Multiple treatment episodes may also utilize increased intraoperative resources (e.g., pre-operative work-up, post-operative recovery, and possibly an overnight hospital stay), increasing time and costs to both the patient and the physician.

[0090] Rather than staging a medical procedure over multiple treatment episodes, physicians have the option of performing multiple procedures sequentially during a single treatment episode, which may be performed by bringing in additional clinical providers to help perform procedures in parallel as part of the single treatment episode.

[0091] However, with respect to multiple treatment episodes, there are drawbacks associated with single treatment episodes as currently performed. As mentioned above, multiple clinical providers may be required to assist in performing a single treatment episode, thereby resulting in increased costs and overcrowded space in the operating room. Furthermore, to perform multiple procedures sequentially over a single treatment episode, the physician may alternate between various techniques, which may involve switching between sterile and non-sterile techniques. Switching between sterile and non-sterile techniques may further involve shifting attention from one surgical site to another, regressing and significantly interrupting clinical workflow.

[0092] Coordination of multiple healthcare providers and / or physicians to perform procedures in parallel during a single treatment episode can be expensive and costly for certain procedures. One example of the use of multiple clinical providers in performing parallel procedures as part of a single treatment episode is combined endoscopic and laparoscopic surgery (CELS), a manual method of performing colonic polypectomy. Polyps can be assessed for whether they can be removed endoscopically based on their size, type, and location. If polyps cannot be removed endoscopically, they can be removed via segmental colectomy, which is associated with a relatively high complication rate and long recovery time. CELS was proposed as a method to allow extraluminal mobilization of the colon (with laparoscopic instruments) to make resection within the lumen (with endoscopic instruments) easier. CELS typically requires at least two physicians (to control the laparoscope and endoscopic instruments, respectively) and two assistants (to hold the laparoscope and colonoscope, respectively). One physician may move the instruments while the remaining provider holds them in place, which can be physically demanding for extended periods of time. There may be additional staff members in the room to control instrument exchanges, sutures or gauze, handle specimens after removal, laparoscopic instruments, and the like.

[0093] Embodiments of the present disclosure relate to systems and methods for performing two or more types / modes of treatment simultaneously (e.g., by a single user or team) as part of a single treatment episode. The systems and methods described herein improve upon the single and multiple treatment episodes described above. In some embodiments, parallel treatments can be performed as part of a single treatment episode by a novel robotic medical system, thereby reducing the need to have the same number of health care providers and / or physicians as with non-robotic assisted parallel medical treatments, such as existing CELS.

[0094] In addition to the above examples of endoscopic diagnosis and surgical resection of cancerous tumors, other exemplary medical procedures may benefit from the systems and methods described herein, including bronchoscopic localization of lung cancer with simultaneous thoracoscopic resection, endoscopic localization of digestive cancer with laparoscopic resection, endoscopic localization and resection of gastrointestinal cancer with laparoscopic assistance, endoscopic imaging or visualization for gastrointestinal reconstruction procedures such as gastrectomy, Roux-en-Y gastric bypass, ureteral stone / tumor localization and percutaneous removal / resection, etc. In some embodiments, such procedures may be performed in a single procedure episode. In some embodiments, such procedures may be performed with a minimal number of clinicians, and in some cases, only a single physician. Furthermore, in some embodiments, simultaneous procedures may be performed using a single type of console to control the simultaneous procedures.

[0095] According to aspects of the present disclosure, a first type of procedure performed during a concomitant / parallel medical procedure can include delivering one or more flexible devices to the patient, while a second type of procedure can include delivering one or more rigid devices to the patient. For example, in one embodiment, the two concomitant procedures can involve an endoscopic procedure (e.g., using a flexible scope) in combination with a laparoscopic procedure (e.g., using a rigid scope). In a medical procedure involving an intrabronchial tumor, a first endoscopic instrument (e.g., a flexible bronchoscope) can be inserted through the bronchus, while a second laparoscopic instrument (e.g., a rigid camera or cutter) can be inserted through an incision that provides access to the tumor.

[0096] In some embodiments, a first type of procedure can be performed through a natural orifice, while a second type of procedure can be performed through an incision. For example, in a medical procedure involving the removal of kidney stones, a first tool (e.g., a laser) can be inserted through a natural orifice in the urethra to break up the stones in the renal pelvis, while a second tool (e.g., a vacuum) can be inserted percutaneously through the incision to aspirate and remove the broken up kidney stones.

[0097] A. Systems and methods for performing associated procedures. In some embodiments, a single robotic medical system can perform two or more medical procedures simultaneously as part of a single treatment episode. FIG. 21 illustrates an embodiment of a bed-based robotic system configured to perform a concomitant procedure according to aspects of the present disclosure. As illustrated in FIG. 21, the robotic medical system 200 includes a first set of one or more robotic arms 205 and a second set of one or more robotic arms 210. The system 200 further includes a platform 215, which can include a bed on which a patient can be placed, with the first set of robotic arms 205 and the second set of robotic arms 210 disposed on bilateral arm supports or rails relative to the platform 215. The first set of robotic arms 205 may be coupled to a first adjustable arm support 220, while the second set of robotic arms 210 may be coupled to a second adjustable arm support 225 located on the opposite side of the platform 215 relative to the first adjustable arm support 220. The bed of the platform 215 may include a head and a foot section. The first arm support 220 and the second arm support 225 may be located between the head and the foot.

[0098] In certain embodiments, the first set of arms 205 may be configured to control one or more flexible instruments 230, such as, for example, a colonoscope, bronchoscope, or urethroscope (e.g., with inner and outer catheters), as part of an endoscopic procedure. The second set of arms 210 may be configured to control one or more rigid instruments 235, such as a rigid camera, vessel sealer, tissue cutter, stapler, needle driver, etc., as part of a laparoscopic procedure. In this embodiment, the first set of arms 205 is aligned in a virtual rail to deliver a flexible ureteroscope, according to some embodiments. The second set of arms 210 delivers one or more laparoscopic instruments through a laparoscopic port. In some embodiments, at least one of the laparoscopic instruments can be rigid, but in some embodiments, the second set of arms 210 may be configured to deliver a combination of rigid and flexible instruments, such as a rigid cutter and a flexible articulating laparoscope. As shown in FIG. 21 , the first set of arms 205 is configured to approach the patient from a different direction than the second set of arms 210. For example, the first set of arms 205 can approach the patient from the base of the platform 215, while the second set of arms 210 can approach the patient from the side of the platform 215. In some embodiments, one or more of the endoscopic or laparoscopic instruments can be navigated in part or in whole via EM or fluoroscopic navigation. In some embodiments, the first set of arms 205 can be locked while the second set of arms 210 is movable. In other embodiments, the first set of arms 205 is movable while the second set of arms 210 is locked.

[0099] As shown in FIG. 21 , the first set of arms 205 are coupled to a first adjustable arm support 220, while the second set of arms 210 are coupled to a second adjustable arm support 225. The first adjustable arm support 220 can be independently adjustable from the second adjustable arm support 225. In some embodiments, the first adjustable arm support 220 is at a different height than the second adjustable arm support 225, while in other embodiments, the first adjustable arm support 220 is at the same height as the second adjustable arm support 225. In some embodiments, the arm supports 220, 225 and / or the arms 205, 210 can be housed below the platform 215. In some embodiments, one or more of the arm supports 220, 225 and / or the arms 205, 210 can be elevated above the base of the platform, thereby avoiding “mop ramps” and inadvertent dirt getting onto these components. In some embodiments, one or more of the arm supports 220, 225 and / or arms 205, 210 can be elevated from a stowed position to a height above the top surface of the bed or platform 215.

[0100] In this embodiment, a pair of arms 210 is connected to a second adjustable arm support 225 and a pair of arms 205 is connected to a first adjustable arm support 220. In other embodiments, the number of arms on each of the adjustable arm supports can be an even number. In other embodiments, the number of arms can be more or less than the number of arms shown in FIG. 21.

[0101] Figure 22 illustrates another embodiment of a bed-based robotic system configured to perform a concomitant procedure according to aspects of the present disclosure. Similar to the embodiment of Figure 21, the embodiment illustrated in Figure 22 includes a first set of one or more robotic arms 205, a second set of one or more robotic arms 210, and a platform 215 having multiple robotic arms 205 and 210 positioned on bilateral arm supports relative to the platform 215. These bilateral arm supports include a first adjustable arm support 220 and a second adjustable arm support 225.

[0102] In contrast to the embodiment shown in FIG. 21, in the embodiment of FIG. 22, the first set of robotic arms 205 includes a single robotic arm configured to control a rigid laparoscopic instrument 235 percutaneously through the patient, while the second set of robotic arms 210 includes a pair of robotic arms configured to control a flexible endoscopic instrument 230. In other embodiments, the robotic arms 205 and 210 may be located on the same side of the bed (or adjacent sides of the bed) and may be configured to control a flexible endoscopic instrument 230 and a rigid laparoscopic instrument 235, respectively. In yet other embodiments, a set of robotic arms 205 and 210 including at least one arm located on each side of the platform 215 may be configured to control a first medical instrument (e.g., a flexible endoscopic instrument 230), and another set of robotic arms 205 and 210 including at least one arm located on each side of the platform 215 may be configured to control a second medical instrument (e.g., a rigid laparoscopic instrument 235).

[0103] In each of the embodiments shown in FIG. 21 and FIG. 22, a single bed-based system with a robotic arm attached thereto may be configured to perform both endoscopic procedures with one or more flexible instruments 230 and laparoscopic procedures with one or more rigid instruments 235. The endoscopic procedures may be performed through a natural orifice (e.g., throat) and the laparoscopic procedures may be performed through an incision (e.g., chest). The procedures may be advantageously performed simultaneously / concomitantly (partially or in whole) by a single user through a single console (further details of which are provided below). In some embodiments, the robotic medical system 200 may be configured to perform two types of medical procedures in series, as needed. For example, a first type of procedure may be performed on a patient. If such a procedure is ineffective on its own, a second type of procedure may be performed to outperform or supplement the first type of procedure as part of a "treatment escalation."

[0104] In each of FIGS. 21 and 22, the robotic arms 205 and 210 can be stowed and subsequently deployed from beneath the platform 215. The robotic arms 205 and 210 are configured to be positioned in multiple positions, for example, near the feet of the patient 240 and / or near the right side of the patient 240, based on commands received from a user. The robotic arms 205 and 210 are configured to be translatable along adjustable arm supports 220 and 225. The robotic arms 205, 210 are capable of multiple degrees of freedom, including two, three, four, five, six, seven, eight or more. In some embodiments, the robotic arms 205, 210 can include one or more redundant degrees of freedom. The robotic arms 205, 210 are coupled to adjustable arm supports 220, 225 configured to provide vertical, lateral, and longitudinal adjustment of the robotic arms 205 and 210. Independent of the movement of the robotic arms 205 and 210, in some embodiments the adjustable arm supports are configured to adjust in three degrees of freedom. In certain embodiments, the adjustable arm supports 220 and 225 may be in the form of a bar or rail along which the base of the robotic arm may translate. The base may couple the robotic arms 205 and 210 to the adjustable arm supports 220 and 225.

[0105] 21 , the first adjustable arm support 220 extends below and beyond the base of the platform 215, but is adjusted horizontally to allow the first robotic arm 205 to be positioned near the feet of the patient 240 as part of an endoscopic procedure. The second adjustable arm support 225 remains substantially aligned with the platform 215, but is adjusted vertically so that the second robotic arm 210 attached thereto may be positioned vertically of the patient 240 as part of a laparoscopic procedure. The first adjustable arm support 220 and the second adjustable arm support 225 allow the robotic arms 205 and 210 to be approached from different directions, including different heights and lateral positions.

[0106] Although the robotic arms 205 and 210 are described as being divided into a first set of robotic arms 205 and a second set of robotic arms 210, the robotic arms 205 and 210 may be divided into other groups (including one or more sets of arms), each configured to perform a separate procedure as part of an associated medical procedure. In some embodiments, an associated procedure (e.g., for diagnosis) may be performed with as few as two arms, one to hold a flexible camera and the other to hold an instrument. In some embodiments, an associated procedure (e.g., for treatment) may be performed using two arms or three arms. In some embodiments, four or more robotic arms 205 and 210 may be provided.

[0107] Depending on the combination of medical procedures being performed simultaneously, the robotic arms 205 and 210 may be configured and / or operated to control various medical instruments. Examples of applications in which one or more of the robotic arms 205 and 210 may be implemented using the robotic medical system 200 include (i) a robotic arm configured to control an introducer or sheath that provides access to a natural body orifice, such as the nose, mouth, vagina, urethra, rectum, or ear, (ii) a robotic arm configured to control an endoscope and / or endoscopic instruments (e.g., flexible instruments) through a natural orifice, with or without the aforementioned introducer or sheath, (iii) a robotic arm configured to hold and direct a thoracoscope or laparoscopic camera (e.g., flexible or rigid device) that provides extraluminal visualization of relevant anatomical spaces (e.g., thoracic, abdominal, peritoneal, and / or retroperitoneal spaces), and / or (iv) one or more robotic arms configured to hold and direct a thoracoscope or laparoscopic instrument (e.g., rigid device). These are merely example applications, and one of ordinary skill in the art will appreciate that the systems described herein are not limited to these implementations.

[0108] There are many advantages to using a single system, such as the robotic medical system of one of Figs. 21 and 22, to perform concomitant endoscopic and laparoscopic procedures with flexible and rigid tools. First, the use of a single system saves the amount of space occupied by the components of the system by requiring less equipment / capital in the operating room. Second, the use of a single system facilitates a single clinician / physician to perform both types of procedures without the assistance of other clinicians / physicians in the operating room. And third, by using a single system, each of the robotically controllable components of the system can be associated with the same global reference frame - for example, the position and / or orientation of an endoscopic instrument can be easily referenced relative to the position and / or orientation of a laparoscopic instrument. In other words, a single system provides knowledge of position and / or orientation data relative to all instruments and manipulators relative to a single coordinate frame, thereby enabling features such as collision prevention and / or avoidance, and / or computer-generated display of the instruments relative to each other.

[0109] 23 illustrates yet another embodiment of a robotic system configured to perform a concomitant procedure according to aspects of the present disclosure. The robotic medical system 300 includes both a platform-based robotic system 301 and a cart-based robotic system 303. The platform-based robotic system 301 includes a first set of one or more robotic arms 305, a second set of one or more robotic arms 310, and a bed or platform 315, where the robotic arms 305 and 310 are positioned on either side of the platform 315. The system 300 further includes a first adjustable arm support 320 coupled to the first set of one or more robotic arms 305, and a second adjustable arm support 325 coupled to the second set of one or more robotic arms 310. The cart-based robotic system 303 includes a third set of one or more robotic arms 330 coupled to a third adjustable arm support 335. In this embodiment, the platform-based robotic system 301 and the cart-based robotic system 303 are advantageously integrated to perform concomitant procedures as part of a single treatment episode.

[0110] The first and second sets of robotic arms 305 and 310 are configured to control one or more rigid instruments 340, such as, for example, a camera, a vessel sealer, a tissue cutter, a stapler, a needle driver, etc., as part of a laparoscopic procedure performed on a patient 350. The third set of robotic arms 330 is configured to control one or more flexible instruments 345, such as, for example, a colonoscope, a bronchoscope, or a urethroscope (e.g., with inner and outer catheters), as part of an endoscopic procedure. However, in other configurations, any combination or subset of the first robotic arm 305, the second robotic arm 310, and the third robotic arm 330 may be configured to control the rigid instruments 340 and / or the flexible instruments 345. In some embodiments, the robotic system 300 may include two or more cart-based systems 303, each of which may be configured to control one or more medical instruments.

[0111] 24 and 25 show two configurations of another embodiment of a bed-based robotic system configured to perform a concomitant procedure according to aspects of the present disclosure. As shown in FIG. 24 and FIG. 25, the robotic medical system 400 includes a first set of one or more robotic arms 405, a second set of one or more robotic arms 410, a platform 415, an adjustable arm support 420, one or more flexible medical instruments 425, and one or more rigid medical instruments 430. In this embodiment, the first set of one or more robotic arms 405 and the second set of one or more robotic arms 410 share the same adjustable arm support 420. An imaging device (e.g., CT, fluoroscope, etc.) is located on the side of the platform 415 opposite the arm support 420. The system 400 further includes an electromagnetic field generator 418 to assist in navigation of the one or more instruments via an EM sensor.

[0112] In a first configuration of the robotic medical system shown in FIG. 24, a first set of robotic arms 405 can be configured to control a flexible medical instrument 405 inserted through the bronchial passages of a patient 440. A second set of robotic arms 410 can be stowed under a platform 415 without controlling a medical instrument. In a second configuration shown in FIG. 25, a first set of robotic arms 410 can be configured to control a flexible instrument 425, while a second set of robotic arms 410 can be configured to control one or more rigid instruments 430 that can be elevated from a stowed position and inserted through an incision made in a patient 440. The robotic arms can be elevated via adjustable arm supports 420. As shown in FIG. 25, an arm rest 422 can be coupled to the platform and / or one or more of the adjustable arm supports 420 to rest the patient's arm during a procedure. 24 and 25, different subsets of robotic arms 405 and 410 can be selected to control medical instruments 425 and 430 depending on how the robotic medical system 400 is configured or utilized. Specifically, two robotic arms 405 can be included as part of the first set of robotic arms 405 to control a flexible instrument in the configuration of FIG. 24, while a single robotic arm 405 can be included as part of the first set of robotic arms 405 to control a flexible instrument in the configuration of FIG. 25. As shown in FIGS. 24 and 25, different combinations of robotic arms 405, 410 on a given adjustable arm support 420 are possible.

[0113] 26 is a flow chart illustrating an exemplary method operable by a robotic system or components thereof for performing at least partially or entirely associated medical procedures according to aspects of the present disclosure. For example, the steps of method 500 illustrated in FIG. 26 may be performed by a processor and / or other components of a medical robotic system (e.g., robot-controllable system 10, or one of the robotic medical systems 200, 300, or 400 described above) or associated systems. For convenience, method 500 will be described as being performed by a "system" in connection with the description of method 500.

[0114] The method 500 begins at block 501. At block 505, the system can control a first robotic arm to insert a first medical instrument through a first opening in the patient. At block 510, the system can control a second robotic arm to insert a second medical instrument through a second opening in the patient. The first robotic arm and the second robotic arm can be part of a first platform, and the first opening and the second opening can be located in two different anatomical regions of the patient. The method 500 ends at block 515.

[0115] As an example of an implementation of method 500, and with reference to the embodiment of FIG. 21 , in block 505, the robotic medical system 200 can control a first set of robotic arms 205 to insert a flexible medical instrument 230 through a first opening in the patient 240. Similarly, in block 510, the robotic medical system 200 can control a second set of robotic arms 210 to insert a rigid medical instrument 230 through a second opening in the patient 240. In some embodiments, the first opening can be a natural orifice of the patient and the second opening can be an incision made in the patient.

[0116] In some embodiments, the first medical instrument can include a first image capturing device (e.g., an endoscope) and the second medical instrument can include a second image capturing device (e.g., a laparoscope). By inserting the first and second medical instruments (each having a camera or other imaging component) through different openings located in two different anatomical regions of the patient, it is possible to provide different views of one or more anatomical regions of the patient. For example, when a flexible instrument is inserted through the patient's colon and a rigid instrument is inserted into the patient's abdominal cavity, the flexible instrument may be able to provide a view of a colon polyp from within the colon, while the rigid instrument may be able to provide a view of the same colon polyp from the abdominal cavity (e.g., from outside the colon). The systems described herein advantageously allow a user to switch between different camera views when viewing a display. In some embodiments, a view from the first image capturing device can be superimposed on a view from the second image capturing device on the display. In some embodiments, the view of the first image capturing device can be placed side-by-side with the second image capturing device in a tiling view on the display. Further details regarding camera view manipulation are provided below.

[0117] Aspects of the present disclosure, including the method 500 of FIG. 26, may enable endoscopic, percutaneous, laparoscopic procedures alone, as well as simultaneous combinations thereof. When utilizing all three modalities during a single procedure, a subset of one or more robotic arms may be assigned to drive and control flexible endoscopes and instruments to provide direct visualization and access to internal body lumens, another subset of one or more robotic arms may be assigned to drive and control laparoscopic / thoracic cameras to provide direct visualization within various body cavities, while another subset of one or more robotic arms may be assigned to drive and control rigid, semi-rigid, or flexible instruments within the body cavities. The robotic arms may be configured and deployed as needed in any case.

[0118] 27A and 27B provide a flow chart illustrating another exemplary method operable by a robotic system or components thereof for performing associated endoscopic and thoracic procedures according to aspects of the present disclosure. For example, the steps of method 600 illustrated in FIGs. 27A and 27B may be performed by a processor and / or other components of a medical robotic system (e.g., robotically-controllable system 10, or one of robotic medical systems 200, 300, or 400) or associated systems. For convenience, method 600 will be described as being performed by a "system" in connection with the description of method 600.

[0119] Method 600 begins at block 601. At block 605, a bed-based platform can be configured to receive a patient who is moved onto the bed by operating room staff. At block 610, the system can deploy a first arm from either the bed or an integrated cart-based system in preparation for an endoscopic instrument. At block 615, the first robotic arm receives a flexible instrument that can be loaded onto the first robotic arm by operating room staff. At block 620, under the control of a physician, the system can drive the flexible instrument into the patient's body through a natural body orifice.

[0120] The system may use the flexible instrument to locate the target pathology in block 625. In the case of a pulmonary lesion, blocks 620 and 625 may include introducing the flexible instrument into the airway and introducing the flexible instrument to the target (e.g., the lesion of interest) under the control of a physician.

[0121] In block 630, the system may deploy a second robotic arm to perform a laparoscopic resection. In some embodiments, block 630 may be executed in response to determining that the medical condition is cancer. In block 645, the second robotic arm may receive a thoracoscope instrument, which may be loaded into the first robotic arm by the operating room staff. The operating room staff may also create a thoracoscope port configured for the thoracoscope instrument to be inserted into the patient. The thoracoscope instrument may include a rigid camera, in this case a thoracoscope, and a thoracoscope instrument. The thoracoscope port may include a cannula that provides access to the patient's thoracic cavity.

[0122] At block 640, the system may perform laparoscopic resection of the target using the thoracoscope instrument under the control of the physician. The flexible instrument and thoracoscope may provide separate views of the target from inside and outside the airway, respectively, to assist the physician in performing the resection. Once the resection is complete, the physician and / or operating room staff may remove the thoracoscope instrument and the proximal thoracoscope port and remove the flexible device from the patient. The method 600 ends at block 645.

[0123] B. Treatment Escalation One advantage of the ability to control both endoscopic and laparoscopic instruments from a single platform (or hybrid bed-based platform and cart-based system) is the ability to assess the level of invasiveness of the surgery as needed. Different procedures have different degrees of invasiveness. For example, a first type of procedure can be a pure endoscopic resection. A second type of procedure can be an endoscopic resection with laparoscopic assistance. A third type of procedure can be a laparoscopic resection. The systems and methods described herein advantageously allow a physician to easily escalate from one type of procedure to another, such as from a first type of procedure to a second type of procedure, from a second type of procedure to a third type of procedure, or from a first type of procedure to a third type of procedure.

[0124] In some embodiments, a physician may intend to perform a first type of procedure (e.g., pure endoscopic resection) to include a first type of procedure (e.g., pure endoscopic resection) and a second type of procedure (e.g., endoscopic resection with laparoscopic assistance). In some embodiments, the first type of procedure may be less invasive than the second type of procedure. For example, in the first type of procedure, the physician may attempt to perform the resection endoscopically without the need to create an incision in the patient. In the second type of procedure, the degree of invasiveness increases when ports and holes are introduced into the patient's abdomen to provide laparoscopic assistance. Ports and holes may be provided for laparoscopes and / or other laparoscopic instruments, for example, to view the tissue and positioning, but the resection is still endoscopic, which keeps the rate of complications and recovery time relatively minimal. The systems and methods described herein may advantageously enable a physician to easily escalate from a first type of procedure to a second type of procedure.

[0125] In some embodiments, a physician may intend to escalate the procedure to include a second type of procedure (e.g., endoscopic resection with laparoscopic assistance) and a third type of procedure (e.g., laparoscopic resection). In some embodiments, the second type of procedure may be less invasive than the third type of procedure. For example, in the second type of procedure, the physician may attempt to perform an endoscopic resection with laparoscopic assistance. In the third type of procedure, the degree of invasiveness increases as the resection is performed laparoscopically. Such resections, e.g., segmental colectomies, may involve higher risks and recovery times relative to the first type of procedure and the second type of procedure. The systems and methods described herein advantageously allow a physician to easily escalate from the second type of procedure to the third type of procedure.

[0126] One exemplary procedure that may be escalated is colon polypectomy. A physician may begin treatment by attempting pure endoscopic resection of a colon polyp. If pure endoscopic resection fails, the physician may rapidly perform endoscopic resection of the colon polyp with the assistance of laparoscopic instruments. In some embodiments, escalation may be performed without bringing additional personnel or capital equipment into the room. If endoscopic resection is still inadequate for resection despite the assistance of laparoscopic instruments, the physician may escalate the procedure to a full laparoscopic procedure and laparoscopic resection. In this example, performing an endoscopic resection with the assistance of laparoscopic instruments may have a higher level of invasiveness than performing a pure endoscopic resection, while performing a laparoscopic resection may have a higher level of invasiveness than performing an endoscopic resection with the assistance of laparoscopic instruments. The level of invasiveness of a given procedure may be determined based on a number of factors, including, but not limited to, the patient's desired or expected recovery time, the absence or presence of an incision for delivering the instrument, the size of the incision required to deliver the medical instrument into the patient's body, the expected morbidity following the procedure, and the expected risk of complications.

[0127] By using a step-wise expansion procedure to treat, a physician can first attempt to perform a minimally invasive procedure before attempting to perform a more invasive procedure. For example, by using a step-wise expansion procedure to treat a colon polyp, a physician can advantageously remove some or all of the colon polyp using a full endoscopic resection, thereby potentially reducing the associated recovery time for the patient without extending treatment over multiple episodes. Although a step-wise expansion procedure is described above in connection with the colon polyp example, a step-wise expansion procedure may be applied to other medical procedures, including, for example, the diagnosis and removal of cancerous nodules.

[0128] 28 is a flow chart illustrating an exemplary method operable by a robotic system or components thereof for performing an associated medical procedure including an escalation procedure according to an embodiment of the present disclosure. For example, the steps of method 700 illustrated in FIG. 28 may be performed by one or more processors and / or other components of a medical robotic system (e.g., robot-controllable system 10, or one of robotic medical systems 200, 300, or 400) or associated systems. For convenience, method 700 will be described as being performed by a "system" in connection with the description of method 700.

[0129] Method 700 begins at block 701. Method 700 may be performed during method 500, for example, to perform a concomitant medical procedure illustrated in FIG. 26 between blocks 505 and 510. However, the timing of performing method 700 is not limited and may be performed before block 505, after block 510, or simultaneously with one or more of blocks 505 and 510. In block 705, the system may control a first robotic arm to perform a first medical procedure. The first medical procedure may include locating a target site. The system may further select an incision site based on locating the target site. The incision may be used to deliver a second medical instrument for use in a second medical procedure in block 710.

[0130] In block 710, in response to determining that the first medical procedure failed to fully treat the patient's medical condition, the system may control a second medical procedure to perform the second medical procedure to fully treat the patient's medical condition. The second medical procedure may have a higher level of invasiveness than the first medical procedure. The first medical procedure and the second medical procedure are performed simultaneously during a single medical episode. In some embodiments, the first medical procedure and the relatively more invasive second medical procedure are performed using a single platform, such as a cart-based platform or a bed-based platform having multiple arms. In other embodiments, the first medical procedure and the relatively more invasive second medical procedure are performed using multiple integrated platforms, such as combining a bed-based platform with a cart-based platform, or combining a cart-based platform with another cart-based platform. In some embodiments, the system may further control a second robotic arm to perform the second medical procedure in response to determining that the target site in the patient's anatomy meets the conditions for treatment via the second medical procedure. The method 700 ends in block 715.

[0131] C. A user interface for control of multiple medical instruments. Another aspect of the present disclosure relates to a user interface that can allow a single user to control all of the robotic arms during the accompanying procedure. In other words, aspects of the present disclosure relate to the use of a novel single interface that can be used to perform endoscopic interventions using one or more flexible devices, as well as laparoscopic interventions using one or more rigid devices.

[0132] 29 illustrates an example console including one or more types of interfaces for controlling a robotic arm according to an embodiment of the present disclosure. As shown in FIG. 29, the console 800 includes a viewer 805 and a controller 810 including two handles (also called positioning platforms) 815 configured to receive input from a user's left and right hands, a pendant 820, an armrest 825, and one or more foot pedals 830.

[0133] FIG. 30 illustrates a close-up view of the controller shown in FIG. 29 according to an aspect of the present disclosure. The controller 810 of FIG. 30 may be similar to the controller 182 shown in FIG. 19. FIG. 31 illustrates a close-up view of one of the handles shown in FIG. 29 and FIG. 30 according to an aspect of the present disclosure. In some embodiments, the handle 815 includes a button 835 and a finger grip 840. The button 835 provides a user interface that allows a user to actuate an end effector of a corresponding medical instrument. The finger grip 840 can provide an interface that allows a user to grasp the handle 815 and manipulate the position of the handle 815 in six degrees of freedom. The handle 815 can also function as a gimbal that allows a user to manipulate the handle in three degrees of freedom (e.g., pitch, yaw, and roll).

[0134] Figure 32 shows a close-up view of the pendant shown in Figure 29 according to an embodiment of the present disclosure. Pendant 820 includes an insert / retract joystick 845, a menu button 850, a quick action button 855, a pause button 860, an articulation and relaxation joystick 865, snapshot, light, and one programmable button 870, and a five button cluster button 875. Pendant 820 may be configured to drive a flexible instrument, such as flexible instrument 230 of Figure 21.

[0135] 29-32 include two or more types of user interfaces (e.g., a gimbal-based interface and a pendant-based interface), in some embodiments, the console 800 may include a single type of interface, such as a handle 815 for performing the associated procedures disclosed herein. For example, in some embodiments, the left hand handle 815 may be configured to control an endoscopic instrument, while the right hand handle 815 may be configured to control a laparoscopic instrument. In other embodiments, the left hand handle and the right hand handle 815 may be used in two different modes - a first mode configured to control one or more flexible endoscopic instruments and a second mode configured to control one or more rigid laparoscopic instruments. A foot pedal 830 or other button on the console 800 may be configured to receive input from a user to switch between the two modes. The use of a single type of interface to control two different instruments (e.g., a flexible endoscope and a rigid laparoscope) is quite novel, as many times, two different types of interfaces are often used to control such various instruments.

[0136] Although a controller 810 such as the controller 810 of FIG. 30 having left and right hand handles 815 may be used to control a laparoscopic instrument, it is not typical to use this type of controller 810 with an accompanying endoscopic procedure involving one or more flexible devices. However, it may be desirable to provide a user with a single user interface to control two or more medical instruments (including endoscopic and laparoscopic instruments) through the same interface, so that the user does not have to continually switch between different user input devices when switching between controlling the two medical instruments. Thus, to provide a single interface that can be physically controlled, the controller 810 may be adapted to control endoscopic instruments in addition to laparoscopic instruments.

[0137] 33 is a flow chart illustrating an exemplary method operable by a robotic system or components thereof for performing a concomitant medical procedure via a single user interface in accordance with aspects of the present disclosure. For example, the steps of method 900 illustrated in FIG. 33 may be performed by a processor and / or other components of a medical robotic system (e.g., robot-controllable system 10, or one of robotic medical systems 200, 300, or 400) or an associated system. For convenience, method 900 will be described as being performed by a "system" in connection with the description of method 900.

[0138] The method 900 begins at block 901. At block 905, the system can use a user interface to manipulate a first instrument inserted through a first opening in the patient via a first robotic arm. At block 910, the system can use a user interface to manipulate a second instrument inserted through a second opening in the patient via a second robotic arm. The first opening and the second opening are located in two different anatomical regions of the patient. In some embodiments, the first opening can be a natural orifice of the patient and the second opening can be an incision made in the patient. The first instrument can be flexible and the second instrument can be rigid. The method 900 ends at block 915.

[0139] In one embodiment, in response to the system receiving a selection from a user to control the endoscopic instrument, one of the handles 815 is mapped to control the insertion and retraction of the endoscopic instrument, while the other handle 815 is mapped to control the articulation and roll of the endoscopic instrument. The handle 815 that controls the insertion and retraction may be tactilely constrained to move in a straight line, and the controller 810 may include a clutch configured to allow the user to adjust the stroke length available to translate the endoscopic instrument into or out of the patient's body. The handle 815 that controls the articulation and roll may be tactilely constrained to prevent the handle 815 from moving in a planar manner. The position of the other articulation / roll handle 815 may be fixed in a coordinate plane, but may allow rotation, pitch, and yaw. Accessory buttons 835 on the articulation / roll handle 815 may be configured to allow the user to irrigate or aspirate the lumen as well as deliver energy to the endoscopic instrument.

[0140] In a second embodiment, one of the handles 815 may be mapped to control the insertion and retraction of the endoscopic instrument, while the other handle 815 is mapped to control the articulation and roll of the endoscopic instrument as in the first embodiment, but the user's driving experience may be slightly modified. One handle 815 may be haptically constrained to move in a straight line, but rather than requiring the user to use a clutch, the user controller 810 may be configured to allow the user to move their hand away from or towards the target to be treated. The amount that the user moves their hand may be mapped to the speed at which the endoscopic instrument moves in or out of the patient's body. The remaining handles 815 may be configured to be controlled in a manner similar to that described in the first embodiment.

[0141] The third embodiment can include a controller 810 with a set of secondary interfaces (not shown) in addition to the primary handle 815 shown in Figures 29-31. The primary left and right handles 815 can be configured similar to laparoscopic or thoracoscope control interfaces. The secondary set of left and right interfaces can be configured to drive an endoscopic instrument. One of the secondary interfaces can include a loop of endoscope-like insertion tube attached to two wheels. The loop of insertion tube can be configured to be translated along an axis and wound left and right. The insertion tube can provide either position or velocity based control. There are toggle buttons on the controller 810 to allow the user to cycle through concentric instruments that need to be translated or rotated within the patient's body. Another secondary interface can include a pseudo endoscopic tip augmented with a series of buttons. The pseudo endoscopic tip can be configured to be manipulated by the user to indicate the desired shape of the distal end of the robotically controlled endoscopic instrument. Buttons on the pseudo endoscopic tip can be utilized to irrigate or aspirate within the lumen as well as deliver energy to the endoscopic instrument. The pseudo endoscopic tip can assume the current position of the endoscopic instrument being controlled and maintain that position until the user instructs it otherwise, or it can behave like a conventional endoscopic instrument and maintain its position when actively instructed by the user. Another secondary interface can include "joystick" style buttons on one or more of the handles 815, which can be used to control one or more flexible instruments in a manner similar to the inputs on the pendant 820. In some embodiments, the handle 815 repositions itself into an alternative configuration (e.g., pointing upwards) to facilitate ergonomic control of the secondary interface joystick buttons.

[0142] In the fourth embodiment, the controller 810 may include a secondary interface for translating and rolling the endoscopic instrument as described in the third embodiment, but one of the two primary left or right hand interfaces may be configured to control articulation and other functions of the endoscopic instrument.

[0143] In a fifth embodiment, the controller 810 can include a secondary interface including a pendant 820 as shown in Figure 32. In this embodiment, the user must physically switch back and forth between the left and right handles 815 and the pendant 820 must move between controlling the laparoscopic and endoscopic instruments. The current view displayed by the viewer 805 can be controlled by the controller 810.

[0144] In some embodiments, the console 800 is configured to limit the number of robotic arms that can be simultaneously controlled while constraining the motion of other robotic arms. For example, in some embodiments, the interface may be used to drive a selected robotic arm while constraining the motion of other robotic arms (e.g., two, three, four or more).

[0145] D. Viewing and switching image displays. 29-32, in some embodiments, the viewer 805 may be configured to display images from endoscopic and laparoscopic imaging sensors, as well as any pre-operative plans or scans loaded into the system or other sources of live video, such as an ultrasound probe. The user may switch between different views in a variety of ways, including but not limited to using accessory buttons 835 located on the left or right hand interface, using accessory foot pedals 830 or switches activated by the user's foot, knee, toe or elbow, using haptic-enabled commands in combination with the instrument clutch to switch or cycle between multiple views using hand-based gestures via the left and right hand interfaces, using a control pendant 825 mounted on the console, or using any combination. Secondary views may be displayed via picture-in-picture, side-by-side, split frame, or cyclical view modes.

[0146] 34 and 35 are exemplary diagrams that may be displayed by a viewer during an associated medical procedure according to aspects of the present disclosure. Specifically, FIG. 34 shows a thoracic view 1000 and FIG. 35 shows an endoscopic view 1050.

[0147] In some embodiments, control of the currently selected medical instrument is synchronized with the coordinate frame of the primary view displayed by the viewer 805. The system may consider the instrument from the primary view approach to be displayed as the primary instrument, but the system may allow the user to control any instrument on the system relative to the primary view. In other words, in a scenario where the user is in thoracoscope view mode 1000, the thoracoscope instrument is the primary instrument. As shown in thoracoscope view mode 1000 of FIG. 34, a first thoracoscope instrument 1005 and a second thoracoscope instrument 1010 can be seen.

[0148] The user may wish to adjust the endoscope positioned within the lung. To do this, the user may display the appropriate secondary view (in this case, the endoscopic view 1050) and utilize one of the interfaces described above in "Section C." to switch from the primary left or right hand instrument to the secondary left or right hand instrument and adjust the endoscopic instrument as needed. As shown in FIG. 35, the endoscopic view 1050 may include a first view 1055 of the camera on the endoscope and a second view 1060 showing the position of the tip of the endoscope relative to the pre-operative model. In addition to switching between the thoracoscope view 1000 and the endoscopic view 1050, the system may further be capable of switching between the thoracoscope view 1000, the endoscopic view 1050, and a third view obtained via a pre-operative scan of the patient.

[0149] FIG. 36 is another exemplary diagram that may be displayed by a viewer during a concomitant medical procedure according to aspects of the present disclosure. In some embodiments, the system may be configured to have a computer-generated overlay of one image on top of another, as shown in FIG. 36. Indeed, in the embodiment shown in FIG. 36, two separate view-on-view embodiments are shown. First, the endoscopic view from the flexible scope (top right corner) is overlaid on the laparoscopic view 1090 from the rigid scope (base image). Second, a graphic or virtual representation 1095 of the flexible scope (in outline) is also overlaid on the laparoscopic view 1090 from the rigid scope. By providing such view-in-view functionality, this helps physicians perform multiple procedures at the same time, minimizing the need for unnecessary personnel performing individual procedures.

[0150] 37 is a flow chart illustrating an exemplary method operable by a robotic system or components thereof for switching displayed images while performing an associated medical procedure according to aspects of the present disclosure. For example, the steps of method 1100 illustrated in FIG. 37 may be performed by a processor and / or other components of a medical robotic system (e.g., robotically-controllable system 10, or one of robotic medical systems 200, 300, or 400) or an associated system. For convenience, method 1100 will be described as being performed by a "system" in connection with the description of method 1100.

[0151] The method 1100 begins at block 1101. At block 1105, the system can deliver a first scope through a first opening of the patient via a first robotic arm to acquire a first image. At block 1110, the system can deliver a second scope through a second opening of the patient via a second robotic arm to acquire a second image. At block 1115, the system can switch between the first image and the second image on the display. In some embodiments, the system can be further configured to switch to a view in which the first image is superimposed on the second image on the display. In other embodiments, the system can be configured to switch between a first image, a second image, and a third image obtained from a preoperative scan of the patient (e.g., a computed tomography (CT) scan or a fluoroscopy scan). The system can be further configured to overlay a virtual image on either the first image or the second image on the display, as shown in FIG. 36. The method 1100 ends at block 1120.

[0152] 3. Exemplary Methods and Workflows for Concomitant Treatment There are many exemplary methods and workflows that are enabled by the robotic medical system described herein. One particular exemplary medical procedure that can be performed using the described robotic medical system is colorectal intervention. In a colorectal intervention procedure, a physician can perform a minimally invasive intervention, which can be a very complex procedure, to attempt to address colorectal disease. In some cases, the treatment of colorectal disease can include the removal of colonic polyps. Although aspects of the present disclosure may use colorectal intervention as a specific example, the systems and methods described herein can also be used in other procedures, such as gastrointestinal and thoracic interventions, as well.

[0153] CELS can be utilized in procedures such as colorectal and gastrointestinal interventions. For example, certain colonic polyps (such as large sessile polyps or those in difficult locations relative to the lumen of the colon) cannot be resected by a purely endoscopic approach using current techniques. These polyps can be resected endoscopically with the assistance of laparoscopic instruments. In such cases, the colon can be repositioned using laparoscopic instruments to allow endoscopic resection of the polyp. In addition or in an alternative procedure, endoscopic tools can be used to localize the polyp to allow targeting and / or tissue-sparing laparoscopic resection of the polyp.

[0154] Because laparoscopic-assisted endoscopic resection and endoscope-assisted laparoscopic resection can each be less invasive (e.g., may result in shorter patient recovery times) than pure laparoscopic resection, the use of endoscopes and combined endoscopic / laparoscopic procedures with CELS can potentially lead to better outcomes for patients with polyps.

[0155] However, there may be some challenges in implementing CELS in practice. One challenge is that CELS may often require several resources and multiple personnel, such as at least four physicians, to (i) control the laparoscopic camera, (ii) control the laparoscopic instruments, (iii) control the endoscopic camera, and (iv) control the endoscopic instruments. In addition to the four physicians, CELS may also require several clinicians and assistants to support certain situations. Therefore, CELS has not been widely adopted and is limited to certain facilities (e.g., high-end academic centers) that can support multiple clinical providers for a single procedure.

[0156] 38A and 38B include a flow chart illustrating an example workflow for performing CELS, according to an embodiment of the present disclosure. For example, the steps of workflow 1200 illustrated in FIG. 38A and 38B may be performed by one or more physicians, clinicians, and / or assistants. For convenience, CELS workflow 1200 is described as being performed by four physicians in connection with the description of workflow 1200.

[0157] Referring to FIG. 38A, workflow 1200 begins at block 1201. At block 1205, workflow 1200 includes one or more physicians introducing an endoscope (e.g., a colonoscope) and one or more other endoscopic instruments into the colon of a patient. At block 1210, workflow 1200 includes one or more physicians identifying and characterizing a target anatomical structure using the endoscope and one or more other endoscopic instruments. At block 1215, workflow 1200 includes one or more physicians placing multiple laparoscopic ports into the patient and introducing a laparoscope and one or more laparoscopic instruments into the patient. The physicians help establish a sterile boundary between a sterile field (e.g., a laparoscopic port) and a non-sterile area (e.g., an endoscope access point, e.g., the patient's anus). The laparoscope and one or more laparoscopic instruments may be inserted into the patient's abdominal cavity.

[0158] 38B, in block 1220, the workflow 1200 includes using the endoscopic camera and instruments and laparoscopic camera and instruments interchangeably to position the target anatomical structure for the intervention. The physician may view different screens for the endoscopic and laparoscopic views. In other embodiments, the same screen may view both the endoscopic and laparoscopic views. In block 1225, the workflow 1200 optionally includes exchanging one or more of the endoscopic and laparoscopic instruments. In block 1230, the workflow 1200 includes performing the intervention at the target anatomical structure using one or more of the endoscopic and laparoscopic instruments. In block 1235, the workflow 1200 includes removing all of the endoscopic and laparoscopic cameras and instruments and closing the laparoscopic ports. The workflow ends in block 1240.

[0159] Aspects of the present disclosure relate to an associated system capable of performing CELS. The system is capable of performing both endoscopic procedures (e.g., procedures involving the use of one or more flexible instruments) and laparoscopic procedures (e.g., procedures involving the use of one or more rigid instruments). Endoscopic and laparoscopic capabilities are advantageously integrated into the associated system in a relatively compact form factor. In addition to the compact form factor, the system can be controlled by a single control unit, thereby advantageously reducing the dependency on multiple physicians and assistants in the operating room. By reducing the dependency on multiple physicians and assistants when performing CELS, this allows the system to be widely used in multiple operating rooms and emergency rooms in hospitals around the world without at least some of the above-mentioned disadvantages of conventional CELS.

[0160] 39 illustrates an embodiment of a bed-based robotic system configured to perform a concomitant procedure according to aspects of the present disclosure. For example, the system 1300 can be used in a method for performing a CELS procedure. The system can be similar to the system described in connection with FIG.

[0161] As shown in Fig. 39, the system 1300 includes a patient platform 1305 including a bed 1307 to which multiple robotic arms 1310 and 1315 are attached on adjustable arm supports 1311 and 1316. The system 1300 further includes a display 1321 that can be configured to display a live video stream. In this embodiment, the display 1321 includes a television screen. In other embodiments, the display 1321 can be a screen that is part of a tower monitor, and the video stream can further be streamed to an external room monitor via HDMI, SDI, or similar means.

[0162] One or more of the first set of robotic arms 1310 are configured to manipulate one or more flexible instruments. In some embodiments, the flexible instruments include a flexible scope, such as an endoscope, and one or more endoscopic instruments. In the embodiment of FIG. 39, the first set of robotic arms 1310 includes two arms, but in other embodiments, more or fewer robotic arms 1310 can be used to control an endoscope and / or endoscopic instruments. One or more of the second set of robotic arms 1315 are configured to manipulate one or more rigid instruments. In some embodiments, the rigid instruments include a rigid scope, such as a laparoscope and one or more laparoscopic instruments. In the embodiment of FIG. 39, the second set of robotic arms 1315 includes three arms, but in other embodiments, more or fewer robotic arms 1315 can be used to control a laparoscope and / or laparoscopic instruments.

[0163] The display 1321 can include a large video screen configured to display feedback from the flexible scope and / or the rigid scope. In some embodiments, the feedback can include a live video stream from the laparoscope and / or the endoscope. In other embodiments, the feedback can include a virtual representation of the location of the laparoscopic and endoscopic instruments relative to a model of the patient's anatomy. In the embodiment of FIG. 39, the system 1300 is configured to display live streams from both the laparoscope and the endoscope simultaneously. For example, the feedback from the flexible scope and the feedback from the rigid scope can be displayed on the video screen in a picture-in-picture view. In another example, the feedback from the flexible scope and the feedback from the rigid scope can be displayed on the video screen in a side-by-side view.

[0164] While FIG. 39 shows an embodiment in which each robotic arm 1310 and 1315 is mounted to a platform 1305, in alternative embodiments, the system 1300 can include one or more carts, each including one or more robotic arms. The carts can communicate with each other such that a physician can control flexible and rigid instruments from a central location. In one embodiment, the robotic arm 1310 configured to manipulate flexible instruments can be mounted to a first cart, while the robotic arm 1315 configured to manipulate rigid instruments can be mounted to a second cart. Examples of systems including cart-based robotic arms are shown, without limitation, in FIGS. 1, 3, 4, and 23.

[0165] FIG. 40 is a flow chart illustrating another exemplary workflow for performing CELS according to an embodiment of the present disclosure. For example, the steps of the workflow 1400 shown in FIG. 40 may be performed by one or more physicians, clinicians, and / or assistants. Some of the steps of the method 1400 of FIG. 40 may be performed by a processor and / or other components of a medical robotic system (e.g., one of the robot-controllable system 10, or robotic medical systems 200, 300, 400, 1300, or 1500, discussed above) or associated systems. Certain steps of the method 1400 may also be performed by the system in response to commands received, for example, from a physician via an input device (e.g., as shown in FIGS. 29-32), or may be performed automatically by the system without intervention from a user.

[0166] Referring to FIG. 40, workflow 1400 begins at block 1401. At block 1405, workflow 1400 includes manipulating a flexible instrument using a first robotic arm of a robotic system. For example, a physician may input a first command to the associated system 1300 of FIG. 39 to manipulate the flexible instrument. In some embodiments, the physician may manipulate the flexible instrument using the first robotic arm through a natural orifice of the patient. At block 1410, workflow 1400 includes manipulating a rigid instrument using a second robotic arm of the robotic system. The physician may input a second command to the associated system to manipulate the rigid instrument. In some embodiments, the physician may manipulate the rigid instrument using the second robotic arm through an incision made in the patient. The physician may be able to switch between control of the flexible instrument and the rigid instrument. In other embodiments, the physician may use separate input devices for control of the flexible instrument and the rigid instrument. Exemplary input devices and interfaces that may be used by the physician are shown in FIGS. 29-32.

[0167] At block 1415, the workflow 1400 includes displaying feedback from the flexible instruments. The feedback from the flexible instruments may be displayed, for example, by the display 1321 of FIG. 39. At block 1420, the workflow 1400 includes displaying feedback from the rigid instruments. As described in more detail below, the feedback from the flexible instruments and the feedback from the rigid instruments may be displayed in a picture-in-picture view, a side-by-side view, and / or feedback from only one of the flexible instruments and the rigid instruments may be displayed at a time. The physician may be able to control how the feedback is displayed. The physician may also be able to select feedback from one of the flexible instruments and the rigid instruments as a primary view and the other feedback as a secondary view. The system may display the primary view and the secondary view on the same view screen of the display. The method 1400 ends at block 1425.

[0168] 41 illustrates another embodiment of a robotic system configured to perform a concomitant procedure in accordance with aspects of the present disclosure. Specifically, the system 1500 of FIG. 41 can be configured to perform a colorectal intervention using a CELS procedure.

[0169] The system 1500 includes robotic arms 1510 and 1515 positioned on a pair of adjustable arm supports 1411 and 1416, thereby allowing the robotic arms 1510 and 1515 to engage both sides of a patient 1503. The adjustable arm supports 1411 and 1416 may be coupled to a platform 1507 or a bed. The first pair of robotic arms 1510 is used to control one or more flexible instruments, such as one or more endoscopes and / or endoscopic tools. In this embodiment, one of the first pair of robotic arms 1510 controls an endoscope 1525 having one or more working channels, while the other of the first pair of robotic arms 1510 controls an endoscopic tool 1530 through the endoscope 1525.

[0170] The endoscopic tools 1530 may include one or more polyp removers or receivers, including, but not limited to, snares (e.g., polyp snares), forceps, nets, graspers, baskets, balloons, injectors (e.g., of dyes, markers, or therapeutic substances), ablation probes, and wires, all of which may or may not be capable of delivering interventional energy, such as electrosurgical energy. Additionally, the endoscopic tools 1530 may include imaging modalities, such as ultrasound imaging or fluorescent imaging. In some embodiments, the endoscope 1525 and / or the endoscopic tools 1530 may be used to view and assist in the removal of one or more polyps, for example, via one or more cameras located at the distal end of the endoscope 1525 and / or the endoscopic tools 1530. In other embodiments, the endoscope 1525 and / or the endoscopic tools 1530 may be used for localization to enable targeting and / or tissue-sparing laparoscopic resection.

[0171] In the embodiment shown in FIG. 41, the second set of four robotic arms 1515 are used to control one or more rigid instruments, such as one or more laparoscopes and / or laparoscopic tools. For example, one of the second set of robotic arms 1515 controls a laparoscope 1535 for viewing the abdomen of the patient 1503, while the other three of the second set of robotic arms 1515 control laparoscopic tools 1540, each delivered through a laparoscopic port (not shown) located on the patient 1503. The laparoscopic tools 1540 may include, but are not limited to, one or more of a variety of graspers, retractors, and / or other tools for manipulation of tissue overall, dissection and cutting tools that may or may not be capable of delivering electrosurgical energy, such as paddle forceps, Maryland forceps, scissors, hooks, and the like, and ligation and suturing tools, such as staplers, vessel sealers, needle drivers, and auto-suturing devices. In some embodiments, the laparoscope 1535 and / or laparoscopic tools 1540 can be used to view and assist in the removal of one or more polyps, in other embodiments, the laparoscope 1535 and / or laparoscopic tools 1540 can be used to reposition the colon to allow for better endoscopic resection.

[0172] In certain embodiments, the robotic arms 1510 and 1515 are placed in a stored position when not in use. For example, a CELS procedure may begin with the use of a flexible instrument controlled by the first robotic arm 1510 in the initial stages or during the procedure. In some embodiments, the system may introduce the flexible instrument into the patient through the patient's natural orifice. The system may manipulate the flexible instrument using the first robotic arm of the robotic system through the natural orifice.

[0173] If pure endoscopic resection is not successful in completely treating the medical condition (e.g., a colon polyp), the system may receive an input signal via a user input device to deploy a second robotic arm 1515. In response to receiving the input signal, the system may deploy the second robotic arm 1515 of the robotic system from a stowed position to a set-up position. Deployment of additional robotic arms may occur as part of a procedure escalation (e.g., as described in section "2.B. Procedure Escalation").

[0174] The system may then manipulate the rigid instrument through the incision made in the patient using a second robotic arm 1515 of the robotic system. The system may also display feedback from at least one of the flexible and rigid instruments during the CELS procedure.

[0175] 42 is an exemplary still image captured from a video screen of an accompanying system during a simulated colorectal intervention according to an embodiment of the present disclosure. Still image 1600 includes a live feed from a laparoscope 1605 as well as a live feed from an endoscope 1610 formatted in a picture-in-picture view. Live feed from laparoscope 1605 shows a view of the colon 1615 from the patient's abdomen as well as a pair of laparoscopic end effectors 1620, each attached to a laparoscopic tool. Live feed from endoscope 1610 shows an internal view of the colon.

[0176] Advantageously, an operator (e.g., physician, assistant, etc.) can switch the view displayed on the video screen and / or the format in which the view is displayed. In other embodiments, rather than providing two different live feeds 1605 and 1610 on the video screen, a user can provide an instruction to freeze one of the live feeds 1605 and 1610 to a still image, while the other live feed 1605 and 1610 can remain live. In other embodiments, rather than providing two different live feeds 1605 and 1610, a user can select one of the display images to present a virtual view (e.g., a virtual representation of the target anatomy, which may be generated based on preoperative imaging of the target anatomy), while the other image can be one of the live feeds 1605 and 1610. In some embodiments, the virtual view can include a view of an endoscopic camera and / or instruments, or a laparoscopic camera and / or instruments, based on measured joint and encoder positions of the robotic arms and manipulators. 42, the live feed from the laparoscope 1605 is displayed prominently and may be considered the "primary" feed, while the live feed from the endoscope 1610 is displayed in a corner region and may be considered the "secondary" feed. The user may be able to switch between the primary and secondary views such that the live feed from the endoscope 1610 comprises the primary view and the live feed from the laparoscope 1605 comprises the secondary view.

[0177] The associated systems described herein can be used to provide several improved methods and workflows for associated CELS. The following sections describe some of the advantages and improvements that can be achieved using the workflows enabled by the associated systems. The workflows described herein can be performed sequentially and / or simultaneously, depending on the particular CELS procedure being performed (e.g., based on the type of procedure, patient, etc.).

[0178] A. Patient Preparation Prior to performing a CELS procedure using the associated systems disclosed herein, a physician, clinician, and / or assistant may perform several pre-operative preparation procedures. One such procedure may include patient preparation.

[0179] Patient preparation can include any step associated with preparing a patient prior to performing surgery (e.g., before an incision is made). This can include, for example, a clinician, assistant, and / or physician identifying sterile and non-sterile sites on a patient. For example, in preparation for a colonic intervention, a physician can identify sterile sites (e.g., areas of the patient's abdomen where laparoscopic ports will be placed) and non-sterile sites (e.g., the anus) and apply sterile drapes to form a sterile boundary between the sterile and non-sterile sites.

[0180] Methods for performing patient preparation can be significantly improved using the accompanying systems described herein. FIG. 43 is a flow chart illustrating an exemplary patient preparation procedure for performing CELS according to an embodiment of the present disclosure. For example, the steps of the method 1700 shown in FIG. 43 can be performed by one or more physicians, clinicians, and / or assistants. Some of the steps of the method 1700 of FIG. 43 can be performed by a processor and / or other components of a medical robotic system (e.g., one of the robot-controllable system 10, or the robotic medical systems 200, 300, 400, 1300, or 1500 discussed above) or associated systems. Certain steps of the method 1700 can also be performed by the system in response to commands received, for example, from a physician via an input device (e.g., as shown in FIGS. 29-32), or can be performed automatically by the system without intervention from a user.

[0181] 43, method 1700 begins at block 1701. At block 1705, method 1700 includes establishing a sterile boundary. In some embodiments, a processor of the system may be used to define and identify the sterile boundary. At block 1710, method 1700 includes the processor identifying sterile and non-sterile zones. In some embodiments, the processor may also identify transition zones between the sterile and non-sterile zones. In some embodiments, the different zones may be identified via a visual boundary.

[0182] At block 1715, method 1700 includes a processor maintaining the first robotic arm in the sterile zone and the second robotic arm in the non-sterile zone. Thus, the processor may prevent the robotic arm or instrument from entering an undesired zone (e.g., from the sterile zone to the non-sterile zone or vice versa). In some embodiments, the processor may identify the first robotic arm as a "sterile" robotic arm that may remain in the sterile zone and the second robotic arm as a "non-sterile" robotic arm that may remain in the non-sterile zone, thereby preventing them from entering an undesired area. In some embodiments, the processor may prevent the first robotic arm and the second robotic arm from moving into a transition zone, thereby leaving a buffer between the zones. In some embodiments, the processor may remind the user via a visual or audio cue that the robotic arm is moving into a transition or non-sterile zone. In some embodiments, the processor may prompt the user to confirm that they continue to move into the transition or non-sterile zone. Method 1700 ends at block 1720.

[0183] B. Air supply During a CELS procedure, it may be desirable to insufflate one or more regions of the patient's body. For example, the abdomen can be insufflated or inflated (e.g., via a tube that pumps air through one or more cannulas / ports placed in the abdomen). In addition, the colon can be insufflated or inflated (e.g., via a tube that pumps air through a working channel of an endoscope). In a colonic interventional procedure, it can be difficult to maintain a proper balance between the two insufflation regions. For example, abdominal insufflation can cause compression of organs (such as the colon), thereby making it difficult to visualize the colon. Insufflation of two different regions (here, the abdomen and the colon) can often compete with each other.

[0184] FIG. 44 is a flow chart illustrating an exemplary insufflation procedure for performing CELS according to an embodiment of the present disclosure. For example, the steps of method 1800 shown in FIG. 44 may be performed by one or more physicians, clinicians, and / or assistants. Some of the steps of method 1800 of FIG. 44 may be performed by a processor and / or other components of a medical robotic system (e.g., one of the robot-controllable system 10, or robotic medical systems 200, 300, 400, 1300, or 1500, discussed above) or associated systems. Certain steps of method 1800 may also be performed by the system in response to commands received, for example, from a physician via an input device (e.g., as shown in FIGS. 29-32), or may be performed automatically by the system without intervention from a user.

[0185] 44, method 1800 begins at block 1801. At block 1805, a processor manipulates a flexible instrument through a first region of the patient and manipulates a rigid instrument through a second region of the patient. In some embodiments, the flexible instrument is manipulated through a first opening (e.g., a natural orifice such as the patient's anus) and the rigid instrument is manipulated through a second opening (e.g., an artificial incision that may be made in the patient's abdomen).

[0186] At block 1810, the processor may insufflate the patient at least one of the patient first region and the patient second region. In some embodiments, the processor may insufflate both the patient first region and the patient second region.

[0187] At block 1815, the processor may optionally adjust the insufflation of the second region based on the measurement of the insufflation of the first region, or vice versa. For example, the processor may determine when sufficient insufflation of two different regions is achieved, thereby creating an appropriate balance of insufflation between the regions. In some embodiments, one or more of the tubes configured to pump air to the patient may also include a pressure sensor configured to take a measurement of the current insufflation in the corresponding region of the patient. Thus, the processor may be able to determine the insufflation in each of the first and second regions based on the measurement from the pressure sensor. In addition, the system may include a video with live footage, so that the physician can instruct the CELS system to increase the amount of inflation in any of the multiple regions in real time. The method 1800 ends at block 1820.

[0188] The improved CELS system described above advantageously allows for the integration of insufflation mechanisms and techniques into the proposed system, thereby making it easier to provide whole-patient distention management. In contrast to conventional CELS procedures, where a first clinician may have to manage distention in one area (e.g., the abdomen) and another clinician may have to manage distention in another area (e.g., the colon), using the described insufflation procedures, the CELS system can allow for dynamic user control of insufflation volume in multiple patient areas (e.g., in both the abdomen and / or colon), thereby improving endoscopic and laparoscopic visualization.

[0189] C. Imaging As previously described, feedback from one or more of the flexible and rigid instruments may be displayed on a video screen during the CELS procedure. In some cases, internal images of different anatomical and target regions (e.g., cancer sites) may be difficult to distinguish from other parts of the patient's anatomy displayed on the video screen. Visualization of these anatomical and target regions may be aided by the introduction of visual or fluorescent markers. For example, visual or fluorescent markers (such as indigo carmine solution, methylene blue, indocyanine green, or other compounds) may be injected into the patient to better identify and characterize the region.

[0190] FIG. 45 is a flow chart illustrating an exemplary imaging procedure for performing CELS according to an embodiment of the present disclosure. For example, the steps of method 1900 shown in FIG. 45 may be performed by one or more physicians, clinicians, and / or assistants. Some of the steps of method 1900 of FIG. 45 may be performed by a processor and / or other components of a medical robotic system (e.g., one of the robot-controllable system 10, or robotic medical systems 200, 300, 400, 1300, or 1500, discussed above) or associated systems. Certain steps of method 1900 may also be performed by the system in response to commands received, for example, from a physician via an input device (e.g., as shown in FIGS. 29-32), or may be performed automatically by the system without intervention from a user.

[0191] Referring to FIG. 45, the method 1900 begins at block 1901. At block 1905, the processor may introduce a marker into the target area of ​​the patient using one of the flexible and rigid instruments. In some embodiments, the marker is delivered into the patient's vein. Examples of markers include visual or fluorescent markers (e.g., indigo carmine solution, methylene blue, indocyanine green, or other compounds). At block 1910, the processor may display feedback from one of the flexible and rigid instruments, including the target area, on a video screen. Thus, the physician may be able to view the target area via the image displayed on the video screen. The method 1900 ends at block 1915.

[0192] The improved CELS system provided above can provide several benefits, such as the ability to communicate and transfer information between different treatment modalities (e.g., endoscopic and laparoscopic modalities). For example, in one embodiment, an endoscope can be used to visualize fluorescently stained areas that could previously only be viewed by a laparoscope, and vice versa, by displaying a live video stream from one or more of the endoscope and laparoscope. In another embodiment, fluorescent dye can be delivered into a vein such that both the endoscope and laparoscope can view the same stained areas. Thus, the improved CELS system makes it easier to view fluorescently stained or marked areas using both the endoscope and laparoscope. In addition to using fluorescent imaging, the system can use narrow band imaging, which can also benefit from having an integrated endoscope and / or laparoscope to identify and characterize lesions.

[0193] D. Navigation Navigation using the associated systems described herein may include driving one or more scopes and / or instruments through the patient's anatomy. Flexible endoscopes and their associated endoscopic instruments can be driven in novel ways using the associated systems described herein. In some embodiments, the flexible instruments are configured to be driven through an outer sheath, for example, of a flexible colonoscope. The flexible instruments can include one or more working channels configured to facilitate delivery of surgical instruments therethrough. For example, in a colorectal intervention, the flexible colonoscope can be viewed as the "mother" instrument, and any instruments within the colonoscope that pass through the working channels of the colonoscope can be viewed as subordinate "daughter" instruments, and such mother and daughter instruments can be navigated together.

[0194] In another embodiment, the outer sheath can be viewed as the "mother" and the colonoscope, and any instruments passing through the colonoscope can be viewed as subordinate "daughters" traveling through the outer sheath. In other words, the improved CELS system allows for navigation of the elongated members (e.g., the outer sheath, the scope, and the working instrument).

[0195] E. Integration of Flexible and Rigid Instruments The improved CELS system described herein uniquely integrates both flexible and rigid instruments. The system, as described herein, allows for navigation and control of both types of instruments, thereby enabling new procedures, including escalation of such procedures.

[0196] In one example of a procedure involving control of both flexible and rigid instruments, the flexible instruments (such as the outer sheath, inner sheath, and working instruments) can be driven by the robotic arms in an initial attempt to perform resection of the target anatomical structure. Following the initial attempt, laparoscopic ports and instruments can be provided as desired to perform resection as desired. In some embodiments, the laparoscopic ports can be approximately 3-14 mm (e.g., to accommodate a 12 mm laparoscope, 8 mm instruments, and instruments as small as 3 mm). In some embodiments, larger hand or gel ports can be used instead of or in addition to standard laparoscopic ports. When both an endoscope and laparoscopic scopes and tools are used, the improved CELS system described herein can switch control between each of the endoscope and laparoscopic scopes and tools, thereby allowing the user to control both, for example, using one or more controllers.

[0197] Advantageously, in some embodiments, the same controller (e.g., a single input device such as one or more handles 815 shown in FIG. 31 or pendant 820 shown in FIG. 32) can be used to control both endoscopic and laparoscopic scopes and tools. The single input device can be any type of controller, including a multi-DOF (e.g., 7-DOF) master controller (see, e.g., master controller 810 shown in FIG. 30) with a gimbal or gamepad type device (e.g., pendant 820). A single input device can be used to switch between control of endoscopes, flexible endoscopic instruments, laparoscopes, and rigid laparoscopic instruments.

[0198] In other embodiments, multiple controllers can be used to control both endoscopic and laparoscopic scopes and tools. An advantage of having multiple controllers is that it allows multiple users (e.g., a set of physicians) to simultaneously control different aspects of the CELS system as desired. In addition, another advantage of having multiple controllers is that the controllers can be located in different locations - for example, one controller can be at the surgeon console and another at the bedside. In one embodiment, a set of users can control rigid and flexible instruments synchronously, whereby one can control rigid instruments from a multi-DOF (e.g., 7-DOF) master at the surgeon console and the other can control flexible instruments from a bedside pendant. In another embodiment, a set of users can control rigid and flexible instruments synchronously, whereby one can control rigid instruments using one type of controller and the other can control flexible instruments using the same type of controller.

[0199] However, in other embodiments, a single user may input commands to control rigid and flexible instruments via the pendant and master controller. For example, a single user may input commands to control rigid instruments using one type of controller and commands to control flexible instruments using another type of controller. This may allow a user to switch between control of endoscopic and laparoscopic instruments by inputting commands into separate devices rather than changing the input mode of the system.

[0200] F. UI / UX In a conventional CELS procedure, one or more surgeons may view an image from the endoscope on one screen and an image from the laparoscope on a second, separate screen. The surgeon often turns his or her head back and forth to analyze the different views from the endoscope and laparoscope.

[0201] The improved CELS system described herein may enable the establishment of live video streams from either or both the laparoscope and endoscope (e.g., colonoscope) through a single video pipeline. To enable this, the system may include two independent video processors that stream video received from the endoscope and laparoscope to an image output path that may output to a tower monitor and / or an operating room monitor. Using a single CELS system to control both video streams allows the user to determine whether a given monitor should display a laparoscopic view, an endoscopic view, or both (e.g., picture-in-picture or side-by-side). Thus, the system may be configured to display feedback from the laparoscope and endoscope in a picture-in-picture and / or side-by-side view.

[0202] The improved CELS system can be configured to switch between laparoscopic and endoscopic views on any connected display unit, whereby the display unit can be either a tower monitor, an operating room monitor, a physician console display, or an alternative third party bedside visualization unit. In some embodiments, the processor can identify one of the endoscopic feedback and the laparoscopic feedback as the primary view and the other as the secondary view. The processor can then switch between the primary and secondary views based on input received from the user.

[0203] Switching the view of any of the connected display units can include (i) switching between full screen laparoscopic and endoscopic views on a single monitor, (ii) switching between full screen laparoscopic or endoscopic views and picture-in-picture or side-by-side (or any combination thereof), (iii) switching which monitor is displaying what content (e.g., switching monitor A from laparoscopic to endoscopic view and monitor B from endoscopic to laparoscopic view, or vice versa). Thus, the system may allow any combination of views to be displayed independently on each display unit.

[0204] G. Exemplary Benefits and Improvements The above-described system using a robotic arm to control endoscopic instruments, laparoscopic instruments, or both provides several benefits. One benefit is allowing robotic control of one or both tool types (flexible / endoscopic and rigid / laparoscopic), which reduces the need to have multiple physicians and personnel to perform a particular procedure. For example, rather than having three of four physicians remain stationary while the fourth manipulates the instruments, the robotic arm can hold the static instruments stationary. In such a case, the flexible instruments can be manipulated to polysnare the polyps while the laparoscopic instruments, laparoscopic instruments, laparoscope, and colonoscope remain stationary.

[0205] In addition, the system can control instruments through the robotic arm, allowing motion and sensor-based determination of the location of the instrument shaft and tip. This information can allow the user to view a graphics-based rendering of the location and position of the robotic arm and instrument at all times. This can include 3D volume renderings of the robot and instruments, 2D line drawings of the robot and instruments, or visual overlays of flexible instruments from within a rigid laparoscope (and vice versa, as shown in FIG. 36).

[0206] 4. Mounting system and terminology Implementations disclosed herein provide systems, methods, and devices for performing associated medical procedures.

[0207] It should be noted that, as used herein, the terms "couple," "coupled," "coupled," or other variations of the word couple, may indicate either an indirect connection or a direct connection. For example, when a first component is "coupled" to a second component, the first component may be either indirectly connected to the second component through another component, or directly connected to the second component.

[0208] The functions associated with the systems, methods, and workflows for performing the associated procedures described herein may be stored as one or more instructions in a processor-readable medium or computer-readable medium. The term "computer-readable medium" refers to any available medium that can be accessed by a computer or processor. By way of example and without limitation, such media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disk read-only memory (CD-ROM), or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. It should be noted that computer-readable media may be tangible and non-transitory. As used herein, the term "code" may refer to software, instructions, code, or data that is executable by a computing device or processor.

[0209] The methods disclosed herein include one or more steps or acts for achieving the described method. Method steps and / or acts may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or acts is required for the proper operation of the described method, the order and / or use of specific steps and / or acts may be modified without departing from the scope of the claims.

[0210] As used herein, the term "plurality" refers to two or more. For example, a plurality of components refers to two or more components. The term "determining" encompasses a wide variety of acts, and thus "determining" can include calculating, computing, processing, calculating, investigating, looking up (e.g., looking in a table, database, or another data structure), ascertaining, and the like. "Determining" can also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. "Determining" can also include resolving, selecting, choosing, establishing, and the like.

[0211] The phrase "based on" does not mean "based only on," unless expressly specified otherwise. In other words, the phrase "based on" describes both "based only on" and "based at least on."

[0212] The foregoing description of the disclosed implementations is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these implementations will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the scope of the present invention. For example, those skilled in the art will recognize that many corresponding alternative and equivalent structural details may be employed, such as equivalent methods of fastening, mounting, coupling, or engaging tool components, equivalent mechanisms for producing specific actuation motions, and equivalent mechanisms for delivering electrical energy. Thus, the present invention is not intended to be limited to the implementations shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0213] [Embodiment] (1) A robot system comprising: A flexible instrument; A rigid instrument; a first robotic arm configured to manipulate the flexible instrument; a second robotic arm configured to manipulate the rigid instrument; and a display configured to display visual data from the flexible instrument and the rigid instrument. (2) the flexible instrument includes a flexible scope; the rigid instrument comprises a rigid scope; the display includes a video screen; The system further comprises: At least one processor; at least one computer readable memory in communication with the at least one processor and configured to cause the one or more processors to: receiving first visual data from the flexible scope; receiving second visual data from the rigid scope; generating a signal based on the first visual data and the second visual data; and at least one computer-readable memory having stored thereon computer-executable instructions for causing the video screen to render a video sequence from at least one of the flexible scope and the rigid scope. (3) The system of embodiment 2, wherein the video screen is further configured to render the video sequences from the flexible scope and the rigid scope in a picture-in-picture view. (4) The system of embodiment 2, wherein the video screen is further configured to display the video sequences from the flexible scope and the rigid scope in a side-by-side view. (5) the first visual data from the flexible scope comprises a primary view and the second visual data from the rigid scope comprises a secondary view; 3. The system of claim 2, wherein the video screen is further configured to switch between rendering the primary view and the secondary view.

[0214] (6) One or more processors; 2. The system of claim 1, further comprising: at least one computer-readable memory in communication with the one or more processors and storing computer-executable instructions for causing the one or more processors to identify sterile and non-sterile zones. (7) The system of claim 6, further comprising a drape positioned between the sterile zone and the non-sterile zone. (8) The computer-executable instructions cause the one or more processors to: maintaining one of the first robotic arm or the second robotic arm within the sterilization zone; 7. The system of claim 6, further comprising: maintaining the other of the first robotic arm and the second robotic arm within the non-sterile zone. (9) The computer-executable instructions cause the one or more processors to: identifying a transition zone between the sterile zone and the non-sterile zone; 9. The system of claim 8, further comprising: preventing the first robotic arm and the second robotic arm from moving into the transition zone. (10) One or more processors; at least one computer readable memory in communication with the one or more processors and configured to provide the one or more processors with: manipulating the flexible instrument through a first area of ​​a patient; The system of embodiment 1, further comprising at least one computer-readable memory having stored therein computer-executable instructions for manipulating the rigid instrument through a second area of ​​the patient.

[0215] (11) The system of embodiment 10, wherein the flexible instrument is configured to be manipulated through a natural orifice and the rigid instrument is configured to be manipulated through an artificial incision. (12) The computer-executable instructions cause the one or more processors to: 11. The system of claim 10, further comprising controlling insufflation of the patient at the first region of the patient and at the second region of the patient. (13) The computer-executable instructions cause the one or more processors to: 13. The system of claim 12, further comprising adjusting the insufflation of the second region based on the measurement of the insufflation of the first region. (14) A bed, an adjustable arm support coupled to the bed; 2. The system of claim 1, wherein the first robotic arm and the second robotic arm are positioned on the adjustable arm support. (15) a pendant configured to receive input to manipulate the flexible instrument using the first robotic arm; and 2. The system of claim 1, further comprising: a master controller configured to receive input for manipulating the rigid instrument using the second robotic arm.

[0216] (16) A non-transitory computer-readable storage medium that, when executed, causes at least one computing device to: manipulating a flexible instrument using a first robotic arm of a robotic system; manipulating a rigid instrument using a second robotic arm of the robotic system; and displaying feedback from the flexible instrument; and and displaying feedback from the rigid instrument. (17) A surgical method comprising: manipulating a flexible instrument using a first robotic arm of a robotic system; manipulating a rigid instrument using a second robotic arm of the robotic system; and displaying feedback from the flexible instrument; and and displaying feedback from the rigid instrument. (18) The flexible instrument includes a flexible scope, the rigid instrument comprises a rigid scope; 18. The method of claim 17, wherein at least one of the feedback from the flexible scope and the feedback from the rigid scope is displayed on a video screen. (19) The method of embodiment 18, wherein the feedback from the flexible scope and the feedback from the rigid scope are displayed in a picture-in-picture view. (20) The method of embodiment 18, wherein the feedback from the flexible scope and the feedback from the rigid scope are displayed in a side-by-side view.

[0217] (21) The method of embodiment 18, wherein one of the feedback from the flexible scope and the rigid scope includes a primary view, and the other of the feedback from the flexible scope and the rigid scope includes a secondary view. (22) The method of embodiment 21, further comprising switching between the primary view and the secondary view such that the feedback from the flexible scope includes the primary view and the feedback from the rigid scope includes the secondary view. 23. The method of claim 17, further comprising identifying a sterile zone and a non-sterile zone. 24. The method of claim 23, wherein a drape is positioned between the sterile zone and the non-sterile zone. (25) maintaining one of the first robotic arm or the second robotic arm within the sterilization zone; 24. The method of claim 23, further comprising maintaining the other of the first robotic arm and the second robotic arm within the non-sterile zone.

[0218] 26. The method of claim 25, further comprising identifying a transition zone between the sterile zone and the non-sterile zone. (27) The method of embodiment 26, further comprising preventing the first robotic arm and the second robotic arm from moving into the transition zone. (28) The method of embodiment 17, wherein the flexible instrument is manipulated through a first area of ​​the patient and the rigid instrument is manipulated through a second area of ​​the patient. (29) The method of embodiment 28, wherein the flexible instrument is manipulated through a first opening and the rigid instrument is manipulated through a second opening. (30) The method of embodiment 29, wherein the first opening comprises a natural opening and the second opening comprises an artificial incision.

[0219] 31. The method of claim 30, wherein the natural orifice includes the patient's anus and the artificial incision is made in the patient's abdomen. (32) The method of embodiment 28, further comprising insufflating the patient at one of the first region of the patient and the second region of the patient. (33) The method of embodiment 32, further comprising insufflating both the first region of the patient and the second region of the patient. (34) The method of embodiment 33, further comprising adjusting the insufflation of the second region based on the measurement of the insufflation of the first region. (35) The method of embodiment 17, wherein the first robotic arm and the second robotic arm are positioned on adjustable arm supports.

[0220] 36. The method of claim 35, wherein the adjustable arm support is connected to a bed. (37) The method of claim 36, wherein the bed includes a pair of adjustable arm supports, each of the pair of adjustable arm supports having one or more robotic arms attached thereto. 38. The method of claim 17, wherein the flexible instrument is configured to be driven through an outer sheath. (39) The method of embodiment 38, wherein the flexible instrument comprises one or more working channels configured to facilitate delivery of surgical instruments therethrough. (40) The method of embodiment 39, wherein the surgical instrument comprises a polyp snare.

[0221] (41) The method of embodiment 17, further comprising introducing a marker into a target area of ​​the patient using one of the flexible instrument and the rigid instrument. (42) The method of embodiment 41, wherein the marker is delivered to the patient's vein. (43) manipulating the flexible instrument using the first robotic arm in response to input received via a pendant; 18. The method of embodiment 17, wherein manipulating the rigid instrument using the second robotic arm is performed in response to input received via a master controller. (44) The method of embodiment 43, wherein the input received via the pendant and the input received via the master controller are received from a single user. (45) manipulating the flexible instrument using the first robotic arm in response to input received via a master controller; 18. The method of embodiment 17, wherein manipulating the rigid instrument using the second robotic arm is performed in response to input received via the master controller.

[0222] (46) The method of embodiment 17, further comprising performing a combined endoscopic and laparoscopic surgery (CELS) procedure using the flexible instrument and the rigid instrument. (47) The method of embodiment 46, wherein the CELS treatment comprises performing a colorectal intervention on the patient. (48) A surgical method comprising: manipulating a flexible instrument using a first robotic arm of a robotic system; manipulating a rigid instrument using a second robotic arm of the robotic system; and displaying a first feedback from the flexible instrument on a viewscreen as a primary view; and displaying a second feedback from the rigid instrument on the same viewscreen as a secondary view. (49) The flexible instrument includes a flexible scope, the rigid instrument comprises a rigid scope; 49. The method of embodiment 48, wherein the first feedback from the flexible scope and the second feedback from the rigid scope are displayed in a picture-in-picture view. (50) The flexible instrument includes a flexible scope; the rigid instrument comprises a rigid scope; 49. The method of claim 48, wherein the first feedback from the flexible scope and the second feedback from the rigid scope are displayed in a side-by-side view.

[0223] (51) The method of embodiment 48, wherein one of the first feedback and the second feedback includes a primary view, and the other of the first feedback and the second feedback includes a secondary view. (52) The method of embodiment 51, further comprising switching the primary view and the secondary view. (53) The method of embodiment 48, further comprising identifying sterile and non-sterile zones. (54) maintaining one of the first robotic arm or the second robotic arm within the sterilization zone; 54. The method of embodiment 53, further comprising maintaining the other of the first robotic arm and the second robotic arm within the non-sterile zone. 55. The method of claim 54, further comprising identifying a transition zone between the sterile zone and the non-sterile zone.

[0224] (56) The method of embodiment 55, further comprising preventing the first robotic arm and the second robotic arm from moving into the transition zone. (57) The method of embodiment 48, wherein the flexible instrument is manipulated through a first area of ​​the patient and the rigid instrument is manipulated through a second area of ​​the patient. (58) The method of embodiment 57, wherein the flexible instrument is manipulated through a first opening and the rigid instrument is manipulated through a second opening. (59) The method of embodiment 58, wherein the first opening comprises a natural opening and the second opening comprises an artificial incision. (60) The method of embodiment 59, wherein the natural orifice includes the patient's anus and the artificial incision is made in the patient's abdomen.

[0225] (61) The method of embodiment 57, further comprising insufflating the patient at one of the first region of the patient and the second region of the patient. (62) The method of embodiment 61, further comprising insufflating both the first area of ​​the patient and the second area of ​​the patient. (63) The method of embodiment 62, further comprising adjusting the insufflation of the second region based on the measurement of the insufflation of the first region. (64) The method of embodiment 48, further comprising introducing a marker into a target area of ​​the patient using one of the flexible instrument and the rigid instrument. (65) The method of embodiment 64, wherein the marker is delivered to the patient's vein.

[0226] (66) Manipulating the flexible instrument using the first robotic arm is in response to input received via a pendant; 49. The method of embodiment 48, wherein manipulating the rigid instrument using the second robotic arm is performed in response to input received via a master controller. (67) The method of embodiment 66, wherein the input received via the pendant and the input received via the master controller are received from a single user. (68) Manipulating the flexible instrument using the first robotic arm is in response to input received via a master controller; 49. The method of embodiment 48, wherein manipulating the rigid instrument using the second robotic arm is performed in response to input received via the master controller. (69) The method of embodiment 48, further comprising performing a combined endoscopic and laparoscopic surgery (CELS) procedure using the flexible instrument and the rigid instrument. (70) The method of embodiment 69, wherein the CELS treatment comprises performing a colorectal intervention on the patient.

[0227] (71) A surgical method comprising: manipulating the flexible instrument using a first robotic arm of a robotic system through a natural orifice of the patient; manipulating a rigid instrument through an incision made in the patient using a second robotic arm of the robotic system; and displaying feedback information from the flexible instrument and the rigid instrument. (72) The flexible instrument includes a flexible scope, the rigid instrument comprises a rigid scope; 72. The method of claim 71, wherein the feedback information from the flexible scope and the rigid scope is displayed on a video screen. (73) The method of embodiment 72, wherein the first feedback from the flexible scope and the second feedback from the rigid scope are displayed in a picture-in-picture view. (74) The method of embodiment 72, wherein the first feedback from the flexible scope and the second feedback from the rigid scope are displayed in a side-by-side view. (75) The method of embodiment 71, wherein one of the first feedback and the second feedback includes a primary view, and the other of the first feedback and the second feedback includes a secondary view.

[0228] (76) The method of embodiment 75, further comprising switching the primary view and the secondary view. (77) The method of embodiment 71, further comprising identifying sterile and non-sterile zones. (78) maintaining one of the first robotic arm or the second robotic arm within the sterilization zone; 78. The method of embodiment 77, further comprising maintaining the other of the first robotic arm and the second robotic arm within the non-sterile zone. 79. The method of claim 78, further comprising identifying a transition zone between the sterile zone and the non-sterile zone. (80) The method of embodiment 79, further comprising preventing the first robotic arm and the second robotic arm from moving into the transition zone.

[0229] 81. The method of claim 71, wherein the natural orifice includes the patient's anus and the incision is made in the patient's abdomen. (82) The method of embodiment 71, wherein the flexible instrument is manipulated through a first area of ​​the patient and the rigid instrument is manipulated through a second area of ​​the patient. (83) The method of embodiment 82, further comprising insufflating the patient at one of the first region of the patient and the second region of the patient. (84) The method of embodiment 83, further comprising insufflating both the first region of the patient and the second region of the patient. (85) The method of embodiment 84, further comprising adjusting the insufflation of the second region based on the measurement of the insufflation of the first region.

[0230] (86) The method of embodiment 71, further comprising introducing a marker into a target area of ​​the patient using one of the flexible instrument and the rigid instrument. (87) The method of embodiment 86, wherein the marker is delivered to the patient's vein. (88) Manipulating the flexible instrument using the first robotic arm is in response to input received via a pendant; 72. The method of embodiment 71, wherein manipulating the rigid instrument using the second robotic arm is performed in response to input received via a master controller. (89) The method of embodiment 88, wherein the input received via the pendant and the input received via the master controller are received from a single user. (90) Manipulating the flexible instrument using the first robotic arm is in response to input received via a master controller; 72. The method of embodiment 71, wherein manipulating the rigid instrument using the second robotic arm is performed in response to input received via the master controller.

[0231] (91) The method of embodiment 71, further comprising performing a combined endoscopic and laparoscopic surgery (CELS) procedure using the flexible instrument and the rigid instrument. (92) A surgical method comprising: Introducing a flexible instrument into the patient through a natural orifice of the patient; manipulating the flexible instrument using a first robotic arm of a robotic system through the natural orifice; deploying a second robotic arm of the robotic system from a stowed position to a setup position in response to receiving an input signal via a user input device; manipulating a rigid instrument using the second robotic arm of the robotic system through an incision made in the patient; and displaying feedback from at least one of the flexible instrument and the rigid instrument. (93) The method of embodiment 92, wherein the user input device includes at least one of a master controller and a pendant.

Claims

1. 1. A robot system comprising: a flexible instrument including a flexible scope configured to be driven through an outer sheath, the flexible instrument comprising one or more working channels configured to facilitate delivery of a surgical instrument therethrough, the surgical instrument comprising a polyp snare; A rigid instrument; a first robotic arm configured to manipulate the flexible instrument; a second robotic arm configured to manipulate the rigid instrument; and a display configured to display visual data from the flexible instrument and the rigid instrument.

2. the rigid instrument comprises a rigid scope; the display includes a video screen; The robot system comprises: At least one processor; At least one computer readable memory in communication with the at least one processor and configured to provide the at least one processor with: receiving first visual data from the flexible scope; receiving second visual data from the rigid scope; generating a signal based on the first visual data and the second visual data; 2. The system of claim 1, further comprising: at least one computer-readable memory having stored thereon computer-executable instructions for: transmitting the signal to the video screen, the signal configured to cause the video screen to render a video sequence from at least one of the flexible scope and the rigid scope.

3. The system of claim 2 , wherein the video screen is further configured to render the video sequences from the flexible scope and the rigid scope in a picture-in-picture view.

4. The system of claim 2 , wherein the video screen is further configured to display the video sequences from the flexible scope and the rigid scope in a side-by-side view.

5. the first visual data from the flexible scope comprises a primary view, and the second visual data from the rigid scope comprises a secondary view; The system of claim 2 , wherein the video screen is further configured to switch between rendering the primary view and the secondary view.

6. one or more processors; 10. The system of claim 1, further comprising: at least one computer readable memory in communication with the one or more processors and storing computer executable instructions for causing the one or more processors to identify sterile and non-sterile zones.

7. The system of claim 6 further comprising a drape positioned between the sterile zone and the non-sterile zone.

8. The computer executable instructions cause the one or more processors to: maintaining one of the first robotic arm or the second robotic arm within the sterilization zone; 7. The system of claim 6, further comprising: maintaining the other of the first robotic arm and the second robotic arm within the non-sterile zone.

9. The computer-executable instructions cause the one or more processors to: identifying a transition zone between the sterile zone and the non-sterile zone; 10. The system of claim 8, further comprising: preventing the first robotic arm and the second robotic arm from moving into the transition zone.

10. one or more processors; at least one computer readable memory in communication with the one or more processors and configured to provide the one or more processors with: manipulating the flexible instrument through a first area of ​​a patient; 10. The system of claim 1, further comprising: at least one computer readable memory having computer executable instructions stored thereon for causing the rigid instrument to maneuver through a second area of ​​the patient.

11. The system of claim 10 , wherein the flexible instrument is configured to be manipulated through a natural orifice and the rigid instrument is configured to be manipulated through an artificial incision.

12. The computer executable instructions cause the one or more processors to: The system of claim 10 , further comprising controlling insufflation of the patient at the first region of the patient and at the second region of the patient.

13. The computer executable instructions cause the one or more processors to: The system of claim 12 , further comprising adjusting the airflow of the second region based on the measurement of the airflow of the first region.

14. A bed and an adjustable arm support coupled to the bed; The system of claim 1 , wherein the first robotic arm and the second robotic arm are positioned on the adjustable arm support.

15. a pendant configured to receive input to manipulate the flexible instrument using the first robotic arm; 10. The system of claim 1, further comprising: a master controller configured to receive input for manipulating the rigid instrument using the second robotic arm.

Images