Systems, methods, and workflows for associated procedures

Abstract

To provide a system, a method, and a workflow that can be used in a robot capable of performing various medical treatments including both low invasive treatment using an abdominoscope or the like and non-invasive treatment using an endoscope or the like.SOLUTION: A method for concomitant treatment includes: operating a flexible tool 13 by using a first robot arm 12 of a robot-controllable system 10 of a cart base; operating a rigid tool by using a second robot arm of the robot system; displaying a feedback from the flexible tool; and displaying a feedback from the rigid tool.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Provisional Application No. 62 / 831,064, filed Apr. 8, 2019, which is hereby incorporated by reference in its entirety.

[0002] (Field of the Invention) The systems and methods disclosed herein relate to medical procedures, more particularly, systems and methods for performing attendant procedures.

Background Art

[0003] To control the insertion and / or operation of one or more medical instruments, robotic medical systems can be used to perform various medical procedures. In certain medical conditions, two or more medical procedures may be performed to fully treat the medical condition. A robot - controllable medical system can include one or more robotic arms or any other instrument positioning device. A robotic medical system can also include a controller used to control the positioning of the instrument between respective procedures via the operation of the robotic arm and / or instrument positioning device.

Summary of the Invention

Means for Solving the Problems

[0004] The systems, methods, and devices of the present disclosure each have several innovative aspects, and no single one of them alone encompasses the desirable attributes disclosed herein.

[0005] In one aspect, a surgical method is provided that includes operating a flexible instrument using a first robotic arm of a robotic system, operating a rigid instrument using a second robotic arm of the robotic system, displaying feedback from the flexible instrument, and displaying feedback from the rigid instrument.

[0006] In another embodiment, a surgical method is provided, which includes manipulating a flexible instrument using a first robotic arm of a robotic system; manipulating a rigid instrument using a second robotic arm of a robotic system; displaying a first feedback from the flexible instrument as a primary view on a view screen; and displaying a second feedback from the rigid instrument as a secondary view on the same view screen.

[0007] In yet another embodiment, a surgical method is provided which includes manipulating a flexible instrument using a first robotic arm of a robotic system through the patient's natural orifice, manipulating a rigid instrument using a second robotic arm of a robotic system through an incision formed in the patient, and displaying feedback information from the flexible and rigid instruments.

[0008] In yet another embodiment, a surgical method is provided which includes introducing a flexible instrument into the patient through the patient's natural orifice; manipulating the flexible instrument using a first robotic arm of a robotic system through the natural orifice; deploying a second robotic arm of a robotic system from a stowed position to a setup position in response to receiving an input signal via a user input device; manipulating a rigid instrument using the second robotic arm of a robotic system through an incision formed in the patient; and displaying feedback from at least one of the flexible instrument and the rigid instrument.

[0009] In yet another embodiment, a robotic system is provided, the robotic system including a flexible instrument, a rigid instrument, a first robotic arm configured to operate the flexible instrument, a second robotic arm configured to operate the rigid instrument, and a display configured to display field-of-view data from the flexible instrument and the rigid instrument.

[0010] In yet another embodiment, a non-temporary computer-readable storage medium storing instructions is provided, which, when executed, causes at least one computing device to operate a flexible device using a first robotic arm of the robotic system, operate a rigid device using a second robotic arm of the robotic system, display feedback from the flexible device, and display feedback from the rigid device. [Brief explanation of the drawing]

[0011] The disclosed embodiments will be described below in conjunction with the attached drawings, and the disclosed embodiments will be illustrative but not limited to them, and similar designations will indicate similar elements. [Figure 1] This is an embodiment of a cart-based robotic system positioned for diagnostic and / or therapeutic bronchoscopy procedures. [Figure 2] This is a further embodiment of the robot system shown in Figure 1. [Figure 3] This is an embodiment of the robotic system shown in Figure 1, which is positioned for ureteroscopy. [Figure 4] This is an embodiment of the robotic system shown in Figure 1, which is deployed for vascular procedures. [Figure 5] This is an embodiment of a table-based robotic system positioned for bronchoscopy procedures. [Figure 6] This is an alternative diagram of the robot system shown in Figure 5. [Figure 7] This is an exemplary system configured to house a robotic arm. [Figure 8] This is an embodiment of a table-based robotic system configured for ureteroscopy procedures. [Figure 9] This is an embodiment of a table-based robotic system configured for laparoscopic procedures. [Figure 10] Figures 5 to 9 show embodiments of a table-based robot system with pitch or tilt adjustment. [Figure 11]Provide a detailed illustration of the interface between the table and columns of the table-based robot system of FIGS. 5-10. [Figure 12] Show an alternative embodiment of the table-based robot system. [Figure 13] Show an end view of the table-based robot system of FIG. 12. [Figure 14] Show an end view of the table-based robot system with a robot arm attached. [Figure 15] Show an exemplary instrument driver. [Figure 16] Show an exemplary medical instrument with a pair of instrument drivers. [Figure 17] Show an alternative design of the instrument driver and instrument where the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument. [Figure 18] Show an instrument having an instrument base insertion architecture. [Figure 19] Show an exemplary controller. [Figure 20] Show a block diagram of a position identification system for estimating the position of one or more elements of the robot system of FIGS. 1-~10, such as the position of the instrument of FIGS. 16-18, according to an exemplary embodiment. [Figure 21] Show an embodiment of a bed-based robot system configured to perform an attendant procedure according to an aspect of the present disclosure. [Figure 22] Show another embodiment of a bed-based robot system configured to perform an attendant procedure according to an aspect of the present disclosure. [Figure 23] Show yet another embodiment of a robot system configured to perform a non-attendant procedure according to an aspect of the present disclosure. [Figure 24] Show two configurations of another embodiment of a bed-based robot system configured to perform an attendant procedure according to an aspect of the present disclosure. [Figure 25] Show two configurations of another embodiment of a bed-based robot system configured to perform an attendant procedure according to an aspect of the present disclosure. [Figure 26] A flowchart showing an exemplary method operable by a robotic system or a component thereof for performing an attendant medical procedure according to an aspect of the present disclosure. [Figure 27A] Provided is a flowchart showing another exemplary method operable by a robotic system or a component thereof for performing an attendant endoscopic and thoracoscopic procedure according to an aspect of the present disclosure. [Figure 27B] Provided is a flowchart showing another exemplary method operable by a robotic system or a component thereof for performing an attendant endoscopic and thoracoscopic procedure according to an aspect of the present disclosure. [Figure 28] A flowchart showing an exemplary method operable by a robotic system or a component thereof for performing an attendant medical procedure including treatment escalation according to an aspect of the present disclosure. [Figure 29] An exemplary console including one or more types of interfaces for controlling a robotic arm according to an aspect of the present disclosure is shown. [Figure 30] An enlarged view of a controller shown in FIG. 29 according to an aspect of the present disclosure is shown. [Figure 31] An enlarged view of one of the handles shown in FIGS. 29 and 30 according to an aspect of the present disclosure is shown. [Figure 32] An enlarged view of a pendant shown in FIG. 29 according to an aspect of the present disclosure is shown. [Figure 33] A flowchart showing an exemplary method operable by a robotic system or a component thereof for performing an attendant medical procedure via a single user interface according to an aspect of the present disclosure. [Figure 34] An exemplary diagram that can be displayed by a viewer during an attendant medical procedure according to an aspect of the present disclosure. [Figure 35] An exemplary diagram that can be displayed by a viewer during an attendant medical procedure according to an aspect of the present disclosure. [Figure 36]This is another exemplary figure that may be displayed by a viewer during an incidental medical procedure relating to an aspect of this disclosure. [Figure 37] This flowchart illustrates an exemplary method, operable by a robotic system or its components, for switching displayed images while performing an associated medical procedure, according to an aspect of the present disclosure. [Figure 38A] This is a flowchart illustrating an exemplary workflow for performing combined endoscopic and laparoscopic surgery (CELS) according to aspects of this disclosure. [Figure 38B] This is a flowchart illustrating an exemplary workflow for performing combined endoscopic and laparoscopic surgery (CELS) according to aspects of this disclosure. [Figure 39] This document describes one embodiment of a bed-based robotic system configured to perform ancillary procedures as described in this disclosure. [Figure 40] This flowchart shows another exemplary workflow for performing CELS according to the aspect of this disclosure. [Figure 41] Another embodiment of a robotic system configured to perform incidental actions according to aspects of this disclosure is shown. [Figure 42] These are exemplary still images captured from a video screen of a system associated with a colorectal intervention, according to an aspect of this disclosure. [Figure 43] This flowchart shows exemplary patient preparation procedures for performing CELS according to aspects of this disclosure. [Figure 44] This is a flowchart illustrating an exemplary air supply procedure for performing CELS according to an aspect of this disclosure. [Figure 45] This flowchart shows an exemplary imaging procedure for performing CELS according to an aspect of this disclosure. [Modes for carrying out the invention]

[0012] 1.Overview Aspects of this disclosure can be integrated into a medical system usable with a robot capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy and non-invasive procedures such as endoscopy. Among endoscopic procedures, the system can perform bronchoscopy, ureteroscopy, gastroscopy, and the like.

[0013] In addition to performing a wide range of procedures, the system can offer additional benefits to assist physicians, such as enhanced imaging and guidance. Furthermore, the system can provide physicians with the ability to perform procedures from an ergonomic position without requiring awkward arm movements and postures. Moreover, the system can provide physicians with improved ease of use, allowing one or more of the system's instruments to be controlled by a single user.

[0014] Various embodiments are described below, along with the drawings, for illustrative purposes. It should be understood that many other implementations of the disclosed concepts are possible, and various advantages may be achieved in the disclosed implementations. Headings are included herein for reference and to help locate the various sections. These headings are not intended to limit the scope of the concepts described therein. Such concepts may be applicable throughout this specification.

[0015] A. Robot System - Cart Robot-operable medical systems can be configured in various ways depending on the specific procedure. Figure 1 shows an embodiment of a cart-based robot-controllable system 10 positioned for a diagnostic and / or therapeutic bronchoscopy procedure. During a bronchoscopy, the system 10 may include a cart 11 having one or more robotic arms 12 for delivering medical instruments, such as a maneuverable endoscope 13 which may be a bronchoscope specifically designed for bronchoscopy, to a natural orifice access point for delivering the diagnostic and / or therapeutic instrument (i.e., the patient's mouth positioned on a table in this embodiment). As shown, the cart 11 can be positioned close to the patient's upper torso to provide access to the access point. Similarly, the robotic arms 12 can be actuated to position the bronchoscope relative to the access point. The arrangement in Figure 1 can also be used when performing gastrointestinal (GI) procedures using a gastroscopy (a specialized endoscope for GI procedures). Figure 2 draws an exemplary embodiment of the cart in more detail.

[0016] Continuing to refer to Figure 1, once the cart 11 is properly positioned, the robotic arm 12 can insert the maneuverable endoscope 13 into the patient robotically, manually, or a combination thereof. As shown, the maneuverable endoscope 13 may include at least two nesting parts, such as an inner leader portion and an outer sheath portion, each portion coupled to a separate instrument driver (also called an instrument drive mechanism (IDM)) from a set of instrument drivers 28, each instrument driver coupled to the distal end of an individual robotic arm. This linear arrangement of the instrument drivers 28, which facilitates the coaxial alignment of the leader portion with the sheath portion, creates a “virtual rail” 29 that can be repositioned in space by manipulating one or more robotic arms 12 at different angles and / or positions. The virtual rail described herein is shown in the figure using dashed lines, and therefore the dashed lines do not indicate the physical structure of the system. Translation of the instrument drivers 28 along the virtual rail 29 causes the inner leader portion to nest with the outer sheath portion, or moves the endoscope 13 forward or backward from the patient. The angle of the virtual rail 29 may be adjusted, translated, and pivoted based on clinical use or physician preference. For example, in bronchoscopy, the angle and position of the virtual rail 29 as shown represent a compromise that provides physician access to the endoscope 13 while minimizing friction caused by bending the endoscope 13 into the patient's mouth.

[0017] The endoscope 13 may be directed downwards into the patient's trachea and lungs after insertion using precise commands from the robotic system until it reaches the target destination or surgical site. To enhance navigation through the patient's lung network and / or reach the desired target, the endoscope 13 may be maneuvered to extend the inner leader portion in a nested manner from the outer sheath portion, thereby obtaining increased articulation and a larger bending radius. The use of a separate instrument driver 28 also allows the leader portion and the sheath portion to be driven independently of each other.

[0018] For example, the endoscope 13 may be directed to deliver a biopsy needle to a target, such as a lesion or nodule in the patient's lung. The needle may be deployed down the working channel along the length of the endoscope to obtain a tissue sample to be analyzed by a pathologist. Depending on the pathological results, additional tools may be deployed down the working channel of the endoscope for further biopsies. After identifying the nodule as malignant, the endoscope 13 may endoscopically deliver instruments to excise the potentially cancerous tissue. In some cases, diagnostic and therapeutic procedures may be delivered in separate steps. In these situations, the endoscope 13 may also be used to deliver a criterion to “mark” the location of the target nodule. In other examples, diagnostic and therapeutic procedures may be delivered during the same procedure.

[0019] System 10 may also include a movable tower 30 connected to the cart 11 via support cables, which can provide support for control, electronics, fluid mechanics, optics, sensors, and / or power to the cart 11. Placing such functions within the tower 30 allows for a smaller form factor cart 11 that can be more easily adjusted and / or repositioned by the operating physician and their staff. Additionally, the separation of functions between the cart / table and the support tower 30 reduces clutter in the operating room and facilitates improved clinical workflow. The cart 11 may be placed close to the patient, while the tower 30 may be stored in a separate location so as not to interfere during the procedure.

[0020] In supporting the robotic system described above, the tower 30 may include components of a computer-based control system that store computer program instructions in a non-temporary computer-readable storage medium, such as a persistent magnetic memory drive or a solid-state drive. The execution of these instructions may control the entire system or its subsystems, whether the execution takes place within the tower 30 or in the cart 11. For example, when executed by the processor of the computer system, the instructions may cause components of the robotic system to actuate associated carriages and arm mounts, actuate a robotic arm, or control a medical device. For example, in response to receiving a control signal, motors in the joints of the robotic arm may position the arm in a particular posture.

[0021] The tower 30 may also include a pump, flow meter, valve control, and / or fluid access to provide controlled irrigation and suction functions to a system that can be deployed through the endoscope 13. These components may also be controlled using the computer system of the tower 30. In some embodiments, the irrigation and suction capabilities may be delivered directly to the endoscope 13 through separate cables.

[0022] The tower 30 may include voltage and surge protectors designed to provide filtered and protected power to the cart 11, thereby avoiding the need to place power transformers and other auxiliary power components within the cart 11, making the cart 11 smaller and more portable.

[0023] Tower 30 may also include support equipment for sensors deployed throughout the robotic system 10. For example, Tower 30 may include optoelectronic equipment for detecting, receiving, and processing data received from optical sensors or cameras through the robotic system 10. In combination with a control system, such optoelectronic equipment may be used to generate real-time images for display in any number of consoles located throughout the system, including within Tower 30. Similarly, Tower 30 may also include an electronic subsystem for receiving and processing signals from deployed electromagnetic (EM) sensors. Tower 30 may also be used to house and position an EM field generator for detection by EM sensors within or on a medical device.

[0024] Tower 30 may also include console 31, in addition to other consoles available in the rest of the system, such as a console mounted on top of a cart. Console 31 may include a user interface and display screen, such as a touchscreen, for the operator, a physician. Consoles within System 10 are generally designed to provide both pre-operative and real-time information of the procedure, as well as robot control, and navigation and positioning information for the endoscope 13. If console 31 is not the only console available to the physician, console 31 can be used by a second operator, such as a nurse, to monitor the patient's health or life and the operation of the system, and to provide procedure-specific data, such as navigation and positioning information. In other embodiments, console 30 is housed in a separate body from Tower 30.

[0025] The tower 30 may be connected to the cart 11 and endoscope 13 via one or more cables or connectors (not shown). In some embodiments, the support functions from the tower 30 are provided to the cart 11 through a single cable, which simplifies and organizes the operating room. In other embodiments, specific functions may be connected by separate wiring and connectors. For example, power may be supplied to the cart through a single power cable, while support for control, optics, fluid mechanics, and / or navigation may be provided through separate cables.

[0026] Figure 2 provides a detailed diagram of an embodiment of a cart from the cart-based robot-controllable system shown in Figure 1. The cart 11 generally includes an elongated support structure 14 (often referred to as the “column”), a cart base 15, and a console 16 located at the top of the column 14. The column 14 may include one or more carriages, such as carriages 17 (alternatively referred to as “arm supports”), for supporting the deployment of one or more robot arms 12 (three are shown in Figure 2). The carriage 17 may include individually configurable arm mounts that rotate along a vertical axis to adjust the base of the robot arm 12 for better positioning relative to the patient. The carriage 17 also includes a carriage interface 19 that allows the carriage 17 to translate vertically along the column 14.

[0027] The carriage interface 19 is connected to the column 14 through slots such as slots 20 positioned on both sides of the column 14 to guide the vertical translation of the carriage 17. The slots 20 include a vertical translation interface for positioning and holding the carriage at various vertical heights relative to the cart base 15. The vertical translation of the carriage 17 allows the cart 11 to adjust the reach of the robotic arm 12 to meet various table heights, patient sizes, and physician preferences. Similarly, individually configurable arm mounts on the carriage 17 allow the robotic arm base 21 of the robotic arm 12 to be angled in various configurations.

[0028] In some embodiments, a slot cover may be added to the slot 20, which is coplanar and parallel to the slot surface, to prevent dirt and fluid from entering the internal chamber of the column 14 and the vertical translation interface as the carriage 17 translates vertically. The slot cover may be deployed through a pair of spring spools positioned near the vertical top and bottom of the slot 20. The cover is coiled within the spool until it unfolds from a coiled state to expand and contract as the carriage 17 translates vertically up and down. The spring load of the spool provides a force to retract the cover into the spool as the carriage 17 translates toward the spool, while also maintaining a seal when the carriage 17 translates toward the spool. The cover may be attached to the carriage 17, for example, using a bracket in the carriage interface 19, to facilitate the proper expansion and contraction of the cover as the carriage 17 translates.

[0029] Column 14 may include internal mechanisms such as gears and motors, designed to use vertically aligned main screws to mechanically translate the carriage 17 in response to control signals generated in response to user input, such as input from a console 16.

[0030] The robotic arm 12 may generally include a robotic arm base 21 and end effectors 22 separated by a series of links 23 connected by a series of joints 24, each joint including an independent actuator, and each actuator including an independently controllable motor. Each independently controllable joint represents an independent degree of freedom available to the robotic arm. Each arm 12 has seven joints and therefore provides seven degrees of freedom. The numerous joints result in numerous degrees of freedom, enabling “redundant” degrees of freedom. Redundant degrees of freedom allow the robotic arm 12 to position their respective end effectors 22 in specific positions, orientations, and trajectories in space using different linkage positions and joint angles. This allows the system to position and orient medical instruments from desired points in space, while simultaneously allowing physicians to move the arm joints to clinically advantageous positions away from the patient to create better access while avoiding arm collisions.

[0031] The cart base 15 balances the weight of the column 14, carriage 17, and arm 12 on the floor. Therefore, the cart base 15 accommodates heavier components such as electronics, motors, power supplies, and components that enable either movement and / or fixing of the cart. For example, the cart base 15 includes casters 25 with rotatable wheels, allowing the cart to be easily moved around the room before treatment. After reaching a suitable position, the casters 25 may be stopped using wheel locks to hold the cart 11 in place during treatment.

[0032] Positioned at the vertical end of column 14, the console 16 provides both a user interface and a display screen (or a dual-purpose device such as a touchscreen 26) for receiving user input, and provides the user, a physician, with both preoperative and intraoperative data. Potential preoperative data on the touchscreen 26 may include preoperative planning, navigation and mapping data derived from preoperative computed tomography (CT) scans, and / or notes from preoperative patient interviews. Intraoperative data on the display may include optical information provided by tools, sensor and coordinate information from sensors, and essential patient statistics such as respiration, heart rate, and / or pulse. The console 16 may be positioned and tilted to allow the physician to access the console from the column 14 side opposite the carriage 17. From this position, the physician can view the console 16, the robotic arm 12, and the patient while operating the console 16 from behind the cart 11. As shown in the figure, the console 16 also includes a handle 27 to assist in operating and stabilizing the cart 11.

[0033] Figure 3 shows an embodiment of a robot-controllable system 10 configured for ureteroscopy. In a ureteroscopy procedure, a cart 11 may be positioned to deliver a ureteroscope 32, a procedure-specific endoscope designed to traverse the patient's urethra and ureters, to the patient's lower abdominal region. In ureteroscopy, it is sometimes desirable that the ureteroscope 32 be directly aligned with the patient's urethra to reduce friction and force on sensitive anatomical structures in the region. As shown, the cart 11 may be positioned at the leg of a table so that a robotic arm 12 can position the ureteroscope 32 for direct, linear access to the patient's urethra. From the leg of the table, the robotic arm 12 can insert the ureteroscope 32 directly into the patient's lower abdomen through the urethra, along a virtual rail 33.

[0034] After being inserted into the urethra using control techniques similar to those used in bronchoscopy, the ureteroscope 32 may be navigated to the bladder, ureters, and / or kidneys for diagnostic and / or therapeutic purposes. For example, the ureteroscope 32 can be directed to the ureters and kidneys, and a lithotripsy device positioned below the working channel of the ureteroscope 32 can be used to break up a kidney stone accumulation. After the lithotripsy is complete, the resulting stone fragments may be removed using a basket positioned below the ureteroscope 32.

[0035] Figure 4 shows an embodiment of a robot-controllable system similarly positioned for vascular procedures. In a vascular procedure, the system 10 may be configured such that a cart 11 can deliver a medical instrument 34, such as a maneuverable catheter, to an access point in the femoral artery within the patient's leg. The femoral artery offers both a larger diameter for navigation and a relatively less detourable, winding path to the patient's heart, which simplifies navigation. As in a ureteroscopy procedure, the cart 11 may be positioned toward the patient's leg and lower abdomen to allow a robotic arm 12 to provide a virtual rail 35 with direct, linear access to the femoral artery access point in the patient's thigh / hip region. After insertion into the artery, the medical instrument 34 may be directed and inserted by translating an instrument driver 28. Alternatively, the cart may be positioned around the patient's upper abdomen to reach alternative vascular access points, such as the carotid and brachial arteries near the shoulder and wrist.

[0036] B. Robot System - Table Embodiments of a robot-operated medical system may also incorporate a patient table. Incorporating a table reduces the amount of capital equipment in the operating room by eliminating carts and allows for greater access to the patient. Figure 5 shows one embodiment of such a robot-controllable system arranged for a bronchoscopy procedure. The system 36 includes a support structure or column 37 for supporting a platform 38 (illustrated as “table” or “bed”) on the floor. Similar to cart-based systems, the end effector of the robotic arm 39 of the system 36 includes an instrument driver 42 designed to manipulate elongated medical instruments, such as the bronchoscope 40 in Figure 5, through or along a virtual rail 41 formed from the linear alignment of the instrument driver 42. In practice, a C-arm for providing fluoroscopic imaging may be positioned above the patient’s upper abdominal region by placing the radiator and detector around the table 38.

[0037] Figure 6 provides an alternative diagram of the system 36 without a patient and medical equipment for consideration purposes. As shown, the column 37 may include one or more carriages 43, illustrated as a ring shape within the system 36, which may serve as the base for one or more robotic arms 39. The carriages 43 may translate along a vertical column interface 44 along the length of the column 37 to provide different bandage points where the robotic arms 39 can be positioned to reach a patient. The carriages 43 may rotate around the column 37 using a mechanical motor positioned within the column 37 to allow the robotic arms 39 to have access to multiple sides of a table 38, such as both sides of a patient. In embodiments with multiple carriages, the carriages may be positioned individually on the column and may translate and / or rotate independently of each other. The carriages 43 do not need to surround the column 37, or even be circular, but a ring shape as shown facilitates the rotation of the carriages 43 around the column 37 while maintaining structural balance. The rotation and translation of the carriage 43 allows the system to position medical instruments such as endoscopes and laparoscopes at different access points on the patient. In other embodiments (not shown), the system 36 may include a patient table or bed having adjustable arm supports in the form of parallel-extending bars or rails. One or more robotic arms 39 may be mounted on adjustable arm supports that can be adjusted vertically (e.g., via shoulders having elbow joints). By providing vertical adjustment, the robotic arms 39 can advantageously be compactly housed under the patient table or bed and then raised during treatment.

[0038] The arm 39 may be mounted to the carriage through a set of arm mounts 45, which include a series of joints that can rotate individually and / or extend in a nested manner, to provide additional configurability for the robot arm 39. Additionally, the arm mounts 45 may be positioned on the carriage 43 such that, when the carriage 43 is rotated appropriately, the arm mounts 45 can be positioned on one side of the table 38 (as shown in Figure 6), on both sides of the table 38 (as shown in Figure 9), or on any of the adjacent sides of the table 38 (not shown).

[0039] The column 37 structurally provides support for the table 38 and a path for the vertical translation of the carriage. Internally, the column 37 may be equipped with a main screw for guiding the vertical translation of the carriage and a motor for mechanizing the translation of the carriage based on the main screw. The column 37 may also transmit power and control signals to the carriage 43 and the robot arm 39 mounted on it.

[0040] The table base 46 performs a similar function to the cart base 15 of the cart 11 shown in Figure 2, and accommodates heavier components to balance the table / bed 38, column 37, carriage 43, and robot arm 39. The table base 46 may also incorporate rigid casters to provide stability during treatment. Casters that extend from the bottom of the table base 46 extend in opposite directions on both sides of the base 46 and may be retracted when it is necessary to move the system 36.

[0041] Continuing with Figure 6, the system 36 may also include a tower (not shown) that divides the functions of the system 36 between the table and the tower to reduce the form factor and bulk of the table. Similar to the embodiments disclosed previously, the tower may provide the table with various support functions such as processing, computing, and control capabilities, power, fluid mechanics, and / or optical and sensor processing. The tower may also be movable to be positioned away from the patient to improve physician access and keep the operating room tidy. Furthermore, positioning components within the tower allows for more storage space within the table base for potential storage of a robotic arm. The tower may also include a master controller or console that provides both a user interface for user input such as a keyboard and / or pendant, and a display screen (or touchscreen) for preoperative and intraoperative information such as real-time imaging, navigation, and tracking information. In some embodiments, the tower may also include a holder for a gas tank used for air delivery.

[0042] In some embodiments, the table base may house and store a robot arm when not in use. Figure 7 shows a system 47 housing a robot arm in one embodiment of a table-based system. In system 47, the carriage 48 may be translated vertically into the base 49 to house the robot arm 50, arm mount 51, and carriage 48 within the base 49. The base cover 52 may be translated and retracted to open, allowing the carriage 48, arm mount 51, and arm 50 to be positioned around the column 53, and closed to house and protect them when not in use. The base cover 52 may be sealed with a membrane 54 along the edge of its opening to prevent the ingress of dirt and fluid when closed.

[0043] Figure 8 shows one embodiment of a robot-controllable table-based system configured for a ureteroscopy procedure. In a ureteroscopy, the table 38 may include a swivel section 55 for positioning the patient off-angle from the column 37 and the table base 46. The swivel section 55 may rotate or pivot about a pivot point (e.g., located below the patient's head) to position the bottom of the swivel section 55 away from the column 37. For example, the rotation of the swivel section 55 allows a C-arm (not shown) to be positioned above the patient's lower abdomen without competing for space with the column (not shown) below the table 38. By rotating a carriage 35 (not shown) around the column 37, the robotic arm 39 may directly insert the ureteroscope 56 into the patient's inguinal region along a virtual rail 57 to reach the urethra. In ureteroscopy, a stirrup 58 may be fixed to the swivel portion 55 of the table 38 to support the position of the patient's legs during the procedure and to allow clear access to the patient's inguinal region.

[0044] In laparoscopic procedures, minimally invasive instruments may be inserted into the patient's anatomical structures through small incisions in the patient's abdominal wall. In some embodiments, the minimally invasive instruments include elongated rigid members, such as shafts, used to access anatomical structures within the patient. After the patient's abdominal cavity is expanded, the instruments may be oriented to perform surgical or medical tasks such as grasping, cutting, ablation, and suturing. In some embodiments, the instruments may include scopes, such as laparoscopes. Figure 9 shows an embodiment of a robot-controlled table-based system configured for laparoscopic procedures. As illustrated in Figure 9, the carriage 43 of the system 36 may rotate and be vertically adjusted to position a pair of robotic arms 39 on either side of the table 38 so that instruments 59 can be positioned using arm mounts 45 to pass through minimal incisions on either side of the patient and reach the patient's abdominal cavity.

[0045] To accommodate laparoscopic procedures, the robot-operable table system may also tilt the platform to a desired angle. Figure 10 shows an embodiment of a robot-controllable medical system having pitch or tilt adjustment. As shown in Figure 10, the system 36 can adapt to the tilt of the table 38 to position one side of the table at a greater distance from the floor than the other side. In addition, the arm mount 45 may rotate to match the tilt so that the arm 39 maintains the same planar relationship as the table 38. To adapt to steep angles, the column 37 may also include a nested portion 60 that allows the column 37 to extend vertically to prevent the table 38 from contacting the floor or colliding with the base 46.

[0046] Figure 11 provides a detailed illustration of the interface between the table 38 and the column 37. The pitch rotation mechanism 61 may be configured to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom. The pitch rotation mechanism 61 may also be enabled by positioning orthogonal axes 1 and 2 at the column-table interface, each axis being actuated by separate motors 3 and 4 in response to an electric pitch angle command. Rotation along one screw 5 allows for tilt adjustment along one axis 1, and rotation along the other screw 6 allows for tilt adjustment along the other axis 2. In some embodiments, ball joints may be used to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom.

[0047] For example, pitch adjustment is particularly useful when positioning the table in the Trendelenburg position, that is, when positioning the patient's lower abdomen higher off the floor than the patient's lower abdomen for lower abdominal surgery. The Trendelenburg position allows gravity to slide the patient's internal organs towards the patient's upper abdomen, emptying the abdominal cavity for minimally invasive tools to enter and perform lower abdominal surgical or medical procedures such as laparoscopic prostatectomy.

[0048] Figures 12 and 13 show isometric and end views of another embodiment of the table-based surgical robot system 100. The surgical robot system 100 includes one or more adjustable arm supports 105 (see, for example, Figure 14) which can be configured to support one or more robot arms relative to the table 101. In the illustrated embodiment, a single adjustable arm support 105 is shown, but additional arm supports can be provided on the opposite side of the table 101. The adjustable arm supports 105 can be moved relative to the table 101 to adjust and / or change the position of the adjustable arm support 105 and / or any robot arm mounted thereon relative to the table 101. For example, the adjustable arm support 105 can be adjusted with one or more degrees of freedom relative to the table 101. The adjustable arm supports 105 provide high versatility to the system 100, including the ability to easily accommodate one or more adjustable arm supports 105 and any robot arms mounted thereon under the table 101. The adjustable arm support 105 can be raised from its stowed position to a position below the upper surface of the table 101. In another embodiment, the adjustable arm support 105 can be raised from its stowed position to a position above the upper surface of the table 101.

[0049] The adjustable arm support 105 can provide several degrees of freedom, including lift (e.g., vertical translation), lateral translation, and tilt. In the illustrated embodiments of Figures 12 and 13, the arm support 105 consists of four degrees of freedom, indicated by arrows in Figure 12. The first degree of freedom allows adjustment of the adjustable arm support 105 in the z direction ("Z-lift"). For example, the adjustable arm support 105 may include a carriage 109 configured to move up and down along or relative to the column 102 supporting the table 101. The second degree of freedom allows the adjustable arm support 105 to tilt. For example, the adjustable arm support 105 may include a rotational joint, which can allow the adjustable arm support 105 to align with the bed in a Trendelenburg position. The third degree of freedom allows the adjustable arm support 105 to "pivot upward," which can be used to adjust the distance between the side of the table 101 and the adjustable arm support 105. The fourth degree of freedom allows for the translation of the arm support 105, which is adjustable along the longitudinal length of the table.

[0050] The surgical robot system 100 shown in Figures 12 and 13 may include a table supported by a column 102 attached to a base 103. The base 103 and column 102 support the table 101 with respect to a support surface. The floor axis 131 and support axis 133 are shown in Figure 13.

[0051] The adjustable arm support 105 can be mounted on the column 102. In other embodiments, the arm support 105 can be mounted on the table 101 or the base 103. The adjustable arm support 105 may include a carriage 109, a bar or rail connector 111, and a bar or rail 107. In some embodiments, one or more robot arms mounted on the rail 107 can translate and move relative to each other.

[0052] The carriage 109 may be attached to the column 102 by a first joint 113, thereby allowing the carriage 109 to move relative to the column 102 (e.g., up and down on the first or vertical axis 123). The first joint 113 may provide a first degree of freedom ("Z-lift") to the adjustable arm support 105. The adjustable arm support 105 may include a second joint 115 that provides a second degree of freedom (tilt) to the adjustable arm support 105. The adjustable arm support 105 may include a third joint 117 that can provide a third degree of freedom ("upward pivot") to the adjustable arm support 105. An additional joint 119 (shown in Figure 13) may be provided to mechanically restrain the third joint 117 so as to maintain the orientation of the rail 107 when the rail connector 111 is rotated around the third axis 127. The adjustable arm support 105 may include a fourth joint 121 that can provide a fourth degree of freedom (translation) of the adjustable arm support 105 along a fourth axis 129.

[0053] Figure 14 shows an end view of a surgical robot system 140A having two adjustable arm supports 105A and 105B mounted on the ventral side of a table 101. A first robotic arm 142A is mounted on a bar or rail 107A of the first adjustable arm support 105B. The first robotic arm 142A includes a base 144A mounted on the rail 107A. The distal end of the first robotic arm 142A includes an instrument drive mechanism 146A that can be attached to one or more robotic medical instruments or tools. Similarly, a second robotic arm 142B includes a base 144B mounted on the rail 107B. The distal end of the second robotic arm 142B includes an instrument drive mechanism 146B. The instrument drive mechanism 146B can be configured to be attached to one or more robotic medical instruments or tools.

[0054] In some embodiments, one or more of the robot arms 142A, 142B include an arm having seven or more degrees of freedom. In some embodiments, one or more of the robot arms 142A, 142B may include eight degrees of freedom, including an insertion axis (one degree of freedom including insertion), a wrist (three degrees of freedom including wrist pitch, yaw, and roll), an elbow (one degree of freedom including elbow pitch), a shoulder (two degrees of freedom including shoulder pitch and yaw), and a base 144A, 144B (one degree of freedom including translation). In some embodiments, the insertion degrees of freedom can be provided by the robot arms 142A, 142B, but in other embodiments, the instrument itself provides insertion via an instrument-based insertion architecture.

[0055] C. Appliance Drivers and Interfaces The end effector of the system's robotic arm includes (i) an instrument driver (alternatively called an “instrument drive mechanism” or “instrument device manipulator”) incorporating electromechanical means for operating a medical instrument, and (ii) a removable or detachable medical instrument which may lack any electromechanical components such as a motor. This dichotomy may be based on the need to sterilize medical instruments used in medical procedures, the fact that their complex mechanical assemblies and delicate electronics make it impossible to adequately sterilize expensive capital equipment. Therefore, medical instruments may be designed to be removed, detached, and replaced from the instrument driver (and thus its system) for individual sterilization or disposal by the physician or physician's staff. In contrast, the instrument driver does not need to be replaced or sterilized and may be covered for protection.

[0056] Figure 15 shows an exemplary instrument driver. Positioned at the distal end of a robotic arm, the instrument driver 62 includes one or more drive units 63 arranged with parallel axes to provide controlled torque to a medical instrument via a drive shaft 64. Each drive unit 63 includes an individual drive shaft 64 for interacting with the instrument, a gearhead 65 for converting the rotation of the motor shaft into a desired torque, a motor 66 for generating the drive torque, an encoder 67 for measuring the speed of the motor shaft and providing feedback to the control circuit, and a control circuit 68 for receiving control signals and operating the drive unit. Each drive unit 63 is independently controlled and motorized, and the instrument driver 62 can provide multiple (four as shown in Figure 15) independent drive outputs to the medical instrument. During operation, the control circuit 68 receives control signals, transmits motor signals to the motor 66, compares the resulting motor speed measured by the encoder 67 with a desired speed, modulates the motor signals to generate the desired torque.

[0057] For procedures requiring a sterile environment, the robotic system may incorporate a drive interface, such as a sterilization adapter connected to a sterile drape, positioned between the instrument driver and the medical instrument. The primary purpose of the sterilization adapter is to transmit angular motion from the instrument driver's drive shaft to the instrument's drive input while maintaining physical separation between the drive shaft and the drive input, and thus sterility. Thus, an exemplary sterilization adapter may consist of a series of rotary inputs and outputs intended to mate with the instrument driver's drive shaft, and a drive input to the instrument. The sterile drape connected to the sterilization adapter is made of a thin, flexible material, such as transparent or translucent plastic, and is designed to cover the instrument driver, the robotic arm, and capital equipment such as a cart (in a cart-based system) or a table (in a table-based system). The use of the drape allows the capital equipment to be positioned close to the patient while still being located in an area where sterilization is not required (i.e., a non-sterile field). On the other side of the sterile drape, the medical instrument may interface with the patient in an area where sterilization is required (i.e., a sterile field).

[0058] D. Medical devices Figure 16 shows an exemplary medical instrument with a pair of instrument drivers. Like other instruments designed for use with robotic systems, the medical instrument 70 includes an elongated shaft 71 (or elongated body) and an instrument base 72. The instrument base 72, also called an “instrument handle” due to its design intended for manual interaction by a physician, may include a rotatable drive input 73, e.g., a receptacle, pulley, or spool, which is generally designed to mate with a drive output 74 extending through a drive interface on the instrument driver 75 at the distal end of a robotic arm 76. When physically connected, latched, and / or coupled, the mated drive input 73 of the instrument base 72 can share a rotational axis with the drive output 74 in the instrument driver 75, allowing for the transmission of torque from the drive output 74 to the drive input 73. In some embodiments, the drive output 74 may include a spline designed to mate with a receptacle on the drive input 73.

[0059] The elongated shaft 71 is designed to be delivered through either an anatomical opening or lumen, such as in an endoscope, or a minimally invasive incision, such as in a laparoscopy. The elongated shaft 71 may be either flexible (e.g., having properties similar to an endoscope) or rigid (e.g., having properties similar to a laparoscope), or may include a customized combination of both flexible and rigid portions. When designed for laparoscopy, the distal end of a rigid elongated shaft may be connected to an end effector extending from a joined list formed from a clevis having at least one degree of freedom, and to a surgical tool or medical instrument, such as a gripper or scissors, which can be actuated based on force from a tendon when the drive input rotates in response to torque received from the drive output 74 of the instrument driver 75. When designed for endoscopy, the distal end of a flexible elongated shaft may include a maneuverable or controllable bend that can articulate and flex based on torque received from the drive output 74 of the instrument driver 75.

[0060] Torque from the instrument driver 75 is transmitted downstream of the elongated shaft 71 via tendons along the shaft 71. These individual tendons, such as pull wires, may be individually fixed to individual drive inputs 73 within the instrument handle 72. From the handle 72, the tendons are directed downward along the elongated shaft 71 into one or more pull lumens and fixed to the distal portion of the elongated shaft 71, or to the wrist of the distal portion of the elongated shaft. During surgical procedures such as laparoscopy, endoscopy, or hybrid procedures, these tendons may be connected to distally mounted end effectors such as wrists, grippers, or scissors. Under such configurations, the torque applied to the drive inputs 73 transmits tension to the tendons, thereby acting on the end effectors in some way. In some embodiments, during surgical procedures, the tendons can rotate the joints around an axis, thereby moving the end effectors in one direction or another. Alternatively, the tendon may be connected to one or more jaws of the gripping device at the distal end of the elongated shaft 71, and the gripping device will be closed by tension from the tendon.

[0061] In endoscopy, tendons may be connected to flexures or articulation points positioned along an elongated shaft 71 (e.g., at the distal end) via adhesive, control rings, or other mechanical fixation. When fixedly attached to the distal end of a flexure, the torque applied to the drive input 73 is transmitted downstream of the tendon, causing the softer flexure (sometimes called an articulation point or region) to flex or articulate. Along the non-flexure portions, it may be advantageous to equilibrium the radial forces resulting from tension in the pull wires by arranging individual tendons in a spiral or helical shape, with individual pull lumens facing along (or inward) the wall of the endoscope shaft. The angles of the spirals and / or spacing between these may be modified or designed for a particular purpose, with narrower spirals resulting in less shaft compression under load, while smaller amounts of spirals result in greater shaft compression under load, but also exhibit flexion limitations. At the other end of the spectrum, the lumen may be oriented parallel to the longitudinal axis of the elongated shaft 71 to enable controlled articulation at the desired flexed or articulated portion.

[0062] In endoscopic procedures, the elongated shaft 71 houses several components that support robotic procedures. The shaft may also constitute a working channel for positioning, irrigating, and / or aspirating surgical tools (or medical instruments) into the surgical area at the distal end of the shaft 71. The shaft 71 may also house wires and / or optical fibers for signal exchange with an optical assembly at the distal tip, which may include an optical camera. The shaft 71 may also house optical fibers for transporting light from a proximal light source, such as a light-emitting diode, to the distal end of the shaft.

[0063] At the distal end of the instrument 70, the distal tip may include an opening for a working channel for delivering the tool to the surgical site for diagnosis and / or treatment, irrigation, and aspiration. The distal tip may also include a port for a camera, such as a fiberscope or digital camera, for capturing images of the internal anatomical space. In connection with this, the distal tip may also include a port for a light source for illuminating the anatomical space when a camera is used.

[0064] In the embodiment shown in Figure 16, the drive shaft axis, and therefore the drive input axis, is perpendicular to the axis of the elongated shaft. However, this arrangement complicates the rolling capability of the elongated shaft 71. By allowing the elongated shaft 71 to roll along its axis while the drive input 73 is stationary, undesirable entanglement of the tendon occurs as it extends from the drive input 73 and enters the lumen within the elongated shaft 71. Such resulting tendon entanglement can interfere with any control algorithm intended to predict the movement of the flexible elongated shaft during endoscopic procedures.

[0065] Figure 17 shows an alternative design of an instrument driver and instrument in which the axis of the drive unit is parallel to the axis of the instrument's elongated shaft. As shown, the circular instrument driver 80 includes four drive units, each having a drive output unit 81 aligned parallel to the end of a robot arm 82. The drive units and their respective drive output units 81 are housed in a rotary assembly 83 of the instrument driver 80, which is driven by one of the drive units in the assembly 83. In response to the torque provided by the rotary drive unit, the rotary assembly 83 rotates along a circular bearing that connects the rotary assembly 83 to the non-rotating portion 84 of the instrument driver. Power and control signals may be transmitted from the non-rotating portion 84 of the instrument driver 80 to the rotary assembly 83 through electrical contacts, or maintained through rotation by a brushed slip ring connection (not shown). In other embodiments, the rotary assembly 83 may be integrated with the non-rotatable portion 84 and therefore respond to a separate drive unit that is not parallel to the other drive units. The rotation mechanism 83 enables the instrument driver 80 to rotate the drive unit and their respective drive output units 81 as a single unit around the instrument driver shaft 85.

[0066] Similar to the embodiments disclosed previously, the device 86 may include an elongated shaft portion 88 and a device base 87 (shown by a transparent outer skin for illustrative purposes) which includes a plurality of drive inputs 89 (such as receptacles, pulleys, and spools) configured to receive a drive output portion 81 in the device driver 80. Unlike the embodiments disclosed previously, the device shaft 88 extends from the center of the device base 87, and its axis is substantially parallel to the axis of the drive inputs 89, rather than being orthogonal as in the design of Figure 16.

[0067] When connected to the rotary assembly 83 of the instrument driver 80, the medical instrument 86, including the instrument base 87 and instrument shaft 88, rotates together with the rotary assembly 83 around the instrument driver shaft 85. Since the instrument shaft 88 is positioned at the center of the instrument base 87, the instrument shaft 88 is coaxial with the instrument driver shaft 85 when installed. Therefore, the rotation of the rotary assembly 83 causes the instrument shaft 88 to rotate around its own longitudinal axis. Furthermore, as the instrument base 87 rotates together with the instrument shaft 88, any tendons connected to the drive input 89 within the instrument base 87 do not become entangled during rotation. Thus, the parallelism of the axes of the drive output 81, the drive input 89, and the instrument shaft 88 allows for shaft rotation without entanglement of control tendons.

[0068] Figure 18 shows a device having an insertable device base architecture according to several embodiments. The device 150 can be connected to one of the device drivers described above. The device 150 includes an elongated shaft 152, an end effector 162 connected to the shaft 152, and a handle 170 connected to the shaft 152. The elongated shaft 152 includes a tubular member having a proximal portion 154 and a distal portion 156. The elongated shaft 152 includes one or more channels or grooves 158 along its outer surface. The grooves 158 are configured to receive one or more wires or cables 180 through them. Thus, one or more cables 180 extend along the outer surface of the elongated shaft 152. In other embodiments, the cables 180 may also extend through the elongated shaft 152. Operation of one or more of the cables 180 (e.g., via a device driver) results in the operation of the end effector 162.

[0069] The fixture handle 170, also referred to as the fixture base, may generally include one or more mechanical input sections 174, such as a receptacle, pulley, or spool, and a mounting interface 172, which are designed to reciprocately engage with one or more torque couplers on the mounting surface of the fixture driver.

[0070] In some embodiments, the instrument 150 includes a series of pulleys or cables that allow an elongated shaft 152 to translate relative to the handle 170. In other words, the instrument 150 itself includes an instrument-based insertion architecture that adapts to the insertion of the instrument, thereby minimizing reliance on the robotic arm to provide the insertion of the instrument 150. In other embodiments, the robotic arm may be significantly involved in the insertion of the instrument.

[0071] E. Controller Any robotic system described herein may include an input device or controller for operating an instrument attached to a robotic arm. In some embodiments, the controller may be linked to the instrument (e.g., communicatively, electronically, electrically, wirelessly, and / or mechanically) so that operation of the controller triggers a corresponding operation of the instrument, for example, via master-slave control.

[0072] Figure 19 is a perspective view of an embodiment of the controller 182. In this embodiment, the controller 182 includes a hybrid controller that can have both impedance control and admittance control. In other embodiments, the controller 182 can utilize only impedance or passive control. In other embodiments, the controller 182 can utilize only admittance control. Being a hybrid controller, the controller 182 can advantageously have lower perceived inertia during use.

[0073] In the illustrated embodiment, the controller 182 is configured to enable the operation of two medical devices and includes two handles 184. Each of the handles 184 is connected to a gimbal 186. Each gimbal 186 is connected to a positioning platform 188.

[0074] As shown in Figure 19, each positioning platform 188 includes a SCARA arm (selective compliance assembly robot arm) 198 connected to a column 194 by a prism joint 196. The prism joint 196 is configured to translate along the column 194 (for example, along a rail 197) so that each of the handles 184 is translated in the z-direction, providing a first degree of freedom. The SCARA arm 198 is configured to enable the movement of the handles 184 in the xy-plane, providing two additional degrees of freedom.

[0075] In some embodiments, one or more load cells are positioned within the controller. For example, in some embodiments, load cells (not shown) are positioned within each body of the gimbal 186. By providing load cells, a portion of the controller 182 can operate under admittance control, thereby advantageously reducing the perceived inertia of the controller during use. In some embodiments, the positioning platform 188 is configured for admittance control, while the gimbal 186 is configured for impedance control. In other embodiments, the gimbal 186 is configured for admittance control, and the positioning platform 188 is configured for impedance control. Thus, in some embodiments, the translational or positional degrees of freedom of the positioning platform 188 may depend on admittance control, while the rotational degrees of freedom of the gimbal 186 may depend on impedance control.

[0076] F. Navigation and Control Conventional endoscopy may involve the use of fluoroscopy (e.g., delivered via a C-arm) and other forms of radiation-based imaging modalities to provide intracavitary guidance to the physician operator. In contrast, the robotic systems envisioned by this disclosure can provide non-radiation-based navigation and localization means to reduce physician exposure to radiation and reduce the amount of equipment in the operating room. As used herein, the term “localization” may mean determining and / or monitoring the position of an object in a reference coordinate system. Techniques such as preoperative mapping, computer vision, real-time EM tracking, and robot command data may be used individually or in combination to achieve a radiation-free operating environment. In other cases where radiation-based imaging modalities are still used, preoperative mapping, computer vision, real-time EM tracking, and robot command data may be used individually or in combination to enhance information obtained solely by radiation-based imaging modalities.

[0077] Figure 20 is a block diagram illustrating a positioning system 90 for estimating the position of one or more elements of a robotic system, such as the position of an instrument, according to an exemplary embodiment. The positioning system 90 may be a set of one or more computer devices configured to execute one or more instructions. The computer devices may be embodied by a processor (or more processors) and computer-readable memory within one or more components considered above. For example, but not limited to, the computer devices may be located in the tower 30 shown in Figure 1, the carts shown in Figures 1 to 4, the beds shown in Figures 5 to 14, and so on.

[0078] As shown in Figure 20, the positioning system 90 may include a positioning module 95 that processes input data 91-94 to generate position data 96 for the distal end of a medical device. The position data 96 may be data or logic representing the location and / or orientation of the distal end of the device relative to a reference system. The reference system may be the anatomical structure of a patient or a reference system to a known object such as an EM field generator (see the following discussion on EM field generators).

[0079] Here, various input data 91-94 are described in more detail. Preoperative mapping can be achieved by utilizing the acquisition of low-dose CT scans. Preoperative CT scans are reconstructed into three-dimensional images that are visualized, for example, as "slices" of notch diagrams of the patient's internal anatomical structures. When analyzed as a whole, image-based models of anatomical cavities, spaces, and structures of the patient's anatomical structures, such as the patient's lung network, can be generated. Techniques such as center-line geometry can be determined and approximated from the CT images to create a three-dimensional volume of the patient's anatomical structures, referred to as model data 91 (also referred to as "preoperative model data" if generated using only preoperative CT scans). The use of center-line geometry is discussed in U.S. Patent Application No. 14 / 523,760, the entirety of which is incorporated herein. Network phase models may also be derived from CT images and are particularly suitable for bronchoscopy.

[0080] In some embodiments, the instrument may be equipped with a camera to provide visual data 92. The localization module 95 may process the visual data to enable one or more vision-based localization tracking. For example, preoperative model data may be used in conjunction with the visual data 92 to enable computer vision-based tracking of a medical instrument (e.g., an endoscope, or an instrument that advances through the working channel of an endoscope). For example, using preoperative model data 91, a robotic system may generate a library of predicted endoscopic images from the model based on the expected movement path of the endoscope, with each image linked to a location in the model. During surgery, this library can be referenced by the robotic system to assist localization by comparing real-time images captured by a camera (e.g., a camera at the distal end of the endoscope) with those in the image library.

[0081] Other computer vision-based tracking techniques use tracking capabilities to determine the movement of the camera, and therefore the endoscope. Some features of the localization module 95 may identify circular geometric shapes in preoperative model data 91 corresponding to anatomical lumens and track changes in those geometric shapes to determine which anatomical lumen was selected, as well as the relative rotation and / or translational movement of the camera. The use of phase maps may further enhance the vision-based algorithms or techniques.

[0082] Optical flow, another computer vision-based technique, may analyze the displacement and translation of image pixels in a video sequence within visual data 92 to infer camera movement. Examples of optical flow techniques include motion detection, object segmentation calculation, luminance, motion compensation coding, and stereoscopic parallax measurement. By comparing multiple frames across multiple iterations, the movement and position of the camera (and therefore the endoscope) can be determined.

[0083] The positioning module 95 can use real-time EM tracking to generate the real-time location of the endoscope within a global coordinate system that can be registered in the patient's anatomical structure represented by a preoperative model. In EM tracking, EM sensors (or trackers) comprising one or more sensor coils embedded in one or more positions and orientations within a medical instrument (e.g., an endoscope) measure fluctuations in the EM field generated by one or more static EM field generators positioned at known locations. The position information detected by the EM sensors is stored as EM data 93. The EM field generators (or transmitters) can be positioned close to the patient to generate a low-intensity magnetic field that the embedded sensors can detect. The magnetic field induces a small current within the sensor coils of the EM sensors, and this current can be analyzed to determine the distance and angle between the EM sensors and the EM field generators. These distances and orientations can be intraoperatively "aligned" to the patient's anatomical structure (e.g., a preoperative model) to determine a geometric transformation that aligns the position in the preoperative model of the patient's anatomical structure with a single position in the coordinate system. Once registered, an EM tracker embedded in one or more locations on a medical instrument (e.g., the distal tip of an endoscope) can provide a real-time display of the medical instrument's progression through the patient's anatomical structure.

[0084] Robot command and kinematic data 94 may also be used by a positioning module 95 to provide positioning data 96 for the robotic system. Device pitch and yaw resulting from joint motion commands can be determined during preoperative calibration. Intraoperatively, these calibration measurements can be used in combination with known insertion depth information to estimate the position of the instrument. Alternatively, these calculations can be analyzed in combination with EM, vision, and / or phase modeling to estimate the position of medical instruments within the network.

[0085] As shown in Figure 20, several other input data can be used by the positioning module 95. For example, although not shown in Figure 20, an instrument utilizing a shape-sensing fiber can provide shape data that the positioning module 95 can use to determine the position and shape of the instrument.

[0086] The positioning module 95 can use a combination of input data 91-94. In some cases, such a combination allows the positioning module 95 to use a probabilistic approach, assigning confidence weights to the locations determined from each of the input data 91-94. Therefore, if the EM data is unreliable (for example, if EM interference is present), the reliability of the location determined by the EM data 93 will decrease, and the positioning module 95 may rely more heavily on the visual data 92 and / or the robot command and kinematic data 94.

[0087] As discussed above, the robotic systems considered herein can be designed to incorporate one or more combinations of the technologies described above. A computer-based control system for a robotic system based on a tower, bed, and / or cart may store computer program instructions in a non-temporary computer-readable storage medium such as a persistent magnetic memory drive or a solid-state drive, which, when executed, cause the system to receive and analyze sensor data and user commands, generate control signals for the entire system, and display navigation and localization data such as the position of instruments in a global coordinate system and an anatomical map.

[0088] 2. Introduction to robot-assisted ancillary procedures. The treatment of a particular medical condition may involve performing two or more medical procedures to completely treat the medical condition. For example, the diagnosis and management of a lung lesion may involve multiple treatment episodes to perform medical procedures, including flexible endoscopy and thoracoscopy. After a lesion is discovered from a radiological study, such as through the analysis of a CT scan, a physician may perform endoscopic diagnosis and subsequent treatment in the course of multiple treatment episodes. In one embodiment, if a physician suspects early-stage cancer in a patient, the physician may order the patient to undergo an endoscopic procedure for the initial diagnosis of cancer. During the endoscopic procedure, the nodule may be biopsied, and if the physician determines that removal of the nodule is necessary, the physician may order the patient to undergo a second treatment episode for surgical removal of the nodule.

[0089] There are drawbacks to performing multiple procedural episodes. The clinical costs and time requirements from both caregivers and patients for diagnosing and treating the patient's condition increase with such a multi-episode approach. In addition, during surgical resection, procedures (e.g., endoscopy) may need to be performed repeatedly to help precisely locate the tumor and provide a surgical target for surgical resection. Furthermore, when staging medical procedures across multiple procedural episodes, patients may have to experience multiple anesthetic episodes, which can increase the risks and inconveniences for the patient. Also, multiple procedural episodes may utilize increased intraoperative resources (e.g., pre-operative workup, post-operative recovery, and possibly an overnight hospital stay), increasing time and costs for both patients and physicians.

[0090] Rather than progressively staging medical procedures across multiple procedural episodes, physicians have the option of performing multiple procedures sequentially within a single procedural episode. Such a single procedural episode may be performed as part of a single procedural episode by inviting additional clinical providers to assist in performing procedures concurrently.

[0091] However, with respect to multiple procedure episodes, there are drawbacks associated with single procedure episodes, as is currently practiced. As mentioned above, multiple clinicians may need to assist in performing a single procedure episode, thereby resulting in increased costs and overcrowded space in the operating room. Furthermore, to perform multiple procedures sequentially over a single procedure episode, physicians may alternate between various techniques, which may involve switching between sterile and non-sterile techniques. Switching between sterile and non-sterile techniques may further involve shifting attention to the regressive and significantly interrupted clinical workflow, moving from one surgical site to another.

[0092] Coordination of multiple healthcare providers and / or physicians to perform procedures in parallel during a single procedural episode is costly and may be expensive for certain procedures. An example of the use of multiple clinicians when performing parallel procedures as part of a single procedural episode is combined endoscopic and laparoscopic surgery (CELS), a manual method for performing colon polyp removal. Polyps can be assessed based on their size, type, and location to determine whether they can be removed endoscopically. If polyps cannot be removed endoscopically, they can be removed via segmental colectomy, which carries a relatively high complication rate and a long recovery time. CELS was proposed as a way to enable extraluminal mobilization of the colon (using laparoscopic instruments) and to facilitate intraluminal resection (using endoscopic instruments). CELS typically requires at least two physicians (to control the laparoscope and endoscopic instruments, respectively) and two assistants (to hold the laparoscope and colonoscope, respectively). While one physician moves the instruments, the other provider may hold them in place, which can be physically demanding over extended periods. Additional staff may be present in the room to manage instrument changes, sutures or gauze, removed handle specimens, and laparoscopic instruments.

[0093] Embodiments of this disclosure relate to systems and methods for simultaneously performing two or more types / modes of treatment as part of a single treatment episode (e.g., by a single user or team). The systems and methods described herein improve upon the single and multiple treatment episodes described above. In some embodiments, parallel treatments can be performed as part of a single treatment episode by a novel robotic medical system, thereby reducing the need to have the same number of healthcare providers and / or physicians as with non-robot-assisted parallel medical treatments, such as existing CELS systems.

[0094] In addition to the above examples of endoscopic diagnosis and surgical resection of cancerous tumors, other exemplary medical procedures that can benefit from the systems and methods described herein include bronchoscopic localization of lung cancer with simultaneous thoracoscopic resection, endoscopic localization of gastrointestinal cancer with laparoscopic resection, endoscopic localization and resection of gastrointestinal cancer with laparoscopy assistance, endoscopic imaging or visualization for gastrointestinal reconstruction such as gastrectomy and Roux-en-Y gastric bypass, ureteral stone / tumor localization and percutaneous removal / resection, etc. In some embodiments, such procedures can be performed in a single procedure episode. In some embodiments, such procedures can be performed by a minimum number of clinicians, and possibly by only one physician. Furthermore, in some embodiments, simultaneous procedures can be performed using a single type of console to control the simultaneous procedures.

[0095] According to aspects of this disclosure, a first type of procedure performed during an incidental / parallel medical procedure may include delivering one or more flexible devices to the patient, while a second type of procedure may include delivering one or more rigid devices to the patient. For example, in one embodiment, two incidental procedures may involve an endoscopic procedure (e.g., using a flexible scope) in combination with a laparoscopic procedure (e.g., using a rigid scope). In a medical procedure involving a tumor in the bronchi, a first endoscopic instrument (e.g., a flexible bronchoscope) may be inserted through the bronchi, while a second laparoscopic instrument (e.g., a rigid camera or cutter) may be inserted through an incision providing access to the tumor.

[0096] In some embodiments, a first type of procedure can be performed through a natural opening, while a second type of procedure can be performed through an incision. For example, in a medical procedure involving the removal of kidney stones, a first tool (e.g., a laser) can be inserted through the natural opening of the urethra to break up the stones in the renal pelvis, while a second tool (e.g., a vacuum) can be inserted percutaneously through an incision to aspirate and remove the broken kidney stones.

[0097] A. Systems and methods for performing related procedures. In some embodiments, a single robotic medical system can perform two or more medical procedures simultaneously as part of a single treatment episode. Figure 21 shows an embodiment of a bed-based robotic system configured to perform ancillary procedures according to embodiments of the present disclosure. As shown in Figure 21, the robotic medical system 200 includes a first set of one or more robotic arms 205 and a second set of one or more robotic arms 210. The system 200 further includes a platform 215, which may include a bed on which a patient can be placed, and the first set 205 and the second set 210 of robotic arms are positioned on arm supports or rails on both sides of the platform 215. The first set 205 of robotic arms may be coupled to a first adjustable arm support 220, while the second set 210 of robotic arms may be coupled to a second adjustable arm support 225 located on the opposite side of the platform 215 from the first adjustable arm support 220. The bed on the platform 215 may include a head and foot. The first arm support 220 and the second arm support 225 may be positioned between the head and the feet.

[0098] In certain embodiments, the first set of arms 205 may be configured to control one or more flexible instruments 230, such as a colonoscope, bronchoscope, or urethroscope (e.g., having internal and external catheters), as part of an endoscopic procedure. The second set of arms 210 may be configured to control one or more rigid instruments 235, such as a rigid camera, vascular sealer, tissue cutter, stapler, or needle driver, as part of a laparoscopy procedure. In this embodiment, the first set of arms 205 are aligned in a virtual rail to deliver a flexible ureteroscope, according to some embodiments. The second set of arms 210 delivers one or more laparoscopic instruments through a laparoscopic port. In some embodiments, at least one of the laparoscopic instruments can be rigid, but in some embodiments, the second set of arms 210 may be configured to deliver a combination of rigid and flexible instruments, such as a rigid cutter and a flexible articulated laparoscope. As shown in Figure 21, the first set of arms 205 is configured to approach the patient from a different direction than the second set of arms 210. For example, the first set of arms 205 can approach the patient from the base of the platform 215, while the second set of arms 210 can approach the patient from the side of the platform 215. In some embodiments, one or more of the endoscopic or laparoscopic instruments can be navigated partially or entirely via EM or fluoroscopic navigation. In some embodiments, the first set of arms 205 can be locked while the second set of arms 210 is movable. In other embodiments, the first set of arms 205 is movable while the second set of arms 210 is locked.

[0099] As shown in Figure 21, the first set of arms 205 are connected to the first adjustable arm support 220, while the second set of arms 210 are connected to the second adjustable arm support 225. The first adjustable arm support 220 can be adjusted independently of the second adjustable arm support 225. In some embodiments, the first adjustable arm support 220 is at a different height than the second adjustable arm support 225, while in other embodiments, the first adjustable arm support 220 is at the same height as the second adjustable arm support 225. In some embodiments, the arm supports 220, 225 and / or arms 205, 210 can be housed below the platform 215. In some embodiments, one or more of the arm supports 220, 225 and / or arms 205, 210 can be raised above the base of the platform, thereby avoiding "mop tilt" and accidental dirt getting onto these components. In some embodiments, one or more arm supports 220, 225 and / or arms 205, 210 can be raised from their hoisting position to a height higher than the top surface of the bed or platform 215.

[0100] In this embodiment, a pair of arms 210 are connected to a second adjustable arm support 225, and a pair of arms 205 are connected to a first adjustable arm support 220. In other embodiments, the number of arms in each adjustable arm support can be even. In other embodiments, the number of arms may be greater or less than the number of arms shown in Figure 21.

[0101] Figure 22 shows another embodiment of a bed-based robotic system configured to perform ancillary procedures according to aspects of the present disclosure. Similar to the embodiment in Figure 21, the embodiment shown in Figure 22 includes one or more robotic arms 205 in a first set, one or more robotic arms 210 in a second set, and a platform 215 having a plurality of robotic arms 205 and 210 positioned on arm supports on both sides of the platform 215. These arm supports on both sides include a first adjustable arm support 220 and a second adjustable arm support 225.

[0102] In contrast to the embodiment shown in Figure 21, in the embodiment of Figure 22, the first set of robot arms 205 includes a single robot arm configured to control a rigid laparoscopic instrument 235 percutaneously on a patient, while the second set of robot arms 210 includes a pair of robot arms configured to control a flexible endoscopic instrument 230. In other embodiments, the robot arms 205 and 210 may be located on the same side of the bed (or adjacent sides of the bed) and may be configured to control the flexible endoscopic instrument 230 and the rigid laparoscopic instrument 235, respectively. In yet another embodiment, a set of robot arms 205 and 210 including at least one arm located on each side of the platform 215 may be configured to control a first medical instrument (e.g., a flexible endoscopic instrument 230), and another set of robot arms 205 and 210 including at least one arm located on each side of the platform 215 may be configured to control a second medical instrument (e.g., a rigid laparoscopic instrument 235).

[0103] In each of the embodiments shown in Figures 21 and 22, a single bed-based system having a robotic arm attached thereto may be configured to perform both endoscopic procedures with one or more flexible instruments 230 and laparoscopic procedures with one or more rigid instruments 235. Endoscopic procedures can be performed through a natural orifice (e.g., the throat), and laparoscopic procedures can be performed through an incision (e.g., the chest). The procedures can be advantageously performed simultaneously / accompanied (partially or entirely) by a single user via a single console (further details of which are provided below). In some embodiments, the robotic medical system 200 may be configured to perform two types of medical procedures in series, as needed. For example, a first type of procedure may be performed on a patient. If such a procedure is ineffective on its own, a second type of procedure may be performed as part of a “procedure escalation” to supersede or complement the first type of procedure.

[0104] In Figures 21 and 22, the robotic arms 205 and 210 can be housed and subsequently deployed from beneath the platform 215. The robotic arms 205 and 210 are configured to be positioned at multiple locations, for example, near the feet of patient 240 and / or near the right side of patient 240, based on commands received from the user. The robotic arms 205 and 210 are configured to be translationally translatable along adjustable arm supports 220 and 225. The robotic arms 205 and 210 can have multiple degrees of freedom, including two, three, four, five, six, seven, eight or more. In some embodiments, the robotic arms 205 and 210 can include one or more redundant degrees of freedom. The robotic arms 205 and 210 are connected to adjustable arm supports 220 and 225, which are configured to provide vertical, lateral, and longitudinal adjustment of the robotic arms 205 and 210. In some embodiments, the adjustable arm supports are configured to be adjustable with 3 degrees of freedom, independently of the movement of the robot arms 205 and 210. In certain embodiments, the adjustable arm supports 220 and 225 may be in the form of rods or rails along which the base of the robot arm can be translated. The base can be connected to the robot arms 205 and 210 and the adjustable arm supports 220 and 225.

[0105] Referring to a specific configuration of the robotic system shown in Figure 21, the first adjustable arm support 220 extends below and beyond the base of the platform 215, thereby being adjusted horizontally to allow the first robotic arm 205 to be positioned near the patient 240's feet as part of an endoscopic procedure. The second adjustable arm support 225 remains substantially aligned with the platform 215, but is adjusted vertically so that the second robotic arm 210 attached to it can be positioned vertically to the patient 240 as part of a laparoscopic procedure. The first and second adjustable arm supports 220 and 225 allow the robotic arms 205 and 210 to approach from different directions, including different heights and lateral positions.

[0106] Although the robot arms 205 and 210 are described as being divided into a first set of robot arms 205 and a second set of robot arms 210, the robot arms 205 and 210 may be divided into other groups (including sets of one or more arms), each configured to perform a separate procedure as part of an associated medical procedure. In some embodiments, the associated procedure (e.g., for diagnosis) may be performed by as few as two arms, one to hold a flexible camera and the other to hold an instrument. In some embodiments, the unassociated procedure (e.g., for treatment) may be performed using two or three arms. In some embodiments, four or more robot arms 205 and 210 may be provided.

[0107] Depending on the combination of medical procedures performed simultaneously, the robotic arms 205 and 210 may be configured and / or operated to control various medical instruments. Examples of applications in which one or more of the robotic arms 205 and 210 may be implemented using the robotic medical system 200 include: (i) a robotic arm configured to control an introducer or sheath that provides access to a natural body orifice such as the nose, mouth, vagina, urethra, rectum, or ear; (ii) a robotic arm configured to control an endoscope and / or endoscopic instrument (e.g., a flexible instrument) through a natural orifice, with or without the aforementioned introducer or sheath; (iii) a robotic arm configured to hold and direct a thoracoscopy or laparoscopy camera (e.g., a flexible or rigid device) that provides extraluminal visualization of the relevant anatomical space (e.g., the thoracic cavity, abdomen, peritoneum, and / or reperitoneal space); and / or (iv) one or more robotic arms configured to hold and direct a thoracoscopy or laparoscopy instrument (e.g., a rigid device). These are merely illustrative uses, and those skilled in the art will understand that the systems described herein are not limited to these implementations.

[0108] There are many advantages to using a single system, such as one of the robotic medical systems shown in Figures 21 and 22, to perform associated endoscopic and laparoscopic procedures using flexible and rigid tools. Firstly, the use of a single system saves the amount of space occupied by the system's components by reducing the equipment / capital in the operating room. Secondly, the use of a single system facilitates a single clinician / physician performing both types of procedures without the need for assistance from other clinicians / physicians in the operating room. And thirdly, by using a single system, each of the system's robot-controllable components can be associated with the same global reference frame—for example, the position and / or orientation of endoscopic instruments can be easily referenced relative to the position and / or orientation of laparoscopic instruments. In other words, a single system provides knowledge of the position and / or orientation data for all instruments and manipulators relative to a single coordinate frame, thereby enabling features such as collision avoidance and / or collision prevention, and / or computer-generated displays of the instruments relative to each other.

[0109] Figure 23 shows yet another embodiment of a robotic system configured to perform ancillary procedures according to aspects of the present disclosure. The robotic medical system 300 includes both a platform-based robotic system 301 and a cart-based robotic system 303. The platform-based robotic system 301 includes a first set of one or more robotic arms 305, a second set of one or more robotic arms 310, and a bed or platform 315, wherein the multiple robotic arms 305 and 310 are arranged on both sides of the platform 315. The system 300 further includes a first adjustable arm support 320 connected to the first set of one or more robotic arms 305, and a second adjustable arm support 325 connected to the second set of one or more robotic arms 310. The cart-based robotic system 303 includes a third set of one or more robotic arms 330 connected to a third adjustable arm support 335. In this embodiment, the platform-based robot system 301 and the cart-based robot system 303 are advantageously integrated to perform ancillary procedures as part of a single procedure episode.

[0110] The first and second sets of robot arms 305 and 310 are configured to control one or more rigid instruments 340, such as a camera, vascular sealer, tissue cutter, stapler, or needle driver, as part of a laparoscopic procedure performed on a patient 350. The third set of robot arms 330 is configured to control one or more flexible instruments 345, such as a colonoscope, bronchoscope, or urethroscope (e.g., having internal and external catheters), as part of an endoscopic procedure. However, in other configurations, any combination or subset of the first robot arms 305, the second robot arm 310, and the third robot arm 330 may be configured to control rigid instruments 340 and / or flexible instruments 345. In some embodiments, the robot system 300 may include two or more cart-based systems 303, each of which may be configured to control one or more medical instruments.

[0111] Figures 24 and 25 show two configurations of another embodiment of a bed-based robotic system configured to perform ancillary procedures according to aspects of the present disclosure. As shown in Figures 24 and 25, the robotic medical system 400 includes one or more robotic arms 405 in a first set, one or more robotic arms 410 in a second set, a platform 415, an adjustable arm support 420, one or more flexible medical instruments 425, and one or more rigid medical instruments 430. In this embodiment, the first set of one or more robotic arms 405 and the second set of one or more robotic arms 410 share the same adjustable arm support 420. An imaging device (e.g., CT, fluoroscopy, etc.) is positioned on the side of the platform 415 opposite the arm support 420. The system 400 further includes an electromagnetic field generator 418 for assisting the navigation of one or more instruments via an EM sensor.

[0112] In the first configuration of the robotic medical system shown in Figure 24, a first set of robotic arms 405 can be configured to control a flexible medical instrument 405 inserted through the tracheal canal of a patient 440. A second set of robotic arms 410 can be housed beneath a platform 415 without controlling any medical instruments. In the second configuration shown in Figure 25, the first set of robotic arms 410 can be configured to control a flexible instrument 425, while the second set of robotic arms 410 can be configured to control one or more rigid instruments 430 that can be lifted from their housing position and inserted through an incision formed in the patient 440. The robotic arms can be raised via adjustable arm supports 420. As shown in Figure 25, an armrest 422 can be connected to one or more of the platform and / or adjustable arm supports 420 to keep the patient's arm still during the procedure. As shown in Figures 24 and 25, different subsets of robot arms 405 and 410 can be selected to control medical instruments 425 and 430, depending on how the robotic medical system 400 is configured or utilized. Specifically, two robot arms 405 may be included as part of a first set of robot arms 405 to control the flexible instrument in the configuration of Figure 24, while a single robot arm 405 may be included as part of a first set of robot arms 405 to control the flexible instrument in the configuration of Figure 25. Different combinations of robot arms 405, 410 on a given adjustable arm support 420 are possible, as shown in Figures 24 and 25.

[0113] Figure 26 is a flowchart illustrating an exemplary method operable by a robotic system or its components for performing at least partially or entirely associated medical procedures according to aspects of the present disclosure. For example, the steps of Method 500 illustrated in Figure 26 may be performed by a medical robotic system (e.g., robot-controllable system 10, or one of the robotic medical systems 200, 300, or 400 described above) or a processor and / or other components of an associated system. For convenience, Method 500 will be described as being performed by a “system” in connection with the description of Method 500.

[0114] Method 500 begins in block 501. In block 505, the system can control a first robotic arm to insert a first medical instrument through a first orifice of the patient. In block 510, the system can control a second robotic arm to insert a second medical instrument through a second orifice of the patient. The first and second robotic arms may be part of a first platform, and the first and second orifices may be positioned in two different anatomical regions of the patient. Method 500 ends in block 515.

[0115] As an example of an embodiment of Method 500, and referring to the embodiment in Figure 21, the robotic medical system 200 controls a first set of robotic arms 205 in block 505 to insert a flexible medical instrument 230 through a first opening in the patient 240. Similarly, in block 510, the robotic medical system 200 controls a second set of robotic arms 210 to insert a rigid medical instrument 230 through a second opening in the patient 240. In some embodiments, the first opening may be a natural opening in the patient, and the second opening may be an incision formed in the patient.

[0116] In some embodiments, the first medical device may include a first imaging device (e.g., an endoscope), and the second medical device may include a second imaging device (e.g., a laparoscope). By inserting the first and second medical devices (each having a camera or other imaging component) through different openings located in two different anatomical regions of the patient, it is possible to provide different views of one or more anatomical regions of the patient. For example, when a flexible device is inserted through the patient's colon and a rigid device is inserted into the patient's abdominal cavity, the flexible device may provide a view of a colon polyp from within the colon, while the rigid device may provide a view of the same colon polyp from the abdominal cavity (e.g., from outside the colon). The systems described herein advantageously allow the user to switch between different camera views when viewing a display. In some embodiments, a view from the first imaging device may be superimposed on a view from the second imaging device on a display. In some embodiments, a view from the first imaging device may be arranged in parallel with the second imaging device in a tiling view on a display. Further details regarding camera view operation are provided below.

[0117] Aspects of the present disclosure, including Method 500 in Figure 26, may enable single endoscopic surgery, percutaneous procedures, laparoscopic surgery, and simultaneous combinations thereof. When all three modalities are utilized during a single procedure, one or more subsets of robotic arms may be assigned to drive and control flexible endoscopes and instruments to provide direct visualization and access to lumens within the body, another subset of one or more robotic arms may be assigned to drive and control laparoscopic / thoracoscopic cameras to provide direct visualization within various body cavities, while another subset of one or more robotic arms may be assigned to drive and control rigid, semi-rigid, or flexible instruments within body cavities. The robotic arms may be configured and deployed as needed, if applicable.

[0118] Figures 27A and 27B provide flowcharts illustrating another exemplary method, operable by a robotic system or its components, for performing associated endoscopic and thoracoscopic procedures according to aspects of the present disclosure. For example, the steps of Method 600 shown in Figures 27A and 27B may be performed by a medical robotic system (e.g., robot-controllable system 10, or one of robotic medical systems 200, 300, or 400) or a processor and / or other components of an associated system. For convenience, Method 600 will be described as being performed by a “system” in connection with the description of Method 600.

[0119] Method 600 begins in block 601. In block 605, a bed-based platform may be configured to receive a patient being moved onto the bed by operating room staff. In block 610, the system may deploy a first arm from either the bed or an integrated cart-based system in preparation of endoscopic instruments. In block 615, the first robotic arm receives a flexible instrument that may be loaded onto the first robotic arm by operating room staff. In block 620, under the control of a physician, the system may drive the flexible instrument into the patient's body through a natural bodily orifice.

[0120] In block 625, the system can locate the target pathology using a flexible instrument. In the case of a lung lesion, blocks 620 and 625 may include introducing the flexible instrument into the airway and introducing the flexible instrument to the target (e.g., the lesion of interest) under the control of a physician.

[0121] In block 630, the system can deploy a second robotic arm to perform laparoscopic resection. In some embodiments, block 630 may be performed in response to a determination that the condition is cancerous. In block 645, the second robotic arm can receive thoracoscopic instruments that can be loaded into the first robotic arm by the operating room staff. The operating room staff may also create a thoracoscopic port configured for insertion of the thoracoscopic instruments into the patient. The thoracoscopic instruments may include a rigid camera, in this case a thoracoscope, and thoracoscopic instruments. The thoracoscopic port may include a cannula that provides access to the patient's thoracic cavity.

[0122] In block 640, the system can perform laparoscopic resection of the target using thoracoscopic instruments under the control of a physician. The flexible instrument and thoracoscope provide separate views of the target from the inside and outside of the airway, respectively, and can assist the physician in performing the resection. Once the resection is complete, the physician and / or operating room staff can remove the thoracoscopic instruments and the adjacent thoracoscopic port and remove the flexible instrument from the patient. Method 600 ends in block 645.

[0123] B. Gradual expansion of treatment One advantage of the ability to control both endoscopic and laparoscopic instruments from a single platform (or a hybrid bed-based platform and cart-based system) is the ability to assess the level of invasiveness of the surgical procedure as needed. Different procedures have different degrees of invasiveness. For example, the first type of procedure may be a purely endoscopic resection. The second type of procedure may be an endoscopic resection with laparoscopic assistance. The third type of procedure may be a laparoscopic resection. The systems and methods described herein are advantageous because they allow physicians to easily scale from one type of procedure to another, such as from the first type of procedure to the second type, from the second type to the third type, or from the first type to the third type.

[0124] In some embodiments, a physician may intend to perform a first type of procedure (e.g., pure endoscopic resection) to include a second type of procedure (e.g., endoscopic resection with laparoscopy assistance). In some embodiments, the first type of procedure may be less invasive than the second type of procedure. For example, in the first type of procedure, the physician may attempt to perform the resection endoscopically without making an incision in the patient. In the second type of procedure, the degree of invasiveness increases when ports and openings are introduced into the patient's abdomen to provide laparoscopy assistance. The ports and openings may provide a laparoscope and / or other laparoscopic instruments, for example, to view tissue and positioning, but the resection is still endoscopic, which keeps the rate of complications and recovery time relatively minimal. The systems and methods described herein may advantageously allow a physician to easily and incrementally scale from the first type of procedure to the second type of procedure.

[0125] In some embodiments, a physician may intend a stepwise expansion of procedures to include a second type of procedure (e.g., endoscopic resection with laparoscopic assistance) and a third type of procedure (e.g., laparoscopic resection). In some embodiments, the second type of procedure may be less invasive than the third type of procedure. For example, in the second type of procedure, the physician may attempt to perform endoscopic resection using laparoscopic assistance. In the third type of procedure, the degree of invasiveness increases when the resection is performed laparoscopically. Such resections, e.g., segmental colectomy, may carry higher risks and recovery times than the first and second types of procedures. The systems and methods described herein advantageously enable a physician to easily stepwise expand from the second type of procedure to the third type of procedure.

[0126] One exemplary procedure that can be progressively expanded is colon polypectomy. A physician may initiate treatment by attempting a purely endoscopic resection of a colon polyp. If the purely endoscopic resection fails, the physician can rapidly perform a endoscopic resection of the colon polyp with the assistance of laparoscopic instruments. In some embodiments, the progressive expansion can be carried out without bringing additional personnel or capital equipment into the room. If endoscopic resection remains unsuitable for resection despite the assistance of laparoscopic instruments, the physician can progressively expand the procedure to a fully laparoscopic procedure and then to laparoscopic resection. In this embodiment, performing endoscopic resection with the assistance of laparoscopic instruments may have a higher level of invasiveness than performing a purely endoscopic resection, while performing laparoscopic resection may have a higher level of invasiveness than performing endoscopic resection with the assistance of laparoscopic instruments. The level of invasiveness of a given procedure may be determined based on a number of factors, but is not limited to, the patient's desired or expected recovery time, the presence or absence of an incision for delivering the instrument, the size of the incision required to deliver the medical instrument into the patient's body, the expected post-procedure morbidity, and the expected risk of complications.

[0127] By using a stepwise expansion approach, physicians can attempt minimally invasive procedures first before attempting more invasive ones. For example, by treating colon polyps using a stepwise expansion approach, physicians can advantageously remove part or all of the colon polyp using complete endoscopic resection, thereby potentially reducing the patient's associated recovery time without extending treatment across multiple episodes. While stepwise expansion has been described above in relation to colon polyps, it can be applied to other medical procedures, such as the diagnosis and resection of cancerous nodules.

[0128] Figure 28 is a flowchart illustrating an exemplary method, operable by a robotic system or its components, for performing an associated medical procedure, including a stepwise expansion procedure, according to aspects of the present disclosure. For example, the steps of Method 700 shown in Figure 28 may be performed by a medical robotic system (e.g., robot-controllable system 10, or one of robotic medical systems 200, 300, or 400) or one or more processors and / or other components of an associated system. For convenience, Method 700 will be described as being performed by a “system” in connection with the description of Method 700.

[0129] Method 700 is initiated in block 701. Method 700 may be performed during Method 500, for example, to perform an accompanying medical procedure illustrated in Figure 26 between blocks 505 and 510. However, the timing of performing Method 700 is not limited and it may be performed before block 505, after block 510, or simultaneously with one or more of blocks 505 and 510. In block 705, the system may control a first robotic arm to perform a first medical procedure. The first medical procedure may include locating a target site. Based on the locating of the target site, the system may further select an incision site. The incision site may be used to deliver a second medical instrument used in a second medical procedure in block 710.

[0130] In block 710, in response to a determination that the first medical procedure failed to completely treat the patient's medical condition, the system may control a second medical procedure to perform a second medical procedure to completely treat the patient's medical condition. The second medical procedure may have a higher level of invasiveness than the first medical procedure. The first and second medical procedures are performed simultaneously during a single medical episode. In some embodiments, the first medical procedure and the relatively more invasive second medical procedure are performed using a single platform, such as a cart-based platform or a bed-based platform with multiple arms. In other embodiments, the first medical procedure and the relatively more invasive second medical procedure are performed using multiple integrated platforms, such as a bed-based platform combined with a cart-based platform, or a cart-based platform combined with another cart-based platform. In some embodiments, in response to a determination that a target site within the patient's anatomical structure meets the condition for treatment via the second medical procedure, the system may further control a second robotic arm to perform the second medical procedure. Method 700 ends in block 715.

[0131] C. User interface for controlling multiple medical devices. Another aspect of the present disclosure relates to a user interface that can enable a single user to control all of the robotic arms during an associated procedure. In other words, aspects of the present disclosure relate to a novel single interface that can be used to perform endoscopic interventions using one or more flexible devices, as well as to the use of one or more rigid devices in laparoscopic interventions.

[0132] Figure 29 shows an exemplary console including one or more types of interfaces for controlling a robotic arm according to an aspect of the present disclosure. As shown in Figure 29, the console 800 includes a viewer 805 and a controller 810 which includes two handles (also called positioning platforms) 815, a pendant 820, an armrest 825, and one or more foot pedals 830 configured to receive input from the user's left and right hands.

[0133] Figure 30 shows an enlarged view of the controller shown in Figure 29, according to an aspect of this disclosure. The controller 810 in Figure 30 may be the same as the controller 182 shown in Figure 19. Figure 31 shows an enlarged view of one of the handles shown in Figures 29 and 30, according to an aspect of this disclosure. In some embodiments, the handle 815 includes a button 835 and a finger grip 840. The button 835 provides a user interface that allows the user to activate the end effector of the corresponding medical device. The finger grip 840 can provide an interface that allows the user to grasp the handle 815 and manipulate the position of the handle 815 in six degrees of freedom. The handle 815 can also function as a gimbal that allows the user to manipulate the handle in three degrees of freedom (e.g., pitch, yaw, and roll).

[0134] Figure 32 shows an enlarged view of the pendant shown in Figure 29, relating to an aspect of the present disclosure. The pendant 820 includes an insert / retract joystick 845, a menu button 850, a quick action button 855, a pause button 860, a joint movement and relaxation joystick 865, a snapshot, light, and one programmable button 870, and a five-button cluster button 875. The pendant 820 may be configured to drive a flexible device such as the flexible device 230 in Figure 21.

[0135] Figures 29–32 include two or more types of user interfaces (e.g., a gimbal-based interface and a pendant-based interface), and in some embodiments, the console 800 may include a single type of interface, such as a handle 815 for performing the associated procedures disclosed herein. For example, in some embodiments, the left-hand handle 815 may be configured to control endoscopic instruments, while the right-hand handle 815 may be configured to control laparoscopic instruments. In other embodiments, the left-hand and right-hand handles 815 may be used in two different modes—a first mode configured to control one or more flexible endoscopic instruments and a second mode configured to control one or more rigid laparoscopic instruments. A foot pedal 830 or other buttons on the console 800 may be configured to receive input from the user to switch between the two modes. The use of a single type of interface to control two different instruments (e.g., a flexible endoscope and a rigid laparoscope) is quite novel, as it is more common to use two different types of interfaces to control such a variety of instruments.

[0136] A controller 810, such as the controller 810 in Figure 30 which has a left-hand handle and a right-hand handle 815, may be used to control laparoscopic instruments, but it is not typical to use this type of controller 810 in conjunction with an accompanying endoscopic procedure involving one or more flexible devices. However, it may be desirable to provide the user with a single user interface to control two or more medical instruments (including endoscopic and laparoscopic instruments) through the same interface, thereby eliminating the need for the user to continuously switch between different user input devices when switching between controlling two medical instruments. Therefore, to provide a single interface that can be physically controlled, the controller 810 may be adapted to control endoscopic instruments in addition to laparoscopic instruments.

[0137] Figure 33 is a flowchart illustrating an exemplary method, according to aspects of the present disclosure, that can be operated by a robotic system or its components for performing an associated medical procedure via a single user interface. For example, the steps of Method 900 shown in Figure 33 may be performed by a medical robotic system (e.g., robot-controllable system 10, or one of robotic medical systems 200, 300, or 400) or a processor and / or other components of an associated system. For convenience, Method 900 will be described as being performed by a “system” in connection with the description of Method 900.

[0138] Method 900 begins in block 901. In block 905, the system can use a user interface to operate a first instrument inserted through a first opening in the patient via a first robotic arm. In block 910, the system can use a user interface to operate a second instrument inserted through a second opening in the patient via a second robotic arm. The first and second openings are located in two different anatomical regions of the patient. In some embodiments, the first opening may be a natural opening in the patient, and the second opening may be an incision formed in the patient. The first instrument may be flexible, and the second instrument may be rigid. Method 900 ends in block 915.

[0139] In one embodiment, depending on the system's reception of a user selection for controlling the endoscopic instrument, one of the handles 815 is mapped to control the insertion and withdrawal of the endoscopic instrument, while the other handle 815 is mapped to control the articulation and rolling of the endoscopic instrument. The handle 815 controlling insertion and withdrawal may be tactilely constrained to move in a straight line, and the controller 810 may include a clutch configured to allow the user to adjust the available stroke length for translating the endoscopic instrument in or out of the patient's body. The handle 815 controlling articulation and rolling may be tactilely constrained so that the handle 815 does not move in a planar manner. The position of the other articulation / roll handle 815 may be fixed in the coordinate plane, but rotation, pitch, and yaw may be permitted. Accessory buttons 835 on the articulation / roll handle 815 may be configured to allow the user to irrigate or aspirate the lumen and deliver energy to the endoscopic instrument.

[0140] In the second embodiment, one of the handles 815 may be mapped to control the insertion and withdrawal of the endoscopic instrument, while the other handle 815 is mapped to control the joint movement and rolling of the endoscopic instrument, as in the first embodiment, although the user's driving experience may be slightly modified. One handle 815 may be tactilely constrained to move in a straight line, but instead of requiring the user to use a clutch, the user controller 810 may be configured to allow the user to move their hand away from or towards the target being treated. The magnitude of the user's hand movement may be mapped to the speed at which the endoscopic instrument moves in or out of the patient's body. The remaining handles 815 may be configured to be controlled in a manner similar to that described in the first embodiment.

[0141] A third embodiment may include a controller 810 having a set of secondary interfaces (not shown) in addition to the primary handle 815 shown in Figures 29 to 31. The primary left and primary right handles 815 may be configured similarly to laparoscopic or thoracoscopic control interfaces. The secondary sets of left and right interfaces may be configured to drive endoscopic instruments. One of the secondary interfaces may include a loop of an endoscope-like insertion tube attached to two wheels. The insertion tube loop may be configured to translate along an axis and wind left and right. The insertion tube can provide either position- or velocity-based control. Toggle buttons may be present on the controller 810 to allow the user to cycle through a concentric instrument that needs to be translated or rotated within the patient's body. Another secondary interface may include a pseudo-endoscopic tip enhanced with a series of buttons. The pseudo-endoscopic tip may be configured to be operated by the user to indicate a desired shape for the distal end of a robot-controlled endoscopic instrument. Buttons on the pseudo-endoscopic tip can be used to irrigate or aspirate in the lumen, as well as to deliver energy to the endoscopic instrument. The simulated endoscope tip assumes the current position of the controlled endoscope instrument and maintains that position until otherwise instructed by the user, or behaves like a conventional endoscope instrument and maintains its position when actively instructed by the user. Another secondary interface may include one or more “joystick” type buttons on the handle 815, which may be used to control one or more flexible instruments in a similar manner to the inputs on the pendant 820. In some embodiments, the handle 815 is repositioned itself to an alternative configuration (e.g., facing upward) to facilitate ergonomic control of the secondary interface joystick buttons.

[0142] In the fourth embodiment, the controller 810 may include a secondary interface for translating and rolling the endoscopic instrument as described in the third embodiment, but one of the two primary left-hand or right-hand interfaces may be configured to control joint movements and other functions of the endoscopic instrument.

[0143] In a fifth embodiment, the controller 810 may include a secondary interface, such as a pendant 820, as shown in Figure 32. In this embodiment, the user must physically switch between the left and right handles 815, and the pendant 820 must move between controlling the laparoscope and endoscopic instruments. The current view displayed by the viewer 805 may be controlled by the controller 810.

[0144] In some embodiments, the console 800 is configured to limit the number of robot arms that can be controlled simultaneously while constraining the motion of other robot arms. For example, in some embodiments, the interface may be used to drive a selected robot arm while constraining the motion of other robot arms (e.g., two, three, four, or more).

[0145] D. Viewing and switching between image displays. Referring to Figures 29 to 32, in some embodiments, the viewer 805 may be configured to display images from an endoscopic image sensor and a laparoscopic image sensor, as well as to display any preoperative plan or scan loaded onto a system of raw video such as an ultrasound probe or other source. The user can switch between different views in various ways, but is not limited to: using accessory buttons 835 located on the left-hand interface or the right-hand interface; using an accessory foot pedal 830 or a switch activated by the user's foot, knee, toe or elbow; using tactile-responsive commands in combination with the instrument clutch to switch or cycle through multiple views using hand-based gestures via the left-hand and right-hand interfaces; using a control pendant 825 mounted on the console; or using any combination. Secondary views may be displayed via picture-in-picture, side-by-side, split-frame, or periodic view modes.

[0146] Figures 34 and 35 are illustrative diagrams that may be displayed by a viewer during an incidental medical procedure according to an aspect of the present disclosure. Specifically, Figure 34 shows a thoracoscopic view 1000, and Figure 35 shows an endoscopic view 1050.

[0147] In some embodiments, the control of the currently selected medical instrument is synchronized with the coordinate frame of the primary view displayed by viewer 805. The system may consider instruments originating from the approach of the primary view displayed as primary instruments, but the system can allow the user to control any instrument on the system relative to the primary view. In other words, in a scenario where the user is in thoracoscopic view mode 1000, the thoracoscopic instruments are primary instruments. As shown in thoracoscopic view mode 1000 in Figure 34, the first thoracoscopic instrument 1005 and the second thoracoscopic instrument 1010 can be seen.

[0148] The user may wish to adjust the endoscope placed in the lung. To do this, the user can display an appropriate secondary view (in this case, the endoscope view 1050) and, using one of the interfaces described above in "Section C," switch from a primary left-hand or right-hand instrument to a secondary left-hand or right-hand instrument and adjust the endoscopic instrument as needed. As shown in Figure 35, the endoscope view 1050 may include a first view 1055 of the camera on the endoscope and a second view 1060 showing the position of the tip of the endoscope relative to the preoperative model. In addition to switching between the thoracoscopic view 1000 and the endoscope view 1050, the system may further allow switching between the thoracoscopic view 1000, the endoscope view 1050, and a third view obtained via a preoperative scan of the patient.

[0149] Figure 36 is another exemplary diagram that may be displayed by a viewer during an accompanying medical procedure according to an aspect of the present disclosure. In some embodiments, the system may be configured to have computer-generated overlays of one image on top of each other, as shown in Figure 36. In fact, the embodiment shown in Figure 36 shows two separate view-in-view embodiments. First, an endoscopic view from the flexible scope (upper right corner) is overlaid on the laparoscopic view 1090 from the rigid scope (base image). Second, a graphic or virtual representation 1095 of the flexible scope (in contour) is also overlaid on the laparoscopic view 1090 from the rigid scope. By providing such view-in-view functionality, this helps physicians perform multiple procedures simultaneously, minimizing the need for unnecessary personnel to perform individual procedures.

[0150] Figure 37 is a flowchart illustrating an exemplary method, operable by a robotic system or its components, for switching displayed images while performing an associated medical procedure, according to aspects of the present disclosure. For example, the steps of Method 1100 shown in Figure 37 may be performed by a medical robotic system (e.g., robot-controllable system 10, or one of robotic medical systems 200, 300, or 400) or a processor and / or other components of an associated system. For convenience, Method 1100 will be described as being performed by a “system” in connection with the description of Method 1100.

[0151] Method 1100 begins in block 1101. In block 1105, the system can deliver a first scope through a first opening in the patient via a first robotic arm to acquire a first image. In block 1110, the system can deliver a second scope through a second opening in the patient via a second robotic arm to acquire a second image. In block 1115, the system can switch between the first and second images on the display. In some embodiments, the system may be further configured to switch between views in which the first image is superimposed on the second image on the display. In other embodiments, the system may be configured to switch between a first image obtained from a preoperative scan of the patient, a second image, and a third image obtained from a third image (e.g., computed tomography (CT) scan or fluoroscopy scan). The system may be further configured to overlay a virtual image on either the first or second image on the display, as shown in Figure 36. Method 1100 ends in block 1120.

[0152] 3. Exemplary methods and workflows for associated procedures There are many exemplary methods and workflows that can be enabled by the robotic medical systems described herein. One particular exemplary medical procedure that can be performed using the robotic medical systems described herein is a colorectal intervention. In a colorectal intervention, a physician can attempt to address a colorectal disease by performing a minimally invasive intervention that could otherwise be a very complex procedure. In some cases, treatment for a colorectal disease may include the removal of a colon polyp. While aspects of this disclosure may use a colorectal intervention as a specific example, the systems and methods described herein can also be used in other procedures such as gastrointestinal and thoracic interventions.

[0153] CELS can be used for procedures such as colorectal and gastrointestinal interventions. For example, certain colonic polyps (such as large sessile polyps or those located in difficult-to-access areas within the colonic lumen) cannot be resected using purely endoscopic techniques with current methods. These polyps can be resected endoscopically with the assistance of laparoscopic instruments. In such cases, the colon can be repositioned using laparoscopic instruments to enable endoscopic resection of the polyp. In addition, or as an alternative procedure, endoscopic tools can be used for laparoscopic resection to locate polyps and / or preserve polyp tissue to enable targeting.

[0154] Since laparoscopic-assisted endoscopic resection and endoscopic-assisted laparoscopic resection can each be less invasive than pure laparoscopic resection (for example, they can shorten the patient's recovery time), the use of endoscopy with CELS and combined endoscopic / laparoscopic procedures may potentially lead to better outcomes for patients with polyps.

[0155] However, there are several challenges to actually implementing CELS. One challenge is that CELS can often require several resources and multiple personnel, such as at least four physicians, to (i) control the laparoscopic camera, (ii) control the laparoscopic instruments, (iii) control the endoscopic camera, and (iv) control the endoscopic instruments. In addition to four physicians, CELS may also require several clinicians and assistants to support certain situations. Therefore, CELS is not widely adopted and is limited to specific facilities (e.g., high-end academic centers) that can support multiple clinicians for a single procedure.

[0156] Figures 38A and 38B include flowcharts illustrating exemplary workflows for performing CELS according to aspects of this disclosure. For example, the steps of workflow 1200 shown in Figures 38A and 38B may be performed by one or more physicians, clinicians, and / or assistants. For convenience, CELS workflow 1200 is described as being performed by four physicians in connection with the description of workflow 1200.

[0157] Referring to Figure 38A, workflow 1200 begins in block 1201. In block 1205, workflow 1200 includes one or more physicians introducing an endoscope (e.g., a colonoscope) and one or more other endoscopic instruments into the patient's colon. In block 1210, workflow 1200 includes one or more physicians using the endoscope and one or more other endoscopic instruments to identify and characterize target anatomical structures. In block 1215, workflow 1200 includes one or more physicians positioning multiple laparoscopic ports in the patient and introducing a laparoscope and one or more laparoscopic instruments into the patient. These physicians assist in establishing a sterile boundary between the sterile field (e.g., laparoscopic ports) and the non-sterile area (e.g., an endoscopic access point, e.g., the patient's anus). The laparoscope and one or more laparoscopic instruments may be inserted into the patient's abdominal cavity.

[0158] Referring to Figure 38B, in block 1220, workflow 1200 includes positioning a target anatomical structure for intervention using interchangeable endoscopic cameras and instruments and laparoscopic cameras and instruments. The physician may view different screens for the endoscopic view and the laparoscopic view. In other embodiments, the same screen may view both the endoscopic view and the laparoscopic view. In block 1225, workflow 1200 includes optionally exchanging one or more of the endoscopic instruments and laparoscopic instruments. In block 1230, workflow 1200 includes performing an intervention on the target anatomical structure using one or more of the endoscopic instruments and laparoscopic instruments. In block 1235, workflow 1200 includes removing all of the endoscopic cameras and instruments and laparoscopic cameras and instruments and closing the laparoscopic port. The workflow ends in block 1240.

[0159] Aspects of this disclosure relate to an accompanying system capable of performing CELS. The system is capable of performing both endoscopic procedures (e.g., procedures involving the use of one or more flexible instruments) and laparoscopic procedures (e.g., procedures involving the use of one or more rigid instruments). The endoscopic and laparoscopic capabilities are advantageously integrated into the accompanying system in a relatively compact form factor. In addition to the compact form factor, the system can be controlled by a single control unit, thereby advantageously reducing reliance on a large number of physicians and assistants in the operating room. By reducing reliance on a large number of physicians and assistants when performing CELS, this makes it possible to widely use the system in numerous operating rooms and emergency rooms in hospitals worldwide without the aforementioned drawbacks of conventional CELS.

[0160] Figure 39 shows one embodiment of a bed-based robotic system configured to perform ancillary procedures according to aspects of the present disclosure. For example, system 1300 can be used in a method for performing CELS procedures. The system may be similar to the system described in relation to Figure 21.

[0161] As shown in Figure 39, the system 1300 includes a patient platform 1305, which includes a bed 1307 to which multiple robotic arms 1310 and 1315 are attached to adjustable arm supports 1311 and 1316. The system 1300 further includes a display 1321 which can be configured to display a live video stream. In this embodiment, the display 1321 includes a television screen. In other embodiments, the display 1321 may be a screen that is part of a tower monitor, and the video stream may be further routed to an external room monitor via HDMI, SDI, or similar means.

[0162] One or more of the first set of robot arms 1310 are configured to operate one or more flexible instruments. In some embodiments, the flexible instruments include flexible scopes such as endoscopes and one or more endoscopic instruments. In the embodiment of Figure 39, the first set of robot arms 1310 includes two arms, but in other embodiments, more or fewer robot arms 1310 can be used to control endoscopes and / or endoscopic instruments. One or more of the second set of robot arms 1315 are configured to operate one or more rigid instruments. In some embodiments, the rigid instruments include rigid scopes such as laparoscopes and one or more laparoscopic instruments. In the embodiment of Figure 39, the second set of robot arms 1315 includes three arms, but in other embodiments, more or fewer robot arms 1315 can be used to control laparoscopes and / or laparoscopic instruments.

[0163] Display 1321 may include a large video screen configured to display feedback from a flexible scope and / or a rigid scope. In some embodiments, the feedback may include live video streams from a laparoscope and / or endoscope. In other embodiments, the feedback may include a virtual representation of the location of laparoscopic and endoscopic instruments relative to a model of the patient's anatomical structure. In the embodiment of Figure 39, system 1300 is configured to display both live streams from the laparoscope and the endoscope simultaneously. For example, feedback from the flexible scope and feedback from the rigid scope may be displayed on the video screen in a picture-in-picture view. In another embodiment, feedback from the flexible scope and feedback from the rigid scope may be displayed on the video screen in a side-by-side view.

[0164] Figure 39 shows one embodiment in which the robot arms 1310 and 1315 are mounted on a platform 1305, but in an alternative embodiment, the system 1300 may include one or more carts, each containing one or more robot arms. The carts can communicate with each other so that a physician can control flexible and rigid instruments from a central location. In one embodiment, robot arm 1310, configured to operate flexible instruments, may be mounted on a first cart, while robot arm 1315, configured to operate rigid instruments, may be mounted on a second cart. Embodiments of systems including cart-based robot arms are shown in Figures 1, 3, 4, and 23, but are not limiting.

[0165] Figure 40 is a flowchart illustrating another exemplary workflow for performing CELS according to an aspect of this disclosure. For example, the steps of workflow 1400 shown in Figure 40 may be performed by one or more physicians, clinicians, and / or assistants. Some of the steps of method 1400 in Figure 40 may be performed by a medical robotic system (e.g., robot-controllable system 10, or one of the robotic medical systems 200, 300, 400, 1300, or 1500 discussed above) or a processor and / or other components of an associated system. Certain steps of method 1400 may also be performed by the system in response to commands received, for example, from a physician, via an input device (e.g., as shown in Figures 29 to 32), or may be performed automatically by the system without user intervention.

[0166] Referring to Figure 40, workflow 1400 begins in block 1401. In block 1405, workflow 1400 includes operating a flexible instrument using a first robotic arm of the robotic system. For example, a physician may input a first command into the accompanying system 1300 in Figure 39 to operate the flexible instrument. In some embodiments, the physician may operate the flexible instrument using the first robotic arm through the patient's natural orifice. In block 1410, workflow 1400 includes operating a rigid instrument using a second robotic arm of the robotic system. A physician may input a second command into the accompanying system to operate the rigid instrument. In some embodiments, the physician may operate the rigid instrument using the second robotic arm through an incision formed in the patient. The physician may be able to switch control between the flexible and rigid instruments. In other embodiments, the physician may use separate input devices for controlling the flexible and rigid instruments. Exemplary input devices and interfaces that can be used by a physician are shown in Figures 29–32.

[0167] In block 1415, workflow 1400 includes displaying feedback from a flexible instrument. Feedback from a flexible instrument may be displayed, for example, by display 1321 in Figure 39. In block 1420, workflow 1400 includes displaying feedback from a rigid instrument. As described in detail below, feedback from flexible instruments and feedback from rigid instruments may be displayed in picture-in-picture view, side-by-side view, and / or feedback from only one of the flexible and rigid instruments may be displayed at a time. The physician may be able to control how the feedback is displayed. The physician may also be able to select feedback from one of the flexible and rigid instruments as the primary view, and the other feedback as the secondary view. The system may display the primary and secondary views on the same view screen of the display. Method 1400 ends in block 1425.

[0168] Figure 41 shows another embodiment of a robotic system configured to perform ancillary procedures according to aspects of the present disclosure. Specifically, the system 1500 of Figure 41 may be configured to perform a colorectal intervention using a CELS procedure.

[0169] The system 1500 includes robotic arms 1510 and 1515 positioned on a pair of adjustable arm supports 1411 and 1416, thereby enabling the robotic arms 1510 and 1515 to engage on both sides of the patient 1503. The adjustable arm supports 1411 and 1416 can be coupled to a platform 1507 or a bed. The first pair of robotic arms 1510 is used to control one or more flexible instruments, such as one or more endoscopes and / or endoscopic tools. In this embodiment, one of the first pair of robotic arms 1510 controls an endoscope 1525 having one or more working channels, while the other of the first pair of robotic arms 1510 controls an endoscopic tool 1530 through the endoscope 1525.

[0170] The endoscopic tool 1530 may include, but is not limited to, one or more polyp removers or receivers, including snares (e.g., polyp snares), forceps, nets, grippers, baskets, balloons, injectors (e.g., dyes, markers, or therapeutic substances), ablation probes, and wires, all of which may or may not be capable of delivering interventional energy, such as electrosurgical energy. In addition, the endoscopic tool 1530 may include imaging modalities such as ultrasound imaging or fluorescence imaging. In some embodiments, the endoscope 1525 and / or endoscopic tool 1530 can be used to visualize and assist in the removal of one or more polyps, for example, via one or more cameras mounted on the distal end of the endoscope 1525 and / or endoscopic tool 1530. In other embodiments, the endoscope 1525 and / or endoscopic tool 1530 can be used for localization and / or tissue-preserving laparoscopic resection to enable targeting.

[0171] In the embodiment shown in Figure 41, the second set of four robotic arms 1515 are used to control one or more rigid instruments, such as one or more laparoscopes and / or laparoscopic tools. For example, one of the robotic arms 1515 of the second set controls a laparoscope 1535 for viewing the abdomen of the patient 1503, while the other three robotic arms 1515 of the second set control laparoscopic tools 1540, each delivered through a laparoscopic port (not shown) located on the patient 1503. Laparoscopic tools 1540 can include, but are not limited to, one or more different gripping instruments, retractors, and / or other tools for manipulating whole tissues, cutting and dissecting tools, such as paddle forceps, Maryland forceps, scissors, hooks, etc., which may or may not be capable of delivering electrosurgical energy, as well as ligation and suturing tools such as staplers, container sealers, needle drivers, and automatic suturing devices. In some embodiments, the laparoscope 1535 and / or laparoscopic tool 1540 can be used to visualize and assist in the removal of one or more polyps. In other embodiments, the laparoscope 1535 and / or laparoscopic tool 1540 can be used to reposition the colon, enabling better endoscopic resection.

[0172] In certain embodiments, the robotic arms 1510 and 1515 are retracted when not in use. For example, a CELS procedure may be initiated by the use of a flexible instrument controlled by the first robotic arm 1510 during the initial stage or during the procedure. In some embodiments, the system may introduce the flexible instrument to the patient through the patient's natural orifice. The system can manipulate the flexible instrument through the natural orifice using the first robotic arm of the robotic system.

[0173] If pure endoscopic resection is unsuccessful in completely treating the condition (e.g., colon polyp), the system may receive an input signal via a user input device and deploy a second robotic arm 1515. In response to receiving the input signal, the system may deploy the second robotic arm 1515 of the robotic system from its stowed position to its setup position. The deployment of the additional robotic arm may be performed as part of a stepwise expansion of the procedure (e.g., as described in section 2.B. Stepwise Expansion of the Procedure).

[0174] The system can then manipulate a rigid instrument using the second robotic arm 1515 of the robotic system through an incision formed in the patient. The system can also display feedback from at least one of the flexible and rigid instruments during the CELS procedure.

[0175] Figure 42 is an exemplary still image captured from a video screen of an accompanying system during a simulated colorectal intervention according to an aspect of the present disclosure. Still image 1600 includes live video from a laparoscope 1605 and live video from an endoscope formatted in picture-in-picture view 1610. Live video from the laparoscope 1605 shows a view of the colon 1615 from the patient's abdomen, as well as a pair of laparoscopic end effectors 1620, each attached to a laparoscopic tool. Live video from the endoscope 1610 shows an internal view of the colon.

[0176] Advantageously, the operator (e.g., a physician, assistant, etc.) can switch the view displayed on the video screen and / or the format in which the view is displayed. In other embodiments, instead of providing two different live images 1605 and 1610 on the video screen, the user can provide a command to freeze one of the live images 1605 and 1610 to make it a still image, while keeping the other live image 1605 and 1610 live. In other embodiments, instead of providing two different live images 1605 and 1610, the user can select one of the display images to present a virtual view (e.g., a virtual representation of the target anatomical structure, which can be generated based on preoperative imaging of the target anatomical structure), while the other image can be one of the live images 1605 and 1610. In some embodiments, the virtual view may include a view of an endoscopic camera and / or instrument, or a laparoscopy camera and / or instrument, based on the measured joint and encoder positions of the robotic arm and manipulator. In the embodiment shown in Figure 42, the live video 1605 from the laparoscope is prominently displayed and can be considered the "primary" video, while the live video 1610 from the endoscope is displayed in a corner area and can be considered the "secondary" video. The user may be able to switch between the primary and secondary views so that the live video 1610 from the endoscope includes the primary view and the live video 1605 from the laparoscope includes the secondary view.

[0177] The accompanying systems described herein can be used to provide several improved methods and workflows for accompanying CELS. The following sections describe some of the advantages and improvements that can be achieved using the workflows enabled by the accompanying systems. The workflows described herein can be performed sequentially and / or simultaneously, depending on the specific CELS procedure being performed (based on, for example, the type of procedure, the patient, etc.).

[0178] A. Patient preparation Prior to performing a CELS procedure using the accompanying systems disclosed herein, a physician, clinician, and / or assistant may perform several preoperative preparatory procedures. One such procedure may include preparing the patient.

[0179] Patient preparation may include any steps related to preparing the patient before surgery (e.g., before the incision is made). This may include, for example, the clinician, assistant, and / or physician identifying the sterile and non-sterile areas of the patient. For example, in preparation for a colonic intervention, the physician may identify the sterile area (e.g., the area of ​​the patient's abdomen where the laparoscopic port will be placed) and the non-sterile area (e.g., the anus), and apply sterile drapes to form a sterile boundary between the sterile and non-sterile areas.

[0180] The methods for preparing a patient can be significantly improved using the accompanying systems described herein. Figure 43 is a flowchart illustrating an exemplary patient preparation procedure to be performed by CELS according to an aspect of this disclosure. For example, the steps of Method 1700 shown in Figure 43 may be performed by one or more physicians, clinicians, and / or assistants. Some of the steps of Method 1700 in Figure 43 may be performed by a medical robotic system (e.g., the robot-controllable system 10 discussed above, or one of the robotic medical systems 200, 300, 400, 1300, or 1500) or a processor and / or other components of an associated system. Certain steps of Method 1700 may also be performed by the system in response to commands received, for example, from a physician, via an input device (e.g., as shown in Figures 29 to 32), or automatically by the system without user intervention.

[0181] Referring to Figure 43, Method 1700 begins in block 1701. In block 1705, Method 1700 includes establishing a sterile boundary. In some embodiments, a system processor can be used to define and identify the sterile boundary. In block 1710, Method 1700 includes the processor identifying a sterile zone and a non-sterile zone. In some embodiments, the processor can also identify a transition zone between the sterile and non-sterile zones. In some embodiments, different zones can be identified via a visual boundary.

[0182] In block 1715, method 1700 includes the processor maintaining a first robotic arm within a sterile zone and a second robotic arm within a non-sterile zone. Thus, the processor can prevent the robotic arms or instruments from entering an undesirable zone (e.g., from a sterile zone to a non-sterile zone, or vice versa). In some embodiments, the processor can identify a first robotic arm as a “sterile” robotic arm that can remain within the sterile zone and a second robotic arm as a “non-sterile” robotic arm that can remain within a non-sterile zone, thereby preventing these robotic arms from entering an undesirable area. In some embodiments, the processor can prevent the first and second robotic arms from moving into a transition zone, thereby leaving a buffer between the zones. In some embodiments, the processor can alert the user via visual or auditory cues that a robotic arm is moving into a transition zone or a non-sterile zone. In some embodiments, the processor can instruct the user to confirm that the robotic arms continue to move into a transition zone or a non-sterile zone. Method 1700 ends in block 1720.

[0183] B. Air supply During CELS procedures, it may be desirable to insufflate one or more areas of the patient's body. For example, the abdomen can be insufflated or inflated (e.g., via a tube that pumps air through one or more cannulas / ports placed within the abdomen). In addition, the colon can be insufflated or inflated (e.g., via a tube that pumps air through the working channel of the endoscope). In colonic interventions, maintaining a proper balance between the two insufflation areas can be challenging. For example, abdominal insufflation can cause compression of organs (e.g., the colon), thereby making visualization of the colon difficult. Insufflation of two different areas (here, the abdomen and the colon) can often compete with each other.

[0184] Figure 44 is a flowchart illustrating an exemplary air supply procedure for performing CELS according to an aspect of this disclosure. For example, the steps of Method 1800 shown in Figure 44 may be performed by one or more physicians, clinicians, and / or assistants. Some of the steps of Method 1800 in Figure 44 may be performed by a medical robotic system (e.g., the robot-controllable system 10 discussed above, or one of the robotic medical systems 200, 300, 400, 1300, or 1500) or a processor and / or other components of an associated system. Certain steps of Method 1800 may also be performed by the system in response to commands received, for example, from a physician, via an input device (e.g., as shown in Figures 29 to 32), or may be performed automatically by the system without user intervention.

[0185] Referring to Figure 44, Method 1800 begins in block 1801. In block 1805, the processor operates a flexible instrument through a first area of ​​the patient and a rigid instrument through a second area of ​​the patient. In some embodiments, the flexible instrument is operated through a first opening (e.g., a natural opening such as the patient's anus), and the rigid instrument is operated through a second opening (e.g., an artificial incision that may be formed in the patient's abdomen).

[0186] In block 1810, the processor can deliver air to the patient in at least one of the patient's first region and the patient's second region. In some embodiments, the processor can deliver air to both the patient's first region and the patient's second region.

[0187] In block 1815, the processor may optionally adjust the inflation of a second region based on measurements of inflation in a first region, or vice versa. For example, the processor may determine when sufficient inflation has been achieved in two different regions, thereby creating an appropriate balance of inflation between the regions. In some embodiments, one or more tubes configured to pressurize air into the patient may also include pressure sensors configured to take measurements of the current inflation in the corresponding regions of the patient. Thus, the processor may be able to determine the inflation in each of the first and second regions based on measurements from the pressure sensors. In addition, the system may include video with live images so that a physician can instruct the CELS system to increase the inflation amount in any of the multiple regions in real time. Method 1800 concludes in block 1820.

[0188] The improved CELS system described above advantageously allows for the integration of the insufflation mechanism and techniques into the proposed system, thereby making it easier to provide whole-patient inflation management. In contrast to conventional CELS procedures, where one clinician may have to manage inflation in one area (e.g., the abdomen) and another clinician in another area (e.g., the colon), the insufflation procedure described allows the CELS system to enable dynamic user control of the volume of air insufflated in multiple patient areas (e.g., both the abdomen and / or the colon), thereby improving endoscopic and laparoscopic visualization.

[0189] C. Imaging As previously explained, feedback from one or more flexible and rigid instruments may be displayed on the video screen during CELS procedures. In some cases, internal images of different anatomical and target areas (e.g., cancerous sites) may be difficult to distinguish from other parts of the patient's anatomical structure displayed on the video screen. Visualizing these anatomical and target areas can be assisted by the introduction of visual or fluorescent markers. For example, visual or fluorescent markers (such as indigo carmine solution, methylene blue, indocyanine green, or other compounds) can be injected into the patient to better identify and characterize the region.

[0190] Figure 45 is a flowchart illustrating an exemplary imaging procedure for performing CELS according to an aspect of the present disclosure. For example, the steps of Method 1900 shown in Figure 45 may be performed by one or more physicians, clinicians, and / or assistants. Some of the steps of Method 1900 in Figure 45 may be performed by a medical robotic system (e.g., the robot-controllable system 10 discussed above, or one of the robotic medical systems 200, 300, 400, 1300, or 1500) or a processor and / or other components of an associated system. Certain steps of Method 1900 may also be performed by the system in response to a command received, for example, from a physician, via an input device (e.g., as shown in Figures 29 to 32), or automatically by the system without user intervention.

[0191] Referring to Figure 45, Method 1900 begins in block 1901. In block 1905, the processor may introduce a marker to a target area of ​​the patient using one of a flexible instrument and a rigid instrument. In some embodiments, the marker is delivered into the patient's vein. Examples of markers include visual markers or fluorescent markers (e.g., indigo carmine solution, methylene blue, indocyanine green, or other compounds). In block 1910, the processor may display feedback, including the target area, from one of the flexible instrument and the rigid instrument on a video screen. Thus, the physician may be able to visualize the target area through the image displayed on the video screen. Method 1900 ends in block 1915.

[0192] The improved CELS system described above can offer several advantages, including the ability to communicate and transfer information between different procedural modalities (e.g., endoscopic and laparoscopic modalities). For example, in one embodiment, a fluorescently stained area that could previously only be visualized by laparoscopy can be visualized by using an endoscope to display live video streams from one or more endoscopes and laparoscopes, and vice versa. In another embodiment, the fluorescent dye can be delivered intravenously so that both the endoscope and laparoscope can visualize the same stained area. Thus, the improved CELS system makes it easier to visualize fluorescently stained or marked areas using both an endoscope and a laparoscope. In addition to the use of fluorescence imaging, the system can use narrowband imaging, which can also benefit from having an integrated endoscope and / or laparoscope to identify and characterize lesions.

[0193] D. Navigation Navigation using the accompanying systems described herein may involve driving one or more scopes and / or instruments through the patient's anatomical structure. Flexible endoscopes and their associated endoscopic instruments can be driven in novel ways using the accompanying systems described herein. In some embodiments, the flexible instrument is configured to be driven through an outer sheath, for example, a flexible colonoscope. The flexible instrument may include one or more working channels configured to facilitate the delivery of surgical instruments through the interior. For example, in a colorectal intervention, the flexible colonoscope may be visualized as a “mother” instrument, and any instrument within the colonoscope passing through the working channels of the colonoscope may be visualized as a subordinate “daughter” instrument, and such mother and daughter instruments can be navigated together.

[0194] In another embodiment, the outer sheath can be visualized as the "mother" and colonoscope, and any instrument passing through the colonoscope can be visualized as a subordinate "daughter" advancing through the outer sheath. In other words, the improved CELS system allows for the navigation of elongated components (e.g., the outer sheath, scope, and working instruments).

[0195] E. Integration of flexible and rigid fixtures The improved CELS system described herein uniquely integrates both flexible and rigid instruments. The system enables navigation and control of both types of instruments, as described herein, thereby enabling new procedures, including the gradual expansion of such procedures.

[0196] In one example of a procedure involving the control of both flexible and rigid instruments, the flexible instruments (such as an outer sheath, inner sheath, and working instrument) can be driven by a robotic arm in an initial attempt to excise a target anatomical structure. Following the initial attempt, laparoscopic ports and instruments can be provided as desired to perform the excision as desired. In some embodiments, the laparoscopic ports can be approximately 3 to 14 mm (to accommodate, for example, a 12 mm laparoscope, an 8 mm instrument, and an instrument of about 3 mm). In some embodiments, a larger hand port or gel port can be used instead of, or in addition to, a standard laparoscopic port. When using both an endoscope and laparoscopic scopes and tools, the improved CELS system described herein allows switching control between the endoscope and each of the laparoscopic scopes and tools, thereby enabling the user to control both, for example, using one or more controllers.

[0197] Advantageously, in some embodiments, both endoscopes and laparoscopes and tools can be controlled using the same controller (e.g., one or more handles 815 shown in Figure 31, or a single input device such as the pendant 820 shown in Figure 32). The single input device can be any type of controller, including a multi-DOF (e.g., 7-DOF) master controller (see, for example, the master controller 810 shown in Figure 30) equipped with a gimbal or gamepad-type device (e.g., the pendant 820). The control of endoscopes, flexible endoscopic instruments, laparoscopes, and rigid laparoscope instruments can be switched using the single input device.

[0198] In other embodiments, multiple controllers can be used to control both endoscopes and laparoscopes and tools. An advantage of having multiple controllers is that it allows multiple users (e.g., a pair of physicians) to simultaneously control different aspects of the CELS system as desired. In addition, another advantage of having multiple controllers is that the controllers can be located in different places—for example, one controller may be on the surgical console and another at the bedside. In one embodiment, a pair of users can synchronously control rigid and flexible instruments, so that one can control rigid instruments from a multi-DOF (e.g., 7-DOF) master on the surgical console and the other can control flexible instruments from a bedside pendant. In another embodiment, a pair of users can synchronously control rigid and flexible instruments, so that one can control rigid instruments using one type of controller and the other can control flexible instruments using the same type of controller.

[0199] However, in other embodiments, a single user may input commands to control rigid and flexible instruments via a pendant and a master controller. For example, a single user may input commands to control rigid instruments using one type of controller and commands to control flexible instruments using another type of controller. This may allow the user to switch control of endoscopic and laparoscopic instruments by inputting commands to separate devices rather than changing the system's input mode.

[0200] F.UI / UX In conventional CELS procedures, one or more surgeons may view images from the endoscope on a first screen and images from the laparoscope on a second, separate screen. Surgeons often rotate their heads forward and backward to analyze different views from the endoscope and laparoscope.

[0201] The improved CELS system described herein can enable the establishment of live video streams from one or both a laparoscope and an endoscope (e.g., a colonoscope) through a single video pipeline. To enable this, the system may include two independent video processors that route video received from the endoscope and laparoscope to an image output path. The image output path can output to a tower monitor and / or an operating room monitor. By controlling both video streams using a single CELS system, the user can decide whether a given monitor should display a laparoscopic view, an endoscopic view, or both (e.g., picture-in-picture or side-by-side). Thus, the system can be configured to display feedback from the laparoscope and endoscope in picture-in-picture view and / or side-by-side view.

[0202] The improved CELS system can be configured to switch between laparoscopic and endoscopic views on any connected display unit, which can be a tower monitor, operating room monitor, physician console display, or an alternative third-party bedside visualization unit. In some embodiments, the processor may identify one of the feedback from the endoscope and the other from the laparoscope as the primary view and the other as the secondary view. The processor can then switch between the primary and secondary views based on input received from the user.

[0203] Switching views on any of the connected display units may include (i) switching between a full-screen laparoscopic view and an endoscopic view on a single monitor, (ii) switching between a full-screen laparoscopic or endoscopic view and picture-in-picture or side-by-side (or any combination thereof), or (iii) switching which monitor is displaying which content (for example, switching monitor A from a laparoscopic view to an endoscopic view, and monitor B from an endoscopic view to a laparoscopic view, or vice versa). Thus, the system may allow any combination of views to be displayed independently on each display unit.

[0204] G. Exemplary advantages and improvements The aforementioned system, which uses a robotic arm to control endoscopic instruments, laparoscopic instruments, or both, offers several advantages. One advantage is the ability to robotically control one or both tool types (flexible / endoscopic and rigid / laparoscope), which reduces the need for a large number of doctors and personnel to perform a particular procedure. For example, instead of three doctors remaining stationary while a fourth doctor manipulates the instrument, a robotic arm can hold a static instrument in place. In such a case, a flexible instrument can be manipulated to snare a polyp while laparoscopic instruments, laparoscopes, and colonoscopes remain stationary.

[0205] In addition, the system allows the robotic arm to control the instrument, enabling motion and sensor-based determination of the instrument shaft and tip location. This information can always allow the user to view a graphics-based rendering of the location and position of the robotic arm and instrument. This can include a 3D volume rendering of the robot and instrument, a 2D line drawing of the robot and instrument, or a visual overlay of the flexible instrument from within a rigid laparoscope (and vice versa, as shown in Figure 36).

[0206] 4. Implementation System and Terminology The implementations disclosed herein provide systems, methods, and apparatus for performing associated medical procedures.

[0207] When used herein, the terms “to connect,” “connected,” “linked,” or other variations of the word “connection” may indicate either indirect or direct connection. For example, when a first component is “connected” to a second component, the first component may be indirectly connected to the second component via another component, or directly connected to the second component.

[0208] Functions relating to the systems, methods, and workflows for performing the incidental procedures described herein may be stored as one or more instructions in processor-readable or computer-readable media. The term “computer-readable media” means any available media that can be accessed by a computer or processor. Examples, but not limited to, such media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disk read-only memory (CD-ROM), or other optical disk storage devices, magnetic disk storage devices, or other magnetic storage devices, or any other media that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. Note that computer-readable media may be tangible and non-temporary. As used herein, the term “code” may mean software, instructions, code, or data that can be executed by a computing device or processor.

[0209] The methods disclosed herein include one or more steps or actions to achieve the described method. The method steps and / or actions may be interchangeable with one another without departing from the claims. In other words, the order and / or use of any particular steps and / or actions may be modified without departing from the claims, unless a particular order of steps or actions is required for the proper operation of the described method.

[0210] As used herein, the term "plural" refers to two or more. For example, "plural components" refers to two or more components. The term "determine" encompasses a wide variety of actions and therefore "determine" can include calculating, arithmetic, processing, calculating, investigating, looking up (e.g., looking at a table, database, or another data structure), confirming, etc. Also, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Also, "determine" can include resolving, selecting, electing, establishing, etc.

[0211] The phrase "based on" does not mean "based solely on" unless explicitly specified otherwise. In other words, the phrase "based on" can mean both "based solely on" and "based at least on."

[0212] The foregoing description of the disclosed implementations is provided to enable any person skilled in the art to manufacture or use the present invention. Various modifications to these implementations will be readily apparent to a person skilled in the art, and the general principles set forth herein may be applied to other implementations without departing from the scope of the present invention. For example, a person skilled in the art will understand that many corresponding alternative and equivalent structural details can be employed, such as equivalent methods for fastening, mounting, linking, or engaging tool components, equivalent mechanisms for producing specific operating motions, and equivalent mechanisms for delivering electrical energy. Accordingly, the present invention is not intended to be limited to the implementations shown herein, but rather to be given the broadest scope consistent with the principles and novel features disclosed herein.

[0213] [Implementation Method] (1) A robotic system, Flexible device, Rigid devices and, A first robotic arm configured to operate the aforementioned flexible device, A second robotic arm configured to operate the rigid device, A robotic system comprising a display configured to display visual data from the flexible device and the rigid device. (2) The flexible instrument includes a flexible scope, The rigid instrument includes a rigid scope, The display includes a video screen, The aforementioned system At least one processor, At least one computer-readable memory, which communicates with the at least one processor and provides to the one or more processors Receiving first visual data from the aforementioned flexible scope, Receiving second visual data from the rigid scope, To generate a signal based on the first visual data and the second visual data, The system according to Embodiment 1, further comprising: at least one computer-readable memory storing a computer-executable instruction for transmitting a signal to a video screen, wherein the signal is configured to cause the video screen to render a video sequence from at least one of the flexible scope and the rigid scope; and (3) The system according to Embodiment 2, wherein the video screen is further configured to render the video sequence from the flexible scope and the rigid scope in a picture-in-picture view. (4) The system according to Embodiment 2, wherein the video screen is further configured to display the video sequence from the flexible scope and the rigid scope in a side-by-side view. (5) The first visual data from the flexible scope includes a primary view, and the second visual data from the rigid scope includes a secondary view. The system according to embodiment 2, wherein the video screen is further configured to switch between rendering the primary view and the secondary view.

[0214] (6) One or more processors, The system according to Embodiment 1, further comprising: at least one computer-readable memory that communicates with the one or more processors and stores computer-executable instructions for causing the one or more processors to identify sterilization zones and non-sterilization zones. (7) The system according to embodiment 6, further comprising a drape positioned between the sterilization zone and the non-sterilization zone. (8) The computer executable instruction is executed on one or more processors, Maintaining either the first robot arm or the second robot arm within the sterilization zone, The system according to embodiment 6, further comprising the action of maintaining the other of the first robot arm and the second robot arm within the non-sterile zone. (9) The computer executable instruction is executed by one or more processors Identifying the transition zone between the sterilization zone and the non-sterilization zone, The system according to embodiment 8, further comprising preventing the first robotic arm and the second robotic arm from moving to the transition zone. (10) One or more processors, At least one computer-readable memory that communicates with the one or more processors and provides to the one or more processors Manipulating the flexible instrument through the first area of ​​the patient, The system according to Embodiment 1, further comprising at least one computer-readable memory storing computer-executable instructions for operating the rigid instrument through a second region of the patient.

[0215] (11) The system according to embodiment 10, wherein the flexible instrument is configured to be operated through a natural opening and the rigid instrument is configured to be operated through an artificial incision. (12) The computer executable instruction is executed by one or more processors The system according to embodiment 10, further enabling control of air delivery to the patient in the first region and the second region of the patient. (13) The computer executable instruction is executed on one or more processors The system according to embodiment 12, further comprising adjusting the air supply in a second region based on the measurement of the air supply in the first region. (14) Bed and, The bed further includes an adjustable arm support connected to the bed, The system according to Embodiment 1, wherein the first robotic arm and the second robotic arm are positioned on the adjustable arm support. (15) A pendant configured to receive input for operating the flexible device using the first robotic arm, The system according to Embodiment 1, further comprising a master controller configured to receive input for operating the rigid device using the second robotic arm.

[0216] (16) A non-temporary computer-readable storage medium that, when executed, is stored in at least one computing device. Using the first robotic arm of the robotic system to manipulate a flexible device, Using the second robotic arm of the robot system to operate a rigid instrument, Displaying feedback from the aforementioned flexible device, A non-temporary computer-readable storage medium that stores commands to display feedback from the rigid device and to perform the same action. (17) Surgical methods, Using the first robotic arm of the robotic system to manipulate a flexible device, Using the second robotic arm of the robot system to operate a rigid instrument, Displaying feedback from the aforementioned flexible device, A surgical method comprising displaying feedback from the rigid device. (18) The flexible instrument includes a flexible scope, The rigid instrument includes a rigid scope, The method according to embodiment 17, wherein at least one of the feedback from the flexible scope and the feedback from the rigid scope is displayed on a video screen. (19) The method according to embodiment 18, wherein the feedback from the flexible scope and the feedback from the rigid scope are displayed in a picture-in-picture view. (20) The method according to embodiment 18, wherein the feedback from the flexible scope and the feedback from the rigid scope are displayed in a side-by-side view.

[0217] (21) The method according to Embodiment 18, wherein one of the feedback from the flexible scope and the rigid scope includes a primary view, and the other of the feedback from the flexible scope and the rigid scope includes a secondary view. (22) The method according to embodiment 21, further comprising switching the primary view and the secondary view such that the feedback from the flexible scope includes the primary view and the feedback from the rigid scope includes the secondary view. (23) The method according to Embodiment 17, further comprising identifying a sterilization zone and a non-sterilization zone. (24) The method according to embodiment 23, wherein a drape is positioned between the sterilization zone and the non-sterilization zone. (25) Maintaining either the first robot arm or the second robot arm within the sterilization zone, The method according to Embodiment 23, further comprising maintaining the other of the first robotic arm and the second robotic arm within the non-sterile zone.

[0218] (26) The method of embodiment 25, further comprising identifying a transition zone between the sterilization zone and the non-sterilization zone. (27) The method according to embodiment 26, further comprising preventing the first robot arm and the second robot arm from moving to the transition zone. (28) The method according to embodiment 17, wherein the flexible instrument is operated through a first area of ​​the patient and the rigid instrument is operated through a second area of ​​the patient. (29) The method according to embodiment 28, wherein the flexible device is operated through a first opening and the rigid device is operated through a second opening. (30) The method according to embodiment 29, wherein the first opening includes a natural opening and the second opening includes an artificial incision.

[0219] (31) The method according to embodiment 30, wherein the natural opening includes the patient's anus and the artificial incision is formed in the patient's abdomen. (32) The method according to embodiment 28, further comprising supplying air to the patient in one of the first region of the patient and the second region of the patient. (33) The method according to embodiment 32, further comprising supplying air to both the first region of the patient and the second region of the patient. (34) The method according to embodiment 33, further comprising adjusting the air supply in the second region based on the measurement of the air supply in the first region. (35) The method according to embodiment 17, wherein the first robot arm and the second robot arm are positioned on an adjustable arm support.

[0220] (36) The method according to embodiment 35, wherein the adjustable arm support is connected to a bed. (37) The method according to embodiment 36, wherein the bed includes a pair of adjustable arm supports, each of the pair of adjustable arm supports having one or more robotic arms attached thereto. (38) The method of embodiment 17, wherein the flexible device is configured to be driven through an outer sheath. (39) The method according to embodiment 38, wherein the flexible instrument includes one or more working channels configured to facilitate the delivery of surgical instruments through the interior. (40) The method according to embodiment 39, wherein the surgical instrument includes a polyp snare.

[0221] (41) The method of embodiment 17, further comprising introducing a marker into the target area of ​​the patient using one of the flexible instrument and the rigid instrument. (42) The method according to embodiment 41, wherein the marker is delivered to the patient's vein. (43) The first robotic arm is used to operate the flexible device in response to an input received via the pendant, The method according to embodiment 17, wherein the operation of the rigid device using the second robotic arm is performed in response to an input received via a master controller. (44) The method according to embodiment 43, wherein the input received via the pendant and the input received via the master controller are received from a single user. (45) The operation of the flexible device using the first robot arm is performed in response to inputs received via the master controller, The method according to Embodiment 17, wherein the operation of the rigid device using the second robotic arm is performed in response to an input received via the master controller.

[0222] (46) The method of embodiment 17, further comprising performing a combined endoscopic and laparoscopic surgical (CELS) procedure using the flexible instrument and the rigid instrument. (47) The method according to embodiment 46, wherein the CELS procedure includes performing a colorectal intervention on the patient. (48) Surgical methods, Using the first robotic arm of the robotic system to manipulate a flexible device, Using the second robotic arm of the robot system to operate a rigid instrument, The first feedback from the aforementioned flexible device is displayed on the view screen as a primary view, A surgical method comprising displaying a second feedback from the rigid instrument as a secondary view on the same view screen. (49) The flexible instrument includes a flexible scope, The rigid instrument includes a rigid scope, The method according to embodiment 48, wherein the first feedback from the flexible scope and the second feedback from the rigid scope are displayed in a picture-in-picture view. (50) The flexible instrument includes a flexible scope, The rigid instrument includes a rigid scope, The method according to embodiment 48, wherein the first feedback from the flexible scope and the second feedback from the rigid scope are displayed in a side-by-side view.

[0223] (51) The method according to Embodiment 48, wherein one of the first feedback and the second feedback includes a primary view, and the other of the first feedback and the second feedback includes a secondary view. (52) The method according to embodiment 51, further comprising switching between the primary view and the secondary view. (53) The method according to embodiment 48, further comprising identifying a sterilization zone and a non-sterilization zone. (54) Maintaining either the first robot arm or the second robot arm within the sterilization zone, The method according to embodiment 53, further comprising maintaining the other of the first robot arm and the second robot arm within the non-sterile zone. (55) The method of embodiment 54, further comprising identifying a transition zone between the sterilization zone and the non-sterilization zone.

[0224] (56) The method according to embodiment 55, further comprising preventing the first robot arm and the second robot arm from moving into the transition zone. (57) The method according to embodiment 48, wherein the flexible instrument is operated through a first area of ​​the patient and the rigid instrument is operated through a second area of ​​the patient. (58) The method according to embodiment 57, wherein the flexible instrument is operated through a first opening and the rigid instrument is operated through a second opening. (59) The method according to embodiment 58, wherein the first opening includes a natural opening and the second opening includes an incision. (60) The method according to embodiment 59, wherein the natural opening includes the anus of the patient and the incision is formed in the abdomen of the patient.

[0225] (61) The method according to embodiment 57, further comprising insufflating the patient in one of the first region of the patient and the second region of the patient. (62) The method according to embodiment 61, further comprising insufflating both the first region of the patient and the second region of the patient. (63) The method according to embodiment 62, further comprising adjusting the insufflation of the second region based on the measurement of the insufflation of the first region. (64) The method according to embodiment 48, further comprising introducing a marker into a target region of the patient using one of the flexible instrument and the rigid instrument. (65) The method according to embodiment 64, wherein the marker is delivered to a vein of the patient.

[0226] (66) Operating the flexible instrument using the first robotic arm is performed in response to an input received via a pendant, Operating the rigid instrument using the second robotic arm is performed in response to an input received via a master controller, the method according to embodiment 48. (67) The method according to embodiment 66, wherein the input received via the pendant and the input received via the master controller are received from a single user. (68) Operating the flexible instrument using the first robotic arm is performed in response to an input received via a master controller, The method according to embodiment 48, wherein the operation of the rigid device using the second robotic arm is performed in response to an input received via the master controller. (69) The method of embodiment 48, further comprising performing a combined endoscopic and laparoscopic surgical (CELS) procedure using the flexible instrument and the rigid instrument. (70) The method according to embodiment 69, wherein the CELS procedure includes performing a colorectal intervention on the patient.

[0227] (71) Surgical methods, Manipulating a flexible instrument using the first robotic arm of the robotic system through the patient's natural orifice, The rigid instrument is operated using the second robotic arm of the robotic system through the incision made in the patient, A surgical method comprising displaying feedback information from the flexible device and the rigid device. (72) The flexible instrument includes a flexible scope, The rigid instrument includes a rigid scope, The method according to embodiment 71, wherein the feedback information from the flexible scope and the rigid scope is displayed on a video screen. (73) The method according to embodiment 72, wherein the first feedback from the flexible scope and the second feedback from the rigid scope are displayed in a picture-in-picture view. (74) The method according to embodiment 72, wherein the first feedback from the flexible scope and the second feedback from the rigid scope are displayed in a side-by-side view. (75) The method according to Embodiment 71, wherein one of the first feedback and the second feedback includes a primary view, and the other of the first feedback and the second feedback includes a secondary view.

[0228] (76) The method of embodiment 75, further comprising switching between the primary view and the secondary view. (77) The method according to embodiment 71, further comprising identifying a sterilization zone and a non-sterilization zone. (78) Maintaining either the first robot arm or the second robot arm within the sterilization zone, The method according to embodiment 77, further comprising maintaining the other of the first robot arm and the second robot arm within the non-sterile zone. (79) The method according to embodiment 78, further comprising identifying a transition zone between the sterilization zone and the non-sterilization zone. (80) The method according to embodiment 79, further comprising preventing the first robot arm and the second robot arm from moving to the transition zone.

[0229] (81) The method according to embodiment 71, wherein the natural opening includes the patient's anus and the incision is formed in the patient's abdomen. (82) The method according to embodiment 71, wherein the flexible instrument is operated through a first area of ​​the patient and the rigid instrument is operated through a second area of ​​the patient. (83) The method of embodiment 82, further comprising supplying air to the patient in one of the first region of the patient and the second region of the patient. (84) The method according to embodiment 83, further comprising supplying air to both the first region of the patient and the second region of the patient. (85) The method of embodiment 84, further comprising adjusting the air supply in the second region based on the measurement of the air supply in the first region.

[0230] (86) The method according to embodiment 71, further comprising introducing a marker into the target area of ​​the patient using one of the flexible instrument and the rigid instrument. (87) The method according to embodiment 86, wherein the marker is delivered to the patient's vein. (88) The first robotic arm is used to operate the flexible device in response to an input received via the pendant, The method according to embodiment 71, wherein the operation of the rigid device using the second robotic arm is performed in response to an input received via a master controller. (89) The method according to embodiment 88, wherein the input received via the pendant and the input received via the master controller are received from a single user. (90) The first robotic arm is used to operate the flexible device in response to an input received via the master controller. The method according to embodiment 71, wherein the operation of the rigid device using the second robotic arm is performed in response to an input received via the master controller.

[0231] (91) The method according to embodiment 71, further comprising performing a combined endoscopic and laparoscopic surgical (CELS) procedure using the flexible instrument and the rigid instrument. (92) Surgical methods, Introducing a flexible device into the patient through the patient's natural orifice, The flexible device is operated using the first robotic arm of the robot system through the aforementioned natural opening, In response to receiving an input signal via a user input device, the second robot arm of the robot system is deployed from the storage position to the setup position. The rigid instrument is operated using the second robotic arm of the robotic system through the incision made in the patient, A surgical method comprising displaying feedback from at least one of the flexible instrument and the rigid instrument. (93) The method according to embodiment 92, wherein the user input device includes at least one of a master controller and a pendant.

Claims

1. It is a robotic system, A flexible instrument comprising a flexible scope configured to be driven through an outer sheath, having one or more working channels configured to facilitate the delivery of surgical instruments through the interior, wherein the surgical instruments include a polyps snare, Rigid devices and, A first robotic arm configured to operate the aforementioned flexible device, A second robotic arm configured to operate the rigid device, A display configured to display visual data from the flexible device and the rigid device, At least one processor, The system includes encoders for measuring the positions of the first robot arm and the second robot arm, The aforementioned at least one processor, Identifying a sterilization zone, a non-sterilization zone, and a transition zone between the sterilization zone and the non-sterilization zone, A robotic system that causes the first robotic arm and the second robotic arm to continue moving within the transition zone from the sterilization zone to the non-sterilization zone or vice versa, based on the positions of the first robotic arm and the second robotic arm measured by the encoder, by displaying a visual cue on the display for the first robotic arm and the second robotic arm to continue moving within the transition zone.

2. The rigid instrument includes a rigid scope, The display includes a video screen, The robot system, At least one computer-readable memory, which communicates with the at least one processor, and provides to the at least one processor, Receiving first visual data from the aforementioned flexible scope, Receiving second visual data from the rigid scope, To generate a signal based on the first visual data and the second visual data, The robot system according to claim 1, further comprising: transmitting a signal to the video screen, wherein the signal is configured to cause the video screen to render a video sequence from at least one of the flexible scope and the rigid scope; and at least one computer-readable memory storing a computer-executable instruction for doing so.

3. The robotic system according to claim 2, wherein the video screen is further configured to render the video sequence from the flexible scope and the rigid scope in a picture-in-picture view.

4. The robotic system according to claim 2, wherein the video screen is further configured to display the video sequence from the flexible scope and the rigid scope in a side-by-side view.

5. The first visual data from the flexible scope includes a primary view, and the second visual data from the rigid scope includes a secondary view. The robotic system according to claim 2, wherein the video screen is further configured to switch between rendering the primary view and the secondary view.

6. The robot system according to claim 1, further comprising: at least one computer-readable memory that communicates with the at least one processor and stores computer-executable instructions for causing the at least one processor to identify the sterilization zone and the non-sterilization zone.

7. The robotic system according to claim 6, further comprising a drape positioned between the sterilization zone and the non-sterilization zone.

8. The computer executable instruction is performed by the at least one processor, Maintaining either the first robot arm or the second robot arm within the sterilization zone, The robot system according to claim 6, further comprising maintaining the other of the first robot arm and the second robot arm within the non-sterile zone.

9. The computer executable instruction is performed by the at least one processor, Identifying the transition zone between the sterilization zone and the non-sterilization zone, The robot system according to claim 8, further comprising preventing the first robot arm and the second robot arm from moving to the transition zone.

10. At least one computer-readable memory, which communicates with the at least one processor, and provides to the at least one processor, Manipulating the flexible instrument through the patient's first region, The robotic system according to claim 1, further comprising: at least one computer-readable memory storing computer-executable instructions for operating the rigid device through a second region of the patient;

11. The robotic system according to claim 10, wherein the flexible instrument is configured to be operated through a natural opening, and the rigid instrument is configured to be operated through an artificial incision.

12. The computer executable instruction is performed by the at least one processor, The robotic system according to claim 10, further enabling control of air delivery to the patient in the first region and the second region of the patient.

13. The computer executable instruction is performed by the at least one processor, The robotic system according to claim 12, further comprising adjusting the air supply in the second region based on the measurement of the air supply in the first region.

14. A bed and The bed further includes an adjustable arm support connected to the bed, The robotic system according to claim 1, wherein the first robotic arm and the second robotic arm are positioned on the adjustable arm support.

15. A pendant configured to receive input for operating the flexible device using the first robotic arm, The robot system according to claim 1, further comprising a master controller configured to receive input for operating the rigid device using the second robot arm.

Images