Systems for hybrid insertion of an elongate flexible device

Abstract

A system comprises a robotically-assisted manipulator assembly, an elongate flexible device, and a control system including a processing unit and memory comprising machine readable instructions. The control system may recognize the elongate flexible device is connected to a carriage of the manipulator assembly and engage a hybrid control mode in which longitudinal motion of the elongate flexible device along an anatomic path is responsive to a manual force and steering motion of a distal portion of the elongate flexible device is responsive to a user input at an operator input system. The control system may also determine a guide assembly has been docked and based on the determination that the guide assembly has been docked, transition from the hybrid control mode to an assisted control mode in which motion of the elongate flexible device is responsive to robotically-assisted control of longitudinal and steering motions of the elongate flexible device.

Description

SYSTEMS FOR HYBRID INSERTION OF AN ELONGATE FLEXIBLE DEVICECROSS-REFERENCED APPLICATIONS

[0001] This application claims priority to and benefit of U.S. Provisional Application No. 63 / 549,920 filed February 5, 2024 and entitled “Systems and Methods for Hybrid Insertion of an Elongate Flexible Device,” which is incorporated by reference herein in its entirety.FIELD

[0002] Examples described herein relate to systems and methods for hybrid insertion of an elongate flexible device into a patient anatomy. More particularly, examples relate to systems and methods for providing hybrid control, including both manual and robotically-assisted control, to navigate an elongate flexible device to a target location in a patient anatomy.BACKGROUND

[0003] Minimally invasive medical techniques may generally be intended to reduce the amount of tissue that is damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions an operator may insert minimally invasive medical instruments such as therapeutic instruments, diagnostic instruments, imaging instruments, and surgical instruments. Some minimally invasive medical instruments may involve the use of endoscopic devices that are navigated to a target location to perform a procedure. Systems and methods are needed to provide an operator with a range of tactile feedback and visual navigation guidance during a device insertion, while providing at least partial robotic assistance with steering and an efficient transition to full robotic assistance during the performance of the procedure.SUMMARY

[0004] The following presents a simplified summary of various examples described herein and is not intended to identify key or critical elements or to delineate the scope of the claims.

[0005] In some examples, a system comprises a robotically-assisted manipulator assembly, an elongate flexible device, and a control system including a processing unit and memory comprising machine readable instructions. The control system may recognize the elongate flexible device is connected to acarriage of the manipulator assembly and engage a hybrid control mode in which longitudinal motion of the elongate flexible device along an anatomic path is responsive to a manual force and steering motion of a distal portion of the elongate flexible device is responsive to a user input at an operator input system. The control system may also determine a guide assembly has been docked and based on the determination that the guide assembly has been docked, transition from the hybrid control mode to an assisted control mode in which motion of the elongate flexible device is responsive to robotically-assisted control of longitudinal and steering motions of the elongate flexible device.

[0006] In some examples, a method for navigating an elongate flexible device may include recognizing the elongate flexible device is connected to a carriage of a manipulator assembly and engaging a hybrid control mode in which longitudinal motion of the elongate flexible device along an anatomic path is responsive to a manual force and steering motion of a distal portion of the elongate flexible device is responsive to a user input at an operator input system. The method may also comprise determining a guide assembly has been docked and based on the determination that the guide assembly has been docked, transitioning from the hybrid control mode to an assisted control mode in which motion of the elongate flexible device is responsive to robotically-assisted control of longitudinal and steering motions of the elongate flexible device.

[0007] In some examples, a system may comprise a robotically-assisted manipulator assembly, an elongate flexible device, and a control system. The control system may recognize the elongate flexible device is connected to a carriage of the manipulator assembly and engage a hybrid control mode in which longitudinal motion of the elongate flexible device along an anatomic path and roll motion of the elongate flexible device relative to a longitudinal axis of the elongate flexible device are each responsive to a manual force and steering motion of a distal portion of the elongate flexible device is responsive to a user input at an operator input system. The control system may also transition from the hybrid control mode to an assisted control mode in which motion of the elongate flexible device is responsive to robotically- assisted control of longitudinal, roll, and steering motions of the elongate flexible device.

[0008] It is to be understood that both the foregoing general description and the following detailed description are illustrative and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTIONS OF THE DRAWINGS

[0009] FIG. 1A illustrates a manipulator assembly and medical instrument system, according to some examples.

[0010] FIG. IB illustrates the manipulator assembly and medical instrument system of FIG. 1A with a guide assembly installed and docked, according to some examples.

[0011] FIG. 2 is a simplified diagram of a patient anatomy, according to some examples.

[0012] FIG. 3 is a flowchart illustrating a method for navigating an elongate flexible device into a patient anatomy, according to some examples.

[0013] FIG. 4 is a flowchart illustrating a method for navigating an elongate flexible device into a patient anatomy, according to some examples.

[0014] FIG. 5 is a flowchart illustrating a method for navigating an elongate flexible device into a patient anatomy, according to some examples.

[0015] FIG. 6 is a schematic diagram for a robotically-assisted manipulator system, according to some examples.

[0016] FIG. 7A is a schematic diagram of an instrument system, according to some examples.

[0017] FIG. 7B illustrates a distal portion of the instrument system of FIG. 7A with an extended example of an instrument, according to some examples.

[0018] FIG. 8A illustrates a perspective view of an operator input system and a display system, according to some examples,

[0019] FIG. 8B illustrates a top view of the operator input system of FIG. 8B, according to some examples.

[0020] Examples of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating examples of the present disclosure and not for purposes of limiting the same.DETAILED DESCRIPTION

[0021] The technology described herein provides systems and methods for introducing an elongate flexible device through an anatomic orifice and along an anatomic path in a patient anatomy. Navigation to the target location may be accomplished using a hybrid control that includes both manual and robot- assisted control of the elongate flexible device. The hybrid control allows an operator to, for example, use robotic-assistance to steer a distal portion of the elongate flexible device while experiencing tactile sensation of at least some forces associated with the manual control of the motion of the device in the insertion degree of freedom. The elongate flexible device may be used to perform medical procedures such as, surgery, biopsy, suturing, ablation, suturing, illumination, irrigation, suction, or other interventional procedures. Some procedures may be performed in the upper or lower gastrointestinal tract including, for example fistula closure, endoscopic submucosal dissections, endoscopic mucosal resections, peroral endoscopic myotomy, endoscopic sleeve gastroplasty, and / or transoral outlet reduction endoscopy. Although the examples provided herein may be used for introducing an elongate flexible device into the gastrointestinal tract to perform gastrointestinal endoscopy procedures, it is understood that the described technology may be used in performing procedures in artificially created lumens or any endoluminal passageway or cavity, including in a patient trachea, colon, intestines, stomach, liver, kidneys and kidney calices, brain, heart, circulatory system including vasculature, fistulas, and / or the like.

[0022] FIGS. 1A and IB provide a perspective view of a manipulator assembly 100 connected to a medical instrument system 102. The manipulator assembly 100 may be a robotically-assisted manipulator assembly. For example, the manipulator assembly 100 may be a component (e.g., the manipulator assembly 502) of a robotically-assisted manipulator system (e.g., the robotically-assisted manipulator system 500) including a control system (e.g., the control system 512) for effecting selective control of the medical instrument system 102. The medical instrument system 102 includes an elongate flexible device 112 coupled at a proximal end to an instrument base 114. In various examples, the elongate flexible device 112 may be a flexible catheter or endoscope (e.g., gastroscope, bronchoscope) and may optionally include one or more working channels sized and shaped to receive one or more medical instruments (see FIGS. 7A-7B). The manipulator assembly 100 includes a base stage 104, an insertion stage 106, and an instrument carriage 108. The instrument base 114 may be coupled to the instrument carriage 108. The manipulator assembly 100 may provide for insertion and retraction of the medical instrument system 102 along an insertion trajectory 103, with respect to the patient anatomy, by moving the instrument carriage 108 and the insertion stage 106 in a telescoping manner relative to base stage 104 and along a linearinsertion axis A. The insertion trajectory 103 may extend in an insertion / retraction direction of motion for the medical instrument system 102. More specifically, the insertion stage 106 may be coupled to and translate along the insertion axis A with respect to the base stage 104. The instrument carriage 108 may be coupled to and translate along the insertion axis A with respect to the insertion stage 106. As shown, the insertion stage 106 and the instrument carriage 108 have respective protective coverings and / or and housings which may at least partially overlap with each other, thereby effecting a telescoping effect, to facilitate sealing and optimal slidable fit. The instrument carriage 108 may include force sensors 109 for measuring gravity, friction, or other forces applied to the medical instrument system 102. In some examples, the force sensors 109 may be located in or on the medical instrument system 102. In yet other examples, force sensors may be located at both the instrument carriage 108 and the medical instrument system 102.

[0023] An imaging assembly 118 may couple to the instrument base 114 and extend through the elongate flexible device 112. The imaging assembly 118 may record concurrent or real-time images of an interventional site (which may be a surgical site, an internal surgical site, a procedure site, etc.) and provide the images to an operator (e.g., operator O, FIG. 6) through one or more displays (e.g., one or more displays of display system 510 in FIG. 6). The instrument carriage 108 may include electronic and optical components providing the imaging assembly 118 with endoscopic capabilities. The imaging assembly 118 may be fixedly coupled or removably coupled to a channel in the elongate flexible device 112. In some embodiments, the imaging assembly 118 may be detached from the manipulator assembly 100 and removed from elongate flexible device 112. Alternative instruments such as biopsy needles, ablation tools, and other flexible instruments may be coupled to manipulator assembly 100 and / or extend through one or more working channels in the elongate flexible device 112.

[0024] The base stage 104 may also include a device connector 116 which supports the elongate flexible device 112 along the insertion trajectory 103. The manipulator assembly 100 may also include a guide assembly 110 which is selectively extendable between the instrument base 114 and the device connector 116 to support a flexible length of the elongate flexible device 112. The guide assembly 110 or anti-buckling device may have a collapsed state (FIG. 1 A) during the installation and set-up of the medical instrument system 102 and may have an extended state (FIG. IB) to support the elongate flexible device 112 during a procedure. The device connector 116 and / or the base stage 104 may be removably coupled to an anatomic orifice device 120. The anatomic orifice device 120 may be, for example, an endotracheal tube, a laryngeal mask airway, or a cannula, and may be fixed to patient anatomy to facilitate insertion ofvarious medical devices into patient anatomy. For example, the anatomic orifice device 120 may be an endotracheal tube inserted into the mouth and trachea of a patient to help provide mechanical ventilation and to provide a conduit for the elongate flexible device 112 to be navigated into the patient esophagus to facilitate imaging, biopsy, and / or treatment. A procedure, such as a gastrointestinal endoscopy procedure, performed with an elongate flexible device 112 may be performed with an additional patient tracheal intubation, in which a tube is placed into the trachea to control delivery of patient oxygen. Alternatively, the elongate flexible device 112 may be used to perform a gastrointestinal procedure without a patient tracheal intubation. The elongate flexible device 112 may be used to perform any of a variety of medical procedures, including exploratory or interventional endoscopy procedures such a gastroscopy, enteroscopy, sigmoidoscopy, colonoscopy, bariatric procedures such as endoscopic sleeve gastroplasty, ablation procedures, or the like.

[0025] FIG. 2 illustrates the elongate flexible device 112 extending through an anatomic orifice 152 such as a mouth of a patient P, within an anatomic passageway 154 such as the esophagus of the patient P, and into an anatomical structure 156. In some examples, the anatomic structure 156 may be a stomach. The anatomy of the patient P may have an anatomical frame of reference (XA, YA, ZA). A distal portion 158 of the elongate flexible device 112 may be used to perform a medical procedure, such as a suturing, biopsy, ablation procedure, at or near target location 160 located in the anatomic structure 156 using any of the methods or systems described herein. In some examples, the target location may be in the gastrointestinal tract, such as in the stomach, in the intestines, at or near a gastroesophogeal junction or the pylorus, or at other locations along the gastrointestinal tract. The elongate flexible device 112 may be advanced or retracted in a longitudinal degree of freedom of motion 170 and / or rotate in a rotational or roll degree of freedom of motion 172 (e.g., roll relative to a longitudinal axis of the elongate flexible device 112). The distal portion 158 of the elongate flexible device 112 may also be articulatable in steering degrees of freedom of motion 174 (e.g., pitch and / or yaw orientations). In some examples, the target location may be determined based on a pre-procedure plan (e.g. using pre-operative images of the patient anatomy) and an intra-operative registration between the patient and a robotically-assisted manipulator system (e.g. system 500) used to implement the pre-procedure plan. In some examples, the target location may be determined based on a combination of kinematic data from the robotically-assisted manipulator system including shape sensor data, image-based feature detection algorithms for analyzing intraoperative images, and intra-operative registration including the detection of one or more anatomical landmarks observed during the procedure or during the procedure set-up.

[0026] In some examples, the elongate flexible device 112 may be initially inserted and navigated to the target location 160 using full manual control. Full manual control involves one or more operators employing hand manipulations and forces to the elongate flexible device 112 to control longitudinal motion 170 (insertion / retraction) and roll motion 172 of the elongate flexible device 112. For example, the operator may apply manual forces to insert and / or retract the elongate flexible device 112 and / or to rotate the elongate flexible device about its longitudinal axis. Manual forces may include any type of force directly applied by the operator’s body to the elongate flexible device 112 or any structure to which the elongate flexible device is coupled. Full manual control may allow the operator to feel the resistive forces from the patient anatomy as the elongate flexible device 112 is delivered. This direct tactile feedback may provide the operator with valuable cues indicating obstructions, tortuous pathways, narrowed pathways or other anatomical characteristics that may cause the operator to take extra precautions or exercise additional care when inserting the elongate flexible device 112. However, full manual control of the elongate flexible device 112 may provide an operator little or no control of the steering degree of freedom 174 of the distal portion 158 of the elongate flexible device 112 in pitch or yaw directions in some embodiments. This is because, in some embodiments, the elongate flexible device 112 may lack manual controls for articulating the distal portion 158 in pitch and yaw and distal articulation is intended for robotic actuation via a manipulator assembly. In some examples, the elongate flexible device 112 may be connected to a robotically-assisted manipulator assembly (e.g., manipulator assembly 100) but may be operated in a full manual control mode of the robotically-assisted medical system that allows an operator to manually manipulate the elongate flexible device with full manual control. In some examples, the elongate flexible device 112 may be selectively detachable from the robotically-assisted manipulator assembly to be operated in full manual control and reattached to operate with robotically-assisted control modes.

[0027] In some examples, the elongate flexible device 112 may be connected to the manipulator assembly 100 and initially inserted and navigated to the target location 160 using an assisted control mode, sometimes referred to as a full robotic assistance mode or fully tele-operated control mode. In the assisted control mode, the longitudinal degree of freedom of motion 170 (insertion / retraction) and the steering degree of freedom 174 of the distal portion 158 of the elongate flexible device 112 (pitch / yaw) may be robotically assisted, responsive to a user input at an operator input system (e.g. operator input system 700) of a robotically-assisted medical system. Optionally, in the assisted control mode, each of the degrees of freedom of motion of the distal portion of the elongate flexible device may be controlled from the operator input system. Optionally, in the assisted control mode, the roll degree of freedom of motion 172 may beheld fixed, may be movable with robotic-assistance in response to input at an operator input system, or may be movable by manual operation. The assisted control mode may also allow an operator to finely control steering motion 174 of the distal portion 158 of the elongate flexible device 112 via the operator input system. The use of an operator input system for robotic assistance of motion in the longitudinal degree of freedom 170 during initial insertion may result in an operator being unable to directly experience tactile feedback cues associated with forces translated directly from the patient anatomy through the body of the elongate flexible device. To compensate for a lack of direct tactile feedback, an operator may rely on other sources of information such as image data provided by an endoscopic imaging system, preoperative images of the patient anatomy, or force sensor information that may be translated into artificial haptic sensations at the operator input system.

[0028] In some examples, the elongate flexible device 112 may be connected to the manipulator assembly 100, and initially inserted and navigated to the target location 160 using a hybrid control mode of the robotically-assisted medical system. The hybrid control mode may incorporate a combination or hybrid of manual and robotic-assisted controls. In the hybrid control mode, the steering degree of freedom 174 of the distal portion 158 of the elongate flexible device 112 may be robotically assisted, responsive to a user input at the operator input system (e.g., operator input system 700) of the robotically-assisted medical system, and the longitudinal degree of freedom of motion 170 (insertion / retraction) may be operated by direct manual control. For example, the operator may apply manual forces to the elongate flexible device 112 to insert and / or retract the elongate flexible device 112 into the anatomic orifice 152 and toward the target location 160. Although the longitudinal degree of freedom of motion may be manually controlled, sensor-based robotically-assisted compensation may be provided to support the manual operation. For example, in the hybrid control mode, robotically-assisted gravity compensation may be provided to the instrument carriage to resist gravity (e.g., to prevent the carriage from plunging rapidly in the insertion direction) and reduce the weight experienced by the operator. As another example, in the hybrid control mode, robotically-assisted friction compensation may be provided to the instrument carriage to reduce friction or inertia along the insertion trajectory. The friction compensation may be provided on any or all the carriage axes driving the elongate flexible device and / or on the longitudinal axis aligned with the insertion trajectory. As another example, in the hybrid control mode, a robotically- assisted light tension may be applied to the elongate flexible device, to help maintain a tautness to the elongate flexible device. Gravity compensation, friction compensation, tension support, or other types of robotic assistance to the operator during manual operation may be based on sensor feedback from sensors(e.g. sensor 109) in the manipulator assembly and / or in the medical instrument system. Optionally, in the hybrid control mode, the roll degree of freedom of motion 172 may be fixed, may be movable with robotic- assistance in response to input at the operator input system, or may be manually operated. Thus, in the hybrid control mode, an operator may manually rotate or twist the elongate flexible device while advancing or retracting the elongate flexible device. The hybrid control mode may also allow an operator to finely control steering of the distal portion 158 of the elongate flexible device via the operator input system while also being able to directly experience tactile forces and the feedback cues associated with forces translated directly from the patient anatomy through the body of the elongate flexible device as the device is moved in the insertion and retraction degree of freedom. For example, if while manually inserting the elongate flexible device, the operator receives a tactile sensation associated with an unsafe contact between the patient anatomy and the elongate flexible device 112, the operator may simultaneously steer the distal portion 158 of the elongate flexible device using the operator input system to avoid the unsafe contact.

[0029] In some examples, the elongate flexible device 112 may be used in a carriage control mode. The carriage control mode may be substantially similar to the hybrid control mode except that the robotic- assistance in the steering degrees of freedom may be disabled so that distal portion steering is not controlled in the carriage control mode. Further, the tension support to the elongate flexible device may be disabled.

[0030] In some examples, the elongate flexible device 112 may be connected to the manipulator assembly 100 and may resist motion in the insertion and retraction degree of freedom while in a holding control mode. In the holding control mode, the instrument carriage (e.g., instrument carriage 108) may hold the elongate flexible device 112 stationary along the insertion trajectory 103 to prevent motion of the elongate flexible device further into the patient anatomy in the longitudinal degree of freedom of motion. For example, an operator may manually insert the elongate flexible device 112 along the insertion direction by hand (e.g., optionally guided under the hybrid control mode described herein). Upon reaching a target amount of insertion, the elongate flexible device 112 may be placed into the holding control mode. In the holding control mode, an operator may release a manual hold on the elongate flexible device 112 and the position of the distal portion 158 of the elongate flexible device may remain stationary (i.e., without further advancement or retraction of the elongate flexible device 112). For example, the elongate flexible device 112 may be placed in the holding control mode when the distal portion 158 is at a target location 160 in the patient anatomy. Optionally, in the holding control mode, a tensioning operation maybe performed in which the instrument carriage 108 is retracted to relieve any buckling of the elongate flexible device 112 while preventing advancement of the distal portion 158 of the elongate flexible device. In some examples, the holding control mode may include a feedback-control property in which the control system maintains the position of the carriage and / or the elongate flexible device via a feedback control to a constant position command. Additionally or alternatively, the holding control mode may include a floating property in which the servo-control compensates for gravity and / or friction inertia while the carriage may be manually positionable along the insertion trajectory. Additionally or alternatively, the holding control mode may include a tension holding property in which a fixed positive tension is maintained on the carriage and / or elongate flexible device.

[0031] FIG. 3 is a flowchart illustrating a method 200 for inserting and navigating an elongate flexible device into a patient anatomy. The method 200 and other methods described herein are illustrated as a set of operations or processes that may be performed in the same or in a different order than the order shown. One or more of the illustrated processes may be omitted in some embodiments of the disclosed methods. Additionally, one or more processes that are not expressly illustrated may be included before, after, in between, or as part of the illustrated processes. In some embodiments, one or more of the processes of a disclosed method may be implemented, at least in part, by a control system executing code stored on non- transitory, tangible, machine-readable media that when run by one or more processors (e.g., the processors of control system 512) may cause the one or more processors to perform one or more of the processes.

[0032] At a process 202, a connection between an elongate flexible device and a carriage of a manipulator assembly may be recognized. For example, an operator may connect the instrument base 114 of the medical instrument system 102 to the instrument carriage 108. A control system (e.g., the control system 512) may recognize the medical instrument system 102, including the elongate flexible device 112, as connected to the instrument carriage 108. In some examples, recognizing the medical instrument system may include recognizing a presence of the medical instrument system or may include recognizing device-specific information such as the type of device, a serial number, or other device profile information. With the medical instrument system 102 coupled to the instrument carriage 108, the distal portion 158 of the elongate flexible device 112 may be inserted through the anatomic orifice device 120 (e.g., an endotracheal tube) and introduced to the patient anatomy.

[0033] At a process 204, a hybrid control mode of a robotically-assisted medical system may be engaged. In the hybrid control mode, longitudinal motion of the distal portion of the elongate flexible device in the longitudinal degree of freedom of motion (e.g., advancement or insertion motion) and / or rollmotion of the elongate flexible device may be operated by direct manual control. For example, an operator may grasp a medial portion of the elongate flexible device 112 between the anatomic orifice device 120 and the instrument carriage 108 and apply a manual force to guide the elongate flexible device into the patient anatomy. Optionally, in the hybrid control mode, an operator may additionally or alternatively manually rotate or twist the elongate flexible device while advancing or retracting the elongate flexible device. In one example, while in the hybrid control mode, the instrument carriage 108 may be stationary at a distal staging position with the elongate flexible device 112 in a prolapsed state. The prolapsed elongate flexible device may be fed into the patient anatomy with little or no movement of the instrument carriage relative to the base stage of the manipulator assembly. This example is described in greater detail in FIG. 5. In an alternative example, while in the hybrid control mode, the instrument carriage may be in motion relative to the base stage, along the insertion trajectory, in response to the manual tensioning force exerted on the elongate flexible device by the operator. This example is described in greater detail in FIG. 4. Manual control of the longitudinal motion 170 of the elongate flexible device 112 in the hybrid control mode may allow the operator to feel the resistive forces from the patient anatomy as the elongate flexible device is delivered. This tactile feedback may provide the operator with valuable cues indicating obstructions, tortuous pathways, narrowed pathways or unsafe contact with the anatomy that may cause the operator to take extra precautions or exercise additional care when inserting the elongate flexible device.

[0034] In the hybrid control mode, the steering degree of freedom 174 of the distal portion 158 of the elongate flexible device 112 may be robotically assisted. For example, the same operator (or a different operator) controlling the longitudinal motion may control the articulation of the distal portion of the elongate flexible device by manipulating an operator input system (e.g., the operator input system 700) of the robotically-assisted medical system. Optionally, the distal portion 158 of the elongate flexible device 112 may be in a default limp state but may transition to an active state in response to a user input at the operator input system. From the active state, the elongate flexible device may revert back or transition to the limp state. The transition to the limp state may occur, for example, when the user input commanding the steering motion ceases for a predetermined period of time or the user’s hand is removed from a touch- sensitive (e.g., a capacitive touch sensing) input device, The transition condition may also be based on the current position of the distal portion 158 relative to patient anatomy or landmarks, and / or whether holding the position fixed is relevant to the medical procedure. In some examples, the transition to the limp state may occur gradually, with a relaxation of the control torques that may result in a smooth, controlledrelaxation of the distal portion. Discontinuities, for example due to friction and / or baseline tension, that may be crossed in a specific progression to achieve a smooth or linearly relaxing behavior. In the hybrid control mode, the steering degrees of freedom of motion may have a limited angle, a limited articulation force, or other lowered function values, as compared to the steering degrees of freedom of motion in the assisted control mode.

[0035] In the hybrid control mode, to support the safety and ease of manual motion in the longitudinal degree of freedom of motion, sensor-based robotically-assisted compensation may be provided to assist with control of the motion in the longitudinal degree of freedom of motion. For example, robotically- assisted gravity compensation may be provided to the instrument carriage to resist gravity (e.g., to prevent the carriage from plunging rapidly in the insertion direction) and reduce the weight experienced by the operator. As another example, robotically-assisted friction compensation may be provided to the instrument carriage to reduce friction or inertia along the insertion trajectory. As another example, in the hybrid control mode, a robotically-assisted light tension may be applied to the elongate flexible device, to help maintain a tautness to the elongate flexible device. Gravity compensation, friction compensation, tension support, or other types of robotic assistance to the operator during manual operation may be based on sensor feedback from sensors (e.g. sensors 109) in the manipulator assembly. Optionally, in the hybrid control mode, the roll degree of freedom of motion 172 may be fixed, may be movable with robotic- assistance in response to input at the operator input system, or may be manually controlled by an operator (e.g., by hand). The hybrid control mode allows an operator to finely control steering of the distal portion 158 of the elongate flexible device via the operator input system while also being able to directly experience tactile forces and the feedback cues associated with forces translated directly from the patient anatomy through the body of the elongate flexible device as the device is moved in the insertion and retraction degree of freedom.

[0036] At the process 204, the hybrid control mode may be engaged in response to any of a variety of stimuli or conditions. For example, the hybrid control mode may be engaged in response to receipt of a mode selection input at the operator input system (e.g., pressing a button) or in response to detection of an insertion direction force at a force sensor on the manipulator assembly or medical instrument system (e.g., detection of a tug on the elongate flexible device). Additionally or alternatively, the hybrid control mode may be engaged in response to a detection of an insertion direction force at a joint sensor of the manipulator assembly, in contradiction to a commanded hold position.

[0037] In the hybrid control mode, the distal portion of the elongate flexible device may be guided tothe target location in preparation for a medical procedure. As the elongate flexible device is manually inserted into the patient, the guide assembly 110 may be in a collapsed configuration to avoid interfering with the operator who is providing holding forces and manipulating the medial portion of the elongate flexible device 112. After the distal portion 158 of the elongate flexible device 112 reaches a target location (e.g., target location 160) and before the elongate flexible device is transitioned to assisted control mode, the operator may extend the guide assembly 110 along the length of the elongate flexible device between the instrument carriage 108 and the anatomic orifice device 120. In some examples the guide assembly 110 may be coupled at a proximal end to the instrument carriage 108 and at a distal end to the anatomic orifice device 120 to support the length of the portion of the elongate flexible device external to the patient.

[0038] At a process 206, a determination may be made that a guide assembly has been docked. For example, a proximal portion of the guide assembly 110 may be coupled to the instrument carriage 108 as shown in FIG. 1 A. The guide assembly 110 may be extended to connect or dock to the device connector 116 at the base stage 104. After the guide assembly is docked, the operator may provide an affirmative indication at the operator input system signaling the docking is complete. Alternatively, a sensor at the base stage may detect that the guide assembly has been connected and docked. The docking of the guide assembly 110 may trigger a transition from the hybrid control mode to the assisted control mode as described at process 208.

[0039] At a process 208, control of the motion of the elongate flexible device may be transitioned to an assisted control mode, sometimes referred to as a full robotic assistance mode. The control mode transition from the hybrid control mode to the assisted control mode may be triggered by a user input at the operator input system or based on the determination that the guide assembly has been docked. In the assisted control mode, the longitudinal degree of freedom of motion 170 and the steering degree of freedom 174 of the distal portion 158 of the elongate flexible device 112 may be robotically assisted, responsive to a user input at an operator input system (e.g. operator input system 700) of the robotically- assisted medical system. Optionally, the robotically-assisted gravity compensation, friction compensation, and tension support may be continued in the assisted control mode. Optionally, in the assisted control mode, the roll degree of freedom of motion 172 may be fixed, operated under robotic assistance, or manually operated. The assisted control mode may also allow an operator to finely control steering motion 174 of the distal portion 158 of the elongate flexible device via the operator input system. In the assisted control mode, an operator may receive information such as image data provided by an endoscopic imagingsystem, pre-operative images of the patient anatomy, or force sensor information translated into artificial haptic sensations at the operator input system to provide guidance during the procedure.

[0040] According to some examples, if the guide assembly 110 is undocked from the device connector 116 and collapsed back toward the instrument carriage 108, the system may transition back to the hybrid control mode. The elongate flexible device may then become limp. If the operator begins to manipulate the operator input system (e.g., a trackball), the distal portion 158 of the elongate flexible device 112 may assume a position holding mode in which the distal portion remains stationary.

[0041] In some examples, methods for navigating an elongate flexible device into a patient anatomy using a hybrid control mode may differ based on whether or how much the instrument carriage moves during the navigation and whether the elongate flexible device external of the patient anatomy is in an elongated configuration or a prolapsed configuration during the navigation. FIG. 4 is a flowchart illustrating a method 300 for navigating an elongate flexible device into a patient anatomy. At a process 302, a connection between an elongate flexible device and a carriage of a manipulator assembly may be recognized. The process 302 may be substantially similar to the process 202, with differences as described. As an example, an operator may connect the instrument base 114 of the medical instrument system 102 to the instrument carriage 108. In this example, the instrument carriage 108 may be at a retracted or proximal position along the insertion trajectory 103 when the medical instrument system 102 is coupled or may be moved to the retracted position along the insertion trajectory 103 after the medical instrument system is coupled. With the instrument carriage 108 in the retracted position, the elongate flexible device 112 may be in an elongated configuration with little or no buckling or prolapse along the length. A control system (e.g., the control system 512) may recognize the medical instrument system 102, including the elongate flexible device 112, as connected to the instrument carriage 108. With the medical instrument system coupled to the instrument carriage 108, the distal portion 158 of the elongate flexible device 112 may be inserted through the anatomic orifice device 120, such as an endotracheal tube, and into the patient’s anatomic orifice.

[0042] At a process 304, a hybrid control mode of a robotically-assisted medical system may be engaged. The process 304 may be substantially similar to the process 204, with differences as described. In the hybrid control mode, the longitudinal motion of the distal portion 158 of the elongate flexible device 112 in the longitudinal degree of freedom of motion (e.g., advancement or insertion motion) 170 may be operated by direct manual control, the steering degree of freedom 174 of the distal portion 158 of the elongate flexible device 112 may be robotically assisted, and sensor feedback forms of robotic assistancesuch as gravity compensation, friction compensation, and tension support may support the operator performing the manual insertion operation. Optionally, in the hybrid control mode, the roll degree of freedom of motion 172 may be fixed, operated under robotic assistance, or manually controlled by an operator.

[0043] At a process 306, the hybrid control mode may be maintained as the instrument carriage with the coupled instrument base is advanced along the insertion trajectory and as the distal portion of the elongate flexible device is advanced toward the target location. For example, the instrument carriage 108 with the coupled instrument base 114 may be advanced from the retracted position along the insertion trajectory 103 as the distal portion 158 of the elongate flexible device 112 is advanced toward the target location 160. Thus, in this example, as the distal portion 158 is navigated to the target location 160, the elongate flexible device 112 may remain extended along the insertion trajectory 103 with little or no buckling or prolapse along the length of the elongate flexible device 112. As the distal portion 158 of the elongate flexible device 112 is manually advanced toward the target location 160, the operator may, occasionally, retract the elongate device slightly to correct course, promote lubrication, change a field of view of the imaging system, or to otherwise assist with the navigation process.

[0044] At a process 308, a determination is made that the distal portion of the elongate flexible device has reached a target location. For example, when the distal portion 158 of the elongate flexible device 112 reaches the target location 160, advancement of the elongate flexible device and the instrument carriage may be stopped, and the arrival at the target location may be recognized. In some examples, the determination that the target has been reached may be based on receipt of a user generated signal at the operator input system (e.g., pressing a button) or based on an analysis of image data of the anatomic environment in the field of view of the imaging system. Additionally or alternatively, a determination that the target location has been reached may be based on co-registered image data, kinematic data, and instrument shape sensor data. For example, the system may determine based on analysis of image data from an endoscopic camera and / or sensor data (e.g., kinematic data, instrument shape sensor data) that the elongate flexible device has navigated past sensitive anatomy and is within a region of the target location 160. For example, for insertion into the stomach or other location within the gastrointestinal tract as a target location, the system may determine that the distal portion 158 of the elongate flexible device 112 has passed the gastroesophageal (GE) junction between the esophagus and the stomach. The system may use the GE junction as an anatomical landmark for determination that the elongate flexible device 112 has reached the target location 160 for purposes of transitioning out of the hybrid control mode. In someexamples, a user may select a procedure type (such as procedures targeting the stomach or the intestines) at the operator input system (cither before or during the procedure), and the system may automatically recognize the gastro-esophageal junction as a target anatomical landmark for transitioning out of the hybrid control mode.

[0045] At a process 310, a transition may be made to a holding control mode. For example, in the holding control mode, the instrument carriage 108 to which the elongate flexible device 112 is connected may be held stationary along the insertion trajectory 103 to prevent motion of the elongate flexible device 112 further into the patient anatomy in the longitudinal degree of freedom of motion. In the holding control mode, the operator may release a manual hold on the elongate flexible device 112 and the distal portion 158 of the elongate flexible device 112 may remain at the target location 160 in the patient anatomy. While the elongate flexible device 112 remains in the holding control mode, the guide assembly 110 may be extended to support the length of the elongate flexible device 112 external to the patient. The guide assembly 110 may be extended by the user manually after releasing the user’s hold on the elongate flexible device 112. Optionally, in the holding control mode, a tensioning operation may be performed in which the instrument carriage 108 is retracted to relieve any buckling of the elongate flexible device 112 while preventing substantial advancement or retraction of the distal portion 158 of the elongate flexible device. The tensioning operation may be responsive to information about the shape of the elongate flexible device. For example, shape information from a shape sensor (e.g., shape sensor 622) extending within the elongate flexible device may indicate if any buckling or curvature has formed in the elongate flexible device and may indicate if the distal portion of the elongate flexible device has moved from the target location. In some examples, the tensioning operation may be a manual operation performed by an operator manually moving the instrument carriage or the elongate flexible device based on optional guidance provided by a shape sensor and / or the imaging system. In other examples, the tensioning operation may be controlled by the control system of the robotically-assisted medical system based on shape sensor and / or imaging data. In some examples, the transition to the holding control mode may be triggered by a user generated signal at the operator input system (e.g., pressing a button) or based on a recognition of a predetermined elapsed duration of time that the elongate flexible device has remained in a stationary position. In some examples, the transition to the holding control mode may be based on detection of anatomical features, such as a target anatomic feature, from an analysis of image data of the anatomic environment in the field of view of the imaging system. The target anatomic feature may be based on the type of procedure as selected by an operator at the start of the procedure. For example, the transition to holding control modemay be based on a detection of the distal portion 158 of the elongate flexible device 112 crossing or otherwise nearing the gastro-csophagcal junction at the stomach. In some examples, a user may select a procedure type (such as procedures targeting the stomach or the intestines) at the operator input system (either before or during the procedure), and the system may automatically recognize the gastro-esophageal junction as a target anatomical feature for transitioning to the holding control mode. In some examples, detection that the system has reached the target anatomic feature may be determined based on a combination of kinematic data from the robotically-assisted manipulator system including shape sensor data and image-based feature detection algorithms for analyzing intra-operative images. In some examples, the transition to the holding control mode may be based on detected lack of motion of the elongate flexible device 112. For example, if the elongate flexible device 112 is determined to be approximately stationary for a predetermined period of time, the transition to the holding control mode may be triggered. A transition out of the holding control mode may be triggered if a threshold force (e.g. an operator applied force) is detected on the elongate flexible device. In some examples, a determination that the elongated flexible device 112 is approximately stationary may additionally or alternatively be based on endoscopic image analysis and / or kinematic data.

[0046] At a process 312, a determination may be made that a guide assembly has been docked. The process 312 may be substantially similar to process 206, with differences as described. While in the holding control mode and after the operator has extended the guide assembly along the length of the elongate flexible device between the instrument carriage and the anatomic orifice device, the determination may be made that the guide assembly docking is complete. For example, after the guide assembly 110 is docked, the operator may provide an affirmative indication at the operator input system signaling the docking is complete. Alternatively, a sensor at the base stage may detect that the guide assembly has been connected.

[0047] At a process 314, control of the motion of the elongate flexible device may be transitioned to an assisted control mode. The process 314 may be substantially similar to the process 208, with differences as described. The control mode transition from the holding control mode (or the hybrid control mode if the holding control mode is omitted) to the assisted control mode may be triggered by a user input at the operator input system or based on the determination that the guide assembly has been docked. In the assisted control mode, the longitudinal degree of freedom of motion 170 and the steering degree of freedom 174 of the distal portion 158 of the elongate flexible device 112 may be robotically assisted, responsive to a user input at the operator input system (e.g. operator input system 700) of the robotically-assisted medical system. Optionally, the robotically-assisted gravity compensation, friction compensation, and tension support may be continued in the assisted control mode. Optionally, in the assisted control mode, the roll degree of freedom of motion 172 may be fixed, operated under robotic assistance, or manually operated. The assisted control mode may also allow an operator to finely control steering motion 174 of the distal portion 158 of the elongate flexible device via the operator input system. In the assisted control mode, an operator may receive information such as image data provided by an endoscopic imaging system, pre-operative images of the patient anatomy, or force sensor information translated into artificial haptic sensations at the operator input system to provide guidance during the procedure.

[0048] FIG. 5 is a flowchart illustrating a method 400 for navigating an elongate flexible device into a patient anatomy. At a process 402, a connection between an elongate flexible device and a carriage of a manipulator assembly may be recognized. The process 402 may be substantially similar to the process 202, with differences as described. An operator may connect the instrument base 114 of the medical instrument system 102 to the instrument carriage 108. A control system (e.g., the control system 512) may recognize the medical instrument system 102, including the elongate flexible device 112, as connected to the instrument carriage 108. In this example, the instrument carriage 108 may be positioned anywhere along the insertion trajectory when the medical instrument system 102 is coupled. The instrument carriage 108 may be advanced along the insertion trajectory 103 to an advanced position near the device connector 116. In the advanced position, the elongate flexible device may be in a slack, buckled, or otherwise prolapsed state. In some examples, the elongate flexible device 112 may be coupled to the device connector 116 as the instrument carriage 108 is advanced to the advanced position. In other examples, the elongate flexible device 112 may be coupled to the device connector 116 after the instrument carriage 108 is advanced to the advanced position. A sufficient length of the elongate flexible device 112 may extend between the instrument carriage 108 and the device connector 116 to allow the operator hand(s) to grasp the elongate flexible device. The distal portion 158 of the elongate flexible device 112 may be inserted into the anatomic orifice device 120, such as an endotracheal tube, and into the patient’s anatomic orifice after the instrument carriage 108 is moved to the advanced position so that the elongate flexible device 112 is not inadvertently advanced within the patient anatomy as the instrument carriage is moved to the advanced position. Optionally, the robotically-assisted medical system may be in a carriage control mode as the carriage is moved to the advanced position. The carriage control mode may be substantially similar to the hybrid control mode except that the robotic-assistance in the steering degrees of freedom may be disabled so that distal portion steering is not controlled in the carriage control mode. Further, the carriagecontrol mode may, optionally, disable tension support to the elongate flexible device.

[0049] At a process 404, a hybrid control mode of a robotically-assisted medical system may be engaged. The process 404 may be substantially similar to the process 204, with differences as described. In the hybrid control mode, the longitudinal motion of the distal portion of the elongate flexible device 112 in the longitudinal degree of freedom of motion (e.g., advancement or insertion motion) 170 may be operated by direct manual control, the steering degree of freedom 174 of the distal portion 158 of the elongate flexible device 112 may be robotically assisted, and sensor feedback forms of robotic assistance such as gravity compensation, friction compensation, and tension support may support the operator performing the manual insertion operation. Optionally, in the hybrid control mode, the roll degree of freedom of motion 172 may be fixed, operated under robotic assistance, or manually controlled by an operator.

[0050] At a process 406, the hybrid control mode may be maintained as the instrument carriage with the coupled instrument base remains generally stationary in the advanced position along the insertion trajectory and as the distal portion of the elongate flexible device is advanced toward the target location. For example, the instrument carriage 108 with the coupled instrument base 114 may remain generally stationary at the advanced position along the insertion trajectory 103 as the distal portion 158 of the elongate flexible device 112 is advanced toward the target location 160. Thus, in this example, as the distal portion 158 is navigated to the target location, the portion of the elongate flexible device 112 outside of the patient anatomy may be buckled or in a prolapsed state. As the distal portion of the elongate flexible device is manually advanced toward the target location, the operator may, occasionally, retract the elongate device slightly to correct course, promote lubrication, change a field of view of the imaging system, or to otherwise assist with the navigation process.

[0051] At a process 408, a determination is made that the distal portion of the elongate flexible device has reached a target location. For example, when the distal portion 158 of the elongate flexible device 112 reaches the target location 160, advancement of the elongate flexible device may be stopped, and the arrival at the target location may be recognized. In some examples, the determination that the target has been reached may be based on receipt of a user generated signal at the operator input system (e.g., pressing a button) or based on an analysis of image data of the anatomic environment in the field of view of the imaging system. Additionally or alternatively, a determination that the target location has been reached may be based on co-registered image data, kinematic data, and instrument shape sensor data.

[0052] At a process 410, the instrument carriage may be moved to a retracted position while a medialportion of the elongate flexible device is held stationary. For example, after the distal portion 158 of the elongate flexible device 112 has reached the target location 160 , a medial portion of the elongate flexible device 112 may be held generally stationary as the movement of the instrument carriage 108 is controlled by the manipulator assembly 100 proximally along the insertion trajectory 103 to extend the portion of the elongate flexible device that remains outside of the patient anatomy from a prolapsed configuration to and extended configuration along the insertion trajectory. The proximal motion of the instrument carriage 108 may be terminated when the elongate flexible device 112 is fully extended and no slack remains. Full extension may be determined based, for example, on shape sensor information, force sensor information, or a user input. As the instrument carriage 108 is moved, the medial portion of the elongate flexible device 112 may be held stationary by manual or mechanical means. For example, the operator may grasp the medial portion of the elongate flexible device 112 to hold it stationary as the instrument carriage 108 is moved proximally. Alternatively, a clamp or solenoid-activated holding device may hold the medial portion of the elongate flexible device. During the process 410, the distal portion of the elongate flexible device at the target location may, optionally, be in a limp, inactivated state. Holding the medial portion generally stationary relative to the insertion trajectory 103 may prevent accidental advancement of the elongate flexible device further into the patient anatomy or displacement from the target location. After the instrument carriage 108 is retracted and the elongate flexible device 112 is fully extended proximally of the patient anatomy, the guide assembly 110 may be extended to support the length of the elongate flexible device external to the patient.

[0053] At a process 412, a determination may be made that a guide assembly has been docked. The process 412 may be substantially similar- to process 206, with differences as described. After the operator has extended the guide assembly 110 along the length of the elongate flexible device 112 between the instrument carriage 108 and the anatomic orifice device 120, the determination may be made that the guide assembly docking is complete. For example, after the guide assembly is docked, the operator may provide an affirmative indication at the operator input system signaling the docking is complete. Alternatively, a sensor at the base stage may detect that the guide assembly has been connected.

[0054] At a process 414, control of the motion of the elongate flexible device may be transitioned to an assisted control mode. The process 414 may be substantially similar to the process 208, with differences as described. The control mode transition to the assisted control mode may be triggered by a user input at the operator input system or based on the determination that the guide assembly has been docked. In the assisted control mode, the longitudinal degree of freedom of motion 170 and the steering degree offreedom 174 of the distal portion 158 of the elongate flexible device 1 12 may be robotically assisted, responsive to a user input at the operator input system (c.g. operator input system 700) of the robotically- assisted medical system. Optionally, the robotically-assisted gravity compensation, friction compensation, and tension support may be continued in the assisted control mode. Optionally, in the assisted control mode, the roll degree of freedom of motion 172 may be fixed, operated under robotic assistance , or manually operated. The assisted control mode may also allow an operator to finely control steering motion 174 of the distal portion 158 of the elongate flexible device via the operator input system. In the assisted control mode, an operator may receive information such as image data provided by an endoscopic imaging system, pre-operative images of the patient anatomy, or force sensor information translated into artificial haptic sensations at the operator input system to provide guidance during the procedure.

[0055] Aspects of this disclosure herein can be part of a computer-assisted, teleoperational manipulator system, sometimes referred to as a robotically-assisted manipulator system or a robotic system. The manipulator system can include one or more manipulators that can be operated with the assistance of an electronic controller (e.g., computer) to move and control functions of one or more instruments when coupled to the manipulators.

[0056] FIG. 6 illustrates an embodiment of a robotically-assisted manipulator system for use with the tools described herein. The manipulator system can be used, for example, in surgical, diagnostic, therapeutic, biopsy, or non-medical procedures, and is generally indicated by the reference numeral 500. As shown in FIG. 6, a robotically-assisted manipulator system 500 can include one or more manipulator assemblies 502 (e.g. the manipulator assembly 100) for operating one or more medical instrument systems 504 (e.g., the medical instrument system 102) in performing various procedures on a patient P positioned on a table T in a medical environment 501. For example, the manipulator assembly 502 can drive catheter or end effector motion, can apply treatment to target tissue, and / or can manipulate control members. The manipulator assembly 502 can be teleoperated, non-teleoperated, or a hybrid teleoperated and nonteleoperated assembly with select degrees of freedom of motion that can be motorized and / or teleoperated and select degrees of freedom of motion that can be non-motorized and / or non-teleoperated. An operator input system 506, which can be inside or outside of the medical environment 501, generally includes one or more input control devices for controlling manipulator assembly 502. Manipulator assembly 502 supports medical instrument system 504 and can optionally include a plurality of actuators or motors that drive inputs on medical instrument system 504 in response to commands from a control system 512. The actuators can optionally include drive systems that when coupled to medical instrument system 504 canadvance medical instrument system 504 into a naturally or surgically created anatomic orifice. Other drive systems can move the distal end of medical instrument in multiple degrees of freedom, which can include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). The manipulator assembly 502 can support various other systems for irrigation, treatment, or other purposes. Such systems can include fluid systems (including, for example, reservoirs, heating / cooling elements, pumps, and valves), generators, lasers, interrogators, and ablation components.

[0057] Robotically-assisted manipulator system 500 also includes a display system 510 for displaying an image or representation of the surgical site and medical instrument system 504 generated by an imaging system 509 which can include an imaging system, such as an endoscopic imaging system. Display system 510 and operator input system 506 can be oriented so an operator O can control medical instrument system 504 and operator input system 506 with the perception of telepresence. A graphical user interface can be displayable on the display system 510 and / or a display system of an independent planning workstation.

[0058] In some examples, the endoscopic imaging system components of the imaging system 509 can be integrally or removably coupled to medical instrument system 504. However, in some examples, a separate imaging device, such as an endoscope, attached to a separate manipulator assembly can be used with medical instrument system 504 to image the surgical site. The endoscopic imaging system 509 can be implemented as hardware, firmware, software, or a combination thereof which interact with or are otherwise executed by one or more computer processors, which can include the processors of the control system 512.

[0059] Robotically-assisted manipulator system 500 can also include a sensor system 508. The sensor system 508 can include a position / location sensor system (e.g., an actuator encoder or an electromagnetic (EM) sensor system) and / or a shape sensor system (e.g., an optical fiber shape sensor) for determining the position, orientation, speed, velocity, pose, and / or shape of the medical instrument system 504. The sensor system 508 can also include temperature, pressure, force, or contact sensors or the like.

[0060] Robotically-assisted manipulator system 500 can also include a control system 512. Control system 512 includes at least one memory 516 and at least one computer processor 514 for effecting control between medical instrument system 504, operator input system 506, sensor system 508, and display system 510. Control system 512 also includes programmed instructions (e.g., a non-transitory machine- readable medium storing the instructions) to implement a procedure using the robotically-assistedmanipulator system including for navigation, steering, imaging, engagement feature deployment or retraction, applying treatment to target tissue (c.g., via the application of energy), or the like.

[0061] Control system 512 can optionally further include a virtual visualization system to provide navigation assistance to operator O when controlling medical instrument system 504 during an image- guided surgical procedure. Virtual navigation using the virtual visualization system can be based upon reference to an acquired pre-operative or intra-operative dataset of anatomic passageways. The virtual visualization system processes images of the surgical site imaged using imaging technology such as computerized tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or the like. The control system 512 can use a pre-operative image to locate the target tissue (using vision imaging techniques and / or by receiving user input) and create a preoperative plan, including an optimal first location for performing treatment. The pre-operative plan can include, for example, a planned size to expand an expandable device, a treatment duration, a treatment temperature, and / or multiple deployment locations.

[0062] FIG. 7A shows a medical instrument system 600 according to some embodiments. In some embodiments, medical instrument system 600 can be used in an image-guided medical procedure. In some examples, medical instrument system 600 can be used for non-teleoperational exploratory procedures or in procedures involving traditional manually operated medical instruments, such as endoscopy. In some embodiments, medical instrument system 600 is interchangeable with, or a variation of, medical instrument system 504 of FIG. 6.

[0063] Medical instrument system 600 includes elongate flexible device 602, such as a flexible catheter or endoscope (e.g., gastroscope, bronchoscope), coupled to a drive unit 604. Elongate flexible device 602 includes a flexible body 616 having proximal end 617 and distal end, or tip portion, 618. In some embodiments, flexible body 616 has an approximately 14-20 mm outer diameter. Other flexible body outer diameters can be larger or smaller. Flexible body 616 can have an appropriate length to reach certain portions of the anatomy, such as the lungs, sinuses, throat, or the upper or lower gastrointestinal region, when flexible body 616 is inserted into a patient’s oral or nasal cavity.

[0064] Medical instrument system 600 optionally includes a tracking system 630 for determining the position, orientation, speed, velocity, pose, and / or shape of distal end 618 and / or of one or more segments 624 along flexible body 616 using one or more sensors and / or imaging devices. The entire length of flexible body 616, between distal end 618 and proximal end 617, can be effectively divided into segments624. Tracking system 630 can optionally be implemented as hardware, firmware, software or a combination thereof which interact with or arc otherwise executed by one or more computer processors, which can include the processors of control system 512 in Fig. 6.

[0065] Tracking system 630 can optionally track distal end 618 and / or one or more of the segments 624 using a shape sensor 622. In some embodiments, tracking system 630 can optionally and / or additionally track distal end 618 using a position sensor system 620, such as an electromagnetic (EM) sensor system. In some examples, position sensor system 620 can be configured and positioned to measure six degrees of freedom, e.g., three position coordinates X, Y, Z and three orientation angles indicating pitch, yaw, and roll of a base point or five degrees of freedom, e.g., three position coordinates X, Y, Z and two orientation angles indicating pitch and yaw of a base point.

[0066] Flexible body 616 includes one or more channels 621 sized and shaped to receive one or more medical instruments 626. In some embodiments, flexible body 616 includes two channels 621 for separate instruments 626, however, a different number of channels 621 can be provided. FIG. 7B is a simplified diagram of flexible body 616 with medical instrument 626 extended according to some embodiments. In some embodiments, medical instrument 626 can be used for procedures and aspects of procedures, such as surgery, biopsy, ablation, mapping, imaging, illumination, irrigation, or suction. Medical instrument 626 can be deployed through channel 621 of flexible body 616 and used at a target location within the anatomy. Medical instrument 626 can include, for example, image capture devices, biopsy instruments, ablation instruments, catheters, laser ablation fibers, and / or other surgical, diagnostic, or therapeutic tools. Medical tools can include end effectors having a single working member such as a scalpel, a blunt blade, a lens, an optical fiber, an electrode, and / or the like. Other end effectors can include, for example, forceps, graspers, balloons, needles, scissors, clip appliers, and / or the like. Other end effectors can further include electrically activated end effectors such as electrosurgical electrodes, transducers, sensors, imaging devices and / or the like. Medical instrument 626 can be advanced from the opening of channel 621 to perform the procedure and then retracted back into the channel when the procedure is complete. Medical instrument 626 can be removed from proximal end 617 of flexible body 616 or from another optional instrument port (not shown) along flexible body 616. The medical instrument 626 can be used with an image capture device (e.g., an endoscopic camera) also within the elongate flexible device 602. Alternatively, the medical instrument 626 can itself be the image capture device.

[0067] Medical instrument 626 can additionally house cables, linkages, or other actuation controls (not shown) that extend between its proximal and distal ends to controllably bend the distal end of medicalinstrument 626. Flexible body 616 can also house cables, linkages, or other steering controls (not shown) that extend between drive unit 604 and distal end 618 to controllably bend distal end 618 as shown, for example, by broken dashed line depictions 619 of distal end 618. In some examples, at least four cables are used to provide independent “up-down” steering to control a pitch motion of distal end 618 and “leftright” steering to control a yaw motion of distal end 618. In embodiments in which medical instrument system 600 is actuated by a robotically-assisted assembly, drive unit 604 can include drive inputs that removably couple to and receive power from drive elements, such as actuators, of the teleoperational assembly. In some embodiments, medical instrument system 600 can include gripping features, manual actuators, or other components for manually controlling the motion of medical instrument system 600. The information from tracking system 630 can be sent to a navigation system 632 where it is combined with information from visualization system 631 and / or the preoperatively obtained models to provide the physician or other operator with real-time position information.

[0068] FIG. 8A is a perspective view of an operator input system 700 (e.g. the operator input system 506), and FIG. 8B is a top view of the operator input system 700. The operator input system 700 includes a housing 702, an input control device 704, and an input control device 706. In some examples, the input control device 704 may be a scroll wheel, and the input control device 706 may be a track ball. In some examples, the input control device 704 has an infinite length of travel and may be spun in either direction (e.g., forward and backward). In some cases, the input control device 706 has an infinite length of travel and may be spun about any number of axes. In some examples, the most common movements of the input control device 706 may be combinations of a left and right rotation, a forward and backward rotation, and a spin in place rotation. In alternative embodiments, one or both of the input control devices 704, 706 may be touch pads, joysticks, touch screens, and / or the like. In some examples, the input control device 704 may receive user inputs to actuate robotically-assisted motion of an elongate flexible device (e.g., elongate flexible device 112) in a longitudinal degree of freedom (e.g. insertion and retraction). In some examples, the input control device 706 may receive user inputs to actuate robotically-assisted motion of the elongate flexible device in steering degrees of freedom (e.g. pitch and yaw motion of the distal portion of the elongate flexible device. An operator input to the operator input system may include any form of touch to any of the input control devices. For example, an operator input may include an operator touch to a capacitive touch input control device. For example, an operator input may include a touch to a movable input control device (e.g. a trackball or scroll wheel) without moving or commanding motion of the input control device.

[0069] In the description, specific details have been set forth describing some examples. Numerous specific details arc set forth to provide a thorough understanding of the examples. It will be apparent, however, to one skilled in the ait that some examples may be practiced without some or all these specific details. The specific examples disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure.

[0070] Elements described in detail with reference to one example, implementation, or application optionally may be included, whenever practical, in other examples, implementations, or applications in which they are not specifically shown or described. For example, if an element is described in detail with reference to one example and is not described with reference to a second example, the element may nevertheless be claimed as included in the second example. Thus, to avoid unnecessary repetition in the description, one or more elements shown and described in association with one example, implementation, or application may be incorporated into other examples, implementations, or aspects unless specifically described otherwise, unless the one or more elements would make an example or implementation nonfunctional, or unless two or more of the elements provide conflicting functions. Not all the illustrated processes may be performed in all examples of the disclosed methods. Additionally, one or more processes that are not expressly illustrated in may be included before, after, in between, or as part of the illustrated processes. In some examples, one or more of the processes may be performed by a control system or may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors may cause the one or more processors to perform one or more of the processes.

[0071] Any alterations and further modifications to the described devices, instruments, methods, and any further application of the principles of the present disclosure are fully contemplated as would normally occur to one skilled in the ait to which the disclosure relates. In addition, dimensions provided herein are for specific examples and it is contemplated that different sizes, dimensions, and / or ratios may be utilized to implement the concepts of the present disclosure. To avoid needless descriptive repetition, one or more components or actions described in accordance with one illustrative example can be used or omitted as applicable from other illustrative examples. For the sake of brevity, the numerous iterations of these combinations will not be described separately. For simplicity, in some instances the same reference numbers are used throughout the drawings to refer to the same or like parts.

[0072] The systems and methods described herein may be suited for imaging and treatment, via natural or surgically created connected passageways, in any of a variety of anatomic systems, including the lung, colon, the intestines, the stomach, the liver, the kidneys and kidney calices, the brain, the heart, the circulatory system including vasculature, and / or the like. While some examples are provided herein with respect to medical procedures, any reference to medical or surgical instruments and medical or surgical methods is non-limiting. For example, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, and sensing or manipulating non-tissue work pieces. Other example applications involve cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, and training medical or non-medical personnel. Additional example applications include use for procedures on tissue removed from human or animal anatomies (without return to a human or animal anatomy) and performing procedures on human or animal cadavers. Further, these techniques can also be used for surgical and nonsurgical medical treatment or diagnosis procedures.

[0073] The methods described herein may be illustrated as a set of operations or processes that may be performed in the same or in a different order than the order shown in the provided flowcharts. One or more of the illustrated processes may be omitted in some examples of the method. Additionally, one or more processes that are not expressly illustrated in the flowchart may be included before, after, in between, or as part of the illustrated processes. In some examples, one or more of the processes of a method may be implemented, at least in part, by a control system executing code stored on non-transitory, tangible, machine-readable media that when run by one or more processors (e.g., the processors of a control system) may cause the one or more processors to perform one or more of the processes.

[0074] One or more elements in examples of this disclosure may be implemented in software to execute on a processor of a computer system such as control processing system. When implemented in software, the elements of the examples of this disclosure may be code segments to perform various tasks. The program or code segments can be stored in a processor readable storage medium or device that may have been downloaded by way of a computer data signal embodied in a carrier wave over a transmission medium or a communication link. The processor readable storage device may include any medium that can store information including an optical medium, semiconductor medium, and / or magnetic medium. Processor readable storage device examples include an electronic circuit; a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storagedevice. The code segments may be downloaded via computer networks such as the Internet, Intranet, etc. Any of a wide variety of centralized or distributed data processing architectures may be employed. Programmed instructions may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein. In some examples, the control system may support wireless communication protocols such as Bluetooth, Infrared Data Association (IrDA), HomeRF, IEEE 802.11, Digital Enhanced Cordless Telecommunications (DECT), ultra- wideband (UWB), ZigBee, and Wireless Telemetry.

[0075] Note that the processes and displays presented might not inherently be related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the operations described. The required structure for a variety of these systems will appear’ as elements in the claims. In addition, the examples of the invention are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.

[0076] This disclosure describes various instruments, portions of instruments, and anatomic structures in terms of their state in three-dimensional space. As used herein, the term position refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term orientation refers to the rotational placement of an object or a portion of an object (e.g., in one or more degrees of rotational freedom such as roll, pitch, and / or yaw). As used herein, the term pose refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (e.g., up to six total degrees of freedom). As used herein, the term shape refers to a set of poses, positions, or orientations measured along an object.

[0077] While certain illustrative examples of the invention have been described and shown in the accompanying drawings, it is to be understood that such examples are merely illustrative of and not restrictive on the broad invention, and that the examples of the invention are not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art.

Claims

CLAIMSWhat is claimed is:1 A system comprising: a robotically-assisted manipulator assembly; an elongate flexible device; and a control system including a processing unit and memory comprising machine readable instructions that, when executed by the processing unit, cause the control system to: recognize the elongate flexible device is connected to a carriage of the manipulator assembly, engage a hybrid control mode in which longitudinal motion of the elongate flexible device along an anatomic path is responsive to a manual force and steering motion of a distal portion of the elongate flexible device is responsive to a user input at an operator input system, determine a guide assembly has been docked, and based on the determination that the guide assembly has been docked, transition from the hybrid control mode to an assisted control mode in which motion of the elongate flexible device is responsive to robotically-assisted control of longitudinal and steering motions of the elongate flexible device.2 The system of claim 1, wherein in the hybrid control mode, a motion of the carriage along an insertion trajectory is responsive to the manual force.3 The system of claim 1, wherein in the hybrid control mode, the carriage remains stationary at a distal staging position.4 The system of claim 1, wherein in the hybrid control mode, the machine readable instructions, when executed by the processing unit further cause the control system to apply gravity compensation to a motion of the carriage, in response to sensor data.

5. The system of claim 1 , wherein in the hybrid control mode, the machine readable instructions, when executed by the processing unit further cause the control system to apply friction compensation to a motion of the carriage, in response to sensor data.6 The system of claim 1, wherein in the hybrid control mode, the distal portion of the elongate flexible device has a default limp state that transitions to an active state in response to the user input at the operator input system.7 The system of claim 6, wherein in the hybrid control mode, the distal portion of the elongate flexible device reverts to the default limp state in response to an elapsed time without an input at the operator input system.8 The system of claim 1, wherein in the hybrid control mode, a roll motion of the elongate flexible device relative to a longitudinal axis of the elongate flexible device is responsive to a manual force.9 The system of claim 1, wherein in the hybrid control mode, a roll motion of the elongate flexible device relative to a longitudinal axis of the elongate flexible device is responsive to the user input at the operator input system.10 The system of claim 1, wherein in the assisted control mode, motion of the elongate flexible device is responsive to robotically-assisted control of roll motion of the elongate flexible device.11 The system of claim 1, wherein the machine readable instructions, when executed by the rocessing unit further cause the control system to determine that the distal portion of the elongate flexible device has reached a target location.12 The system of claim 11, wherein the determining that the distal portion of the elongate flexible device has reached a target location is based on the user input at the operator input system.13 The system of claim 11, wherein the determining that the distal portion of the elongate flexible device has reached a target location is based on at least one of analysis of image data from an imagingsystem carried by the elongate flexible device or kinematic data of the elongate flexible device.

14. The system of claim 13, wherein the determining that the distal portion of the elongate flexible device has reached the target location is based on both the analysis of image data from the elongate flexible device and kinematic data of the elongate flexible device.

15. The system of claims 13 or 14, wherein the target location is identified based upon a procedure type selected by a user input at the operator input system.

16. The system of claim 1, wherein the machine readable instructions, when executed by the rocessing unit further cause the control system to transition to a holding control mode in which the carriage is held stationary while the distal portion of the elongate flexible device is at a target location.

17. The system of claim 16, wherein the transitioning to the holding control mode is based on the user input at the operator input system.

18. The system of claim 17, wherein the transitioning to the holding control mode is based on an elapsed time value during which the carriage does not move along an insertion trajectory.

19. The system of claim 16, wherein the transitioning to the holding control mode is based on at least one of an analysis of image data from the elongate flexible device or kinematic data of the elongate flexible device.

20. The system of claim 19, wherein the transitioning to the holding control mode is based on both the analysis of image data from the elongate flexible device and kinematic data of the elongate flexible device.

21. The system of claim 19 or 20, wherein the transitioning to the holding control mode comprises detecting that the elongate flexible device has crossed or is near a target anatomical feature.

22. The system of claim 21, wherein the target anatomical feature is identified based upon aprocedure type selected by a user input at the operator input system.

23. The system of claim 1, wherein the transitioning to the assisted control mode is based on the user input at the operator input system that the guide assembly has been docked.

24. The system of claim 1, wherein the transitioning to the assisted control mode is based on a confirmation of docking of the guide assembly by a docking sensor.

25. The system of claim 1, wherein engaging the hybrid control mode is in response to the user input at the operator input system.

26. The system of claim 1, wherein engaging the hybrid control mode is in response to a signal from a force sensor of the manipulator assembly.

27. The system of claim 1, wherein engaging the hybrid control mode is in response to a signal from a joint sensor of the manipulator assembly.

28. The system of claim 1, wherein the carriage is in a retracted position on the manipulator assembly when the elongate flexible device is recognized to be in connection with the carriage.

29. The system of claim 1, wherein the machine readable instructions, when executed by the processing unit further cause the control system to determine that the carriage has advanced to a distal staging position along an insertion direction.

30. The system of claim 1, wherein the machine readable instructions, when executed by the processing unit further cause the control system to control motion of the carriage to a retracted position along an insertion direction while a medial portion of the elongate flexible device is held stationary.

31. The system of claim 30, wherein the medial portion is held stationary by a holding device.

32. The system of claim 31, wherein the medial portion is held stationary by a manual force.

33. A method for navigating an elongate flexible device, the method comprising: recognizing the elongate flexible device is connected to a carriage of a manipulator assembly; engaging a hybrid control mode in which longitudinal motion of the elongate flexible device along an anatomic path is responsive to a manual force and steering motion of a distal portion of the elongate flexible device is responsive to a user input at an operator input system; determining a guide assembly has been docked; and based on the determination that the guide assembly has been docked, transitioning from the ybrid control mode to an assisted control mode in which motion of the elongate flexible device is esponsive to robotically-assisted control of longitudinal and steering motions of the elongate flexible device.

34. The method of claim 33, wherein in the hybrid control mode, a motion of the carriage along an insertion trajectory is responsive to the manual force.

35. The method of claim 33, wherein in the hybrid control mode, the carriage remains stationary at a distal staging position.

36. The method of claim 33, further comprising applying gravity compensation to a motion of the carriage, in response to sensor data.

37. The method of claim 33, further comprising applying friction compensation to a motion of the carriage, in response to sensor data.

38. The method of claim 33, wherein in the hybrid control mode, the distal portion of the elongate flexible device has a default limp state that transitions to an active state in response to the user input at the operator input system.

39. The method of claim 38, wherein in the hybrid control mode, the distal portion of the elongate flexible device reverts to the default limp state in response to an elapsed time without an input at the operator input system.

40. The method of claim 33, wherein in the hybrid control mode, a rotational motion of the elongate flexible device is responsive to a manual force.

41. The method of claim 33, wherein in the hybrid control mode, a rotational motion of the elongate flexible device is responsive to the user input at the operator input system.

42. The method of claim 33, further comprising determining that the distal portion of the elongate flexible device has reached a target location.

43. The method of claim 40, wherein the determining that the distal portion of the elongate flexible device has reached a target location is based on the user input at the operator input system.

44. The method of claim 43, wherein the determining that the distal portion of the elongate flexible device has reached a target location is based on at least one of analysis of image data from an imaging system carried by the elongate flexible device or kinematic data of the elongate flexible device.

45. The method of claim 44, wherein the determining that the distal portion of the elongate flexible device has reached the target location is based on both the analysis of image data from the elongate flexible device and kinematic data of the elongate flexible device.

46. The method of claim 44 or 45, wherein the target location is identified based upon a procedure type selected by a user input at the operator input system.

47. The method of claim 33, further comprising transitioning to a holding control mode in which the carriage is held stationary while the distal portion of the elongate flexible device is at a target location.

48. The method of claim 47, wherein the transitioning to the holding control mode is based on the user input at the operator input system.

49. The method of claim 47, wherein the transitioning to the holding control mode is based on an elapsed time value during which the carriage does not move along an insertion trajectory.

50. The method of claim 47, wherein the transitioning to the holding control mode is based on at least one of an analysis of image data from the elongate flexible device or kinematic data of the elongate flexible device.

51. The method of claim 50, wherein the transitioning to the holding control mode is based on both the analysis of image data from the elongate flexible device and kinematic data of the elongate flexible device.

52. The method of claim 50 or 51, wherein the transitioning to the holding control mode comprises detecting that the elongate flexible device has crossed or is near a target anatomical feature.

53. The method of claim 52, wherein the target anatomical feature is identified based upon a rocedure type selected by a user input at the operator input system.

54. The method of claim 33, wherein the transitioning to the assisted control mode is based on the user input at the operator input system that the guide assembly has been docked.

55. The method of claim 33, wherein the transitioning to the assisted control mode is based on a confirmation of docking of the guide assembly by a docking sensor.

56. The method of claim 33, wherein engaging the hybrid control mode is in response to the user input at the operator input system.

57. The method of claim 33, wherein engaging the hybrid control mode is in response to a signal from a force sensor of the manipulator assembly.

58. The method of claim 33, wherein engaging the hybrid control mode is in response to a signal from a joint sensor of the manipulator assembly.

59. The method of claim 33, wherein the carriage is in a retracted position on the manipulatorassembly when the elongate flexible device is recognized to be in connection with the carriage.

60. The method of claim 33, further comprising determining that the carriage has advanced to a distal staging position along an insertion trajectory.

61. The method of claim 33, further comprising controlling motion of the carriage to a retracted osition along an insertion trajectory while a medial portion of the elongate flexible device is held stationary.

62. The method of claim 61, wherein the medial portion is held stationary by a holding device.

63. The method of claim 61, wherein the medial portion is held stationary by a manual force.

64. A system comprising: a robotically-assisted manipulator assembly; an elongate flexible device; and a control system including a processing unit and memory comprising machine readable instructions that, when executed by the processing unit, cause the control system to: recognize the elongate flexible device is connected to a carriage of the manipulator assembly, engage a hybrid control mode in which longitudinal motion of the elongate flexible device along an anatomic path and roll motion of the elongate flexible device relative to a longitudinal axis of the elongate flexible device are each responsive to a manual force and steering motion of a distal ortion of the elongate flexible device is responsive to a user input at an operator input system, and transition from the hybrid control mode to an assisted control mode in which motion of the elongate flexible device is responsive to robotically-assisted control of longitudinal, roll, and steering motions of the elongate flexible device.

Images