System and method for controlling sequential workflows

Abstract

A manual user interface system includes a knob movable in at least one rotational degree of freedom and at least one translational degree of freedom, and an actuator system coupled with the knob and responsive to a plurality of movements of the knob to operate a plurality of instrument devices of a robotic-assisted instrument system through a plurality of predetermined stages of a sequential workflow.

Description

[0001] Cross-reference to related applications This application claims priority and benefit to U.S. Provisional Application No. 63 / 591,537, filed October 19, 2023, entitled "Systems and Methods for Controlling a Sequential Workflow," the entire contents of which are incorporated herein by reference. This patent application also relates to U.S. Provisional Application No. 63 / 591,591, filed October 19, 2023, entitled "Systems and Methods for Actuating an Instrument System," the entire contents of which are incorporated herein by reference. This patent application also relates to International Application PCT / US2024 / 0 [Case No. P06766-WO2], filed October 15, 2024, entitled "Systems and Methods for Actuating an Instrument System," the entire contents of which are incorporated herein by reference. Technical Field

[0002] The examples described herein relate to systems and methods for controlling sequential workflows. More specifically, examples may relate to the use of multi-degree-of-freedom user interface devices that actuate multiple functions of an instrument system during the sequential workflow of a medical procedure. Background Technology

[0003] Minimally invasive medical techniques are typically designed to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. These techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, the operator can insert minimally invasive medical devices, such as therapeutic, diagnostic, imaging, and surgical instruments. Some minimally invasive medical device systems can be controlled using multiple user input devices and may require multiple hands or multiple operators for actuation. There is a need for systems and methods that reduce the complexity of user input required to perform medical procedures. Summary of the Invention

[0004] The following is a brief summary of the various examples described herein, and is not intended to identify key or important elements or to indicate the scope of the claims.

[0005] In some instances, a manual user interface system includes a knob movable in at least one rotational degree of freedom and at least one translational degree of freedom, and an actuator system coupled to the knob and responding to multiple movements of the knob to operate multiple instrumentation devices of a robot-assisted instrumentation system through multiple predetermined stages of a sequential workflow.

[0006] In some instances, the manual user interface system includes a knob and an actuator system movable in at least one rotational degree of freedom. The actuator system is coupled to the knob and operates the device of the robot-assisted instrument system through a forward sequence phase of the workflow in response to rotation of the knob in a first direction of the rotational degree of freedom, and operates the device through a reverse sequence phase of the workflow in response to rotation of the knob in a second direction of the rotational degree of freedom.

[0007] In some instances, a method includes receiving a first user input at a manual user interface. The manual user interface is movable in multiple degrees of freedom. The method also includes, in response to the first user input at the manual user interface, moving a first instrumentation device of the robot-assisted instrumentation system during a first predetermined phase of a sequential workflow, and receiving a trigger associated with the end of the first predetermined phase of the sequential workflow. The method further includes receiving a second user input at the manual user interface and, in response to the second user input, moving a second instrumentation device of the robot-assisted instrumentation system during a second predetermined phase of the sequential workflow.

[0008] In some instances, a method includes receiving first user input in a first degree of freedom of the user interface at a manual user interface to cause a first instrument device to perform a first action, and receiving second user input in a second degree of freedom of the user interface at a manual user interface to cause a second instrument device to perform a second action. The method also includes receiving third user input in the first degree of freedom of the user interface at the manual user interface to cause the second instrument device to perform a third action.

[0009] In some instances, a method includes receiving first user input in a first degree of freedom of the user interface at a manual user interface to cause a first instrument device to perform a first action, and receiving second user input in the same first degree of freedom to cause a second instrument device to perform a second action. The method also includes receiving third user input in the same first degree of freedom of the user interface to cause the second instrument device to perform a third action.

[0010] It should be understood that the foregoing general description and the following detailed description are illustrative and explanatory in nature and are intended to provide an understanding of the contents of this disclosure, without limiting its scope. In this regard, additional aspects, features, and advantages of this disclosure will become apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0011] Figure 1A and 1B The manual user interface system is shown based on some examples.

[0012] Figure 2 The distal end of an instrument system for creating suture patterns is shown according to some examples.

[0013] Figure 3A The diagram illustrates an operator input system that includes a manual user interface, based on several examples.

[0014] Figure 3B The diagram illustrates an operator input system that includes a manual user interface, based on several examples.

[0015] Figure 3C The diagram illustrates an operator input system that includes a manual user interface, based on several examples.

[0016] Figure 4 This is a flowchart illustrating a method for controlling an instrument system based on some examples.

[0017] Figure 5 This is a flowchart illustrating predetermined stages in a sequential workflow for a suturing procedure, which can be performed using an instrument system, based on some examples.

[0018] Figure 6 This is a flowchart illustrating, according to some examples, a method for controlling an instrument system having a single manual user interface device with movement or responsiveness in multiple degrees of freedom.

[0019] Figure 7 This is a flowchart illustrating, according to some examples, a method for controlling an instrument system having a single manual user interface device with a single degree of freedom of movement or responsiveness.

[0020] Figure 8A The diagram illustrates, based on some examples, a graphical user interface displayed on the screen of a manual user interface device.

[0021] Figure 8B The diagram illustrates a control panel that includes a manual user interface device and a display screen, based on several examples.

[0022] Figure 8C The diagram illustrates a graphical user interface that can be displayed on a screen, based on some examples.

[0023] Figure 8DThe graphical user interface shown, based on some examples, includes virtual features that provide a virtual representation of instrument system components that may be obscured by tissue in the field of view of the imaging system.

[0024] Figure 8E The diagram illustrates a graphical user interface that can be displayed on a screen, based on some examples.

[0025] Figure 9A The illustration shows an instrument actuation system for one or more instrument devices that can be used to actuate an instrument system, based on some examples.

[0026] Figure 9B It shows Figure 9A The opposite side of the instrument's actuation device.

[0027] Figure 9C It shows Figure 9A Details of the actuation mechanism.

[0028] Figure 10 The diagram illustrates, based on several examples, a device actuator connected to a device system.

[0029] Figure 11 It is a simplified diagram of the patient's anatomy based on some examples.

[0030] Figure 12 These are schematic diagrams of robot-assisted manipulator systems based on some examples.

[0031] Figure 13A This is a schematic diagram of an instrument system based on an example.

[0032] Figure 13B It shows Figure 13A The distal end of the instrument system, which has an extended instance of the instrument according to the example.

[0033] Figure 14 The diagram schematically illustrates an instrument actuation system for one or more instrument devices that can be used to actuate an instrument system, based on some examples.

[0034] Figure 15A The illustration shows an instrument actuation system for one or more instrument devices that can be used to actuate an instrument system, based on some examples.

[0035] Figure 15B It shows Figure 15A The opposite side of the instrument's actuation device.

[0036] Figure 16A The illustrations show operators manipulating instruments connected to instrument actuators, based on several examples.

[0037] Figure 16BThe remote manual user interface device is shown according to some examples.

[0038] Figure 17 The remote manual user interface device is shown according to some examples.

[0039] Figure 18A The illustration shows an instrument actuation system for one or more instrument devices that can be used to actuate an instrument system, based on some examples.

[0040] Figure 18B It shows Figure 18A The opposite side of the instrument's actuation device.

[0041] Examples of this disclosure and its advantages can be best understood by referring to the following detailed description. It should be understood that the same reference numerals are used to identify the same elements shown in one or more figures, which are shown in order to illustrate examples of this disclosure and not to limit its scope. Detailed Implementation

[0042] The techniques described herein provide endoscopic techniques and processing systems utilizing multi-degree-of-freedom manual user interface devices. These techniques and processing systems can be used to perform medical procedures such as surgery, biopsy, suturing, ablation, illumination, irrigation, or aspiration. Some procedures can be performed in the upper or lower gastrointestinal tract, including, for example, fistula closure, endoscopic submucosal dissection, endoscopic mucosal resection, transoral endoscopic myotomy, endoscopic sleeve gastrectomy, and / or transoral outlet reduction endoscopy. While the examples provided herein can be used for suturing gastrointestinal tissue, it should be understood that the described techniques can be used to perform procedures in artificially created lumens or any endoluminal passages or chambers, including in the patient's trachea, colon, intestines, stomach, liver, kidneys and renal calyces, brain, heart, circulatory system including the vascular system, fistulas, and the like.

[0043] Figure 1A A perspective view of the manual user interface system 100 is shown. Figure 1BA side view of a manual user interface system 100 is shown, which includes a single manual user interface device, such as a knob 102, that can be operated by a user's single hand actuation to perform multiple actions on an instrument system. The knob 102 may include a body 108 having a circular, cylindrical, spherical, hemispherical, mushroom-shaped, or other shape adapted to interaction with a human hand. For example, the knob's size may conform to the space formed by a human hand's fingers. The knob body 108 may include a top surface 110 and a side surface 112. The manual user interface system 100 may also include an actuator system 104 and a controller 106. The actuator system 104 may be coupled to the knob body 108 and may include an actuator 114, such as a brushless DC motor or a brushed DC motor. The actuator system 104 may also include an encoder 116. Data from the encoder 116 may drive the movement of an instrument system (e.g., instrument system 200) through one or more operations, actions, or functions. Controller 106 can communicate with actuator system 104 and with instrument actuation device (i.e., instrument actuation device 802). Controller 106 can be, for example, an Arduino-based controller, microcontroller, computer, or other computing device. In some instances, knob 102 may include display screen 120, which may include a touchscreen to receive user input via touch gestures from one or more fingers. The touchscreen may include, for example, a resistive touch panel, a capacitive touch panel, an infrared touch panel, a surface acoustic wave touch panel, or an optical imaging touch panel. The touchscreen can be used to operate secondary instrument systems such as imaging systems. In some instances, manual user interface system 100 can be a control device for an operator input system (e.g., operator input system 1106), and controller 106 can be a component of a control system (e.g., control system 1112) for a robot-assisted medical system (e.g., medical system 1100).

[0044] Knob 102 can be movable in multiple degrees of freedom in response to input from a user. For example, the knob can be translated 122 in the + and / or -X directions (e.g., right and left), 124 in the + and / or -Y directions (e.g., up and down), and / or 126 in the + and / or -Z directions (e.g., forward and backward). Knob 102 can provide spring-loaded or tactile force button action to return the knob to a central position after receiving translation input. Alternatively or concurrently, knob 102 can move in a clockwise or counterclockwise direction during rotation 130. In some instances, knob 102 may include one or more force sensors 118 that can detect and respond to forces 132 or pressures simultaneously applied to the side surface 112 (e.g., simultaneously in the + / -X directions), such as forces associated with squeezing or gripping actions at knob 102. In some instances, knob 102 may include display screen 120, which may include a touchscreen to receive user input via touch gestures 134 from one or more fingers. The touchscreen may include, for example, a resistive touch panel, a capacitive touch panel, an infrared touch panel, a surface acoustic wave touch panel, or an optical imaging touch panel. In some instances, knob 102 may provide haptic output 136, such as vibration or rotational click (e.g., a click sensation).

[0045] As described below, knob 102 can be used to control multiple instrument devices of an instrument system, controlling a specific instrument device according to the current stage of a predetermined workflow. In various instances, the motion or degrees of freedom of knob 102 can be omitted or locked to constrain available motion inputs or outputs. Constraints can be based on, for example, the currently controlled instrument device or the current stage of a sequential workflow. For example, for some programs or some controlled instrument devices, knob 102 can be restricted to bidirectional rotational and translational motion in the + / -Y direction. For some programs, specific instruments, specific functions of instruments, or specific degrees of freedom of motion can be omitted, constrained, or limited. In various instances, combined motions can be achieved based on user input. For example, when the end effector of an instrument grasps tissue, the knob can move in the translational degree of freedom and also be squeezed to cause translational motion of the robot-assisted instrument device.

[0046] Any of the various instrument systems can be operated with robot assistance in response to user input at the manual user interface system 100. Instrument systems may include suturing, biopsy, ablation, imaging, treatment delivery, or other instrument systems used in medical procedures. Figure 2The distal end of an instrument system 200 is shown, which can be robotically assisted to perform suturing procedures in response to user input at a manual user interface system 100. The instrument system 200 may include an elongated device 202 and a suturing apparatus 204. In some instances, the instrument system 200 may be an endoscopic instrument system, and the elongated device 202 may be a flexible, manipulable endoscope. The elongated device 202 may serve as a platform for driving the movement of mechanisms in the suturing apparatus 204, delivering working parts to the anatomical structure accessed by the instrument system 200, and capturing image data of the anatomical structure. The device 202 may include a flexible body 206, with one or more working channels 208, 209 extending through the flexible body 206. The working channels 208, 209 may extend through the flexible body 206 to provide access for a removable instrument system, suture material, and / or termination components, and to allow instrument exchange during the procedure. In some instances, suture support tools such as tissue alignment tool 218 and needle exchange tool 219 may be detachably extended within the working channel 208. A needle tip capturing mechanism 221 may be positioned at the end of needle exchange tool 219 and may operate independently of the insertion and retraction movements of needle exchange tool 219. Tissue alignment tool 218 and needle exchange tool 219 may be detachable, and various other suture support tools may extend within the working channel. In some instances, tissue alignment tool 218 may include a helical component that can rotate to engage tissue. In other instances, the alignment tool may include opposing jaws, clamps, or other tissue grasping mechanisms to engage and move tissue. Other suture support tools may include, for example, tissue alignment tool sheaths, suture tensioning tools, end-cap transfer tools, tightening tools, needle tip exchange tools, or other tools used as part of a suturing procedure. Working channels 208 and 209 may also, or alternatively, allow fluid passage, deliver vacuum pressure, or otherwise provide a pathway between the proximal and distal ends of the elongated flexible device 202. Each working channel may terminate at an opening in the distal end 212 of the device 202. For example, working channel 208 may terminate at opening 210.

[0047] The suturing device 204 may include a suture needle 214, which may be coupled to a suture 216 or other types of natural or synthetic suture material. The suture 216 may be connected to the suture needle 214. The suture needle 214 may be straight, curved, semi-circular, or another suitable shape for capturing suture occlusion. The suture needle 214 may include a needle arm and at least one end having a tip configured to pierce tissue. The suturing device 204 may also include a tissue occlusion chamber 220 into which tissue may be drawn in a folded, bent, clamped, or other occlusion configuration. In some instances, the suture needle 214 may move along a linear path. In some instances, the suture needle 214 may move from a ready position and return to a ready position along an arcuate path in a single direction. In some instances, the suture needle 214 may move in a bidirectional arcuate path. Optionally, the imaging system 240 may also extend through the flexible body 206 (e.g., imaging system 1109 or a component thereof). Imaging system 240 may be an integrated component of device 202 (e.g., permanently coupled), or it may be slidably received within and detachable from flexible body 206. In some instances, imaging system 240 may extend, hinge, steer, or otherwise move relative to flexible body 206. In some instances, imaging system 240 may use one or more flexible optical fibers to transmit images. Digital image-based imaging systems may have a “chip-on-the-tip” design, in which a remote digital sensor, such as one or more charge-coupled devices (CCDs) or complementary metal-oxide-semiconductor (CMOS) devices, stores image data. Imaging system 240 may capture two-dimensional or three-dimensional image data. For example, a stereo imaging system may employ a stereo camera to capture stereoscopic images of a patient's anatomy.

[0048] The instrument system 200 can be used to perform a suturing procedure that places suture 216 in tissue 224 through a series of suture occlusions 222A-222E. The suture occlusions 222A-222E can form a suture pattern 235. A suture end member 230 can be attached to the suture 216 near the initial suture occlusion 222A, and an end member 232 can be attached to the suture 216 near the final suture occlusion 222E. Tightening of the suture 216 can be achieved by pulling the suture in a tightening direction D1 (e.g., as the end member 230 moves toward the suture occlusion 222A, away from the initial suture occlusion 222A) or a tightening direction D2 (e.g., as the end member 232 moves toward the suture occlusion 222E, away from the final suture occlusion 222E). In some instances, folding can be achieved by at least partially tensioning in both directions D1 and D2. Some aspects of the instrument system and method of use are described in U.S. Provisional Patent Application No. 63 / 518,228, entitled "Systems and Methods for Endoscopic Tissue Suturing," filed August 8, 2023, and in U.S. Provisional Patent Application No. 63 / 518,208, entitled "Systems and Methods for Endoscopic Tissue Suturing," filed August 8, 2023, the entire contents of which are incorporated herein by reference.

[0049] The manual user interface system 100 can control the actuation of multiple instrument devices or components of the instrument system 200. For example, the instrument devices of the instrument system 200 controllable by the manual user interface system may include a suture needle 214, a tissue alignment tool 218, a sheath of the tissue alignment tool 218, a needle tip capturing mechanism 221, and / or a needle exchange tool 219. For example, the manual user interface system 100 can control the linear or rotational movement of the suture needle 214. The manual user interface system 100 can also, or alternatively, control the advance, retraction, and / or rotation of the alignment tool 218. The manual user interface system 100 can also, or alternatively, control the gripping and / or releasing actions of the needle tip capturing mechanism 221. The manual user interface system 100 can also, or alternatively, control the advance and / or retraction of the needle tip exchange tool 219. The manual user interface system 100 can also, or alternatively, control the advance and / or retraction of the sheath of the tissue alignment tool 218. In some instances, the manual user interface system 100 can also, or alternatively, control the advance and / or retraction of a tightening tool. In some instances, the manual user interface system 100 can control the actuation required to perform needle exchange, suture loading, and / or suture tensioning procedures.

[0050] The manual user interface system 100 can be a control device for an operator input system (e.g., operator input system 1106) of a robot-assisted medical system (e.g., medical system 1100). In some instances, such as Figure 3A As shown, the operator input system 300 may include a control panel 301 on which a manual user interface system 302 is mounted. The manual user interface system 302 may be substantially similar to the manual user interface system 100. The control panel 301 may also include control devices 304 and 306. In some instances, control device 304 may be a roller, and control device 306 may be a trackball. The control panel 301 may also include a display screen 308, which may display a graphical user interface optionally including touchscreen control devices. In some instances, the manual user interface system 302 may be used to control the actuation of suture needle 214 and one or more suture support tools such as tissue alignment tool 218 and tightening tool (not shown). The imaging system may be controlled by control devices 304 and 306. The imaging system may be imaging system 240, or it may be another endoscopic imaging system coupled to a separate robot-assisted manipulator. For example, roller control device 304 may control the insertion / retraction movement of imaging system 240, and trackball control device 306 may control the steering of imaging system 240. In alternative instances, the control panel may include other types of control devices to receive user input and control the movement of the imaging system 240. In alternative instances, the manual user interface system 302 may be the only control device mounted on the control panel. In alternative instances, the manual user interface system 302 may be mounted on a control panel with fewer, more, or different user control devices.

[0051] In some instances, such as Figure 3BAs shown, the operator input system 320 may include a control panel 321, and manual user interface systems 322 and 324 are mounted on the control panel 321. Manual user interface systems 322 and 324 may be substantially similar to manual user interface system 100. The control panel 301 may also include a display screen 328, which may display a graphical user interface optionally including a touchscreen control device. In this example, manual user interface system 322 may control the actuation of instrument system 200, and manual user interface system 324 may control the actuation of different instrument systems, such as those connected to different robot-assisted manipulators, or manual user interface systems 322 and 324 may each control the actuation of various aspects of the general instrument system 200. The imaging system may be controlled by touchpad 330 of manual user interface system 322 and touchpad 332 of manual user interface system 324. The imaging system may be imaging system 240, or it may be another endoscopic imaging system connected to a separate robot-assisted manipulator. For example, touchpad 330 may control the insertion / retraction movement of the imaging system, and touchpad 332 may control the steering of the imaging system.

[0052] In some instances, such as Figure 3CAs shown, the operator input system 350 may include a control panel 351, a manual user interface system 352, and a manual user interface system 354 mounted on the control panel 351. The manual user interface systems 352 and 354 may be substantially similar to the manual user interface system 100. In this example, the manual user interface system 352 may be connected to the panel 351 via a pivot mechanism 353, and the manual user interface system 354 may be connected to the panel 351 via a pivot mechanism 355. The pivot mechanisms 353 and 355 may each allow the manual user interface systems 352 and 354 to move in additional or different degrees of freedom. For example, the pivot mechanisms may allow the manual user interface systems to pivot in response to user input commands for pitch and yaw steering of the instrument device for the instrument system 200. The control panel 301 may also include a display screen 358, which may display a graphical user interface optionally including a touchscreen control device. In this example, manual user interface system 352 can control the actuation of instrument system 200, and manual user interface system 354 can control the actuation of different instrument systems, such as those connected to different robot-assisted manipulators, or manual user interface systems 352 and 354 can each control the actuation of various aspects of the general instrument system 200. The imaging system can be controlled by touchpad 360 of manual user interface system 352 and touchpad 362 of manual user interface system 354. The imaging system can be imaging system 240 or another endoscopic imaging system connected to a separate robot-assisted manipulator. For example, touchpad 360 can control the insertion / retraction movement of the imaging system, and touchpad 362 can control the steering of the imaging system.

[0053] In some instances, manual user interface systems (e.g., systems 100, 302, 322, 324, 352, 354) can be used to control the actuation of an instrument system (e.g., instrument system 200) during the execution of a sequential workflow consisting of predetermined stages associated with the functions of the instrument devices of the instrument system's components. As described in the methods and workflows below, multiple or all movements, actuations, and functions of the instrument system can be controlled by a single manual user interface (e.g., knob 102). Compared to methods requiring multiple user interfaces or multiple human users to execute the same sequential procedure, a single knob that controls multiple instrument devices at various speeds, directions, and forces at multiple or all stages of the sequential procedure can significantly simplify the user experience. Figure 4 This is a flowchart illustrating a method 400 for controlling an instrument system. Figure 5This is a flowchart illustrating predetermined stages in a sequential workflow 450 of a suturing procedure that can be performed by the instrument system 200. The methods 400 and other methods described herein for controlling the instrument system may refer to predetermined stages of the sequential workflow 450 of the suturing procedure; however, the methods 400 and manual user interface systems described herein are not limited to use in suturing procedures. In alternative instances, Figure 4 , 6 The methods described in section 7 can be used with instrument systems to perform any of a variety of procedures with a sequential workflow having predetermined stages. Such procedures may include, for example, dissection, biopsy, ablation, irrigation, or other interventional medical procedures.

[0054] For reference Figure 5 As described, the sequential workflow 450 of the suturing procedure can be executed using the instrument system 200. Each stage of the sequential workflow can correspond to one or a series of movements, actuations, or functions of the component instruments of the instrument system 200. In some stages, a single component instrument of the instrument system can be actuated. In other stages, two or more component instruments of the instrument system can be actuated.

[0055] In stage 452, after the instrument system (e.g., instrument system 200) has been positioned appropriately for suturing the occlusion (e.g., suturing occlusion 222A), the controlled component instrument device may be a suture needle (e.g., suture needle 214), which can be moved by a retraction movement. For example, the curved suture needle 214 can be retracted by rotating the needle to the open position. For example, if the curved suture needle 214 is already in the open position, stage 452 may optionally be omitted.

[0056] In stage 454, the controlled component instrument may be a tissue engagement tool (e.g., engagement tool 218) extending from the working channel 208. In some instances, the engagement tool may include a helical component extending toward the tissue surface. The helical component may be pushed forward against the tissue. In other instances, the engagement tool may use opposing jaws, clamps, or other tissue grasping mechanisms to engage the tissue. During stage 454, the tissue engagement tool may be controlled to move in the insertion or retraction direction.

[0057] In stage 456, the controlled component instrument can again be a tissue alignment tool, but the controlled movement can be a rotational movement along a first direction. For example, the helical alignment tool 218 can be rotated such that the helical component enters the tissue in a clockwise helical motion. Optionally, within stage 456, the tissue alignment tool can be rotated in a second direction opposite to the first direction (e.g., counterclockwise) to finely adjust the depth of the helical component within the tissue. For example, if the helical thread tissue alignment tool initially rotates too deep into the tissue, the reverse thread of the alignment tool can remove some of the helical thread from the tissue. Optionally, in stage 456, the tissue alignment tool can be rotated and further extended. In other words, stage 456 can allow control of both the translational and rotational movements of the tissue alignment tool 218.

[0058] In stage 458, the controlled component device may be an imaging system (e.g., imaging system 240 or an imaging system coupled to a different robot-assisted manipulator) that can move forward, retract, change direction, or be otherwise manipulated between or in parallel with any stage of workflow 450 (e.g., during or in combination with any stage of workflow 450).

[0059] At stage 460, the controlled component instrument can again be a tissue alignment tool. The tissue alignment tool and the tissue correspondingly engaged with it can be retracted. For example, once the helical component of the tissue alignment tool 218 has been driven to the desired depth in the tissue, the helical component can be retracted, bringing the tissue closer to the suturing device and into the path of the suture needle 214.

[0060] In stage 462, the controlled component instrument may be a suture needle. The suture needle may be advanced into the occlusive tissue. For example, an arcuate suture needle 214 may be advanced by rotating the needle to a closed position while the tissue occlusion tool 218 holds the joined tissues in the occlusive position.

[0061] In stage 464, the controlled component instrument may be a needle exchange tool that can advance into the suturing device. For example, needle exchange tool 219 can advance through working channel 209. For example, stage 464 may be optionally omitted if the needle tip capturing mechanism 221 attached to needle exchange tool 219 is already in a sufficient position in working channel 209.

[0062] At stage 466, the controlled component instrument may be a needle tip capturing mechanism that can remove the needle tip portion of the suture needle from the body of the suture needle. For example, needle tip capturing mechanism 221 can grasp and remove the tip of the suture needle 214 attached to the suture. Needle tip capturing mechanism 221 can grasp, remove, and retain the needle tip portion, including the attached suture.

[0063] In stage 468, the controlled component instrument may again be a tissue alignment tool that can rotate in a second direction opposite to the first direction to remove the tissue alignment tool from the tissue. For example, a spiral alignment tool 218 may rotate such that the spiral component moves in a counterclockwise spiral motion until the spiral component becomes unthreaded and separates from the tissue.

[0064] At stage 470, the controlled component instrument can again be a retractable tissue alignment tool. For example, after removal from the tissue, the spiral alignment tool 218 can be retracted from the tissue and enter the working channel 208.

[0065] In stage 472, the controlled component instrument can be a suture needle that can be retracted from the tissue. For example, an arcuate suture needle 214 can be retracted by rotating the needle to the open position.

[0066] In stage 474, the controlled component instrument can again be a suture needle. The suture needle can advance and re-engage with its tip. For example, an arcuate suture needle 214 can advance by rotating the needle to a closed position, where it re-engages with a needle tip capturing mechanism 221 that holds the needle tip portion. After re-engagement, the suture needle can be modified with its tip and attached suture. Stage 474 can optionally be performed after the instrument system (e.g., instrument system 200) has moved, for example, to a position suitable for another suture engagement (e.g., suture engagement 222B).

[0067] refer to Figure 4 The method 400 for controlling an instrument system can be described with reference to stages 452 and 454 of a sequential workflow 450, but can also be used with any stage of workflow 450 or other sequential workflows. Method 400 can focus on the transition between two predetermined stages of a sequential workflow, such as stages 452 and 454 of sequential workflow 450. (Refer to...) Figure 4 In process 402, the first user input can be received at a manual user interface system that is movable in multiple degrees of freedom. For example, the manual user interface system 100 includes a knob 102 that is movable in translation and rotation degrees of freedom and can be translated or rotated by user input.

[0068] In process 404, in response to a first user input, the first instrument device can be moved in a first predetermined stage of the sequential workflow. For example, the suture needle 214 of the instrument system 200 can be retracted in response to a user input at knob 102 in a predetermined stage 452 of the sequential workflow 450.

[0069] In process 406, a trigger associated with the end of a first predetermined stage of the sequential workflow can be received. For example, user input can provide a trigger associated with the end of stage 452, where the suture needle is retracted. Triggers indicating a transition between workflow stages can be user input, such as pressing knob 102 (e.g., movement in the -Y direction) or a signal from a sensor indicating that a predetermined threshold has been reached. For example, a force sensor reading can indicate that tissue has been touched or punctured. In other instances, an encoder reading can indicate that the tool has moved a predetermined distance. For example, processes 402-406 can be associated with retracting the suture needle at stage 452 of the sequential workflow 450.

[0070] In process 408, a second user input can be received at the manual user interface system. For example, the knob 102 can be translated or rotated with the same degrees of freedom as the first user input, or it can be moved with different degrees of freedom.

[0071] In process 410, in response to a second user input, the second instrument device may be moved in a second predetermined stage of the sequential workflow. For example, the tissue alignment tool 218 of the instrument system 200 may be extended in response to a user input at knob 102 in a predetermined stage 454 of the sequential workflow 450.

[0072] In some instances, a single manual user interface device, such as knob 102, can move or respond to user input forces in multiple translational and / or rotational directions to perform a sequential workflow using the instruments of an instrument system. At each stage of the sequential workflow, user input at the knob will result in a predetermined movement or action of one or more instruments. The actuated instruments and the type of actuation that occurs may depend on the current stage of the sequential workflow. For example, in the first stage of the sequential workflow, user input at the manual user interface device may actuate a first instrument to perform an action, but in the second stage of the sequential workflow, user input at the manual user interface device (even input with the same degrees of freedom as the knob) may actuate a second instrument to perform a different action. Figure 6 This is a flowchart illustrating a method 500 for controlling an instrument system with a single manual user interface device having movement or responsiveness in multiple degrees of freedom. This type of control can allow users greater discretion when performing stages of a sequential workflow and may be suitable for users with extensive experience or proprietary technology.

[0073] In other instances, the knob may be constrained to move in a single translational or rotational direction or to respond to force in order to execute a sequential workflow. Figure 7 This is a flowchart illustrating a method 550 for controlling an instrument system having a single manual user interface device that moves or responds to force in a single degree of freedom to perform a sequential workflow. For example, the movement of knob 102 can be restricted to rotational movement to perform all stages of the sequential workflow. Alternatively, other degrees of freedom of the single manual user interface device can be assigned to perform administrative functions not directly related to the task of performing the sequential workflow. For example, knob 102 can be restricted to rotational movement to perform individual stages, but administrative functions for registering the start or end of a stage can be assigned to translational movement of knob 102 (e.g., a button press action in the -Y direction). Figure 7 The control types described herein can restrict user discretion during the execution of sequential workflow phases and are suitable for procedures with minimal complexity. In some instances, the control method can be selected based on program complexity, user experience level, or other factors. In other instances, the instrument system can be controlled using a single manual interface that can move in two or more degrees of freedom. In some instances, the degrees of freedom of movement of the single manual interface may be locked or available depending on the type of program, the phase of the sequential workflow, or other factors or conditions.

[0074] refer to Figure 6 As described in method 500, the movement of an instrument system device to perform a predetermined sequential workflow can be controlled by multiple degrees of freedom of movement or input at a single manual user interface device. The sequential workflow of the intervention procedure may include stages defined by the functions of the instrument system. Various functions or actions of the instrument device can be controlled by various inputs to the manual user interface system. The user inputs associated with each instrument device action may be predetermined or may be pre-selected by the user. Method 500 may refer to the various stages of the sequential workflow 450 performed by instrument system 200; however, method 500 is not limited and can be used to perform any sequential workflow. Figure 7 Compared to method 550, Figure 6 Method 500 allows for finer control and greater user discretion, but may require more user awareness and training.

[0075] In process 502, a first user input can be received in a first degree of freedom of motion at the manual user interface system to cause the first instrument device to take a first action. The first action can be associated with a first stage of a sequential workflow. For example, and referring to sequential workflow 450, the first user input can be a rotation of knob 102 (e.g., a first degree of freedom) to retract or move the suture needle 214 (e.g., a second instrument device) to an open position (e.g., the first action) at stage 452 of workflow 450. In this example, a counterclockwise rotation of knob 102 (e.g., around the Y-axis) can retract the suture needle 214. User input at knob 102 can activate a binary action associated with stage 452, such as a single automatic retraction movement of the suture needle. Alternatively, variable user input at knob 102 can result in a corresponding amount of movement of the suture needle. For example, a quarter turn of knob 102 can be associated with a 90-degree rotation (or another predetermined rotation) of the suture needle. Rotation of knob 102 can be a full 360-degree rotation, a 90-degree rotation of the knob, or another predetermined amount of rotation corresponding to a predetermined amount of movement of the instrument. In some instances, user input in the opposite direction of the first degree of freedom can move the first instrument in the opposite direction. Reversing the direction allows the user to finely control the amount of movement of the instrument. In some instances, tactile sensations, such as a clicking sensation, corresponding to the movement of the first instrument can be transmitted to the user via knob 102. In some instances, input at the manual user interface system can signal the completion of a stage and, optionally, can proceed to the next stage of the workflow. For example, user input in the -Y direction (e.g., pressing down) can be a predetermined signal indicating the end of stage 452. In other instances, the completion of a stage can be determined by sensor information, imaging information, or other input sources. In some instances, the instrument can be held or kept stationary at the end of a stage while the manual user interface responds to user input for subsequent stages. For example, after knob 102 is rotated to retract the suture needle and stage 452 has been completed, the suture needle can be held or kept in the retracted position without further input at knob 102. The procedure can be switched to stage 454, and the tissue alignment tool 218 can be extended in response to user input at knob 102 while the suture needle remains in the retracted position.

[0076] In process 504, a second user input can be received at the manual user interface system at the second degree of motion to cause the second instrument device to take a second action. The second action can be associated with a second stage of the sequential workflow. For example, and referring to sequential workflow 450, the second user input could be, at stage 454 of workflow 450, translating or pushing knob 102 forward (e.g., in the +Z direction) (e.g., second degree of freedom) to extend tissue alignment tool 218 (e.g., the second instrument device) from work channel 208 (e.g., the second action). In some instances, variable user input at knob 102 can result in a corresponding amount of movement of the tissue alignment tool. For example, a 2mm movement of knob 102 in the +Z direction can be associated with a 1mm advance (or another predetermined distance) of the tissue alignment tool. In some instances, user input in the opposite direction of the second degree of freedom can move the second instrument device in the opposite direction. Reversing the direction allows the user to optionally finely control the amount of movement of the tissue alignment tool under visual guidance from an imaging system. In some instances, tactile sensations, such as a clicking sensation, corresponding to the movement of the first instrument can be transmitted to the user via knob 102. In some instances, input at the manual user interface system can signal the completion of a stage and, optionally, advance to the next stage of the workflow. For example, user input in the -Y direction (e.g., pressing down) can be a predetermined signal indicating the end of stage 454. In other instances, the completion of this stage can be determined by other input sources.

[0077] In process 506, a third user input can be received at the manual user interface system at the first degree of motion to cause the second instrument device to take a third action. The third action can be associated with a third stage of a sequential workflow. For example, and referring to sequential workflow 450, the third user input could be rotation of knob 102 (e.g., the first degree of freedom) to cause rotation (e.g., the third action) of tissue alignment tool 218 (e.g., the second instrument device) at stage 456 of workflow 450. In some instances, clockwise rotation of knob 102 (e.g., about the Y-axis) can cause the helical tissue alignment tool 218 to rotate toward the tissue, and counterclockwise rotation of the knob can cause the helical tissue alignment tool to rotate away from the tissue. In some instances, variable user input at knob 102 can result in a corresponding amount of rotation of the tissue alignment tool. Reversing the direction can allow the user to optionally finely control the amount of movement of the tissue alignment tool under visual guidance from an imaging system. In some instances, tactile sensations, such as a clicking sensation, corresponding to the movement of the first instrument can be transmitted to the user via knob 102. In some stages of a sequential workflow, two or more actuators of an instrument may be activated simultaneously. For example, in process 506, a spiral tissue alignment tool may extend while rotating. In this example, rotational and translational user input may be transmitted to knob 102. Some stages of a sequential workflow may allow simultaneous input of different degrees of freedom, while other stages may allow user input of a single degree of freedom. In some instances, input at the manual user interface system may signal the completion of a stage and, optionally, may proceed to the next stage of the workflow. For example, user input in the -Y direction (e.g., pressing down) may be a predetermined signal indicating the end of stage 456. In other instances, the completion of a stage may be determined by other input sources.

[0078] In process 508, a fourth user input, in the opposite direction to the second user input, can be received in the second degree of freedom of motion at the manual user interface system to cause the second instrument device to take a fourth action in the opposite direction to the second action. The fourth action can be associated with a fourth stage of the sequential workflow. For example, and referring to sequential workflow 450, the fourth user input could be a translation or pull (e.g., second degree of freedom) of knob 102 in a backward direction (e.g., the -Z direction, opposite to the +Z direction of process 504) at stage 460 of workflow 450, causing retraction (e.g., the fourth action) of the tissue alignment tool 218 (e.g., the second instrument device) and any tissue attached to the helical component of the tissue alignment tool. In some instances, the user input at knob 102 can result in a corresponding amount of movement of the tissue alignment tool. In some instances, a tactile sensation, such as a clicking sensation, corresponding to the movement of the first instrument can be transmitted to the user via knob 102. In some instances, the input at the manual user interface system can signal the completion of a stage and, optionally, can proceed to the next stage of the workflow. For example, user input in the -Y direction (e.g., pressing down) could be a predetermined signal indicating the end of phase 460. In other instances, the completion of a phase could be determined by other input sources.

[0079] In process 510, a fifth user input may be received at the tracking device of the manual user interface system to manipulate the imaging system. For example, the fifth user input may be an interaction with display screen 120 that, at stage 458 of workflow 450, results in manipulation (e.g., insertion / retraction or orientation change) of the imaging system (e.g., imaging system 240 or an imaging system coupled to a different robot-assisted manipulator). Process 510 may be optional and may occur during or between any stages of processes 502-508.

[0080] Alternatively, additional user input at a manual user interface system (e.g., knob 102) may enable the instrument devices of an instrument system (e.g., instrument system 200) to take additional actions at each remaining stage (e.g., stages 462-474) of the sequential workflow 450.

[0081] refer to Figure 7As described in method 550, the movement of an instrument system device through a predetermined sequential workflow can be controlled by movement or input at a single degree of freedom at a single manual user interface device. The sequential workflow of the intervention procedure may include stages defined by the functions of the instrument system. Various functions or actions of the instrument device can be controlled within a single degree of freedom via input to the manual user interface system. User input at a single degree of freedom can be used to advance predetermined stages of the sequential workflow and can be predetermined or pre-selected by the user. In some instances, user input at a single degree of freedom can advance through stages of the sequential workflow in a first direction or move through stages of the sequential workflow in the opposite direction and in the reverse order. For example, rotation of knob 102 in a first direction of the rotational degree of freedom can operate the instrument system through stages of the workflow in a forward sequence, and rotation of the knob in a second direction of the rotational degree of freedom opposite to the first direction can operate the device through stages of the workflow in a reverse sequence. In some instances, rotation in the second direction can rewind or cancel a currently or previously completed stage of the workflow. Although the operation of the instrument during a predetermined sequential workflow phase can be controlled by movement in a single degree of freedom using a manual user interface device, the manual user interface device can move in additional degrees of freedom to perform administrative or supportive tasks, such as providing indication of the end of a predetermined workflow phase or providing input on a knob's touchpad to control the imaging system. Method 550 may refer to the phases of the sequential workflow 450 performed by the instrument system 200; however, Method 550 is not limited and can be used to perform any sequential workflow. Compared to Method 500, Method 550 may allow for more automated control and requires less user discretion, but may allow for less granular control of actuation at each phase of the sequential workflow.

[0082] In process 552, a first user input can be received in a first degree of freedom of motion at the manual user interface system to cause the first instrument device to perform a first action. The first action can be associated with a first stage of a sequential workflow. For example, and referring to sequential workflow 450, the first user input can be rotation of knob 102 (e.g., the first degree of freedom) to retract or move the suture needle 214 (e.g., the first instrument device) to an open position (e.g., the first action) at stage 452 of workflow 450. In this example, counterclockwise rotation of knob 102 (e.g., around the Y-axis) can retract the suture needle 214. In some instances, user input in the opposite direction of the first degree of freedom can move the first instrument device in the opposite direction. Reversing the direction can allow the user to undo or reverse the movement of the instrument device. The rotation of knob 102 to complete the action can be a full 360-degree rotation, a 90-degree rotation of the knob, or another predetermined amount of rotation corresponding to a predetermined amount of movement of the instrument device. In some instances, tactile sensations, such as a clicking sensation, corresponding to the movement of the first instrument can be conveyed to the user via knob 102. In some instances, the instrument can be held or kept stationary at the end of a stage, while the manual user interface responds to user input for subsequent stages. For example, after knob 102 has been rotated to retract the suture needle and stage 452 has been completed, the suture needle can be held or retained in the retracted position without further input at the knob. The procedure can then proceed to stage 454, and the tissue alignment tool 218 can extend in response to user input at knob 102 while the suture needle remains in the retracted position.

[0083] In some instances, input at a manual user interface system in a degree of freedom different from the first degree of freedom can indicate the completion of a stage and optionally prompt a transition to the next stage of the workflow. For example, user input in the -Y direction (e.g., pressing down) can be a predetermined signal indicating the end of stage 452. In some instances, transitions between stages of a sequential workflow can be triggered by sensor data, which may include image data. For example, force sensor data can indicate that the occlusion device has contacted the tissue and can trigger a transition from the advance stage (e.g., stage 454) to the rotation stage (e.g., stage 456). As another example, encoder data can indicate that the suture needle has moved to the retracted position (e.g., stage 452) and can trigger a transition to the occlusion device advance stage (e.g., stage 454). As yet another example, analyzed imaging data or device sensor data (e.g., position sensor information) can indicate that the occlusion device and the connected tissue have been fully retracted into the tissue occlusion chamber, thereby triggering the end of the occlusion device retraction stage (e.g., stage 460) and the transition to the needle advance stage (e.g., stage 462).

[0084] In process 554, a second user input can be received at the same first degree of freedom of motion in the manual user interface system to cause the second instrument device to take a second action. The second action can be associated with a second stage of the sequential workflow. For example, and referring to sequential workflow 450, the second user input can be rotation of knob 102 (e.g., first degree of freedom) to extend the tissue alignment tool 218 (e.g., the second instrument device) from the work channel 208 at stage 454 of workflow 450 (e.g., second action). In some instances, during this stage, user input in the opposite direction of the first degree of freedom can move the second instrument device in the opposite direction. Reversing the direction allows the user to optionally finely control the amount of movement of the tissue alignment tool under visual guidance from an imaging system. Continuing rotation in the opposite direction of the first degree of freedom can regress or cancel the currently or previously completed stage of the workflow. In some instances, tactile sensations corresponding to the movement of the first instrument, such as a clicking sensation, can be transmitted to the user via knob 102. In some instances, input or received sensor information at the manual user interface system can signal the completion of a phase and, optionally, advance to the next phase of the workflow.

[0085] In process 556, a third user input can be received at the manual user interface system at the first degree of motion freedom to cause the second instrument device to take a third action. The third action can be associated with a third stage of a sequential workflow. For example, and referring to sequential workflow 450, the third user input could be rotation of knob 102 (e.g., the first degree of freedom) to cause rotation of tissue alignment tool 218 (e.g., the second instrument device) at stage 456 of workflow 450 (e.g., the third action). In some instances, during stage 456, clockwise rotation of knob 102 (e.g., about the Y-axis) causes the helical tissue alignment tool 218 to rotate toward the tissue, and counterclockwise rotation causes the helical tissue alignment tool to rotate away from the tissue. Reversing the direction allows the user to optionally finely control the amount of movement of the tissue alignment tool under visual guidance from an imaging system. In some instances, tactile sensations corresponding to the movement of the first instrument, such as a clicking sensation, can be transmitted to the user via knob 102. In some instances, input or received sensor information at the manual user interface system can signal the completion of a phase and, optionally, advance to the next phase of the workflow.

[0086] In process 558, a fourth user input can be received at the manual user interface system at the first degree of motion to cause the second instrument device to take a fourth action in the opposite direction to the second action. The fourth action can be associated with a fourth stage of the sequential workflow. For example, and referring to sequential workflow 450, the fourth user input could be rotation of knob 102 (e.g., the first degree of freedom) to retract the tissue alignment tool 218 (e.g., the second instrument device) and any tissue attached to the helical component of the tissue alignment tool at stage 460 of workflow 450 (e.g., the fourth action). In some instances, tactile sensations corresponding to the movement of the first instrument, such as a clicking sensation, can be transmitted to the user via knob 102. In some instances, input or received sensor information at the manual user interface system can signal the completion of the stage, and optionally, progress can be made to the next stage of the workflow.

[0087] In process 560, a fifth user input may be received at the tracking device of the manual user interface system to manipulate the imaging system. For example, the fifth user input may be an interaction with the touchscreen of display 120, which results in manipulation (e.g., insertion / retraction or orientation change) of the imaging system (e.g., imaging system 240 or an imaging system coupled to a different robot-assisted manipulator) at stage 458 of workflow 450. Process 510 may be optional and may occur during or between any stages of processes 552-558.

[0088] Alternatively, additional user input at the manual user interface system (e.g., knob 102) under the first degree of motion freedom may cause the instrument device of the instrument system (e.g., instrument system 200) to take additional actions in each remaining stage (e.g., stages 462-474) of the sequential workflow 450.

[0089] At each stage of the sequential workflow of the execution program, the graphical user interface can provide the user with images, graphical representations, text information, sensor measurement data, or other display information that may be related to the program. Figure 8AA graphical user interface 600 is shown that can be displayed on a display screen 601 of a knob 602. Knob 602 can be substantially similar to knob 102. Display screen 601 can display information related to the current stage of a sequential workflow, and the displayed information can change as the workflow transitions to the next stage. In this example, display screen 601 can display information related to stage 454 of workflow 450. For example, in stage 454, the tissue alignment tool can be a spiral alignment device, and display screen 601 can graphically show measurements of the extension of the alignment device. When the workflow transitions from stage 454 to stage 456, the information displayed on display screen 601 can be converted to information related to the rotation of the tissue alignment tool in stage 456. Alternatively or additionally, the information displayed on display screen 601 can be displayed on other displays of the robot-assisted medical system, such as displays 308, 328, 358, and 1110.

[0090] Figure 8B A control panel 650 (e.g., control panels 301, 321, 351) is shown, including a knob 652 (e.g., knob 102) and a display screen 654. The display screen 654 may display a graphical user interface including icons 656 corresponding to each stage of a sequential workflow (e.g., a stage of sequential workflow 450). In some instances, the stages of the sequential workflow may be defined by markers 658, which divide the sequential workflow into subroutines. A stage indicator 660, such as a highlighted ring, may indicate the stage of the sequential workflow in which the knob is currently active. In some instances, the stage indicator may be an arrow on the display screen of the knob 652 pointing to the currently active stage. In some instances, the icons 656 associated with stages of the sequential workflow may be arranged in an arc shape around a portion of the knob 652. In other instances, the icons may have a linear arrangement, a circular arrangement, or other suitable arrangement for displaying the sequential workflow.

[0091] Figure 8C A graphical user interface 700 that can be displayed on a display screen 701 is shown. The display screen 701 may be located on a knob 102 or on other displays of a robot-assisted medical system, such as displays 308, 328, 358, and 1110. The display screen 701 may provide an endoscopic camera view 702 from an imaging system (e.g., imaging system 240) and may include stage indicators, such as a highlighted title for the current stage 704 (e.g., a stage of sequential workflow 450), and titles for the previous stage 706 and the next stage 708 displayed sequentially with the current stage 704.

[0092] Figure 8DA graphical user interface 700 is shown, which includes virtual or ghosting features 710 that provide a virtual representation of instrument system components that may be obscured by tissue in the imaging system's field of view. Alternatively, ghosting features can provide a preview, virtual representation of the intended position of the instrument system components relative to the tissue at the end of the current stage of the workflow.

[0093] Figure 8E A graphical user interface 720 that can be displayed on a display screen 721 is shown. The display screen 721 can be located on a knob 102 or on other displays of a robot-assisted medical system, such as displays 308, 328, 358, and 1110. The display screen 721 can provide an endoscopic camera view 722 from an imaging system (e.g., imaging system 240) and can also display a stage indicator 724 with information related to the current stage of the sequential workflow. In this example, the display screen 721 can display information related to stage 456 of workflow 450. For example, in stage 456, the tissue alignment tool can be a spiral alignment device, and the display screen 701 can graphically display relevant information, such as measurements or graphical representations of the extension and / or rotation of the alignment device, via the stage indicator 724.

[0094] The aforementioned manual user interface system and the method for controlling an instrument system using the manual user interface system can be implemented in a robot-assisted medical system using an instrument actuation device, which can be a component of a manipulator assembly (e.g., manipulator assembly 1102). For example, the instrument actuation device can be coupled to and supported by a robotic actuated arm mounted on a floor, wall, fixed console, or movable console within a medical environment. In various instances, the arm can be robotically actuated, manually actuated, or have a combination of robotic and manual actuation. In other instances, the instrument actuation device can be carried by an operator or coupled to an instrument system (e.g., a manual endoscope) carried by an operator or coupled to a mounted instrument system. As used herein, a manual endoscope refers to an endoscope whose distal joints are operated by a manual control (e.g., manually at a manual control in the proximal region of the endoscope). Figures 9A-9C An example of an instrument actuation system 800 that can be used to actuate a predetermined sequence of workflows is shown, but other instrument actuation systems may also be suitable. In some instances, the instrument actuation system 800 can be used to actuate instruments in response to non-predetermined operator commands. Figure 9AA device actuation system 800 is illustrated, which can be used to actuate one or more device devices in a device system. The device actuation system 800 may include a manual user interface system 100 that transmits user input from a knob 102 to a device actuation device 802. The device actuation device 802 may include actuation mechanisms 804-814, each actuation mechanism being assigned to provide motion for the degrees of freedom of the device devices in the device system (e.g., device system 200). Actuation mechanisms 804-814 may receive control signals from a controller 106 to actuate the functions of the device devices. The control signals for the actuation mechanisms may be received wirelessly or via a wired connection. Actuation mechanisms 804-814 may be powered, for example, via cables operatively connected to a power source or battery. Figure 9B The mechanical actuator 802 is shown in relation to Figure 9A The opposite side is shown. Figure 9C Detailed parts of the actuation mechanism 804 are shown.

[0095] The device actuation device 802 may include a frame 816, which includes a proximal platform 818, a distal platform 820, and three elongated support members 822, 824, and 826 extending between platforms 818 and 820. The elongated support members 822, 824, and 826 may extend generally parallel to the longitudinal axis A1, and platforms 818 and 820 may extend generally perpendicular to the longitudinal axis A1. In alternative embodiments, fewer or more horizontal support platforms may be included, and fewer or more elongated support members may be included. The device actuation device 802 may also include elongated threaded members 828, 830, and 832 (e.g., lead screws) extending generally parallel to the longitudinal axis A1 between platforms 818 and 820.

[0096] Each actuation mechanism 804-814 can drive independent movement of the instrument device of an instrument system (e.g., instrument system 200) in a degree of freedom (including reciprocating or bidirectional movement in the degree of freedom). For example, actuation mechanism 804 can drive the insertion and retraction movement of the shaft of a tissue alignment tool (e.g., tissue alignment tool 218). Actuation mechanism 806 can drive the insertion and retraction movement of the sheath of the tissue alignment tool. Actuation mechanism 808 can drive a needle tip capturing mechanism (e.g., needle tip capturing mechanism 221). Actuation mechanism 810 can drive the insertion and retraction movement of a needle exchange tool (e.g., needle exchange tool 219). Actuation mechanism 812 can drive bidirectional rotational movement of a tissue alignment tool (e.g., tissue alignment tool 218). Actuation mechanism 814 can drive bidirectional movement of a suture needle (e.g., suture needle 214).

[0097] Actuation mechanism 804 may include servo mechanism or motor 840, actuator housing 842, gear system 844 ( Figure 9CThe actuator housing 842 includes a housing chamber 848 supporting the motor 840 and a channel 850 sized for passage of the elongated support member 822. The device 852 can be detachably coupled to the connector 846. The device 852 may include a shaft 854 and a proximal engagement portion 856 at the proximal end of the shaft 854, the proximal engagement portion 856 engaging the connector 846 in a transverse direction (e.g., generally perpendicular to the longitudinal axis A1). The device connector 846 allows for quick release and replacement of the device 852. A gear system 844 connects the output of the motor 840 to the elongated threaded member 828. When actuated by the motor 840, movement of the gear system 844 causes the actuating mechanism 804 to travel proximal or distal along the elongated threaded member 828 in direction A1. As the actuating mechanism 804 moves in direction A1, the attached device 852 also moves in this direction. In some instances, the instrument 852 may be a tissue alignment tool 218, and the motor 840 may drive the insertion and retraction movements of the tissue alignment tool 218.

[0098] Actuation mechanism 806 may include a servo mechanism or motor 860, actuator housing 862, gear system (not shown, but similar to gear system 844), and device connector 866. Actuator housing 862 includes a housing chamber 868 supporting motor 860 and includes a channel 870 sized for passage of elongated support member 822. Device 872 may be detachably coupled to connector 866. Device 872 may include shaft 874 and a proximal coupling 876 at the proximal end of shaft 874, the proximal coupling 876 engaging connector 866 in a transverse direction (e.g., generally perpendicular to longitudinal axis A1). Device connector 866 may allow for quick release and replacement of device 872. The gear system of actuation mechanism 806 may connect the output of motor 860 to elongated threaded member 828. When actuated by motor 860, movement of gear system may cause actuation mechanism 806 to travel proximal or distal along elongated threaded member 828 in direction A1. When the actuating mechanism 806 moves in direction A1, the attached instrument device 872 also moves accordingly. In some instances, the instrument device 872 may be a sheath / coat for a tissue alignment tool 218, and the motor 860 may drive the insertion and retraction movements of the sheath. The actuating mechanisms 804 and 806 may be driven independently, allowing the sheath to extend or retract relative to the tissue alignment tool 218. In some instances, the shaft 854 of the instrument device 852 may extend within and move relative to the shaft 874.

[0099] Actuation mechanism 808 may include a servo mechanism or motor 880, actuator housing 882, gear system (not shown, but similar to gear system 844), and device connector 886. Actuator housing 882 includes a housing chamber 888 supporting motor 880 and includes a channel 890 sized for passage of elongated support member 824. Device 892 may be detachably coupled to connector 886. Device 892 may include shaft 894 and a proximal coupling 896 at the proximal end of shaft 894, the proximal coupling 896 engaging connector 886 in a transverse direction (e.g., generally perpendicular to longitudinal axis A1). Device connector 886 may allow quick release and replacement of device 892. The gear system of actuation mechanism 808 may connect the output of motor 880 to elongated threaded member 832. When actuated by motor 880, movement of the gear system may cause actuation mechanism 808 to travel proximal or distal along elongated threaded member 832 in direction A1. When the actuation mechanism 808 moves in direction A1, the attached instrument device 892 also moves. In some instances, the instrument device 892 may be a needle tip capturing mechanism 221, and the motor 880 may drive the capturing and releasing movements of the needle tip capturing mechanism 211.

[0100] Actuation mechanism 810 may include a servo mechanism or motor 900, actuator housing 902, gear system (not shown, but similar to gear system 844), and device connector 906. Actuator housing 902 includes a housing chamber 908 supporting motor 900 and includes a channel 910 sized for passage of elongated support member 824. Device 912 may be detachably coupled to connector 906. Device 912 may include shaft 914 and a proximal coupling 916 at the proximal end of shaft 914, the proximal coupling 916 engaging connector 906 in a transverse direction (e.g., generally perpendicular to longitudinal axis A1). Device connector 906 may allow for quick release and replacement of device 912. The gear system of actuation mechanism 810 may connect the output of motor 900 to elongated threaded member 832. When actuated by motor 900, movement of gear system may cause actuation mechanism 810 to travel proximal or distal along elongated threaded member 832 in direction A1. When the actuating mechanism 810 moves in direction A1, the attached instrument device 912 also moves accordingly. In some instances, the instrument device 912 may be a needle exchange tool 219, and the motor 900 may drive the insertion and retraction movements of the needle exchange tool 219. The actuating mechanisms 808 and 810 may be driven independently, allowing the needle exchange tool 219 and the needle tip capturing mechanism 221 to move independently. In some instances, the shaft 894 of the instrument device 892 may extend within and move relative to the shaft 914.

[0101] Actuation mechanism 812 may include a servo mechanism or motor 920 and a gear system (not shown, but similar to gear system 844). Actuator housing 842 of actuation mechanism 804 may include a support platform 843 supporting motor 920. The gear system of actuation mechanism 812 can connect the output of motor 920 to instrument device 852. When actuated by motor 920, movement of the gear system can cause rotational movement of instrument device 852. In some instances, instrument device 852 may be tissue alignment tool 218, and motor 920 can drive bidirectional rotational movement of tissue alignment tool 218. Actuation mechanisms 804 and 812 can be driven independently, allowing instrument device 852 to rotate independently of its linear motion.

[0102] Actuation mechanism 814 may include a servo mechanism or motor 940, actuator housing 942, and a gear system (not shown, but similar to gear system 844). Actuator housing 942 includes a housing chamber 948 supporting motor 940 and a channel 950 sized for passage of elongated support member 826. Actuator housing 942 may be coupled to instrument device 952, which may include drive mechanism 954 and a suture needle (not shown) at the distal end of drive mechanism 954. The gear system of actuation mechanism 814 may connect the output of motor 940 to elongated threaded member 830. When actuated by motor 940, movement of gear system may cause actuation mechanism 814 to travel proximally or distally along elongated threaded member 832 in direction A1. When actuation mechanism 814 moves in direction A1, the attached instrument device 952 also moves in this direction. In some instances, instrument device 952 may include suture needle 214, and motor 940 may drive bidirectional movement of suture needle 214.

[0103] The instrument actuation device 802 may also include a sensing system 960, which includes force sensors 962, 964, 966 (e.g., load sensors) housed within a proximal platform 818. Force sensor 962 may contact the proximal end of an elongated threaded member 828. Force sensor 964 may contact the proximal end of an elongated threaded member 830. Force sensor 966 may contact the proximal end of an elongated threaded member 832. The distal end of the elongated threaded member may be biased toward the force sensor by a biasing member such as a spring. For example, spring 968 may bias the elongated threaded member 828 toward force sensor 962. Spring 970 may bias the elongated threaded member 830 toward force sensor 964. Spring 972 may bias the elongated threaded member 832 toward force sensor 966. Springs 968, 970, 972 may be housed within a distal platform 820. The force sensors detect tensile and / or compressive forces transmitted from the coupled instrument device through the actuator housing. The detected force can be used to generate tactile feedback to knob 102 to provide the user with information about the force encountered by the instrument. For example, when tissue alignment tool 218 encounters resistance (such as tissue) as it advances via movement of motor 840 and / or rotates via movement of motor 920, the resistance on alignment tool 218 can be transmitted to elongated threaded member 828, which transmits the force to force sensor 962. Force sensor 962 can generate tactile feedback to knob 102, for example, by increasing the torque of knob 102 to make it more difficult to turn, thereby simulating the manual feel of advancing the tool into tissue. Spring characteristics (e.g., size, stiffness, material) can differ from each other and can be selected to provide the desired preload force on the elongated threaded member.

[0104] The remote platform 820 may also include a device system interface port 980, such as Figure 10 As shown, the proximal end of instrument system 982 (e.g., instrument system 200 or endoscope) can be coupled to this port. Instrument system 982 can engage with port 980 from a lateral (e.g., generally perpendicular to the longitudinal axis A1) direction. Figure 10 As shown, the instrument actuation device 802 can be connected to the proximal end of the instrument system 982 at port 980. When the instrument system is connected to the instrument actuation device, instrument devices 852, 872, 912, and 952 can extend into the instrument system 982.

[0105] Figure 11 An endoscopic instrument system 1000 (e.g., instrument system 200) extending within an anatomical passage 1002 of an anatomical structure 1004 is shown. In some instances, the anatomical structure 1004 may be the stomach. The anatomical structure 1004 has an anatomical reference frame (X). A Y A Z AThe distal end 1006 of the endoscopic instrument system 1000 can be advanced into an anatomical opening (e.g., a patient's oral cavity) and through an anatomical passage 1002 to perform medical procedures such as suturing at or near a target tissue located in a region 1008 of an anatomical structure 1004 using any of the methods or systems described herein.

[0106] The aspects disclosed herein may be part of a computer-aided teleoperation manipulator system, sometimes referred to as a robot-aided manipulator system or a robot system. A manipulator system may include one or more manipulators that, when coupled to an electronic controller (e.g., a computer), can be operated with the assistance of the electronic controller to move and control the functions of one or more instruments.

[0107] Figure 12 An embodiment of a robot-assisted manipulator system for use with the tools described herein is illustrated. The manipulator system can be used for, for example, surgical, diagnostic, therapeutic, biopsy, or non-medical procedures, and is generally denoted by reference numeral 1100. Figure 12As shown, the robot-assisted manipulator system 1100 may include one or more manipulator components 1102 for operating one or more medical device systems 1104 (e.g., device system 200) while performing various procedures on a patient P placed on a table T in a medical environment 1101. For example, the manipulator component 1102 may drive the movement of a catheter or end effector, may dispose of target tissue, and / or may manipulate control members. The manipulator component 1102 may be a teleoperated, non-teleoperated, or hybrid teleoperated and non-teleoperated component, having selected degrees of freedom of motion that are electrically and / or teleoperated and selected degrees of freedom of motion that are non-electrically and / or non-teleoperated. The operator input system 1106 may be internal or external to the medical environment 1101 and typically includes one or more control devices (e.g., manual user interface system 100) for controlling the manipulator component 1102. Manipulator assembly 1102 supports medical device system 1104 and may optionally include multiple actuators or motors that drive inputs on medical device system 1104 in response to commands from control system 1112. The actuators may optionally include a drive system that, when coupled to medical device system 1104, can advance medical device system 1104 into a natural or surgically created anatomical opening. Other drive systems can move the distal end of the medical device in multiple degrees of freedom, which may include three linear degrees of motion (e.g., linear motion along the X, Y, Z Cartesian coordinate axes) and three rotational degrees of motion (e.g., rotation about the X, Y, and Z Cartesian coordinate axes). Manipulator assembly 1102 may support a variety of other systems for flushing, treatment, or other purposes. Such systems may include fluid systems (e.g., including reservoirs, heating / cooling elements, pumps, and valves), generators, lasers, interrogators, and ablation components.

[0108] The robot-assisted manipulator system 1100 also includes a display system 1110 for displaying images or representations of the surgical site and a medical device system 1104 generated by an imaging system 1109, which may include an imaging system such as an endoscopic imaging system. The display system 1110 and the operator input system 1106 can be oriented so that an operator can control the medical device system 1104 and the operator input system 1106 through telepresent perception. A graphical user interface can be displayed on the display system 1110 and / or on the display system of a standalone planning workstation.

[0109] In some instances, the endoscopic imaging system components of imaging system 1109 may be integrally or detachably coupled to medical device system 1104. However, in some instances, a separate imaging device, such as an endoscope, attached to a separate manipulator assembly may be used with medical device system 1104 to image the surgical site. Endoscopic imaging system 1109 may be implemented as hardware, firmware, software, or a combination thereof, interacting with or otherwise executed by one or more computer processors, which may include the processor of control system 1112.

[0110] The robot-assisted manipulator system 1100 may also include a sensor system 1108. The sensor system 1108 may include a position / location sensor system (e.g., an actuator encoder or electromagnetic (EM) sensor system) and / or a shape sensor system (e.g., a fiber optic shape sensor) for determining the position, orientation, rate, velocity, attitude, and / or shape of the medical device system 1104. The sensor system 1108 may also include temperature, pressure, force, or contact sensors, etc.

[0111] The robot-assisted manipulator system 1100 may also include a control system 1112. The control system 1112 includes at least one memory 1116 and at least one computer processor 1114 for controlling the medical device system 1104, the operator input system 1106, the sensor system 1108, and the display system 1110. In some instances, the control system 1112 may include a controller 106 of the manual user interface system 100. The control system 1112 also includes programming instructions (e.g., a non-transient machine-readable medium storing the instructions) to implement programs using the robot-assisted manipulator system, including navigation, steering, imaging, engagement feature deployment or retraction, disposal of target tissue (e.g., by applying energy), and so on.

[0112] The control system 1112 may optionally include a virtual visualization system to provide navigation assistance to the operator O when controlling the medical device system 1104 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system may be based on a reference to a preoperative or intraoperative dataset of the acquired anatomical access. The virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc. The control system 1112 may use the preoperative images to locate the target tissue (using visual imaging techniques and / or by receiving user input) and create a preoperative plan, including the optimal first location for the procedure. The preoperative plan may include, for example, the plan size of the scalable device, the duration of the procedure, the procedure temperature, and / or multiple deployment locations.

[0113] Figure 13A A medical device system 1200 (e.g., device system 200) according to some embodiments is illustrated. In some embodiments, the medical device system 1200 can be used for image-guided medical procedures. In some instances, the medical device system 1200 can be used for non-telescopic exploratory procedures or procedures involving conventionally manually operated medical instruments such as endoscopes. In some embodiments, the medical device system 1200 can be used with... Figure 12 The medical device system 1104 is interchangeable or a variant thereof.

[0114] The medical device system 1200 includes an elongated flexible device 1202, such as a flexible catheter or endoscope (e.g., a gastroscope, bronchoscope) coupled to a drive unit 1204. The elongated flexible device 1202 includes a flexible body 1216 having a proximal end 1217 and a distal or tip portion 1218. In some embodiments, the flexible body 1216 has an outer diameter of approximately 14-20 mm. Other flexible bodies may have larger or smaller outer diameters. When the flexible body 1216 is inserted into a patient's mouth or nasal cavity, the flexible body 1216 may have an appropriate length to reach portions of anatomical structures such as the lungs, sinuses, throat, or upper or lower digestive tract regions.

[0115] The medical device system 1200 optionally includes a tracking system 1230 for determining the position, orientation, rate, velocity, attitude, and / or shape of the distal end 1218 and / or one or more segments 1224 along the flexible body 1216 using one or more sensors and / or imaging devices. The flexible body 1216 can be effectively divided into segments 1224 along its entire length between the distal end 1218 and the proximal end 1217. The tracking system 1230 can optionally be implemented as hardware, firmware, software, or a combination thereof, which interact with or are executed by one or more computer processors, which may include... Figure 12 The processor of the control system 1112.

[0116] Tracking system 1230 may optionally use shape sensor 1222 to track distal end 1218 and / or one or more segments 1224. In some embodiments, tracking system 1230 may optionally and / or additionally use position sensor system 1220, such as an electromagnetic (EM) sensor system, to track distal end 1218. In some instances, position sensor system 1220 may be configured and positioned to measure six degrees of freedom, such as three position coordinates X, Y, Z and three azimuth angles of pitch, yaw, and roll of an indicator base point, or five degrees of freedom, such as three position coordinates X, Y, Z and two azimuth angles of pitch and yaw of an indicator base point.

[0117] The flexible body 1216 includes one or more channels 1221, the dimensions and shapes of which are adapted to receive one or more medical devices 1226. In some embodiments, the flexible body 1216 includes two channels 1221 for separate devices 1226; however, a different number of channels 1221 may be provided. Figure 13BThis is a simplified diagram of a flexible body 1216 with an extended medical device 1226 according to some embodiments. In some embodiments, the medical device 1226 can be used for procedures and aspects of procedures, such as surgery, biopsy, ablation, mapping, imaging, illumination, irrigation, or aspiration. The medical device 1226 can be deployed through a channel 1221 of the flexible body 1216 and used at a target location within an anatomical structure. The medical device 1226 may include, for example, an image capture device, biopsy instrument, ablation instrument, catheter, laser ablation fiber, and / or other surgical, diagnostic, or therapeutic tool. Medical tools may include end effectors having a single working component, such as a scalpel, blunt blade, lens, fiber optic, electrode, etc. Other end effectors may include, for example, forceps, graspers, balloons, needles, scissors, applicators, etc. Other end effectors may also include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, imaging devices, etc. The medical device 1226 can advance from an opening in the channel 1221 to perform a procedure and then retract into the channel when the procedure is complete. The medical device 1226 can be removed from the proximal end 1217 of the flexible body 1216 or from another optional device port (not shown) along the flexible body 1216. The medical device 1226 can be used with an image capturing device (e.g., an endoscopic camera) also within the elongated flexible device 1202. Alternatively, the medical device 1226 itself can be an image capturing device.

[0118] Medical device 1226 may additionally accommodate cables, links, or other actuation control devices (not shown) extending between its proximal and distal ends to controllably bend the distal end of medical device 1226. Flexible body 1216 may also accommodate cables, links, or other steering control devices (not shown) extending between drive unit 1204 and distal end 1218 to controllably bend distal end 1218, as illustrated, for example, by dashed line depiction 1219 of distal end 1218. In some instances, at least four cables are used to provide independent "up / down" steering to control pitch movement of distal end 1218 and "left / right" steering to control yaw movement of distal end 1218. In embodiments where medical device system 1200 is actuated by a robot-assisted component, drive unit 1204 may include a drive input detachably coupled to and receiving power from a drive element of the teleoperated component, such as an actuator. In some implementations, the medical device system 1200 may include a gripping feature, a manual actuator, or other components for manually controlling the movement of the medical device system 1200. Information from the tracking system 1230 may be sent to the navigation system 1232, where it is combined with information from the visualization system 1231 and / or a preoperatively acquired model to provide real-time location information to the physician or other operator.

[0119] In some instances, the manual user interface system for controlling instrument systems can be mounted on and supported by a fixed or movable control panel in a medical environment. In other instances, the manual user interface system can be carried by the operator or coupled to an instrument system (e.g., a manual endoscope) carried or installed by the operator. Figure 14 An example of a device actuation system 1300 for use with one or more device devices in an actuated instrument system is shown. The device actuation system 1300 may include a manual user interface system 1301 that transmits user input to a device actuation device 1302 to operate multiple actions of the instrument system (e.g., instrument system 200). The manual user interface system 1301 may be a component of an operator input system (e.g., operator input system 1106), and the device actuation device 1302 may be a component of a manipulator assembly (e.g., manipulator assembly 1102). In some instances, the device actuation device 1302 may be coupled to and supported by a robotic actuated arm mounted on a floor, wall, fixed console, or movable console within a medical environment. In various instances, the arm may be robotically actuated, manually actuated, or have a combination of robotic and manual actuation. In other instances, the device actuation device may be carried by an operator or coupled to an operator-carried or otherwise mounted instrument system (e.g., a manual endoscope).

[0120] The manual user interface system 1301 may include a remote manual user interface device 1304, which is movable in one or more degrees of freedom in response to input from a user. The remote manual user interface device 1304 may be detached from or spatially remote from the instrument actuator 1302 and may actuate servo-controlled degrees of freedom of the instrument actuator 1302. For example, the remote manual user interface device 1304 may be mounted to a fixed or movable control panel (e.g., control panel 301) within a medical environment, or may be included in a handheld controller that can be attached to an operator's hand or arm. The remote manual user interface device 1304 may be movable in translational and / or rotational degrees of freedom. For example, the remote manual user interface device 1304 may include an elbow, button, joystick, scroll wheel, trackball, or knob 102. The manual user interface system 1301 may include an actuator system 104 and a controller 106, as previously described, for transmitting operator commands from the remote manual user interface device 1304 to the instrument actuator 1302 to operate the instrument system.

[0121] The manual user interface system 1301 may include a local manual user interface device 1306 that can move in one or more degrees of freedom in response to input from a user. The local manual user interface device 1306 may be mounted to or supported by the instrument actuator 1302 and can control the movement of one or more instruments in an instrument system (e.g., instrument system 200). For example, the local manual user interface device 1306 may include a handle, grip, knob, or other structure coupled to an instrument in the instrument system, which can be directly manipulated by an operator to translate, rotate, or otherwise locally control one or more instruments coupled to the instrument actuator 1302. The local manual user interface device 1306 may also actuate servo-controlled degrees of freedom of movement of the instrument actuator 1302 via the actuator system 104 and the controller 106.

[0122] Figure 15A An example of an instrument actuator 1400 is shown that can be used to perform multiple actions on an instrument system (e.g., instrument system 200). Instrument actuator 1400 may be an example of instrument actuator 1302 and may be responsive to control provided by a remote manual user interface device 1452 or 1480, as described below. Figure 15B The mechanical actuator 1400 is shown in relation to... Figure 15A The opposite side is shown in the diagram. The instrument actuator 1400 may be similar to the instrument actuator 802, but may be adapted to provide additional linear degrees of freedom of motion, and may be adapted to different instrument systems or instrument devices.

[0123] The instrument actuation device 1400 may include actuation mechanisms 1404, 1406, 1408, 1410, 1412, and 1414, each of which is assigned to provide motion for the degrees of freedom of the instrument device of an instrument system (e.g., instrument system 200). Actuation mechanisms 1404, 1406, 1408, 1410, 1412, and 1414 may receive control signals from controller 106 to actuate the functions of the instrument device. The control signals for the actuation mechanisms may be received wirelessly or via a wired connection. Actuation mechanisms 1404, 1406, 1408, 1410, 1412, and 1414 may be powered, for example, via cables operatively connected to a power source or battery.

[0124] The device actuation device 1400 may include a frame 1416, which includes a proximal platform 1418, a distal platform 1420, and four elongated support members 1422, 1423, 1424, and 1426 extending between platforms 1418 and 1420. The elongated support members 1422, 1423, 1424, and 1426 may be inclined relative to the longitudinal axis A2, and platforms 1418 and 1420 may extend substantially perpendicular to the longitudinal axis A2. The inclination of the elongated support members along axis A2 may create space for elongated instruments (e.g., laser instruments, balloon instruments, ablation instruments, grasping instruments, tissue alignment tools, etc.) or instrument cables to pass through. The device actuation device 1400 may also include elongated threaded members 1428 and 1430 (e.g., lead screws) that extend at an angle to the longitudinal axis A2 between platforms 1418 and 1420.

[0125] Each actuation mechanism 1404, 1406, 1408, 1410, 1412, and 1414 can drive independent motion of the instrument devices of an instrument system (e.g., instrument system 200) in degrees of freedom, including reciprocating or bidirectional motions. For example, actuation mechanism 1404 can drive linear degree-of-freedom motion, such as the insertion and retraction motion of the axis of a tissue alignment tool (e.g., tissue alignment tool 218). Actuation mechanism 1406 can drive linear degree-of-freedom motion, such as the opening and closing of the jaws or gripping mechanism or snare of a tissue alignment tool. Actuation mechanism 1408 can drive linear degree-of-freedom motion, such as a needle tip capturing mechanism (e.g., needle tip capturing mechanism 221). Actuation mechanism 1410 can drive linear degree-of-freedom motion, such as the insertion and retraction motion of a needle exchange tool (e.g., needle exchange tool 219). Actuation mechanism 1412 can drive bidirectional rotational motion, such as the rotation of a helical tissue alignment tool (e.g., tissue alignment tool 218). The actuation mechanism 1414 can drive a linear degree of freedom motion with bidirectional movement, such as a suture needle (e.g., suture needle 214).

[0126] Actuation mechanisms 1404, 1406, 1408, 1410, 1412, and 1414 may be structurally and functionally similar to one or more of the aforementioned actuation mechanisms 804-814, and each actuation mechanism may include a servo mechanism or motor, an actuator housing, a transmission or drive system, and an instrument device connector. Instrument devices (e.g., tissue alignment devices, needle deployment tools, suture needles) may be detachably coupled to one or more of the actuation mechanisms 1404, 1406, 1408, 1410, 1412, and 1414 and may be moved by the actuation mechanisms. For example, a tissue alignment tool 1440 may be attached via proximal couplings 1442, 1444, and 1446 to a corresponding connector of a set of actuation mechanisms 1404, 1406, and 1412, which may control the movement of the tissue alignment tool 1440. In this example, the tissue alignment tool 1440 may include a telescopic instrument device or component. In this example, actuation mechanism 1406 can cause pivoting motion of arm 1448, which drives linear motion of components of tissue alignment tool 1440 to cause the gripper jaws to open and close. Actuation mechanism 1412 can actuate one or more gears that rotate tissue alignment tool 1440. Actuation mechanism 1404 can actuate one or more gears that cause linear motion of tissue alignment tool 1440 along elongated threaded member 1428 (e.g., lead screw). In this example, the housings of each servo mechanism of actuation mechanisms 1404, 1406, and 1412 can be coupled or integrally formed such that linear motion of actuation mechanism 1404 along elongated threaded member 1428 also results in corresponding linear motion of actuation mechanisms 1406 and 1412. In this example, linear degree-of-freedom motion can be actuated by two different mechanisms: the movement of actuation mechanism 1404 along the lead screw and the movement of pivot arm 1448 caused by actuation mechanism 1406, to linearly move the proximal coupling 1442 of the organization mating tool 1440.

[0127] The needle deployment tool 1411 can be attached via proximal couplings 1443, 1445, 1447 to a corresponding connector of an assembly of actuating mechanisms 1408, 1410, 1414, which can control the movement of components of the needle deployment tool 1411. The needle deployment tool 1411 can be used to actuate one or more of a needle tip capturing mechanism, a needle exchange tool, and a needle. In this example, the needle deployment tool 1411 may include a telescopic instrument device or component. The needle deployment tool 1411 may include an instrument component that is slidable within an external instrument component and / or otherwise independently movable relative to the external instrument component. In this example, the actuating mechanism 1408 can cause a pivoting movement of an arm 1449, which drives a linear motion to actuate a portion of the needle deployment tool 1411, such as a needle tip capturing mechanism. Actuation mechanism 1410 can actuate one or more gears, which cause linear movement along an elongated threaded member 1430 (e.g., a lead screw) to actuate a portion of needle deployment tool 1411 (e.g., insertion and retraction movement of a needle exchange tool). Actuation mechanism 1414 can actuate one or more gears, which cause linear movement along the elongated threaded member 1430, thereby actuating a portion of needle deployment tool 1411 (e.g., bidirectional movement of a suture needle). In this example, the housings of each servo mechanism of actuation mechanisms 1408, 1410 can be coupled or integral, such that linear movement of actuation mechanism 1410 along the elongated threaded member 1430 also results in a corresponding linear movement of actuation mechanism 1408. In this example, linear degree-of-freedom movement can be actuated by two different mechanisms: movement of actuation mechanisms 1410, 1414 along the lead screw and movement of pivot arm 1449 caused by actuation mechanism 1408, to linearly move proximal coupling 1443 of needle deployment tool 1411. In this example, actuation mechanisms 1410 and 1414 can move independently along the slender threaded member 1430. In various examples, one or more actuation mechanisms may be omitted, and the degrees of freedom of motion associated with the omitted actuation mechanisms can be manually executed by the direct operator.

[0128] like Figure 16A As shown, the remote platform 1420 can be coupled to the proximal end of the instrument system 1450 (e.g., instrument system 200 or endoscope), and the instrument device actuated by the instrument actuator 1400 can extend through or along the instrument system 1450. In some instances, the endoscope 1450 and the attached instrument actuator 1400 can be held or supported by one hand of the operator O1, and the remote manual user interface device 1452 (e.g., remote manual user interface device 1304) can be held or supported by the other hand of the operator O1. Figure 16BAs shown, the remote manual user interface device 1452 may include a body 1454 shaped to fit within the palm of a hand and an input device 1456 extending from the surface of the body 1454 and arranged to engage with the thumb (or any finger) of the hand holding the body 1454. The input device 1456 may include, for example, a joystick. In some instances, the joystick may pivot back and forth to actuate an actuation mechanism 1404, thereby driving insertion and retraction movements of the shaft of the tissue alignment tool 1440. In some instances, the joystick may pivot left and right to actuate an actuation mechanism 1412, thereby driving rotational movement of the shaft of the tissue alignment tool 1440. In some instances, pressing the joystick may actuate a binary switch that actuates one or more actuation mechanisms 1408, 1410, 1414 in a predetermined sequence to, for example, use a needle deployment tool 1411 to deliver suture needles. In other instances, the binary switch may apply ablation energy. The remote manual user interface device 1452 may also include an input device 1458, such as a button, trigger, or other binary switch. In some instances, the input device 1458 may be pressed to, for example, actuate an actuation mechanism 1406 to activate the gripper or snare of the tissue engagement tool 1440. A strap 1460 or other fastening member may be used to secure the remote manual user interface device 1452 to the operator's hand.

[0129] When Figure 16A When connected to the operator's hand, the operator's hand can, for example, simultaneously grasp the body of the endoscope 1450, actuate the knob 1456 with the thumb, and actuate the trigger 1458 with the index finger. Thus, the operator O1 can manipulate the endoscope 1450 (e.g., insertion and rolling movements) while also controlling the instrument actuation device 1400 to operate the instrument.

[0130] Figure 17A remote manual user interface device 1480 is shown, which can be used as an alternative to or in conjunction with a remote manual user interface device 1452. The remote manual user interface device 1480 can be mounted to a fixed or movable control panel (e.g., control panel 301) in a medical environment. The remote manual user interface device 1480 may include a body 1484 and an input device 1486 extending from the surface of the body and arranged to be engaged by an operator's hand. The input device 1486 is movable in multiple degrees of freedom. For example, the input device 1486 can pivot forward and / or backward to actuate an actuation mechanism 1404, thereby driving insertion and retraction movements of the axis of a tissue alignment tool. In some instances, the input device 1486 can rotate clockwise and / or counterclockwise to actuate an actuation mechanism 1412 to drive rotational movement of the axis of the tissue alignment tool. In some instances, input device 1486 can be pressed to actuate a binary switch, which in a predetermined sequence actuates one or more of actuation mechanisms 1408, 1410, and 1414 to, for example, eject suture stitches or apply ablation energy. In some instances, remote manual user interface device 1480 may also include input devices, such as buttons, triggers, or other binary toggles, for actuating actuation mechanism 1406 to activate a gripper or snare.

[0131] Figure 18A and 18B An example of an instrument actuator 1500 is shown that can be used to perform multiple actions on an instrument system (e.g., instrument system 200). The instrument actuator 1500 may be an example of an instrument actuator 1302 and may be responsive to control provided by a local manual user interface device 1502 directly coupled to the instrument actuator 1500. Figure 18B The mechanical actuator 1500 is shown in relation to... Figure 18A The opposite side is shown. The instrument actuation device 1500 may be similar to the instrument actuation device 802, but may be adapted to allow the user to directly control one or more degrees of freedom of motion and receive tactile information associated with that direct control. The instrument actuation device 1500 may also be adapted to accommodate different instrument systems or instrument devices.

[0132] The instrument actuation device 1500 may include actuation mechanisms 1506, 1508, 1510, 1512, and 1514, each actuation mechanism being assigned to provide movement for the degrees of freedom of the instrument device of the instrument system (e.g., instrument system 200). Actuation mechanisms 1506, 1508, 1510, 1512, and 1514 may receive control signals from controller 106 for actuating the functions of the instrument device. Control signals to the actuation mechanisms may be received wirelessly or via a wired connection. Actuation mechanisms 1506, 1508, 1510, 1512, and 1514 may be powered, for example, via cables operatively connected to a power source or battery. In this example, at least one degree of freedom of the instrument device of the instrument system can be directly manipulated by an operator through interaction with a local manual user interface device 1502.

[0133] The instrument actuation device 1500 may include a frame 1516, which includes a proximal platform 1518, a distal platform 1520, and three elongated support members 1522, 1524, and 1526 extending between platforms 1518 and 1520. The elongated support members 1522, 1524, and 1526 may be generally parallel to the longitudinal axis A3, and platforms 1518 and 1520 may extend generally perpendicular to the longitudinal axis A3. The instrument actuation device 1500 may also include one or more elongated threaded members 1528 (e.g., lead screws) extending generally parallel to the longitudinal axis A3 between platforms 1518 and 1520.

[0134] Each actuation mechanism 1506, 1508, 1510, 1512, and 1514 can drive independent movement of the instrument devices of an instrument system (e.g., instrument system 200) in degrees of freedom, including reciprocating or bidirectional movements of the degrees of freedom. For example, actuation mechanism 1506 can drive linear degrees of freedom, such as the opening and closing of the jaws or gripping mechanism of a tissue alignment tool or a snare. Actuation mechanism 1508 can drive a needle tip capturing mechanism (e.g., needle tip capturing mechanism 221). Actuation mechanism 1510 can drive the insertion and retraction movements of a needle exchange tool (e.g., needle exchange tool 219). Actuation mechanism 1512 can drive bidirectional rotational movement of a tissue alignment tool (e.g., tissue alignment tool 218). Actuation mechanism 1514 can drive linear degrees of freedom movement such as bidirectional movement of a suture needle (e.g., suture needle 214).

[0135] Actuation mechanisms 1506, 1508, 1510, 1512, and 1514 may be structurally and functionally similar to one or more of the aforementioned actuation mechanisms 804-814, and each actuation mechanism may include a servo mechanism or motor, an actuator housing, a transmission or drive system, and an instrument connector. Instruments (e.g., tissue alignment devices, needle deployment tools, suture needles) may be detachably coupled to and moved by one or more of the actuation mechanisms 1506, 1508, 1510, 1512, and 1514.

[0136] In various instances, the movement of an instrument such as tissue alignment tool 1505 can be achieved through a hybrid system of servo and direct manual control degrees of freedom. For example, tissue alignment tool 1505 can be attached via a proximal coupling to a set of actuators 1506, 1512 that can control certain degrees of freedom of movement of tissue alignment tool 1505. In this example, actuator 1506 can drive the movement of components of tissue alignment tool 1505 to open and close the gripper jaws. Actuator 1512 can actuate one or more gears or drive systems that rotate tissue alignment tool 1505. In some instances, actuator 1512 can rotate tissue alignment tool 1505 a predetermined number of revolutions in response to an operator's rotational initiation action on handle portion 1504 (e.g., a direct manual five-degree twist). Tactile cues regarding the state of tissue alignment tool 1505 (e.g., a helical tissue alignment tool) as it rotates into tissue may be valuable to the operator. For example, an operator might find it valuable to know whether the tissue alignment tool encounters increased resistance (e.g., indicating tissue twisting) or decreased torque (e.g., indicating the tissue alignment tool has withdrawn from the back of the tissue). In some instances, the actuation mechanism 1512 may include a current sensor for sensing torque. The current driving the servo (e.g., corresponding to torque) can be monitored, and the current in the handle portion 1504 (e.g., corresponding to torque) can be adjusted accordingly to indicate the sensed torque. Thus, the user receives a tactile or sensory feedback of the helical torque at the handle portion 1504. The operator can use this feedback to determine whether to continue initiating rotation, terminate rotation, or initiate reverse rotation.

[0137] In this example, the actuation mechanism for the insertion and retraction movements of the shaft driving the tissue alignment tool 1505 can be omitted, and this degree of freedom of movement can be manually actuated by the operator. More specifically, the operator can grasp the handle portion 1504 of the local manual user interface device 1502 to advance or retract the tissue alignment tool 1505 (including the instrument holder 1507 or other movable support structure for the tool 1505). The handle portion 1504 can be coupled to the proximal end of the tissue alignment tool 1505. To enhance the user's tactile feedback, the holder 1507 may include a constant-force spring to counteract gravity.

[0138] The handle portion 1504 may also include a trigger mechanism 1509 (e.g., a button, switch, or other binary toggle) that can sequentially actuate one or more actuation mechanisms 1508, 1510, 1514 to, for example, move one or more instrument devices, including a needle deployment tool 1511, which are involved in creating suture stitches. The needle deployment tool 1511 can be used to actuate one or more of a needle tip capturing mechanism, a needle exchange tool, and a needle. Telescopic components of the needle deployment tool 1511 (e.g., a chuck, a needle, and a needle exchange tool) can be assembled and installed by first loading the distal end into an endoscope. The needle deployment tool 1511 may include instrument components that can slide within and / or otherwise move independently relative to external instrument components. Then, as... Figure 18B As shown, the proximal connection 1513 of the needle deployment tool 1511 can be connected to the connector of the instrument actuator 1500, for example, using a magnet.

[0139] In this example, actuation mechanism 1508 can cause pivoting motion of arm 1549, which drives linear motion to actuate a portion of needle deployment tool 1511, such as a needle tip capturing mechanism. In some examples, actuation mechanism 1510 can drive insertion and retraction motions of needle deployment tool 1511 (e.g., needle exchange tool 219).

[0140] In this example, operator interaction with the handle portion 1504 of the local manual user interface device 1502 can control multiple instrument devices connected to the instrument actuator 1500. In this example, the distal platform 1520 can be connected to the proximal end of an instrument system (e.g., instrument system 200 or endoscope 1450), and the instrument devices actuated by the instrument actuator 1500 can extend through the instrument system. In some examples, the proximal end of the endoscope and the attached instrument actuator 1500 can be held or supported by one of the operator's hands, and the distal end of the endoscope can be held or supported by the operator's other hand.

[0141] The specification has set forth specific details describing some examples. Numerous specific details have been set forth to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that some examples can be practiced without some or all of these specific details. The specific examples disclosed herein are intended to be illustrative and not limiting. Other elements may be implemented by those skilled in the art, although not specifically described herein, but within the scope and spirit of this disclosure.

[0142] Where feasible, elements described in detail with reference to one instance, implementation, or application may optionally be included in other instances, implementations, or applications not specifically shown or described therein. For example, if an element is described in detail with reference to one instance but not with reference to a second instance, that element may still be considered included in the second instance. Therefore, to avoid unnecessary repetition in the description, unless otherwise specifically described, one or more elements shown and described in association with one instance, implementation, or application may be incorporated into other instances, implementations, or aspects, unless one or more elements render the instance or implementation ineffective, or unless two or more elements produce conflicting functionality. Not all illustrated processes can be performed in all instances of the disclosed methods. Furthermore, one or more processes not explicitly shown may be included before, after, between, or as part of the illustrated process. In some instances, one or more processes may be performed by a control system, or may be implemented at least in part as executable code stored on a non-transient, tangible, machine-readable medium, which, when run by one or more processors, causes one or more processors to perform one or more processes.

[0143] Any changes and further modifications to the described equipment, apparatus, and methods, as well as any further application of the principles of this disclosure, are those that would normally occur to those skilled in the art. Furthermore, the dimensions provided herein are for specific examples, and it is conceivable that the concepts of this disclosure may be implemented using different dimensions, dimensions, and / or ratios. To avoid unnecessary descriptive repetition, one or more parts or actions described according to an illustrative example may be used or omitted from other illustrative examples where applicable. For the sake of brevity, multiple iterations of these combinations will not be described separately. For simplicity, in some cases, the same reference numerals are used throughout the drawings to refer to the same or similar parts.

[0144] The systems and methods described herein may be applicable to imaging and manipulation in any of a variety of anatomical systems via naturally or surgically created access pathways, including the lungs, colon, intestines, stomach, liver, kidneys and renal calyces, brain, heart, circulatory system including the vascular system, and so on. While examples of medical procedures are provided herein, any references to medical or surgical instruments and methods are non-limiting. For example, the instruments, systems, and methods described herein may be used for non-medical purposes, including industrial use, general robotic use, and sensing or manipulating non-tissue artifacts. Other example applications include cosmetic improvements, imaging of human or animal anatomy, data collection from human or animal anatomy, and training of medical or non-medical personnel. Additional example applications include procedures on tissues removed from human or animal anatomy (without returning to human or animal anatomy) and procedures performed on human or animal cadavers. Furthermore, these techniques may also be used for surgical and non-surgical medical procedures or diagnostic procedures.

[0145] The methods described herein can be represented as a set of operations or procedures that may be performed in the same or a different order than those shown in the provided flowcharts. In some instances of the method, one or more procedures shown may be omitted. Furthermore, one or more procedures not explicitly shown in the flowcharts may be included before, after, between, or as part of the procedures shown. In some instances, one or more procedures of the method may be implemented at least in part by a control system that executes code stored on a non-transient, tangible, machine-readable medium, which, when run by one or more processors (e.g., the processor of the control system), causes one or more processors to execute one or more of the procedures.

[0146] One or more elements of the embodiments of this disclosure can be implemented in software for execution on a processor of a computer system, such as a control processing system. When implemented in software, the elements of the embodiments of this disclosure can be code segments that perform various tasks. Programs or code segments can be stored in a processor-readable storage medium or device, which can be downloaded via computer data signals carried on a carrier wave over a transmission medium or communication link. Processor-readable storage devices can include any medium capable of storing information, including optical media, semiconductor media, and / or magnetic media. Examples of processor-readable storage devices include electronic circuits; semiconductor devices, semiconductor memory devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. Code segments can be downloaded via computer networks such as the Internet, intranets, etc. Any of a variety of centralized or distributed data processing architectures can be employed. Programming instructions can be implemented as multiple individual programs or subroutines, or they can be integrated into multiple other aspects of the system described herein. In some instances, the control system may support wireless communication protocols such as Bluetooth, Infrared Data Association (IrDA), HomeRF, IEEE 802.11, Digital Enhanced Cordless Communication (DECT), Ultra Wideband (UWB), ZigBee, and wireless telemetry.

[0147] Note that the processes and displays presented may not be related to any particular computer or other device itself. Various general-purpose systems can be used with the programs taught herein, or it may prove convenient to construct more specialized devices to perform the described operations. The structures required for various such systems will appear as elements in the claims. Furthermore, examples of the invention are described without reference to any particular programming language. It should be understood that the teachings of the invention described herein can be implemented using various programming languages.

[0148] This disclosure describes the states of various instruments, instrument parts, and anatomical structures in three-dimensional space. As used herein, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or part of an object (e.g., in one or more rotational degrees of freedom, such as roll, pitch, and / or yaw). As used herein, the term "attitude" refers to the position of an object or part of an object in at least one translational degree of freedom, and the orientation of the object or part of an object in at least one rotational degree of freedom (e.g., up to six total degrees of freedom). As used herein, the term "shape" refers to a set of attitudes, positions, or orientations measured along an object.

[0149] While certain illustrative examples of the invention are described and shown in the accompanying drawings, it should be understood that these examples are merely illustrative of the invention and not intended to limit it, and that the examples of the invention are not limited to the specific constructions and arrangements shown and described, as various other modifications can be made by those skilled in the art.

Claims

1. A manual user interface system, comprising: A knob that is movable in at least one rotational degree of freedom and at least one translational degree of freedom; and An actuator system, coupled to and responsive to multiple movements of the knob, operates multiple instrumentation devices of a robot-assisted instrumentation system through multiple predetermined stages of a sequential workflow.

2. The manual user interface system of claim 1, wherein the knob is movable in opposite directions in the degree of rotational freedom.

3. The manual user interface system of claim 1, wherein the knob is movable in opposite directions in the translational degree of freedom.

4. The manual user interface system of claim 1, wherein the knob is movable in multiple translational degrees of freedom.

5. The manual user interface system of claim 1, wherein the knob is movable in multiple degrees of rotational freedom.

6. The manual user interface system of claim 1, wherein at least one rotational degree of freedom or at least one translational degree of freedom of the knob can be locked based on the instrument among the plurality of instruments currently in operation.

7. The manual user interface system of claim 1, wherein one or more rotational or translational degrees of freedom of the knob can be locked based on the current stage of the sequential workflow.

8. The manual user interface system of claim 1, wherein the knob includes a force sensor responsive to a squeezing action.

9. The manual user interface system of claim 1, wherein the knob comprises a touchscreen.

10. The manual user interface system of claim 1, wherein the knob is configured to provide tactile output.

11. The manual user interface system of claim 1, further comprising a panel on which the knob is mounted.

12. The manual user interface system of claim 11, further comprising a second knob mounted on the panel.

13. The manual user interface system of claim 11, further comprising a touchscreen on the panel.

14. The manual user interface system of claim 1, further comprising an actuator system coupled to the knob.

15. A manual user interface system, comprising: A knob that is movable in at least one rotational degree of freedom; and An actuator system is coupled to the knob and operates the device of the robot-assisted instrument system in a forward sequence of workflow stages in response to rotation of the knob in a first direction of the rotational degree of freedom, and operates the device in a reverse sequence of workflow stages in response to rotation of the knob in a second direction of the rotational degree of freedom.

16. The manual user interface system of claim 15, wherein the knob is movable in a plurality of rotational degrees of freedom.

17. The manual user interface system of claim 15, wherein the knob is movable in opposite directions in the at least one degree of rotational freedom.

18. The manual user interface system of claim 15, wherein the at least one rotational degree of freedom of the knob can be locked based on the device of the currently operating robot-assisted instrument system.

19. The manual user interface system of claim 15, wherein at least one degree of rotational freedom of the knob can be locked based on the current stage of the workflow.

20. The manual user interface system of claim 15, wherein the knob includes a force sensor responsive to a squeezing action.

21. The manual user interface system of claim 15, wherein the knob comprises a touchscreen.

22. The manual user interface system of claim 15, wherein the knob is configured to provide tactile output.

23. The manual user interface system of claim 15 further includes a panel on which the knob is mounted.

24. The manual user interface system of claim 23 further includes a second knob mounted on the panel.

25. The manual user interface system of claim 23 further includes a touchscreen on the panel.

26. The manual user interface system of claim 15, further comprising an actuator system coupled to the knob.

27. A method comprising: The system receives first user input at a manual user interface, which is movable in multiple degrees of freedom. In response to the first user input at the manual user interface, the first instrument device of the mobile robot-assisted instrument system is moved in a first predetermined stage of the sequential workflow. Receive a trigger associated with the end of the first predetermined stage of the sequential workflow; Receive second user input at the manual user interface; and In response to the second user input, the second instrument device of the robot-assisted instrument system is moved in the second predetermined stage of the sequential workflow.

28. The method of claim 27, wherein the first user input is in rotational degrees of freedom and the second user input is in translational degrees of freedom.

29. The method of claim 27, wherein the first user input moves the manual user interface in a first direction of the rotational degrees of freedom of the plurality of degrees of freedom to operate the first instrument device in a first instrument direction.

30. The method of claim 29, wherein the first user input moves the manual user interface in a second direction of the rotational degree of freedom to operate the first instrument device in a second instrument direction opposite to the first instrument direction.

31. The method of claim 27, wherein the first user input moves the manual user interface with a different degree of freedom than the second user input.

32. The method of claim 27, wherein the first user input moves the manual user interface with the same degrees of freedom as the second user input.

33. The method of claim 27, wherein the trigger includes a third user input at the manual user interface.

34. The method of claim 27, wherein the trigger includes a sensed threshold.

35. The method of claim 27, wherein the first predetermined stage of the sequential workflow is needle arm retraction, and the second predetermined stage of the sequential workflow is extension of the tissue alignment tool.

36. The method of claim 27, wherein the first predetermined stage of the sequential workflow is the extension of the tissue alignment tool, and the second predetermined stage of the sequential workflow is the rotation of the tissue alignment tool.

37. The method of claim 27, further comprising receiving user input at the tracking device of the manual user interface and, in response to the user input at the tracking device, manipulating the imaging device of the robot-assisted instrument system.

38. The method of claim 27, further comprising displaying a graphical user interface comprising a plurality of icons, each of the plurality of icons being associated with a corresponding predetermined stage of the sequential workflow.

39. The method of claim 38, wherein the graphical user interface includes a stage indicator that indicates one of the plurality of icons associated with a current predetermined stage of the sequential workflow.

40. The method of claim 27, wherein the robot-assisted instrument system includes an imaging system, and the method further includes displaying a graphical user interface, the graphical user interface including views from the imaging system and stage indicators for the current predetermined stage of the sequential workflow.

41. A method comprising: The first user input is received in the first user interface degree of freedom at the manual user interface so that the first device takes a first action. The second user input is received in the second user interface degree of freedom at the manual user interface to cause the second instrument to perform a second action; and The third user input is received in the first user interface degree of freedom at the manual user interface to cause the second instrument device to take a third action.

42. The method of claim 41, wherein the first user input is in rotational degrees of freedom and the second user input is in translational degrees of freedom.

43. The method of claim 41, wherein the first user input moves the manual user interface in a first direction of the first user interface degree of freedom to cause the first instrument device to perform the first action.

44. The method of claim 43, wherein the first user input moves the manual user interface in a second direction of the first user interface degree of freedom to reverse the first action.

45. The method of claim 41, wherein the first user interface degree of freedom and the second user interface degree of freedom are different.

46. ​​The method of claim 41, wherein the first action occurs during a first predetermined phase of the sequential workflow, and the second action occurs during a second predetermined phase of the sequential workflow.

47. The method of claim 46, further comprising receiving a trigger input at the manual user interface and, in response to receiving the trigger input, ending a first predetermined phase of the sequential workflow.

48. A method comprising: The first user input is received in the first user interface degree of freedom at the manual user interface so that the first device takes a first action. Receive second user input in the first degree of freedom of the manual user interface to cause the second medical device to perform a second action; and The third user input is received in the first user interface degree of freedom at the manual user interface to cause the second instrument device to take a third action.

49. The method of claim 48, wherein the first user input moves the manual user interface in a first direction of the first user interface degree of freedom to cause the first instrument device to perform the first action.

50. The method of claim 49, wherein the first user input moves the manual user interface in a second direction of the first user interface degree of freedom to reverse the first action.

51. The method of claim 48, wherein the first action occurs during a first predetermined phase of the sequential workflow, and the second action occurs during a second predetermined phase of the sequential workflow.

52. The method of claim 51, further comprising receiving a trigger input at the manual user interface and, in response to receiving the trigger input, ending a first predetermined phase of the sequential workflow.

53. The method of claim 52, wherein the trigger input is received in a second user interface degree of freedom of the manual user interface, and wherein the first user interface degree of freedom and the second user interface degree of freedom are different.

Images