View-based motion of imaging instruments

Abstract

A computer-assisted system includes a manipulator structure configured to support an imaging instrument having a field of view that defines a view frame, and a control system. The manipulator structure and the imaging instrument together have an aggregate set of joints. The control system, in an imaging instrument movement mode: receives a user input to move the field of view with a desired motion, determines, based on the desired motion, a motion command that moves the field of view in accordance with a first alignment relationship that establishes a first one-to-one mapping between first, second, and third translational and rotational degrees of freedom of the input device in an input device base frame and respective first, second, and third translational and rotational degrees of freedom of the field of view in the view frame, and commands the aggregate set of joints based on the motion command.

Description

VIEW-BASED MOTION OF IMAGING INSTRUMENTSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority under 35 U S.C. § 119(e) to U.S. Provisional Patent Application Serial No. 63 / 528,638 filed on July 24, 2023, which is hereby incorporated by reference herein in its entirety. This application further claims the benefit of priority under 35 U S.C. § 119(e) to U.S. Provisional Patent Application Serial No.63 / 652,837, filed on May 29, 2024, which is hereby incorporated by reference herein in its entirety.BACKGROUNDField of Invention

[0002] The present invention generally provides improved computer-assisted devices, systems, and methods.Overview

[0003] Computer-assisted systems can be used to perform a task at a worksite. For example, a computer-assisted system may comprise a handheld, motorized tool assembly. As another example, a computer-assisted system may comprise a robotic system, and may include one or more robotic manipulators to manipulate instruments for performing the task.

[0004] Example computer-assisted systems include industrial and recreational robotic systems. Example computer-assisted systems also include medical robotic systems used in procedures for diagnosis, non-surgical treatment, surgical treatment, etc.

[0005] Some computer-assisted systems include one or more instruments that are articulated in order to perform various procedures. The computer-assisted system can be automated, semi- automated, teleoperated, etc. In a teleoperated example, a human operator manipulates one or more leader input controls to command motion of one or more follower instruments located in a workspace. In some examples, the teleoperated system is configured to support an instrument that includes an imaging device, such as an endoscope or a camera, which enables the operator to observe the workspace. In some instances, the field of view of the imaging such an imaging instrument is directed to enable the operator to see the instrument(s) as the operator commands motion of the instrument(s).

[0006] In some embodiments, the imaging instrument is disposed on or otherwise supported by a manipulator structure. The field of view may be movable by changing the configuration of links and joints of the manipulator structure and / or the imaging instrument. In one or more embodiments, the operator may use an input device to change the field of view using steering commands to move the field of view with a desired motion. Depending on how the motion commands for moving the field of view are applied, the resulting motion may be more or less intuitive to the operator. Thus, it is desirable to command movement of the field of view using a more intuitive scheme, to facilitate operator control.SUMMARY

[0007] In general, in one aspect, one or more embodiments relate to a computer- assisted system comprising: an input device configured to be manipulated by a user; a manipulator structure comprising a plurality of manipulator links extending from a proximal base, the plurality of manipulator links coupled by a plurality of manipulator joints, the manipulator structure configured to support an imaging instrument comprising a plurality of instrument links coupled by at least one instrument joint, the imaging instrument having a field of view that defines a view frame; wherein when the manipulator structure is supporting the imaging instrument, the manipulator structure and the imaging instrument together comprise an aggregate plurality of joints, the aggregate plurality of joints comprising the plurality of manipulator joints and the at least one instrument joint; and a control system comprising one or more processors, the control system configured to, in an imaging instrument movement mode: receive, from the input device, a user input to move the field of view with a desired motion; determine, based on the desired motion, a motion command that moves the field of view in accordance with a first alignment relationship, wherein the first alignment relationship establishes: a first one-to-one mapping between first, second, and third translational degrees of freedom of the input device in an input device base frame and respective first, second, and third translational degrees of freedom of the field of view in the view frame, and a first one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base frame and respective first, second, and third rotational degrees of freedom of the field of view in the view frame; and command the aggregate plurality of joints based on the motion command.

[0008] In general, in one aspect, one or more embodiments relate to method for viewbased steering of an imaging instrument associated with a computer-assisted system, whereinthe computer-assisted comprises: an input device configured to be manipulated by a user; a manipulator structure comprising a plurality of manipulator links extending from a proximal base, the plurality of manipulator links coupled by a plurality of manipulator joints, the manipulator structure configured to support the imaging instrument comprising a plurality of instrument links coupled by at least one instrument joint, the imaging instrument having a field of view that defines a view frame; wherein when the manipulator structure is supporting the imaging instrument, the manipulator structure and the imaging instrument together comprise an aggregate plurality of joints, the aggregate plurality of joints comprising the plurality of manipulator joints and the at least one instrument joint; and a control system; wherein the method, executed by the control system, comprises: when the control system is configured to operate in an imaging instrument movement mode: receiving, from the input device, a user input to move the field of view with a desired motion; determining, based on the desired motion, a motion command that moves the field of view in accordance with a first alignment relationship, wherein the first alignment relationship establishes: a first one-to-one mapping between first, second, and third translational degrees of freedom of the input device in an input device base frame and respective first, second, and third translational degrees of freedom of the field of view in the view frame, and a first one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base frame and respective first, second, and third rotational degrees of freedom of the field of view in the view frame; and commanding the aggregate plurality of joints based on the motion command.

[0009] Other aspects of the invention will be apparent from the following description and the appended claims.

[0010] Benefits of the disclosed technique include, and are not limited to, reducing the operator’s cognitive load, increasing an operator’s accuracy in commanding motion of the field of view, increasing the efficiency for the use of the computer-assisted systems, reducing the time needed to perform procedures with the computer-assisted system, increasing the speed and ease in learning to use the computer-assisted system, and the like.BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figuresmay be represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

[0012] FIG. 1A shows an example computer-assisted system in accordance with one or more embodiments.

[0013] FIG. IB shows an example computer-assisted system in accordance with one or more embodiments.

[0014] FIG. 2A shows elements of a manipulator assembly of a computer-assisted system in accordance with one or more embodiments.

[0015] FIG. 2B shows a reconfiguration of elements of a manipulator assembly in accordance with one or more embodiments.

[0016] FIG. 3 shows an input portion of an example input device in accordance with one or more embodiments.

[0017] FIG. 4 illustrates elements and operation of a manipulator assembly supporting an imaging device in accordance with one or more embodiments.

[0018] FIGs. 5A and 5B show control schemes of a computer assisted system in accordance with embodiments of the disclosure.

[0019] FIG. 5C shows a controller comprising one or more processors, in accordance with embodiments of the disclosure.

[0020] FIG. 6A schematically illustrates the steering of a field of view with and without an alignment filter in accordance with embodiments of the disclosure.

[0021] FIG. 6B shows an implementation of an alignment filter in accordance with embodiments of the disclosure.

[0022] FIG. 6C schematically illustrates operations performed by the alignment filter in accordance with embodiments of the disclosure.

[0023] FIGs. 7 shows a flowchart describing an example method in accordance with one or more embodiments.DETAILED DESCRIPTION

[0024] Specific embodiments of the disclosure will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.

[0025] In the following detailed description of embodiments of the disclosure, numerous specific details are set forth in order to provide a more thorough understanding ofthe invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0026] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements, and is not to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before”, “after”, “single”, and other such terminology. Rather, the use of ordinal numbers is to distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

[0027] This disclosure describes various devices, elements, and portions of computer- assisted systems and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element (e.g., three degrees of translational freedom in a three-dimensional space, such as along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an element or a portion of an element (e.g., three degrees of rotational freedom in three-dimensional space, such as about roll, pitch, and yaw axes, represented in angle-axis, rotation matrix, quaternion representation, and / or the like). As used herein, and for a device with a kinematic series, such as with a repositionable structure with a plurality of links coupled by one or more joints, the term “proximal” refers to a direction toward a base of the kinematic series, and “distal” refers to a direction away from the base along the kinematic series.

[0028] As used herein, the term “pose” refers to the multi-degree of freedom (DOF) spatial position and orientation of a coordinate system of interest attached to a rigid body. In general, a pose includes a pose variable for each of the DOFs in the pose. For example, a full 6-DOF pose for a rigid body in three-dimensional space would include 6 pose variables corresponding to the 3 positional DOFs (e.g., x, y, and z) and the 3 orientational DOFs (e.g., roll, pitch, and yaw). A 3-DOF position only pose would include only pose variables for the 3 positional DOFs. Similarly, a 3-DOF orientation only pose would include only pose variables for the 3 rotational DOFs. Further, a velocity of the pose captures the change in pose over time (e.g., a first derivative of the pose). For a full 6-DOF pose of a rigid body inthree-dimensional space, the velocity would include 3 translational velocities and 3 rotational velocities. Poses with other numbers of DOFs would have a corresponding number of velocities translational and / or rotational velocities.

[0029] Aspects of this disclosure are described in reference to computer-assisted systems, which can include devices that are teleoperated, externally manipulated, autonomous, semiautonomous, and / or the like. Further, aspects of this disclosure are described in terms of an implementation using a teleoperated surgical system, such as the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including teleoperated and nonteleoperated, and medical and non-medical embodiments and implementations. Implementations on da Vinci® Surgical Systems are merely exemplary and are not to be considered as limiting the scope of the inventive aspects disclosed herein. For example, techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperated systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like.Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0030] Referring now to the drawings, in which like reference numerals represent like parts throughout the several views, FIG 1A illustrates an example computer-assisted system 100, according to various embodiments. As shown in Figure 1A, the computer-assisted system 100 includes, without limitation, a manipulator structure comprising a manipulator assembly 110 and a user input system 150. In a teleoperation scenario, an operator (not shown) uses the user input system 150 to operate the manipulator assembly 110, such as in a leader-follower configuration (also often called teleoperation configuration or master-slave configuration in industry) of the computer-assisted system 100. In the leader-followerconfiguration, the user input system 150 is the leader, and the manipulator assembly 110 is the follower of the leader-follower configuration.

[0031] The manipulator assembly 110 can be used to introduce a set of one or more instruments to a work site through a port 130, such as a gel port, a cannula, entry guide, etc. (a cannula is shown in the figure) inserted in an aperture. In a medical scenario, the work site can be on or within a body cavity of a patient, and the aperture can be a minimally invasive incision or a natural body orifice. When used, the port 130 can be free-floating, held in place by a fixture separate from the manipulator assembly 110, or held by a drivable structure 122 (a linkage is shown in the figure) or other part of the manipulator assembly 110. The drivable structure 122 is coupled to additional joints and links 114, 120 of the manipulator assembly 110, and these additional joints and links 114, 120 are mounted on a base 112. In some embodiments, the drivable structure 122 terminates in a manipulator- supporting link 124. In the example shown, the manipulator assembly 110 further comprises a set of manipulators 126 coupled to the manipulator-supporting link 124. Each of the manipulators 126 include a carriage (or other instrument-coupling link) configured to couple to an instrument, and each of the manipulators 126 include one or more joint(s) that can be driven to move the carriage. For example, a manipulator 126 can include a prismatic joint that, when driven, linearly moves the carriage and any instrument(s) coupled to the carriage. In some embodiments, this linear motion is along an insertion axis of the instrument, as further described below with reference to FIG. 2A. In this configuration, the elements of the manipulator assembly 110 that are proximal to the manipulators 126 may be understood as a proximal repositionable structure.

[0032] In some embodiments, the additional joints and additional links 114 and 120 are used to position the port 130 at the aperture or another location. FIG. 1A illustrates a prismatic joint for vertical adjustment (as indicated by arrow “A”) and a rotary joints for horizontal adjustment (as indicated by arrow “B”). The drivable structure 122 is used to robotically pivot the port 130 (and the instruments disposed within it at the time) in yaw, pitch and roll angular rotations about the remote center of motion as indicated by arrows C, D and E, respectively. These joints may be considered part of the proximal repositionable structure.

[0033] In some embodiments, actuation of the degrees of freedom provided by joint(s) of the instrument(s) is provided by actuators disposed in, or whose motive force (e.g., linear force or rotary torque) is transmitted to, the instrument(s). Examples of actuatorsinclude rotary motors, linear motors, solenoids, and / or the like. The actuators drive transmission elements in the manipulators and / or in the instruments to control the degrees of freedom of the instrument(s). For example, the actuators can drive rotary discs of the manipulator that couple with rotary discs of the instrument(s), where driving the rotary discs of the instruments drives transmission elements in the instrument that couple to move the joint(s) of the instrument, or to move the end effector(s) of the instrument, as further discussed below with reference to FIG.2. Accordingly, the degrees of freedom of the instrument(s) are controlled by actuators that drive the instrument(s) in accordance with control signals determined based on inputs from the associated input devices (e.g., input devices 152 of the user input system 150). The control signals are determined in order to cause instrument motion or other actuation as indicated by movement of the input control devices or any other control signal. Furthermore, in some embodiments, appropriately positioned sensors, e.g., encoders, potentiometers, and / or the like, are provided to enable measurement of indications of the joint positions, or other data that can be used to derive joint position, such as joint velocity. The actuators and sensors are disposed in, transmit to, and / or receive signals from the manipulator(s) 126.

[0034] While a particular configuration of the manipulator structure including the manipulator assembly 110 is shown in FIG. 1 A, those skilled in the art will appreciate that embodiments of the disclosure can be used with any design of manipulator structure. For example, a manipulator assembly can have any number and any types of degrees of freedom, may or may not be configured to couple to a port, use a port other than a cannula, and / or other configuration different from what is shown in FIG 1A.

[0035] In the embodiments shown in FIG 1A, the user input system 150 includes one or more input devices 152 operated by the operator (not shown). The one or more input devices 152 are contacted and manipulated by the hands of the operator, with one input device for each hand. Examples of such hand-input-devices include any type of device manually operable by human operator, e.g., joysticks, trackballs, button clusters, and / or other types of haptic devices typically equipped with multiple degrees of freedom. A more detailed description of an input device is provided below in reference to FIG. 3. Additionally, in some embodiments, position, force, and / or tactile feedback devices (not shown) are employed to transmit position, force, and / or tactile sensations from the instruments back to the operator's hands through the input devices 152.

[0036] The input devices 152 are supported by the user input system 150 and are shown as mechanically grounded, and in other implementations may be mechanically ungrounded. An ergonomic support 156 is provided in some implementations. For example, FIG. 1A shows an ergonomic support 156 including forearm rests on which the operator may rest his or her forearms while manipulating the input devices 152. In some examples, the operator performs tasks at a work site near the manipulator assembly 110 during a medical procedure by controlling the manipulator assembly 110 using the input devices 152.

[0037] A display unit 154 is included in the user input system 150. The display unit 154 displays images for viewing by the operator. The display unit 154 provides the operator with a view of the worksite with which the manipulator assembly 110 interacts. The view can include, for example, stereoscopic images or three-dimensional images to provide a depth perception of the worksite and the instrument(s) of the manipulator assembly 110 in the worksite. The display unit 154 can be moved in various degrees of freedom to accommodate the operator’s viewing position and / or to provide control functions. Where a display unit (such as the display unit 154 is also used to provide control functions, such as to command the manipulator assembly, the display unit also includes an input device (e.g., another input device 152).

[0038] When using the user input system 150, the operator can sit in a chair or other support in front of the user input system 150, position his or her eyes to see images displayed by the display unit 154, grasp and manipulate the input devices 152, and rest his or her forearms on the ergonomic support 156 as desired. In some implementations, the operator can stand at the workstation or assume other poses, and the display unit 154 and input devices 152 may differ in construction, be adjusted in position (height, depth, etc.), etc.

[0039] FIG. IB illustrates an example system 170, according to various embodiments. The system 170 corresponds to the computer-assisted system 100 and includes one or more computing systems 172. A computing system 172 includes a processing system and is used to process input provided by the user input system 150, e.g., from the input device(s) 152 manipulated by an operator. In some embodiments, a computing system 172 is further used to provide an output, e.g., a video image to the display unit 174. Examples of display unit 174 include LCDs, LEDs, organic LED displays, projectors, etc. In some embodiments, one or more computing systems 172 are used to control the manipulator assembly 110.

[0040] In one or more embodiments, the computing system(s) 172 executes one or more control methods. The control methods include instructions for controlling one or morecomponents of the manipulator assembly 110. In one or more embodiments, joint movements of the manipulator assembly 110 are controlled by one or more control methods driving one or more joints using actuators of the manipulator assembly 110, the joint movements being calculated by a processor of a processing system of the computing system(s) 172. The control methods process control signals from the user input system 150 or elsewhere, and / or sensor signals (e.g., positional encoder data from joint position sensors, image data from image instruments such as ultrasonic probes or cameras or endoscopes, and / or the like), to calculate commands for the joint actuators.

[0041] In some embodiments, the control methods perform at least some of the calculations of the joint commands using vectors and / or matrices, some of which have elements corresponding to positions, velocities, and / or forces / torques of the joints. The range of alternative joint configurations available to the control methods can be conceptualized as a joint space. For example, in some embodiments, the joint space has as many dimensions as the manipulator assembly has degrees of freedom, and a particular configuration of the manipulator assembly represents a particular point in the joint space, with each coordinate corresponding to a joint state of an associated joint of the manipulator assembly.

[0042] As used herein, the term “state” of a joint or multiple joints refers to the control variables associated with the joint or the multiple joints, respectively. For example, the state of an angular joint refers to the angle defined by that joint within its range of motion, and / or to the angular velocity (or speed or direction) of the joint. Similarly, the state of an axial or prismatic joint refers to the joint's axial or linear position, and / or to its axial or linear velocity (or speed or direction). While one or more of the control methods described herein include position controllers, they often also have velocity control aspects. Alternative embodiments can rely primarily or entirely on velocity controllers, force controllers, acceleration controllers, and / or the like without departing from the disclosure. Various aspects of control systems that can be used in such devices are described in U.S. Pat. No.6,699,177, which is incorporated herein by reference. In general, as long as the movements described are based on the associated calculations, the calculations of movements of the joints and movements of an end effector described herein are performed using a position control technique, a velocity control technique, an acceleration control technique, a force or torque control technique, a combination of some or all of the foregoing, and / or the like.

[0043] In some embodiments, the control modes include one or more other types of control modes. For example, during a robotic task being performed under the control of inputdevices 152 operated by a user, various joints of the manipulator assembly can be commanded to a same position and controlled to maintain static positions. However, in another control mode, one or more of the joints can be commanded to be “floating”, and facilitate motion of that joint due to externally applied force. For example, a joint held in place by a brake can be floated by partially or entirely releasing the brake. As another example, a joint driven by actuator(s) can be held in place by commanding the actuator(s) to hold the joint position, and be floating by updating the command to the actuator(s) to the then-current position, velocity, and / or acceleration of the joint. .

[0044] In various embodiments, multiple different control modes are combined during operation of the manipulator assembly. For example, some joints could be controlled to maintain position and resist or rebound from attempted external articulation of those joints, while other joints could be controlled to move those other joints. Parameters such as joint position, velocity, or acceleration of the joints are detected by joint sensors. The sensor signals are used to provide kinematic information of the manipulator assembly.

[0045] The architecture of the control methods used for controlling the manipulator assembly can be of any appropriate form. As a specific example, the control architecture can be hierarchical, and could include a high-level controller and multiple joint controllers. A commanded movement is received by the high-level controller in, for example, a Cartesiancoordinate space (referred to herein as Cartesian-space). The commanded movement could be, for example, based on a movement command (e.g., in the form of a position and / or velocity) received from the user input system 250, or any other system that provides a movement command. The commanded movement is converted into commanded joint positions or joint velocities (e.g., linear or angular joint positions, linear or angular joint velocities). In some embodiments, the conversion is performed using an inverse kinematics algorithm. Subsequently, the joint controllers convert the received commanded joint positions or velocities into commanded currents to drive the actuators producing joint movements. The joint movements together produce a manipulator assembly movement that reflects the commanded movement.

[0046] In some embodiments, a joint controller controls a joint position. In some embodiments, the joint controller controls other variables such as joint velocity and / or joint force (linear force or angular torque). A joint controller receives a feedback signal in the form of a sensed joint state from an associated joint sensor, which it can use for closed- loop control. The sensed joint state includes, for example and without limitation, a joint position,a joint velocity (or component of velocity such as speed or direction), a joint acceleration (or component of acceleration), and / or the like, representing the joint movement. The sensed joint state is derived from the signals obtained from the joint sensor. A joint sensor can be, for example, an encoder, a potentiometer, an accelerometer, a hall effect sensor, and / or the like. In some embodiments, a state observer or estimator (not shown) is used. Each joint controller can implement any appropriate control scheme, such as a proportional integral derivative (PID), proportional derivative (PD), full state feedback, sliding mode, and / or various other control schemes, without departing from the disclosure.

[0047] In one or more embodiments, the control methods further perform at least one of the steps described in FIG. 7 below.

[0048] A computing system 172 may include, without limitation, one or more computer processors, non-persistent storage (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, etc.), a communication interface (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and / or numerous other elements and functionalities. In some embodiments, a computer processor of a computing system 172 is an integrated circuit for processing instructions. For example, the computer processor can be one or more cores or micro-cores of a processor.

[0049] In some embodiments, a communication interface of a computing system 172 includes an integrated circuit for connecting the computing system 172 to a network (not shown) and / or to another device, such as another computing system 172. Further, in some embodiments, the computing system 172 includes one or more output devices, such as a display unit 174, a printer, a speaker, external storage, or any other output device. Software instructions in the form of computer readable program code to perform embodiments of the disclosure are stored, in whole or in part, temporarily or permanently, on non-transitory computer readable medium. Specifically, the software instructions correspond to computer readable program code that, when executed by a processor(s), is configured to perform one or more embodiments of the disclosure. In some embodiments, a computing system 172 is connected to or be a part of a network. The network may include multiple nodes. Each node corresponds to a different computing system, group of computing systems, group of nodes, and / or the like.

[0050] In some embodiments, the manipulator assembly 110 couples to an instrument when used to perform a procedure. The instrument can include an imaging device, e.g., an endoscope or an ultrasonic probe, usable to capture images of the worksite and output the captured images to an auxiliary system 180. In some embodiments, the auxiliary system 180 processes the captured images using one or more image processing techniques prior to any subsequent display. For example, the auxiliary system 180 can overlay the captured images with a virtual control interface prior to displaying the combined images to the operator via the user input system 150. In some embodiments, one or more separate display units 174 are coupled with a computing system 172 and / or the auxiliary system 180 for local and / or remote display of images, such as images of the procedure site or other related images.

[0051] FIG. 2A shows elements of an example manipulator assembly 200, in accordance with one or more embodiments. In the example shown in FIG. 2A, the elements of the manipulator assembly 200 correspond to a distal portion of the manipulator assembly 110 of FIG. 1 A. The manipulator assembly 200 can be used to introduce a set of instruments 260, 270, 280 to a work site through a port 130 inserted in an aperture.

[0052] The manipulator assembly 200 includes a manipulator- supporting link 202 that may be rotated about roll axis 290 (arrow E in FIGs. 1A and 2A) of the manipulatorsupporting link 202. The three instruments 260, 270, and 280 are each supported by the manipulator- supporting link 202 through a respective manipulator 126. As previously discussed in reference to FIG. 1A, each of the instruments 260, 270, 280 is also supported by a manipulator 126 (proximal of the manipulator-supporting link 202 and therefore not visible in FIG. 2A) where the instrument is coupled to a carriage. Translational movement of the carriage of the manipulator 126 may be used for an insertion / retraction of each of the instruments 260, 270, 280 (arrow F in FIG. 2A, for simplicity shown only for instrument 280). Each of the instruments 260, 270, 280 include an instrument shaft 262, 272, 282 and an end effector 264, 274, 284. In the example of FIG. 2A, instruments 260 and 270 are illustrated as equipped with jawed end effectors and instrument 280 is illustrated as equipped with an imaging instrument (e.g., an endoscopic camera). Each of the end effectors may be movable in one or more degrees of freedom. For example, the end effector 284 may be able to change orientation in pitch and yaw degrees of freedom (arrows G and H, respectively). End effectors 264, 274, and 284 may include additional degrees of freedom than shown. For example, for jawed instruments, these degrees of freedom may further enable and open / close operation of jaws of the end effector. In the example of FIG. 2A, instruments 260 and 270further include a degree of freedom for rotation of the instrument shaft 262, 272 (arrow I, only shown for instrument 270).

[0053] In order to increase the range of motion of the instrument 280 (which may not have a degree of freedom for rotation about the instrument shaft 282, a roll movement about the roll axis 290, as indicated by arrow E, may be performed. Such a roll movement changes the position and the orientation of the end effector 284. A roll movement of the end effector 284 is thus possible, even in absence of a roll degree of freedom at the instrument 280. As illustrated in FIG. 2B, discussed below, the additional roll movement enables the instrument to assume additional configurations as compared to otherwise identical systems incapable of roll movement. However, it may be desirable to avoid affecting the configuration of the end effectors 264, 274 of instruments 260, 280, when performing the roll movement about roll axis 290. One such scenario assumes that the end effector 284 of instrument 280 is or includes an imaging device, which may be used to capture still or moving images (e.g., videos) of operations performed using the end effectors 264, 274 of instruments 260, 270.This enables an operator viewing images captured by the imaging device to better observe the actions of instrument 260, 270. If the operator wants to change the field of view of the imaging device, for example to view the operations performed using the end effectors 264, 274 from a different angle, this may be facilitated by roll movement about the roll axis 290. In this scenario, a change in the configuration of the end effectors 264, 274, caused by the roll movement, would be undesirable.

[0054] To avoid such an unintended change in position, orientation, internal configuration (e.g., jaw opening for jawed instruments) state of the end effectors 264, 274, the instruments 260, 270 may be reconfigured to compensate for the roll movement about the roll axis 290. In the example illustrated in FIG. 2A, instruments 260, 270 are equipped with shaft offset joints 266, 276. Reconfiguration of the shaft offset joints 266, 276, along with insertion / retraction and / or rotation along the instrument shafts 262, 272 of the instruments 260, 270 may be used for this compensation.

[0055] FIG. 2B shows a reconfiguration of elements of a manipulator assembly (e.g., manipulator assembly 110, 200) in accordance with one or more embodiments. In the example, the instrument 280 is a stereoscopic imaging instrument, with two cameras and an illuminator (all shown in the illustration 230) disposed in between. In the schematic illustration 230, the instrument 280 is viewed facing the two cameras, and only the front facing surface of the instrument 280 is shown. The illustration 230 further includes acrosshair 292 that identifies the orientation of the manipulator- supporting link (not shown). Rotation of the manipulator-supporting link, thus maps to a corresponding rotation of the crosshair 292. The left panel of the illustration 230 shows the end effector 284 of the imaging instrument 280 in four different configurations. Three configurations are illustrated using solid lines, identifying camera configurations (in the example in two-dimensional space) that can be reached with the manipulator- supporting link oriented as indicated by the crosshair 292. One configuration is illustrated using dashed lines, identifying a camera configuration that cannot be reached with the manipulator-supporting link oriented as indicated by the crosshair 292. The right panel of the illustration 230 shows the end effector 284 of the imaging instrument 280 in the configuration that previously could not be reached. This configuration can now be reached because the manipulator-supporting link 202 has been rotated as indicated by the crosshair 292 rotated as indicated by arrow E, also shown in FIG. 2A. As FIG. 2B illustrates, the instrument 280 can be rotated, even though the instrument itself does not have the corresponding rotational degree of freedom. While the roll E about the roll axis 290 does not directly result in a pure rotation of the instrument 280 because the instrument shaft 282 is offset from the roll axis 290, the shaft offset joints 286, combined with insertion / retraction along instrument shaft 282 enable a pure rotation of the instrument 280.

[0056] FIG. 3 is a perspective view of an input portion 300 of an example input device such may be used for the input devices 152 depicted in FIG. 1A. In some implementations, the input portion 300 includes one or more gimbal mechanisms. In the example of FIG. 3, the input portion 300 includes a handle 302 which is contacted by an operator to manipulate the input device. Further, in this example, the handle 302 includes two grips that each include a finger loop 304 and a grip member (depicted as grip members 306a, 306b). The two grip members 306a, 306b are positioned on opposite sides of a central portion 303 of the handle 302, and the grip members 306a, 306b can be grasped, held, or otherwise contacted by an operator’s fingers. Each finger loop 304 is attached to a respective grip member 306a or 306b and can be used to secure an operator’s fingers to the associated grip member 306a or 306b. In this example, finger contacts 305 can be connected or formed at the unconnected end of the grip members 306a, 306b to provide surfaces to contact the operator’s fingers. The operator may also contact other portions of handle 302 while grasping the grip members 306a, 306b.

[0057] Each grip member 306a, 306b can be moved in an associated degree of freedom (depicted as degrees of freedom 308a, 308b). In some examples, the grip members 306a, 306b are each coupled to the central portion 303 of the handle 302 at respective rotational couplings, allowing rotational movement of the grip members about associated grip axes 307a, 307b, with respect to the central portion 303. As such, each grip member 306a, 306b can be moved in an associated degree of freedom (i.e., degree of freedom 308a about grip axis 307a and degree of freedom 308 about grip axis 307b) by an operator contacting the grip members (306a, 306b). In various implementations, a single grip member (e.g., 306a) and finger loop (304) can be provided, or only one of the grip members (e.g., 306a) can be moved in the corresponding degree of freedom (e.g., 308a) while the other grip member (e.g., 306b) can be fixed with reference to the handle 302. For example, the positions of grip members 306a, 306b in their degrees of freedom can control corresponding rotational positions of an instrument or component thereof.

[0058] One or more grip sensors (not shown) can be coupled to the handle 302 and / or other components of the input portion 300 and can detect the positions of the grip members 306a, 306b in their respective degrees of freedom 308a, 308b. The grip sensors can send signals describing sensed positions and / or motions to the control of computing system(s) 172 of the computer-assisted system 100. In some modes (e.g., a teleoperation mode described later in the instant disclosure) or implementations, the computing system 172 can provide control signals to a device manipulated by the computer-assisted system 100 (e.g., manipulator assembly 110). For example, the positions of the grip members 306a, 306b in their respective degrees of freedom 308a, 308b can be used to control any of various degrees of freedom of an instrument (or, in some instances, the distal end of an instrument) supported by the manipulator assembly 110.

[0059] Various implementations of an input device, such as the that depicted in the input portion 300 of FIG. 3, can provide one or more active actuators (e.g., motors, voice coils, etc.) to output active forces on the grip members 306a, 306b in the degrees of freedom 308a, 308b. For example, a sensor and / or actuator can be housed in central portion 303 or in housing 309 and coupled to the grip members 306a, 306b by a transmission. Some implementations can provide one or more passive actuators (e.g., brakes) or springs between the grip members 306a, 306b and the central portion 303 of the handle 302 to provide resistance in particular directions of the grips (e.g., movement in directions toward each other in degrees of freedom 308a, 308b).

[0060] The handle 302 can additionally be provided with a rotational degree of freedom 310 about a roll axis 312 defined between a first end and a second end of the handle 302. The roll axis 312 is a longitudinal axis in this example that extends approximately along the center of the central portion 303 of handle 302. The handle 302 can be rotated about the roll axis 312 with respect to a base member of the input portion 300, such as a base member that includes the housing 309. For example, an operator can rotate the grip members 306a, 306b and central portion 303 as a single unit around the roll axis 312), with respect to the housing 309, to provide control of manipulator assembly 110 or other elements.

[0061] Additionally, one or more input sensors (not shown) can be coupled to the handle 302 to detect the orientation of the handle 302 in the rotational degree of freedom 310. For example, the sensor can send signals describing the orientation to the computing system(s) 172 that can provide control signals to the manipulator assembly (110) as described above. For example, rotation of the handle 302 in the rotational degree of freedom 310 can control a particular degree of freedom of an instrument 260, 270, 280 of the manipulator assembly 110 that is different than another degree of freedom controlled by the degrees of freedom 308a, 308b of the grip members 306a, 306b.

[0062] Some implementations of the input portion 300 can provide one or more actuators to output forces on the handle 302 (including grip members 306a, 306b and finger loops 304 in the rotational degree of freedom 310. For example, a sensor and / or actuator can be housed in the housing 309 and coupled to the handle 302 by a shaft extending through the central portion 303) of the handle 302.

[0063] In various implementations, the handle 302 can be provided with additional degrees of freedom. For example, a rotational degree of freedom 320 about a yaw axis 322 can be provided to the handle 302 at a rotational coupling between an elbow shaped link 324 and a link 326, where the elbow shaped link 324 is coupled to the handle 302 (e.g., at the housing 309). In this example, the yaw axis 322 intersects and is orthogonal to the roll axis 312. Additional degrees of freedom can similarly be provided. For example, the link 326 can be elbow- shaped and a rotational coupling can be provided between the other end of link 326 and another link (not shown). A rotational degree of freedom 328 about an axis 330 can be provided to the handle 302 at the rotational coupling. In some examples, the input portion 300 can allow movement of the handle 302 within the workspace of the input system 150 with a plurality of degrees of freedom, e.g., six degrees of freedom including three rotational degrees of freedom and three translational degrees of freedom. One or more additionaldegrees of freedom can be sensed by associated input sensors and / or actuated by actuators (motors, etc.) similarly as described above for the degrees of freedom 308a, 308b, 310, etc., the sensors and actuators coupled to the input portion 300. In various implementations, sensors can sense positions of the handle in a degree of freedom, or sense orientations of the handle in a degree of freedom, or sense positions and orientations of the handle in multiple degrees of freedom. For example, positions in a translational degree of freedom and orientations in a rotational degree of freedom can be sensed by one or more associated input sensors. In some examples, a position in a translational degree of freedom and / or orientation in a rotational degree of freedom can be derived from rotations of components (e.g., links of a linkage) coupled to the handle 302 as sensed by rotational sensors. Some implementations can include linear sensors that can directly sense translational motion of one or more components coupled to the handle 302. In some implementations, each additional degree of freedom of the handle 302 can control a different degree of freedom (or other motion) in the manipulator assembly 110.

[0064] In an example implementation, the handle 302 is mechanically grounded, i.e., supported in space by a kinematic chain with an end stationary at mechanical ground, such as a floor, wall, or ceiling. For example, the housing 309 can be coupled to a mechanical linkage that is coupled to the ground or an object connected to ground, providing a stable platform for the use of the input portion 300. For example, a grounded mechanical linkage can be connected to a base member, e.g., with one or more rotary couplings, ball joints, or other couplings, including linear joints. The mechanical linkage can provide six or more degrees of freedom to the handle 302.

[0065] In the example of FIG. 3, the handle 302 includes one or more switches 350. As depicted, the one or more switches 350 can be coupled to the central portion 303 or to mechanisms within central portion 303. For example, two switches 350 can be positioned on opposite sides of axis 312, and / or additional switches can be provided. In some examples, a switch 350 has a portion that can slide parallel to the axis 312, e.g., as directed by an operator’s finger, or the switch portion can be depressed. In some implementations, the switch 350 can be moved to various positions to provide particular command signals, e.g., to select functions, options, or modes of the input system 150 and / or input device. In some implementations, one or more of the switches 350 can be implemented as a button (e.g., depressed in a direction, such as perpendicular to the axis 312 or other direction), a rotary dial, a switch that moves perpendicular to the axis 312, or other type of input control.Switches 350 can use electromagnetic sensors, mechanical switches, magnetic sensors, or other types of sensors to detect positions of the switch.

[0066] As previously stated, the computer-assisted system 100 can be a teleoperated system with a leader-follower control scheme. The leader-follower control scheme is also called the master-slave control scheme in literature. With the leader-follower control scheme, the input system 150 forms part or all of a “leader” ( “master”) device that commands at least a portion of the manipulator structure, such as part or all of manipulator assembly 110) (the “follower” or “slave” device).

[0067] In general, a control system, e.g., computing system(s) 172, receives control signals from the input system 150 and generates actuation signals which are sent to command actuators of manipulator assembly 110. The control system can also receive sensor signals that indicate positions, orientations, states, and / or changes of various follower components (e.g., manipulator structure elements) from the manipulator assembly 110 and send actuation signals to the input system 150 to provide force, torque, and / or position feedback to the operator.

[0068] In one or more embodiments, an input system 150 is equipped with two input portions 300 (one for each of the operator’s hands) is used to control elements instruments, e.g., instruments 260, 270, and 280. A specific implementation of control systems and methods that implement a view-based steering of an imaging instrument (e.g., instrument 280) is described below in reference to FIG. 4.

[0069] While FIGs. 1A-3 show various configurations of components, other configurations may be used without departing from the scope of the disclosure. For example, various components may be combined to create a single component. As another example, the functionality performed by a single component may be performed by two or more components. While examples of particular manipulator assemblies, particular repositionable structures, instruments, input systems, and input portions are shown, the disclosure generalizes to any type of manipulator assemblies, repositionable structures, instruments, input systems, and input portions with any number of degrees of freedom. Further, while components are often described in context of medical scenarios such as surgical scenarios, embodiments of the disclosure are equally applicable to other domains that involve robotic manipulation, e.g., non-surgical scenarios or systems, non-medical scenarios or systems, and / or the like.

[0070] FIG. 4 shows elements and operation of a manipulator structure 400 of a computer-assisted system, in accordance with one or more embodiments. The manipulator structure s shown comprises a manipulator assembly 410, which is supporting an imaging instrument 420.

[0071] When operating a computer-assisted system to perform a procedure, a user may use visual feedback from the environment. The environment may include a target site, and the user may, for example, want to visually inspect the target site and / or interact with the target site, e.g., by performing a procedure. The visual feedback may be provided by the imaging instrument 420 based on the field of view of the imaging instrument. The visual feedback may be provided to the user in the form of a 3D or 2D view.

[0072] The user may want to change the view for various reasons, e.g., to better observe the target site. The imaging instrument’s position and / or orientation may be updated as desired by the user, by changing the configuration of the joints of the manipulator assembly 410 and / or the imaging instrument 420.

[0073] In one or more embodiments, the configuration (position and / or orientation) of the imaging instrument 420 is updated, based on a user input received at, for example, the input portion 300 of an input device of input system 150 manipulated by the user as subsequently described.

[0074] The manipulator assembly 410 comprises a plurality of links extending from a proximal base 412 of the manipulator structure 400. The plurality of links is coupled together by a plurality of joints. The proximal base 412 is proximal to the plurality of links, and may be stationary or may be movable. For example, the proximal base 412 may be installed on tracks enabling a linear motion, on a rollable cart, on another manipulator structure actuatable to move the proximal base 412, etc. A manipulator structure base frame 414 is associated with the proximal base 412. Certain operations, such as kinematic operations may be performed in the manipulator structure base frame 414. For example, the imaging instrument position and / or orientation may be determined in the manipulator structure base frame using forward kinematics of the manipulator structure 410 and the imaging instrument 420. For a proximal base 412 that is stationary, the manipulator structure base frame may be a world frame or may have a continuous offset from a world frame.

[0075] The manipulator structure 410 may be any type of manipulator structure. The manipulator structure may be a general-purpose robotic structure, or it may be a robotic structure designed for a specific application, such as a medical device robotic structure.Examples of manipulator structures include the elements of the manipulator assembly (110 in FIG. 1A; 200 in FIG. 2A) that support an instrument, e.g., imaging instrument 420.

[0076] The manipulator structure 410 may include any number of links 416. The links 416 are coupled by joints 418 that may be of any type, such as revolute joints and prismatic joints.

[0077] The imaging instrument 420 may be any type of imaging instrument. The imaging instrument may be a general-purpose monoscopic or stereoscopic camera or it may be a specialized instrument such as an endoscope. An example of an imaging instrument is provided in FIG. 2A. Additional details may be found in U.S. Patent No. 8,620,473, e.g., in FIG. 8 showing an articulable imaging system 1750 and the associated description. The imaging instrument may be articulable separately from other instruments that may be present, e.g., as shown in FIG. 2A. Accordingly, the imaging instrument 420 may have its own joints, in addition to the joints of the manipulator structure 410. Any number of joints may be present, as long as the number of joints is sufficient to enable movement of the field of view as discussed below and, more generally, the combination of the manipulator structure 410 and the imaging instrument 420 is not limited to a particular kinematic configuration.

[0078] In one or more embodiments, the user may control the operation of the manipulator assembly 400 using different operating modes. Depending on the selected operating mode, the manipulator assembly 400 may respond differently to a user input provided by the user. Examples of robotic systems operating in different operating modes when receiving user inputs are provided in U.S. Patent No. 9,586,323. U.S. Patent No. 9,586,323 is hereby incorporated by reference in its entirety.

[0079] One or more of the operating modes may be imaging instrument movement modes, which may be used to enable the user to move the field of view 422 of the imaging instrument 420, where motion of an input device (e.g., handle 302 of the input portion 300 of the input device), when operated by the user, commands motion of the imaging instrument 420, thereby changing the field of view of the imaging instrument according to a desired motion. A description is subsequently provided in reference to FIGs. 5A-5C, 6A-6C, and 7.

[0080] Continuing with the discussion of FIG. 4, a view frame 424 is fixed relative to the field of view 422. A view axis 426 extends as shown in FIG. 4. The field of view 422 may be a cone or frustrum with the view axis at its center, and may be determined by the optical characteristics of the imaging instrument 420. While the field of view 422 may be modifiable using optical and digital operations (optical / digital zoom, digital translation and / orroll operations), in the following discussion the field of view 422 is assumed to be changed by movement of the imaging instrument, e.g., relative to the manipulator structure base frame 414.

[0081] Movement of the field of view 422 may be based on a user input received from a user at an input device. The user input may specify a desired motion of the field of view. The desired motion may include one or more components including a roll component, one or more translation components, a pitch component, a yaw component, and / or an insertion / retraction component, as illustrated in FIG. 4. The components may be summarized as a roll component and one or more non-roll components (comprising the one or more translation components, the pitch component, the yaw component, and / or the insertion / retraction component).

[0082] The motion of the field of view may be described relative to the manipulator structure base frame 414, or relative to any other reference frame. The motion of the field of view may also be described relative to the view frame 424. For example, a current view frame may be obtained relative to a previous view frame, enabling the description of the current view frame relative to the previous view frame.

[0083] The determining of motion commands that can be used to cause motion of the joints in order to update the field of view based on the desired motion, in accordance with embodiments of the disclosure, is subsequently discussed.

[0084] FIGs. 5A and 5B show control schemes 500, 550 of a computer assisted system, in accordance with embodiments of the disclosure. Both control schemes may be used to update the field of view of an imaging instrument in multiple degrees of freedom (e.g., roll, pitch, yaw, translation, and / or insertion / retraction), as previously described. In one or more embodiments, the control schemes 500, 550 are used to update the field of view of an imaging instrument in three rotational and three translational degrees of freedom, i.e., six degrees of freedom.

[0085] The control schemes 500, 550 include a robotic controller 510 and an alignment filter 520. These components are subsequently discussed.

[0086] An input device, e.g., an input device of input system 150 as previously described, may be used to control the field of view of the imaging instrument. The input device may further be used to control other instruments (different from the imaging instrument). The input device may obtain a user input to move the field of view with a desired motion. The desired motion may be obtained in any reference frame, e.g., in an inputdevice base frame (e.g., fixedly attached to an input system 150 that supports an input portion 300), a world frame, the manipulator structure base frame, a frame centered on the user, e.g., a view or eye-centered frame, etc.

[0087] In one or more embodiments, the robotic controller 510 in combination with the alignment filter 520 are configured to determine a motion command based on the desired motion. The motion command may then be used to command the joints of the manipulator structure and / or the imaging instrument to cause motion of these joints such that the field of view of the imaging instrument changes according to the desired motion. The robotic controller 510 may be any kind of robotic controller and may use any appropriate technique. In some examples, the robotic controller 510 uses a Jacobian-based approach. An example is provided in FIG. 5C.

[0088] Referring to FIG. 5C, in one or more embodiments, the robotic controller 510 generates a motion command q to control the joints 418 based on a desired motion obtained at the input device, converted into an aligned desired motion Xstate by the alignment filter 520. The motion command q is in an n-dimensional joint space, with n being the number of joints 418 being controlled when positioning and / or orientating the imaging instrument 420. Joints of the manipulator structure 410 and / or joints of the imaging instrument 420 may be involved. The input (i.e., the aligned desired motion xstate) to the robotic controller 510 may be provided in cartesian coordinates, e.g., in the manipulator structure base frame 414, or in any other frame. In some embodiments, the view frame is used to provide the aligned desired motion Xstate- A description of the aligned desired motion Xstate, generated by the alignment filter 520 based on the desired motion, is provided below.

[0089] FIG. 5C shows a kinematic controller that includes an inverse kinematics stage J+block) that determines a joint configuration (joint angles / positions) q corresponding to the aligned desired motion Xstate received at the input of the robotic controller 510 from the alignment filter 520. Individual joints are controlled by joint controllers that drive joint actuators with currents to position or orient the joints as instructed by q. The inversekinematics stage may operate on increments (differentiation block) of a cartesian error generated by subtracting the current cartesian state from the aligned desired motion Xstate- The current cartesian state is generated using a forward kinematics model (forward kinematics block) operating on q. A gain (z block) is used to set the control gain of the feedback control loop. The output of J+is in increments and is integrated (J block in FIG. 5C) to obtain q. As previously noted, q may then be used to drive individual joint controllers.

[0090] Control architectures in accordance with embodiments of the disclosure may differ from the example shown in FIG. 5C. For example, the feedback loop may be closed over signals obtained from joint sensors, without departing from the disclosure. Furthermore, in some embodiments, controlling the joints is performed under consideration of potentially one or more constraints. Such constraints may be imposed by range of motion limitations on a cartesian and / or joint level, velocity limits, spatial constraints such as the maintaining of a remote center (e.g., as described in reference to FIG. 1A), collision avoidance, and other constraints that may be satisfiable when controlling a manipulator structure 410 / imaging instrument with kinematic redundancies. Other robotic controllers may output a velocity as an alternative to position / orientation or in addition to position / orientation.

[0091] In some embodiments, the robotic controller 510 includes multiple interacting kinematic controllers. Referring to the configuration of manipulator assembly 200 shown in FIG. 2A and the reconfiguration of the elements of the manipulator assembly shown in FIG. 2B, the robotic controller 510 may include a first kinematic controller that controls all joints of the manipulator structure and the instruments, but that does not include functionality to perform the instrument roll operation illustrated in FIG. 2B. Such a robotic controller may further include a second kinematic controller that augments the first kinematic controller by adding the instrument roll functionality to the robotic controller 510. The second kinematic controller may be dedicated to determining a joint angle for the joint that provides the roll degree of freedom of the manipulator-supporting link. The configurations that are not reachable using the first kinematic controller alone are, thus, made reachable through a roll of the manipulator-supporting link about the roll axis of the manipulator-supporting link, controlled by the second kinematic controller. The second kinematic controller is, thus, configured to roll the manipulator- supporting link 202 until the instrument 280 is oriented such that the desired configuration of the end effector 284 is achievable using movement of the instrument joints (e.g., joints 286) as illustrated in FIG. 2B.

[0092] Because a roll movement of the manipulator- supporting link 202 alters position and / or orientation of the instruments 260, 270, 280, thereby causing interactions with the operations performed by the first kinematic controller, the output of the second kinematic controller is injected into the feedback path of the first controller.

[0093] While the configuration as described relies on a first kinematic controller and a second kinematic controller, in an alternative configuration, a single kinematic controllermay determine the roll movement of the manipulator- supporting link together with the movements of the other joints of the manipulator assembly.

[0094] Analogous to the robotic controller 510 as described in reference to FIG. 5C, both the first and the second kinematic controllers may receive input (aligned desired motion xstate) in cartesian coordinates. The first kinematic controller may receive the full sixdimensional state, whereas the second kinematic controller may receive a three-dimensional state that is limited to the rotational components needed to determine the roll movement of the manipulator-supporting link. The first kinematic controller may output a motion command for all n joints, whereas the second kinematic controller may output a motion command for the joint responsible for the roll movement of the manipulator- supporting link only.

[0095] In one or more embodiments, the alignment filter 520 is functionally located between the input device and the robotic controller 510. The alignment filter 520 receives signals that represent the user input provided by the user at the user input device in the form of a desired motion and alters these signals to generate signals that represent an aligned desired motion used to drive the robotic controller 510.

[0096] In some embodiments, the alterations performed by the alignment filter enable an intuitive steering of the field of view of the imaging instrument by the user via the input device. A discussion of intuitive control and control behaviors in presence and absence of the alignment filter 520 are subsequently discussed in reference to FIG. 6A, followed by a discussion of implementations of the alignment filter as shown in FIG. 6B.

[0097] FIG. 6A schematically illustrates the steering of a field of view of an imaging instrument with and without an alignment filter, in accordance with embodiments of the disclosure.

[0098] A user, when intending to steer the field of view of the imaging instrument may expect the field of view to follow the user’s input at the input device in a certain manner, making the control intuitive. Deviations from such an expected behavior may be perceived as making the steering of the field of view less intuitive. Therefore, some control schemes for controlling motion of the field of view based on a user input may be more or less intuitive than others. What is perceived as intuitive may depend on circumstances. Frequently, a movement that is executed as expected in the user’s visual reference frame is considered intuitive by the user. Accordingly, what is considered intuitive may depend on the user’s current view. For example, when observing the entire manipulator assembly 400 (e.g., as anexternal observer looking at the entire system), a motion of the imaging instrument 420 that follows a commanded movement in the manipulator structure base frame 414 may be considered intuitive. A command to the left would move the imaging instrument 420 to the left, in the manipulator structure base frame 414; a command in an upward direction would move the imaging instrument 420 in an upward direction the manipulator structure base frame 414, etc. In contrast, this type of motion may no longer feel intuitive when the user is viewing the image captured by the imaging instrument 420, e.g., through the display unit 154. In this case, because the user’s current view is limited to the field of view of the imaging instrument itself, a corresponding motion that follows the commanded movement in the view frame 424 would be considered intuitive. The example of FIG. 6A illustrates this based on a sequence of user inputs. While FIG. 6A, for the sake of simplicity, is limited to a two- dimensional example, the concepts as described generalized to a three-dimensional space with three translational and three rotational degrees of freedom. Further, while the user inputs are shown as being provided sequentially in time at ti, t2, and t3, the user inputs may overlap in time or may occur simultaneously.

[0099] The sequence of three user inputs (first row of illustrations in FIG. 6A) includes a command to translate to the right, a counterclockwise roll command, and another command to translate to the right. Depending on the control scheme that is applied, the resulting motions of the field of view of the imaging instrument differ. The second row of illustrations in FIG. 6A illustrates the resulting three motions of the field of view when the user input is applied in the view frame. Each of the three motions of the field of view is illustrated from a current spatial configuration of the field of view (solid lines) to a subsequent or future spatial configuration of the field of view (dashed lines). While, as previously noted, the illustrations of FIG. 6A are limited to a two-dimensional change of the spatial configuration (a translation and a rotation), more generally, the spatial configuration of the field of view (current and future), may be based on all six degrees of freedom. The third row of illustrations in FIG. 6A illustrates the resulting motion of the field of view from a current spatial configuration of the field of view (solid lines) to a subsequent or future spatial configuration of the field of view (dashed lines) when the user input is applied in a frame different from the view frame, e.g., in the manipulator structure base frame. The behavior shown in the third row may be obtained using the robotic controller 510 without the alignment filter 520. In contrast, the behavior shown in the second row may be obtainedwhen the alignment filter 520 is present and realigns the user input to be in the view frame as further discussed below.

[0100] Referring to the second row of illustrations in FIG. 6A, a user input to the right results in a translation of the field of view to the right. This is true regardless of whether the motion of the field of view is observed in the manipulator structure base frame or in the view frame. Next, the counterclockwise roll input results in a counterclockwise rotation of the field of view. This is also true regardless of whether the motion of the field of view is observed in the manipulator structure base frame or in the view frame. The next user input to the right results in a translation of the field of view to the right, in the view frame. The translation is not to the right when observed in the manipulator structure base frame. Instead, the translation is to the right when observed in the view frame, because the illustrations in the second row are representative of a view-based steering of the imaging instrument in which the field of view is updated relative to the current field of view. When viewed from another frame, e.g., the manipulator structure base frame, a dissociation between the spatial configuration (position, orientation) of the input device and the spatial configuration of the field of view may occur. In the example of FIG. 6A, this is the result of the user input including a roll component. Over time, this dissociation may increase based on the motion history of field of view. Accordingly, when observed in the manipulator structure base frame, the same user input can result in a completely different movement of the field of view, depending on the history of user inputs and corresponding motion history of the field of view. In other words, this control scheme is path-dependent with no 1:1 mapping between input device configuration and field of view, in the manipulator structure base frame. When observed in the manipulator structure base frame, the motion command is, therefore, best understood as being determined based on the current spatial configuration of the field of view reflecting the motion history of the field of view. In the example of FIG. 5A, the motion history is formed by the motion of the field of view at time steps ti, t2, and t3. More generally, a motion history may be based on any number of time steps, e.g., starting with an entry into the imaging instrument movement mode that produces the behavior as illustrated in the second row of illustrations in FIG. 6A. Additional details related to the entering of the imaging instrument movement mode are provided below. In contrast, adjustment of the field of view in response to a user input remains consistent when observed in the view frame itself. In other words, the current spatial configuration of the field of view is independent from the motion history. A user controlling the field of view of the image device having the field of 1view may perceive this behavior as intuitive. Implementations that enable this behavior may be obtained using the alignment filter 520 of FIGs. 5 A and 5B. A more detailed description of the operations performed by the alignment filter is provided below in reference to FIG. 6B.

[0101] Referring to the third row of illustrations, the user input to the right results in a translation of the field of view to the right. This is true regardless of whether the motion of the field of view is observed in the manipulator structure base frame or in the view frame. The counterclockwise roll input results in a counterclockwise rotation of the field of view. This is also true regardless of whether the motion of the field of view is observed in the manipulator structure base frame or in the view frame. The next user input to the right results in a translation of the field of view to the right, in the manipulator structure base frame. The translation is not to the right in the view frame. Accordingly, the illustrations in the third row are representative of a non- view-based steering of the imaging instrument. This may be intuitive when steering the imaging instrument based on an external view of the manipulator assembly including the imaging instrument. However, it is unlikely to be intuitive to a user steering the field of view while viewing an image provided by the imaging instrument itself.

[0102] Referring to FIG. 5B, the control scheme 550 may produce the behavior as shown in the second row of illustrations of FIG. 6A when the alignment filter 520 is active. Alternatively, the control scheme may produce the behavior as shown in the third row of illustrations of FIG. 6A when the alignment filter 520 is bypassed. Being able to switch between these two different control behaviors may be beneficial, for example, when using the same input device to steer different types of instruments. For example, the view-based steering (second row of illustrations in FIG. 6A, using alignment filter) may be used to enable a user to control the field of view of an imaging instrument through manipulation of the input device. Alternatively, the non- view-based steering (third row of illustrations in FIG. 6A, bypassing alignment filter) may be used to control other instruments. A switch as shown in FIG. 5B make the alignment filter selectively activable, thereby enabling switching between control behaviors.

[0103] FIG. 6B shows the alignment filter 520 in accordance with embodiments of the disclosure. The desired motion Xdes at the input of the alignment filter 520, in some embodiments, is provided in the form of values describing position and orientation of the input device. Sets of these values may be provided at a fixed rate, and a set of values received by the alignment filter 520 may represent the position and orientation of the input device at a point in time. The alignment filter may use Xdes to forward-propagate the currentaligned desired motion (including, for example, the current position and orientation of the field of view) at time t=T to determine the aligned desired motion at time t=T+. In other words, the spatial configuration of the field of view, represented by the aligned desired motion Xstate is incrementally forward-propagated in time. This operation may be performed for translations and for rotations of the field of view and enables the use of the aligned desired motion Xstate at the output of the alignment filter 520 as the input to the robotic controller 510. Additional operations such as transformations into one or more different reference frames may be performed to provide the aligned desired motion in a reference frame suitable as an input to the robotic controller 510. When these operations by the alignment filter 520 are performed at a sufficiently high rate, the path-dependent behavior as shown in the second row of illustrations in FIG. 6A is obtained. A user may, thus, be able to smoothly control the field of view of the imaging device in a manner that is intuitive when steering the imaging instrument while relying on visual feedback in the form of images provided by the imaging instrument. A more detailed description specific to the alignment filter 520 as shown in FIG. 6B is subsequently provided.

[0104] The incremental updating of the field of view relative to the current field of view is enabled by an incremental processing of the desired motion Xdes in the form of a motion increment (further discussed below) received at the input of the alignment filter 520, and results in the generation of the desired aligned motion Xstate at the output of the alignment filter 520. The incremental processing comprises the three steps of (1) forward-propagating a positional component dstate in Xstate to obtain the forward-propagated positional component Estate '■> (2) forward-propagating an orientational component Rstate in Xstate to obtain the forward-propagated positional component Rgtate', and (3) combining d^tateand d^tateto form Xstate which is provided to the robotic controller 510. In other words, for each motion increment of Xdes, Xstate is forward-propagated using the motion increment. Each of these steps is subsequently described.

[0105] The forward-propagation of dstate may be performed in multiple sub-steps. First, ddes is obtained from Xdes received at the input of the alignment filter 520. Xdes may be provided in a six-dimensional space (three positions, three orientations), and obtaining ddes may involve separating the positional components from the orientational components in Xdes. A positional motion increment A ddes may be obtained by determining the difference (e.g., using subtraction) between ddes obtained at the current time steps and d~lesobtained at the previous time step of the incremental execution of the alignment filter 520. Next, thepositional motion increment Addes is rotated to align Addes with the current field of view. The alignment, thus, compensates for a current orientational offset between the input device and the field of view of the imaging device. A detailed description of the alignment is provided below based on the examples shown in FIG. 6C. The aligned positional motion increment is subsequently combined (e.g., using addition) with dstate to obtain dtate.

[0106] The forward-propagation of Rstate may be performed analogous to the forwardpropagation of dstate- First, Rdes is obtained from Xdes received at the input of the alignment filter 520. Obtaining Rdes may involve separating the orientational components from the positional components in Xdes- A orientational motion increment ARdes may be obtained by determining the difference (e.g., by converting the two orientations to rotation matrices, multiplication, and extraction of the orientation difference from the matrix after the multiplication) between Rdes obtained at the current time steps and Resobtained at the previous time step of the incremental execution of the alignment filter 520. Next, the orientational motion increment ARdes is rotated to align ARdes with the current field of view. The alignment, thus, compensates for the current orientational offset between the input device and the field of view of the imaging device. The aligned orientational motion increment is subsequently applied to Rstate to obtain Rstate.

[0107] The forward-propagated components d^tateand Rstatemay subsequently be combined to form the desired motion Xstate- xstate may be obtained from d^tateand Rotate using vector concatenation and may be used to drive the robotic controller 510.

[0108] FIG. 6C provides an illustration of the operations performed by the alignment filter, previously discussed in reference to FIG. 6B. The top illustration 600 schematically shows an input device (left panel) and a field of view of an imaging device (right panel). For the sake of simplicity, assume that the solid horizontal line of the input device in the illustration 600 corresponds to the handle 302 of the input portion 300 in FIG. 3. By operating the handle 302 of the input device using a single hand, a user can change the field of view of the imaging device, driven by the combination of the alignment filter 520 and the robotic controller 510. In the illustration 600, the current spatial configurations of the input device and the field of view are indicated using solid lines, and the future spatial configurations (e.g., one time step into the future) are indicated using dashed lines. In the example, the handle is positionally displaced by Addes- The direction of the motion increment is registered against a vertical reference (dotted line). More generally, the motion increment, including the direction of the motion increment may be registered in an input device baseframe. In the example, Addes is perpendicular to the vertical reference, i.e., Addes is in a purely horizontal direction. While a vertical reference is used to establish the direction of Addes, any other reference may be used to determine the direction of Addes- The reference is assumed to be stationary in, for example, a world reference frame, an input device base frame, etc. The reference is used for illustrative purposes. A dedicated reference does not necessarily exist in an actual implementation. Instead, direction may be determined in the corresponding reference frames. Here, the direction of Addes may be determined in the input device base frame or another reference frame.

[0109] Referring to the right panel of the illustration 600, the field of view is shown as rotated by 45° relative to the input device. Accordingly, the reference (dotted line) used to identify the orientation of the field of view is also rotated by 45° relative to the corresponding reference associated with the input device. As a result of the rotational offset between the two references, a rotation is applied to Addes such that the alignment of the rotated Addes relative to the reference associated with the field of view is identical to the alignment of Addes relative to the reference associated with the input device. In other words, the direction of Addes (after performing the alignment) in the view frame is identical to the direction of Addes in the input device base frame. The aligned Addes is subsequently used to cause movement of the view frame from the current spatial configuration (solid lines) to the future spatial configuration (dashed lines).

[0110] While the illustration 600 describes the operation of the alignment filter for a particular positional component (a horizontal movement at the input device, the same operations may be performed for positional components in any direction or combinations of directions in all three positional degrees of freedom. Furthermore, operations of the alignment filter for orientational components may be performed in a similar manner.

[0111] Due to the incremental operation of the alignment filter, an alignment relationship (position, orientation) between the spatial configuration of the input device and the spatial configuration of the field of view needs to be established to allow the incremental operation of the alignment filter starting from this spatial relationship. The alignment relationship may be used by the alignment filter 520 when forward propagating xstate using Xdes- Specifically, the rotation of the Addes and ARdes is performed using the alignment relationship.

[0112] Alignment relationships may be defined in different manners. For example, to accomplish the control behavior described in the second row of FIG. 6A, an alignmentrelationship that includes a one-to-one mapping between degrees of freedom of the input device and respective degrees of freedom of the field of view may be defined. The alignment relationship may be defined in any number of degrees of freedom, e.g., in all six degrees of freedom. Certain reference frames may be used to establish the alignment relationship. For example, the degrees of freedom of the input device may be described in the input device base frame, whereas the degrees of freedom of the field of view may be described in the view frame.

[0113] Other alignment relationships, e.g., a second alignment relationship, may be used, though. For example, to accomplish the control behavior described in the third row of FIG. 6A, a second one-to-one mapping between degrees of freedom of the input device and respective degrees of freedom of the field of view may be used, in which the degrees of freedom of the input device may be described in the input device base frame, whereas the degrees of freedom of the field of view may be described in the manipulator structure base frame. Such a relationship may be applied to any number of degrees of freedom, e.g., four, five, or six degrees of freedom. Other alignment relationships may further be used in conjunction with a second instrument different from the imaging instrument. In such an alignment relationship, a one-to-one mapping between degrees of freedom of the input device and respective degrees of freedom of the second instrument may be used, in which both the degrees of freedom of the input device and the degrees of freedom of the second instrument are described in the view frame. Such an alignment relationship may be applied to any number of degrees of freedom. Referring to FIG. 2A, the alignment relationship may be used to control the instruments 260 and 270. The alignment relationship may further be used to control other (secondary) imaging instruments such as ultrasound probes, additional endoscopes, etc.

[0114] The illustration 600 shows a particular alignment relationship between the input device and the field of view (the input device shown as horizontally oriented, and the field of view shown as tilted at a 45° angle).

[0115] In one or more embodiments, such an alignment relationship is established through a latching of the input device onto the view frame. The latching may be triggered by certain events. For example, the latching may be triggered whenever entering the imaging instrument movement mode to control the field of view by user input provided at the input device. The user may signal the intention to enter the imaging instrument movement mode by operating a control element such as a button, a foot pedal, a software switch, etc. As partof the latching, a spatial configuration of the input device and a spatial configuration of the field of view is obtained at the entry into the imaging movement mode. The spatial configuration of the input device may be obtained in the input device base frame, whereas the spatial configuration of the field of view may be obtained in the manipulator structure base frame. The obtained spatial configurations relative to one another establish the alignment relationship.

[0116] After the entry into the imaging instrument movement mode, the alignment relationship may be held static, even as the field of view moves (including translation and / or rotation) in response to the desired motion that reflects the user input at the input device. The alignment relationship may be held static until an exit from the imaging instrument movement mode. For example, referring to FIG. 6A, the three user inputs to move the field of view with desired motions may result in three motion commands, each in accordance with the alignment relationship. An exit of the imaging instrument movement mode may be user- instructed and may occur for any reason, for example, when the user decides to control a different instrument, decides to take a break from controlling the imaging instrument to move the view frame, etc.

[0117] After the exit from the imaging instrument movement mode, another entry into the imaging instrument movement mode may be detected, e.g., when the user chooses to resume moving the view frame. This may trigger the establishment of another alignment relationship, which is independent from the previously used alignment relationship. The resulting motion commands after the renewed entry into the imaging instrument movement as the user continues to provide input to move the field of is then in accordance with the newly obtained alignment relationship.

[0118] As a result of the independence of the alignment relationships before and after exit and re-entry of the imaging instrument movement mode, the user may use the exit and re-entry to re-position and / or re-orient the input device as desired relative to the field of view. This may enable the user to change the hand posture for more ergonomic operation, to re- center the input device when close to the edge of the input device’s workspace, etc.

[0119] The illustration 610 in comparison to the illustration 600 shows the result of a changed alignment relationship. While the orientation of the field of view in illustration 610 is as shown in illustration 600, the orientation of the input device is different, as a result of the input device being latched onto the view frame in a different configuration. As a result, the same Addes as the input device causes the same movement of the field of view, despite thechanged orientation of the input device. While not illustrated, the alignment relationship may implement additional mapping characteristics such as a scaling, without departing from the disclosure.

[0120] As discussed in reference to FIGs. 5A and 5B, the aligned desired motion generated by the alignment filter 520 serves as an input to the robotic controller 510. In the previously introduced implementation that relies on a first kinematic controller and a second kinematic controller, the aligned desired motion of the alignment filter may be provided as input to both the first kinematic controller and the second kinematic controller. While the first kinematic controller may receive the aligned desired motion in all six degrees of freedom, the second kinematic controller may receive the aligned motion in the three rotational degrees of freedom only. In addition, the aligned desired motion may undergo one or more transformations between reference frames, prior to being provided to the first and second kinematic controllers, and the aligned desired motion may be provided to the first and second kinematic controllers in different reference frames.

[0121] The operations described in reference to FIGs. 6 A, 6B, and 6C may be performed by any type of the previously described manipulator assemblies and other manipulator assemblies. Specifically referring to the manipulator assembly 200 of FIG. 2A, if such a manipulator assembly is used, the motion command may include a commanded roll component of the manipulator-supporting link 202 when the desired motion includes a desired roll component of the field of view about the view axis of the field of view, thereby enabling execution of the commanded roll component, even when the imaging instrument lacks any joints operable to provide the desired roll component of the field of view.

[0122] In such a configuration, the roll command component of the motion command may be provided in the manipulator structure base frame, whereas all other command components (i.e., the non-roll command components) that are a result of may be provided in the view frame. Assuming, for example, that the desired motion includes a desired non-roll component with a rotation about an axis perpendicular to the view axis, the non-rollcommand component of the motion command is in the view frame, whereas the roll command component is in the manipulator structure base frame.

[0123] Because, as previously described in reference to FIG. 2A, roll of the manipulator- supporting link, without further compensation, would result in a potentially undesired movement of another instrument (or multiple instruments) supported by the manipulator- supporting link, an additional motion command for the instrument joints of theother instrument(s) may be determined such that the undesired movement resulting from the roll of the manipulator-supporting link is compensated for. The compensation may be such that a position or orientation of a distal portion of the other instrument(s) is maintained during the performance of the commanded roll component. Alternatively, the compensation may be such that a position or orientation of the distal portion of the other instrument(s) is restored after the performance of the commanded roll component.

[0124] Turning to FIG. 7, a flowchart in accordance with one or more embodiments is shown. The flowchart of FIG. 7 depicts a method 700 for computer-assisted systems. The method 700 may be used for steering an imaging instrument. One or more of the steps in FIG. 7 may be performed by various components of systems, previously described with reference to FIGs. 1A, IB, 2A, 2B, 3, 4, and 5A-5C. While these figures illustrate particular configurations of computer assisted systems, the method is equally applicable to other configurations. On a high level, the described steps may be performed for a computer assisted system that includes a manipulator structure with a plurality of manipulator links that are coupled by a plurality of manipulator joints. The manipulator structure may support an imaging instrument that includes a plurality of instrument links coupled by at least one instrument joint. The imaging instrument has a field of view that defines a view frame.When the manipulator structure is supporting the imaging instrument, the manipulator structure and the imaging instrument together comprise an aggregate plurality of links coupled by an aggregate plurality of joints, the aggregate plurality of links comprising the plurality of manipulator links and the plurality of instrument links, and the aggregate plurality of joints comprising the plurality of manipulator joints and the at least one instrument joint.

[0125] The method may be executed on one or more processors, e.g., of the computing system(s) 172 of the computer-assisted system.

[0126] While the various steps in the flowchart are presented and described sequentially, one of ordinary skill will appreciate that some or all of the steps may be executed in different orders, may be combined or omitted, and some or all of the steps may be executed in parallel. Additional steps may further be performed. Furthermore, the steps may be performed actively or passively. For example, some steps may be performed using polling or be interrupt driven in accordance with one or more embodiments of the disclosure.

[0127] While not shown, the operations of the method 700 may be executed in a loop, thereby enabling the incremental execution of the alignment filter, as previously described. The loop may be executed in real-time and at a frequency sufficient to enable a user tooperate the systems as described with little to no perceivable delay. As a result, a user receiving visual content based on the field of view of the imaging instrument may get the impression that movement of the visual content is directly controllable by their user input. For example, a user input to the right will always cause a shift of the visual content as seen by the user, e.g., through display unit 154, based on the behavior illustrated in the center row of FIG. 6A. A positional correspondence is thus continuously maintained, unlike in joystick control methods where the correspondence is in the velocity domain. At least some steps of the method 700 may be executed while in an imaging instrument movement mode. Other steps may be executed when not in the imaging instrument movement mode. Furthermore, additional operations that are not shown in the flowchart may be performed when in the imaging instrument movement mode, without departing from the disclosure.

[0128] Turning to the method 700, in Block 702, a user input to move the field of view with a desired motion is received from an input device, as previously described.

[0129] In Block 704, a motion command is determined based on the desired motion. The motion command is determined such that it moves the field of view in accordance with an alignment relationship. The alignment relationship includes a one-to-one mapping between first, second, and third translational degrees of freedom of the input device in an input device base frame and respective first, second, and third translational degrees of freedom of the field of view in the view frame. The alignment relationship further includes a one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base frame and respective first, second, and third rotational degrees of freedom of the field of view in the view frame. The determining of the motion command may be performed as previously described in reference to FIGs. 5 A, 5B, 5C, 6A, 6B, and 6C.

[0130] In Block 706, the aggregate plurality of joints (i.e., the joints of the manipulator structure and the joints of the imaging instrument) is commanded based on the motion command.

[0131] With the execution of the motion command, the spatial position and / or orientation of the imaging instrument may adjust such that the field of view moves in accordance with the user input.

[0132] While the execution of blocks 702-706 is described for a first imaging instrument movement mode that relies on a first alignment relationship, blocks 702-706 may also be performed for a second imaging instrument movement mode that relies on a secondalignment relationship that is independent from the first alignment relationship. Furthermore, operations for moving additional instruments (different from the imaging instrument) may be performed, either simultaneously or non-simultaneously with the execution of blocks 702- 706.

[0133] Furthermore, in some embodiments, haptic feedback may be provided to the user operating the input device to move the field of view. The haptic feedback may be designed to inform the user of motion limitations associated with moving the field of view. Such constraints may be hardware or physical constraints (e.g., collisions of the imaging instrument with objects in the environment or range of motion limitations), or software limitations. To ensure that the feedback provided to the user is informative and easy to interpret, the alignment relationship used for the control of the field of view is also applied to determine the feedback command for providing haptic feedback to the user at the input device in presence of the motion limitation. The haptic feedback may, thus, indicate the inhibiting of the continuation of the movement towards the motion limitation to the user. For example, a virtual wall may be implemented by the haptic feedback, thereby preventing or at least inhibiting the user from continuing commanding a movement of the imaging device.

[0134] Although only a few example embodiments have been described in detail above, those skilled in the art will readily appreciate that many modifications are possible in the example embodiments without materially departing from this invention. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the following claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-assisted system comprising: an input device configured to be manipulated by a user; a manipulator structure comprising a plurality of manipulator links extending from a proximal base, the plurality of manipulator links coupled by a plurality of manipulator joints, the manipulator structure configured to support an imaging instrument comprising a plurality of instrument links coupled by at least one instrument joint, the imaging instrument having a field of view that defines a view frame; wherein when the manipulator structure is supporting the imaging instrument, the manipulator structure and the imaging instrument together comprise an aggregate plurality of joints, the aggregate plurality of joints comprising the plurality of manipulator joints and the at least one instrument joint; and a control system comprising one or more processors, the control system configured to, in an imaging instrument movement mode: receive, from the input device, a user input to move the field of view with a desired motion; determine, based on the desired motion, a motion command that moves the field of view in accordance with a first alignment relationship, wherein the first alignment relationship establishes: a first one-to-one mapping between first, second, and third translational degrees of freedom of the input device in an input device base frame and respective first, second, and third translational degrees of freedom of the field of view in the view frame, and a first one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base frame and respective first, second, and third rotational degrees of freedom of the field of view in the view frame; and command the aggregate plurality of joints based on the motion command.

2. The computer-assisted system of claim 1, wherein, when observed in a manipulator structure base frame fixed relative to the proximal base, the motion command is determined further based on a current spatial configuration of the field of view reflecting a motion history of the field of view.

3. The computer-assisted system of claim 2, wherein the motion history begins with an entry into the imaging instrument movement mode.

4. The computer-assisted system of claim 2, wherein, when observed in the view frame, the current spatial configuration of the field of view is independent from the motion history.

5. The computer-assisted system of claim 1, wherein to determine the motion command, the control system is configured to: for each motion increment of the desired motion, forward-propagate an aligned desired motion using the motion increment; and generate the motion command using the aligned desired motion.

6. The computer-assisted system of claim 5, wherein to forward-propagate the aligned desired motion using the motion increment, the control system is configured to: rotate the motion increment according to the first alignment relationship; and combine the aligned desired motion with the rotated motion increment.

7. The computer-assisted system of claim 5, wherein the control system comprises a selectively activable alignment filter, and wherein to forward-propagate the aligned desired motion using the motion increment, the alignment filter is active.

8. The computer-assisted system of claim 1, wherein the control system is further configured to: determine the first alignment relationship at a first entry into the imaging instrument movement mode.

9. The computer-assisted system of claim 8, wherein the control system is further configured to: hold the first alignment relationship static even as the field of view moves.

10. The computer-assisted system of claim 9, wherein to hold the alignment relationship static even as the field of view moves, the control system is configured to: hold the first alignment relationship static even as the field of view rotates.

11. The computer-assisted system of claim 8, wherein the control system is further configured to: hold, from the first entry into the imaging instrument movement mode until a first exit from the imaging instrument movement mode, the first alignment relationship static even as the field of view rotates.

12. The computer-assisted system of claim 8, wherein the user input is a first user input, the desired motion is first desired motion, wherein the motion command is a first motion command, and wherein the control system is further configured to: receive, from the input device and after the first desired motion has been received, a second user input to move the field of view with a second desired motion; determine, based on the second desired motion, a second motion command that moves the field of view in accordance with the first alignment relationship; and command the aggregate plurality of joints based on the second motion command.

13. The computer-assisted system of claim 8, wherein the control system is further configured to: determine, at a second entry into the imaging instrument movement mode and independently of the first alignment relationship, a second alignment relationship, wherein the second entry occurs after the first entry.

14. The computer-assisted system of claim 8, wherein to determine the first alignment relationship, the control system is configured to: obtain, at the first entry, a spatial configuration of the input device; obtain, at the first entry, a spatial configuration of the field of view; and determine, based on the spatial configuration of the input device and the spatial configuration of the field of view, the first alignment relationship.

15. The computer-assisted system of claim 8, wherein to determine the first alignment relationship, the control system is configured to: identify a spatial relationship between the spatial configuration of the input device in the input device base frame and the spatial configuration of the field of view in a base frame, the base frame fixed relative to the proximal base.

16. The computer-assisted system of claim 1, wherein the imaging instrument movement mode is a first imaging instrument movement mode, the user input is a first user input, the desired motion is a first desired motion, the motion command is a first motion command, and wherein the control system is further configured to, in a second imaging instrument movement mode: receive, from the input device, a second user input to move the field of view with a second desired motion; determine, based on the second desired motion, a second motion command that moves the field of view in accordance with a second alignment relationship; wherein the second alignment relationship does not establish: a one-to one mapping between first, second, and third translational degrees of freedom of the input device in the input device base frame and respective first, second, and third translational degrees of freedom of the field of view in the view frame, and a one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base frame and respective first, second, and third rotational degrees of freedom of the field of view in the view frame.

17. The computer-assisted system of claim 16, wherein the second alignment relationship establishes: a second one-to one mapping between a first translational degree of freedom of the input device in the input device base frame and a first translational degree of freedom of the field of view in the base frame; and a second one-to-one mapping between a first rotational degree of freedom of the input device in the input device base frame and a first rotational degree of freedom of the field of view in the base frame.

18. The computer-assisted system of any of claims 1 to 17, wherein the manipulator structure comprises a manipulator- supporting link having a roll axis; wherein the desired motion comprises a desired roll component of the field of view about a view axis of the field of view; and wherein the motion command comprises a roll command component of the manipulator- supporting link about the roll axis.

19. The computer-assisted system of claim 18, further comprising the imaging instrument, wherein the imaging instrument lacks any joints operable to provide the desired roll component of the field of view.

20. The computer-assisted system of claim 18, wherein when the desired motion comprises a desired roll component comprising a rotation about the view axis and a desired non-roll component comprising a rotation about an axis perpendicular to the view axis, and wherein the motion command comprises: a roll command component in a base frame fixed relative to the proximal base, the roll command component being based on the desired roll component; and a non-roll command component in the view frame, the non-roll command component based on the desired non-roll component.

21. The computer-assisted system of claim 20, wherein the motion command is a first motion command, and the imaging instrument is a first instrument; wherein the manipulator structure is further configured to support a second instrument, the second instrument comprising a plurality of second instrument links coupled by one or more second instrument joints; and wherein the control system is further configured to: determine a second motion command, for the one or more second instrument joints, to maintain a position or orientation of a distal portion of the second instrument during a performance of the roll command component or to restore the position or orientation of the distal portion of the second instrument after the performance of the roll command component; and command the one or more second instrument joints based on the second motion command.

22. The computer-assisted system of any of claims 1 to 17, wherein the user input is a first user input, the desired motion is a first desired motion, the motion command is a first motion command, the imaging instrument is a first instrument, and the imaging instrument movement mode is a first instrument movement mode; and wherein the control system is further configured to, in a second instrument movement mode: receive, from the input device, a second user input to move a second instrument with a second desired motion; and determine, based on the second desired motion, a second motion command that moves the second instrument in accordance with a second alignment relationship; wherein the second alignment relationship establishes: a second one-to-one mapping between the first, second, and third translational degrees of freedom of the input device in the view frame and respective first, second, and third translationaldegrees of freedom of the second instrument in the view frame, and a second one-to-one mapping between the first, second, and third rotational degrees of freedom of the input device in the view frame and respective first, second, and third rotational degrees of freedom of the second instrument in the view frame.

23. The computer-assisted system of any of claims 1 to 17, wherein the input device is configured for operation by a single hand of the user.

24. The computer-assisted system of any of claims 1 to 17, wherein the control system is further configured to, in the imaging instrument movement mode: detect a motion limitation experienced by the imaging instrument when performing the motion command; and determine, based on the motion limitation and in accordance with the first alignment relationship, a feedback command for providing haptic feedback to the user by the input device.

5. A method for view-based steering of an imaging instrument associated with a computer-assisted system, wherein the computer-assisted comprises: an input device configured to be manipulated by a user; a manipulator structure comprising a plurality of manipulator links extending from a proximal base, the plurality of manipulator links coupled by a plurality of manipulator joints, the manipulator structure configured to support the imaging instrument comprising a plurality of instrument links coupled by at least one instrument joint, the imaging instrument having a field of view that defines a view frame; wherein when the manipulator structure is supporting the imaging instrument, the manipulator structure and the imaging instrument together comprise an aggregate plurality of joints, the aggregate plurality of joints comprising the plurality of manipulator joints and the at least one instrument joint; and a control system; wherein the method, executed by the control system, comprises: when the control system is configured to operate in an imaging instrument movement mode: receiving, from the input device, a user input to move the field of view with a desired motion; determining, based on the desired motion, a motion command that moves the field of view in accordance with a first alignment relationship, wherein the first alignment relationship establishes: a first one-to-one mapping between first, second, and third translational degrees of freedom of the input device in an input device base frame and respective first, second, and third translational degrees of freedom of the field of view in the view frame, and a first one-to-one mapping between first, second, and third rotational degrees of freedom of the inputdevice in the input device base frame and respective first, second, and third rotational degrees of freedom of the field of view in the view frame; and commanding the aggregate plurality of joints based on the motion command.

26. The method of claim 25, wherein, when observed in a manipulator structure base frame fixed relative to the proximal base, the motion command is determined further based on a current spatial configuration of the field of view reflecting a motion history of the field of view.

27. The method of claim 26, wherein the motion history begins with an entry into the imaging instrument movement mode.

28. The method of claim 26, wherein, when observed in the view frame, the current spatial configuration of the field of view is independent from the motion history.

29. The method of claim 25, further comprising, to determine the motion command: for each motion increment of the desired motion, forward-propagating an aligned desired motion using the motion increment; and generate the motion command using the aligned desired motion.

30. The method of claim 29, further comprising, to forward-propagate the aligned desired motion using the motion increment: rotating the motion increment according to the first alignment relationship; and combining the aligned desired motion with the rotated motion increment.

31. The method of claim 25, further comprising: determining the first alignment relationship at a first entry into the imaging instrument movement mode.

32. The method of claim 31, further comprising: holding the first alignment relationship static even as the field of view moves.

33. The method of claim 32, further comprising, to hold the alignment relationship static even as the field of view moves: holding the first alignment relationship static even as the field of view rotates.

34. The method of claim 31, further comprising: holding, from the first entry into the imaging instrument movement mode until a first exit from the imaging instrument movement mode, the first alignment relationship static even as the field of view rotates.

35. The method of claim 31, wherein the user input is a first user input, the desired motion is first desired motion, wherein the motion command is a first motion command, and wherein the method further comprises: receiving, from the input device and after the first desired motion has been received, a second user input to move the field of view with a second desired motion; determining, based on the second desired motion, a second motion command that moves the field of view in accordance with the first alignment relationship; and commanding the aggregate plurality of joints based on the second motion command.

36. The method of claim 31, further comprising: determining, at a second entry into the imaging instrument movement mode and independently of the first alignment relationship, a second alignment relationship, wherein the second entry occurs after the first entry.

37. The method of claim 31, wherein the method further comprises, to determine the first alignment relationship: obtaining, at the first entry, a spatial configuration of the input device; obtaining, at the first entry, a spatial configuration of the field of view; anddetermining, based on the spatial configuration of the input device and the spatial configuration of the field of view, the first alignment relationship.

38. The method of claim 31, wherein the method further comprises, to determine the first alignment relationship: identifying a spatial relationship between the spatial configuration of the input device in the input device base frame and the spatial configuration of the field of view in a base frame, the base frame fixed relative to the proximal base.

39. The method of claim 25, wherein the imaging instrument movement mode is a first imaging instrument movement mode, the user input is a first user input, the desired motion is a first desired motion, the motion command is a first motion command, and wherein the method further comprises, in a second imaging instrument movement mode: receiving, from the input device, a second user input to move the field of view with a second desired motion; determining, based on the second desired motion, a second motion command that moves the field of view in accordance with a second alignment relationship; wherein the second alignment relationship does not establish: a one-to one mapping between first, second, and third translational degrees of freedom of the input device in the input device base frame and respective first, second, and third translational degrees of freedom of the field of view in the view frame, and a one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base frame and respective first, second, and third rotational degrees of freedom of the field of view in the view frame.

40. The method of claim 39, wherein the second alignment relationship establishes: a second one-to one mapping between a first translational degree of freedom of the input device in the input device base frame and a first translational degree of freedom of the field of view in the base frame; anda second one-to-one mapping between a first rotational degree of freedom of the input device in the input device base frame and a first rotational degree of freedom of the field of view in the base frame.

41. The method of any of claims 25 to 40, wherein the manipulator structure comprises a manipulator- supporting link having a roll axis; wherein the desired motion comprises a desired roll component of the field of view about a view axis of the field of view; and wherein the motion command comprises a roll command component of the manipulator- supporting link about the roll axis.

42. The method of claim 41, further comprising the imaging instrument, wherein the imaging instrument lacks any joints operable to provide the desired roll component of the field of view.

43. The method of claim 41, wherein when the desired motion comprises a desired roll component comprising a rotation about the view axis and a desired non-roll component comprising a rotation about an axis perpendicular to the view axis, and wherein the motion command comprises: a roll command component in a base frame fixed relative to the proximal base, the roll command component being based on the desired roll component; and a non-roll command component in the view frame, the non-roll command component based on the desired non-roll component.

44. The method of claim 43, wherein the motion command is a first motion command, and the imaging instrument is a first instrument; wherein the manipulator structure is further configured to support a second instrument, the second instrument comprising a plurality of second instrument links coupled by one or more second instrument joints; and wherein the method further comprises:determining a second motion command, for the one or more second instrument joints, to maintain a position or orientation of a distal portion of the second instrument during a performance of the roll command component or to restore the position or orientation of the distal portion of the second instrument after the performance of the roll command component; and commanding the one or more second instrument joints based on the second motion command.

45. The method of any of claims 25 to 40, wherein the user input is a first user input, the desired motion is a first desired motion, the motion command is a first motion command, the imaging instrument is a first instrument, and the imaging instrument movement mode is a first instrument movement mode; and wherein the method further comprises, in a second instrument movement mode: receiving, from the input device, a second user input to move a second instrument with a second desired motion; and determining, based on the second desired motion, a second motion command that moves the second instrument in accordance with a second alignment relationship; wherein the second alignment relationship establishes: a second one-to-one mapping between the first, second, and third translational degrees of freedom of the input device in the view frame and respective first, second, and third translational degrees of freedom of the second instrument in the view frame, and a second one-to-one mapping between the first, second, and third rotational degrees of freedom of the input device in the view frame and respective first, second, and third rotational degrees of freedom of the second instrument in the view frame.

46. The method of any of claims 25 to 40, wherein the input device is configured for operation by a single hand of the user.

47. The method of any of claims 25 to 40, wherein the method further comprises, in the imaging instrument movement mode: detecting a motion limitation experienced by the imaging instrument when performing the motion command; and determining, based on the motion limitation and in accordance with the first alignment relationship, a feedback command for providing haptic feedback to the user by the input device.

48. A non-transitory machine-readable medium comprising a plurality of machine- readable instructions executed by one or more processors associated with a computer- assisted system, the plurality of machine-readable instructions causing the one or more processors to perform the method of any of claims 25 to 47.

Images