View content-based steering of imaging instruments

Abstract

A computer-assisted system includes a control system that receives, from an input device, a user input to move an image of a region in the field of view of an imaging device with a desired motion, and determines, based on the desired motion, a motion command that moves the imaging instrument such that the image of the region in the field of view moves in accordance with an alignment relationship. The alignment relationship establishes a one- to-one mapping between first, second, and third translational and rotational degrees of freedom of the input device in an input device base reference frame and respective first, second, and third translational and rotational degrees of freedom of the image of the region in the view reference frame. The control system further commands joints of the imaging device and joints of a manipulator structure that supports the imaging instrument based on the motion command.

Description

VIEW CONTENT-BASED STEERING OF IMAGING INSTRUMENTSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of U.S. Provisional Patent Application 63 / 704,673, filed on October 8, 2024, which are hereby incorporated by reference herein in their entirety.BACKGROUNDField of Invention

[0002] The present invention generally provides improved computer-assisted devices, systems, and methods.Overview

[0003] Computer-assisted systems can be used to perform a task at a worksite. For example, a computer-assisted system may comprise a handheld, motorized tool assembly. As another example, a computer-assisted system may comprise a robotic system, and may include one or more robotic manipulators to manipulate instruments for performing the task.

[0004] Example computer-assisted systems include industrial and recreational robotic systems. Example computer-assisted systems also include medical robotic systems used in procedures for diagnosis, non-surgical treatment, surgical treatment, etc.

[0005] Some computer-assisted systems include one or more instruments that are articulated in order to perform various procedures. The computer-assisted system can be automated, semi-automated, teleoperated, etc. In a teleoperated example, a human operator manipulates one or more leader input controls to command motion of one or more follower instruments located in a workspace. In some examples, the teleoperated system is configured to support an instrument that includes an imaging device, such as an endoscope or a camera, which enables the operator to observe the workspace. In some instances, the field of view of the imaging such an imaging instrument is directed to enable the operator to see the instrument s) as the operator commands motion of the instrum ent(s).

[0006] In some embodiments, the imaging instrument is disposed on or otherwise supported by a manipulator structure. The imaging instrument may be movable by changing the configuration of links and joints of the manipulator structure and / or the imaging instrument. In one or more embodiments, the operator may use an input device to provide a user input to move the imaging instrument. Depending on how a motion command formoving the imaging instrument is generated based on the user input, the resulting motion may be more or less intuitive to the operator. Thus, it is desirable to command movement of the imaging instrument using a more intuitive scheme, to facilitate operator control.SUMMARY

[0007] In general, in one aspect, one or more embodiments relate to a computer- assisted system comprising: an input device configured to be manipulated by a user; a display device configured to display images, the images being of a field of view of an imaging instrument, the field of view defining a view reference frame; a manipulator structure comprising a plurality of manipulator links extending from a proximal base, the plurality of manipulator links coupled by a plurality of manipulator joints, the manipulator structure configured to support the imaging instrument, the imaging instrument comprising a plurality of instrument links coupled by at least one instrument joint, wherein when the manipulator structure is supporting the imaging instrument, the manipulator structure and the imaging instrument together comprise an aggregate plurality of joints, the aggregate plurality of joints comprising the plurality of manipulator joints and the at least one instrument joint; and a control system comprising one or more processors, the control system configured to, in an imaging instrument movement mode: receive, from the input device, a user input to move an image of a region in the field of view with a desired motion; determine, based on the desired motion, a motion command that moves the imaging instrument such that the image of the region in the field of view moves in accordance with a first alignment relationship, wherein the first alignment relationship establishes: a first one-to-one mapping between first, second, and third translational degrees of freedom of the input device in an input device base reference frame and respective first, second, and third translational degrees of freedom of the image of the region in the view reference frame, and a first one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third rotational degrees of freedom of the image of the region in the view reference frame; and command the aggregate plurality of joints based on the motion command.

[0008] In general, in one aspect, one or more embodiments relate to a method for view content-based steering of an imaging instrument associated with a computer-assisted system, wherein the computer-assisted comprises: an input device configured to be manipulated by a user; a display device configured to display images, the images being of a field of view of the imaging instrument, the field of view defining a view reference frame; amanipulator structure comprising a plurality of manipulator links extending from a proximal base, the plurality of manipulator links coupled by a plurality of manipulator joints, the manipulator structure configured to support the imaging instrument, the imaging instrument comprising a plurality of instrument links coupled by at least one instrument joint, wherein when the manipulator structure is supporting the imaging instrument, the manipulator structure and the imaging instrument together comprise an aggregate plurality of joints, the aggregate plurality of joints comprising the plurality of manipulator joints and the at least one instrument joint; and a control system; wherein the method, executed by the control system, comprises: when the control system is configured to operate in an imaging instrument movement mode: receiving, from the input device, a user input to move an image of a region in the field of view with a desired motion; determining, based on the desired motion, a motion command that moves the imaging instrument such that the image of the region in the field of view moves in accordance with a first alignment relationship, wherein the first alignment relationship establishes: a first one-to-one mapping between first, second, and third translational degrees of freedom of the input device in an input device base reference frame and respective first, second, and third translational degrees of freedom of the image of the region in the view reference frame, and a first one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third rotational degrees of freedom of the image of the region in the view reference frame; and commanding the aggregate plurality of joints based on the motion command.

[0009] Other aspects of the invention will be apparent from the following description and the appended claims.

[0010] Benefits of the disclosed technique include, and are not limited to, reducing the operator’s cognitive load, increasing an operator’s accuracy in steering the imaging instrument, increasing the efficiency for the use of the computer-assisted systems, reducing the time needed to perform procedures with the computer-assisted system, increasing the speed and ease in learning to use the computer-assisted system, and the like.BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures may be represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

[0012] FIG. 1 A shows an example computer-assisted system in accordance with one or more embodiments.

[0013] FIG. IB shows an example computer-assisted system in accordance with one or more embodiments.

[0014] FIG. 2A shows elements of a manipulator assembly of a computer-assisted system in accordance with one or more embodiments.

[0015] FIG. 2B shows a reconfiguration of elements of a manipulator assembly in accordance with one or more embodiments.

[0016] FIG. 3 shows an input portion of an example input device in accordance with one or more embodiments.

[0017] FIG. 4 illustrates elements and operation of a manipulator assembly supporting an imaging instrument in accordance with one or more embodiments.

[0018] FIGs. 5A and 5B show control schemes of a computer assisted system in accordance with embodiments of the disclosure.

[0019] FIG. 5C shows a controller comprising one or more processors, in accordance with embodiments of the disclosure.

[0020] FIG. 6A schematically illustrates a view content-based steering of an imaging instrument in accordance with embodiments of the disclosure.

[0021] FIG. 6B shows an implementation of an imaging instrument control filter in accordance with embodiments of the disclosure.

[0022] FIG. 7 shows a flowchart describing an example method in accordance with one or more embodiments.DETAILED DESCRIPTION

[0023] Specific embodiments of the disclosure will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.

[0024] In the following detailed description of embodiments of the disclosure, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0025] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not to imply or create any particular ordering of the elements, and is not to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before”, “after”, “single”, and other such terminology. Rather, the use of ordinal numbers is to distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

[0026] This disclosure describes various devices, elements, and portions of computer- assisted systems and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element (e.g., three degrees of translational freedom in a three-dimensional space, such as along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an element or a portion of an element (e.g., three degrees of rotational freedom in three-dimensional space, such as about roll, pitch, and yaw axes, represented in angle-axis, rotation matrix, quaternion representation, and / or the like). As used herein, and for a device with a kinematic series, such as with a repositionable structure with a plurality of links coupled by one or more joints, the term “proximal” refers to a direction toward a base of the kinematic series, and “distal” refers to a direction away from the base along the kinematic series.

[0027] As used herein, the term “pose” refers to the multi-degree of freedom (DOF) spatial position and orientation of a coordinate system of interest fixed relative to a rigid body. In general, a pose includes a pose variable for each of the DOFs in the pose. For example, a full 6-DOF pose for a rigid body in three-dimensional space would include 6 pose variables corresponding to the 3 positional DOFs (e.g., x, y, and z) and the 3 orientational DOFs (e.g., roll, pitch, and yaw). A 3 -DOF position only pose would include only pose variables for the 3 positional DOFs. Similarly, a 3 -DOF orientation only pose would include only pose variables for the 3 rotational DOFs. Further, a velocity of the pose captures the change in pose over time (e.g., a first derivative of the pose). For a full 6-DOF pose of a rigid body in three-dimensional space, the velocity would include 3 translational velocities and 3 rotational velocities. Poses with other numbers of DOFs would have a corresponding number of velocities translational and / or rotational velocities.

[0028] Aspects of this disclosure are described in reference to computer-assisted systems, which can include devices that are teleoperated, externally manipulated,autonomous, semiautonomous, and / or the like. Further, aspects of this disclosure are described in terms of an implementation using a teleoperated surgical system, such as the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including teleoperated and nonteleoperated, and medical and non-medical embodiments and implementations. Implementations on da Vinci® Surgical Systems are merely exemplary and are not to be considered as limiting the scope of the inventive aspects disclosed herein. For example, techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperated systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like. Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0029] Referring now to the drawings, in which like reference numerals represent like parts throughout the several views, FIG 1 A illustrates an example computer-assisted system 100, according to various embodiments. As shown in Figure 1 A, the computer-assisted system 100 includes, without limitation, a manipulator structure comprising a manipulator assembly 110 and a user input system 150. In a teleoperation scenario, an operator (not shown) uses the user input system 150 to operate the manipulator assembly 110, such as in a leader-follower configuration (also often called teleoperation configuration or master-slave configuration in industry) of the computer-assisted system 100. In the leader-follower configuration, the user input system 150 is the leader, and the manipulator assembly 110 is the follower of the leader-follower configuration.

[0030] The manipulator assembly 110 can be used to introduce a set of one or more instruments to a work site through a port 130, such as a gel port, a cannula, entry guide, etc. (a cannula is shown in the figure) inserted in an aperture. In a medical scenario, the work site can be on or within a body cavity of a patient, and the aperture can be a minimally invasiveincision or a natural body orifice. When used, the port 130 can be free-floating, held in place by a fixture separate from the manipulator assembly 110, or held by a drivable structure 122 (a linkage is shown in the figure) or other part of the manipulator assembly 110. The drivable structure 122 is coupled to additional joints and links 114, 120 of the manipulator assembly 110, and these additional joints and links 114, 120 are mounted on a base 112. In some embodiments, the drivable structure 122 terminates in a manipulator-supporting link 124. In the example shown, the manipulator assembly 110 further comprises a set of manipulators 126 coupled to the manipulator-supporting link 124. Each of the manipulators 126 include a carriage (or other instrument-coupling link) configured to couple to an instrument, and each of the manipulators 126 include one or more joint(s) that can be driven to move the carriage. For example, a manipulator 126 can include a prismatic joint that, when driven, linearly moves the carriage and any instrument(s) coupled to the carriage. In some embodiments, this linear motion is along an insertion axis of the instrument, as further described below with reference to FIG. 2A. In this configuration, the elements of the manipulator assembly 110 that are proximal to the manipulators 126 may be understood as a proximal repositionable structure.

[0031] In some embodiments, the additional joints and additional links 114 and 120 are used to position the port 130 at the aperture or another location. FIG. 1 A illustrates a prismatic joint for vertical adjustment (as indicated by arrow “A”) and a rotary joint for horizontal adjustment (as indicated by arrow “B”). The drivable structure 122 is used to robotically pivot the port 130 (and the instruments disposed within it at the time) in yaw, pitch and roll angular rotations about the remote center of motion as indicated by arrows C, D and E, respectively. These joints may be considered part of the proximal repositionable structure.

[0032] In some embodiments, actuation of the degrees of freedom provided by joint(s) of the instrument(s) is provided by actuators disposed in, or whose motive force (e.g., linear force or rotary torque) is transmitted to, the instrument(s). Examples of actuators include rotary motors, linear motors, solenoids, and / or the like. The actuators drive transmission elements in the manipulators and / or in the instruments to control the degrees of freedom of the instrum ent(s). For example, the actuators can drive rotary discs of the manipulator that couple with rotary discs of the instrument(s), where driving the rotary discs of the instruments drives transmission elements in the instrument that couple to move the joint(s) of the instrument, or to move the end effector(s) of the instrument, as further discussed below with reference to FIG.2. Accordingly, the degrees of freedom of theinstrument(s) are controlled by actuators that drive the instrument(s) in accordance with control signals determined based on inputs from the associated input devices (e.g., input devices 152 of the user input system 150). The control signals are determined in order to cause instrument motion or other actuation as indicated by movement of the input control devices or any other control signal. Furthermore, in some embodiments, appropriately positioned sensors, e.g., encoders, potentiometers, and / or the like, are provided to enable measurement of indications of the joint positions, or other data that can be used to derive joint position, such as joint velocity. The actuators and sensors are disposed in, transmit to, and / or receive signals from the manipulator(s) 126.

[0033] While a particular configuration of the manipulator structure including the manipulator assembly 110 is shown in FIG. 1 A, those skilled in the art will appreciate that embodiments of the disclosure can be used with any design of manipulator structure. For example, a manipulator assembly can have any number and any types of degrees of freedom, may or may not be configured to couple to a port, use a port other than a cannula, and / or other configuration different from what is shown in FIG 1 A.

[0034] In the embodiments shown in FIG 1A, the user input system 150 includes one or more input devices 152 operated by the operator (not shown). The one or more input devices 152 are contacted and manipulated by the hands of the operator, with one input device for each hand. Examples of such hand-input-devices include any type of device manually operable by human operator, e.g., joysticks, trackballs, button clusters, and / or other types of haptic devices typically equipped with multiple degrees of freedom. A more detailed description of an input device is provided below in reference to FIG. 3. Additionally, in some embodiments, position, force, and / or tactile feedback devices (not shown) are employed to transmit position, force, and / or tactile sensations from the instruments back to the operator's hands through the input devices 152.

[0035] The input devices 152 are supported by the user input system 150 and are shown as mechanically grounded, and in other implementations may be mechanically ungrounded. An ergonomic support 156 is provided in some implementations. For example, FIG. 1A shows an ergonomic support 156 including forearm rests on which the operator may rest his or her forearms while manipulating the input devices 152. In some examples, the operator performs tasks at a work site near the manipulator assembly 110 during a medical procedure by controlling the manipulator assembly 110 using the input devices 152.

[0036] A display unit 154 is included in the user input system 150. The display unit 154 displays images for viewing by the operator. The display unit 154 provides the operatorwith a view of the worksite with which the manipulator assembly 110 interacts. The view can include, for example, stereoscopic images or three-dimensional images to provide a depth perception of the worksite and the instrum ent(s) of the manipulator assembly 110 in the worksite. The display unit 154 can be moved in various degrees of freedom to accommodate the operator’s viewing position and / or to provide control functions. Where a display unit (such as the display unit 154 is also used to provide control functions, such as to command the manipulator assembly, the display unit also includes an input device (e.g., another input device 152).

[0037] When using the user input system 150, the operator can sit in a chair or other support in front of the user input system 150, position his or her eyes to see images displayed by the display unit 154, grasp and manipulate the input devices 152, and rest his or her forearms on the ergonomic support 156 as desired. In some implementations, the operator can stand at the workstation or assume other poses, and the display unit 154 and input devices 152 may differ in construction, be adjusted in position (height, depth, etc.), etc.

[0038] FIG. IB illustrates an example system 170, according to various embodiments. The system 170 corresponds to the computer-assisted system 100 and includes one or more computing systems 172. A computing system 172 includes a processing system and is used to process input provided by the user input system 150, e.g., from the input device(s) 152 manipulated by an operator. In some embodiments, a computing system 172 is further used to provide an output, e.g., a video image to the display unit 174. Examples of display unit 174 include LCDs, LEDs, organic LED displays, projectors, etc. In some embodiments, one or more computing systems 172 are used to control the manipulator assembly 110.

[0039] In one or more embodiments, the computing system(s) 172 executes one or more control methods. The control methods include instructions for controlling one or more components of the manipulator assembly 110. In one or more embodiments, joint movements of the manipulator assembly 110 are controlled by one or more control methods driving one or more joints using actuators of the manipulator assembly 110, the joint movements being calculated by a processor of a processing system of the computing system(s) 172. The control methods process control signals from the user input system 150 or elsewhere, and / or sensor signals (e.g., positional encoder data from joint position sensors, image data from image instruments such as ultrasonic probes or cameras or endoscopes, and / or the like), to calculate commands for the joint actuators.

[0040] In some embodiments, the control methods perform at least some of the calculations of the joint commands using vectors and / or matrices, some of which haveelements corresponding to positions, velocities, and / or forces / torques of the joints. The range of alternative joint configurations available to the control methods can be conceptualized as a joint space. For example, in some embodiments, the joint space has as many dimensions as the manipulator assembly has degrees of freedom, and a particular configuration of the manipulator assembly represents a particular point in the joint space, with each coordinate corresponding to a joint state of an associated joint of the manipulator assembly.

[0041] As used herein, the term “state” of a joint or multiple joints refers to the control variables associated with the joint or the multiple joints, respectively. For example, the state of an angular joint refers to the angle defined by that joint within its range of motion, and / or to the angular velocity (or speed or direction) of the joint. Similarly, the state of an axial or prismatic joint refers to the joint's axial or linear position, and / or to its axial or linear velocity (or speed or direction). While one or more of the control methods described herein include position controllers, they often also have velocity control aspects. Alternative embodiments can rely primarily or entirely on velocity controllers, force controllers, acceleration controllers, and / or the like without departing from the disclosure. Various aspects of control systems that can be used in such devices are described in U.S. Pat. No. 6,699,177, which is incorporated herein by reference. In general, as long as the movements described are based on the associated calculations, the calculations of movements of the joints and movements of an end effector described herein are performed using a position control technique, a velocity control technique, an acceleration control technique, a force or torque control technique, a combination of some or all of the foregoing, and / or the like.

[0042] In some embodiments, the control modes include one or more other types of control modes. For example, during a robotic task being performed under the control of input devices 152 operated by a user, various joints of the manipulator assembly can be commanded to a same position and controlled to maintain static positions. However, in another control mode, one or more of the joints can be commanded to be “floating”, and facilitate motion of that joint due to externally applied force. For example, a joint held in place by a brake can be floated by partially or entirely releasing the brake. As another example, a joint driven by actuator(s) can be held in place by commanding the actuator(s) to hold the joint position, and be floating by updating the command to the actuator(s) to the then-current position, velocity, and / or acceleration of the joint. .

[0043] In various embodiments, multiple different control modes are combined during operation of the manipulator assembly. For example, some joints could be controlled to maintain position and resist or rebound from attempted external articulation of those joints,while other joints could be controlled to move those other joints. Parameters such as joint position, velocity, or acceleration of the joints are detected by joint sensors. The sensor signals are used to provide kinematic information of the manipulator assembly.

[0044] The architecture of the control methods used for controlling the manipulator assembly can be of any appropriate form. As a specific example, the control architecture can be hierarchical, and could include a high-level controller and multiple joint controllers. A commanded movement is received by the high-level controller in, for example, a Cartesiancoordinate space (referred to herein as Cartesian-space). The commanded movement could be, for example, based on a movement command (e.g., in the form of a position and / or velocity) received from the user input system 250, or any other system that provides a movement command. The commanded movement is converted into commanded joint positions or joint velocities (e.g., linear or angular joint positions, linear or angular joint velocities). In some embodiments, the conversion is performed using an inverse kinematics algorithm. Subsequently, the joint controllers convert the received commanded joint positions or velocities into commanded currents to drive the actuators producing joint movements. The joint movements together produce a manipulator assembly movement that reflects the commanded movement.

[0045] In some embodiments, a joint controller controls a joint position. In some embodiments, the joint controller controls other variables such as joint velocity and / or joint force (linear force or angular torque). A joint controller receives a feedback signal in the form of a sensed joint state from an associated joint sensor, which it can use for closed-loop control. The sensed joint state includes, for example and without limitation, a joint position, a joint velocity (or component of velocity such as speed or direction), a joint acceleration (or component of acceleration), and / or the like, representing the joint movement. The sensed joint state is derived from the signals obtained from the joint sensor. A joint sensor can be, for example, an encoder, a potentiometer, an accelerometer, a hall effect sensor, and / or the like. In some embodiments, a state observer or estimator (not shown) is used. Each joint controller can implement any appropriate control scheme, such as a proportional integral derivative (PID), proportional derivative (PD), full state feedback, sliding mode, and / or various other control schemes, without departing from the disclosure.

[0046] In one or more embodiments, the control methods further perform at least one of the steps described in FIG. 7 below.

[0047] A computing system 172 may include, without limitation, one or more computer processors, non-persistent storage (e.g., volatile memory, such as random accessmemory (RAM), cache memory), persistent storage (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, etc.), a communication interface (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and / or numerous other elements and functionalities. In some embodiments, a computer processor of a computing system 172 is an integrated circuit for processing instructions. For example, the computer processor can be one or more cores or micro-cores of a processor.

[0048] In some embodiments, a communication interface of a computing system 172 includes an integrated circuit for connecting the computing system 172 to a network (not shown) and / or to another device, such as another computing system 172. Further, in some embodiments, the computing system 172 includes one or more output devices, such as a display unit 174, a printer, a speaker, external storage, or any other output device. Software instructions in the form of computer readable program code to perform embodiments of the disclosure are stored, in whole or in part, temporarily or permanently, on non-transitory computer readable medium. Specifically, the software instructions correspond to computer readable program code that, when executed by a processor(s), is configured to perform one or more embodiments of the disclosure. In some embodiments, a computing system 172 is connected to or be a part of a network. The network may include multiple nodes. Each node corresponds to a different computing system, group of computing systems, group of nodes, and / or the like.

[0049] In some embodiments, the manipulator assembly 110 couples to an instrument when used to perform a procedure. The instrument can include an imaging device, e.g., an endoscope or an ultrasonic probe, usable to capture images of the worksite and output the captured images to an auxiliary system 180. In some embodiments, the auxiliary system 180 processes the captured images using one or more image processing techniques prior to any subsequent display. For example, the auxiliary system 180 can overlay the captured images with a virtual control interface prior to displaying the combined images to the operator via the user input system 150. In some embodiments, one or more separate display units 174 are coupled with a computing system 172 and / or the auxiliary system 180 for local and / or remote display of images, such as images of the procedure site or other related images.

[0050] FIG. 2A shows elements of an example manipulator assembly 200, in accordance with one or more embodiments. In the example shown in FIG. 2A, the elements of the manipulator assembly 200 correspond to a distal portion of the manipulator assembly110 of FIG. 1 A. The manipulator assembly 200 can be used to introduce a set of instruments 260, 270, 280 to a work site through a port 130 inserted in an aperture.

[0051] The manipulator assembly 200 includes a manipulator-supporting link 202 that may be rotated about roll axis 290 (arrow E in FIGs. 1 A and 2A) of the manipulatorsupporting link 202. The three instruments 260, 270, and 280 are each supported by the manipulator-supporting link 202 through a respective manipulator 126. As previously discussed in reference to FIG. 1 A, each of the instruments 260, 270, 280 is also supported by a manipulator 126 (proximal of the manipulator-supporting link 202 and therefore not visible in FIG. 2A) where the instrument is coupled to a carriage. Translational movement of the carriage of the manipulator 126 may be used for an insertion / retraction of each of the instruments 260, 270, 280 (arrow F in FIG. 2A, for simplicity shown only for instrument 280). Each of the instruments 260, 270, 280 include an instrument shaft 262, 272, 282 and an end effector 264, 274, 284. In the example of FIG. 2A, instruments 260 and 270 are illustrated as equipped with jawed end effectors and instrument 280 is illustrated as equipped with an imaging instrument (e.g., an endoscopic camera). Each of the end effectors may be movable in one or more degrees of freedom. For example, the end effector 284 may be able to change orientation in pitch and yaw degrees of freedom (arrows G and H, respectively). End effectors 264, 274, and 284 may include additional degrees of freedom than shown. For example, for jawed instruments, these degrees of freedom may further enable and open / close operation of jaws of the end effector. In the example of FIG. 2A, instruments 260 and 270 further include a degree of freedom for rotation of the instrument shaft 262, 272 (arrow I, only shown for instrument 270).

[0052] In order to increase the range of motion of the instrument 280 (which may not have a degree of freedom for rotation about the instrument shaft 282, a roll movement about the roll axis 290, as indicated by arrow E, may be performed. Such a roll movement changes the position and the orientation of the end effector 284. A roll movement of the end effector 284 is thus possible, even in absence of a roll degree of freedom at the instrument 280. As illustrated in FIG. 2B, discussed below, the additional roll movement enables the instrument to assume additional configurations as compared to otherwise identical systems incapable of roll movement. However, it may be desirable to avoid affecting the configuration of the end effectors 264, 274 of instruments 260, 280, when performing the roll movement about roll axis 290. One such scenario assumes that the end effector 284 of instrument 280 is or includes an imaging device, which may be used to capture still or moving images (e.g., videos) of operations performed using the end effectors 264, 274 of instruments 260, 270.This enables an operator viewing images captured by the imaging device to better observe the actions of instrument 260, 270. If the operator wants to change the field of view of the imaging device, for example to view the operations performed using the end effectors 264, 274 from a different angle, this may be facilitated by roll movement about the roll axis 290. In this scenario, a change in the configuration of the end effectors 264, 274, caused by the roll movement, would be undesirable.

[0053] To avoid such an unintended change in position, orientation, internal configuration (e.g., jaw opening for jawed instruments) state of the end effectors 264, 274, the instruments 260, 270 may be reconfigured to compensate for the roll movement about the roll axis 290. In the example illustrated in FIG. 2A, instruments 260, 270 are equipped with shaft offset joints 266, 276. Reconfiguration of the shaft offset joints 266, 276, along with insertion / retraction and / or rotation along the instrument shafts 262, 272 of the instruments 260, 270 may be used for this compensation.

[0054] FIG. 2B shows a reconfiguration of elements of a manipulator assembly (e.g., manipulator assembly 110, 200) in accordance with one or more embodiments. In the example, the instrument 280 is a stereoscopic imaging instrument, with two cameras and an illuminator (all shown in the illustration 230) disposed in between. In the schematic illustration 230, the instrument 280 is viewed facing the two cameras, and only the front facing surface of the instrument 280 is shown. The illustration 230 further includes a crosshair 292 that identifies the orientation of the manipulator-supporting link (not shown). Rotation of the manipulator-supporting link, thus maps to a corresponding rotation of the crosshair 292. The left panel of the illustration 230 shows the end effector 284 of the imaging instrument 280 in four different configurations. Three configurations are illustrated using solid lines, identifying camera configurations (in the example in two-dimensional space) that can be reached with the manipulator-supporting link oriented as indicated by the crosshair 292. One configuration is illustrated using dashed lines, identifying a camera configuration that cannot be reached with the manipulator-supporting link oriented as indicated by the crosshair 292. The right panel of the illustration 230 shows the end effector 284 of the imaging instrument 280 in the configuration that previously could not be reached. This configuration can now be reached because the manipulator-supporting link 202 has been rotated as indicated by the crosshair 292 rotated as indicated by arrow E, also shown in FIG. 2A. As FIG. 2B illustrates, the instrument 280 can be rotated, even though the instrument itself does not have the corresponding rotational degree of freedom. While the roll E about the roll axis 290 does not directly result in a pure rotation of the instrument 280 because theinstrument shaft 282 is offset from the roll axis 290, the shaft offset joints 286, combined with insertion / retraction along instrument shaft 282 enable a pure rotation of the instrument 280.

[0055] FIG. 3 is a perspective view of an input portion 300 of an example input device such may be used for the input devices 152 depicted in FIG. 1A. In some implementations, the input portion 300 includes one or more gimbal mechanisms. In the example of FIG. 3, the input portion 300 includes a handle 302 which is contacted by an operator to manipulate the input device. Further, in this example, the handle 302 includes two grips that each include a finger loop 304 and a grip member (depicted as grip members 306a, 306b). The two grip members 306a, 306b are positioned on opposite sides of a central portion 303 of the handle 302, and the grip members 306a, 306b can be grasped, held, or otherwise contacted by an operator’s fingers. Each finger loop 304 is attached to a respective grip member 306a or 306b and can be used to secure an operator’s fingers to the associated grip member 306a or 306b. In this example, finger contacts 305 can be connected or formed at the unconnected end of the grip members 306a, 306b to provide surfaces to contact the operator’s fingers. The operator may also contact other portions of handle 302 while grasping the grip members 306a, 306b.

[0056] Each grip member 306a, 306b can be moved in an associated degree of freedom (depicted as degrees of freedom 308a, 308b). In some examples, the grip members 306a, 306b are each coupled to the central portion 303 of the handle 302 at respective rotational couplings, allowing rotational movement of the grip members about associated grip axes 307a, 307b, with respect to the central portion 303. As such, each grip member 306a, 306b can be moved in an associated degree of freedom (i.e., degree of freedom 308a about grip axis 307a and degree of freedom 308 about grip axis 307b) by an operator contacting the grip members (306a, 306b). In various implementations, a single grip member (e.g., 306a) and finger loop (304) can be provided, or only one of the grip members (e.g., 306a) can be moved in the corresponding degree of freedom (e.g., 308a) while the other grip member (e.g., 306b) can be fixed with reference to the handle 302. For example, the positions of grip members 306a, 306b in their degrees of freedom can control corresponding rotational positions of an instrument or component thereof.

[0057] One or more grip sensors (not shown) can be coupled to the handle 302 and / or other components of the input portion 300 and can detect the positions of the grip members 306a, 306b in their respective degrees of freedom 308a, 308b. The grip sensors can send signals describing sensed positions and / or motions to the control of computing system(s) 172of the computer-assisted system 100. In some modes (e.g., a teleoperation mode described later in the instant disclosure) or implementations, the computing system 172 can provide control signals to a device manipulated by the computer-assisted system 100 (e.g., manipulator assembly 110). For example, the positions of the grip members 306a, 306b in their respective degrees of freedom 308a, 308b can be used to control any of various degrees of freedom of an instrument (or, in some instances, the distal end of an instrument) supported by the manipulator assembly 110.

[0058] Various implementations of an input device, such as the that depicted in the input portion 300 of FIG. 3, can provide one or more active actuators (e.g., motors, voice coils, etc.) to output active forces on the grip members 306a, 306b in the degrees of freedom 308a, 308b. For example, a sensor and / or actuator can be housed in central portion 303 or in housing 309 and coupled to the grip members 306a, 306b by a transmission. Some implementations can provide one or more passive actuators (e.g., brakes) or springs between the grip members 306a, 306b and the central portion 303 of the handle 302 to provide resistance in particular directions of the grips (e.g., movement in directions toward each other in degrees of freedom 308a, 308b).

[0059] The handle 302 can additionally be provided with a rotational degree of freedom 310 about a roll axis 312 defined between a first end and a second end of the handle 302. The roll axis 312 is a longitudinal axis in this example that extends approximately along the center of the central portion 303 of handle 302. The handle 302 can be rotated about the roll axis 312 with respect to a base member of the input portion 300, such as a base member that includes the housing 309. For example, an operator can rotate the grip members 306a, 306b and central portion 303 as a single unit around the roll axis 312), with respect to the housing 309, to provide control of manipulator assembly 110 or other elements.

[0060] Additionally, one or more input sensors (not shown) can be coupled to the handle 302 to detect the orientation of the handle 302 in the rotational degree of freedom 310. For example, the sensor can send signals describing the orientation to the computing system(s) 172 that can provide control signals to the manipulator assembly (110) as described above. For example, rotation of the handle 302 in the rotational degree of freedom 310 can control a particular degree of freedom of an instrument 260, 270, 280 of the manipulator assembly 110 that is different than another degree of freedom controlled by the degrees of freedom 308a, 308b of the grip members 306a, 306b.

[0061] Some implementations of the input portion 300 can provide one or more actuators to output forces on the handle 302 (including grip members 306a, 306b and fingerloops 304 in the rotational degree of freedom 310. For example, a sensor and / or actuator can be housed in the housing 309 and coupled to the handle 302 by a shaft extending through the central portion 303) of the handle 302.

[0062] In various implementations, the handle 302 can be provided with additional degrees of freedom. For example, a rotational degree of freedom 320 about a yaw axis 322 can be provided to the handle 302 at a rotational coupling between an elbow shaped link 324 and a link 326, where the elbow shaped link 324 is coupled to the handle 302 (e.g., at the housing 309). In this example, the yaw axis 322 intersects and is orthogonal to the roll axis 312. Additional degrees of freedom can similarly be provided. For example, the link 326 can be elbow-shaped and a rotational coupling can be provided between the other end of link 326 and another link (not shown). A rotational degree of freedom 328 about an axis 330 can be provided to the handle 302 at the rotational coupling. In some examples, the input portion 300 can allow movement of the handle 302 within the workspace of the input system 150 with a plurality of degrees of freedom, e.g., six degrees of freedom including three rotational degrees of freedom and three translational degrees of freedom. One or more additional degrees of freedom can be sensed by associated input sensors and / or actuated by actuators (motors, etc.) similarly as described above for the degrees of freedom 308a, 308b, 310, etc., the sensors and actuators coupled to the input portion 300. In various implementations, sensors can sense positions of the handle in a degree of freedom, or sense orientations of the handle in a degree of freedom, or sense positions and orientations of the handle in multiple degrees of freedom. For example, positions in a translational degree of freedom and orientations in a rotational degree of freedom can be sensed by one or more associated input sensors. In some examples, a position in a translational degree of freedom and / or orientation in a rotational degree of freedom can be derived from rotations of components (e.g., links of a linkage) coupled to the handle 302 as sensed by rotational sensors. Some implementations can include linear sensors that can directly sense translational motion of one or more components coupled to the handle 302. In some implementations, each additional degree of freedom of the handle 302 can control a different degree of freedom (or other motion) in the manipulator assembly 110.

[0063] In an example implementation, the handle 302 is mechanically grounded, i.e., supported in space by a kinematic chain with an end stationary at mechanical ground, such as a floor, wall, or ceiling. For example, the housing 309 can be coupled to a mechanical linkage that is coupled to the ground or an object connected to ground, providing a stable platform for the use of the input portion 300. For example, a grounded mechanical linkagecan be connected to a base member, e.g., with one or more rotary couplings, ball joints, or other couplings, including linear joints. The mechanical linkage can provide six or more degrees of freedom to the handle 302.

[0064] In the example of FIG. 3, the handle 302 includes one or more switches 350. As depicted, the one or more switches 350 can be coupled to the central portion 303 or to mechanisms within central portion 303. For example, two switches 350 can be positioned on opposite sides of axis 312, and / or additional switches can be provided. In some examples, a switch 350 has a portion that can slide parallel to the axis 312, e.g., as directed by an operator’s finger, or the switch portion can be depressed. In some implementations, the switch 350 can be moved to various positions to provide particular command signals, e.g., to select functions, options, or modes of the input system 150 and / or input device. In some implementations, one or more of the switches 350 can be implemented as a button (e.g., depressed in a direction, such as perpendicular to the axis 312 or other direction), a rotary dial, a switch that moves perpendicular to the axis 312, or other type of input control. Switches 350 can use electromagnetic sensors, mechanical switches, magnetic sensors, or other types of sensors to detect positions of the switch.

[0065] As previously stated, the computer-assisted system 100 can be a teleoperated system with a leader-follower control scheme. The leader-follower control scheme is also called the master-slave control scheme in literature. With the leader-follower control scheme, the input system 150 forms part or all of a “leader” ( “master”) device that commands at least a portion of the manipulator structure, such as part or all of manipulator assembly 110) (the “follower” or “slave” device).

[0066] In general, a control system, e.g., computing system(s) 172, receives control signals from the input system 150 and generates actuation signals which are sent to command actuators of manipulator assembly 110. The control system can also receive sensor signals that indicate positions, orientations, states, and / or changes of various follower components (e.g., manipulator structure elements) from the manipulator assembly 110 and send actuation signals to the input system 150 to provide force, torque, and / or position feedback to the operator.

[0067] In one or more embodiments, an input system 150 is equipped with two input portions 300 (one for each of the operator’s hands) is used to control elements instruments, e.g., instruments 260, 270, and 280. A specific implementation of control systems and methods that implement a view-based steering of an imaging instrument (e.g., instrument 280) is described below in reference to FIG. 4.

[0068] While FIGs. 1 A-3 show various configurations of components, other configurations may be used without departing from the scope of the disclosure. For example, various components may be combined to create a single component. As another example, the functionality performed by a single component may be performed by two or more components. While examples of particular manipulator assemblies, particular repositionable structures, instruments, input systems, and input portions are shown, the disclosure generalizes to any type of manipulator assemblies, repositionable structures, instruments, input systems, and input portions with any number of degrees of freedom. Further, while components are often described in context of medical scenarios such as surgical scenarios, embodiments of the disclosure are equally applicable to other domains that involve robotic manipulation, e.g., non-surgical scenarios or systems, non-medical scenarios or systems, and / or the like.

[0069] FIG. 4 shows elements and operation of a manipulator assembly 400 of a computer-assisted system, in accordance with one or more embodiments. The manipulator structure as shown comprises a manipulator structure 410, which is supporting an imaging instrument 420.

[0070] When operating a computer-assisted system to perform a procedure, a user may use visual feedback from the environment. The environment may include a target site, and the user may, for example, want to visually inspect the target site and / or interact with the target site, e.g., by performing a procedure. The visual feedback may be provided by the imaging instrument 420 based on the field of view of the imaging instrument. The visual feedback may be provided to the user in the form of a 3D or 2D view.

[0071] The user may want to change what is visible in the field of view for various reasons, e.g., to better observe the target site or, more generally, a region in the field of view, where the region could be anything in an environment that is observable by the imaging instrument. The imaging instrument’s position and / or orientation may be updated as desired by the user, by changing the configuration of the joints of the manipulator structure 410 and / or the imaging instrument 420.

[0072] In one or more embodiments, the configuration (position and / or orientation) of the imaging instrument 420 is updated, based on a user input received at, for example, the input portion 300 of an input device of input system 150 manipulated by the user as subsequently described.

[0073] The manipulator structure 410 comprises a plurality of links extending from a proximal base 412 of the manipulator assembly 400. The plurality of links is coupledtogether by a plurality of joints. The proximal base 412 is proximal to the plurality of links, and may be stationary or may be movable. For example, the proximal base 412 may be installed on tracks enabling a linear motion, on a rollable cart, on another manipulator structure actuatable to move the proximal base 412, etc. A manipulator structure base reference frame 414 is associated with the proximal base 412. Certain operations, such as kinematic operations may be performed in the manipulator structure base reference frame 414. For example, the imaging instrument position and / or orientation may be determined in the manipulator structure base reference frame using forward kinematics of the manipulator structure 410 and the imaging instrument 420. For a proximal base 412 that is stationary, the manipulator structure base reference frame may be a world reference frame or may have a continuous offset from a world reference frame.

[0074] The manipulator structure 410 may be any type of manipulator structure. The manipulator structure may be a general-purpose robotic structure, or it may be a robotic structure designed for a specific application, such as a medical device robotic structure. Examples of manipulator structures include the elements of the manipulator assembly (110 in FIG. 1 A; 200 in FIG. 2A) that support an instrument, e.g., imaging instrument 420.

[0075] The manipulator structure 410 may include any number of links 416. The links 416 are coupled by joints 418 that may be of any type, such as revolute joints and prismatic joints.

[0076] The imaging instrument 420 may be any type of imaging instrument. The imaging instrument may be a general-purpose monoscopic or stereoscopic camera or it may be a specialized instrument such as an endoscope. An example of an imaging instrument is provided in FIG. 2A. Additional details may be found in U.S. Patent No. 8,620,473, e.g., in FIG. 8 showing an articulable imaging system 1750 and the associated description. The imaging instrument may be articulable separately from other instruments that may be present, e.g., as shown in FIG. 2A. Accordingly, the imaging instrument 420 may have its own joints, in addition to the joints of the manipulator structure 410. Any number of joints may be present, as long as the number of joints is sufficient to enable movement of the imaging instrument 420 as discussed below and, more generally, the combination of the manipulator structure 410 and the imaging instrument 420 is not limited to a particular kinematic configuration.

[0077] In one or more embodiments, the user may control the operation of the manipulator assembly 400 using different operating modes. Depending on the selected operating mode, the manipulator assembly 400 may respond differently to a user inputprovided by the user. Examples of robotic systems operating in different operating modes when receiving user inputs are provided in U.S. Patent No. 9,586,323. U.S. Patent No. 9,586,323 is hereby incorporated by reference in its entirety.

[0078] One or more of the operating modes may be imaging instrument movement modes, which may be used to enable the user to move the imaging instrument 420, where motion of an input device (e.g., handle 302 of the input portion 300 of the input device), when operated by the user (e.g., using a single hand), commands motion of the imaging instrument 420, thereby causing a change of the location (position and or orientation) of a region captured in the imaging instrument’s field of view field according to a desired motion. The user is, thus, given the perception of being able to move the environment viewed through the imaging instrument according to the desired motion while, in reality, the imaging instrument is the component that moves, thereby causing a movement of an image of the region in the field of view. This movement of the image relative to the field of view results in the perception of moving the environment (with the region) itself. A description is subsequently provided in reference to FIGs. 5A-5C, 6A, 6B, and 7.

[0079] Continuing with the discussion of FIG. 4, a view reference frame 424 is defined by the field of view 422 (in this particular example, view reference frame 424 is fixed relative to the field of view 422). A view axis 426 extends as shown in FIG. 4. The field of view 422 may be a cone or frustrum with the view axis at its center, and may be determined by the optical characteristics of the imaging instrument 420. While the field of view 422 may be modifiable using optical and digital operations (optical / digital zoom, digital translation and / or roll operations), in the following discussion the field of view 422 is assumed to be changed by movement of the imaging instrument, e.g., relative to the manipulator structure base reference frame 414. The field of view 422 defines the part of the environment surrounding the imaging instrument 420 that is visible through the imaging instrument. Regions of the environment that are within the field of view are visible, regions of the environment that are outside the field of view are not visible. In FIG. 4, a region 428 of the environment in the field of view is visible as an image of the region through the imaging instrument 420.

[0080] Motion of the image of the region 428 within or relative to the field of view 422 resulting from movement of the imaging instrument 420 may be based on a user input received from a user at an input device. The user input may specify a desired motion of the image of the region 428. The desired motion may include one or more components including a roll component, one or more translation components, a pitch component, a yaw component,and / or an insertion / retraction component, as illustrated in FIG. 4. The components may be summarized as a roll component and one or more non-roll components (comprising the one or more translation components, the pitch component, the yaw component, and / or the insertion / retraction component).

[0081] The determining of motion commands that can be used to cause motion of the joints in order to update the imaging instrument’s position and / or orientation such that the image of the region 428 moves in the field of view 422 based on the desired motion, in accordance with embodiments of the disclosure, is subsequently discussed.

[0082] FIGs. 5A and 5B show control schemes 500, 550 of a computer assisted system, in accordance with embodiments of the disclosure. Both control schemes may be used to update the imaging instrument’s spatial configuration in multiple degrees of freedom (e.g., roll, pitch, yaw, translation, and / or insertion / retraction), as previously described. In one or more embodiments, the control schemes 500, 550 are used to update the imaging instrument’s spatial configuration in three rotational and three translational degrees of freedom, i.e., six degrees of freedom.

[0083] The control schemes 500, 550 include a robotic controller 510 and an imaging instrument control filter 520. These components are subsequently discussed.

[0084] An input device, e.g., an input device of input system 150 as previously described, may be used to control the imaging instrument. The input device may further be used to control other instruments (different from the imaging instrument). The input device may obtain a user input to move the imaging instrument such that an image of a region of an environment captured by the imaging instrument moves relative to the field of view of the imaging instrument according to a desired motion. To the user, this generates the perception of being able to control movement of the environment (including the region) relative to a stationary imaging instrument, while in fact it is the imaging instrument that is being moved. The desired motion may be obtained in any reference frame, e.g., in an input device base reference frame, a world reference frame, the manipulator structure base reference frame, a reference frame centered on the user, e.g., a view or eye-centered reference frame, etc. The input device base reference frame is defined relative an input system 150 that supports an input portion 300 (in this particular example, by being fixed relative to a portion of the input system 150).

[0085] In one or more embodiments, the robotic controller 510 in combination with the imaging instrument control filter 520 are configured to determine a motion command based on the desired motion. The motion command may then be used to command the jointsof the manipulator structure and / or the imaging instrument to cause motion of these joints such that the image of the region in the environment captured in the field of view moves in the field of view (i.e., relative to the field of view) according to the desired motion. The robotic controller 510 may be any kind of robotic controller and may use any appropriate technique. In some examples, the robotic controller 510 uses a Jacobian-based approach. An example is provided in FIG. 5C.

[0086] Referring to FIG. 5C, in one or more embodiments, the robotic controller 510 generates a motion command q to control the joints 418 based on a desired motion obtained at the input device, converted into an imaging instrument desired motion Xstate by the imaging instrument control filter 520. The motion command q is in an n-dimensional joint space, with n being the number of joints 418 being controlled when positioning and / or orientating the imaging instrument 420. Joints of the manipulator structure 410 and / or joints of the imaging instrument 420 may be involved. The input (i.e., the camera desired motion Xstate) to the robotic controller 510 may be provided in cartesian coordinates, e.g., in the manipulator structure base reference frame 414, or in any other reference frame. In some embodiments, the view reference frame is used to provide the imaging instrument desired motion Xstate. A description of the imaging instrument desired motion Xstate, generated by the imaging instrument control filter 520 based on the desired motion, is provided below.

[0087] FIG. 5C shows a kinematic controller that includes an inverse kinematics stage (J+block) that determines a joint configuration (joint angles / positions) q corresponding to the imaging instrument desired motion Xstate received at the input of the robotic controller 510 from the imaging instrument control filter 520. Individual joints are controlled by joint controllers that drive joint actuators with currents to position or orient the joints as instructed by q. The inverse-kinematics stage may operate on increments (differentiation block) of a cartesian error generated by subtracting the current cartesian state from the imaging instrument desired motion Xstate. The current cartesian state is generated using a forward kinematics model (forward kinematics block) operating on q. A gain (2 block) is used to set the control gain of the feedback control loop. The output of J+is in increments and is integrated (J block in FIG. 5C) to obtain q. As previously noted, q may then be used to drive individual joint controllers.

[0088] Control architectures in accordance with embodiments of the disclosure may differ from the example shown in FIG. 5C. For example, the feedback loop may be closed over signals obtained from joint sensors, without departing from the disclosure. Furthermore, in some embodiments, controlling the joints is performed under consideration of potentiallyone or more constraints. Such constraints may be imposed by range of motion limitations on a cartesian and / or joint level, velocity limits, spatial constraints such as the maintaining of a remote center (e.g., as described in reference to FIG. 1 A), collision avoidance, and other constraints that may be satisfiable when controlling a manipulator structure 410 / imaging instrument with kinematic redundancies. Other robotic controllers may output a velocity as an alternative to position / orientation or in addition to position / orientation.

[0089] In some embodiments, the robotic controller 510 includes multiple interacting kinematic controllers. Referring to the configuration of manipulator assembly 200 shown in FIG. 2A and the reconfiguration of the elements of the manipulator assembly shown in FIG. 2B, the robotic controller 510 may include a first kinematic controller that controls all joints of the manipulator structure and the instruments, but that does not include functionality to perform the instrument roll operation illustrated in FIG. 2B. Such a robotic controller may further include a second kinematic controller that augments the first kinematic controller by adding the instrument roll functionality to the robotic controller 510. The second kinematic controller may be dedicated to determining a joint angle for the joint that provides the roll degree of freedom of the manipulator-supporting link. The configurations that are not reachable using the first kinematic controller alone are, thus, made reachable through a roll of the manipulator-supporting link about the roll axis of the manipulator-supporting link, controlled by the second kinematic controller. The second kinematic controller is, thus, configured to roll the manipulator-supporting link 202 until the instrument 280 is oriented such that the desired configuration of the end effector 284 is achievable using movement of the instrument joints (e.g., joints 286) as illustrated in FIG. 2B.

[0090] Because a roll movement of the manipulator-supporting link 202 alters position and / or orientation of the instruments 260, 270, 280, thereby causing interactions with the operations performed by the first kinematic controller, the output of the second kinematic controller is injected into the feedback path of the first controller.

[0091] While the configuration as described relies on a first kinematic controller and a second kinematic controller, in an alternative configuration, a single kinematic controller may determine the roll movement of the manipulator-supporting link together with the movements of the other joints of the manipulator assembly.

[0092] Analogous to the robotic controller 510 as described in reference to FIG. 5C, both the first and the second kinematic controllers may receive input (imaging instrument desired motion Xstate) in cartesian coordinates. The first kinematic controller may receive the full six-dimensional state, whereas the second kinematic controller may receive a three-dimensional state that is limited to the rotational components needed to determine the roll movement of the manipulator-supporting link. The first kinematic controller may output a motion command for all n joints, whereas the second kinematic controller may output a motion command for the joint responsible for the roll movement of the manipulatorsupporting link only.

[0093] In one or more embodiments, the imaging instrument control filter 520 is functionally located between the input device and the robotic controller 510. The imaging instrument control filter 520 receives signals that represent the user input provided by the user at the user input device in the form of a desired motion and alters these signals to generate signals that represent an imaging instrument desired motion used to drive the robotic controller 510.

[0094] In some embodiments, the alterations performed by the imaging instrument control filter enable an intuitive steering of the imaging instrument by the user via the input device. A discussion of intuitive control and control behaviors in presence and absence of the imaging instrument control filter 520 are subsequently discussed in reference to FIG. 6A, followed by a discussion of implementations of the imaging instrument control filter as shown in FIG. 6B.

[0095] FIG. 6A schematically illustrates a view content-based steering of an imaging instrument in accordance with one or more embodiments.

[0096] A user steering the imaging instrument may expect the view provided to them by the imaging instrument to change in a certain manner to make the steering of the imaging instrument intuitive. Deviations from such an expected behavior may be perceived as making the steering of the imaging instrument less intuitive. Therefore, some control schemes for controlling the imaging instrument based on a user input may be more or less intuitive than others. What is perceived as intuitive may depend on circumstances. Frequently, a movement that is executed as expected in the user’s visual reference frame is considered intuitive by the user. Accordingly, what is considered intuitive may depend on the user’s current view. For example, when observing the entire manipulator assembly 400 (e.g., as an external observer looking at the entire system), a motion of the imaging instrument 420 that follows a commanded movement in the manipulator structure base reference frame 414 may be considered intuitive. A command to the left would move the imaging instrument 420 to the left, in the manipulator structure base reference frame 414; a command in an upward direction would move the imaging instrument 420 in an upward direction the manipulator structure base reference frame 414, etc. In contrast, this type of motion may no longer feelintuitive when the user is viewing the image captured by the imaging instrument 420, e.g., through the display unit 154. In this case, because the user’s current view is limited to the field of view of the imaging instrument itself, a corresponding motion of the view content (i.e., what may be seen using the imaging instrument) that follows the commanded movement would be considered intuitive.

[0097] The example of FIG. 6A illustrates this based on a sequence of user inputs. While FIG. 6A, for the sake of simplicity, is limited to a two-dimensional example, the concepts as described generalize to a three-dimensional space with three translational and three rotational degrees of freedom. Further, while the user inputs are shown as being provided sequentially, the user inputs may overlap in time or may occur simultaneously.

[0098] The sequence of three user inputs as shown in the left column of FIG. 6 A includes user inputs to translate to the right, perform a clockwise roll, and translate in an upward direction.

[0099] As illustrated in the center column of the illustrations of FIG. 6 A, the view content, i.e., what may be seen by a user using the imaging instrument updates according to the user input. Assume that a view of content in an environment is captured by the imaging instrument. In the illustrations of the center column, the view content is represented by the irregular-shaped image of the region, and the view as displayed to the user in a display is represented by a rectangle. Further assume that the view as displayed corresponds to the field of view of the imaging instrument. The content in the field of view (and therefore the content visible in the display) updates according to the user inputs as illustrated in the center column: First, the image of the region is translated to the right, next the image of the region is reoriented clockwise, and finally the image of the region is translated in an upward direction (solid lines identify the image of the region prior to the updating, and dashed lines identify the image of the region after the updating). Based on the direct correspondence of the user input and the resulting movement of the image of the region (or, more generally, the resulting updating of the view content in the field of view) as displayed to the user, the user may get the perception of having direct control over movement of the view content in the display.

[0100] As illustrated in the right column of FIG. 6A, it is movement of the imaging instrument that generates the perception of having direct control over movement of the view content in the display. In some embodiments, the movement of the imaging device is determined relative to an initial spatial configuration of the imaging instrument. Such an initial spatial configuration may be determined using a latching operation performed at a point in time prior to the execution of the movement of the imaging device, as discussed indetail below. In the example as illustrated in the right column of FIG. 6 A, assume that the initial spatial configuration is established by the “x” symbol. In the example, the initial spatial configuration is defined at the center of the field of view of the imaging instrument. The initial spatial configuration may be defined in other manners, without departing from the disclosure. Further assume that the irregular-shaped region is stationary in the environment, whereas the field of view of the imaging instrument (rounded rectangle) is movable relative to the environment (solid lines identify the field of view prior to the updating, and dashed lines identify the field of view after the updating). In the control algorithms described in reference to FIG. 6B, the movement of the field of view is determined relative to the initial spatial configuration. Accordingly, as illustrated in the right column, top row, a movement of the imaging instrument to the right (relative to the initial spatial configuration) is used to generate the perception of the region moving to the left (image of the region moving to the left in the illustration of the display output), in the field of view. Further, as illustrated in the right column, center row, a counterclockwise reorientation in combination with a translational movement of the imaging instrument (relative to the initial spatial configuration) is used to generate the perception of the region being reoriented in a clockwise direction (image of the region rotating in clockwise direction in the illustration of the display output). Although the perception involves a pure reorientation of the region (without a translational component, a combination of a translational and a rotational movement of the imaging instrument is needed to generate the perception as a result of the prior translation (top row). Finally, as shown in the right column, bottom row, a translational movement of the imaging instrument (relative to the initial spatial configuration) is used to generate the perception of the region moving in an upward direction (image of the region moving upward in the illustration of the display output).

[0101] While the illustrations of FIG. 6 A are limited to translational and rotational movements in a two-dimensional space, the illustrations are intended to generalize to a full six degree of freedom control scheme and may include sequential or simultaneous execution of combinations of translational and rotational movements. While not shown, the control scheme may include additional operations such as a scaling between the user input and the resulting movement. A description of the operations that enable the view content-based steering as illustrated in FIG. 6A is subsequently provided. Further, while, for illustrative purposes, an image of a region is shown as moving in the display output provided to the user, it is the entire view content that updates along with the image of the region, based on the user inputs.

[0102] Referring back to FIG. 5B, the control scheme 550 may produce the behavior as illustrated in FIG. 6A when the imaging instrument control filter 520 is active.Alternatively, the control scheme may produce other behaviors when the imaging instrument control filter 520 is bypassed. Being able to switch between different control behaviors may be beneficial, for example, when using the same input device to steer different types of instruments. For example, the view content-based steering may be used to enable a user to control the imaging instrument through manipulation of the input device. Alternatively, the steering when bypassing imaging instrument control filter may be used to control other instruments. Different control behaviors may further be used for steering of the imaging instrument, e.g., based on user preference. A switch as shown in FIG. 5B makes the imaging instrument control filter selectively activable, thereby enabling switching between control behaviors.

[0103] FIG. 6B shows the imaging instrument control filter 520 in accordance with embodiments of the disclosure. The desired motion Xdes at the input of the imaging instrument control filter 520, in some embodiments, is provided in the form of values describing position and orientation of the input device. Sets of these values may be provided at a fixed rate, and a set of values received by the imaging instrument control filter 520 may represent the position and orientation of the input device at a point in time. The imaging instrument control filter may use Xdes to determine the desired imaging instrument motion Xstate. The desired imaging instrument motion Xstate at the output of the imaging instrument control filter 520 may then serve as the input to the robotic controller 510.

[0104] In one or more embodiments, Xstate may be determined from Xdes using the combination of coordinate transformations Tstate= Tworid atc / ied)Tde1sTworid atc / ied), where:Tdes is the input matrix based on xdes (More generally, Tdes may be understood as an input command obtained from an input device (e.g., from the input portion 300). The input command may include a position and an orientation. The input command may be adjusted, e.g., by applying offsets, gains, etc.);Tworid(iatched) is the transform between a reference frame representing the spatial configuration of the input device at the time of latching (i.e., Tdes at the time of latching) and a reference frame associated with the environment or the world (e.g., manipulator structure base reference frame 414) as observed using the imaging instrument (Accordingly, TWOrid(iatched) may be understood as a snapshot of the positionand orientation of the imaging device in the reference frame (e.g., a cartesian reference frame) used by the robotic controller 510 at its input to control movement of the imaging instrument. Upon entrance to image control mode, Tdes = Tworld(latched) and hence the commanded cartesian position and orientation Tstate = Tdes. As a result, the imaging device remains steady (does not jump) when entering the image control mode because the commanded cartesian position and orientation is the current position and orientation.);Tstate is the output matrix from which Xstate may be directly obtained (In other words, Tstate is the output of the imaging instrument control filter 520. It may serve as the command (e.g., in the form of a cartesian position and orientation) to the robotic controller 510.); andGenerally, Tbs a matrix [x, R] that includes a position x (position vector) and an orientation R (rotation matrix).

[0105] As previously noted, a latching operation is performed to determine Tworid(iatched) . The latching operation, in one or more embodiments, is used to establish an alignment relationship (position, orientation) between the input device and the environment or world as observed through the imaging instrument. The alignment relationship is set such that, at the point in time when the alignment relationship is determined, the imaging instrument control filter 520 would output an Xstate that exactly corresponds to the current spatial configuration of the imaging instrument. Accordingly, no corrective movement of the imaging instrument would be caused if the imaging instrument control filter 520 and the robotic controller 510 were executed at the time of the latching. During operation of the imaging instrument control filter 520 (after performing the latching operation), an Xdes that deviates from the Xdes at the time of the latching operation results in an Xstate that differs from the current spatial configuration of the imaging instrument, thereby causing a driving of the imaging instrument towards Xstate^ by the robotic controller 510.

[0106] An alignment relationship determined when performing a latching operation may be established in any number of degrees of freedom, e.g., in all six degrees of freedom. A first alignment relationship may initially be established when a latching operation is performed for the first time.

[0107] A latching operation may be triggered by certain events. For example, a latching operation may be triggered whenever entering the imaging instrument movement mode to control the imaging instrument by user input provided at the input device. The usermay signal the intention to enter the imaging instrument movement mode by operating a control element such as a button, a foot pedal, a software switch, etc.

[0108] After the entry into the imaging instrument movement mode, the alignment relationship may be held static, even as the imaging instrument with the field of view moves (including translation and / or rotation) in response to the desired motion that reflects the user input at the input device. The alignment relationship may be held static until an exit from the imaging instrument movement mode. For example, referring to FIG. 6A, the three user inputs to move the field of view with desired motions may result in three motion commands, each in accordance with the alignment relationship. An exit of the imaging instrument movement mode may be user-instructed and may occur for any reason, for example, when the user decides to control a different instrument, decides to take a break from controlling the imaging instrument to move the view reference frame, etc.

[0109] After the exit from the imaging instrument movement mode, another entry into the imaging instrument movement mode may be detected, e.g., when the user chooses to resume moving the view reference frame. This may trigger the establishment of another alignment relationship, which is independent from the previously used alignment relationship. The resulting motion commands after the renewed entry into the imaging instrument movement as the user continues to provide input to move the field of is then in accordance with the newly obtained alignment relationship.

[0110] As a result of the independence of the alignment relationships before and after exit and re-entry of the imaging instrument movement mode, the user may use the exit and re-entry to re-position and / or re-orient the input device as desired relative to the field of view. This may enable the user to change the hand posture for more ergonomic operation, to re- center the input device when close to the edge of the input device’s workspace, etc. While not illustrated, the alignment relationship may implement additional mapping characteristics such as a scaling, without departing from the disclosure.

[0111] Other alignment relationships may further be used. For example, an alignment relationship that enables a steering of the field of view in the manipulator structure base reference frame may be used.

[0112] Also, another alignment relationship may further be used in conjunction with a second instrument different from the imaging instrument. In such an alignment relationship, a one-to-one mapping between degrees of freedom of the input device and respective degrees of freedom of the second instrument may be used, in which both the degrees of freedom of the input device and the degrees of freedom of the second instrument are described in theview reference frame. Such an alignment relationship may be applied to any number of degrees of freedom. Referring to FIG. 2A, the alignment relationship may be used to control the instruments 260 and 270. The alignment relationship may further be used to control other (secondary) imaging instruments such as ultrasound probes, additional endoscopes, etc.

[0113] While not shown in FIG. 6, the imaging instrument control filter 520 may perform additional operations such as transformations into one or more different reference frames to provide the desired imaging instrument motion in a reference frame suitable as an input to the robotic controller 510. When these operations by the imaging instrument control filter 520 are performed at a sufficiently high rate, the behavior as shown in FIG. 6A is obtained. A user may, thus, be able to smoothly control the imaging instrument in a manner that is intuitive when steering the imaging instrument while relying on visual feedback in the form of images provided by the imaging instrument.

[0114] As discussed in reference to FIGs. 5A and 5B, the desired imaging instrument motion generated by the imaging instrument control filter 520 serves as an input to the robotic controller 510. In the previously introduced implementation that relies on a first kinematic controller and a second kinematic controller, the desired imaging instrument motion of the imaging instrument control filter may be provided as input to both the first kinematic controller and the second kinematic controller. While the first kinematic controller may receive the desired imaging instrument motion desired imaging instrument motion in all six degrees of freedom, the second kinematic controller may receive the aligned motion in the three rotational degrees of freedom only. In addition, the desired imaging instrument motion may undergo one or more transformations between reference frames, prior to being provided to the first and second kinematic controllers, and the desired imaging instrument motion may be provided to the first and second kinematic controllers in different reference frames.

[0115] The operations described in reference to FIGs. 6A and 6B may be performed by any type of the previously described manipulator assemblies and other manipulator assemblies. Specifically referring to the manipulator assembly 200 of FIG. 2A, if such a manipulator assembly is used, the motion command may include a commanded roll component of the manipulator-supporting link 202 when the desired motion includes a desired roll component of the field of view about the view axis of the field of view, thereby enabling execution of the commanded roll component, even when the imaging instrument lacks any joints operable to provide the desired roll component of the field of view.

[0116] In such a configuration, the roll command component of the motion command may be provided in the manipulator structure base reference frame, whereas all othercommand components (i.e., the non-roll command components) that are a result of may be provided in the view reference frame. Assuming, for example, that the desired motion includes a desired non-roll component with a rotation about an axis perpendicular to the view axis, the non-roll-command component of the motion command is in the view reference frame, whereas the roll command component is in the manipulator structure base reference frame.

[0117] Because, as previously described in reference to FIG. 2A, roll of the manipulator-supporting link, without further compensation, would result in a potentially undesired movement of another instrument (or multiple instruments) supported by the manipulator-supporting link, an additional motion command for the instrument joints of the other instrument(s) may be determined such that the undesired movement resulting from the roll of the manipulator-supporting link is compensated for. The compensation may be such that a position or orientation of a distal portion of the other instrument(s) is maintained during the performance of the commanded roll component. Alternatively, the compensation may be such that a position or orientation of the distal portion of the other instrument(s) is restored after the performance of the commanded roll component.

[0118] Turning to FIG. 7, a flowchart in accordance with one or more embodiments is shown. The flowchart of FIG. 7 depicts a method 700 for computer-assisted systems. The method 700 may be used for steering an imaging instrument. One or more of the steps in FIG. 7 may be performed by various components of systems, previously described with reference to FIGs. 1 A, IB, 2A, 2B, 3, 4, 5A, and 5B. While these figures illustrate particular configurations of computer assisted systems, the method is equally applicable to other configurations. On a high level, the described steps may be performed for a computer assisted system that includes a manipulator structure with a plurality of manipulator links that are coupled by a plurality of manipulator joints. The manipulator structure may support an imaging instrument that includes a plurality of instrument links coupled by at least one instrument joint. The imaging instrument has a field of view that defines a view reference frame. When the manipulator structure is supporting the imaging instrument, the manipulator structure and the imaging instrument together comprise an aggregate plurality of links coupled by an aggregate plurality of joints, the aggregate plurality of links comprising the plurality of manipulator links and the plurality of instrument links, and the aggregate plurality of joints comprising the plurality of manipulator joints and the at least one instrument joint.

[0119] The method may be executed on one or more processors, e.g., of the computing system(s) 172 of the computer-assisted system.

[0120] While the various steps in the flowchart are presented and described sequentially, one of ordinary skill will appreciate that some or all of the steps may be executed in different orders, may be combined or omitted, and some or all of the steps may be executed in parallel. Additional steps may further be performed. Furthermore, the steps may be performed actively or passively. For example, some steps may be performed using polling or be interrupt driven in accordance with one or more embodiments of the disclosure.

[0121] While not shown, the operations of the method 700 may be executed in a loop, thereby enabling repeated execution of the imaging instrument control filter, as previously described. The loop may be executed in real-time and at a frequency sufficient to enable a user to operate the systems as described with little to no perceivable delay. As a result, a user receiving visual content based on the field of view of the imaging instrument may get the impression that movement of the visual content is directly controllable by their user input. For example, a user input to the right will always cause a shift of the visual content, e.g., an image of a region of the environment surrounding the imaging instrument as seen by the user through display unit 154, based on the behavior illustrated in FIG. 6A. A positional correspondence is thus continuously maintained, unlike in joystick control methods where the correspondence is in the velocity domain. At least some steps of the method 700 may be executed while in an imaging instrument movement mode. Other steps may be executed when not in the imaging instrument movement mode. Furthermore, additional operations that are not shown in the flowchart may be performed when in the imaging instrument movement mode, without departing from the disclosure.

[0122] Turning to the method 700, in Block 702, a user input to move an image of a region in the field of view with a desired motion is received from an input device, as previously described.

[0123] In Block 704, a motion command is determined based on the desired motion. The motion command is determined such that it moves the imaging instrument with its field of view in accordance with a first alignment relationship. The first alignment relationship includes a one-to-one mapping between first, second, and third translational degrees of freedom of the input device in an input device base reference frame and respective first, second, and third translational degrees of freedom of the image of the region in the view in the view reference frame. The first alignment relationship further includes a one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third rotational degrees of freedom of the image of the region in the view reference frame. As a result of the reversecorrespondence between a movement of the field of view relative to the region and the movement of the image of the region in the field of view, a second alignment relationship is further applicable when determining the motion command. The second alignment relationship establishes a directionally reversed one-to-one mapping between the first, second, and third translational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third translational degrees of freedom of the field of view in a manipulator structure base reference frame fixed relative to the proximal base, and a directionally reversed one-to-one mapping between the first, second, and third rotational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third rotational degrees of freedom of the field of view in the manipulator structure base reference frame. The determining of the motion command may be performed as previously described in reference to FIGs. 5A, 5B, 5C, 6A, and 6B.

[0124] Other alignment relationships may further be used. For example, a motion command may also be determined using an alignment relationship unlike the above alignment relationship. Such an alternative alignment relationship may establish a one-to-one mapping between a first translational degree of freedom of the input device in the input device base reference frame and a first translational degree of freedom of the field of view in a manipulator structure base reference frame fixed relative to the proximal base and a one-to- one mapping between a first rotational degree of freedom of the input device in the input device base reference frame and a first rotational degree of freedom of the field of view in the manipulator structure base reference frame. Another alignment relationship may establish a one-to-one mapping between the first, second, and third translational degrees of freedom of the input device in the view reference frame and respective first, second, and third translational degrees of freedom of an instrument in the view reference frame, and a one-to- one mapping between the first, second, and third rotational degrees of freedom of the input device in the view reference frame and respective first, second, and third rotational degrees of freedom of the instrument in the reference view frame.

[0125] In Block 706, the aggregate plurality of joints (i.e., the joints of the manipulator structure and the joints of the imaging instrument) is commanded based on the motion command.

[0126] With the execution of the motion command, the spatial position and / or orientation of the imaging instrument may adjust such that the image of the region in the field of view moves in accordance with the user input.

[0127] While the execution of blocks 702-706 is described for a first imaging instrument movement mode that relies on a first alignment relationship, blocks 702-706 may also be performed for a second imaging instrument movement mode that relies on a second alignment relationship that is independent from the first alignment relationship. Furthermore, operations for moving additional instruments (different from the imaging instrument) may be performed, either simultaneously or non-simultaneously with the execution of blocks 702- 706.

[0128] Furthermore, in some embodiments, haptic feedback may be provided to the user operating the input device to move the field of view. The haptic feedback may be designed to inform the user of motion limitations associated with moving the image of the region in the field of view. Such constraints may be hardware or physical constraints (e.g., collisions of the imaging instrument with objects in the environment or range of motion limitations), or software limitations. To ensure that the feedback provided to the user is informative and easy to interpret, the alignment relationship used for the control of the imaging instrument is also applied to determine the feedback command for providing haptic feedback to the user at the input device in presence of the motion limitation.

[0129] The haptic feedback may, thus, indicate the inhibiting of the continuation of the movement towards the motion limitation to the user. For example, a virtual wall may be implemented by the haptic feedback, thereby preventing or at least inhibiting the user from continuing commanding a movement of the imaging instrument.

[0130] Although only a few example embodiments have been described in detail above, those skilled in the art will readily appreciate that many modifications are possible in the example embodiments without materially departing from this invention. Accordingly, all such modifications are intended to be included within the scope of this disclosure as defined in the following claims.

Claims

CLAIMSWhat is claimed is:

1. A computer-assisted system comprising: an input device configured to be manipulated by a user; a display device configured to display images, the images being of a field of view of an imaging instrument, the field of view defining a view reference frame; a manipulator structure comprising a plurality of manipulator links extending from a proximal base, the plurality of manipulator links coupled by a plurality of manipulator joints, the manipulator structure configured to support the imaging instrument, the imaging instrument comprising a plurality of instrument links coupled by at least one instrument joint, wherein when the manipulator structure is supporting the imaging instrument, the manipulator structure and the imaging instrument together comprise an aggregate plurality of joints, the aggregate plurality of joints comprising the plurality of manipulator joints and the at least one instrument joint; and a control system comprising one or more processors, the control system configured to, in an imaging instrument movement mode: receive, from the input device, a user input to move an image of a region in the field of view with a desired motion; determine, based on the desired motion, a motion command that moves the imaging instrument such that the image of the region in the field of view moves in accordance with a first alignment relationship, wherein the first alignment relationship establishes: a first one-to-one mapping between first, second, and third translational degrees of freedom of the input device in an input device base reference frame and respective first, second, and third translational degrees of freedom of the image of the region in the view reference frame, and a first one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third rotational degrees of freedom of the image of the region in the view reference frame; andcommand the aggregate plurality of joints based on the motion command.

2. The computer-assisted system of claim 1, wherein to determine the motion command, the control system is configured to use a second alignment relationship, wherein the second alignment relationship establishes: a directionally reversed one-to-one mapping between the first, second, and third translational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third translational degrees of freedom of the field of view in a manipulator structure base reference frame fixed relative to the proximal base, and a directionally reversed one-to-one mapping between the first, second, and third rotational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third rotational degrees of freedom of the field of view in the manipulator structure base reference frame.

3. The computer-assisted system of claim 1, wherein the control system is further configured to: determine the first alignment relationship at a first entry into the imaging instrument movement mode.

4. The computer-assisted system of claim 3, wherein the control system is further configured to: hold the first alignment relationship static even as the imaging instrument moves.

5. The computer-assisted system of claim 4, wherein to hold the first alignment relationship static even as the imaging instrument moves, the control system is configured to: hold the first alignment relationship static even as the field of view rotates.

6. The computer-assisted system of claim 3, wherein the control system is further configured to: hold, from the first entry into the imaging instrument movement mode until a first exit from the imaging instrument movement mode, the first alignment relationship static even as the field of view rotates.

7. The computer-assisted system of claim 3, wherein the control system is further configured to: determine, at a second entry into the imaging instrument movement mode and independently of the first alignment relationship, a second alignment relationship, wherein the second entry occurs after the first entry.

8. The computer-assisted system of claim 3, wherein to determine the first alignment relationship, the control system is configured to: obtain, at the first entry, a spatial configuration of the input device; obtain, at the first entry, a spatial configuration of the field of view; and determine, based on the spatial configuration of the input device and the spatial configuration of the field of view, the first alignment relationship.

9. The computer-assisted system of claim 8, wherein to determine the first alignment relationship, the control system is configured to: at the first entry, identify the spatial configuration of the imaging device in a reference frame used to represent the motion command.

10. The computer-assisted system of any of claims 1 to 9, wherein the user input is a first user input, the desired motion is first desired motion, wherein the motion command is a first motion command, and wherein the control system is further configured to: receive, from the input device and after the first desired motion has been received, a second user input to move the image of the region in the field of view with a second desired motion; determine, based on the second desired motion, a second motion command that moves the imaging instrument such that the image of the region in the field of view moves in accordance with the first alignment relationship; andcommand the aggregate plurality of joints based on the second motion command.

11. The computer-assisted system of any of claims 1 to 9, wherein the imaging instrument movement mode is a first imaging instrument movement mode, the user input is a first user input, the desired motion is a first desired motion, the motion command is a first motion command, and wherein the control system is further configured to, in a second imaging instrument movement mode: receive, from the input device, a second user input to move the field of view with a second desired motion; and determine, based on the second desired motion, a second motion command that moves the field of view in accordance with a second alignment relationship; wherein the second alignment relationship does not establish: a one-to-one mapping between first, second, and third translational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third translational degrees of freedom of the image of the region in the view reference frame, and a one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third rotational degrees of freedom of the image of the region in the view reference frame.

12. The computer-assisted system of claim 11, wherein the second alignment relationship establishes: a second one-to-one mapping between a first translational degree of freedom of the input device in the input device base reference frame and a first translational degree of freedom of the field of view in a manipulator structure base reference frame fixed relative to the proximal base; and a second one-to-one mapping between a first rotational degree of freedom of the input device in the input device base reference frame and a first rotational degree of freedom of the field of view in the manipulator structure base reference frame.

13. The computer-assisted system of any of claims 1 to 9, wherein the manipulator structure comprises a manipulator-supporting link having a roll axis;wherein the desired motion comprises a desired roll component of the image of the region in the field of view about a view axis of the field of view; and wherein the motion command comprises a roll command component of the manipulator-supporting link about the roll axis.

14. The computer-assisted system of claim 13, further comprising the imaging instrument, wherein the imaging instrument lacks any joints operable to provide the desired roll component of the field of view.

15. The computer-assisted system of claim 13, wherein the motion command comprises: the roll command component in a base reference frame fixed relative to the proximal base; and a non-roll command component in the view reference frame.

16. The computer-assisted system of claim 15, wherein the motion command is a first motion command, and the imaging instrument is a first instrument; wherein the manipulator structure is further configured to support a second instrument, the second instrument comprising a plurality of second instrument links coupled by one or more second instrument joints; and wherein the control system is further configured to: determine a second motion command, for the one or more second instrument joints, to maintain a position or orientation of a distal portion of the second instrument during a performance of the roll command component or to restore the position or orientation of the distal portion of the second instrument after the performance of the roll command component; and command the one or more second instrument joints based on the second motion command.

17. The computer-assisted system of any of claims 1 to 9, wherein the user input is a first user input, the desired motion is a first desired motion, the motion command is a first motion command, the imaging instrument is afirst instrument, and the imaging instrument movement mode is a first instrument movement mode; and wherein the control system is further configured to, in a second instrument movement mode: receive, from the input device, a second user input to move a second instrument with a second desired motion; and determine, based on the second desired motion, a second motion command that moves the second instrument in accordance with a second alignment relationship; wherein the second alignment relationship establishes: a second one-to-one mapping between the first, second, and third translational degrees of freedom of the input device in the view reference frame and respective first, second, and third translational degrees of freedom of the second instrument in the view reference frame, and a second one-to-one mapping between the first, second, and third rotational degrees of freedom of the input device in the view reference frame and respective first, second, and third rotational degrees of freedom of the second instrument in the view reference frame.

18. The computer-assisted system of any of claims 1 to 9, wherein the input device is configured for operation by a single hand of the user.

19. The computer-assisted system of any of claims 1 to 9, wherein the control system is further configured to, in the imaging instrument movement mode: detect a motion limitation experienced by the imaging instrument when performing the motion command; and determine, based on the motion limitation and in accordance with the first alignment relationship, a feedback command for providing haptic feedback to the user by the input device.

0. A method for view content-based steering of an imaging instrument associated with a computer-assisted system, wherein the computer-assisted comprises: an input device configured to be manipulated by a user; a display device configured to display images, the images being of a field of view of the imaging instrument, the field of view defining a view reference frame; a manipulator structure comprising a plurality of manipulator links extending from a proximal base, the plurality of manipulator links coupled by a plurality of manipulator joints, the manipulator structure configured to support the imaging instrument, the imaging instrument comprising a plurality of instrument links coupled by at least one instrument joint, wherein when the manipulator structure is supporting the imaging instrument, the manipulator structure and the imaging instrument together comprise an aggregate plurality of joints, the aggregate plurality of joints comprising the plurality of manipulator joints and the at least one instrument joint; and a control system; wherein the method, executed by the control system, comprises: when the control system is configured to operate in an imaging instrument movement mode: receiving, from the input device, a user input to move an image of a region in the field of view with a desired motion; determining, based on the desired motion, a motion command that moves the imaging instrument such that the image of the region in the field of view moves in accordance with a first alignment relationship, wherein the first alignment relationship establishes: a first one-to-one mapping between first, second, and third translational degrees of freedom of the input device in an input device base reference frame and respective first, second, and third translational degrees of freedom of the image of the region in the view reference frame, and a first one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base reference frame and respective first,second, and third rotational degrees of freedom of the image of the region in the view reference frame; and commanding the aggregate plurality of joints based on the motion command.

21. The method of claim 20, further comprising using a second alignment relationship to determine the motion command, wherein the second alignment relationship establishes: a directionally reversed one-to-one mapping between the first, second, and third translational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third translational degrees of freedom of the field of view in a manipulator structure base reference frame fixed relative to the proximal base, and a directionally reversed one-to-one mapping between the first, second, and third rotational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third rotational degrees of freedom of the field of view in the manipulator structure base reference frame.

22. The method of claim 20, further comprising: determining the first alignment relationship at a first entry into the imaging instrument movement mode.

23. The method of claim 22, further comprising: holding the first alignment relationship static even as the imaging instrument moves.

24. The method of claim 23, further comprising, to hold the first alignment relationship static even as the imaging instrument moves: holding the first alignment relationship static even as the field of view rotates.

25. The method of claim 22, further comprising: holding, from the first entry into the imaging instrument movement mode until a first exit from the imaging instrument movement mode, the first alignment relationship static even as the field of view rotates.

26. The method of claim 22, further comprising: determining, at a second entry into the imaging instrument movement mode and independently of the first alignment relationship, a second alignment relationship, wherein the second entry occurs after the first entry.

27. The method of claim 22, further comprising, to determine the first alignment relationship: obtaining, at the first entry, a spatial configuration of the input device; obtaining, at the first entry, a spatial configuration of the field of view; and determining, based on the spatial configuration of the input device and the spatial configuration of the field of view, the first alignment relationship.

28. The method of claim 27, further comprising, to determine the first alignment relationship: at the first entry, identifying the spatial configuration of the imaging device in a reference frame used to represent the motion command.

29. The method of any of claims 20 to 28, wherein the user input is a first user input, the desired motion is first desired motion, wherein the motion command is a first motion command, and wherein the method further comprises: receiving, from the input device and after the first desired motion has been received, a second user input to move the image of the region in the field of view with a second desired motion; determining, based on the second desired motion, a second motion command that moves the imaging instrument such that the image of the region in the field of view moves in accordance with the first alignment relationship; and commanding the aggregate plurality of joints based on the second motion command.

30. The method of any of claims 20 to 28, wherein the imaging instrument movement mode is a first imaging instrument movement mode, the user input is a first user input, the desired motion is a first desired motion, the motion command is a first motioncommand, and wherein the method further comprises when the control system is configured to operate in a second imaging instrument movement mode: receiving, from the input device, a second user input to move the field of view with a second desired motion; and determining, based on the second desired motion, a second motion command that moves the field of view in accordance with a second alignment relationship; wherein the second alignment relationship does not establish: a one-to-one mapping between first, second, and third translational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third translational degrees of freedom of the image of the region in the view reference frame, and a one-to-one mapping between first, second, and third rotational degrees of freedom of the input device in the input device base reference frame and respective first, second, and third rotational degrees of freedom of the image of the region in the view reference frame.

31. The method of claim 30, wherein the second alignment relationship establishes: a second one-to-one mapping between a first translational degree of freedom of the input device in the input device base reference frame and a first translational degree of freedom of the field of view in a manipulator structure base reference frame fixed relative to the proximal base; and a second one-to-one mapping between a first rotational degree of freedom of the input device in the input device base reference frame and a first rotational degree of freedom of the field of view in the manipulator structure base reference frame.

32. The method of any of claims 20 to 28, wherein the manipulator structure comprises a manipulator-supporting link having a roll axis; wherein the desired motion comprises a desired roll component of the image of the region in the field of view about a view axis of the field of view; and wherein the motion command comprises a roll command component of the manipulator-supporting link about the roll axis.

33. The method of claim 32, wherein the motion command comprises: the roll command component in a base reference frame fixed relative to the proximal base; and a non-roll command component in the view reference frame.

34. The method of claim 33, wherein the motion command is a first motion command, and the imaging instrument is a first instrument; wherein the manipulator structure is further configured to support a second instrument, the second instrument comprising a plurality of second instrument links coupled by one or more second instrument joints; and wherein the method further comprises: determining a second motion command, for the one or more second instrument joints, to maintain a position or orientation of a distal portion of the second instrument during a performance of the roll command component or to restore the position or orientation of the distal portion of the second instrument after the performance of the roll command component; and commanding the one or more second instrument joints based on the second motion command.

35. The method of any of claims 20 to 28, wherein the user input is a first user input, the desired motion is a first desired motion, the motion command is a first motion command, the imaging instrument is a first instrument, and the imaging instrument movement mode is a first instrument movement mode; and wherein the method further comprises when the control system is configured to operate in a second instrument movement mode: receiving, from the input device, a second user input to move a second instrument with a second desired motion; and determining, based on the second desired motion, a second motion command that moves the second instrument in accordance with a second alignment relationship; wherein the second alignment relationship establishes:a second one-to-one mapping between the first, second, and third translational degrees of freedom of the input device in the view reference frame and respective first, second, and third translational degrees of freedom of the second instrument in the view reference frame, and a second one-to-one mapping between the first, second, and third rotational degrees of freedom of the input device in the view reference frame and respective first, second, and third rotational degrees of freedom of the second instrument in the view reference frame.

36. The method of any of claims 20 to 28, wherein the input device is configured for operation by a single hand of the user.

37. The method of any of claims 20 to 28, further comprising, when the control system is configured to operate in an imaging instrument movement mode: detecting a motion limitation experienced by the imaging instrument when performing the motion command; and determining, based on the motion limitation and in accordance with the first alignment relationship, a feedback command for providing haptic feedback to the user by the input device.

38. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions executed by one or more processors associated with a computer-assisted system, the plurality of machine-readable instructions causing the one or more processors to perform the method of any of claims 20 to 37.

Images