Systems and methods for inter-arm registration

Abstract

Systems and methods for inter-arm registration include a computer-assisted system having a control unit coupled to a repositionable arm of a computer-assisted device. The control unit is configured to: receive, from an imaging device, successive images of an instrument mounted to the repositionable arm; determine an observed velocity of a feature of the instrument; determine an expected velocity of the feature of the instrument based on kinematics of the repositionable arm; transform the observed velocity and / or the expected velocity to a common coordinate system using a registration transform; determine an error between a direction of the observed velocity and a direction of the expected velocity in the common coordinate system; and update the registration transform based on the determined error. In some embodiments, the instrument is a medical instrument and the imaging device is an endoscope. In some embodiments, the control unit is further configured to control the instrument using the registration transform.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 957,537, filed January 6, 2020, which is incorporated by reference herein. TECHNICAL FIELD

[0003] The present disclosure relates generally to the operation of devices having instruments mounted to repositionable arms of computer-assisted devices, and more particularly to determining registration between instruments on different computer-assisted devices. BACKGROUND

[0004] Increasingly, devices are being replaced by computer-assisted electronic devices. This is especially true in industrial, entertainment, educational, and other environments. As a medical example, today’s hospitals find a large number of electronic devices in operating rooms, interventional suites, intensive care units, emergency rooms, and / or the like. For example, glass and mercury thermometers are being replaced by electronic thermometers, intravenous drip tubes now include electronic monitors and flow regulators, and traditional handheld surgical and other medical instruments are being replaced by computer-assisted medical devices.

[0005] These computer-assisted devices can be used to perform operations and / or procedures on materials located in a workspace, such as a patient’s tissue. When the workspace is separate from an operator who controls the computer-assisted device, the operator typically controls the computer-assisted device using teleoperation and monitors the computer-assisted device’s activities using an imaging device positioned to capture images or video of the workspace. In computer-assisted devices having instruments mounted to repositionable arms and / or manipulators, teleoperation typically involves the operator using one or more input controls to provide movement commands for the instruments, e.g., movement commands implemented by driving one or more joints in the respective repositionable arms and / or manipulators. In some computer-assisted devices, one of the instruments can be an imaging device mounted to a repositionable arm, such that the operator can change the orientation and / or direction of the imaging device’s field of view in order to be able to capture images of the workspace from different locations and orientations.

[0006] Because the operator relies on images of other instruments captured by the imaging device to teleoperate the other instruments, it is useful to understand the geometric relationship (e.g., registration) between the imaging device and the other instruments, such that movements of the operator’s hands relative to an eye coordinate system of the operator’s station can be properly translated into motion of the other instruments.

[0007] Accordingly, it would be advantageous to have methods and systems that determine registration between an imaging device and one or more instruments that are teleoperated. SUMMARY

[0008] Consistent with some embodiments, a computer-assisted system includes a control unit coupled to a repositionable arm of a computer-assisted device. The control unit is configured to: receive, from an imaging device, successive images of an instrument mounted to the repositionable arm; determine an observed velocity of a feature of the instrument; determine an expected velocity of the feature of the instrument based on kinematics of the repositionable arm; transform the observed velocity, the expected velocity, or both the observed velocity and the expected velocity to a common coordinate system using a registration transform; determine an error between a direction of the observed velocity and a direction of the expected velocity in the common coordinate system; and update the registration transform based on the determined error.

[0009] Consistent with some embodiments, a method of operating a computer-assisted system using a control unit includes: receiving, from an imaging device, successive images of an instrument mounted to a repositionable arm of a computer-assisted device; determining an observed velocity of a feature of the instrument; determining an expected velocity of the feature of the instrument based on kinematics of the repositionable arm; transforming the observed velocity, the expected velocity, or both the observed velocity and the expected velocity to a common coordinate system using a registration transform; determining an error between a direction of the observed velocity and a direction of the expected velocity in the common coordinate system; and updating the registration transform based on the determined error.

[0010] Consistent with some embodiments, a computer-assisted system includes a control unit coupled to a repositionable arm of a computer-assisted device. The control unit is configured to: receive, from an imaging device, successive images of an instrument mounted to the repositionable arm; determine an observed orientation of an alignment feature of the instrument; determine an expected orientation of the alignment feature of the instrument based on kinematics of the repositionable arm; transform the observed orientation, the expected orientation, or both the observed orientation and the expected orientation to a common coordinate system using a registration transform; determine an error between the observed orientation and the expected orientation in the common coordinate system; and update the registration transform based on the determined error.

[0011] Consistent with some embodiments, a method of operating a computer-assisted system using a control unit includes: receiving, from an imaging device, successive images of an instrument mounted to a repositionable arm of a computer-assisted device; determining an observed orientation of an alignment feature of the instrument; determining an expected orientation of the alignment feature of the instrument based on kinematics of the repositionable arm; transforming the observed orientation, the expected orientation, or both the observed orientation and the expected orientation to a common coordinate system using a registration transform; determining an error between the observed orientation and the expected orientation in the common coordinate system; and updating the registration transform based on the determined error.

[0012] Consistent with some embodiments, a non-transitory machine-readable medium includes a plurality of machine-readable instructions that, when executed by one or more processors, are adapted to cause the one or more processors to perform any of the methods described herein. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a simplified diagram of a computer-assisted system according to some embodiments.

[0014] Figure 2 is a simplified diagram of a control system according to some embodiments.

[0015] Figure 3 is a simplified diagram of a registration method according to some embodiments.

[0016] Figures 4A-4C is a simplified diagram of a method according to some embodiments. Figure 3 of several processes of the method of

[0017] Figure 5 is a simplified diagram of a process of the method of Figure 3 according to additional embodiments.

[0018] In the drawings, elements having the same name share the same or similar function. DETAILED DESCRIPTION

[0019] This specification and the accompanying drawings should not be construed as limiting the aspects, embodiments, implementations, or modules of the invention— the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes can be made without departing from the spirit and scope of this specification and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure the application. Like reference numbers in two or more figures represent the same or similar elements.

[0020] In this specification, specific details of some embodiments consistent with the present disclosure are set forth to provide a thorough understanding of embodiments. Numerous specific details are set forth in order to provide a thorough understanding of embodiments. However, it will be apparent to those skilled in the art that embodiments can be practiced without some or all of these specific details. In some instances, well-known structures have not been described in detail in order to avoid obscuring the concepts of the present disclosure. Additionally, one or more features associated with one embodiment can be incorporated into other embodiments, unless specifically described otherwise or if the one or more features would make the embodiment unworkable.

[0021] Moreover, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. For example, spatially relative terms— such as "under", "below", "lower", "over", "upper", "proximal", "distal", and the like— can be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. The spatially relative terms are intended to encompass different positions (i.e., orientations) of the elements and / or their operations, and are not intended to be limited to specific positions (i.e., orientations) unless otherwise specifically noted. For example, an element described as "above" or "below" another element can be "above" or "below" the other element, as well as "above" or "below" the other element, as illustrated in the figures. Thus, the exemplary term "above" can encompass both positions and orientations above and below. The devices can be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes include various special device positions and orientations. Similarly, the terms "first", "second", "third", etc. are used herein not necessarily to denote different or successive stages or areas, but to identify particular elements, steps, or areas. Furthermore, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. And, the terms "comprises", "comprising", "includes", "including" and / or "contains", "containing", are intended to be open-ended terms that specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. Components described as coupled can be electrically or mechanically directly coupled, or they can be indirectly coupled via one or more intermediate components.

[0022] Elements described with reference to one embodiment, implementation, or module can be included in other embodiments, implementations, or modules as practicable, even if not specifically shown or described in those other embodiments, implementations, or modules. For example, if an element is described with reference to one embodiment and not described with reference to a second embodiment, the element can still be claimed to be included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application can be incorporated into other embodiments, implementations, or aspects, unless specifically described otherwise, unless one or more elements would render the embodiment or implementation inoperable, or unless two or more elements provide conflicting functionality.

[0023] In some cases, well-known methods, procedures, components, and circuits have not been described in detail since they are already widely known.

[0024] The present disclosure describes various devices, elements, and portions of computer-assisted devices and elements in terms of their states in three-dimensional space. As used herein, the term "position" refers to the location of an element or portion of an element in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an element or portion of an element (three rotational degrees of freedom— e.g., roll, pitch, and yaw, angular axis, rotation matrix, quaternion representation, and / or the like). As used herein, the term "shape" refers to a set position or orientation measured along an element. As used herein, and with respect to devices having repositionable arms, the term "proximal" refers to the direction along its kinematic chain toward the base of the computer-assisted device, and "distal" refers to the direction along the kinematic chain away from the base.

[0025] As used herein, the term "pose" refers to the six degrees of freedom (DOF) spatial position and orientation of a coordinate system of interest attached to a rigid body. As used herein, the term "feature" refers to fixed points, alignment axes, and / or similar geometric elements that are measured directly using an imaging device or derived from such measurements using a mathematical model.

[0026] As used herein, the term "observed feature" or "primary feature" refers to a feature that is measured or detected directly by an imaging device; and the term "derived feature," "secondary feature," or "computed feature" refers to a feature that is estimated or computed from measured features using a mathematical model. A collection of one or more features (primary and / or secondary features) is referred to as a feature set.

[0027] Aspects of the present disclosure are described with reference to computer-assisted systems and devices, which can include teleoperated, remotely controlled, autonomous, semi-autonomous, robotic, and / or similar systems and devices. Moreover, aspects of the present disclosure are described in terms of implementations using a surgical system, such as the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc., of Sunnyvale, California. However, those skilled in the art will understand that inventive aspects disclosed herein can be embodied and implemented in various ways, including robotic as well as, if applicable, non-robotic embodiments and implementations. da Surgical System. However, those skilled in the art will understand that inventive aspects disclosed herein can be embodied and implemented in various ways, including robotic as well as, if applicable, non-robotic embodiments and implementations. da The implementation methods described in the surgical system are merely exemplary and should not be construed as limiting the scope of the inventive aspects disclosed herein. For example, the techniques described with reference to surgical instruments and methods can be used in other contexts. Therefore, the instruments, systems, and methods described herein can be used with humans, animals, parts of human or animal anatomy, industrial systems, general robotics, or remote operating systems. As further examples, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial use, general robotic use, sensing or manipulating non-tissue artifacts, cosmetic enhancements, imaging of human or animal anatomy, collecting data from human or animal anatomy, setting up or dismantling systems, and training medical or non-medical personnel and / or similar individuals. Additional example applications include procedures for tissue removed from human or animal anatomy (without returning the human or animal anatomy) and procedures for human or animal cadavers. Furthermore, these techniques can also be used in medical treatment or diagnostic procedures, with or without surgical aspects.

[0028] Figure 1 This is a simplified diagram of a computer-aided system 100 according to some embodiments. For example... Figure 1 As shown, the computer-aided system 100 includes two computer-aided devices 110 and 120. Computer-aided device 110 includes a repositionable structure with a repositionable arm 112 configured to support an instrument. This instrument can be an imaging instrument, a manipulation instrument, a flushing or aspiration instrument, or any other suitable instrument. Figure 1 In some examples, the instrument includes an imaging device 114. In some examples, the instrument includes an imaging device 114, which is a single-view or stereo camera, a still or video camera, an endoscope, a hyperspectral device, an infrared or ultrasonic device, an ultrasound device, a fluorescence microscope device, and / or the like. Similarly, the computer-aided device 120 includes a repositionable structure having a repositionable arm 122 configured to support the instrument 124.

[0029] In some examples, instrument 124 can be an imaging device, a non-imaging device, and / or the like. In some medical examples, the instrument can be a medical instrument, such as forceps, a clip applier, a clamp, a retractor, a cauterizing instrument, an aspiration instrument, a suturing device, an anastomosis device, a cutting device, and / or the like. In some examples, instrument 124 can include an end effector capable of performing multiple tasks, such as both grasping material (e.g., tissue of a patient) located in the workspace and delivering energy to the grasped material. In some examples, the energy can include ultrasound, radiofrequency, electrical, magnetic, thermal, optical, and / or other types of energy. In some examples, each of repositionable arm 112, repositionable arm 122, imaging device 114, and / or instrument 124 can include one or more joints. In some medical examples, imaging device 114 and / or instrument 124 can be inserted into a workspace (e.g., an anatomical body of a patient or cadaver, a veterinary subject, an anatomical model, and / or the like) through a respective cannula, access port, and / or the like. In some embodiments, computer-assisted system 100 can be found in an operating room and / or an interventional suite.

[0030] Figure 1 The field of view 130 of imaging device 114 is also shown by indicating an area within the workspace in which imaging device 114 can capture images of the workspace and objects and events within the workspace. In the illustrated configuration, at least a distal portion of instrument 124 is located within field of view 130, allowing imaging device 114 to capture images of at least the distal portion of instrument 124 when instrument 124 is not completely occluded visually from imaging device 114. According to some embodiments and as further described below, the images of instrument 124 obtained by imaging device 114 can be used to determine a registration transform between computer-assisted device 110 and computer-assisted device 120. Figure 1

[0031] Figure 1 ​Various coordinate systems useful in describing how to determine registration transformations are also shown. Coordinate system b0 corresponds to a base coordinate system of computer-assisted device 110. In some examples, coordinate system b0 can have its origin at a center point on a base of computer-assisted device 110 and can be aligned with one or more principal axes of computer-assisted device 110. In some examples, the center point on the base of computer-assisted device 110 can be on a horizontal, flat surface on which computer-assisted device 110 can be wheeled, slid, and / or otherwise repositioned. In some examples, the z-axis of coordinate system b0 can correspond to a vertically upward direction. Similarly, coordinate system b1 corresponds to a base coordinate system of computer-assisted device 120. In some examples, coordinate system b1 can have its origin at a center point on a base of computer-assisted device 120 and can be aligned with one or more principal axes of computer-assisted device 120. In some examples, the center point on the base of computer-assisted device 120 is on a horizontal, flat surface on which computer-assisted device 120 can be wheeled, slid, and / or otherwise repositioned. In some examples, the z-axis of coordinate system b1 can correspond to a vertically upward direction. In some embodiments, coordinate systems b0 and / or b1 need not necessarily be on the same or parallel planes, such as in the case where one or more of computer-assisted devices 110 and / or 120 are mounted on a table, wall, ceiling, and / or the like.

[0032] Figure 1 An imaging device coordinate system c (e.g., for a camera) is also shown, which can be used to describe the position and / or orientation of objects (e.g., the distal end of instrument 124) in images captured by imaging device 114. In some examples, the origin of coordinate system c can be at the distal end of imaging device 114. In some examples, the origin of coordinate system c can be at the midpoint of an imaging plane associated with imaging device 114. In some examples, the z-axis of coordinate system c can be oriented in the viewing direction of imaging device 114. In some examples, imaging device 114 can be a steerable imaging device and / or a flexible continuum robot-like imaging device, such that coordinate system c can be located at or near the distal tip of the imaging device. In some examples, coordinate system c can not be attached to a physical point and can be rigidly or otherwise virtually coupled to a reference point on the imaging device.

[0033] Figure 1 An instrument coordinate system t (e.g., for instrument 124) is also shown, which can be used to describe the position and / or orientation of a population of points on or near instrument 124 relative to a known fixed point on instrument 124. In some examples, one or more kinematic models of computer-assisted device 120 and / or instrument 124 can be used to determine instrument coordinate system t relative to base coordinate system b lThe orientation of the instrument 124. In some examples, the fixed point may correspond to the distal end of the instrument 124. In some examples, the fixed point may correspond to easily identifiable point features on the instrument 124, such as reference points, indicative markings (including markings for indicating instrument type or origin); representative structures, such as tool tips or U-shaped clamps about which one or more jaws and / or links of the instrument 124 can rotate; points and / or similar points at a predetermined displacement relative to any of the aforementioned fixed points. In some examples, the principal axis of the coordinate system t may correspond to an alignment feature of the instrument 124. The alignment feature may be a real or virtual line of symmetry, a line associated with the principal rotational and / or translational motion of the instrument 124, a line integral with the shape of the instrument 124, and / or similar. In some examples, the alignment feature may be an axis defined from a reference to an indicating axis, an indicative mark indicating the axis, the axis of the instrument 124, an axis corresponding to a unique structure (e.g., located between the two jaws of the instrument 124 and oriented with the gripping surfaces of the jaws when the jaws are closed), an axis formed by line segments connecting any of the aforementioned fixed points, a virtual axis relative to a predefined starting point of any of the aforementioned alignment features or axes, and / or the like. In some examples, the coordinate system t may not be attached to a physical point and may be rigidly or otherwise virtually coupled to a reference point and / or fixed point on the instrument 124.

[0034] like Figure 1 As further illustrated in the embodiments, the geometric relationship between coordinate systems b0 and b1 is altered because the bases of both computer-aided devices 110 and 120 can move and / or orient relative to each other and independently (e.g., they have independent kinematic chains). In some examples, the geometric relationship between coordinate systems b0 and b1 can be achieved using a 6-DOF registration transformation. To characterize it. As shown in Equation 1, the registration transformation Includes a 3-DOF rotational section and 3DOF translation part Rotating part It is a 3x3 matrix describing the 3D rotation difference between coordinate systems b0 and b1. In some examples, the rotation part... This describes the rotation about three axes of the coordinate system. In some examples, the three axes may correspond to the x, y, and z axes; roll, pitch, and yaw axes, and / or similar. In some examples, the rotation part may be represented using angular axes, quaternions, and / or similar equivalent notations. Translation part It is a 3x1 vector describing the 3D displacement between coordinate systems b0 and b1. In some examples, the registration transformation... It can be used to determine the complete transformation from the instrument coordinate system t to the imaging device coordinate system c (e.g.,c T t ). In some examples, the registration transform can be defined as a correction to the assumed imaging device coordinate system c (e.g., c ′T c ), where c' is a corrected coordinate system of the imaging device 114 that aligns the motion of the instrument 124 and the motion of one or more input controls used to control the instrument 124.

[0035]

[0036] In some embodiments, when the computer-assisted devices 110 and 120 are located on a common planar surface and have a common vertical upward direction (e.g., coordinate system b0 and coordinate system b1 are both located on the same horizontal reference plane and have the same vertical upward axis), the reference transform can be simplified to a single rotation about the vertical upward axis and can be simplified to a two-dimensional (2D) lateral translation between coordinate system b0 and coordinate system b1. In some examples, when the bases of the computer-assisted devices 110 and 120 are located on parallel planes, each plane being orthogonal to the vertical upward axis and separated by a distance, the reference transform can be simplified to a single rotation about the vertical upward axis and can be a three-dimensional (3D) translation vector.

[0037] In some embodiments requiring determination of a 6DOF registration transform, the inclination or elevation angle between the bases of the computer-assisted devices 110 and 120 can be known through one or more inclinometers and / or accelerometers, thus reducing the number of DOFs of the registration transform that need to be computed by other means.

[0038] Computer-assisted device 110 and computer-assisted device 120 are both coupled to control unit 140 via respective interfaces. Each of the respective interfaces can include one or more cables, connectors, and / or buses, and can also include one or more networks with one or more network switching and / or routing devices. Control unit 140 includes a processor 150 coupled to a memory 160. The operation of control unit 140 is controlled by processor 150. And although control unit 140 is shown with only one processor 150, it should be understood that processor 150 can represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), tensor processing units (TPUs), and / or the like in control unit 140. Control unit 140 can be implemented as a standalone subsystem and / or as an add-on to a board of a computing device or as a virtual machine.

[0039] Memory 160 can be used to store software executed by control unit 140 and / or one or more data structures used during the operation of control unit 140. Memory 160 can include one or more types of machine-readable media. Some common forms of machine-readable media can include floppy diskettes, flexible disks, hard disks, magnetic tapes, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium from which a processor or computer is adapted to read.

[0040] As shown, memory 160 includes a control module 170 responsible for controlling one or more aspects of the operation of computer-assisted device 110 and / or computer-assisted device 120, including, for example, controlling movement and / or operation of each of computer-assisted device 110, computer-assisted device 120, repositionable arm 112, repositionable arm 122, imaging device 114, and / or instrument 124; determining registration transforms between computer-assisted device 110 and computer-assisted device 120, and / or the like, as described in more detail below. And although control module 170 is characterized as a software module, control module 170 can be implemented using software, hardware, and / or a combination of hardware and software.

[0041] As described above and further emphasized here, Figure 1Just one example, which should not unduly limit the scope of the claims, is provided. One of ordinary skill in the art will recognize a number of variations, alternatives, and modifications. According to some embodiments, the computer-assisted system 100 can also include one or more operator stations having respective one or more input controls for determining desired motions of the computer-assisted device 110, the computer-assisted device 120, the repositionable arm 112, the repositionable arm 122, the imaging device 114, and / or the instrument 124. In some examples, the operator stations are in the form of a control console. In some examples, a single operator station can be shared between the computer-assisted device 110 and the computer-assisted device 120. In some examples, the computer-assisted device 110 and the computer-assisted device 120 can have separate operator stations. In some examples, the one or more operator stations can be coupled to the computer-assisted device 100, the computer-assisted device 120, and / or the control unit 140 using respective interfaces. Each of the respective interfaces can include one or more cables, connectors, and / or buses, and can also include one or more networks having one or more network switching and / or routing devices.

[0042] According to some embodiments, the computer-assisted system 100 can include any number of computer-assisted devices having corresponding repositionable arms and / or instruments that are similar in design to or different from the computer-assisted device 110 and / or the computer-assisted device 120. In some examples, each computer-assisted device can include two, three, four, or more repositionable arms and / or instruments.

[0043] According to some embodiments, each of the computer-assisted devices 110 and / or 120 can have a separate control unit similar to the control unit 140. In some examples, each of the separate control units can be coupled to each other via an interface. The interface can include one or more cables, connectors, and / or buses, and can also include one or more networks with one or more network switching and / or routing devices. In some examples, the separate control units can use the interface to share and / or exchange information about their respective computer-assisted devices. In some examples, the exchanged information can include kinematic information about the position of one or more joints in the respective repositionable arms 112 and / or 122 and / or the respective imaging devices 114 and / or 124; kinematic information about the position and / or orientation of the imaging devices 114 and / or instruments 124, and / or the like. In some examples, the exchanged information can include one or more sensor readings, mode indications, interrupts, state transitions, events, fault conditions, safety warnings, and / or the like. In some examples, the exchanged information can include one or more images captured by the imaging devices 114, information derived from the one or more images, calibration and / or configuration parameters specific to the instruments 124 and / or imaging devices 114, and / or the like.

[0044] According to some embodiments, the imaging device 114 can instead be mounted to a fixture or be a handheld imaging device and not be mounted to a repositionable arm of a computer-assisted device. In some examples, when the imaging device 114 is not mounted to a repositionable arm of a computer-assisted device, the coordinate system b0may be the same coordinate system as coordinate system c.

[0045] In computer-assisted systems with multiple separate and / or modular computer-assisted devices and repositionable arms that do not share a common known basis (e.g., as shown in the embodiment of Figure 1 With multiple computer-assisted devices and / or repositionable arms that do not share a common known basis (e.g., as shown in the embodiment of ) to establish registration. In some embodiments, this registration difficulty also exists when using a handheld imaging device and the base coordinate system of the imaging device is not bound to a base of the repositionable structure (e.g., the base coordinate system of the imaging device can be bound to the imaging device itself). In some embodiments, the registration transform can be used for collision avoidance between computer-assisted devices, overlaying user interface elements on captured images, coordinating motion of multiple computer-assisted devices (e.g., for autonomous task sequences), and / or the like.

[0046] According to some embodiments, iteratively establishing a registration transform by using the imaging device to monitor motion of the instrument and using the monitoring to continually refine the registration transform provides several advantages over other methods for registering computer-assisted devices. In some examples, because the refinement is iterative, the resulting registration transform need not be static and can be updated as the instrument is used, the computer-assisted device is repositioned, and / or the like. In some examples, iterative registration can optionally be used to avoid having to pre-exercise the instrument through a set of known movements to determine the registration transform when in the field of view of the imaging device. In some examples, one of the goals of iterative registration is to reduce error in the rotational portion of the registration transform (e.g., ) to below 20 degrees, and preferably below 10 degrees or even 5 degrees. In some examples, one of the goals of iterative registration is to reduce error in the translational portion of the registration transform (e.g., ) to below 5 centimeters, and in some examples below 2 centimeters.

[0047] Figure 2 is a simplified diagram of a control system 200 according to some embodiments. In some embodiments, the control system 200 can be part of the control module 170. In some embodiments, the control system 200 can be used to control movement of the imaging device 114 and / or the instrument 124 of Figure 1 In some examples, the movement can include one or more of a position, an orientation, a pose, a motion, and / or the like of the imaging device 114 and / or the instrument 124. In some examples, the movement can be commanded by an operator using one or more input controls. Other aspects of the control system 200 are described in more detail in U.S. Patent Application Publication No. 2019 / 0143513, which is incorporated by reference herein.

[0048] In some embodiments, both the operator input system and the computer-aided device for manipulating the apparatus may include multiple links connected by joints to facilitate control of multiple degrees of freedom of the apparatus. When the operator moves one or more input controls of the operator input system from a first posture to a second posture during the execution of a program, sensors associated with the joints of the operator input system provide information indicating such commanded movement in the joint space of the one or more input controls. In some examples, the commanded movement includes commanded acceleration provided by sensors associated with the links of the operator input system. Sensors associated with the apparatus and / or the computer-aided device provide information indicating the movement of the apparatus in the apparatus joint space for feedback purposes.

[0049] like Figure 2 As shown, control system 200 is illustrated for controlling instrument 215 using input controls 210. In some examples, instrument 215 may be associated with imaging device 114 and / or instrument 124. In some examples, control system 200 may be adapted to use multiple input controls to control instrument 215. In some examples, control system 200 may be adapted to control two or more instruments.

[0050] Input control processing unit 221 receives information on the joint positions and / or velocities of the joints in input control 210 and / or the operator input system. In some examples, the joint positions may be sampled at the control system processing rate. Input control forward kinematics processing unit 222 receives the joint positions and velocities from input control processing unit 221 and transforms them from the joint space of input control 210 to corresponding positions and velocities in a reference coordinate system associated with the operator input system. In some examples, input control forward kinematics processing unit 222 performs this transformation using Jacobian and reference system-related information. In some examples, the reference coordinate system of the operator input system may be a reference coordinate system relative to the operator's eye. In some embodiments, an operating mode of control system 200 ensures that the motion of instrument 215 in the imaging device reference system corresponds to the motion of input control 210 in a reference coordinate system relative to the operator's eye.

[0051] The scale and offset processing unit 224 receives pose, velocity, and acceleration commands from the input control forward kinematics processing unit 222, scales the commanded movement according to a scale factor selected for executing the program, and accounts for an offset to generate a desired pose and velocity of the instrument 215. Scale adjustments are useful in cases where the desired instrument 215 makes small movements relative to large movements of the input controls 210 to allow the instrument 215 to move more precisely at a work site. For example, the offset determines a corresponding position and / or orientation of the instrument coordinate frame (e.g., instrument coordinate frame t) relative to the position and orientation of the input controls 210 in the reference coordinate frame of the operator input system.

[0052] The simulated instrument processing unit 228 receives the desired instrument position and velocity commands from the scale and offset processing unit 224 and limits the desired instrument pose, velocity, acceleration, and / or the like to specified limits, such as enforcing correct and intuitive operation of the instrument 215 by keeping the instrument 215 and any associated joints within motion limits and / or the like. The simulated instrument processing unit 228 generates simulated instrument and computer-assisted device joint states (e.g., positions, velocities, accelerations, and / or the like). In some examples, the simulated instrument and computer-assisted device joint states are determined based on an instrument 215 and / or manipulator Jacobian of a computer-assisted device to which the instrument 215 is mounted.

[0053] The inverse scale and offset processing unit 226 receives simulated joint position and velocity commands from the simulated instrument processing unit 228 and performs an inverse function (opposite to the function of the scale and offset processing unit 224) on the simulated joint position and velocity commands. The Cartesian controller 227 receives the input of the scale and offset processing unit 224 and the output of the inverse scale and offset processing unit 226. The Cartesian controller 227 then generates an error signal as the difference between the input of the scale and offset processing unit 224 and the output of the inverse scale and offset processing unit 226 and generates a Cartesian force “F CART ” based on the error signal.

[0054] The input control transpose kinematics processing unit 235 receives the Cartesian force F CART from the summing node 234 and generates a corresponding torque in joint space using, for example, a Jacobian transpose matrix and kinematic relationships associated with the operator input system. In systems where the operator input system has actuator-driven joints for motion range limits or force feedback, the input control output processing unit 236 receives output from the input control transpose kinematics processing unit 235 and generates electrical signals for controlling the actuators of the operator input system and the input controls 210. In some examples, the operator can feel control of the actuators of the operator input system and the input controls 210 as haptic feedback.

[0055] When the input control processing unit 221 receives input controls 210 and operator input system joint positions from sensors, the instrument input processing unit 229 also receives instrument positions from sensors in the instrument 215 and computer-assisted device. In some examples, the instrument positions are received by the instrument input processing unit 229 at the control system processing rate. The instrument input processing unit 229 includes an actuator-side input processing unit 241 and a load-side input processing unit 242. The actuator-side input processing unit 241 receives joint measurement data (e.g., pose, velocity, acceleration, and / or the like) from actuator-side sensors in the instrument 215 and / or computer-assisted device. The load-side input processing unit 322 receives link data (e.g., position, motion, and / or the like) of links in the instrument 215 and computer-assisted device from load-side sensors. The joint control unit 238 receives joint measurement data and link data from the instrument input processing unit 229 as well as the simulated joint commands from the simulated slave processing unit 228 and generates instrument command signals for joint actuators in the instrument 215 and / or computer-assisted device as well as input control feedback command signals for joint actuators in the input controls 210 and / or operator input system.

[0056] The instrument command signals are generated by the joint control unit 238 to drive the joints of the instrument 215 and / or computer-assisted device until feedback errors computed in the joint control unit 238 are zeroed out. The instrument output processing unit 230 receives the instrument command signals from the joint control unit 238, converts them to appropriate electrical signals, and supplies the electrical signals to joint actuators of the instrument 215 and / or computer-assisted device to drive the actuators accordingly.

[0057] The input control feedback command signals are generated by the joint control unit 238 that reflect the forces exerted against the instrument 215 and / or computer-assisted device supporting the instrument 215 back to the operator input system and input controls 210 so that the operator can feel haptic feedback in some form. In some examples, the joint control unit 238 can generate the input control feedback command signals based on joint position and / or velocity tracking errors in the instrument 215 and / or computer-assisted device. The kinematics mapping unit 231 receives the input control feedback command signals from the joint control unit 238 and generates corresponding Cartesian forces at the tip of the instrument 215 with respect to a reference coordinate frame associated with the operator input system.

[0058] The gains 233 adjust the size of the Cartesian forces to ensure system stability while providing sufficient force sensation to the operator. The gain-adjusted Cartesian forces are then passed through a summation node 234 and processed through the input control transpose kinematics processing unit 235 and the input control output processing unit 236 along with the Cartesian forces provided by the Cartesian controller 227, as previously described.

[0059] Figure 3 This is a simplified diagram of a registration method 300 according to some embodiments. One or more of the processes 305-350 of method 300 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when run by one or more processors (e.g., processor 150 in control unit 140), causes one or more processors to execute one or more of processes 305-350. In some embodiments, method 300 may be executed by one or more modules, such as control module 170. In some embodiments, method 300 may be used to determine a registration transformation (e.g., a registration transformation) between a first computer-assisted device (e.g., computer-assisted device 120) having an instrument (e.g., instrument 124) mounted thereon and a second computer-assisted device (e.g., computer-assisted device 110) having an imaging device (e.g., imaging device 114) mounted thereon capable of capturing one or more images of the instrument. In some embodiments, method 300 may also be used to determine a registration transformation between a computer-aided device having instruments mounted thereon and a handheld imaging device. In some embodiments, process 340 is optional and may be omitted. In some embodiments, method 300 may include additional processes not shown. In some embodiments, one or more of processes 305-350 may be performed at least in part by one or more units of control system 200.

[0060] At procedure 305, a seed transform is used to initialize the registration transform. In some examples, the seed transform is a coarse estimate of the registration transform, which will be iteratively refined using procedures 310-350. In some examples, one or more sensors may be used to determine the seed transform. In some examples, the one or more sensors may include an inertial measurement unit (IMU) for an imaging device, an instrument, or both an imaging device and an instrument. In some examples, the one or more sensors may also include one or more gravity sensors (e.g., accelerometers and / or inclinometers) and / or one or more magnetometers. In some examples, the one or more sensors may be associated with one or more tracking systems (e.g., electromagnetic and / or optical trackers).

[0061] In some embodiments, the seed transform can be determined by tracking one or more known movements of the imaging device and / or instrument. In some examples, the one or more known movements can overlap with motion of the imaging device and / or instrument performed for other purposes. In some examples, the one or more known movements can be commanded by the operator using the operator station. In some examples, the one or more known movements can include one or more small motions that are automatically initiated and / or initiated in response to an operator triggered start. In some examples, an image (e.g., a still image) captured by the imaging device just prior to the initiation of the one or more small motions can be displayed to the operator as the one or more small motions occur, such that the one or more small motions are not visible to the naked eye of the operator. In some examples, an image captured live by the imaging device can be displayed to the operator after the one or more small motions are complete.

[0062] In some embodiments, the one or more known movements can be one or more motions in known directions of the imaging device and / or instrument coordinate system that are superimposed on the movements of the imaging device and / or instrument commanded by the operator. In some examples, the one or more small motions can be consistent with an Eulerian magnification technique. In some examples, the one or more small motions can include motions that occur at a frequency higher than typical movements commanded by the operator during teleoperation (e.g., motions at a frequency greater than 10 Hz). In some examples, the one or more small motions can be separated from the operator commanded movements during data processing using a high pass filter.

[0063] In some embodiments, the one or more known movements and / or one or more small motions can continue until a sufficient number of images showing the relative position and / or orientation of the instrument with respect to the imaging device are captured such that, when combined with the forward and / or inverse kinematic models of the first and second computer-assisted devices, they can provide sufficient data to determine one or more estimates of each of the relevant unknowns of the registration transform (e.g., the unknowns from the rotational and / or translational portions of the seed transform). In some examples, multiple estimates of the unknowns can be aggregated by averaging, least squares estimation, and / or the like. In some examples, at least two images are captured when the computer-assisted devices to which the instrument and imaging device are respectively mounted are on a common flat surface and have a common vertical upward direction and a common vertical upward axis. In some examples, at least three images are captured when the computer-assisted devices to which the instrument and imaging device are respectively mounted are not on a common flat surface or do not have a common vertical upward axis. In some examples, the duration of the one or more known movements and / or one or more small motions and the number of images sampled are determined based on a tradeoff between the accuracy of the seed transform, the time to complete process 305, and / or the number of features (e.g., points, line segments, axes, directed vectors, free vectors, and the like) observed in the images.

[0064] In some embodiments, the translational portion of the seed transform can be determined from one or more images of the instrument captured by the imaging device. In some examples, when the position of a point feature (e.g., a fiducial, an indicative marker, a representative structure such as a tool tip or a U-shaped clamp pin, and / or the like) on the instrument can be determined relative to the imaging device coordinate system (e.g., coordinate system c), the relative position of the first and second computer-assisted devices and the forward and / or inverse kinematic models can be used to determine the translational portion of the seed transform (e.g., the translational portion ). In some examples, when the imaging device is capable of capturing partial or full 3D information (e.g., it is a stereo endoscope, provides depth information in addition to images, and / or the like), the relative position of the point feature can be determined directly from the imaging data. In some examples, when the imaging device is only capable of capturing 2D information, multiple images can be used to resolve the relative position of the point feature with imaging device movement, instrument movement, and / or both imaging device and instrument movement between images. In some examples, the translational portion of the seed transform can use multiple images and / or image sets to more accurately determine the relative position of the point feature. In some examples, the relative position estimates from each of the multiple images or from the image sets can be aggregated together, such as by using averaging, least squares estimation, and / or the like. In some examples, the relative positions determined for multiple point features and / or sets of features can be aggregated to determine the translational portion of the seed transform.

[0065] At process 310, the instrument is controlled using the registration transform. In some embodiments, the registration transform can be used to help more intuitively map movement of one or more input controls on the operator station to movement of the instrument (e.g., such that movement of the one or more input controls relative to an eye coordinate frame of the operator station more closely matches corresponding movement of the instrument relative to the imaging device). In some examples, movement of the one or more input controls in the eye coordinate frame of the operator station is converted to corresponding movement of the instrument relative to the imaging device. The registration transform is used to map the corresponding movement of the instrument relative to the imaging device to corresponding movement of the instrument relative to a base coordinate frame of the computer-assisted device to which the instrument is mounted In some examples, this mapping can be determined using Equation 2, where is a forward kinematics transform from the instrument to a base coordinate frame of the computer-assisted device to which the instrument is mounted, is the registration transform, and is a forward kinematics transform from the imaging device to a base coordinate frame of the computer-assisted device to which the imaging device is mounted.

[0066]

[0067] In some examples, movement of the one or more input controls in the eye coordinate frame of the operator station (e.g., the output of the scaling and offset processing unit 224), corresponding movement of the instrument relative to the imaging device and / or corresponding movement of the instrument relative to a base coordinate frame of the computer-assisted device to which the instrument is mounted may be expressed as 3DOF position, 3DOF orientation, 3DOF translational velocity, 3DOF angular velocity, and / or a subset thereof, of a reference coordinate frame of interest. These commanded motion properties at the operator station are referred to as “desired” or “intended,” and are associated with the reference coordinate frame of interest of the operator station. In some examples, the reference coordinate frame of interest can be associated with the one or more input controls. In some examples, the same motion properties measured on the instrument using images from the imaging device are referred to as “observed.”

[0068] At process 315, one or more images of the instrument are obtained. In some examples, the one or more images can be captured by the imaging device. In some examples, the data can include left and right 3D images from a stereo imaging device, 3D images from multiple imaging devices set up to capture images from different perspectives, and / or 3D depth or intensity maps from the imaging device.

[0069] At a process 320, observed geometric properties of one or more features of the instrument are extracted from the one or more images. In some examples, the one or more observed geometric properties can correspond to one or more positions, one or more velocities, one or more rotational velocities, and / or the like. In some examples, each observed geometric property can be modeled as a point, a transmissible vector, a free vector, a bound vector, and / or suitable combinations of any of these. In some examples, the one or more features of the instrument can include one or more primary features and / or one or more secondary features.

[0070] In some examples, a computer vision algorithm is used to determine the primary features from the one or more images to detect known fiducials on the instrument, detectable features of the instrument, and / or the like. In some examples, a machine learning and / or artificial intelligence (AI) module can be used to observe the primary features of interest on the instrument. In some examples, the machine learning and / or AI module can be composed of a deep convolutional neural network that is trained to detect specific features on the instrument, such as identifiable point features on the instrument (such as fiducials, indicative markers (including markers to indicate the type or source of the instrument), representative structures (such as a tool tip or a U-shaped clip pin about which one or more jaws and / or links of the instrument can rotate), and / or the like. In some examples, the one or more primary features can correspond to alignment features of the instrument, such as real or virtual lines of symmetry, lines associated with primary rotational and / or translational motion of the instrument, lines integral to the shape of the instrument, and / or the like. In some examples, the alignment features can be an axis determined from a fiducial indicating an axis line, an indicative marker indicating an axis line, an axis of a shaft of the instrument, an axis corresponding to a unique structure (e.g., located between two jaws of the instrument and oriented with the gripping faces of the two jaws when the jaws are closed), an axis formed by a line segment connecting any of the above point features, and / or the like.

[0071] In some examples, the one or more secondary features can correspond to one or more points of interest on a kinematic chain of a computer-assisted device holding the instrument, such as a remote center of motion. In some examples, the one or more secondary features can correspond to points and / or axes that have a predefined linear and / or angular displacement from any of the one or more primary features. In some examples, the secondary features can be determined using one or more estimators that use kinematic information of the imaging device and / or the computer-assisted device to which the instrument is mounted and / or kinematic information of the one or more primary features detected in the one or more images. In some examples, each estimator can be a Kalman filter, a state observer, and / or the like. In some examples, the one or more secondary features can be derived from a 3D model of a portion of the instrument that is at least partially visible in the one or more images. In some examples, the 3D model of the part can be registered to the observed part using model matching and / or the like. In some examples, the one or more secondary features can be derived by a data fusion process that combines information from the one or more primary features with information from one or more other sensors, such as one or more accelerometers, inclinometers, IMUs, and / or the like. In some examples, a combination or subset of the one or more primary features and / or the one or more secondary features can be used depending on occlusion of a portion of the instrument in the field of view of the imaging device; accuracy of observing and / or estimating features, lighting conditions, color variations; replacement of the instrument during a procedure; absence of the instrument in the field of view of the imaging device, and / or the like. Additional examples of process 320 are described in further detail with respect to Figure 4A Further details are described.

[0072] At process 325, expected geometric properties of the one or more features are extracted from instrument kinematics. In some examples, the expected geometric properties of the one or more features correspond to the same observed geometric properties of the same one or more features from which the observed geometric properties were extracted during process 320 from the one or more images. In some examples, the expected geometric properties of the one or more features are time synchronized with the same observed geometric properties of the same one or more features such that the expected geometric properties and the observed geometric properties are based on information about the one or more features obtained at approximately the same time period. In some examples, the instrument kinematics correspond to information about a kinematic chain of the instrument and a computer-assisted device to which the instrument is mounted. In some examples, the information about the kinematic chain of the instrument can be determined based on one or more joint positions and / or orientations of the computer-assisted device to which the instrument is mounted, one or more joint positions and / or orientations of a repositionable arm to which the instrument is mounted, one or more joint positions and / or orientations of the instrument, and one or more corresponding kinematic models. Additional examples of process 325 are described in further detail with respect to Figure 4B Further details are described.

[0073] At process 330, the expected geometric features and the observed geometric features are transformed to a common coordinate system using a registration transform. The common coordinate system allows for direct comparison between (i) the first estimate of the geometric properties of the one or more features of the instrument determined based on the one or more images during process 320 and (ii) the second estimate of the same or related geometric properties of the one or more features of the instrument determined based on the instrument kinematics determined during process 325, information about the kinematic chain of the computer-assisted device to which the instrument is mounted, and information about the kinematic chain of the computer-assisted device to which the imaging device is mounted. In some examples, the common coordinate system is the coordinate system of the instrument (e.g., coordinate system t), the coordinate system of the imaging device (e.g., coordinate system c), the base coordinate system of the computer-assisted device to which the instrument is mounted (e.g., coordinate system bi), and / or the base coordinate system of the computer-assisted device to which the imaging device is mounted (e.g., coordinate system bo). In some examples, the common coordinate system can be selected to be any other reference coordinate system different from coordinate systems bo, bi, t, or c. In some examples, the reference coordinate system can be selected based on the procedure being performed, the type of instrument, the type of imaging device, operator preference, and / or the like, as long as a suitable kinematic transform can be determined to transform both the expected geometric properties and the observed geometric properties to the reference coordinate system.

[0074] In some examples, when the common coordinate system is the coordinate system of the imaging device, the observed geometric properties of the one or more features of the instrument determined during process 320 are already available in the coordinate system of the imaging device, and the expected geometric properties of the one or more features of the instrument in the coordinate system of the imaging device can be transformed to the expected geometric properties in the coordinate system of the imaging device using Equation 3 wherein is a forward kinematic transform from the instrument to the base coordinate system of the computer-assisted device to which the instrument is mounted, is the registration transform, and is a forward kinematic transform from the imaging device to the base coordinate system of the computer-assisted device to which the imaging device is mounted.

[0075]

[0076] In some examples, when the common coordinate system is the coordinate system of the instrument, the expected geometric properties of the one or more features of the instrument determined during process 325 are already available in the coordinate system of the instrument, and the observed geometric properties of the one or more features of the instrument determined during process 320 can be transformed to the coordinate system of the instrument using the approach of Equation 2.

[0077] Similar methods can be used for other common coordinate systems, including coordinate system b0, coordinate system b1, and / or any of the reference coordinate systems described above. In some examples, variations of equations 2 and 3 can be used with appropriate forward and / or inverse kinematics transforms of the reference transform and the instrument, imaging device, and / or forward and / or inverse kinematics transforms between the common coordinate system and the base coordinate system of the appropriate computer-assisted device.

[0078] In some examples, information about the kinematic chain of the imaging device can be determined based on one or more joint positions and / or orientations of the computer-assisted device to which the imaging device is mounted, one or more joints and / or orientations of the repositionable arm to which the imaging device is mounted, one or more joint positions and / or orientations of the imaging device, and one or more corresponding kinematic models. In some examples, information about the kinematic chain of the imaging device can be supplemented using system calibration and / or configuration parameters, such as imaging device calibration, imaging device settings, system settings, and / or the like.

[0079] In some examples, when the imaging device is handheld, the position and / or orientation of the imaging device can be determined from information obtained from one or more sensors, one or more tracking units, one or more IMUs, and / or the like.

[0080] At process 335, an error between the expected geometric property and the observed geometric property is determined in the common coordinate system. In some examples, the error can describe an error in the registration transform. In some examples, the error can additionally and / or alternatively be determined from a difference between a change in the expected geometric property and a change in the observed geometric property. In some examples, the registration transform error AT can be computed by solving an optimization problem that minimizes a difference between the expected geometric feature and the observed geometric feature, as shown in equation 4.

[0081]

[0082] In some examples, when the expected geometric property and the observed geometric property correspond to point features, alignment features, and / or the like of one or more features that can be matched using the registration transform error AT, the optimization problem can be solved by a weighted least squares method. In some examples, when there is no correspondence between the expected geometric property and the observed geometric property, the optimization problem can be solved by a method such as an iterative closest point algorithm and / or the like. Additional examples of determining the error are described below with respect to Figure 4C and / or Figure 5 are described in further detail.

[0083] At optional process 340, the error is filtered. In some examples, the error determined during process 335 is passed through a frequency domain filter to smooth transitions between previous and current registration transform updates. In some examples, separate filters can be used for the rotational and translational portions of the error depending on the baseline noise level in determining the expected geometric properties and / or the observed geometric properties. In some examples, the filters used to process different degrees of freedom within the translational or rotational portions can also be different.

[0084] At process 345, it is determined whether the magnitude of the error or filtered error is within a suitable range for updating the registration transform. In some examples, the translational and / or rotational portions of the error or filtered error can be compared to corresponding configurable upper limits to determine whether the error or filtered error can provide a safe and / or allowable update to the registration transform. In some examples, the corresponding configurable upper limits can be a constant depending on the maximum allowable difference in the registration transform between successive iterations of method 300. In some examples, the corresponding configurable upper limits can be variable depending on the maximum allowable instantaneous difference in the registration transform between successive iterations of method 300 in a particular configuration of the computer-assisted device, instrument, and / or imaging device. In some examples, the corresponding configurable upper limits can be determined based on one or more of the procedure being performed, the type of instrument, the type of imaging device, the type of any of the computer-assisted devices, operator preferences, and / or the like.

[0085] In some examples, the magnitude of the translational and / or rotational portions of the error or filtered error can be compared to corresponding configurable lower limits to determine whether the error or filtered error indicates that a registration transform update should be performed. In some examples, the corresponding configurable lower limits can be set according to a maximum alignment error that is perceptible to an operator, a maximum alignment error that can significantly reduce the efficiency of an operator during a procedure, and / or the like. In some examples, the corresponding configurable lower limit can be 5 degrees for the rotational portion of the error or filtered error and / or 2 cm for the translational portion of the error or filtered error. In some examples, the corresponding configurable lower limits can be determined based on one or more of the procedure being performed, the type of instrument, the type of imaging device, the type of any of the computer-assisted devices, operator preferences, and / or the like.

[0086] When the magnitude of the error or filtered error is determined to be within a suitable range (e.g., the error is greater than a configurable lower bound and / or less than a configurable upper bound), the registration transform is updated using process 350. When the magnitude of the error or filtered error is determined not to be within a suitable range, the method 300 repeats without updating the registration transform by returning to process 310. In some embodiments, in addition to checking the range of error magnitude, the rate of convergence of the registration error is also used as a check to determine whether to update the registration transform using process 350. As modeled, the registration error should monotonically decrease over time as successive corrections are made. In some examples, the registration error trend can be calculated to ensure that the error magnitude is gradually converging towards the lower bound on average.

[0087] At process 350, the registration transform is updated. In some examples, the error or filtered error is used to determine a change in the rotational portion of the registration transform and / or a change in the translational portion of the registration transform.

[0088] In some examples, the change in the rotational portion of the registration transform is determined from the angular error determined during process 335 and optionally filtered by process 340. In some examples, the change in the rotational portion of the registration transform can be a 3x3 rotation matrix (AR) that describes the error in the rotational angles of the rotational portion of the registration transform. In some examples, the rotational portion of the registration transform is updated by combining the change in the rotational portion of the registration transform determined from the current iteration (i) of the method 300 (e.g., rotation matrix AR(i)) with the current rotational portion of the registration transform at the current iteration to determine the updated rotational portion of the registration transform for the next iteration (i+1) of the method 300, as shown in Equation 5.

[0089]

[0090] In some examples, the change in the translational portion of the registration transform is determined by combining the translational error determined from the current iteration (i) of the method 300 (e.g., translation vector At(i)) with the translational portion of the registration transform at the current iteration of the method 300 to update the translational portion of the registration transform at the next iteration of the method 300, as shown in Equation 6.

[0091]

[0092] In some examples, the rotational portion of the registration for the next iteration of the method 300 is determined by using Equation 1 with the translational portion of the registration transform for the next iteration of the method 300 ​​The registration transformation is updated in combination. After the registration transformation is updated at process 350, the method 300 repeats by returning to process 310.

[0093] In some embodiments, the processes 310-350 of the method 300 can be performed within a closed-loop real-time control system in the control unit 140, such that the registration transformation updates are available at each sample of the control system. As a result, as the instrument continues to be controlled, updates to the registration transformation can be repeated by repeating the processes 310-350 during each cycle of the control system. In some examples, the sensor inputs to the control system and the registration transformation outputs from the control system can be at different sampling rates compared to the execution rate of the control system itself; however, such peripheral communications are typically synchronized with the control system such that the expected geometric properties of the one or more features of the instrument and the observed geometric properties are time-synchronized.

[0094] Figures 4A-4C is a simplified diagram of several processes of the method 300 according to some embodiments. More specifically, Figures 4A-4C Embodiments of the method 300 correspond to implementations in which the 6DOF pose and velocity of the one or more input controls are known in the eye coordinate system of the operator station, such that movement of the one or more input controls relative to the eye coordinate system of the operator station can be used to determine corresponding expected movement of the instrument in the imaging device coordinate system. In some examples, the expected movement of the instrument in the imaging device coordinate system can correspond to an expected 3DOF position, an expected 3DOF orientation, an expected 3DOF translational velocity, and / or an expected 3DOF angular velocity of the instrument or a subset thereof. Figure 4A Process 410 is shown for implementing the process 320 of the method 300 under these implementations, Figure 4B Processes 420 and 430 are shown for implementing the process 325 of the method 300 under these implementations, and Figure 4C Process 440 is shown for implementing the process 335 of the method 300 under these implementations.

[0095] Referring to Figure 4A At process 410, an observed velocity of a feature of the instrument is determined based on an image obtained from the imaging device. In some examples, consecutive images over time are obtained from the imaging device. In some examples, the consecutive images can correspond to an image obtained during the process 315 of the current iteration through the processes 310-350 and an image obtained during the process 315 of a previous iteration through the processes 310-350.

[0096] In some examples, the observed velocity corresponds to a change in position of a point feature on the instrument. In some examples, the point feature can correspond to a primary point feature (such as a fiducial, an indicative marker), a representative structure (such as a tool tip or clevis pin about which the jaws of the instrument can rotate), and / or the like visible in the image. In some examples, the point feature can correspond to a secondary point feature determined from one or more primary features. In some examples, the observed velocity can be modeled as a 3D velocity using, for example, Dx, Dy, and Dz values.

[0097] In some examples, the velocity corresponds to a change in orientation of an alignment feature of the instrument. In some examples, the alignment feature can be a primary alignment feature, such as an axis determined from a fiducial indicative of an axis, an indicative marker indicative of an axis, an axis corresponding to a unique structure (e.g., located between two jaws of the instrument and oriented with the gripping faces of the two jaws when the jaws are closed), and / or the like. In some examples, the alignment feature can be a secondary alignment feature determined from one or more primary features. In some examples, the change in orientation of the alignment feature can be modeled as a change in rotation about three independent axes. In some examples, the three independent axes can be roll, pitch, and yaw; x, y, and z; and / or the like. In some examples, the change in orientation of the alignment feature can be modeled using a quaternion.

[0098] In some examples, the observed velocity can be modeled as a 3D translational velocity of a point cloud composed of a set of features. In some examples, the observed velocity can be modeled as a 6DOF spatial velocity of a reference coordinate frame on the instrument, including a 3DOF translational velocity and a 3DOF angular velocity. In some examples, the reference coordinate frame can be attached to the wrist center of the instrument. In some examples, the reference coordinate frame can be attached to a link distal to the distal-most joint through which the orientation of the distal portion of the instrument can be articulated. In some examples, the reference coordinate frame can be located anywhere on or near the instrument and can be selected based on the type of instrument, the procedure being performed, operator preference, and / or the like. In some examples, when the observed velocity is a set of observed velocities of a set of features, the multiple observed velocities can be mapped to one point feature using a linear or non-linear weighted average and / or other aggregation.

[0099] Reference Figure 4BAt process 420, an expected velocity of the instrument is determined from movement of the one or more input controls. In some examples, the expected velocity can be synchronized in time with the observed velocity determined during process 410. In some examples, the expected velocity of the instrument is determined from a velocity of the one or more input controls and / or a change in position and / or a change in orientation of the one or more input controls between successive iterations through processes 310-350. In some examples, a velocity of the one or more input controls can be mapped to a velocity of the instrument using a known transformation between the one or more input controls in the eye coordinate system of the operator station. In some examples, the mapping can include a predetermined motion scaling between the input controls and the instrument, such as can be implemented by scaling and offset unit 224. In some examples, the mapping can include signal processing, such as filtering, offsetting, and / or the like.

[0100] At optional process 430, the expected velocity of the instrument is transformed to an expected velocity of the feature. When the feature whose observed velocity is determined during process 410 is different from the feature of the instrument controlled by the one or more input controls, the expected velocity of the instrument determined during process 420 from movement of the one or more input controls is transformed to an expected velocity of the feature. In some examples, one or more kinematic and / or geometric models of the instrument can be used to transform the expected velocity of the instrument to the expected velocity of the feature.

[0101] Reference is made to Figure 4C At process 440, an error between a direction of the observed velocity and a direction of the expected velocity of the feature is determined. In some examples, the error between the direction of the observed velocity and the direction of the expected velocity can correspond to an angular difference between an observed change in position and an expected change in position (e.g., translational velocity) of a point feature or an angular difference between an observed change in orientation and an expected change in orientation of a rotational and / or aligned feature. In some examples, the error between the observed velocity and the expected velocity of the one or more features is determined using the methods described with respect to process 335. In some examples, one or more of the following errors can be significantly smaller than a requirement for the registration transformation error, or can be well characterized and compensated for, during determination of the error between the observed velocity and the expected velocity: a control system tracking error of the manipulator inverse kinematics, a calibration error of the manipulator kinematics, and / or a control system tracking error of the input control and instrument teleoperation loop (e.g., as described with respect to control system 200).

[0102] Figure 5 is a simplified diagram of process 335 of method 300 according to additional embodiments. More specifically, Figure 5 Embodiments of correspond to implementations in which a transformation between the one or more input controls and the instrument is unknown and / or not used in the eye coordinate system of the operator station. Figure 5Processes 510-530 are shown for implementing process 335 of the methods under these implementations.

[0103] At process 510, a difference between an observed orientation of an alignment feature and an expected orientation is determined. In some examples, the difference can correspond to an angular difference between the observed orientation of the alignment feature and the expected orientation. In some examples, the observed orientation of the alignment feature and the expected orientation can be time-synchronized. In some examples, the alignment feature can be a primary alignment feature, such as an axis determined from a fiducial indicating an axis, an indicative marker indicating an axis, an axis corresponding to a unique structure (e.g., located between two jaws of an instrument and oriented with the gripping faces of the two jaws when the jaws are closed), and / or the like. In some examples, the alignment feature can be a secondary alignment feature determined from one or more primary features. In some examples, when the computer-assisted device on which the imaging device is mounted and the computer-assisted device on which the instrument is mounted are located on a common planar surface and have a common vertical upward axis, the difference can correspond to an angular difference about the common vertical upward axis. In some examples, when the first computer-assisted device and the second computer-assisted device are not located on a common planar surface or do not have a common vertical upward axis, the difference can correspond to an angular difference about three axes of a common coordinate system. In some examples, the change in orientation of the alignment feature can be modeled as a change in rotation about three independent axes. In some examples, the three independent axes can be roll, pitch, and yaw; x, y, and z; and / or the like. In some examples, the change in orientation of the alignment feature can be modeled using quaternions. In some examples, the difference between the observed orientation of the alignment axis and the actual orientation can be determined using the methods described with respect to process 335.

[0104] At process 520, a gradient of an error term is determined based on the difference. In some examples, the error term can be equal to a vector norm of the difference. In some examples, the error term can be equal to the difference. In some examples, the gradient is computed based on the difference determined during process 410 of the current iteration of processes 310-350 and the difference determined during process 410 of a previous iteration of processes 310-350.

[0105] At process 530, an error in the rotation angle is determined based on the gradient. In some examples, Equation 7 can be used to determine the error in the rotation angle dQ, where k is one or more tuning parameters. In some examples, k is a scalar parameter when the first computer-assisted device and the second computer-assisted device are located on a common flat surface and have a common vertical upward axis. In some examples, k is a 3x3 parameter matrix when the first computer-assisted device and the second computer-assisted device are not located on a common flat surface and have a common vertical upward direction axis. In some examples, k can be tuned based on one or more of operator preference, type of instrument and / or imaging device, type of first computer-assisted device and / or second computer-assisted device, and / or the like.

[0106] dQ = -k * gradient(error term) (Equation 7)

[0107] As described above and further emphasized here, Figure 3 , Figures 4A-4C and / or Figure 5 are merely examples which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. According to some embodiments, process 305 can be repeated periodically. In some examples, a new seed transform can be determined by repeating process 305, such as when a base of one or more of the first computer-assisted device or the second computer-assisted device is moved. In some examples, when a coordinate system of the instrument (e.g., coordinate system t) can be determined from the captured images, the entire registration transform can be determined from the determination of the coordinate system of the instrument. In some examples, the coordinate system of the instrument can be determined from a point (e.g., a point feature) on the instrument and two or more axes that can be used to span across the 3D image space. In some examples, one or more fiducials, one or more markers, and / or one or more structures on the instrument can be used to determine the point and the two or more axes.

[0108] According to some embodiments, the registration transform can be used to map keep-out regions of the workspace onto images of the workspace captured by the imaging device and displayed to the operator. In some examples, the keep-out regions correspond to regions within the workspace that are not allowed to be entered by the instrument or are urged away from. In some examples, the keep-out regions can be indicated by one or more visual cues overlaid on the captured images (e.g., barriers, shaped or cross-hatched regions, and / or the like). In some examples, the keep-out regions can correspond to haptic walls and / or haptic response regions for providing haptic feedback to the operator using one or more input controls; for example, the haptic feedback can resist input control movement through the haptic walls and / or into the haptic regions. In some examples, the keep-out regions are measured using depth maps or intensity maps generated by the imaging device. In some examples, the keep-out regions are detected by a trained machine learning module using depth, intensity, and / or color datasets and other variables such as lighting conditions. In some medical examples, the keep-out regions correspond to safety critical anatomical structures such as tissue, bone, nerves, blood vessels, and / or the like.

[0109] According to some embodiments, the magnitude of the error determined during process 335 and / or process 340 can be used to control the speed at which the instrument is teleoperated using the one or more input controls during process 310. In some examples, the speed of teleoperation is inversely scaled according to the magnitude of the error, such that a larger error results in a lower maximum speed of teleoperation, and such that a smaller error results in a higher allowed speed of teleoperation. In some examples, the speed of teleoperation can be adjusted by changing a scaling factor between the amount of movement of the one or more input controls and the corresponding amount of movement of the instrument. In some examples, the speed of teleoperation can be adjusted by setting a speed cap on the rate at which the instrument can move. In some examples, the magnitude of the error can be used to set a gain on the amount of haptic feedback applied to the one or more input controls, where a higher magnitude of error results in a larger amount of haptic feedback. In some examples, the speed at which the instrument is teleoperated can be scaled only when the teleoperation moves the instrument in a direction that increases the magnitude of the error.

[0110] According to some embodiments, the registration transform can be initialized during process 305 using techniques other than the seed transform. In some examples, any other registration technique can be used to initialize the registration transform, such that processes 310-350 can be used to continuously update the registration transform determined using that other registration technique. In some examples, the other registration technique can include one or more of touch-based registration (where the instrument is teleoperated to one or more known positions and / or orientations), registration using one or more registration devices (e.g., shape sensors, articulated links, and / or the like), and / or the like.

[0111] According to some embodiments, the method 300 can be adapted to situations where there are different configurations of computer-assisted devices, repositionable arms, and / or instruments. In some examples, when there are additional computer-assisted devices having different base coordinate systems, the method 300 can be applied separately to each additional computer-assisted device to determine and iteratively update a respective registration transform for each additional computer-assisted device. In some examples, when the first computer-assisted device has additional instruments (both with or without separate corresponding repositionable arms), each instrument can be used to determine a seed transform and / or a registration transform for the first computer-assisted device. In some examples, the contributions from each instrument (e.g., translational portion values, seed transforms, and / or errors used to determine changes in rotational registration transforms) can be determined separately and then aggregated together (e.g., by averaging, least squares estimation, and / or the like). In some examples, the processes 310-340 can be performed separately for each instrument, with each instrument contributing to updating the registration transform in turn before looping through each instrument again. In some examples, the contributions of each instrument to the update of the registration transform can be weighted based on the size of the error determined during the process 335 and / or the process 340 for the respective instrument. In some examples, the weighting of each instrument’s update to the registration transform can be proportional to the size of the error for that instrument. In some examples, the weights can be normalized relative to the sum of the error sizes.

[0112] According to some embodiments, when multiple instruments can be seen in the captured image, the movement of the instrument determined from the captured image can be correlated to the movement of one or more input controls used to control the instrument to determine which instrument is associated with each of the one or more input controls. In some examples, the association can be aided by the operator by manipulating one instrument at a time, manually selecting the association, and / or the like.

[0113] Some examples of the control unit, such as the control unit 140, can include a non-transitory, tangible machine-readable medium including executable code that, when executed by one or more processors (e.g., the processor 150) can cause the one or more processors to perform the processes of the method 300 and / or Figure 3 A、 Figure 3 B、 Figure 4A and / or Figure 4B the processes of the method 300. The control unit can include a non-transitory, tangible machine-readable medium including executable code that, when executed by one or more processors (e.g., the processor 150) can cause the one or more processors to perform the processes of the method 300 and / or Figure 3 A、 Figure 3 B、 Figure 4A and / or Figure 4BSome common forms of machine-readable media that record instructions for execution by such computer or processor include floppy diskettes, floppy disks, hard disks, magnetic tapes, any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tapes, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium from which a computer or processor can read instructions.

[0114] While the illustrative embodiments have been described, numerous modifications, alterations, and changes to the illustrative embodiments can be appreciated by those skilled in the art without departing from the scope and spirit of the present disclosure. It should be understood that this disclosure is not restricted to the illustrative embodiments, but rather, is intended to cover various modifications, alternatives, and equivalents. Therefore, the scope and spirit of the disclosure should be judged in terms of the claims and their full scope or equivalents, including the following claims, and not based on the details of the description.

Claims

1. A computer-aided system, comprising: A control unit coupled to a repositionable arm of a computer-aided device; The control unit is configured as follows: Receive continuous images of the instrument mounted on the repositionable arm from the imaging device; The observed speed at which the features of the instrument are determined based on the continuous images; The expected speed of the device's features is determined based on the kinematics of the repositionable arm; Use a registration transformation to transform the observed velocity, the expected velocity, or both the observed velocity and the expected velocity to a common coordinate system; Determine the error between the direction of the observed velocity and the direction of the expected velocity in the common coordinate system; as well as The registration transformation is updated based on the determined error.

2. The computer-aided system according to claim 1, wherein the instrument is a medical device and the imaging device is an endoscope.

3. The computer-aided system of claim 1, wherein the control unit is further configured to use the registration transformation to control the instrument.

4. The computer-aided system according to claim 1, wherein: The feature is a point feature of the instrument; and The observed velocity of the feature includes the positional change of the point feature in the continuous images.

5. The computer-aided system according to claim 1, wherein: The feature is the alignment feature of the instrument; and The observed speed of the feature includes the orientation change of the alignment feature in the continuous images.

6. The computer-aided system according to claim 1, wherein: The feature is a feature set that includes multiple features; and The observed velocity of the feature set is an aggregation of the observed velocities of the features in the feature set.

7. The computer-aided system of claim 1, wherein the feature is a primary feature observable in the continuous images, or a secondary feature determined from one or more primary features observable in the continuous images.

8. The computer-aided system of claim 1, wherein the feature corresponds to a reference of the instrument, an indicative mark of the instrument, or a structure of the instrument.

9. The computer-aided system of claim 1, wherein the feature is a virtual feature having a predefined displacement or predefined rotation relative to the instrument.

10. The computer-aided system of claim 1, wherein the imaging device is mounted to the second repositionable arm of the second computer-aided device.

11. The computer-aided system of claim 1, wherein the expected speed includes a change in position or orientation of the feature due to movement of one or more input controls for manipulating the instrument.

12. The computer-aided system of claim 11, wherein, in order to determine the expected speed of the instrument, the control unit is configured to: The expected speed of the device is determined from the movement of the one or more input controls; and The expected speed of the instrument is transformed into the expected speed of the feature.

13. The computer-aided system according to any one of claims 1 to 11, wherein the registration transformation is between the base coordinate system of the computer-aided device and the coordinate system associated with the imaging device.

14. The computer-aided system of claim 13, wherein the coordinate system associated with the imaging device is a base coordinate system of the second computer-aided device, and the imaging device is mounted to a second repositionable arm of the second computer-aided device.

15. The computer-aided system according to any one of claims 1 to 11, wherein, in order to update the registration transform, the control unit is configured to update the rotation portion of the registration transform based on the error, but not the translation portion of the registration transform based on the error.

16. The computer-aided system of claim 15, wherein, in order to update the rotation portion of the registration transform, the control unit is configured to combine a rotation matrix based on the error with a previous value of the rotation portion.

17. The computer-aided system according to any one of claims 1 to 11, wherein the control unit is further configured to initialize the registration transformation from a seed transformation or another registration method.

18. The computer-aided system of claim 1, wherein the control unit is further configured to initialize the registration transformation from a seed transformation, the seed transformation being determined based on an image captured by the imaging device of the instrument during a known relative movement between the instrument and the imaging device.

19. The computer-aided system of claim 18, wherein the known relative movement corresponds to a motion with a frequency higher than a threshold frequency.

20. The computer-aided system of claim 18, wherein the control unit is further configured to display a static image to the operator while performing the known relative movement.

21. The computer-aided system according to any one of claims 1 to 11, wherein, in order to update the registration transformation based on a determined error, the control unit is configured to update the registration transformation only when the error is less than a configurable upper limit.

22. The computer-aided system according to any one of claims 1 to 11, wherein, in order to update the registration transformation based on a determined error, the control unit is further configured to update the registration transformation when the error is greater than a configurable lower limit.

23. The computer-aided system according to any one of claims 1 to 11, wherein the control unit is further configured to control the instrument using an updated registration transformation.

24. The computer-aided system according to any one of claims 1 to 11, wherein the control unit is further configured to use the magnitude of the error to control the speed at which one or more input controls remotely operate the instrument.

25. The computer-aided system of claim 24, wherein, in order to control the speed at which the one or more input controls remotely operate the instrument, the control unit is configured to: The upper limit speed of the movement of the device is set based on the size; or The gain of the haptic feedback applied to the one or more input controls is set based on the size.

26. A non-medical method for operating a computer-aided system, the method comprising: The control unit receives continuous images of the repositionable arm of the instrument mounted on the computer-aided device from the imaging equipment. The observed speed by which the control unit determines the features of the device based on the continuous images; The control unit determines the expected speed of the instrument's feature based on the kinematics of the repositionable arm; The control unit uses a registration transformation to transform the observed velocity, the expected velocity, or both the observed velocity and the expected velocity, to a common coordinate system. The control unit determines the error between the direction of the observed velocity and the direction of the expected velocity in the common coordinate system. as well as The registration transformation is updated by the control unit based on the determined error.

27. The method of claim 26, further comprising controlling the instrument by the control unit using the registration transformation.

28. The method according to claim 26, wherein: The feature is a point feature of the instrument; and The observed velocity of the feature includes the positional change of the point feature in the continuous images.

29. The method according to claim 26, wherein: The feature is the alignment feature of the instrument; and The observed speed of the feature includes the orientation change of the alignment feature in the continuous images.

30. The method of claim 26, wherein: The feature is a feature set that includes multiple features; and The observed velocity of the feature set is an aggregation of the observed velocities of the features in the feature set.

31. The method of claim 26, wherein the feature is a primary feature observable in the continuous images, or a secondary feature determined from one or more primary features observable in the continuous images.

32. The method of claim 26, wherein the feature corresponds to a reference of the instrument, an indicative mark of the instrument, or a structure of the instrument.

33. The method of claim 26, wherein the feature is a virtual feature having a predefined displacement or predefined rotation relative to the instrument.

34. The method of claim 26, wherein the expected speed includes a change in position or orientation of the feature due to movement of one or more input controls for manipulating the instrument.

35. The method of any one of claims 26 to 34, wherein the registration transformation is between the base coordinate system of the computer-aided device and the coordinate system associated with the imaging device, and wherein the coordinate system associated with the imaging device is the base coordinate system of a second computer-aided device, the imaging device being mounted to a second repositionable arm of the second computer-aided device.

36. The method of any one of claims 26 to 34, wherein updating the registration transform comprises updating the rotation portion of the registration transform based on the error, but not updating the translation portion of the registration transform based on the error.

37. The method according to any one of claims 26 to 34, further comprising initializing the registration transformation by the control unit from a seed transformation or another registration method.

38. The method according to any one of claims 26 to 34, further comprising: The control unit determines the seed transformation based on the image captured by the imaging device of the instrument when a known relative movement occurs between the instrument and the imaging device; as well as The registration transformation is initialized from the seed transformation.

39. The method according to any one of claims 26 to 34, further comprising updating the registration transformation by the control unit only when the error is less than a configurable upper limit.

40. The method according to any one of claims 26 to 34, further comprising updating the registration transformation by the control unit when the error is greater than a configurable lower limit.

41. The method according to any one of claims 26 to 34, further comprising the control unit using the magnitude of the error to control the speed at which one or more input controls remotely operate the instrument.

42. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions, said machine-readable instructions, when executed by one or more processors of a computer-aided system, adapted to cause said one or more processors to perform a method, said method comprising: Receive continuous images of an instrument mounted on a computer-aided repositionable arm from an imaging device; The observed speed at which the features of the instrument are determined based on the continuous images; The expected speed of the device's features is determined based on the kinematics of the repositionable arm; Use a registration transformation to transform the observed velocity, the expected velocity, or both the observed velocity and the expected velocity to a common coordinate system; Determine the error between the direction of the observed velocity and the direction of the expected velocity in the common coordinate system; as well as The registration transformation is updated based on the determined error.

43. The non-transitory machine-readable medium of claim 42, wherein the method further comprises: The registration transformation is used to control the device.

44. The non-transitory machine-readable medium according to claim 42, wherein: The feature is a point feature of the instrument; and The observed velocity of the feature includes the positional change of the point feature in the continuous images.

45. The non-transitory machine-readable medium according to claim 42, wherein: The feature is the alignment feature of the instrument; and The observed speed of the feature includes the orientation change of the alignment feature in the continuous images.

46. ​​The non-transitory machine-readable medium according to claim 42, wherein: The feature is a feature set that includes multiple features; and The observed velocity of the feature set is an aggregation of the observed velocities of the features in the feature set.

47. The non-transitory machine-readable medium of claim 42, wherein the feature is a primary feature observable in the continuous image, or a secondary feature determined from one or more primary features observable in the continuous image.

48. The non-transitory machine-readable medium of claim 42, wherein the feature corresponds to a reference of the instrument, an indicative mark of the instrument, or a structure of the instrument.

49. The non-transitory machine-readable medium of claim 42, wherein the feature is a virtual feature having a predefined displacement or predefined rotation relative to the instrument.

50. The non-transitory machine-readable medium of claim 42, wherein the expected speed includes a change in position or orientation of the feature due to movement of one or more input controls for manipulating the instrument.

51. The non-transitory machine-readable medium according to any one of claims 42 to 50, wherein the registration transformation is between the base coordinate system of the computer-aided device and the coordinate system associated with the imaging device, and wherein the coordinate system associated with the imaging device is the base coordinate system of a second computer-aided device, the imaging device being mounted to a second repositionable arm of the second computer-aided device.

52. The non-transitory machine-readable medium according to any one of claims 42 to 50, wherein updating the registration transformation comprises: The rotation portion of the registration transform is updated based on the error, but the translation portion of the registration transform is not updated based on the error.

53. The non-transitory machine-readable medium according to any one of claims 42 to 50, wherein the method further comprises: The registration transformation is initialized from a seed transformation or another registration method.

54. The non-transitory machine-readable medium according to any one of claims 42 to 50, wherein the method further comprises: The seed transformation is determined based on the image captured by the imaging device of the instrument when a known relative movement occurs between the instrument and the imaging device. as well as The registration transformation is initialized from the seed transformation.

55. The non-transitory machine-readable medium according to any one of claims 42 to 50, wherein the method further comprises: The registration transformation is updated only when the error is less than the configurable upper limit.

56. The non-transitory machine-readable medium according to any one of claims 42 to 50, wherein the method further comprises: The registration transformation is updated when the error exceeds a configurable lower limit.

57. The non-transitory machine-readable medium according to any one of claims 42 to 50, wherein the method further comprises: The magnitude of the error is used to control the speed at which one or more input controls remotely operate the device.

Images