Master / slave registration and control for remote control operations

Abstract

The invention relates to master / slave registration and control for remote control operations. A remotely operated system is disclosed that includes a display, a main input device, and a control system. The control system is configured to determine an orientation of an end effector reference frame relative to a field-of-view reference frame, determine an orientation of a master input device reference frame relative to a display reference frame, establish an alignment relationship between the master input device reference frame and the display reference frame, and determine an orientation of the end effector reference frame relative to the field-of-view reference frame based on the alignment relationship. A change in posture of the end effector is commanded in response to a change in posture of the primary input device. The alignment relationship is independent of a positional relationship between the master input device reference frame and the display reference frame. In one aspect, the teleoperated system is a telecontrolled medical system, such as a telecontrolled surgical system.

Description

[0001] This application is a divisional application of Chinese patent application 201880038183.2, entitled “Master / slave alignment and control for remote control operation”, filed on November 13, 2018.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Application No. 62 / 587,175, filed November 16, 2017, which is incorporated herein by reference in its entirety.

[0004] Copyright Notice

[0005] A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. Technical Field

[0006] The present disclosure relates generally to teleoperated systems, and more particularly to spatial registration and control in teleoperated systems. Background Art

[0007] Examples of teleoperated systems include industrial and entertainment systems. Examples of teleoperated systems also include medical teleoperated systems used in procedures for diagnosis, non-surgical treatments, surgical treatments, and the like. A teleoperated robotic medical system that can be used for telesurgery or other teleoperated medical procedures can include one or more remotely controllable robotic manipulators. In some embodiments, the remotely controllable robotic manipulators can also be configured to be manually articulated or moved.

[0008] Minimally invasive robotic telesurgery systems are specific examples of teleoperated systems that enable surgeons to perform surgical procedures on patients without having to directly hold and move the instruments by hand, using some form of remote control of instrument movement. Surgical systems that incorporate robotics to perform minimally invasive surgery under at least partial computer control extend the benefits of minimally invasive surgery. For example, certain surgical procedures that are difficult or impossible to perform using manually operable minimally invasive surgical tools become possible through the use of such surgical systems. Examples of such surgical systems are commercialized by Intuitive Surgical, Inc. of Sunnyvale, California, USA. Surgical systems.

[0009] In order to establish a desired control relationship between a master input device (also referred to as a "master control device" or "master device") and a corresponding tool for a conventional teleoperated system, motion gestures of the master control device and the tool are determined in three-dimensional (3D) space. In surgical systems, the patient-side unit provides a known and defined kinematic structure that allows the tool posture to be determined. Similarly, the surgeon's console provides a known and defined kinematic structure on which the master input device posture can be determined. New teleoperated system architectures may lack a single mechanical base common to the tools that can be used to determine the kinematic relationship between the tools. Similarly, new teleoperated system architectures may lack a mechanical base common to the master input devices that can be used to determine the kinematic relationship between the master input devices or between the master input device and other devices (such as a display). Therefore, there is a need for improved spatial alignment and control in teleoperated systems. Summary of the invention

[0010] The following summary of the invention introduces certain aspects of the subject matter of the invention in order to provide a basic understanding. This summary of the invention is not an extensive overview of the subject matter of the invention, and it is not intended to identify key or important elements or to delineate the scope of the subject matter of the invention. Although this summary of the invention contains information related to various aspects and embodiments of the subject matter of the invention, its purpose is to present some aspects and embodiments in a general form as a prelude to the more detailed description below.

[0011] In one aspect, a teleoperated system includes an imaging device, an end effector, a display, a master input device, and a computer control system. The imaging device has a field of view. A reference frame is defined for the field of view, the end effector, the display, and the master input device. The control system determines complete orientation information of the reference frame, but less than all position information of the reference frame. Complete orientation information is also referred to as full orientation information, and full position information is also referred to as full position information. In one aspect, the control system does not determine any position information about the reference frame. In another aspect, the control system determines partial position information about the reference frame. The control system establishes a master / slave control relationship between the master device and the end effector by using complete orientation information of the reference frame, but less than complete position information of the reference frame (partial position information or no position information).

[0012] In various aspects, the teleoperated system includes various means for determining complete orientation information of the reference frame and also, where applicable, less than complete position information of the reference frame. These means include using one or more of: a temporary local mechanical relationship between a pair of teleoperated system units, a fixed sensor locator system, a fixed feature locator system, a simultaneous positioning and mapping system, a machine vision system using images from an imaging device, a fiber optic shape sensor, an accelerometer, a magnetometer, a gyroscope, a vibration detector, and a vibration injector.

[0013] In one aspect, a teleoperated system includes a display, a master input device, and a control system. The control system includes one or more processors and a memory. The memory includes programming instructions suitable for causing the one or more processors to perform operations. The operations include: determining the orientation of an end effector reference system relative to a field of view reference system, determining the orientation of an input device reference system relative to a display reference system, establishing an alignment relationship, and (based on the alignment relationship) commanding a posture change of the end effector in response to a posture change of the master input device. The end effector reference system is defined for the end effector of the tool, and the field of view reference system is movable relative to the field of view reference system and is defined for the field of view of the imaging device. The input device reference system is defined for the master input device, and the display reference system is defined for the image. The alignment relationship includes an alignment relationship between the end effector and the field of view or an alignment relationship between the input device and the display, wherein the alignment relationship between the end effector and the field of view is between the end effector reference system and the field of view reference system and is independent of the positional relationship between the end effector reference system and the field of view reference system, and wherein the alignment relationship between the input device and the display is between the input device reference system and the display reference system and is independent of the positional relationship between the input device reference system and the display reference system.

[0014] In one aspect, a method for operating a medical system includes determining an orientation of an end effector reference system relative to a field of view reference system, determining an orientation of an input device reference system relative to a display reference system, establishing an alignment relationship, and (based on the alignment relationship) commanding a posture change of the end effector in response to a posture change of a master input device. The end effector reference system is movable relative to the field of view reference system and is defined for an end effector of a tool, and the field of view reference system is defined for a field of view of an imaging device. The input device reference system is defined for a master input device of the medical system, and the display reference system is defined for a display of the medical system. The alignment relationship includes an alignment relationship of the end effector to the field of view or an alignment relationship of the input device to the display, wherein the alignment relationship of the end effector to the field of view is between the end effector reference system and the field of view reference system and is independent of a positional relationship between the end effector reference system and the field of view reference system, and wherein the alignment relationship of the input device to the display is independent of a positional relationship between the master input device reference system and the display reference system.

[0015] In one aspect, a teleoperated system includes a display, a host device, and a control system. A display reference system is defined for the display, and a host device reference system is defined for the host device. The control system includes a memory storing instructions that, when executed by the control system, cause the control system to perform operations. The operations include: determining the complete orientation of the field of view reference system, determining the complete orientation of the end effector reference system, determining the complete orientation of the display reference system, determining the complete orientation of the host device reference system, establishing a teleoperated master / slave control relationship between the host device and the end effector, and performing a master / slave control relationship between the master device and the end effector. The field of view reference system is defined for the field of view of the imaging device, and the end effector reference system is defined for the end effector of the tool. Establishing the teleoperated master / slave control relationship includes establishing an alignment relationship. The alignment relationship includes an alignment relationship of the end effector to the field of view or an alignment relationship of the master device to the display, wherein the alignment relationship of the end effector to the field of view is between the end effector reference system and the field of view reference system and is based on less than complete position information relating the end effector reference system to the field of view reference system, and wherein the alignment relationship of the master device to the display is between the master device reference system and the display reference system and is based on less than complete position information relating the master device reference system to the display reference system. Executing the master / slave control relationship includes changing the posture of the end effector to correspond to the posture change of the master device.

[0016] In one aspect, the alignment relationship is between a master device reference system and a display reference system and is based on: a complete orientation of the master device reference system, a complete orientation of the display reference system, and less than complete position information relating the master device reference system to the display reference system.

[0017] In one aspect, a teleoperated surgical system includes: means for determining the full orientation of a field of view reference system; means for determining the full orientation of an end effector reference system; means for determining the full orientation of a display reference system of a display on which an image of the end effector is displayed when the surgical end effector is within the field of view of an endoscopic camera; means for determining the full orientation of a master reference system of a master device; means for establishing a teleoperated master / slave control relationship between the master device and the surgical end effector; and means for executing a master / slave control relationship between the master device and the end effector. The field of view reference system is defined for the field of view of the imaging device, and the end effector reference system is defined for the surgical end effector. The means for establishing a teleoperated master / slave control relationship establishes such a relationship by establishing an alignment relationship between the master reference system and the display reference system. The alignment relationship between the master reference system and the display reference system is based on the full orientation of the master reference system and the full orientation of the display reference system, and is independent of the positional relationship between the master reference system and the display reference system. The means for executing a master / slave control relationship executes such a relationship by changing the posture of the end effector to correspond to the change in the posture of the master device.

[0018] In one aspect, a non-transitory machine-readable medium includes a plurality of machine-readable instructions that, when executed by one or more processors associated with a medical device, are suitable for causing the one or more processors to perform any operations or methods described herein.

[0019] Aspects described herein may also include zero, any one, or any combination of the following.

[0020] In some aspects, the alignment relationship includes an alignment relationship of the end effector with the field of view. In some aspects, the alignment relationship includes an alignment relationship of the host device with the display.

[0021] In some aspects, the alignment relationship is a first alignment relationship, and the operation also includes establishing a second alignment relationship, wherein the second alignment relationship includes an alignment relationship between the input device and the display, wherein the first alignment relationship includes an alignment relationship between the end effector and the field of view, and wherein the second alignment relationship includes an alignment relationship between the end effector and the field of view, wherein the first alignment relationship includes an alignment relationship between the input device and the display.

[0022] In some aspects, the operations (or methods) further include establishing a teleoperated master-slave control relationship based on the alignment relationship.

[0023] In some aspects, determining the orientation of the end effector reference frame relative to the field of view reference frame includes: determining the complete orientation of the field of view reference frame, and determining the complete orientation of the end effector reference frame. In some aspects, determining the orientation of the master input device reference frame relative to the display reference frame includes: determining the complete orientation of the display reference frame, and determining the complete orientation of the master input device reference frame.

[0024] In some aspects, the operation (or method) does not include determining a full position of at least one reference frame, the at least one reference frame selected from the group consisting of: a field of view reference frame, an end effector reference frame, a display reference frame, and a primary input device reference frame. In some aspects, the operation (or method) does not include determining a full position of the end effector reference frame relative to the field of view reference frame. In some aspects, the operation (or method) does not include determining a full position of the primary input device reference frame relative to the display reference frame. In some aspects, the operation (or method) also includes determining a less than full position of the end effector reference frame relative to the field of view reference frame. In some aspects, the operation (or method) also includes determining a less than full position of the primary input device reference frame relative to the display reference frame.

[0025] In some aspects, the system is a teleoperated medical system and the tool is a medical tool. In various aspects, the medical tool is a diagnostic tool or a therapeutic tool. In some aspects, the system is a teleoperated surgical system and the tool is a surgical tool.

[0026] In some aspects, the system further comprises a manipulator arm configured to removably support the tool, the manipulator arm comprising a plurality of joints and a plurality of links. In some aspects, commanding a pose change of the end effector comprises using or commanding the manipulator arm to change a pose of the end effector.

[0027] In some aspects, establishing the alignment relationship includes establishing the alignment relationship in response to an indication to initiate teleoperation. In some aspects, the indication to initiate teleoperation includes receiving a user command to initiate teleoperation. In some aspects, the indication to initiate teleoperation includes exiting from a clutch mode. In some aspects, in the clutch mode, the master-slave control relationship is temporarily suspended. In some aspects, in the clutch mode, the control system does not command a posture change of the end effector in response to a posture change of the master input device.

[0028] In some aspects, the operation (or method) further comprises: updating the alignment relationship. In some aspects, updating the alignment relationship comprises: updating the alignment relationship when executing the master / slave control relationship. In some aspects, updating the alignment relationship comprises: updating the alignment relationship at predetermined time intervals.

[0029] In some aspects, the system is a teleoperated surgical system including an imaging device and a tool having an end effector. The imaging device includes an endoscopic camera and the tool includes a surgical tool.

[0030] In some aspects, less than complete position information is position information that does not relate a host device reference system to a display reference system. In one aspect, less than complete position information is partial position information that relates a host device reference system to a display reference system.

[0031] In some aspects, the operations (or methods) further include determining partial position information of at least one reference frame. The reference frame is selected from the group consisting of: a field of view reference frame, an end effector reference frame, a display reference frame, and a host device reference frame.

[0032] In some aspects, changing the posture of the end effector to correspond to the posture change of the master device includes changing the movement direction of the end effector to correspond to the movement direction change of the master device.

[0033] In some aspects, the system also includes a device for determining the spatial position between two or more units of the system, or also includes a spatial determination system that incorporates one or more of the following: a temporary local mechanical relationship between a pair of remotely operated system units, a fixed sensor locator system, a fixed feature locator system, a simultaneous positioning and mapping system, a machine vision system using images from an imaging device, a fiber optic shape sensor, an accelerometer, a magnetometer, a gyroscope, a vibration detector, and a vibration injector.

[0034] Embodiments and aspects are generally described in terms of teleoperated surgical systems, but they are not limited to teleoperated surgical systems. Embodiments in various other teleoperated systems are contemplated, including but not limited to teleoperated systems for military, research, material handling applications, safety, emergency, and manufacturing applications. Therefore, the disclosed technology is applicable to medical and non-medical processes, as well as medical and non-medical tools (e.g., manipulating tools or cameras). For example, the camera or other tools, systems, and methods of any embodiment described herein can be used for non-medical purposes, including industrial uses, general robotic uses, and sensing or manipulating non-tissue workpieces. Other exemplary applications relate to cosmetic improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, setting up or disassembling systems, and training medical or non-medical personnel. Additional exemplary applications include using processes for tissue removed from human or animal anatomical structures (not returning to human or animal anatomical structures), and performing processes on human or animal corpses. In addition, these technologies can also be used for medical treatment or diagnostic processes that include or do not include surgical aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1is a schematic plan view of a teleoperated surgical system.

[0036] Figure 2 is a front view of the patient side unit of the teleoperated surgical system.

[0037] Figure 3 is a perspective view of a teleoperated surgical tool.

[0038] Figure 4 is a front elevation view of the user control unit.

[0039] Figures 5A-5J Schematic diagram of various teleoperated system architectures, spatial alignment, and associated control aspects.

[0040] Fig. 6A is a schematic diagram showing various space determination methods in a remote control system.

[0041] Figure 6B-6C is a schematic diagram showing the determination of a magnetic bearing. DETAILED DESCRIPTION

[0042] The present specification and drawings showing aspects, embodiments, implementations or applications of the invention should not be construed as limiting - the claims define the protected invention. Various mechanical, compositional, structural, electrical and operational changes may be made without departing from the spirit and scope of the present specification and claims. In some cases, well-known circuits, structures or technologies are not shown or described in detail to avoid obscuring the invention. Like numbers in two or more figures represent the same or similar elements.

[0043] In addition, the specific words selected to describe one or more embodiments and optional elements or features are not intended to limit the present invention. For example, spatial relative terms (such as "below", "below", "lower", "above", "upper", "near side", "distal side", etc.) can be used to describe the relationship between an element or feature and another element or feature shown in the figure. These spatially relative terms are intended to cover the different positions (i.e., translation placement) and orientations (i.e., rotation placement) of the equipment in use or operation other than the position and orientation shown in the figure. For example, if the equipment in the figure is flipped, the element described as "below" or "below" other elements or features will be "above" or "above" other elements or features. Therefore, the exemplary term "below" can include the position and orientation above and below. It is possible to orient the equipment in other ways (for example, rotate 90 degrees or be in other orientations), and the spatial relative descriptors used in this article can be interpreted accordingly. Similarly, the description of movement along (translation) and around (rotation) various axes includes various special equipment positions and orientations. The combination of the position and orientation of the body defines the "posture" of the body.

[0044] Similarly, unless the context indicates otherwise, geometric terms such as "parallel," "perpendicular," "circular," or "square" are not intended to require absolute mathematical precision. Rather, such geometric terms allow for variations due to manufacturing or equivalent functions. For example, if an element is described as "circular" or "substantially circular," the description still encompasses components that are not exactly circular (e.g., components that are slightly elliptical or multi-sided polygons). The words "including" or "having" mean including, but not limited to.

[0045] It should be understood that while the specification is sufficiently clear, concise and accurate, strict and exhaustive linguistic precision is not always possible or desirable. For example, considering a video signal, a skilled reader will understand that an oscilloscope described as displaying a signal does not display the signal itself but rather a representation of the signal, and a video monitor described as displaying a signal does not display the signal itself but rather the video information carried by the signal.

[0046] In addition, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are also intended to include the plural forms. Also, the terms "include", "comprise", "include", "have", etc. specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts and / or groups. Also, unless otherwise specified, one or more individually listed items or each of them should be considered as optional items, so that various combinations of items are described without exhaustively listing every possible combination. The auxiliary verb "may" also means that the feature, step, operation, element or part is optional.

[0047] Whenever feasible, elements described in detail with reference to one embodiment, implementation, or application may be optionally included, but not specifically shown or described in other embodiments, implementations, or applications. For example, if an element is described in detail with reference to one embodiment and not described with reference to a second embodiment, the element may still be required to be included in the second embodiment. Therefore, in order to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects, unless otherwise specifically stated, unless one or more elements would render an embodiment or implementation ineffective, or two or more elements provide conflicting functions.

[0048] Unless stated otherwise, elements described as being coupled (eg, mechanically, electrically, communicatively, etc.) may be directly coupled or they may be indirectly coupled via one or more intermediate components.

[0049] Aspects of the present invention are based in part on the use of a surgical device commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. An example of such a surgical system is Surgical system (model IS4200), Surgical System (Model IS4000) and Surgical System (Model IS3000). However, the skilled artisan will appreciate that aspects of the invention disclosed herein may be embodied and implemented in a variety of ways, including computer-assisted and hybrid combinations of manual and computer-assisted embodiments and implementations. The embodiments on the surgical system are exemplary only and should not be considered as limiting the scope of aspects of the invention. For example, the disclosed technology is applicable to medical and non-medical processes, as well as medical and non-medical tools (e.g., manipulation tools or cameras). For example, the tools (e.g., manipulation tools or cameras), systems and methods of any embodiment described herein can be used for non-medical purposes, including industrial uses, general robotic uses, and sensing or manipulating non-tissue artifacts. Other exemplary applications relate to cosmetic improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, setting up or disassembling systems, and training medical or non-medical personnel. Additional exemplary applications include using processes for tissues removed from human or animal anatomical structures (not returned to human or animal anatomical structures), and performing processes on human or animal corpses. In addition, these technologies can also be used for medical treatment or diagnostic processes that include or do not include surgical aspects.

[0050] Those skilled in the art will understand that a computer is a machine that follows programmed instructions to perform mathematical or logical functions on input information to produce processed output information. A computer includes a logical computing unit that performs mathematical or logical functions, and a memory system that stores programming instructions, input information, and output information. The term "computer" and similar terms, such as "processor" or "controller" or "control system", should be considered synonymous. Those skilled in the art will understand that the functions of a computer can be centralized or distributed between two or more locations, and can be implemented with various combinations of hardware, firmware, and software.

[0051] Teleoperated medical systems have been developed that increase operator dexterity or ability to improve ergonomics, etc. For example, minimally invasive teleoperated surgical systems ("teleoperated surgical systems") have been developed that operate at least in part with computer assistance and that operate using a master / slave model in which a user-operated master input device controls a motor-driven slave surgical tool. The user grasps and moves the master input device by remote control to operate the slave surgical tool, rather than directly holding and moving the tool by hand. The slave surgical tool follows the movement of the master input device.

[0052] In minimally invasive teleoperated surgery, an imaging device (such as an endoscopic camera at the surgical site) captures tissue and moving images from the working end of a surgical tool. For convenience, "camera" is generally used herein to refer to an imaging device for capturing one or more images. Examples of imaging devices include imaging devices based on optics, ultrasound technology, magnetic resonance imaging (MRI), CT (computed tomography), X-rays, etc. Examples of endoscopic cameras include single-view, stereo and 3D cameras, as well as cameras that image in the visible spectrum, infrared, ultraviolet, some other part of the spectrum, or a combination of the foregoing. "Tools" are used herein to include imaging and non-imaging instruments. As used herein, the term "end effector" refers to any distal component or portion of a tool, such as the tip of a manipulation, suction, irrigation or cauterization tool, or the tip of an imaging device such as an endoscopic camera (e.g., examples of end effectors include graspers, scissors, cauterization hooks, suction / irrigation nozzles, blunt tips, distal tips of catheters or other flexible devices, lenses of optical imaging devices, probe tips of ultrasonic imaging devices, etc.). The user views the image while operating the master device and sees the slave end effector move in correspondence with the master device's movements. A computer control system provides a control interface between the master device and the slave surgical tool.

[0053] The user usually operates the master device from a position away from the patient (e.g., across the operating room, in a different room, or in a building completely different from the patient). In many telesurgery cases, the user is not in the sterile area and therefore does not interact directly with the patient. However, in some telesurgery cases, the user operating the master device is close enough to the patient so as to optionally interact directly with the patient in the sterile area. The master device is usually free to move in all six Cartesian degrees of freedom (DOF) so that the change of the master device position (translation along the Cartesian coordinate axis) and the change of the master device orientation (rotation around the Cartesian coordinate axis) can cause the corresponding slave tool translation and rotation. This specification is in the context of a Cartesian reference system, and it will be appreciated by those skilled in the art that other suitable three-dimensional reference systems (e.g., cylindrical reference system, spherical reference system) can be used.

[0054] The master device can be in various forms. For example, the master device can be the most distal link in a kinematic chain with redundant mechanical DOFs, a joystick, an exoskeleton glove, etc. In some cases, the master device tracks gestures so that if the translation and rotation of the hand are tracked with sufficient accuracy to perform surgery, the user's hand alone or a portion of the user's hand can act as a virtual master device. The master device can optionally have one or more mechanical DOFs to control corresponding end effector mechanical DOFs, such as a forceps mechanism for the end effector clamp, or a switch (e.g., a button or slider) for the end effector knife movement between the jaws. In addition, the master device can optionally have one or more inputs (such as switches) to control additional end effectors or surgical system features, such as electrosurgical energy application, stapler control, engaging and disengaging the master / slave control relationship ("clutching") between the master device and the slave tool, changing the system operating mode, changing the master device control from one slave surgical tool to a second slave surgical tool, displaying menu selections, etc.

[0055] Surgical tools have many forms, and they include tools for therapeutic and diagnostic functions. Exemplary surgical tools include tissue graspers, needle drivers, scissors, retractors, electrosurgical cauterization tools, staplers, surgical clip applicators, ultrasonic cutters, suction / irrigation tools, catheters, ultrasonic probes, etc. In some cases, cameras (such as endoscopic cameras) or other image capture technologies can be considered surgical tools. Cameras and associated image processing technologies can be used for specialized functions, such as near-infrared image capture, fluorescence energy capture, hyperspectral imaging, etc. These special imaging functions increase the effectiveness of interventions.

[0056] Many teleoperated surgical systems incorporate robotic technology (even though they may not take autonomous actions, they are often referred to as "surgical robots"). Exemplary teleoperated surgical systems are shown in the following patents: U.S. Pat. No. US6,331,181 B1 (filed on October 15, 1999) (describing a multi-port system in which cameras and other surgical tools enter the body only via separate ports), US 8,784,435 B2 (filed on August 12, 2010) (describing a single-port system in which cameras and other surgical tools enter the body via a single common port), and US 8,801,661 B2 (filed on November 7, 2013) (describing a system using flexible surgical tools). The entire disclosures of U.S. Pat. Nos. 6,331,181, 8,784,435, and 8,801,661 are incorporated herein by reference in their entirety.

[0057] A teleoperated surgical system typically includes one or more motor-driven, teleoperated manipulators. The surgical tool is removably mounted on the manipulator, and the manipulator typically moves the entire tool as well as components of the tool, including the tool's end effector. The movement of the manipulator corresponds to the movement of the user's main device, so that the movement of the end effector accurately follows the movement of the main device. Various teleoperated surgical manipulator architectures are known, such as serial kinematic chains, spherical linkages, orthogonal prismatic joints (linear and circular curves), etc. The surgical tool itself can be a single rigid body or a kinematic chain, and thus the kinematic pose of the tool determines the pose of its end effector in space. Similarly, the kinematic pose of the manipulator determines the pose of the surgical tool in space.

[0058] The manipulator is typically maintained in a fixed position and orientation by a non-remotely operated setup structure (such as a motion arm). The setup structure typically includes at least one kinematic pair of links coupled by removable and lockable joints so that the manipulator can be repositioned in space and then maintained in a new posture. The lockable joint(s) may be powered (motorized) or unpowered. And the lockable joint(s) may be locked in various ways, such as by using manual or electrically controlled brakes, or by controlling the powered joints to maintain a fixed relationship between the links in the kinematic pair. The kinematic posture of the setup structure determines the posture of the manipulator in space by keeping the proximal ("base") link of the manipulator fixed.

[0059] In turn, the setup structure can have a proximal-most link ("base") that is optionally fixed to a mechanical ground (e.g., a floor, wall, ceiling, or structure fixed to a floor, wall, or ceiling) or optionally movable with reference to the mechanical ground (e.g., a cart that rolls on a floor, moves along one or more rails on a wall, ceiling, or operating table, etc.). The movable setup structure base also serves to pose the manipulator in space. The manipulator, the optional setup structure (fixed or movable), and the optional base (fixed or movable) together serve as a support structure for surgical tools mounted on the manipulator with reference to the mechanical ground. Any structure that holds a manipulator, an imaging device (such as a camera), or another tool fixed in space relative to the mechanical ground can be used as a support structure. For example, a motorized or non-motorized clamp that holds an endoscopic camera stable in space while the camera captures images of a surgical site can be used as a camera support structure. As another example, an endoscopic camera can be held in place by a support structure that includes a kinematic chain; the kinematic chain can be a passive kinematic chain, or include one or more driven joints. Additional examples of mechanical support structures are described below.

[0060] Figure 11 is a schematic plan view showing components of an exemplary teleoperated system, particularly a multi-port teleoperated surgical system 100 for performing minimally invasive surgery. System 100 is similar to the system described in more detail in U.S. Patent No. US 6,246,200 B1 (filed on August 3, 1999) (disclosing "Manipulator Positioning Linkage for Robotic Surgery"), the disclosure of which is incorporated herein by reference. Additional relevant details are described in U.S. Patent No. US 8,529,582 B2 (filed on May 20, 2011) (disclosing "Instrument Interface of a Robotic Surgical System") and US 8,823,308 B2 (filed on July 1, 2011) (disclosing "Software Center and Highly Configurable Robotic Systems for Surgery and Other Uses"), the disclosures of which are also incorporated herein by reference. A system user 102 (typically a surgeon or other skilled clinician when the system 100 is used for surgery) performs a minimally invasive surgical procedure on a patient 104 lying on an operating table 106. The system user 102 sees a moving image (monoscopic (2D) or stereoscopic (3D)) presented by a display 108 and manipulates one or more master devices 110 at the user's control unit 112. In response to the user's master device movements, a computer 113 acts as a dedicated control system and directs the movement of a remotely operated tool 114 (in this surgical example, the tool 114 is a surgical tool). As described in more detail below, the master device 110 is computationally aligned with the tool 114. Based on this alignment, the computer 113 generates commands that correlate the movement of the master device with the movement of the end effector of the tool 114, so that the motion of the end effector follows the movement of the master device in the hands of the system user 102 in a manner that is intuitive to the user.

[0061] As described above, computer 113 typically includes data processing hardware and machine-readable code embodying software programming instructions to implement the methods described herein (e.g., including associated control systems). Figure 1 The computer 113 is shown as a single box in the simplified diagram of FIG, but the computer may include two or more centralized or distributed data processing units, where at least a portion of the processing is optionally performed near the input device, a portion is performed near the manipulator, etc. Any of a variety of centralized or distributed data processing architectures may be employed. Similarly, the programming code may be implemented as multiple separate programs or subroutines, or may be integrated into various other teleoperated surgical system components.

[0062] like Figure 1 As shown, the system 100 also includes a manipulator assembly 116, which includes two teleoperated manipulators 120 for the tool 114 and a teleoperated manipulator 124 for the tool including an imaging device. For ease of explanation, the imaging device is shown and described as a camera 126 below, and the camera 126 can be any suitable imaging device. For example, the camera 126 can be configured to image optically, ultrasonically or using any other suitable technology. In this surgical example, the camera 126 is an endoscopic camera configured to image in the visible spectrum. Other numbers and combinations of manipulators are optional (e.g., one, three or more manipulators for tools, two or more manipulators for cameras). The manipulator assembly 116 also includes a manipulator setting structure 118 that supports the manipulator 120 during the process. And, the manipulator assembly 116 includes an imaging device setting structure shown as a camera setting structure 122; the camera setting structure supports the manipulator 124. The tool setting structure and the camera setting structure both have base links, and these base links are coupled to a single movable cart. For each tool 114 and camera 126, the associated manipulator, setup structure, and cart illustrate the support structure for the tool or camera.

[0063] As shown, an image of the internal surgical site is displayed to the user 102 via a display 108 in the user control unit 112. The internal surgical site is optionally simultaneously displayed to the assistant 128 via an auxiliary display 130 (2D or 3D). However, as described above and in more detail below, in some teleoperated systems, the user 102 may be in close proximity to the patient 104 (e.g., in a position similar to the illustrated position of the assistant 128) during the surgical procedure. In these teleoperated surgical system architectures, the user may view an image of the surgical site on a 2D or 3D display that is mounted on the floor, wall, ceiling, or other device, as shown by the location of the display 130.

[0064] A sterile assistant 128 (e.g., a nurse, an assistant surgeon, or another skilled clinician) performs various optional tasks before, during, and after the surgical procedure. For example, the assistant 128 can adjust the posture of the manipulators 120, 124, adjust the setup structures 118, 122, swap tools 114 with other tools 132 on the manipulators, operate non-teleoperated medical tools and devices within the patient, hold an endoscopic camera, and perform other tasks associated with teleoperated surgery and general surgery.

[0065] Figure 2 is a front view showing the manipulator assembly 116. Specifically, Figure 2A teleoperated surgical system patient side unit is shown, which shows an embodiment of a multi-port teleoperated surgical manipulator assembly commercialized by Intuitive Surgical of Sunnyvale, California, USA as Surgical System. In a multi-port teleoperated surgical system, tools enter the body through two or more separate incisions or natural orifices. In this example, the manipulator assembly 116 includes four teleoperated surgical tools and camera manipulators supported by a movable patient-side unit.

[0066] In other embodiments of the multi-port teleoperated surgical system, one or more of the manipulators 120, 124 and associated setup structures 118, 122 are mounted, alone or in combination, on one or more separate movable units or fixed to a machine floor, as described herein. It can be seen that various combinations of manipulators and their associated support structures can be used.

[0067] In a single-port teleoperated surgical system, all tools enter the body through a single incision or natural orifice. Examples of manipulator assemblies for a single-port teleoperated surgical system are shown and described in U.S. Pat. No. 8,784,435 B2 (filed Aug. 12, 2010) (disclosing "Surgical System Entry Guide"), and examples of teleoperated surgical systems and flexible surgical tools are shown and described in U.S. Pat. No. 8,801,661 B2 (filed Nov. 7, 2013) (disclosing "Robotic Catheter System and Methods").

[0068] According to one aspect of the present invention, one or more master devices described herein may be used to control two or more different teleoperated surgical system configurations, whether they are two or more different multi-port systems, two or more multi-port, single-port systems, two or more flexible tool systems, or any combination of such multi-port, single-port and flexible tool systems.

[0069] Figure 3is a perspective view of a teleoperated tool 114 that can be used with a teleoperated system. Specifically, a teleoperated surgical tool is shown; the surgical tool includes a distal end effector 140, an optional wrist 141, a proximal end chassis 142, a housing 143 above the chassis 142 (the chassis 142 and housing 143 can optionally be combined), and an elongated shaft 144 coupled between the end effector 140 and the chassis 142. The end effector 140 is coupled to the shaft 144 directly or via the optional wrist 141. Various wrist 141 architectures allow the orientation of the end effector 140 to be changed relative to the shaft 144 in various combinations of pitch, yaw, and roll. Optionally, the end effector roll function is performed by the roll shaft 144. Various mechanisms (combinations of pulleys, cables, levers, gears, universal joints, motors, etc.) are mounted on the chassis 142 and are used to receive mechanical or electrical input from an associated manipulator of the tool 114. These inputs are used to orient and operate the end effector 140. The chassis 142 will typically include a mechanical or electrical interface 146 suitable for coupling to the manipulators 120, 124. As described in more detail in U.S. Pat. No. 6,331,181 B1, the tool 114 will typically include a memory 148, wherein the memory is typically electrically coupled to a data interface (the data interface typically forms part of the interface 146). When the tool is mounted on the manipulator, the data interface allows for the storage 148 and the computer 113 (see Figure 1 ) to carry out data communication.

[0070] End effector 140 also illustrates an endoscopic camera-type tool with an image capture component in place, such as the proximal or distal end of the camera-type tool, and with or without wrist 141. Thus, tool 114, by its structure, also illustrates camera 126 for kinematic purposes, and subsequent references to the kinematic characteristics of tool 114 and the like and control aspects related thereto also apply to camera 126 and the like.

[0071] Various alternative teleoperated surgical tools of different types and different end effectors 140 may be used. Tools associated with at least some manipulators are configured to be removed from their associated manipulators and replaced with alternative tools during a surgical procedure. Additional details are provided in U.S. Patent No. US 8,823,308 B2.

[0072] In some operating environments, the tool 114 and the end effector 140 can be optionally combined into a combination with multiple functions. Additional details related to these combinations are provided in U.S. Patent No. US 7,725,214 B2 (filed on June 13, 2007) (disclosing "Minimally Invasive Surgical System"), the disclosure of which is incorporated herein by reference in its entirety. Details related to the interface between the tool 114 and the manipulator 120 are provided in the following patents: U.S. Patent No. US 7,955,322 B2 (filed on December 20, 2006) (disclosing "Wireless Communication in a Robotic Surgical System") and US 8,666,544 B2 (filed on July 10, 2013) (disclosing "Cooperative Minimally Invasive Telesurgical System"), the disclosures of these patent applications are incorporated herein by reference in their entirety, and are also provided in U.S. Patent No. US 8,529,582 B2.

[0073] Figure 4 is a front elevation view of the user control unit, showing Figure 1 An example of a user control unit 112 is shown. The user control unit 112 includes a display 108 in which an image of a working site (e.g., a surgical site in the surgical example) is displayed to a user 102 (e.g., a surgeon or other skilled clinician in the surgical example). A support 111 is provided on which the user 102 can rest his forearms while holding two master devices 110, one for each hand. The master devices 110 are positioned behind the support 111 and generally in the space below and behind the display 108. When using the control unit 112, the user 102 generally sits in front of the control unit 112, positions the eyes in front of the display 108, and holds the master devices 110 (one for each hand) while resting the forearms on the support 111. The master devices are positioned so that the image of the associated end effector is located between the master device and the eye, so that when the user sees the end effector in place of the hand, the movement of the master device intuitively moves the master device. The user control unit 112 may optionally include a computer 113 or a portion of a computer 113 for establishing and maintaining a teleoperated control relationship between the master device 110 and an associated tool 114 and its end effector 140 .

[0074] The effective teleoperation control relationship between the master device and its slave tool (e.g., the slave surgical tool in the surgical example) and the end effector requires spatial alignment between the master device and the end effector. Alignment must provide a reasonably accurate relationship between the master device motion perceived by the user (e.g., proprioception) and the end effector motion (e.g., visual perception) perceived by the user. For example, if the user moves the hand holding the master device to the left, the user expects to perceive the associated slave manipulator to move to the left. If the perceived spatial motion matches, the user can easily control the movement of the slave device by moving the master device. However, if the perceived spatial motion does not match (e.g., the master device moving to the left causes the slave to move upward and to the right), slave control is difficult. The required alignment is completed using known kinematic relationships and reference frame transformations in the teleoperation system (e.g., the teleoperation surgical system in the surgical example). These relationships are described below in Cartesian terms, but other 3D coordinate systems can also be used for systems that function in 3D space.

[0075] 1. Architecture and Reference System

[0076] Figure 5A is a schematic diagram of teleoperated system components (e.g., teleoperated surgical system components in the surgical example) and associated Cartesian reference frames. Figure 5A As shown, camera 126 (e.g., an endoscopic camera in the endoscopic surgery example) has a field of view (FOV) 127. The distal portion of tool 114 (e.g., a surgical tool in the surgical example) has a wrist 141 and an end effector 140, wherein its wrist 141 and its end effector 140 are within FOV 127. An end effector reference frame 150 is associated with end effector 140, and a tool reference frame 151 is associated with a proximal portion (body) of tool 114, such as a portion external to the patient (chassis, housing, proximal shaft, etc.). If the tool does not have a wrist 141, reference frames 150 and 151 can be combined into a single reference frame sufficient to locate and orient the proximal portion of tool 114 and end effector 140, or reference frames 150, 151 can remain separate, each with an origin at a different location. Similarly, field of view reference frame 152 is associated with FOV 127, and imaging device reference frame 153 is associated with a main portion (such as a proximal portion of the patient exterior) of camera 126. If camera 126 does not have wrist 141, reference frames 152 and 153 may be combined into a single reference frame, or they may remain separate, each with an origin at a different location.

[0077] For systems where the physical dimensions of all tools 114 (e.g., surgical tools in the surgical example) including the camera 126 and mechanical linkages are known and all joint angles between these mechanical linkages can be determined (using direct rotation sensors, motor position sensors, fiber optic shape sensors, etc.), the kinematic relationship between the reference frame 150 or 151 and the reference frame of any other linkage in the tool 114 can be determined by using well-known kinematic calculations. Similarly, the kinematic relationship between the reference frame 152 or 153 and any other linkage in the camera 126 can be determined. Therefore, for such systems where the end effector 140 operates within the FOV 127, the alignment between the reference frames 150 and 152 will allow the user to easily control the end effector 140 in the FOV 127.

[0078] Figure 5A Also shown is user 102 viewing display 108 and grasping master device 110. Display 108 displays an image within FOV 127. As shown, an image 140a of end effector 140 is displayed on display 108. Display reference frame 154 is associated with display 108, and master device reference frame 156 is associated with master device 110. When master device 110 translates and rotates in 3D space, its associated reference frame 156 translates and rotates accordingly. Known methods can be used to sense the translation and rotation (posture changes) of these reference frames 156 and mathematically transform them into the reference frame 150 of end effector 140 so as to provide a control relationship between master device 110 and end effector 140 by using well-known kinematic calculations. As the position and orientation of reference frame 156 of master device 110 changes, the position and orientation of reference frame 150 of end effector 140 changes accordingly, so that the movement of end effector 140 is subordinate to and follows the movement of master device 110. User 102 views the position and orientation changes of end effector 140 on display 108. In order to establish the desired simple and intuitive control relationship between the master device movement and the end effector image movement, relationships are established between reference frames 150 and 152 and between reference frames 154 and 156. Once these reference frame relationships are established, the movement of reference frame 150 relative to reference frame 152 can be controlled to accurately or acceptably match the movement of reference frame 156 relative to reference frame 154.

[0079] Figure 5B is another schematic diagram of teleoperated system components and the associated Cartesian reference frame. Figure 5B The mechanical support structure is shown with reference to the mechanical ground support Figure 5A Various components shown. Figure 5BReference frames associated with these support structures are also shown. For simplicity in this and the following figures, each support structure is depicted as a kinematic pair - two links coupled by a movable joint. However, it should be understood that the support structures can be a variety of optional configurations, such as a single link with zero DOF, a single kinematic pair with 1 or more DOF, or a combination of kinematic pairs with 2 or more DOF. And, support structures with 2 or more DOF can optionally have joints that give the support structures redundant DOF. Any of a variety of kinematic joints can be used (rotational, prismatic, spherical, etc. as shown).

[0080] like Figure 5B As shown, the main device 110 is supported by a main device support structure 160, which starts from a mechanical ground portion 160a (also referred to as "base 160a") and extends distally until coupled with the main device 110. Reference frame 161 is associated with one of the links of the main device support structure 160. Similarly, the display 108 is supported by a display device support structure 162, which starts from a mechanical ground base 162a and extends distally until coupled with the display 108. Reference frame 163 is associated with one of the links of the display device support structure 162. Similarly, the camera 126 is supported by an imaging device support structure (shown as camera support structure 164), which starts from a mechanical ground base 164a and extends distally until coupled with the camera 126. Camera support structure reference frame 165 is associated with one of the links of the camera support structure 164. And similarly, the tool 114 is supported by a tool support structure 166 that begins at a mechanically grounded base 166a and extends distally until coupled to the tool 114. A reference frame 167 is associated with one of the links of the tool support structure 166.

[0081] Figure 5BIt is also shown that the master device support structure 160 can be optionally configured as a support structure when there is a break in the kinematic chain between the master device 110a and the grounded portion 160a of the master device support structure. This configuration exists when the master device 110 is not mechanically grounded (i.e., the master device 110a is an "ungrounded" master device). In order to communicate with the master device 110a, a tether or wireless connection including a communication line can be used. It will be recalled that the master device can be one or more unenhanced hands of the user 102, and therefore the spatial sensing of hand gestures is a wireless implementation of an ungrounded master device. Control commands 160b from the master device 110 - position and orientation changes referenced to the master device reference frame 156 and any additional control inputs from the master device itself (buttons, joysticks, finger movements, hand gestures, etc.) - are received via the master device control input receiver 160c, via a tether, via a wireless signal from the master device, or by free space gesture sensing. The control commands are then routed to the computer 113 for processing and corresponding slave control actions. Examples of ungrounded master devices are given in the following patents: U.S. Patent Nos. 8,521,331 B2 (filed on November 13, 2009) (disclosing "Patient-side Surgeon Interface for a Minimally Invasive, Teleoperated Surgical Instrument"), US 8,935,003 B2 (filed on September 21, 2010) (disclosing "Method and System for Hand Presence Detection in a Minimally Invasive Surgical System"), and US 8,996,173 B2 (filed on September 21, 2010) (disclosing "Method and Apparatus for Hand Gesture Control in a Minimally Invasive Surgical System"), which are incorporated herein by reference.

[0082] When determining joint positions, well-known forward or inverse kinematic calculations are used to transform between the master device reference frame 156 and the master device support structure reference frame 161; between the display reference frame 154 and the display support structure reference frame 163; between the imaging device field of view reference frame (referred to as the camera FOV reference frame 152), the imaging device reference frame (camera reference frame 153) and the imaging device support structure reference frame (camera support structure reference frame 165); and between the end effector reference frame 150, the tool reference frame 151 and the tool support structure reference frame 167. For example, see U.S. Pat. Nos. 5,631,973 (filed May 5, 1994) (disclosing "Method for Telemanipulation with Telepresence"), 5,808,665 (filed September 9, 1996) (disclosing "Endoscopic Surgical Instrument and Method for Use"), and US 6,424,885 B1 (filed August 13, 1999) (disclosing "Camera Referenced Control in a Minimally Invasive Surgical Apparatus"), the disclosures of which are incorporated herein by reference in their entirety.

[0083] Figure 5B Also shown is a world reference frame 168 that is stationary relative to the depicted mechanical ground. Reference frame 168 may be directly associated with the mechanical ground, or it may be associated with some other structure that remains stationary relative to the mechanical ground. Also, a gravity vector (g) 169 is shown with reference to world reference frame 168 arbitrarily aligned with the z-axis of reference frame 168, but it should be understood that the world reference frame may alternatively be at any orientation relative to the gravity vector. Magnetic North Pole is another example of a reference axis that may be associated with a stationary world reference frame.

[0084] Figure 5C is another schematic diagram of teleoperated system components and the associated Cartesian reference frame. Figure 5C It is shown that the support structures for the system components can be optionally combined in various ways, and the reference frame can be associated with the combined support structures. For example, the main device support structure 160 and the display device support structure 162 can be combined into a common control support structure 170 as shown. The control support structure 170 extends from a proximal base 170a at the machine floor, and then branches distally to support the display 108 and the main device 110. An example of such a control support structure common to the main device and the display is Figure 4 A user control unit 112 is shown.

[0085] The control support structure 170 may also be configured to support a configuration of an ungrounded master device 110a, as described above and as Figure 5C shown.

[0086] A control support structure reference frame 171 is associated with the control support structure 170. Well-known forward or inverse kinematic calculations are used to transform between the display reference frame 154, the host device reference frame 156, and the control support reference frame 171. Also, as shown, the control support reference frame 171 can be used in an ungrounded host configuration.

[0087] As another example, it is shown how the support structures can be optionally combined. Figure 5C The camera support structure 164 and the tool support structure 166 are shown combined into a single equipment support structure 172 (e.g., a surgical equipment support structure in a surgical example). The equipment support structure 172 extends from a proximal base 172a at the machine floor and then branches distally to support the tool 114 and the camera 126. An example of such an equipment support structure is Figure 2 Manipulator assembly 116 is shown.

[0088] The device support reference frame 173 is associated with the device support structure 172. Where necessary, well-known forward or inverse kinematics calculations may be used to transform between the end effector reference frame 150, the tool reference frame 151, the FOV reference frame 152, the camera reference frame 153, and the device support reference frame 173. It will be recalled that in some cases a person may hold the camera 126 and thus act as the camera support structure.

[0089] Figure 5C Also shown are a world reference frame 168 and a gravity vector (g) 169 relative to the control support structure 170 and the device support structure 172 and relative to their associated reference frames. The world reference frame 168 and the gravity vector 169 are shown to illustrate that they can be relative to the reference frames 171 or 173 as needed. Figure 5B The various reference frames shown are used. Additional depictions are included below.

[0090] Those skilled in the art will appreciate that various support structures can support a single object as shown, or alternatively, they can support two or more similar or dissimilar objects. For example, the master device support structure 160 can support two master devices 110 (e.g., one master device 110 for each left and right hand to control a corresponding separate tool 114, as shown in the user control unit 112). Alternatively, the master device support structure 160 can support three or more master devices (e.g., two master devices 110 to control a corresponding separate tool 114, and a third master device to control a third tool 114 or a camera 126). A combination of one or more kinematically grounded master devices and one or more ungrounded master devices can be supported. And, if two or more master devices 110 are supported, the display 108 may or may not be supported. Similarly, the tool support structure 166 and the device support structure 172 can support two or more tools 114, support or not support the camera 126 (e.g., the manipulator assembly 116).

[0091] In addition, the teleoperated system may optionally include a combination of two or more main device support structures 160, display device support structures 162, camera support structures 164, tool support structures 166, control support structures 170, and device support structures 172. For example, The Xi surgical system has a control support structure 170 that supports a single display 108 and two main devices 110 , and it has an equipment support structure that supports an endoscopic camera 126 and up to three tools 114 . The Xi Surgical System optionally includes a second control support structure 170 that supports a second single display 108 and a second set of two master devices 110, and this second support structure may be used, for example, in a training situation.

[0092] Those skilled in the art will appreciate that the various reference frames shown may be kinematically optionally combined in various ways into a single reference frame. For example, the main device support reference frame 161 or the control support reference frame 171 can be used as the main device reference frame 156. Similarly, the display reference frame 154 can be used as the main device reference frame 156. Similarly, the camera body reference frame 153 (imaging device body reference frame) can be used as the FOV reference frame. In order to avoid unnecessary description, not all of the various combinations are listed, but all such combinations are within aspects of the present invention. Generally, a reference frame associated with any link (including the farthest link) in one kinematic chain can be used as a reference frame associated with any link (including the farthest link) in a second kinematic chain. According to aspects of the present invention, a kinematic relationship between two reference frames is established as needed.

[0093] As described above, when the display 108 and one or more master devices 110 are supported in a common control support structure 170, and when kinematic pose information about the display and the master device is determined, well-known kinematic calculations can be used to establish the required control alignment between the reference frames associated with the display and one or more master devices. This is because the position and orientation relationship between the display 108, one or more master devices 110, and the control support structure 170 is known. Similarly, when the camera 126 and one or more tools 114 are supported in a common device support structure 172, well-known kinematic calculations can be used to establish the required control alignment between the reference frames associated with the camera and its FOV, one or more individual end effectors 140 corresponding to one or more individual tools 114, and the device control structure, because the position and orientation relationship between these objects is known. In order to establish the control alignment required for teleoperation (e.g., telesurgery in the surgical example), the control support reference frame 171 is transformed into the device support reference frame 173, and thus the master device / display and camera end effector / FOV reference frame alignment is established.

[0094] For example, the new teleoperated system architecture may lack a single mechanical base common to the tools that can be used to determine the kinematic relationship between the tools. Likewise, the new teleoperated system architecture may lack a mechanical base common to the master input devices that can be used to determine the kinematic relationship between the master input devices or between (one or more) master input devices and other devices (such as a display). Therefore, there is a need for improved spatial registration and control in teleoperated systems.

[0095] For example, a teleoperated system may include two or more units carrying tools, wherein the units may be movable relative to each other so that the kinematic relationship between the units is not easily defined by mounting to the same mechanical base. In addition, a tool (such as a manipulator or a camera) may be supported by a passive support structure that is not equipped with any sensors, or the tool may be held by a person, and therefore the computer control system cannot determine the kinematic information of the tool from the support structure of the tool. Therefore, there is no single mechanical base common to these units (e.g., in a medical embodiment, between an endoscopic camera and one or more other medical tools) that can be used to determine the kinematic relationship between tools held by different units. In addition, one or more units may be added or removed as needed during a procedure (e.g., during a surgical procedure in a surgical example).

[0096] A similar situation may exist with respect to master control devices used to control tool motion in these new systems. The master control devices may not share a common mechanical base, and the kinematic relationship between a display (e.g., a display showing an image of a work site captured by an imaging device) and one or more master control devices (e.g., a master input device used to control the gesture of one or more tools, such as one or more manipulation or imaging tools) may not be determined solely from kinematic data. Furthermore, one or more master input devices may be added or removed as needed during the process.

[0097] Therefore, in the case of using one or more separate main equipment, display, imaging device and tool support structures, however, it is more difficult to establish the position and orientation relationship between the main equipment and the display and between the imaging device and other tools. If the position and orientation are to be used to establish the required control alignment, the position and orientation of each individual support structure must be determined. And in addition, if the position or orientation of the individual support structure changes during use (for example, within a surgical operation in the surgical example), the new position and orientation of the changed support structure must be determined to establish the required control alignment again. However, it is generally difficult to determine the position and orientation of the individual support structures with sufficient accuracy.

[0098] Figure 5D2 is a schematic plan view of a medical example showing a patient and two patient-side units, which illustrates an exemplary situation in which separate camera and tool support structures are used during a medical procedure. Patient 200 is shown on an operating table 202. Schematic camera support structure 204 is shown as a mobile unit that can move on the floor of an operating room. As described above, camera support structure 204 supports an endoscopic camera 206, which has a FOV positioned toward a working site 208 (a medical site in this example, such as a surgical site) in the patient's body. Schematic tool support structure 210 is included, which is also shown as a mobile unit that can move on the floor of the operating room. Tool support structure 210 supports tool 212, which is positioned to position its end effector 214 at the surgical site within the patient and FOV. Camera support structure 204 represents a single camera support structure supporting a single camera, a single camera support structure supporting two or more cameras, and two or more separate camera support structures each supporting one or more cameras. Likewise, tool support structure 210 represents a single tool support structure supporting a single tool, a single tool support structure supporting two or more tools, and two or more separate tool support structures each supporting one or more tools. Additionally, camera support structure 204 and tool support structure 210 optionally represent combined equipment support structures as described above. Therefore, to avoid an unnecessarily lengthy description of all possible variations, those skilled in the art will appreciate that the following description of camera support structure 204 and tool support structure 210 also applies to various other support structures that each may represent.

[0099] As shown, the camera support structure 204 is in a certain pose 204a relative to the mechanical ground (in this example, the floor 216). The camera support structure reference frame 218 is associated with a single link of the camera support structure's kinematic chain (e.g., the base link at the mechanical ground, the links of the setup structure, the links of the manipulator, the links of the camera itself; the pose of the camera's farthest link (which may be the camera body itself) is also used to define the reference frame of the camera FOV). The orientation of the camera support structure reference frame 218 changes as the orientation of the associated individual link changes, and kinematic calculations are then used to determine the orientation of any other links in the camera support structure.

[0100] exist Figure 5D 2, which is shown in a first pose 210a relative to the machine floor (floor 216). The tool support structure reference frame 220 is associated with a single link of the kinematic chain of the camera support structure (e.g., a base link at the machine floor, a link of the setup structure, a link of the manipulator, a link of the tool itself, including the end effector). Figure 5DAlso shown is a tool support structure 210 in a second optional posture 210b relative to the machine ground, which shows that the tool support structure can be placed in various positions and orientations within and during teleoperation (e.g., during teleoperated surgery in the surgical example). The reference frame 220 changes as its associated link on the tool support structure changes, as shown by arrow 222. Those skilled in the art will understand that, as shown, the various postures of the tool support structure 210 also represent various postures of one or more additional individual or combined tool structures, as well as optional postures of the camera support structure 204 and optionally one or more additional individual or combined camera support structures, and one or more cameras and tool support structures combined into one or more separate individual device support structures. However, for all of these optional combinations, the support structure supporting the camera is separate from the support structure holding the tool that needs to be registered in the camera's FOV reference frame.

[0101] Figure 5E is another schematic plan view of a patient and two patient side units, showing a second example of a varying orientation of the separate camera and tool support structures. Figure 5E , a camera and tool support structure is mounted on a table 202 (e.g., a medical table in the medical example, such as a surgical table in the surgical example). For example, the camera and tool support structure can be mounted in various positions along the side rails of the table, which serve as a machine floor. A camera support structure 224 for camera 206 is mounted to the table 202 at a base location 224a. Similarly, a tool support structure 226 for tool 114 is mounted to the table 202 at a first base location 226a. The tool support structure 226 is optionally mounted to the table 202 at a second base location 226b, which again illustrates that the tool support structure can be placed in various positions and orientations within and during a teleoperated operation (e.g., during teleoperated surgery in the surgical example). In Figure 5D In FIG. 1 , the position and orientation of the tool support structure base are shown as being changed, and in FIG. 1 , the position and orientation of the tool support structure base are shown as being changed. Figure 5E , only the position of the base of the tool support structure is shown as changing, as the base orientation does not change when it is in different positions along the table rails. However, the position and orientation of the tool support structure base may change in other configurations, such as when the tool support structure is moved from one side of the table to the opposite side of the table. In addition, the table pose may change, which changes the base orientation and position accordingly. Again, as described above, various combinations of cameras and tool support structures are possible. And again, for all of these optional combinations, the support structure supporting the camera is either completely physically separated from the support structure holding the tool that needs to be registered in the camera's FOV reference frame, or is mechanically coupled via a shared support structure that also holds the tool but does not share kinematic information and is therefore effectively kinematically separated.

[0102] 2. Controlled alignment

[0103] In order to effectively move the distal end of the tool relative to the camera FOV reference frame, an alignment relationship is determined between the camera FOV and the end effector of the tool (i.e., between the reference frame associated with the camera FOV and the reference frame associated with the end effector). In addition, an alignment relationship is determined between the master device and the display (between the reference frame associated with the master controller and the reference frame associated with the display), which outputs an image from the camera showing the end effector. Examples of establishing such an alignment relationship and forcing the master device to a pose corresponding to the displayed end effector pose are found in U.S. Patent No. 6,424,885 B1 and U.S. Patent No. 6,459,926 (filed on September 17, 1999), the disclosures of which are incorporated by reference in their entirety. In these examples, the master device is mounted at the end of a robot master manipulator arm. In order to establish the necessary master / slave control relationship, the master manipulator arm moves the master device to a pose in the display reference frame that corresponds to the pose of the slave end effector in the camera FOV reference frame. This movement aligns the master device pose with the displayed end effector pose and thus establishes a visually and ontologically intuitive control relationship between the master device and the end effector. Thus, in various embodiments, the relationship between the end effector reference frame and the FOV reference frame and between the display reference frame and the master device reference frame is determined.

[0104] In an embodiment, the user's perception of intuitive master / slave control depends on at least two main perceived correlations between the master and the slave. First, it depends on the user's perception of the correlation between the master (master device) orientation in space and the slave (end effector) orientation in space - that is, the perceived orientation correlation. Second, it depends on the user's perception of the correlation between the movement direction of the master (master device) and the movement direction of the slave (end effector) - the perceived movement direction correlation.

[0105] Therefore, the user's proprioception of the orientation of the master device should be within an acceptable tolerance of the user's visual perception of the orientation of the corresponding end effector image in the display. For many tools and / or master devices, the long axis of the master device should be considered to be oriented in the direction of the long axis of the end effector in the display. However, for other tools and / or master devices, the master device and / or end effector orientation axis used for control may not be the long axis. For example, a pistol grip master device may not have a long axis that is perceived to be aligned with the long axis of the end effector, but the user can still perceive the orientation correlation between the pistol grip master device and the end effector. In addition, if the gripping function of the end effector intuitively corresponds to the gripping movement of the master device, the orientation of the plane of the gripping movement of the master device should be perceived as corresponding to the orientation of the plane of the gripping movement of the end effector in the display. (This is a match between the roll angle of the master device around its actual or perceived long axis and the roll angle of the end effector around the long axis of the end effector.)

[0106] Likewise, the user's proprioception of the direction of movement of the master device should be within acceptable tolerances of the user's visual perception of the direction of movement of the corresponding end-effector image in the display.

[0107] Individual users will have different personal tolerances for perceived orientation and movement direction correlation. For example, some users may tolerate a 20-40 degree mismatch in perceived orientation correlation. And, some users are affected by as little as 5 degrees of perceived movement direction correlation mismatch. We found that when the teleoperated surgical system first establishes a control relationship between the master device and the end effector to begin master / slave operation, and as the system continues to update and maintain the control relationship between the master device and the end effector during master / slave operation, the perceived master / slave orientation and movement direction correlation is more important than the perceived position correlation for adequate performance.

[0108] Reference Fig. 5F , oriented to the Z axis EL Associated with the display image of the left end effector and oriented about the axis Z ER Associated with the displayed image of the right end effector. Likewise, the orientation is aligned to the axis Z ML Associated with the left master and oriented about axis Z MR Associated with the right master. Orientation axis is in Z ML1 and Z MR1 The spacing of the master device positions at the end effector generally corresponds to the end effector orientation axis Z. EL and Z ER However, the main device orientation axis Z ML2 and Z MR2 The spacing distance is greater than the end effector orientation axis Z EL and Z ERHowever, the main device orientation axis Z ML2 and Z MR2 The position of the main device provides effective intuitive control. In a similar manner, the main device orientation axis Z ML2 and Z MR2 The vertical position difference (e.g., the left side is higher than the right side) or the depth position difference (e.g., the left side is farther than the right side) also provides effective intuitive control. ML1 The right main device orientation axis and Z MR2 The position of the left master orientation axis is used to establish and maintain effective control.

[0109] Still reference Fig. 5F , move the 3D space by V at the first master device position MR1 Associated with the right master, and moves the parallel 3D space V at the second master position MR2 Associated with the right master. Although the right master is at a separate location in 3D space, these mobile V MR1 and V MR2 are all perceived as the 3D spatial movement V of the right end effector ER relevant for effective intuitive control.

[0110] 3. Orientation Alignment and Orientation Alignment Only

[0111] In order to provide the desired perceptual correlation of orientation and movement direction between the master device and the end effector for effective intuitive control by the user, the control system determines and aligns the relationship between the associated reference frames to initiate master / slave control for teleoperation (e.g., for telesurgery in the surgical example) and to maintain and update master / slave control during teleoperation (e.g., for telesurgery in the surgical example).

[0112] According to aspects of the invention, the desired alignment relationships between the camera FOV and the end effector and between the host device and the display are established with reference only to the orientation of these objects rather than to their positions, that is, with reference to the orientation of the reference system associated with these objects rather than to the positions of the reference system associated with these objects. For control, an absolute orientation is determined between the end effector alignment orientation axis and the camera FOV alignment orientation axis, and an absolute orientation is established between the host device alignment orientation axis and the display alignment orientation axis. When determining the orientation relationship for control, the end effector can be located inside or outside the camera FOV.

[0113] According to aspects of the invention, the desired alignment relationship between the camera FOV and the end effector, and between the host device and the display, is established with reference to the full orientation of these objects but with reference to less than their full position. The full orientation is also referred to as the "full orientation", and the full position is also referred to as the "full position". Examples of less than full position information include no position information, and including partial position information (such as position information along only one or two axes). Therefore, in examples utilizing Cartesian coordinates, the required alignment is established with reference to the orientation of the reference system associated with these objects around all three Cartesian coordinate axes (e.g., rotated around the X-axis, Y-axis, and Z-axis), and with reference to the position of the reference system associated with these objects along zero, one, or two Cartesian coordinate axes (e.g., not along the X-axis, Y-axis, or Z-axis, or only along one or two of the X-axis, Y-axis, and Z-axis). Control is then established using the full orientation information and less than full position information.

[0114] The following description focuses on the use of orientation information, and it should be understood that, additionally, less than full position information (ie, along zero, one, or two Cartesian axes) may optionally be combined with full orientation information to establish and maintain the described control.

[0115] A full uniform transformation may be used, but is not necessary to establish and maintain effective alignment between reference frames for intuitive control. Once the orientation of any individual link in the kinematic chain for these objects (which may be the object itself) is known, an effective control relationship can be established with less than full position information for that individual link. Less than full position information for a link may mean no position information for that link, and some embodiments use only orientation information to establish control relationships. Using only orientation for alignment simplifies the alignment task because it does not require determining the absolute or relative position of these objects or their supporting structures, nor does it require constructing a combined support structure that establishes a common reference frame for these objects or supporting structures. Less than full position information may also mean partial position information, and some embodiments use both orientation information and partial position information when establishing control relationships. Using partial rather than full position information also simplifies the alignment task by reducing the amount of position information that is determined. Thus, if Figure 5B-5E As shown, the various separate individual objects and their supporting structures can be properly aligned for control. Establishing the alignment relationships required for effective control is accomplished through the well-known kinematic transformation from one reference frame to another.

[0116] Once the initial orientation alignment relationships required for effective teleoperation control are established between the various reference frames (e.g., orientation-only alignment relationships, or full orientation and partial position alignment relationships), changes in position and orientation relative to these reference frames are used to perform teleoperation (e.g., telesurgery in the surgical example). For example, changes in the position of the master device are performed as corresponding 1:1 or scaled position changes of the tool's end effector, changes in the orientation of the master device are performed as corresponding orientation changes of the tool's end effector, and changes in the posture of the master device are performed as corresponding posture changes of the tool's end effector. However, while these control relationships are in effect during teleoperation, alignment between the reference frames can be maintained by using only the orientation of the reference frame and not its position, or by using the orientation with reference to less than all of its position information.

[0117] In one aspect, the end effector orientation is determined relative to the axis orientation of the tool based on wrist function. In some cases, all three end effector orientation DOFs relative to an axis are independent of the axis to which the end effector is coupled. In other cases, less than three end effector orientation DOFs relative to an axis are independent of the axis. For example, the end effector pitch and yaw DOFs can be independent of the axis, and the end effector roll orientation about the z-axis is determined by the axis roll DOF. As another example, the end effector yaw DOF can be independent of the axis, but the pitch and roll of the end effector are determined by the corresponding axis pitch and roll DOFs. Therefore, in some cases where a transformation from the tool reference frame to the end effector reference frame occurs to establish the alignment required for control, three, two, or one rotation may be required, depending on the distal configuration of the tool.

[0118] In one aspect, the orientation of the end effector reference frame is determined relative to a reference frame other than the tool reference frame. For example, as described in more detail below, the orientation of the end effector reference frame is determined relative to the orientation of a reference frame of a field of view (FOV). For example, in an endoscopic surgical example, the orientation of the end effector reference frame (of a tool at a surgical site) is determined relative to the orientation of a field of view reference frame (for an endoscopic camera whose FOV partially or fully covers the surgical site, also referred to as the camera FOV). In this example, the end effector can be located inside or outside the field of view associated with the field of view reference frame.

[0119] In one aspect, the orientation of the input device frame is determined relative to a reference frame of an image displayed by a display and is viewable by a user interacting with the input device.

[0120] According to another aspect of the invention, when an event occurs in the remote control system (such as a signal received by the computer of the remote control system indicating that the user wishes to start remote control), an initial orientation alignment relationship required for control (e.g., only orientation alignment relationship, or full orientation and partial position alignment relationship) is established. Such an event may be a button press or other actively controlled event, so that remote control is not accidentally started.

[0121] Examples of such system events may be transitioning between teleoperation of one or more tools and teleoperation of one or more endoscopic cameras, replacing a first tool with a second tool on a teleoperated manipulator, and other actions expected to occur throughout a procedure (e.g., through a surgical procedure in the surgical example). As a specific example, one event that may optionally be used to trigger a request to establish a control relationship is exiting a "clutch" mode, in which the master / slave relationship between the master control device and the end effector is temporarily suspended. Clutch mode is analogous to the use of a mechanical clutch that engages and disengages a coupling between objects. As Figure 5G As shown, at a first time t1, the master device orientation axis ZM is in a first position as shown, and is perceived to be related to the orientation axis ZE of the image of the corresponding slave end effector in terms of orientation and movement. That is, as shown, the user remotely controls the end effector and senses that at time t1, the z-axis ZM of the master device is aligned with the z-axis ZE of the image of the end effector. Then, the user enters the clutch mode, moves the master device to the second position shown by the arrow, and exits the clutch mode at time t2. At t2, the user again perceives that the orientation axis ZM of the master device is related to the orientation axis ZE of the image of the end effector shown in terms of orientation and movement, but the position of the orientation axis ZM at t2 is different from the position at t1.

[0122] According to another inventive aspect, the initial orientation alignment relationship required to establish control (e.g., only the orientation alignment relationship, or the full orientation and partial position alignment relationship) is updated during remote control operation to further refine the relationship to correct for drift that may occur in various sensors and other system components. These updates are optionally performed using only orientation information rather than position information. Updates can optionally occur at various times, such as at predetermined time intervals or at one or more system events. As an example of updating at predetermined time intervals, the end effector posture is updated approximately every 1ms (milliseconds) to correspond to the main posture, and therefore the alignment relationship is updated every 100 cycles (approximately every 100ms). In addition, updates can be performed at a combination of system events and predetermined time intervals, such as when remote control operation is reestablished after the main device clutch moves, and thereafter updated at a predetermined number of clock cycles.

[0123] Alternatively, the alignment relationship between the reference frames can be refined by a ratcheting process, for example, as described in U.S. Patent No. 8,423,186B2 (filed on June 30, 2009), which is incorporated herein by reference, to converge on the perceived orientation alignment of the master device and the end effector after an event such as clutching. For example, where the orientation of the input device reference frame relative to the display reference frame is a first relative orientation, and the orientation of the end effector reference frame relative to the field of view reference frame is a second relative orientation, and there is a certain difference between the first relative orientation and the second relative orientation, the system can update the second alignment relationship multiple times to gradually reduce the difference.

[0124] As another example, the system can integrate the orientation difference and apply a portion of the integrated difference to the commanded motion. In this alternative, the end effector reference frame changes with the commanded motion, and the commanded motion dynamically reduces the orientation difference. In an embodiment, the system determines: the commanded orientation change of the end effector, the residual orientation difference, and the maximum reduction in the difference (such as the percentage of the commanded orientation change that still maintains orientation intuitiveness, limited to a maximum of + / - 20% of the commanded motion in some cases). The system then modifies the commanded motion by adding an amount based on the residual orientation difference (such as the residual orientation difference scaled by a scale factor). This scale factor can be limited by a maximum scale factor.

[0125] According to another inventive aspect of using only orientations in alignment relationships, the orientations of various objects and links in the kinematic chain may be determined in various ways, as described below. Furthermore, even though position information of these objects and links may also be determined, in these aspects, only the orientation information is used to initially establish the alignment relationships required for control and then maintain and update the alignment relationships.

[0126] Figure 5H is a schematic diagram of teleoperated (e.g., telesurgery in the surgical example) system components and reference frames associated with those components. Where applicable, components similar to Figures 5A-5F As shown, camera tool 502 (also referred to as "camera 502") has an associated camera tool reference frame 503. Camera tool 502 has FOV 504 (2D or 3D) with an associated FOV reference frame 505. If the distal objective end 506 of camera tool 502 is not steerable relative to the body of the camera tool, the camera tool reference frame 503 and the FOV reference frame 505 can be combined.

[0127] The camera support structure 508 supports the camera tool 502 at the distal end. In some embodiments, the camera support structure has a single link, and in other embodiments, it has two or more links, and each pair of links is coupled by a joint. As shown, one link of the camera support structure 508 is identified as a camera support target link 510, and a dedicated spatial indicator target 512 is optionally fixed to the target link 510. The camera tool support structure reference system 513 is appropriately associated with the target link 510 or the target 512. The camera support structure 508 also includes a proximal base link 514 at the mechanical floor 516 (e.g., coupled to the floor 516a, coupled to the wall 516b, coupled to the ceiling 516c, or coupled to the structure itself at the mechanical floor, such as a table (fixed or movable) or a movable cart). As indicated by the directional arrow, the base link 514 is optionally movable relative to the floor, but is otherwise in a fixed relationship with the floor during initial alignment and operation. In some alternative embodiments, camera support structure 508 is omitted and camera 502 is supported by a person during the process. In such alternative embodiments, target 512 is optionally fixed to camera tool 502, and camera tool reference frame 503 and camera tool support structure reference frame 513 are combined.

[0128] The tool 520 has an associated tool reference frame 521. The tool 520 has a distal end effector 522 which, when applied, has an associated end effector reference frame 523 as it is movable relative to the body of the tool 520.

[0129] The tool support structure 524 supports the tool 520 at the distal end. In some embodiments, the tool support structure has a single link, and in other embodiments, it has two or more links, and each pair of links is coupled by a movable joint. As shown, one link of the tool support structure 524 is identified as a tool support target link 526, and a dedicated target 528 is optionally fixed to the target link 526. The tool support structure reference system 529 is appropriately associated with the target link 526 or the target 528. The tool support structure 524 also includes a proximal base link 530 at the mechanical floor 516 (e.g., coupled to the floor 516a, coupled to the wall 516b, coupled to the ceiling 516c, or coupled to the structure itself at the mechanical floor, such as a table (fixed or movable) or a cart). The base link 530 is optionally movable relative to the ground, but is otherwise fixed to the ground during alignment and operation.

[0130] As required by the system configuration, the spatial orientation determination unit 531 determines the camera support target link 510, the camera support spatial indicator target 512, the tool support spatial indicator target link 526, the camera support spatial indicator target 528 and the camera 502, or a subset of these, depending on the manner in which the orientation is determined. The orientation determination unit 531 optionally uses any of a variety of known means to determine the orientation, such as kinematics, electromagnetic positioning and other RF-based methods, ultrasonic or acoustic positioning, optical tracking based on dedicated targets or natural features, and fiber optic shape sensors. The details are described below. The orientation determination unit 531 is optionally centralized in a single location or distributed in two or more locations, can be optionally integrated into one or more teleoperated system units and support structures, and can be optionally worn by the user.

[0131] In some embodiments where only orientation information is used for an alignment relationship, the system is configured to sense only orientation information for such an alignment relationship. For example, a kinematic support structure is instrumented to sense only joint orientation. Omitting sensors simplifies the design and saves costs. Therefore, in some optional embodiments, two mechanically connected support structures are instrumented to sense only orientation and not position. For example, in a kinematic chain support structure for a tool manipulator or master device, a rotation sensor is used to sense the angle of the rotary joint, but a position sensor for sensing the position of the prismatic joint is omitted. Similarly, for ungrounded devices, only orientation and not position is optionally sensed, because only orientation can be used to establish and maintain an intuitive control relationship between a master and a slave device. In some other embodiments where only orientation information is used for an alignment relationship, position information is partially or completely sensed.

[0132] Also shown is an optional world reference frame 532 and a gravity vector (g), which can be used to define or determine one or more reference frames or sense orientation changes of one or more reference frames. The world reference frame 532 is optionally used for kinematic transformations between various system component reference frames.

[0133] Figure 5H Also shown is a display 534 on which an image from the camera tool 502 is displayed. A display reference frame 535 is associated with the display 534. The display 534 is supported with reference to the ground 516. When the distal end of the tool 520 is in the FOV 504, the corresponding image is displayed on the display 534 and viewed by a user 540 (e.g., a surgeon or skilled clinician or other personnel in a medical example).

[0134] A user 540 holds a master device 536, which has an associated master device reference frame 537. The master device 536 is optionally mechanically grounded or ungrounded, as indicated by the dashed line to ground 516. Whether grounded or ungrounded, the orientation of the master device 536 (the orientation of the master device reference frame 537) is sensed and determined by a master orientation determination unit 538, which selectively determines the orientation using any of a variety of known means, such as kinematics, optical tracking, or other wireless tracking (e.g., technology supplied by Leap Motion, Inc. of San Francisco, California, USA). For control purposes, the orientation of the master device reference frame 537 may be determined with reference to a reference frame on a kinematic support structure (if applicable) or with reference to a fixed reference frame (such as a master orientation unit reference frame 539 or a world reference frame 532).

[0135] Figure 5H Also shown is a user 540 oriented (standing, seated) to the ground 516. An optional user reference frame 542 is associated with the user 540. The user reference frame 542 is a body-centered reference frame defined with reference to a point on the user's body (e.g., the eyes, another location on the body, clothing or equipment worn by the user, etc.).

[0136] The centralized or distributed computer control system 550 receives information about the orientation of the various teleoperated system components and performs the initial alignment relationship required to establish effective teleoperated control and the rotational transformations necessary to maintain the alignment relationship as needed. When the alignment relationship is established and any other conditions required to enter the master / slave control mode are met, the computer control system 550 outputs commands to operate the teleoperated system in the master / slave control mode. Examples of optional requirements for entering the master / slave control mode are determining that the end effector is in the camera FOV, determining that the user is viewing the display (e.g., see U.S. Provisional Patent Application No. 62 / 467,506 (filed on March 6, 2017), which is incorporated by reference, and other safety-related conditions.

[0137] Typically, the computer control system 550 establishes the required initial orientation alignment relationship to enter the master / slave follow mode between the master device 536 and the surgical tool 520. A relative orientation transformation relationship is established between the end effector reference system and the FOV reference system and between the master device reference system and the display reference system. A direct transformation relationship between the master device reference system and the end effector reference system is not required. As described above, the transformation relationship used for initial alignment does not take into account the position information of the end effector or the master device. The transformation chain varies based on the system architecture and various reference systems of the components that can be used in the system architecture. In some embodiments, the required initial orientation alignment relationship is an orientation-only alignment relationship, and the associated transformation relationship is an orientation-only transformation relationship that only transforms the orientation. In some embodiments, the required initial orientation alignment relationship is a full orientation and partial position alignment relationship, and the associated transformation relationship is a full orientation and partial position transformation relationship, which only partially transforms the position, such as only transforming the position along one or two axes in three-dimensional space.

[0138] For example, refer to Figure 5H , the transformation relationship from the master device reference frame 537 to the end effector reference frame 523 may include a transformation from the master device reference frame 537 to the master orientation unit reference frame 539, to the tool support structure reference frame 529, to the tool reference frame 521, to the end effector reference frame 523. Optionally, a transformation to and from the world reference frame 532 is typically included between the reference frames associated with the control unit and the patient side unit.

[0139] In addition, the computer control system 550 establishes an initial orientation transformation relationship between the master device reference system 537 and the display reference system 535. In some embodiments, the initial orientation transformation relationship is an orientation-only transformation relationship. In some embodiments, the initial orientation transformation relationship is a full orientation and partial position transformation relationship.

[0140] When the initial master / slave relationship is established between the master device and the end effector, a reference frame transformation chain between the master device and the end effector is established for the master device for any master device position and orientation in the space where the user holds the master device. The user may choose to visually align the position and orientation of the master device and the displayed end effector image, but the user is not required to do so in order to establish control alignment. That is, the user may choose to hold the master device without visually aligning the position and orientation of the master device and the displayed end effector image. For example, the user may hold the master device at the level of the chest or abdomen outside the user's field of view, optionally with the forearms resting on the armrests for stability and fatigue reduction. As another example, when the initial control alignment is established, the user may be oriented at an oblique angle away from the display while holding the master device. For example, the user's shoulder may be rotated 45 degrees from the display so that the user can operate the master device with one hand and a manual tool (e.g., a laparoscopic tool in a surgical example) with the other hand.

[0141] In one aspect of establishing a master / slave relationship between a master device and an end effector, master / slave teleoperation is optionally permitted provided that the master device is within a certain orientation tolerance. The tolerance may be based on the orientation of the master device within the master orientation unit reference frame ( Figure 5H , element 539). Alternatively, the tolerance can be based on the orientation of the master device within the display reference frame ( Figure 5H , element 535), the orientation is actually based on a tolerance or on the orientation of a displayed image of the end effector. Alternatively, the tolerance can be based on the orientation of the master device in some other reference frame. The orientation tolerance can be applied to one rotation (e.g., roll), two rotations (e.g., pitch and yaw), or all three rotations in Cartesian space. And, the orientation tolerance can be different for each of these rotations. For example, if the master device contains the grip DOF in a plane, the roll tolerance relative to the corresponding grip DOF plane of the end effector can be smaller (e.g., ±10°) or larger (e.g., ±20°) than the pitch or yaw tolerance (e.g., ±15°) relative to the pitch and yaw of the end effector.

[0142] Fig.5I 5 is a diagrammatic view illustrating the requirement that the master device orientation be within a certain tolerance of the displayed end effector orientation in order to establish alignment for control. As depicted, an image 522a of the end effector 522 is shown on the display 534. Based on the kinematic relationship established between the FOV reference system 505 and the end effector reference system 523, the orientation 560 of the end effector image 522a in the display reference system 535 is determined. (For clarity, reference numeral 560 is shown twice in the figure—once relative to the display, and once relative to the display reference system.) Then, an alignment tolerance 562 is determined relative to the orientation 560. Fig.5I , the tolerance is illustrated by a cone having orientation 560 as its central axis. Other alignment tolerance shapes may alternatively be used, such as elliptical cones, pyramids, and similar shapes that may be defined with reference to an orientation axis.

[0143] Determine the kinematic relationship between the display reference frame and the host device reference frame. Then, as a first example, Fig.5I As shown, it is determined that the master device 536a has an orientation 564a. It is determined that the orientation 564a is within the alignment tolerance 562, and therefore the master / slave control between the master device 536a and the end effector 522 is allowed. As a second example, it is determined that the master device 536b has an orientation 564b. Orientation 564b is not within the alignment tolerance 562, and therefore the master / slave control between the master device 536b and the end effector 522 is not allowed before the orientation 564b is determined to be within the alignment tolerance 562. In some cases, as long as the master device orientation is within the alignment tolerance range, the control alignment will be automatically established, and optionally, visual, audio, tactile or similar indications are output to the user as a signal that the master / slave control relationship is valid. The ratchet function as described above can be used. In order to establish the required control relationship in other cases, in addition to the master orientation being within the alignment tolerance, the system must receive another event, such as a button press, a verbal command, or a similar input requesting the establishment of a control relationship. This method for establishing a tolerance for the orientation of the master device in order to initiate master / slave control is applicable when the tolerance is based on other reference frames.

[0144] In some cases where the master control is at the remote end of the manipulator, the control system 550 may optionally command the manipulator to reference the master orientation unit reference frame ( Figure 5H , element 539) or the display reference frame ( Figure 5H , element 535) or some other reference frame to orient the master device's orientation alignment axis. For example, the control system 550 can command the arm to place the master device in an aligned orientation relative to the displayed end effector image, and it commands the arm to place the master device in a defined default position relative to the display, or relative to the display's current position, rather than a position corresponding to the displayed image of the end effector.

[0145] If two master devices are used to control a single object, the perceived orientation and direction of the movement correlation between the master device and the object can be the perceived orientation and direction of the movement correlation between the master device and the object that act together. An example can be two master devices as handles with a straight connecting axis between them. When the master devices move together to change the position of the connecting axis, the position of the object moves accordingly (e.g., the camera FOV moves up and down, moves left and right, moves inside and outside). When the master devices move together to change the orientation of the connecting axis, the orientation of the object changes accordingly (e.g., the camera FOV tilts up and down, translates left and right, and rolls clockwise-counterclockwise). Here again, only the orientation information needs to be used, and the position information is not required. For example, the master devices can be close to each other or far apart on the connecting axis, and as the object is controlled, the spacing on the connecting axis may change (e.g., farther apart provides more refined orientation control; closer apart provides an increased range of motion).

[0146] like Figure 5J As shown, for example, a connection axis X is defined between the left and right main devices. ML-MR . You can also define the normal axis Z ML-MR To control. ML-MR When moving left, the camera will translate or rotate to move accordingly, optionally giving the user the perception of moving the scene to the right (FOV moving right as shown) or moving the camera to the left (FOV moving left). The camera may translate or rotate as shown. Insertion and exit are performed along axis Z. ML-MR It can also be controlled along another mutually orthogonal axis Y ML-MR Similarly, the FOV position or orientation can be controlled by rolling around the connecting axis and its Cartesian orthogonal line (for example, the FOV can be controlled by rolling around the X axis and its Cartesian orthogonal line). ML-MR Roll and tilt, FOV is rotated by Y ML-MR Roll and pan, and FOV is rotated by rotating around Z ML-MR As described herein, methods similar to those used to establish an initial alignment relationship may be used to maintain the alignment relationship during teleoperation.

[0147] Determine spatial relationships

[0148] As described above, the operating environment of a teleoperated system may include two or more manipulators for various tool and camera combinations. For example, the patient-side environment of a teleoperated surgical system may include two or more manipulators for various surgical tools and endoscopic camera combinations. Moreover, one or more of these manipulators may not have a predetermined fixed spatial relationship relative to other manipulators. Similarly, there may not be a predetermined fixed spatial relationship between one or more host devices and a display screen. In this case, it is impossible to establish and maintain the control relationship necessary for teleoperation based solely on sensing the angular relationship between various kinematic pairs (e.g., by using a joint angle or similar sensor) and the kinematic transformation of each individual unit. Determine the spatial relationship between units in order to establish and maintain effective intuitive control.

[0149] Fig. 6A is a schematic diagram of a teleoperated system (specifically, a telesurgery system) incorporating aspects of the invention for determining spatial relationships. Fig. 6A Several aspects of the invention are described herein, and various optional teleoperated surgical system configurations are further described below. Not all depicted and described aspects are required in a single embodiment. Where applicable, Fig. 6A The object depicted in Figures 5A-5J Objects depicted in (e.g., tools, endoscopic cameras, support structures, control systems, spatial sensors, movable units, etc.).

[0150] As shown, there are two teleoperated surgical tools 602a, 602b, each of which has a corresponding end effector 603a, 603b. Each tool 602a, 602b is actuated by a corresponding manipulator 604a, 604b, each of which is mounted on a corresponding base 606a, 606b to constitute a corresponding patient-side unit 608a, 608b. Similarly, an endoscopic camera 610 is shown, and it has a FOV 612. The endoscopic camera 610 is actuated by a corresponding manipulator 614 mounted on a base 616, and together they constitute a patient-side unit 618. As shown, the patient-side units 608a, 608b and 618 are movable relative to each other—none of them has a common mechanical support structure to fully constrain their relative spatial relationship. Therefore, the patient-side units 608a, 608b and 618 are generally as described above.

[0151] Each surgical tool 602a, 602b enters the body via a corresponding optional cannula 620a, 620b. Likewise, the endoscopic camera 610 enters the body through an optional cannula 622. The tool end effectors 603a, 603b are located within the FOV 612. Three optional vibration sensing / injection units 624a, 624b, and 624c are shown, each attached to a corresponding one of the cannula 620a, 620b, and 622. Alternatively, the vibration sensing / injection units 624a, 624b, and 624c can be coupled to any location on the patient-side unit or other teleoperated surgical system units.

[0152] The captured image data 625 travels from the endoscope camera 610 to an optional machine vision processor 626. The image data 625 also travels to a display image processing unit 628, which in turn processes the captured image data and outputs display image data 630 for display. The machine vision processor 626 outputs machine vision spatial data 632 for use as described below.

[0153] Fig. 6A Also shown are master devices 634a, 634b operated by a user. Each master device 634a, 634b is supported by a corresponding mechanical support structure 636a, 636b. As shown, the support structures 636a, 636b are mechanically coupled in a fixed spatial relationship and are each mounted to a removable common base 638. Optionally, each support structure 636a, 636b is mounted to a separate removable base. Also shown is an optional grounded or ungrounded master control device configuration 640, in which the posture of the master device 641 is sensed by the master device spatial sensing unit 642 (see also Figure 5H , orientation determination unit 538; Figure 5B and Figure 5C , control input receiver 160c) senses. Spatial sensing unit 642 may be fixed in space, or it may optionally be mounted to a removable base 644. Display 646 may optionally be mounted at a fixed location, mounted on a removable support structure 648, or worn by a user. Support structure 648 may have a base that is fixed in space, or the base may be mounted to a corresponding removable mechanical base 650, or the base may be mounted to a base common to a base corresponding to a master control device, such as base 638 or base 644. Thus, the components associated with master control devices 634a, 634b and display 646 are generally as described above.

[0154] Fig. 6A Also shown is a control system 652 for a teleoperated surgical system (see also Figure 5H , control system 550; Figure 4, computer 113). The control system 652 executes programmed instructions to implement alignment and other system control functions described herein. The control system 652 communicates signals with the patient side units 608a, 608b, and 618. It also communicates signals with the master devices 634a, 634b. Through this signal communication, the control system 652 receives spatial information 654a, 654b associated with the end effectors 603a, 603b, spatial information 654c associated with the FOV 612, and spatial information 654d, 654e associated with the master devices 634a, 634b. Optionally, if the control system 652 is not mechanically coupled to the master device, the control system receives spatial information 654f associated with the display 646.

[0155] Optional spatial indicators 656a-656f are mounted to bases 606a, 606b, 616, 638, 644, and 650. As shown, spatial detection unit 658 (centralized or distributed sensing) is associated with spatial indicators 656a-656c, and spatial detection unit 660 (centralized or distributed sensing) is associated with spatial indicators 656d-656f. However, optionally, a single spatial detector unit may be associated with all of the spatial indicators in the teleoperated surgical system. Figure 5H , targets 512, 528 and orientation determination unit 531 are examples of spatial indicators and detectors.

[0156] Thus, inventive aspects determine spatial relationships between teleoperated surgical system units that are not in a permanent fixed mechanical relationship. The relationships are determined in order to establish, achieve, and maintain intuitive motion based on input from a user master control device, but with acceptable or no interference with the operating room environment. For example, when there is no fixed mechanical relationship between an end effector and a manipulator of an endoscopic camera, or directly between the end effector and the endoscopic camera itself, a spatial relationship between them is determined. The determined relationship is then used to establish the transformations necessary for the teleoperated control relationship. At a minimum, only the orientation is determined. However, in some embodiments, some position information may be determined for one or both axes, or all posture information may be determined.

[0157] In some aspects, the spatial determination method uses external hardware, such as spatial indicators 656a-656f and spatial detection units 658, 660. However, the size and location of this external hardware is set so that it does not interfere with the operating room environment. In other aspects, the spatial determination method does not require additional hardware and is included in the existing hardware of the teleoperated surgical system. For example, an additional data processing unit such as a video data processor or a machine vision processor can be added inside the existing unit.

[0158] Various means may be used to determine the spatial relationship between teleoperated surgical system units that are movable relative to each other by positioning the units to a common single reference frame as desired. The single reference frame may be a world reference frame defined separately from the teleoperated surgical system (e.g., see Figure 5H , reference frame 532). Alternatively, a single reference frame may be associated with a device in the teleoperated surgical system, such as the base of a teleoperated manipulator unit.

[0159] In the following disclosure, reference is made to a "reference base" of the unit, which in some embodiments is the actual physical base of the unit resting on the floor or fixed to another support structure in the world reference frame. However, one skilled in the art will appreciate that a "reference base" may be arbitrarily defined at any point on the patient-side unit that remains stationary in the world reference frame during teleoperated surgery. Since each "reference base" is movable, the relationship of one or more reference bases is determined once the reference base is in a posture that will remain stationary during teleoperated surgery. Fig. 6A , bases 606a, 606b, 616, 638, 644, and 650 are examples of such removable reference bases and will be used as an illustration of the reference base. The principles of operation are illustrated with respect to patient side units 608a, 608b, and 618, and these principles are applicable to user control units and can be used for specific system configurations. The features and functions associated with the spatial indicators 656d-656f and the spatial detection unit 660 are similar to the features and functions of the spatial indicators 656a-656c and the spatial detection unit 658.

[0160] One method of determining spatial relationships is to establish a temporary local mechanical relationship between a pair of units (e.g., two patient-side manipulator units) by attaching a temporary kinematically instrumented direct mechanical coupling (e.g., a coupled linkage with a joint angle sensor) between the units. The instrumented coupling allows the kinematic relationship to be determined once the units are placed for telesurgery and repositioned during telesurgery. However, in some cases, such mechanical positioning methods are not suitable for use in the operating room. For example, the equipment used for these methods may interfere with sterile drapes and other operating room equipment. Alternatively, sterile drapes and other operating room equipment may interfere with the equipment used for these methods. In addition, the equipment used for these methods may consume too much space in the patient-side environment, may be expensive, and may be difficult to operate due to the need for frequent calibration and other maintenance.

[0161] Another spatial relationship determination method is to adapt the indoor locator system method for use with a teleoperated surgical system in an operating room environment. (In this context, the term "locator system" can be configured to provide some or all of the parameters of orientation, some or all of the parameters of position, or some or all of the parameters of orientation and position.) These locator system methods may find and track an actively transmitted object, or they may find and track the surrounding environment of an object.

[0162] One aspect of the locator system approach is to place one or more sensors on each unit to detect one or more synthetic or natural features on one or more other units. Synthetic features can actively transmit energy ("beacons"; for example, infrared or visible light, RFID, ultrasound) or can be passive (for example, dedicated spatial indicator targets). One or more sensors are used to determine the spatial relationship between one or more pairs of units, which are then used for the described teleoperated control. However, in some cases, the line of sight method is not suitable for the operating room because the line of sight may be blocked by operating room equipment, sterile curtains, etc. And, if three or more units are involved, multiple lines of sight must be clear between multiple pairs of units. But in some cases, the line of sight is almost always clear, such as between the unit and the operating table (for example, see U.S. Patent Application No. 15 / 522,180) (filed on April 26, 2017; National Phase No. PCT / US2015 / 057664 of the International Application) (disclosure "System and Method for Registering to a Surgical Table", which is incorporated herein by reference).

[0163] Another aspect of the locator system approach is to place one or more sensors in respective fixed locations in the operating room environment that allow line of sight to the unit (e.g., higher up on a wall, or on a ceiling) and track synthetic or natural features on the various movable units. As described above, the synthetic features can be beacons or passive. The spatial indicators 656a-656c also illustrate natural physical features that can be sensed. The advantage of using two or more sensors is that multiple possible lines of sight ensure that a unit is always detected, and multiple lines of sight between two or more sensors and a single unit can provide redundancy and can optimize the determination of the unit's posture.

[0164] As an exemplary embodiment of this fixed sensor approach, a single optical sensor is placed in a fixed position in an operating room environment, and the single optical sensor detects passive dedicated synthetic optical targets or natural features on one or more patient-side units. Fig. 6A The spatial indicators 656a-656c in FIG. 1 show such objects or natural features (see also Figure 5H, targets 512 and 528). Calibration establishes the spatial relationship between the coordinate system of each target and its corresponding unit base reference system. For example, the spatial detection unit 658 acts as an optical tracker and determines the spatial relationship between the spatial indicators 656a and 656b. Then, a forward kinematic transformation is used to determine the posture of each end effector 603a, 603b relative to its corresponding target. Since all target reference frames can be represented as a common base reference frame or other specified common reference frame in a single optical tracker reference frame, the relative transformation between the end effector reference frames can be calculated by using a combination of measured optical tracker data and the forward kinematics of the patient-side units 608a, 608b.

[0165] As another exemplary embodiment of the fixed sensor method, two or more optical sensors are placed in a fixed position in the operating room environment, and the optical sensors detect passive dedicated synthetic optical targets or natural features on one or more patient side units. Control is then established as described above.

[0166] As another exemplary embodiment of the fixed sensor method, one or more RFID or ultrasonic beacons are placed on each unit, and one or more sensors are fixed in the operating room environment to detect the beacons. One or more postures of one or more units are determined from the sensed beacons, and control is then established as described above.

[0167] As another exemplary embodiment of the fixed sensor approach, a combination of synthetic features and / or natural features is sensed. This combined sensor type approach provides robustness and reliability compared to a single sensor type approach. For example, a clear target pattern on one patient side unit and a natural feature on a second patient side unit are sensed, or a combination of a clear target pattern and a natural feature are sensed on each patient side unit.

[0168] The second aspect of the locator system approach is to place one or more sensors on each mobile unit, and track one or more synthetic or natural features fixed in the operating room environment (e.g., high on a wall, or on the ceiling) at a location that is easily sensed by the unit. As with the fixed sensor aspect, in this fixed feature aspect, the synthetic features may actively transmit energy (a "beacon"; e.g., light, RF, ultrasound) or may be passive (e.g., a dedicated spatial indicator target). In this fixed feature aspect, Fig. 6A Such sensors are shown as space indicators 656a-656c in FIG. Figure 5H , elements 512 and 528), and the spatial detection unit 658 shows one or more fixed features in the operating room environment. The control system receives spatial information from one or more units and then establishes control as described above.

[0169] In a manner similar to the fixed sensor and fixed feature locator system approach, another alternative is to use a simultaneous localization and mapping (SLAM) technology customized for use with a teleoperated surgical system in an operating room environment. There are various SLAM methods. For example, see U.S. Patent No. US 9,329,598 B2 (filed on April 13, 2015) (disclosing "Simultaneous Localization and Mapping for a Mobile Robot") and US 7,689,321 B2 (filed on February 10, 2010) (disclosing "Robust Sensor Fusion for Mapping and Localization in aSimultaneous Localization and Mapping (SLAM) system"), which are incorporated herein by reference. Detection and tracking of mobile objects (DATMO) technology can be used in conjunction with SLAM. For example, see U.S. Patent No. US9,727,786B2 (filed on November 14, 2014) (disclosing "Visual Object Tracking System with Model Validation and Management"), which is incorporated herein by reference. Multiple sensors ensure adequate coverage and overlap of the operating room configuration, taking into account other operating room equipment (operating table, anesthesia station, etc.) and the need to move the patient-side unit relative to such equipment. The SLAM and / or DATMO sensors can be fixed in the operating room environment, mounted on a movable unit, or both. The base reference frame orientation required for control is determined, and control is then established as described above.

[0170] As an alternative to the modified indoor locator system approach, machine vision methods can be used to track the tool directly in the stereo images captured by the endoscope camera. The tracking information is used to determine the pose of the end effector(s) directly in the reference frame associated with the camera field of view. Fig. 6A , the machine vision processor 626 transmits spatial data 632 about the end effector to the control system 652. The tracked relationships are used to determine the relative pose of the stationary manipulator base during telesurgery.

[0171] In one embodiment, machine vision is continuously used to track the pose of the end effectors. In an alternative embodiment, once the relationship between the manipulator bases is determined based on machine vision and kinematic information, the alignment between the end effectors can be determined based solely on the kinematic information and no further machine vision tracking is required. Fig. 6A, the control system 652 receives kinematic data such as spatial information 654a-654c from the patient side units 608a, 608b and 618. This use of kinematic data greatly reduces the computational load of continuous machine vision tracking. In yet another alternative embodiment, machine vision tracking is used to update posture information at regular intervals (e.g., every 100ms, every 1s, etc.), and this periodic update embodiment is still much less computationally intensive than continuous machine vision tracking.

[0172] As another alternative, the spatial determination system is based on fiber optic shape sensors integrated with cables associated with each unit. The cable interconnection between units, or between two units and a common node (such as a control system unit) includes a fiber optic shape sensor (e.g., a sensor incorporating fiber Bragg grating technology). Shape sensor technology is used to determine the spatial relationship between interconnected units. Cables that transmit control or video data can be modified to include fiber optic shape sensors.

[0173] The above aspects can be used to determine all posture information (all orientation and position information), or they can be used to determine less than all posture information (e.g., in three-dimensional Cartesian space, only partial orientation information around one or two Cartesian axes and / or only partial position information along one or two Cartesian axes). As discussed above, in various embodiments including many manipulator assemblies for the patient side, only the relative orientation between the camera and one or more tools is required for effective control alignment of the tool associated with the camera. And therefore, only the relative orientation between these objects or the individual links in the kinematic chain supporting these objects is required. Similarly, in some embodiments, for user control, only the relative orientation between the displayed image of the end effector and the main device is required. Therefore, these aspects can be simplified or more robust because they only require estimating half the number of variables (i.e., orientation, not position). If the need to determine and track all posture information is eliminated and only the orientation information is determined, additional spatial determination methods can be used.

[0174] In an alternative orientation determination method, the spatial indicators 656a-656f illustrate a combination of a 3-axis accelerometer and a 3-axis magnetometer. It will be recalled that the spatial indicators may be located at any link in the movable unit, or on the object itself, such as an endoscope camera.

[0175] like Figure 6BAs shown, for example, spatial indicators 656a, 656b are a combination of a 3-axis accelerometer and a 3-axis magnetometer mounted to respective bases 606a, 606b, respectively. The combination of the two spatial indicators 656a, 656b each determines the gravity vector g and the orientation relative to the magnetic north pole N, and therefore they are constant for both units. As shown, for the local magnetic field 670, b is the local magnetic field vector, g is the gravity vector, and the orientation is the magnetic field projected onto plane 672 perpendicular to g to indicate the magnetic north pole.

[0176] n=g×b

[0177]

[0178] Based on the magnetic north pole orientation, gravity vector orientation, and kinematic information, the spatial orientation of the respective bases 606a, 606b, and therefore the relative spatial orientation of the respective end effectors 603a, 603, is determined for an initial alignment relationship. Likewise for the FOV 612 and user control unit. Then, once the initial control alignment is established, the kinematic information can be used to provide the overall pose information for the master / slave teleoperation.

[0179] Magnetic field interference may be present in the operating room environment due to local magnetic materials, electric motors, electromagnetic field generators, nearby ferrous materials, etc. Typically, the patient-side unit should be placed so that the accelerometer / magnetometer combination is away from these things. However, if the North Pole orientation error causes the North Pole orientation of two or more patient-side units to differ greatly enough to cause the initial alignment between the two units to affect intuitive master / slave control, alignment correction must be performed. For example, the same movement of the left and right master input devices may result in different movements of their corresponding end effectors (e.g., when the associated left master device moves directly to the left, the left end effector moves directly to the left (as viewed in the display); however, when the associated right master device moves directly to the left in the same direction, the right end effector moves up and to the left (as viewed in the display)).

[0180] Therefore, an alignment adjustment function is provided, and the user can adjust and fine-tune the relationship of each tool image relative to the perceived orientation of the corresponding master device. Since the orientation misalignment is only caused by different determined North Pole orientations, the adjustment is a one-time DOF adjustment for each tool relative to the FOV reference frame. Figure 6C As shown, for example, a first position is determined for a first patient side unit base reference system 674a. and determining a second orientation for the second patient side unit base reference system 674b Because the orientations are different, the user can adjust the angle θ shown therebetween to obtain an alignment for intuitive control. In this way, the same movement of the primary input device (e.g., directly to the left as perceived by the user) will result in the same movement of the corresponding tool (e.g., directly to the left as viewed in the display). This one-time DOF adjustment can be made more easily by the user than adjusting the entire 3D rotation to make the correction, and it is shown that this one-time DOF adjustment method can be applied to any spatial determination of orientation where one rotational alignment produces a non-intuitive control relationship. This may occur, for example, if the support structure base is moved or the person holding the endoscopic camera moves.

[0181] Another alternative method for making rotational corrections is to determine the misalignment using external tracking or machine vision methods as described above, and use the determined misalignment to make corrections. As described above, these correction methods can be performed at intervals to reduce the computational load. In addition, the combination of the accelerometer / magnetometer method and the second spatial determination method provides a more robust and computationally less demanding solution because the second method is simplified due to its determination of orientation in a single DOF. For example, the accelerometer / magnetometer method can be used to provide an initial estimate of the orientation of the tool end effector in the field of view of an endoscope camera, and then by using this initial estimate, the machine vision tracking task can be expedited or made computationally less demanding.

[0182] As another alternative spatial determination method, a 3-axis gyroscope is coupled to each unit at a fixed position. Each gyroscope is calibrated to a known orientation, and then the gyroscope is used to determine the subsequent orientation when the unit moves. Calibration may be completed in various ways, such as a known mechanical alignment (e.g., a fixture on a wall or a table such as an operating table), or by using an accelerometer / magnetometer method as described above. Although gyroscopes may tend to drift over a long period of time, the gyroscope method can be used in conjunction with another spatial determination method to provide a redundant incremental check of the base orientation during use (e.g., during surgery in the surgical example). For example, accelerometers, magnetometers, and gyroscope measurements can be used together to determine the relative orientation and transformation between the base links. As another example, gyroscope measurements can be combined with other spatial determination methods to add robustness and redundancy, and simplify or accelerate estimation. In addition, gyroscopes can be used to detect transient disturbances in the magnetic field that cause deviations in azimuth measurements that are inconsistent with gyroscope data. In this regard, gyroscope data can be optionally weighted more heavily until the magnetometer signal stabilizes. Alternatively, detection of transient magnetic field disturbances may be used to signal a problem or malfunction to a user.As described above, a single DOF adjustment that allows a user to fine tune the perceived alignment in a reduced parameter space may be incorporated into implementations that incorporate gyroscope information.

[0183] As another alternative spatial determination method, only acceleration sensors are used, and vibration sensors (e.g., see FIG. 6 , sensors 624a-624c including vibration sensing / injection units) are used to determine the relative spatial relationship between units. In one embodiment, ambient common mode vibrations (e.g., vibrations from the floor) are sensed at each patient-side unit or sleeve. Assuming that each unit senses the same ambient vibrations, the common mode signal is identified by an accelerometer associated with each unit and fixed in a known orientation for each unit. The gravity vector and the sensed horizontal direction of the vibration at each unit are used to determine the relative orientation between the units. In an alternative embodiment, common mode vibrations are injected. For example, the sleeve vibrates so that its remote center of motion at the body wall vibrates in a known direction. The unit senses the injected vibration direction and determines the relative spatial orientation.

[0184] In yet another embodiment using vibration information, one or more beacons are placed (e.g., on the operating room floor) to inject periodic and time-synchronized common-mode vibrations so that each unit can sense the vibrations. An accelerometer or matched resonator on the unit senses the vibrations. Time-of-flight measurements are used to establish the distance to the one or more vibration beacons, and triangulation is used to determine the relative spatial orientation of the units. For example, assuming a speed of sound in concrete of 3500 m / s, a resolution of 1 cm requires a time resolution of approximately 3 μs. This approach advantageously eliminates the need for a clear line of sight between the beacon and the sensor. In all methods involving vibration injection, the vibration frequency can be selected outside the audible range.

[0185] In another embodiment, the orientation degree of freedom is optionally measured by using two or more different methods. For example, accelerometers can be used to determine the orientation on two axes, and machine vision is used to determine the azimuth orientation on the remaining axis. The determinations are fused, and the fused result provides a complete 3-axis solution.

[0186] Thus, the spatial determination system is used to determine the relative orientation and required transformations between the plurality of teleoperated system units in order to establish and maintain a required user intuitive perception of control alignment between the manual master control device and the respective tool. Advantageously, the master / slave control required alignment is established and maintained using only the orientation information or the orientation information combined with less than complete position information.

[0187] Further embodiments

[0188] Many embodiments have been described in terms of teleoperated surgical systems, but it should be understood that the inventive aspects are not limited to teleoperated surgical systems. Embodiments in various other teleoperated systems are contemplated. For example, aspects may be implemented in teleoperated systems related to military applications (e.g., bomb disposal, reconnaissance, operations under enemy fire), research applications (e.g., marine submersibles, earth orbiting satellites, and manned stations), material handling applications (e.g., nuclear "hot cells" or other hazardous materials), emergency response (e.g., search and rescue, firefighting, nuclear reactor investigations), unmanned ground vehicles (e.g., agriculture, manufacturing, mining, construction), etc.

Claims

1. A remote control system comprising: monitor; Primary input device; as well as A control system comprising one or more processors and a memory, the memory comprising programming instructions adapted to cause the one or more processors to perform operations comprising: determining an orientation of an end effector reference frame relative to a field of view reference frame, the end effector reference frame being movable relative to the field of view reference frame, the end effector reference frame being defined for an end effector of a tool and the field of view reference frame being defined for a field of view of an imaging device, determining an orientation of an input device reference frame relative to a display reference frame, the input device reference frame being defined for the primary input device and the display reference frame being defined for an image displayed by the display, establishing a first alignment relationship, the first alignment relationship comprising an alignment relationship between an end effector and a field of view or an alignment relationship between an input device and a display, wherein the alignment relationship between the end effector and the field of view is between the end effector reference system and the field of view reference system and is independent of a positional relationship between the end effector reference system and the field of view reference system, and wherein the alignment relationship between the input device and the display is between the input device reference system and the display reference system and is independent of a positional relationship between the input device reference system and the display reference system, and Based on the first alignment relationship, a posture change of the end effector is commanded in response to a posture change of the master input device.

2. The teleoperated system of claim 1, wherein the first alignment relationship comprises an alignment relationship of the end effector with a field of view.

3. The remote controlled system of claim 2, wherein: The operations further include: establishing a second alignment relationship, the second alignment relationship comprising an alignment relationship between the input device and the display; and Commanding a pose change of the end effector is also based on the second alignment relationship.

4. The teleoperated system of claim 3, wherein the orientation of the input device reference system relative to the display system is a first relative orientation, and the orientation of the end effector reference system relative to the field of view reference system is a second relative orientation, wherein the first relative orientation differs from the second relative orientation by a difference, and wherein the operations further comprise: The second alignment relationship is updated multiple times to gradually reduce the difference.

5. The remotely operated system of claim 1, wherein the first alignment relationship comprises an alignment relationship of the input device with a display.

6. The remote control system according to claim 1, wherein the pose of the end effector is relative to the field of view reference system; wherein said posture of said primary input device is relative to said display reference frame; and The change of the posture of the end effector includes: The orientation of the end effector relative to the field of view reference system changes, and the orientation change corresponds to the orientation change of the primary input device relative to the display reference system.

7. The remote controlled system of claim 1, wherein the operation further comprises: A remote-controlled master-slave control relationship is established based on the first alignment relationship.

8. The remote control system according to claim 1, Wherein determining the orientation of the end effector reference system relative to the field of view reference system comprises: determining the complete orientation of the field of view reference system, and determining an integral orientation of the end effector reference system; and Wherein determining the orientation of the input device reference system relative to the display reference system comprises: determining the full orientation of the display reference system, and The overall orientation of the input device reference frame is determined.

9. The teleoperated system of claim 1, wherein the programming instructions are not adapted to cause the one or more processors to: determining a complete position of at least one reference frame selected from the group consisting of: the field of view reference frame, the end effector reference frame, the display reference frame, and the input device reference frame; or determining the complete position of the end effector reference system relative to the field of view reference system; or The complete position of the input device reference system relative to the display reference system is determined.

10. The remote controlled system of claim 1, wherein the operation further comprises: determining a less than complete position of the end effector reference system relative to the field of view reference system; or A less-than-complete position of the input device reference system relative to the display reference system is determined.

Images