Systems and methods for determining registration and control of a robotic manipulator or associated tool

Abstract

A robotic system includes a first manipulator assembly and a second manipulator assembly that are movable independently in an operating environment and have bases. A processing unit is configured to receive first sensor data from a plurality of first sensors disposed on the first manipulator assembly, where the first sensor data provides spatial information about the operating environment external to the first manipulator assembly. A first spatial relationship of the second manipulator assembly relative to the first manipulator assembly is determined using data including the first sensor data. A first alignment relationship between the first manipulator assembly and the second manipulator assembly is established based on the first spatial relationship. Based on the first alignment relationship, motion of the second manipulator assembly is commanded in response to commands from a first input device that is operable by an operator.

Description

[0001] Cross-reference of related applications

[0002] This application claims the benefit of U.S. Provisional Application 62 / 993,960, filed March 24, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to systems and methods for performing robot procedures, and more particularly to systems and methods for determining registration of robot manipulators for controlling the movement of robot manipulators and / or associated tools. Background Technology

[0004] Robotic manipulator components include one or more robotic manipulators that can be operated to control the movement of tools within a workspace. For example, such robotic manipulators can be used to perform non-medical and medical procedures. As a specific example, remotely operated surgical manipulators can be used to perform minimally invasive medical techniques.

[0005] In medical technology, the aim is to improve patient outcomes and facilitate clinicians' diagnostic or treatment procedures. For example, for medical procedures involving access to a patient's anatomy, minimally invasive techniques can be performed through natural openings in the patient's anatomy or through one or more incisions. Through these natural openings or incisions, clinicians can insert medical instruments to reach target tissue locations. Minimally invasive medical instruments include tools such as therapeutic, diagnostic, and surgical instruments. Minimally invasive medical instruments may also include imaging tools, such as endoscopic tools, which provide the user with a field of view within the patient's anatomy. Robotic medical systems allow users to control medical devices via manipulators. Manipulators may include two or more links connected together by one or more joints. Joints may include actively controlled joints whose position or movement is actively driven by actuators. Joints may also include passive joints whose position or movement is not actively driven by actuators.

[0006] Robotic manipulators can be remotely operated or otherwise computer-assisted. To execute and view robotic procedures at a surgical site (e.g., a surgical site within a patient's body), two or more manipulators can be used to hold and manipulate tools, including, for example, surgical instruments and imaging tools. The operator can use a master control unit selectively associated with the tools and the manipulators holding them. In such robotic systems, control of the tools in response to operator manipulation of the master control unit can have multiple definable reference frames and corresponding reference frame transformations to map coordinates in one reference frame to corresponding coordinates in another. However, precise control of the tools can be difficult to achieve when one or more of the position and / or orientation of the reference frames and / or reference frame transformations are unknown. In such cases, the success rate and accuracy of the procedures may be reduced. In medical robotic contexts, greater convenience and efficiency can be achieved through more precise control of the tools.

[0007] In remotely operated medical systems comprising multiple manipulator components, it is desirable to know the position and / or orientation of the manipulator components relative to each other. For example, this information can be used to enhance operation or avoid collisions. In some remotely operated medical systems, the manipulator components share known references, such as a common mounting base, making it possible to deduce the relative positions of the manipulator components (and their end effectors) using the kinematic relationships between the manipulator components and their known references.

[0008] In some cases, remotely operated medical systems include independent manipulator components that do not share a known reference (e.g., manipulator components mounted on separate mobile carts or at different, unknown locations to a common table). In such systems, one or more parameters relating to the positioning or orientation of the respective bases of the manipulator components relative to each other are unknown or may change between or during procedures (e.g., if the mounting base locations are moved). Therefore, while the kinematics of each manipulator may provide information about its individual position or orientation relative to its own base, such individual manipulator kinematics may not provide information about the orientation and position of the manipulator components relative to each other. Therefore, it would be advantageous to provide improved methods and systems for registering independent manipulator components in robotic systems such as remotely operated medical systems. Summary of the Invention

[0009] Embodiments of the present invention are described in the appended claims.

[0010] Consistent with some embodiments, the robot system includes a first manipulator assembly and a second manipulator assembly in an operating environment, and each has a separate movable base. A processing unit is configured to receive first sensor data from a plurality of first sensors disposed on the first manipulator assembly, wherein the first sensor data provides spatial information about the operating environment outside the first manipulator assembly. Data including the first sensor data is used to determine a first spatial relationship between the second manipulator assembly and the first manipulator assembly. A first alignment relationship is established between the first and second manipulator assemblies based on the first spatial relationship. Based on the first alignment relationship, movement of the second manipulator assembly is commanded in response to a command from a first input device operable by an operator.

[0011] Consistent with other embodiments, the method of operating a robot system includes receiving first sensor data from a plurality of first sensors disposed on a first manipulator assembly in an operating environment. The first sensor data provides spatial information about the operating environment outside the first manipulator assembly. The first manipulator assembly includes a plurality of first links physically coupled to a first base. The operating environment includes a second manipulator assembly including a plurality of second links physically coupled to a second base, the second base being movable independently relative to the first base. The method further includes determining a first spatial relationship between the first manipulator assembly and the second manipulator assembly using data including the first sensor data, establishing a first alignment relationship between the first manipulator assembly and the second manipulator assembly based on the first spatial relationship, and, based on the first alignment relationship, commanding movement of the second manipulator assembly in response to a command from a first input device operated by an operator.

[0012] Other embodiments include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

[0013] It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the disclosure without limiting its scope. In this regard, additional aspects, features, and advantages of the disclosure will become apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0014] When with attachment Figure 1 When reading this disclosure, the various aspects are best understood from the following detailed description. It should be emphasized that, in accordance with industry standard practice, the various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or decreased for clarity of discussion. Furthermore, reference numerals and / or letters may be repeated in various examples within this disclosure. This repetition is for simplicity and clarity and does not in itself determine the relationship between the various embodiments and / or configurations discussed.

[0015] Figure 1 This is a schematic diagram of a robotic medical system according to an embodiment of the present disclosure.

[0016] Figure 2 This is a perspective view of a manipulator assembly with an external environment detection sensor system according to an embodiment of the present disclosure.

[0017] Figure 3 A flowchart illustrating an embodiment of the present disclosure is shown, which provides a method for performing a registration process for manipulator components that are individually movable from each other.

[0018] Figure 4A This is a perspective view of two manipulator components according to an embodiment of the present disclosure; Figure 4B According to embodiments of this disclosure Figure 4A A top view of the two manipulator components; Figure 4C According to another embodiment of this disclosure Figure 4A A top view of the two manipulator components; Figure 4D This is according to yet another embodiment of the present disclosure. Figure 4A A top view of the two manipulator components; Figure 4E This is according to yet another embodiment of the present disclosure. Figure 4A A schematic diagram of the two manipulator components.

[0019] Figure 5A This is a perspective view of three manipulator components according to one embodiment of the present disclosure; Figure 5B According to one embodiment of this disclosure Figure 5A A top view of the three manipulator components; Figure 5C According to another embodiment of this disclosure Figure 5A A top view of the three manipulator components; Figure 5D This is according to yet another embodiment of the present disclosure. Figure 5A A top view of the three manipulator components; Figure 5E This is according to yet another embodiment of the present disclosure. Figure 5A A top view of the three manipulator components.

[0020] Figure 6 A flowchart is shown according to another embodiment of the present disclosure, which provides a method for performing a registration process for manipulator components that can move independently of each other.

[0021] Embodiments of this disclosure and their advantages will be best understood by referring to the following detailed description. It should be understood that similar reference numerals are used to identify similar elements shown in one or more of the accompanying drawings, wherein the illustrations are for illustrative purposes only and are not intended to limit the scope of these embodiments. Detailed Implementation

[0022] To facilitate an understanding of the principles of this disclosure, reference will now be made to the embodiments illustrated in the accompanying drawings, and these embodiments will be described using specific language. However, it should be understood that this disclosure is not intended to limit its scope. In the following detailed description of various aspects of this disclosure, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that the embodiments of this disclosure can be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail in order to avoid unnecessarily obscuring various aspects of the embodiments of this disclosure.

[0023] Any changes and further modifications to the described apparatus, tools, and methods, as well as any further application of the principles of this disclosure, are fully contemplated, as this would typically occur to those skilled in the art relating to this disclosure. In particular, it is fully contemplated that features, components, and / or steps described with respect to one embodiment can be combined with features, components, and / or steps described with respect to other embodiments of this disclosure. Furthermore, the dimensions provided herein are for specific examples, and the concepts of this disclosure are contemplated for implementation using different sizes, dimensions, and / or ratios. To avoid unnecessary descriptive repetition, one or more components or actions described according to one illustrative embodiment may be used or omitted in other illustrative embodiments, if applicable. For brevity, numerous iterations of these combinations will not be described separately. For simplicity, in some cases, the same reference numerals are used throughout the drawings to refer to the same or similar parts.

[0024] While some of the examples described herein frequently refer to surgical procedures or tools, or medical procedures or tools, the disclosed techniques are also applicable to non-medical procedures and non-medical tools. For example, the tools, systems, and methods described herein can be used for non-medical purposes, including industrial applications, general robotic applications, and the sensing or manipulation of non-tissue workpieces. Other example applications involve surgical or non-surgical cosmetic improvements, imaging or data collection from human or animal anatomy, training medical or non-medical personnel, performing procedures on tissue removed from human or animal anatomy (without returning the human or animal anatomy), and performing procedures on human or animal cadavers.

[0025] The following examples will describe various tools and parts thereof according to their states in three-dimensional space. As used herein, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom, such as along the Cartesian X, Y, Z axes, which can be described using variations in Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or part of an object (three rotational degrees of freedom—e.g., which can be described using roll, pitch, and yaw). As used herein, the term "attitude" refers to the position of an object or part of an object in at least one translational degree of freedom, and the orientation of the object or part of the object in at least one rotational degree of freedom. For an asymmetric rigid body in three-dimensional space, the complete attitude can be described using six parameters out of six total degrees of freedom.

[0026] Refer to the attached diagram. Figure 1 An example robotic system is shown. Specifically, in Figure 1 In this context, computer-aided robotic medical systems, which can be remotely operated and used for medical procedures, such as diagnostic, therapeutic, or surgical procedures, are generally indicated by reference numeral 10. As will be described, the remote operating system of this disclosure is under the remote operational control of an operator. In some embodiments, the manipulators or other parts of the robotic system can be directly controlled through manual interaction with the manipulators (or other parts) themselves. Thus, "remotely operated manipulators" as used in this application includes manipulators that can be partially or fully controlled remotely, and includes manipulators that can be controlled simultaneously or in a time-multiplexed manner through both remote operation and direct manual control. Furthermore, in some embodiments, non-remotely operated or robotic medical systems may be under partial control of a computer programmed to execute procedures or sub-procedures. In other alternative embodiments, procedures or sub-procedures may be executed using a fully automated medical system under the full control of a computer programmed to execute procedures or sub-procedures.

[0027] like Figure 1As shown, the robotic medical system 10 generally includes a manipulator assembly 12 mounted to or near a worktable O on which a patient P is positioned. The manipulator assembly described herein typically includes one or more robotic manipulators and tools mounted thereon, although the term "manipulator assembly" also includes manipulators without tools mounted thereon. The manipulator assembly 12 in this example may be referred to as a patient-side trolley because it includes a trolley and is designed for use alongside a patient. Medical tools 14 (also referred to as tool 14) and 15 are operatively coupled to the manipulator assembly 12. Within this disclosure, 15 includes an imaging device and may also be referred to as an imaging tool 15. Imaging tool 15 may include an endoscopic imaging system using optical imaging techniques, or another imaging system using other techniques (e.g., ultrasound, fluorescence, etc.). An operator input system 16 allows an operator, such as a surgeon or other type of clinician S, to view images representing the surgical site and control the operation of medical tool 14 and / or imaging tool 15.

[0028] The operator input system 16 for the robotic medical system 10 can be "mechanically grounded" by connecting to a base with linkages (such as connecting to an operator console), or it can be "mechanically ungrounded" and not connected in this way. Figure 1 As shown, the operator input system 16 is connected to the operator console 38, which is typically located in the same room as the operating table O during surgical procedures. However, it should be understood that the operator S may be located in a different room or a completely different building from the patient P. The operator input system 16 typically includes one or more control devices for controlling the medical tool 14. The operator input system 16 is also referred to herein as the “master manipulator,” “master control device,” “master input device,” and “input device.” The control device(s) may include one or more of any number of various input devices, such as hand grips, joysticks, trackballs, data gloves, trigger guns, foot pedals, hand controllers, voice recognition devices, touchscreens, body motion or presence sensors, and so on. In some embodiments, the control device(s) will be provided with the same degrees of freedom as the medical tool as the robotic component to provide telepresence to the operator; that is, the operator has a perception that the control device(s) is integrated with the tool, giving the operator the feeling of directly controlling the tool as if on-site during the procedure. In other embodiments, the control device(s) may have more or fewer degrees of freedom than the associated medical tool and still provide telepresence to the operator. In some embodiments, the control devices(s) are manual input devices that move in six degrees of freedom and may also include an actuable handle for actuating medical tools (e.g., for disabling gripper-end effectors, applying potentials to electrodes, capturing images, delivering drug therapy, etc.).

[0029] Manipulator assembly 12 supports and manipulates medical instrument 14 while operator S views the surgical site via operator console. Images of the surgical site can be acquired by medical instrument 15, such as via an imaging system including a monocular or stereoscopic endoscope, which can be manipulated by manipulator assembly 12 to orient the medical instrument 15. An electronic trolley can be used to process the images of the surgical site for subsequent display to operator S via operator console. The number of medical instruments 14 used at one time generally depends on factors such as medical diagnostic or treatment (e.g., surgery) procedures and space constraints within the operating room. Manipulator assembly 12 may include a kinematic structure of one or more links coupled by one or more non-servo-controlled joints, and servo-controlled robotic manipulators. In various embodiments, the non-servo-controlled joints can be manually positioned or locked to allow or inhibit relative movement between links physically coupled to the non-servo-controlled joints. Manipulator assembly 12 includes multiple motors that drive inputs on the medical instrument 14. These motors move in response to commands from a control system (e.g., control system 20). The motor includes a drive system that, when coupled to the medical instrument 14, can advance the medical instrument into an anatomical opening created naturally or surgically. Other electrically driven systems can move the distal end of the medical instrument in multiple degrees of freedom, including three linear degrees of motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three rotational degrees of motion (e.g., rotation about the X, Y, Z Cartesian axes). Furthermore, the motor can be used to actuate articulated end effectors of the instrument for grasping tissue in pawls of biopsy devices, etc. The medical instrument 14 may include an end effector having a single working component, such as a scalpel, blunt blade, needle, imaging sensor, optical fiber, electrode, etc. Other end effectors may include multiple working components, and examples include forceps, grippers, scissors, applicators, staplers, bipolar electrocautery devices, etc.

[0030] The robotic medical system 10 also includes a control system 20. The control system 20 includes at least one memory 24 and at least one processor 22, and typically multiple processors, for implementing control between the medical instrument 14, the operator input system 16, and other auxiliary systems 26, which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, lighting systems, steering control systems, flushing systems, and / or suction systems. The control system 20 also includes programming instructions (e.g., a computer-readable medium storing the instructions) to implement some or all of the methods described according to the aspects disclosed herein. Although the control system 20... Figure 1The simplified schematic is shown as a single block, but the system may include two or more data processing circuits, with some of the processing optionally executed on or near the manipulator assembly 12, others at the operator input system 16, and so on. Any of a variety of centralized or distributed data processing architectures may be employed. Similarly, programming instructions may be implemented as multiple separate programs or subroutines, or they may be integrated into multiple other aspects of the remote operating system described herein. In one embodiment, the control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0031] In some embodiments, the control system 20 may include one or more actuator controllers that receive force and / or torque feedback from the medical instruments 14 or from the manipulator assembly 12. In response to this feedback, the actuator controllers transmit signals to the operator input system 16. The actuator controllers(s) may also transmit signals instructing the manipulator assembly 12 to move the medical instruments(s) 14 and / or 15 within an internal surgical site extending into the patient's body via an opening in the body. Any suitable conventional or specialized controller may be used. The controller may be separate from or integrated with the manipulator assembly 12. In some embodiments, the controller and manipulator assembly are provided as part of an integrated system, such as a remotely operated arm positioned near the patient's body during medical procedures.

[0032] The control system 20 can be coupled to the medical instrument 15 and may include a processor to process the captured images for subsequent display, such as displaying them to an operator using an operator console or wearing a head-mounted display system, displaying them on one or more fixed or portable monitors near the control system, or displaying them on another suitable display located locally and / or remotely. For example, in the case of using a stereoscopic endoscope, the control system 20 may process the captured images to present the operator with a coordinated stereoscopic image of the surgical site. This coordination may include alignment between the stereoscopic images and may include adjusting the stereoscopic working distance of the stereoscopic endoscope.

[0033] In alternative embodiments, the robotic system may include more than one manipulator assembly and / or more than one operator input system. The exact number of manipulator assemblies will depend on factors such as surgical procedures and space constraints within the operating room. Operator input systems may be juxtaposed, or they may be positioned in separate locations. Multiple operator input systems allow more than one operator to control one or more manipulator assemblies in various combinations.

[0034] In various embodiments, the operator console 38 includes left-eye and right-eye displays for presenting the operator S with a coordinated stereoscopic view of the surgical environment to achieve depth perception. The operator input system 16 of the operator console 38 includes one or more input controls that subsequently enable the manipulator assembly 12 to manipulate one or more medical instruments 14 and / or 15. The input controls can be used, for example, to deactivate gripper-end effectors, apply electromotive force to electrodes, deliver medication, etc. In various alternatives, the input controls may additionally or alternatively include joystick devices, trackballs, data gloves, trigger guns, hand controllers, voice recognition devices, touchscreens, body motion or presence sensors, etc. In some embodiments, and for some associated medical instruments 14, the input controls will provide the same degrees of freedom as their associated medical instruments 14 to provide a remote presentation to the operator S, or to give the operator S the feeling that the input controls 36 are integrated with the instruments 14, giving the operator S the feeling of direct control over the instruments 14. In other embodiments, the input controls may have more or fewer degrees of freedom than the associated medical instruments, and still provide remote control to the operator S. For this purpose, position, force, and tactile feedback sensors can be used to transmit position, force, and tactile sensations from tool 14 back to the operator S via an input control device. The operator input system 16 of the operator console 38 may also include an input control device, which includes a foot pedal for receiving input from the user's feet.

[0035] Now for reference Figure 2 This illustrates a manipulator assembly 300 with a single manipulator 302. The manipulator assembly 300 can be configured as a patient-side trolley (e.g., Figure 1 Example manipulator component 12), or mounted to a patient table or table rail (e.g., surgical table, examination table), ceiling, wall, or floor. Figure 2 In the example, the manipulator assembly 300 includes a manipulator 302, and a replaceable tool 350 is shown mounted on the manipulator 302. The manipulator 302 and the tool 350 may also be referred to herein as apparatus 350.

[0036] In some embodiments, tool 350 may be configured to manipulate industrial workpieces or manipulate human or animal tissue for reasons other than medical treatment or diagnosis. In some embodiments, tool 350 may include tools for performing medical procedures. Tool 350 includes a mounting portion 352 and a shaft 354. Figure 2In the example shown, mounting portion 352 includes a bracket located on the proximal portion of tool 350. As used herein, the term proximal generally refers to a direction or location away from the workpiece or patient, and distal generally refers to a direction or location closer to the workpiece or patient. The bracket is configured to removably engage tool 350 with a fifth connector that allows the carriage 353 of manipulator 302 to move. Figure 2 As shown, the fifth connector includes a prismatic connector aligned along the insertion direction of the tool 350. The shaft 354 is coupled to the end effector 360 via a wrist 358. The end effector 360 has a tool tip 362. In some embodiments, the manipulator assembly 300 may include a support for a port device (e.g., a cannula for some medical procedures) that guides or restricts movement of the tool 350 relative to the manipulator assembly 300. The tool 350 associated with each manipulator assembly 300 may also be operated by the operator via an operator input system (e.g., ...). Figure 1 Example of operator input system 16) control.

[0037] More specifically, the example manipulator 302 includes links L1, L2, L3, L4, and a fifth link (e.g., denoted as L5, including a carriage 353), which are connected in a kinematic chain via joints J1, J2, J3, J4, and a fifth joint (e.g., denoted as J5). A mounting portion 352 of the tool 350 is mounted to L5, which is physically coupled to link L4. Each of the joints (e.g., J1, J2, J3, J4, and J5) is motor-controlled. In the example, movement of J5 moves L5 relative to L4, providing insertion and withdrawal movements to the tool 350. Other manipulator designs may not implement this J5 of the movable carriage 353; or, other manipulator designs may not have a carriage 353 at all and may be coupled to the tool 350 in another manner, with the manipulator inserting and withdrawing from the tool 350 by moving one or more other joints (e.g., joints J2-J4). Thus, at least a portion of the manipulator assembly 300 is configured to move using electrically or actively coupled joints. In this embodiment, the motor of manipulator 302 is controlled by a control system (e.g., control system 20), and if manipulator assembly 300 has other manipulators, it can coordinate with the motors of other manipulators of the same manipulator assembly 300, or with other manipulator assemblies, to assume a desired posture. This can assist in advancing, installing tools, preparing steps, storing, moving, and manipulating target anatomical structures within the patient's body, placing remote centers of motion, making room for assistants, obstacles, or equipment surrounding the patient, and applying forces to anatomical structures, such as for activities like palpating tissue. Furthermore, encoders and other sensors associated with each motor or connector of manipulator assembly 200 provide feedback to the control system, enabling the control system to receive data related to the motion state, status, torque applied by or on the connector, and settings of the connector / motor of manipulator assembly 300, sensing or detecting or determining the motion state of the connector / motor, the status of manipulator assembly 300, the torque applied by or on the connector, and settings.

[0038] Although each of the joints (e.g., J1, J2, J3, J4, and J5) can be controlled by individual or multiple joint or actuator controllers, the joints and actuator controllers can be controlled by a common joint control unit of a common control system (e.g., control system 20, master / slave control system, etc.). Therefore, tool 350, the tip 362 and end effector 360 of tool 350, and manipulator 302 can be controlled by a user (e.g., operator S) of their associated control devices (e.g., Figure 1 The example is the operator input system) manipulated to control.

[0039] Notice, Figure 2The kinematic configuration of the manipulator assembly 300 shown is merely exemplary and is not intended to limit it beyond what is specifically described in the appended claims. Those skilled in the art will understand that other configurations can be used. For example, one or more of the connectors (e.g., connectors J1, J2, J3, J4, J5) may be non-servo-controlled and may be configured to allow manual positioning or locking. As another example, the manipulator assembly 300 may include a variety of numbers and types of connectors and combinations thereof (e.g., rotary connectors, prismatic connectors). In one example, the manipulator assembly 300 may include a parallelogram linkage. In another example, the manipulator assembly 300 may include a prismatic connector proximal to the base link L0 and one or more rotary connectors distal to the base link L0. In this example, the one or more rotary connectors distal to the base link L0 may rotate in a particular plane or in three dimensions. In yet another example, the manipulator assembly 300 may include a single-port platform comprising a base manipulator carrying a plurality of sub-manipulators. In this example, each of the sub-manipulators may be connected in series to the base manipulator.

[0040] exist Figure 2 In some examples, an external environment detection sensor system 304 (also referred to as external environment sensor system 304 or sensor system 304) is attached to the manipulator assembly 300. In various examples, the sensors of the external environment detection sensor system 304 may be located at one or more of the links (L0, L1, L2, L3, L4, or L5) and connectors (J1, J2, J3, J4, or J5) of the manipulator assembly 300. In some examples where the manipulator assembly 300 includes a clamp, the sensors(s) of the manipulator assembly may be coupled to the clamp(s).

[0041] The external environment detection sensor system 304 can provide information about the external environment of the manipulator assembly 300 (e.g., to the control system 20). The external environment detection sensor system 304 may include one or more sensors, including, for example, optical sensors, depth sensors, time-of-flight sensors, transmitter-receiver sensors, any other suitable sensors, and / or combinations thereof. In some examples, the optical sensor includes an imaging device that detects visible or non-visible light. The optical sensor will detect images of other manipulator assemblies and process the resulting images to identify and locate portions of external objects (e.g., other manipulator assemblies). For example, different manipulator assemblies can be identified by markings, colors, shapes, support tools, movements specific to the manipulator assembly visible to such sensors. Depth information can be provided by integrated or separate depth sensors, through triangulation using multiple imaging devices or stereo imaging devices, or any suitable technique. In some examples, the time-of-flight sensor includes a laser rangefinder, an LED rangefinder, lidar, radar, etc. In embodiments, when the sensors include optical sensors or time-of-flight sensors, the control system can detect and process occlusion because those sensors can only provide information about external objects if they are able to view at least a portion of the external object.

[0042] In some embodiments, the sensors may include accelerometers, electromagnetic sensors, RFID sensors, inclinometers, or inertial measurement units (IMUs). Accelerometers, inclinometers, and IMUs may not directly provide manipulator assembly-to-manipulator assembly registration data; instead, they may be used to provide orientation information relative to a world reference frame, which can be used to provide some of the rotational transformations between manipulator assemblies, or as a check on rotational transformations calculated in other ways.

[0043] In various embodiments, the manipulator assembly 300 may have different arrangements of external environment detection sensor systems. Figure 2 In some examples, the manipulator assembly 300 may include multiple sensors, each located on a different link or joint of the manipulator assembly 300. In some examples, a single link or joint may have multiple sensors of the external environment detection sensor system 304. Note that although in Figure 2 In this embodiment, the external environment detection sensor system 304 is attached to each link (or tool rigidly mounted to the link) and each connector of the manipulator 302, but in some embodiments, the manipulator 302 may include links and / or connectors to which no external environment detection sensor system is attached.

[0044] like Figure 2As shown in the example, different tools 350 and / or end effectors 360 can be mounted to the manipulator assembly 300 to perform different functions. In this example, the external environment detection sensor system 304 attached to the manipulator 302 can be used to provide data for controlling the movement of the different tools 350 and / or end effectors 360.

[0045] In some embodiments, the sensor data provided by the external environment detection sensor system 304 includes spatial information of a detected external object (e.g., another manipulator component) relative to the manipulator assembly 300. In some examples, the sensor data includes one or more images detected by one or more image sensors of the external environment detection sensor system 304. In some examples, the sensor data may also include identification information for identifying the detected external object (e.g., another manipulator component). In some examples, the sensor data may include identification information for identifying the sensor position (e.g., a linkage of a manipulator component).

[0046] In various embodiments, the image sensors described herein may include a variety of sensors for various types of sensing technologies, which can be used to provide images of various dimensions (e.g., images of two-dimensional (2D), three-dimensional (3D), or any other suitable higher-dimensional representation of space). In various examples, 3D images can be provided, for example, directly from a 3D image sensor, constructed from a range of 2D sensor information and any / or other suitable sensor information, and / or constructed using any other suitable technology. For example, a 3D image can be constructed from 2D sensor information using depth information including a depth map. In various examples, depth information can be provided by a variety of technologies, including, for example, stereo imaging, depth cameras, laser ranging technologies, etc. Therefore, the image sensors described herein may include any sensor configured to generate 2D, 3D, or higher-dimensional representations of space, including, for example, capacitive sensors designed to provide capacitive 2D representations of capacitance in a region (e.g., touchscreens on mobile phones), level sensors, switches, IR cameras, LiDAR, depth cameras, radar, sonar, ultrasonic sensors, optical cameras, any other suitable sensors, and / or combinations thereof.

[0047] See below for reference. Figures 3 to 6 The description describes a control system (e.g., for use in an environment comprising multiple manipulator components each having a separately movable base) Figure 1The example control system 20 can receive sensor data including environmental information from outside the corresponding manipulator component. This sensor data may be provided by one or more external environment detection sensor systems on the corresponding manipulator component, and the control system can use the sensor data to perform registration of multiple manipulator components. This registration can be used to control those manipulator components and / or associated tools, and to provide enhanced operation and / or collision avoidance. In some embodiments, the sensor data (e.g., for registration) is used to determine environmental information, such as the location or shape of non-manipulator items (e.g., obstacles including, for example, one or more operators or patients). This determination can be performed continuously or periodically, and provides dynamic and / or real-time obstacle detection, which can be further used to enhance operation, avoid collisions, etc.

[0048] refer to Figure 3 As an example, the flowchart provides a method 380 for performing a registration process for manipulator components that can move independently relative to each other. Method 380 begins at process 382, ​​where an operating environment for the operation of a robotic system is provided, and the robotic system includes a first manipulator component and a second manipulator component having independently movable bases. Method 380 may proceed to process 384, where the control system receives sensor data, including first sensor data from multiple sensors on the first manipulator component, to provide an operating environment external to the first manipulator component. Method 380 may proceed to process 386, where the control system uses the sensor data including the first sensor data to determine a spatial relationship between the first manipulator component and the second manipulator component. Method 380 may then proceed to process 388, where the control system establishes a first alignment relationship (e.g., a transformation for registration) between the first manipulator component and the second manipulator component based on the first spatial relationship. Method 380 may proceed to process 390, where the control system switches from a registration mode (e.g., including processes 382-388) to a tool control mode (e.g., in a medical example, performing actions on a patient at a workbench during a medical procedure). When operating in tool control mode, the control system can control the movement of the second manipulator assembly or the associated tool therein relative to the imaging device's reference frame (also referred to as the "imaging device reference frame") in response to the movement of the master control device associated with the tool.

[0049] refer to Figure 4A , Figure 4B , Figure 4C , Figure 4D and Figure 4E The examples illustrate various configurations of the external environment detection sensor system in the robotic system 400, including two manipulator components with independently movable bases. (Reference) Figure 4A The example shows a robot system 400 (e.g., Figure 1The robotic medical system 10 includes two manipulator assemblies 402 and 404, respectively, mounted on separate bases 406 and 422. Manipulator assembly 402 includes base 406, structural support 408, and manipulator 410. Figure 4A In this example, the imaging tool 15 is mounted on the manipulator 410, and thus the manipulator assembly 402 can be considered as further including the mounted imaging tool 15. The imaging tool 15 includes an axis 412 and an imaging device 414. The imaging device 414 may include, for example, an optical imager, an ultrasonic imager, an electromagnetic imager such as a fluorescence imager, a thermal imager, a thermoacoustic imager, and any other suitable imager. The imaging device 414 has a field of view 416.

[0050] like Figure 4A As shown, the base 406 has a reference frame 418, also referred to as the imaging base reference frame 418 (denoted as b1). The imaging device 414 has a reference frame 420, also referred to as the imaging device reference frame 420 (denoted as c). The transformation from the base reference frame 418 to the imaging device reference frame 420 is represented as follows: b1 T c It can be determined based on the forward kinematics of the manipulator component 402.

[0051] like Figure 4A As shown, the robot system 400 also includes a manipulator assembly 404. The manipulator assembly 404 includes a base 422, which is physically separate from and independent of the base 406 of the manipulator assembly 402. The manipulator assembly 404 includes a structural support 424 and a manipulator 426. Figure 4A In the example, tool 14 is mounted on manipulator 426, and therefore manipulator assembly 404 can be considered to further include the mounted tool 14. Tool 14 includes shaft 428, wrist 430 coupled to the distal end of shaft 428, and end effector 432 coupled to wrist 430. Base 422 has a reference frame 434, which is also referred to as tool base reference frame 434 (denoted as b2). Shaft 428 of tool 14 has a reference frame 436, which is also referred to as shaft reference frame 436 (denoted as s). The transformation from tool base reference frame 434 to shaft reference frame 436 can be expressed as follows: b2 T s And it can be determined (e.g., based on the forward kinematics of manipulator component 404).

[0052] In the examples, manipulator assemblies 402 and 404 may be positioned in different trolleys that are movable relative to each other. In another example, manipulator assemblies 402 and 404 may include clamps that allow them to be clamped to different components (e.g., bed frame, bed rails, ceiling clamps, etc.). In some examples, each manipulator assembly includes clamps for removably coupling the manipulator assembly to a surgical table rail, which allows the manipulator assembly to be positioned in different configurations around the surgical table depending on the surgical procedure to be performed. In some examples, one or more manipulator assemblies are coupled to their respective mounting systems. In those examples, each manipulator assembly is movable independently relative to the other manipulator assembly and may be positioned in different configurations beside the surgical table depending on the surgical procedure to be performed.

[0053] In various embodiments, the positions and orientations of bases 406 and 422 relative to each other are unknown. Therefore, the transformation from imaging base reference frame 418b1 to tool base reference frame 434b2... b1 T b2 This unknown alignment between bases 406 and 422 can make intuitive control of the slave tool / end effector difficult. To provide an effective control relationship between the master device and its slave tool / end effector (also known as master-tool alignment), spatial alignment is required between the master device and the tool / end effector. This spatial alignment provides a reasonably accurate relationship between the operator's perceived movement of the master device (e.g., proprioception) and the operator's perceived resulting movement of the tool, including the shaft and end effector (e.g., visual perception). For example, if the operator moves their hand gripping the master device to the left, the operator expects to perceive that the associated slave tool / end effector also moves to the left. If the perceived spatial movements match, the operator can easily control the movement of the slave tool / end effector by moving the master device. However, if the perceived spatial movements do not match (e.g., the master device moving to the left causes the slave tool / end effector to move upwards and to the right), the operator will find it difficult to control the movement of the slave tool by moving the master device. As detailed below, the registration process using an external environment sensor system can be used to determine the unknown alignment between bases 406 and 422 (also referred to as the alignment between manipulator assemblies 402 and 404), which can then be used to determine host-tool alignment and host-tool transformation.

[0054] Manipulator components 402 and 404 (e.g., Figure 2One or more of the manipulator components 300 may include corresponding external environment sensor systems (e.g., external environment sensor system 304). The registration process may use sensor data from the external environment sensor systems coupled to the manipulator components to determine the alignment of the manipulator components and the host-tool alignment. Additional information (e.g., known kinematic relationships and reference frame transformations in the robot system) may also be used. In some examples, such additional information may include link data provided to the control system by link sensor systems attached to the links of the manipulator and / or to the tools supported by the manipulator, wherein the link data may include, for example, measurements and / or estimates of the states of the links (e.g., attitude, velocity, acceleration). These relationships are described below in Cartesian terms, although other 3D coordinate systems may be used.

[0055] Various configurations of the sensor system 304 can be provided. See below. Figures 4B to 4E As shown, multiple sensors in the sensor system 304 of the manipulator assembly provide redundancy and redundant data, which mitigates problems introduced by occlusion and improves the overall accuracy of the dataset. In some examples, the manipulator assembly, including the clamps with sensors, is coupled to the same rail of the surgical table (i.e., coupled to the same side of the surgical table). In such examples, the sensors of the clamps coupled to different manipulator assemblies can be within each other's field of view. These sensors can communicate with each other and / or with the control system, which allows the control system to determine the position of one manipulator assembly relative to another. The control system can determine those relative positions using data that includes redundant data provided by all sensors (e.g., using kinematic and / or dynamic calculations).

[0056] In other examples, the manipulator assemblies are coupled to different rails on the surgical table (i.e., coupled to different or opposite sides of the surgical table). In those examples, there may be obstruction between some sensors of the clamps connected to different manipulator assemblies (e.g., a first sensor of a first manipulator assembly may be obstructed from or outside the field of view of a second sensor of a second manipulator assembly). However, because of the redundant sensors on each manipulator assembly, other sensors of the manipulator assemblies (e.g., coupled to other joints or links) do not obstruct each other, and those unobstructed sensors can provide sufficient spatial relationship information, enabling the control system to determine the spatial relationship of one manipulator assembly relative to another.

[0057] refer to Figure 4BThe example illustrates a robot system 400 with an external environment sensor system. Manipulator assemblies 402 and 404 are mounted at different locations on a guide rail 450 near the control table O. The external environment sensor system 304 of manipulator assembly 402 includes sensors 304-1 and 304-2. In some embodiments, sensor 304-1 includes an imaging device with a field of view 452-1, wherein sensor data (e.g., image data) from sensor 304-1 does not include sufficient information (e.g., an image of all or part of manipulator assembly 404) for determining the spatial relationship between manipulator assemblies 402 and 404. In the example, as... Figure 4B As shown, the manipulator component 404 is not in the field of view 452-1.

[0058] exist Figure 4B In the example, sensor 304-2 includes an imaging device with a field of view 452-2. Figure 4B In one example, manipulator assembly 404 is in the field of view 452-2, but is partially or completely occluded from sensor 304-2 (e.g., by intermediate object 453), such that the sensor data (e.g., image data) from sensor 304-2 does not include sufficient information (e.g., an image of all or part of manipulator assembly 404) for determining the spatial relationship between manipulator assemblies 402 and 404. In other examples, manipulator assembly 404 is not occluded by sensor 304-1, and the sensor data (e.g., image data) from sensor 304-2 includes sufficient information (e.g., an image of all or part of manipulator assembly 404) for the control system to determine the spatial relationship between manipulator assemblies 402 and 404.

[0059] exist Figure 4B In one example, the manipulator assembly 404 does not include the external environment sensor system 304. In other examples, the manipulator assembly 404 also includes the external environment sensor system 304.

[0060] refer to Figure 4C The example shows another example of a robot system 400 with an external environment sensor system. Figure 4C The robot system 400 is basically similar to Figure 4B The robot system 400, except for the differences described below, includes an external environment sensor system 304 comprising sensors 304-3 and 304-4. In some embodiments, sensor 304-3 includes an imaging device having a field of view 452-3, wherein sensor data (e.g., image data) from sensor 304-3 does not include sufficient information (e.g., an image of all or part of manipulator assembly 402) for determining the spatial relationship between manipulator assemblies 402 and 404. In the example, as... Figure 4CAs shown, manipulator assembly 402 is not in the field of view 452-3. In another example, manipulator assembly 402 is in the field of view 452-3, but is occluded by sensor 304-3 (e.g., by a third component), such that the sensor data (e.g., image data) from sensor 304-3 does not include sufficient information (e.g., an image of all or part of manipulator assembly 402) for determining the spatial relationship between manipulator assemblies 402 and 404.

[0061] exist Figure 4C In this example, sensor 304-4 includes an imaging device with a field of view 452-4, which includes a manipulator assembly 402, and sensor data (e.g., image data) from sensor 304-4 includes sufficient information (e.g., an image of all or part of manipulator assembly 402) for determining the spatial relationship between manipulator assemblies 402 and 404. In this example, manipulator assembly 402 is not obstructed by sensor 304-4.

[0062] refer to Figure 4D The example illustrates yet another instance of a robotic system 400 with an external environment sensor system. Figure 4D In the example, multiple sensors in the external environment sensor system may include reflection-based transmitter-receiver sensors. For example, the external environment sensor system 304 of the manipulator assembly 402 includes transmitter-receiver sensor 304-5, which can detect spatial relationships (e.g., distance, orientation) of nearby objects. Transmitter-receiver sensor 304-5 may be an optical time-of-flight sensor, including a transmitter 454 (e.g., configured to emit infrared light 458) and a receiver 456 (e.g., configured to receive reflected infrared light 460) to determine the distance to an object (e.g., the manipulator assembly 404).

[0063] refer to Figure 4E The example shows yet another example of a robotic system 400 with an external environment sensor system. Figure 4C The robot system 400 is basically similar to Figure 4BThe robot system 400, except for the differences described below, includes an external environment sensor system 304 comprising sensors 304-1 and 304-2 for manipulator components 404. Sensor 304-1 includes an imaging device with a field of view 452-1, wherein sensor data (e.g., image data) from sensor 304-1 does not include sufficient information (e.g., an image of all or part of manipulator components 402) for determining the spatial relationship between manipulator components 402 and 404. Sensor 304-2 includes an imaging device with a field of view 452-2, wherein sensor data (e.g., image data) from sensor 304-2 includes a partial view of manipulator components 402. In various embodiments, such partial view sensor data from sensor 304-2 can provide sufficient information for determining the spatial relationship between manipulator components 402 and 404, wherein the control system can use both the partial view sensor data and the complete kinematic information of both manipulator components 402 and 404 to determine the spatial relationship.

[0064] refer to Figure 5A , Figure 5B , Figure 5C , Figure 5D and Figure 5E The example illustrates various configurations of the external environment detection sensor system in Robot System 500, which includes a third component in addition to two manipulator assemblies with independently movable bases. Although in Figures 5A to 5E In the example, the third component is another manipulator assembly on a separately movable base, but in various embodiments, the third component can be any suitable component with a movable or fixed base in the operating environment including the robot system 500, including, for example, a column, clamp, gripper, etc.

[0065] refer to Figure 5A The example illustrates Robot System 500. Robot System 500 is essentially similar to... Figure 4A Robot system 400 differs from the one described below. Robot system 500 includes a third component in addition to the two manipulator components. Figure 5A In the example, the third component includes a manipulator assembly 502. Manipulator assembly 502 includes a base 504, which is physically separate and independent from and independent of the bases 406 and 422 of manipulator assembly 404. Manipulator assembly 502 includes a structural support 508 and a manipulator 510. Figure 5AIn the example, tool 512 is mounted on manipulator 510, and therefore manipulator assembly 502 can be considered to further include the mounted tool 512. Tool 512 includes shaft 514, wrist 516 coupled to the distal end of shaft 514, and end effector 518 coupled to wrist 516. Base 504 has a reference frame 506, which can also be represented as b3. The shaft 514 of tool 512 has a reference frame, and the transformation from base reference frame 506 to shaft reference frame can be determined based on the forward kinematics of manipulator assembly 502.

[0066] In various embodiments, the relative positions and orientations of bases 504, 406, and 422 are unknown. As discussed in detail below, in various embodiments, a third component (e.g., manipulator assembly 502) can be used to determine the transformation from imaging base reference frame b1418 to tool base reference frame b2434. b1 T b2 In the example, the transformation b1 T b2 This can be determined as follows:

[0067] b1 T b2 = b1 T b3 * b3 T b2 (1)

[0068] in b1 T b3 It is a transformation from the imaging base reference frame b1 418 to the base reference frame b3 506, and in which b3 T b2 Transformation T is the transformation from base reference frame b3 506 to base reference frame b2 422. Transformation T can include a complete 6x6 transformation matrix, a 3x3 rotation matrix (also referred to as R), or any suitable transformation format.

[0069] refer to Figure 5BThe example illustrates a robot system 500 with an external environment sensor system. The robot system 500 includes manipulator assemblies 402, 404, and 502 mounted on rails near a control table O. Sensor data from sensor 304-2 of the external environment sensor system 304 of manipulator assembly 402 does not provide sufficient spatial information about manipulator assembly 404 relative to manipulator assembly 402 (e.g., due to occlusion 520), although manipulator assembly 404 is within the field of view of sensor 304-2 if occlusion 520 is not present. Sensor data from sensor 304-7 of the external environment sensor system 304 of manipulator assembly 402 also does not provide sufficient spatial information about manipulator assembly 404 relative to manipulator assembly 402 (e.g., due to occlusion or a limited field of view of image sensor 304-7). Sensor data from sensors 304-8 of the external environment sensor system 304 of the manipulator assembly 404 does not provide sufficient spatial information about the manipulator assembly 402 relative to the manipulator assembly 404 (e.g., due to occlusion or limited field of view of the image sensor 304-8).

[0070] exist Figure 5B In the example, sensor data from sensors 304-7 of the external environment sensor system 304 of the manipulator assembly 402 provides sufficient spatial information about the manipulator assembly 502 relative to the manipulator assembly 402. Therefore, the control system can use the sensor data from sensors 304-7, for example, by determining the transformation... b1 T b3 To determine the first alignment relationship between manipulator assemblies 402 and 502 (the first alignment relationship is the alignment relationship between the corresponding bases b1 and b3).

[0071] Sensor data from sensors 304-8 of the external environment sensor system 304 of the manipulator assembly 404 provides sufficient spatial information about the manipulator assembly 502 relative to the manipulator assembly 404. Therefore, the control system can use the sensor data from sensors 304-8, for example, by determining transformations. b3 T b2 To determine a second alignment relationship between manipulator assemblies 404 and 502 (the second alignment relationship is the alignment relationship between the corresponding bases b2 and b3).

[0072] Then, the control system can determine the alignment relationship between the manipulator assemblies 402 and 404 based on the first alignment relationship and the second alignment relationship, for example, by using equation (1) based on b1 T b3 and b3 T b2 Determine transformation b1 T b2 .

[0073] In some examples, the third component (e.g., manipulator assembly 502) may not have a corresponding external environment sensor system 304. In some examples, the third component, manipulator assembly 502, may have a corresponding external environment sensor system 304, but the corresponding sensor data is not used by the control system to determine the alignment relationship between manipulator assemblies 402 and 404.

[0074] refer to Figure 5C The example illustrates another example of a robot system 500 with an external environment sensor system. The robot system 500 includes manipulator assemblies 402, 404, and 502 mounted on rails near the manipulator table O. The sensors 304-7 of the external environment sensor system 304 of the manipulator assembly 402 are substantially similar to... Figure 5B The sensor 304-7 does not provide sufficient spatial information about the manipulator assembly 404 relative to the manipulator assembly 402 (e.g., due to occlusion or limited field of view of the image sensor 304-7).

[0075] exist Figure 5C In the example, sensor data from sensors 304-7 of the external environment sensor system 304 of the manipulator assembly 402 provides sufficient spatial information about the manipulator assembly 502 relative to the manipulator assembly 402. Therefore, the control system can use the sensor data from sensors 304-7, for example, by determining the transformation... b1 T b3 To determine the first alignment relationship between manipulator assemblies 402 and 502 (the first alignment relationship is the alignment relationship between corresponding bases b1 and b3).

[0076] Sensor data from sensors 304-9 of the external environment sensor system 304 of the manipulator assembly 502 does not provide sufficient spatial information about the manipulator assembly 402 relative to the manipulator assembly 502 (e.g., due to occlusion or limited field of view of the image sensor 304-9). On the other hand, sensor data from sensors 304-9 of the external environment sensor system 304 of the manipulator assembly 502 provides sufficient spatial information about the manipulator assembly 404 relative to the manipulator assembly 502. Therefore, the control system can use the sensor data from sensors 304-9, for example, by determining the transformation... b3 T b2 To determine the second alignment relationship between manipulator assemblies 404 and 502 (the second alignment relationship is the alignment relationship between corresponding bases b2 and b3).

[0077] Then, the control system can determine the alignment relationship between the manipulator assemblies 402 and 404 based on the first alignment relationship and the second alignment relationship, for example, by using equation (1) based on b1 T b3 and b3 T b2 Determine transformation b1 T.

[0078] In some examples, the manipulator assembly 404 may not have a corresponding external environment sensor system 304. In some examples, the manipulator assembly 404 may have a corresponding external environment sensor system 304, but the corresponding sensor data is not used by the control system to determine the alignment relationship between the manipulator assemblies 402 and 404.

[0079] refer to Figure 5D The example illustrates yet another instance of a robotic system 500 with an external environment sensor system. Figure 5D In the example, sensor data from sensors 304-10 of the external environment sensor system 304 of the manipulator assembly 502 provides sufficient spatial information about the manipulator assembly 402 relative to the manipulator assembly 502. Therefore, the control system can use the sensor data from sensors 304-10, for example, by determining the transformation... b1 T b3 To determine the first alignment relationship between manipulator assemblies 402 and 502 (the first alignment relationship is the alignment relationship between corresponding bases b1 and b3).

[0080] exist Figure 5D In the example, sensor data from sensors 304-9 of the external environment sensor system 304 of the manipulator assembly 502 provides sufficient spatial information about the manipulator assembly 404 relative to the manipulator assembly 502. Therefore, the control system can use the sensor data from sensors 304-9, for example, by determining the transformation... b3 T b2 To determine the second alignment relationship between manipulator assemblies 404 and 502 (the second alignment relationship is the alignment relationship between corresponding bases b2 and b3).

[0081] Then, the control system can determine the alignment relationship between the manipulator assemblies 402 and 404 based on the first alignment relationship and the second alignment relationship, for example, by using equation (1) based on b1 T b3 and b3 T b2 Determine transformation b1 T b2 .

[0082] In some examples, manipulator components 402 and 404 may not have a corresponding external environment sensor system 304. In some examples, one or more of manipulator components 402 and 404 have a corresponding external environment sensor system 304, but the corresponding sensor data is not used by the control system to determine the alignment relationship between manipulator components 402 and 404.

[0083] refer to Figure 5E The example shows yet another example of a robotic system 500 with an external environment sensor system.

[0084] Sensor data from sensors 304-10 of the external environment sensor system 304 of the manipulator assembly 502 does not provide sufficient spatial information about the manipulator assembly 404 relative to the manipulator assembly 502 (e.g., due to occlusion or limited field of view of the image sensor 304-10). On the other hand, sensor data from sensors 304-10 of the external environment sensor system 304 of the manipulator assembly 502 provides sufficient spatial information about the manipulator assembly 402 relative to the manipulator assembly 502. Therefore, the control system can use the sensor data from sensors 304-10, for example, by determining the transformation... b1 T b3 To determine the second alignment relationship between manipulator assemblies 402 and 502 (the second alignment relationship is the alignment relationship between corresponding bases b2 and b3).

[0085] exist Figure 5E In the example, sensor data from sensors 304-11 of the external environment sensor system 304 of the manipulator assembly 404 does not provide sufficient spatial information about the manipulator assembly 402 relative to the manipulator assembly 404 (e.g., due to occlusion or limited field of view of the image sensor 304-11). On the other hand, sensor data from sensors 304-11 provides sufficient spatial information about the manipulator assembly 502 relative to the manipulator assembly 404. Therefore, the control system can use sensor data from sensors 304-7, for example, by determining the transformation... b1 T b3 To determine the first alignment relationship between manipulator assemblies 402 and 502 (the first alignment relationship is the alignment relationship between corresponding bases b1 and b3).

[0086] Then, the control system can determine the alignment relationship between the manipulator assemblies 402 and 404 based on the first alignment relationship and the second alignment relationship, for example, by using equation (1) based on b1 T b3 and b3 T b2 Determine transformation b1 T b2 .

[0087] In some examples, the manipulator assembly 402 may not have a corresponding external environment sensor system 304. In some examples, the manipulator assembly 402 may have a corresponding external environment sensor system 304, but the corresponding sensor data is not used by the control system to determine the alignment relationship between the manipulator assemblies 402 and 404.

[0088] refer to Figure 6 As an example, the flowchart provides a method 600 for performing a registration process for manipulator components that are individually movable relative to each other. Method 600 begins with process 602, which provides an operating environment for the operation of a robot system, and the robot system includes a first manipulator component and a second manipulator component having individually movable bases. Figures 4A to 5E The example provides an operating environment for a robot system (e.g., robot system 400 or 500) that includes manipulator components 402 and 404.

[0089] Method 600 can proceed to process 604, wherein the control system receives sensor data, including first sensor data, from multiple sensors on the first manipulator assembly to provide an operating environment external to the first manipulator assembly. Figures 4A to 5E In one example, the control system receives first sensor data (if available) from multiple sensors of an external environment sensor system 304 located on the manipulator assembly 402. The first sensor data may come from sensors located on various parts of the manipulator assembly 402 (e.g., different links and / or joints). In some examples, information about the operating environment outside the first manipulator assembly provides spatial information (e.g., orientation, position, etc.) of other manipulator assemblies (e.g., manipulator assembly 404) relative to the manipulator assembly 402.

[0090] Note that although in the description herein, manipulator component 402 is used as an example of a first manipulator component and manipulator component 404 is used as an example of a second manipulator component, in various examples the terms "first" and "second" may be reversed such that manipulator component 402 and 404 may be the second manipulator component and the first manipulator component, respectively.

[0091] Method 600 can proceed to process 606, wherein the control system receives sensor data, including second sensor data, from multiple sensors on the second manipulator assembly to provide an operating environment external to the second manipulator assembly. Figures 4A to 5EIn one example, the control system receives second sensor data (if available) from multiple sensors of an external environment sensor system 304 located on the manipulator assembly 404. The second sensor data may originate from sensors located on various parts of the manipulator assembly 404 (e.g., different links and / or joints). In some examples, information about the operating environment outside the second manipulator assembly provides spatial information (e.g., orientation, position, etc.) of other manipulator assemblies (e.g., manipulator assembly 402) relative to the manipulator assembly 404.

[0092] Method 600 can proceed to process 608, in which the control system determines whether the received sensor data (i.e., the received first sensor data and second sensor data) is sufficient to establish an alignment relationship (including, for example, transformation) between the first manipulator assembly and the second manipulator assembly.

[0093] In some embodiments, in process 608, the control system determines that the received sensor data is sufficient to establish an alignment relationship between the first manipulator assembly and the second manipulator assembly. In these embodiments, method 600 may proceed to process 616 to perform an adjustment operation.

[0094] In some embodiments, the adjustment operation of process 616 may include process 618, wherein the control system may provide an instruction (e.g., on a display) to the operator to add a sensor intermediary to the operating environment. Upon receiving the instruction, the operator may add the sensor intermediary to the operating environment. Figures 5A to 5E In the example, the operator can add a third component (e.g., manipulator component 502 or other components) as a sensor intermediary. Additional sensor data associated with the sensor intermediary can be provided. For example, as... Figure 5B , Figure 5C and Figure 5E As shown, the additional sensor data may include sensor data provided by the sensor system 304 of the manipulator assembly 402 and / or 404 (e.g., sensors 304-7, 304-8, 304-11), and includes spatial relationship information of the third component relative to the manipulator assembly 402 and / or 404. For further examples, such as... Figure 5C , Figure 5D and Figure 5E As shown, the additional sensor data may include sensor data provided by the sensor system 304 of the third component (e.g., sensors 304-9, 304-10), and includes spatial relationship information of the manipulator components 402 and / or 404 relative to the third component. This additional sensor data can be used to establish the alignment relationship between the manipulator components 402 and 404 in subsequent steps.

[0095] In some embodiments, the adjustment operation of process 616 may include process 620, in which the control system may provide the operator with instructions (e.g., on a display) to move one or more of the first and second manipulator assemblies. These instructions may also include suggested positions for the first and second manipulator assemblies (e.g., positions on rails, positions below the top of the console) to eliminate obstructions and improve the sufficiency of sensor data used to determine alignment relationships. In some examples, for a table-mounted system, the instructions may suggest that the manipulator assemblies be mounted to the console at the intermediate portion of structural support 408, and that structural support 408 extends below the console (i.e., below the clamp attachment points). In those examples, time-of-flight or other sensors may be used to provide an unobstructed view below the console.

[0096] In some examples where the robotic system includes a third component, the instruction may also include a suggested location for the third component (e.g., third manipulator assembly 502 or any other suitable third component). Upon receiving the instruction, the operator may move one or more manipulator assemblies and / or the third component to a new location (e.g., the suggested location).

[0097] In some embodiments, the adjustment operation of process 616 may include process 622, wherein the control system may automatically move one or more of the manipulator components and / or third components to improve the sufficiency of sensor data.

[0098] Then, method 600 can proceed to process 624, in which the control system receives updated sensor data after performing (multiple) adjustment operations, and proceeds to process 608 to determine whether the updated sensor data is sufficient to establish an alignment relationship between the first manipulator assembly and the second manipulator assembly.

[0099] In some embodiments, in process 608, the control system determines that the sensor data is sufficient to determine an alignment relationship (e.g., one or both, or orientation and positional relationship) between the first manipulator assembly and the second manipulator assembly. In those embodiments, method 600 may proceed to process 610 to establish the alignment relationship.

[0100] In some embodiments, process 610 may include process 611, wherein the control system uses first and / or second sensor data to determine a relative spatial relationship between a first manipulator assembly and a second manipulator assembly. Based on the determined relative spatial relationship, an alignment relationship between the first manipulator assembly and the second manipulator assembly is determined.

[0101] In some embodiments, where the updated sensor data includes additional sensor data associated with the sensor intermediary, process 610 may include process 612, wherein the control system uses the additional sensor data associated with the sensor intermediary to determine the alignment relationship between the first manipulator assembly and the second manipulator assembly. Figures 5A to 5E In the example, the control system determines the alignment between the first manipulator assembly and the second manipulator assembly based on additional sensor data associated with a sensor intermediary (e.g., a third component), such as based on equation (1).

[0102] Then, method 600 can proceed to process 614, in which the control system switches from registration mode to tool control mode (e.g., performing operations on a patient on the operating table during a medical procedure). When operating in tool control mode, the control system can control the movement of the tool relative to the imaging device reference frame in response to movement of the master control device associated with the tool. In order to move the tool effectively in the imaging device reference frame, the control system uses the alignment relationship between manipulator assemblies 402 and 404 (including, for example, the base transformation determined during the registration process). b1 T b2 This determines the alignment relationship between the imaging device reference system and the end effector reference system.

[0103] For example, the control system can calculate the transformation from the imaging device reference frame c to the end effector reference frame as follows. c T end effector :

[0104] c T end effector = c T b1 * b1 T b2 * b2 T end effector (2)

[0105] in c T b1 This is a transformation from the imaging device reference frame 420 to the imaging base reference frame 418. b2 T end effector It is a transformation from the tool base reference frame 434 to the end effector reference frame. c T b1 and b2 T end effector These are transformations that can be determined based on the forward and reverse kinematics of manipulator components 402 and 404, respectively, and b1 T b2 It has already been determined in process 610 through the registration process.

[0106] In some embodiments, during process 614, the control system may derive a master-tool transformation in response to state variable signals provided by the imaging system, such that the tool image on the display appears substantially connected to the master unit. These state variables typically indicate the Cartesian position of the imaging device's field of view, such as that provided by a manipulator supporting the imaging device. The control system may use the base transformation determined by the registration process. b1 T b2 This leads to the derivation of the host-tool transformation, enabling the control system to respond to the movement of the host device and appropriately control the movement of tool 14 relative to the imaging device reference frame.

[0107] In some embodiments, an operator reference frame is defined relative to the display or the operator viewing the display. In those embodiments, the control system can determine the alignment between the input device and the operator reference frame, and based on this alignment, command the movement of the second manipulator assembly in response to changes in the posture of the input device corresponding to the second manipulator assembly.

[0108] In some embodiments, the control system determines an alignment relationship between a first manipulator assembly and a third component, wherein the third component is another manipulator assembly. In those embodiments, movement of the third manipulator assembly can be in response to commands from an input device, such as commands based on changes in the attitude of the input device, based on the alignment relationship between the first and third manipulator assemblies. The input device for controlling the third manipulator assembly can be different from the input device for controlling the second manipulator assembly.

[0109] In various embodiments, each of the manipulator components of the robotic system (e.g., a first manipulator component, a second manipulator component, a third manipulator component) may include any number of manipulators. For example, a manipulator component may include a single manipulator, such as... Figure 2 As depicted in the example, multiple manipulators are mounted on a common physical base, such as Figure 1As illustrated in the examples, a manipulator assembly comprising multiple manipulators may have two, three, four, or more manipulators mounted to a common physical base. In a first example, registration is performed between manipulator assemblies, such as between manipulator assemblies where each manipulator assembly has one manipulator, between manipulator assemblies where each manipulator assembly has multiple manipulators mounted to a common base, or between manipulator assemblies where a first manipulator assembly has a single manipulator and a second manipulator assembly has multiple manipulators mounted to a common base. In this first example, registration between manipulator assemblies may be performed using an external environment detection sensor system or by any suitable technique, according to the techniques described in this disclosure. In a second example, registration is performed between manipulators of the same manipulator assembly that share a common base. In this second example, where the manipulators share a common physical base, registration between manipulators can be achieved through kinematic modeling with sufficient kinematic information about the manipulator configuration (such as information provided by sensors, such as shape sensors, connector sensors, etc.). Providing manipulator shape information or connector position information allows relative registration to be achieved using the kinematic information of two manipulators. Alternatively or additionally, registration may be performed using an external environment detection sensor system or by any suitable technique, based on the techniques described in this disclosure; this type of registration may be used to verify registration achieved by kinematics to provide an alternative registration technique in case the primary registration technique is insufficient (e.g., in the event of failure of one or more sensors used for the primary registration technique, or in the event that the sensor data used for the primary registration technique is too noisy).

[0110] In various embodiments, the registration process can be performed before, during, or after an operation (e.g., a medical procedure). In a medical example, the registration process can be performed outside the patient or inside the patient before the medical procedure (e.g., during setup). In another example, the registration process can be performed during the medical procedure. In yet another example, the registration process can be performed as a backup and / or calibration check registration method, where another registration process (e.g., a registration process based on the mounting position of the manipulator components) is the primary registration process. In yet another example, the registration process can be used in a robotic system with manipulators on the same pedestal to check and confirm the registration of those manipulators with their corresponding tools. In yet another example, the guiding mounting position of the manipulator components can be used to narrow down the search, provide initial guesses, and / or provide a confirmatory check for the registration process.

[0111] In this disclosure, the specific words chosen to describe one or more embodiments and optional elements or features are not intended to limit the invention. For example, spatially relative terms—such as “below,” “under,” “lower,” “above,” “upper,” “near,” “far”, etc.—are used to describe the relationship between one element or feature and another element or feature as shown in the figures. These spatially relative terms are intended to include different positions (i.e., translational positions) and orientations (i.e., rotational placements) of the device in use or operation, in addition to the positions and orientations shown in the figures. For example, if the device in the figures is flipped, then an element described as “below” or “under” other elements or features will be “above” or “on” other elements or features. Thus, the exemplary term “below” can include both above and below positions and orientations. The device may be oriented in other ways (e.g., rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein are therefore interpreted. Similarly, descriptions of movement along (translation) and about (rotation) various axes include various specific device positions and orientations. The combination of body position and orientation defines the body posture.

[0112] Similarly, geometric terms such as “parallel” and “perpendicular” are not intended to require absolute mathematical precision unless the context indicates otherwise. Instead, such geometric terms allow for variations due to manufacturing or equivalent functions.

[0113] Furthermore, unless the context otherwise indicates, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well. And the terms “comprises,” “comprising,” “includes,” “have,” etc., specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as connected may be directly connected electrically or mechanically, or they may be indirectly connected via one or more intermediate components. The auxiliary verb “may” similarly implies that a feature, step, operation, element, or component is optional.

[0114] Elements described in detail with reference to one embodiment, implementation, or application may optionally be included in other embodiments, implementations, or applications that are not specifically shown or described therein, provided that it is practicable. For example, if an element is described in detail with reference to one embodiment and not with reference to a second embodiment, that element may still be referred to as being included in the second embodiment. Therefore, to avoid unnecessary repetition in the following description, the illustration and description of one or more elements in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless specifically described otherwise, unless the one or more elements render the embodiment or implementation ineffective, or unless two or more of the elements provide conflicting functionality.

[0115] Any changes and further modifications to the described apparatus, instruments, and methods, as well as any further application of the principles of this disclosure, are fully contemplated as would normally occur to those skilled in the art to which this disclosure pertains. In particular, it is fully contemplated that features, components, and / or steps described with respect to one embodiment may be combined with features, components, and / or steps described in other embodiments of this disclosure. Furthermore, the dimensions provided herein are for specific examples, and it is contemplated that the concepts of this disclosure may be implemented using different sizes, dimensions, and / or scales. To avoid unnecessary descriptive repetition, one or more components or actions described according to one illustrative embodiment may be used or omitted in other illustrative embodiments as appropriate. For brevity, numerous restates of these combinations will not be described separately. For simplicity, in some cases, the same reference numerals are used throughout the drawings to refer to the same or similar parts.

[0116] Various instruments and parts thereof are described in their state in three-dimensional space. As used herein, the term “position” refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, and Z coordinates). As used herein, the term “orientation” refers to the rotational placement of an object or part of an object (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term “attitude” refers to the position of an object or part of an object in at least one translational degree of freedom, and the orientation of the object or part of the object in at least one rotational degree of freedom (up to six total degrees of freedom). As used herein, the term “shape” refers to a set of attitudes, positions, or orientations measured along an object.

[0117] While some of the examples described herein refer to surgical procedures or instruments, or medical procedures and medical devices, the disclosed techniques may optionally be applied to non-medical procedures and non-medical devices. For example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial use, general robotic use, and sensing or manipulating non-tissue artifacts. Other example applications involve cosmetic improvement, imaging of human or animal anatomy, collecting data from human or animal anatomy, and training medical or non-medical personnel. Additional example applications include performing procedures on tissue removed from human or animal anatomy (without returning the human or animal anatomy), and performing procedures on human or animal cadavers. Furthermore, these techniques can also be used in surgical and non-surgical medical treatment or diagnostic procedures.

[0118] Furthermore, although some of the examples presented in this disclosure discuss remotely operated robot systems or remotely operable systems, the disclosed techniques are also applicable in part or in whole to computer-aided systems that are directly and manually moved by an operator. A computer is a machine that performs mathematical or logical functions on input information according to programmed instructions to produce processed output information. A computer includes logic units that perform mathematical or logical functions, and memory that stores programming instructions, input information, and output information. The term "computer" and similar terms such as "processor," "controller," or "control system" are similar.

[0119] Although certain exemplary embodiments of the invention have been described and illustrated in the accompanying drawings, it should be understood that such embodiments are merely illustrative and not intended to limit the scope of the invention, and that embodiments of the invention are not limited to the specific constructions and arrangements shown and described, as various other modifications may be apparent to those skilled in the art.

Claims

1. A robotic system comprising: a first manipulator assembly comprising a plurality of first links physically coupled to a first base, the first manipulator assembly configured to support a first tool; a plurality of first sensors disposed on the first manipulator assembly and not disposed on the first tool; a second manipulator assembly comprising a plurality of second links physically coupled to a second base, the second base separately movable relative to the first base, wherein the first manipulator assembly and the second manipulator assembly are in an operating environment; and a processing unit comprising one or more processors, the processing unit configured to: receive first sensor data from the plurality of first sensors, wherein the first sensor data provides spatial information about the operating environment external to the first manipulator assembly, determine, using data comprising the first sensor data, a first spatial relationship of the second manipulator assembly relative to the first manipulator assembly, establish, based on the first spatial relationship, a first alignment relationship between the first manipulator assembly and the second manipulator assembly, and based on the first alignment relationship, command movement of the second manipulator assembly in response to a command from a first input device operable by an operator.

2. The robotic system of claim 1, wherein the first spatial relationship comprises a position of the second manipulator assembly relative to the first manipulator assembly.

3. The robotic system of claim 1, wherein the first spatial relationship comprises an orientation of the second manipulator assembly relative to the first manipulator assembly.

4. The robotic system of claim 1, wherein the first spatial relationship comprises a position and an orientation relative to the first manipulator assembly.

5. The robotic system of claim 1, wherein: the spatial information about the operating environment external to the first manipulator assembly comprises an orientation or a position of the second manipulator assembly relative to the first manipulator assembly.

6. The robotic system of claim 1, wherein the plurality of first sensors comprises one or more image sensors, and wherein the first sensor data comprises one or more images detected by the one or more image sensors.

7. The robotic system of claim 1, wherein the second manipulator assembly is configured to support a second tool, and wherein the first sensor data comprises data based on signals transmitted between the plurality of first sensors and a plurality of second sensors disposed on the second manipulator assembly and not disposed on the second tool.

8. The robotic system of any of claims 1-6, wherein the second manipulator assembly is configured to support a second tool, the robotic system further comprising: a plurality of second sensors disposed on the second manipulator assembly and not disposed on the second tool; wherein the processing unit is further configured to: ​ receive second sensor data from the plurality of second sensors, wherein the second sensor data provides spatial information about the operating environment external to the second manipulator assembly, wherein the data used to determine the first spatial relationship further comprises the second sensor data.

9. The robotic system of claim 8, wherein: the spatial information about the operating environment external to the second manipulator assembly comprises an orientation or a position of the first manipulator assembly relative to the second manipulator assembly.

10. The robotic system of claim 8, further comprising: a third component having a movable base distinct from the first manipulator assembly and the second manipulator assembly; wherein the spatial information about the operating environment external to the first manipulator assembly provided by the first sensor data comprises a spatial relationship of the third component relative to the first manipulator assembly, wherein the second sensor data provides information about a spatial relationship of the second manipulator assembly relative to the third component, and the processing unit is configured to: determine a second alignment relationship between the first manipulator assembly and the third component based on the first sensor data, determine a third alignment relationship between the second manipulator assembly and the third component based on the second sensor data, and determine the first alignment relationship using the second alignment relationship and the third alignment relationship.

11. The robotic system of claim 8, wherein the plurality of first sensors comprises sensors located on different links of the plurality of first links, and wherein the plurality of second sensors comprises sensors located on different links of the plurality of second links.

12. The robotic system of any of claims 1-7, further comprising: a third component having a movable base distinct from the first manipulator assembly and the second manipulator assembly; wherein the data further comprises third sensor data from a plurality of third sensors disposed on the third component, the third sensor data providing spatial information about the operating environment external to the third component, and wherein the processing unit is further configured to: determine the first alignment relationship further using the third sensor data.

13. The robotic system of claim 12, wherein the processing unit is configured to determine the first alignment relationship further using the third sensor data by: determining a second alignment relationship between the second manipulator assembly and the third component based on the third sensor data; determining a third alignment relationship of the first manipulator assembly relative to the third component based on the first sensor data or the third sensor data; and determining the first alignment relationship using the second alignment relationship and the third alignment relationship.

14. The robotic system of any of claims 1-6, further comprising: a third component having a movable base distinct from the first manipulator assembly and the second manipulator assembly; and wherein the data used to determine the first spatial relationship further comprises the second sensor data. ​ a plurality of second sensors disposed on the second manipulator assembly; wherein the data further comprises third sensor data from a plurality of third sensors disposed on the third component, the third sensor data providing spatial information about the operating environment external to the third component, wherein the processing unit is further configured to: receive second sensor data from the plurality of second sensors, wherein the second sensor data provides spatial information about the operating environment external to the second manipulator assembly, determine a second alignment relationship between the second manipulator assembly and the third component based on the second sensor data or the third sensor data; determine a third alignment relationship between the first manipulator assembly and the third component based on the third sensor data; and use the second alignment relationship and the third alignment relationship to determine the first alignment relationship.

15. The robotic system of claim 12, wherein the processing unit is configured to further use the third sensor data to determine the first alignment relationship by: in response to determining that the data without the third sensor data is insufficient to establish the first alignment relationship, further using the third sensor data to determine the first alignment relationship.

16. The robotic system of claim 12, further comprising: a second input device operable by the operator, wherein the third component comprises a third manipulator assembly, and wherein the processing unit is further configured to: establish a second alignment relationship between the first manipulator assembly and the third manipulator assembly, and based on the second alignment relationship, command motion of the third manipulator assembly in response to a command from the second input device.

17. The robotic system of any of claims 1-7, further comprising: a display configured to be viewed by the operator, wherein an operator reference frame is defined relative to the display or relative to the operator viewing the display, wherein the first alignment relationship comprises an alignment relationship between a field of view of an imaging device supported by the first manipulator assembly and an end effector of a tool supported by the second manipulator assembly, and wherein the processing unit is further configured to: determine a second alignment relationship between the first input device and the operator reference frame; further command the motion of the second manipulator assembly based on the second alignment relationship, wherein the command comprises a change in pose of the first input device.

18. The robotic system of any of claims 1-7, wherein the processing unit is further configured to: in response to determining that the data is insufficient to establish the first alignment relationship, perform an adjustment operation.

19. The robotic system of claim 18, wherein the adjustment operation comprises: providing an indication to an operator to add a sensor intermediary to the operating environment.

20. The robotic system of claim 18, wherein the adjustment operation comprises: ​ providing an indication to an operator to move the first manipulator assembly or the second manipulator assembly to improve sufficiency of the data.

21. The robotic system of claim 18, wherein the adjusting operation comprises: commanding movement of the first manipulator assembly or the second manipulator assembly to improve sufficiency of the data.

22. The robotic system of claim 18, wherein determining that the data is insufficient comprises determining that the plurality of first sensors are occluded from detecting the second manipulator assembly.

23. The robotic system of any of claims 1-7, wherein the first manipulator assembly comprises a jaw mechanism configured to clamp to an operating table such that a first portion of the first manipulator assembly is below a top of the operating table; wherein one or more of the plurality of first sensors are located in the first portion of the first manipulator assembly; and wherein the first manipulator assembly and the second manipulator assembly are configured to perform an operation on a patient on the operating table during a medical procedure.

24. The robotic system of any of claims 1-7, wherein the processing unit is further configured to: determine the first alignment relationship further based on at least one of: a first installation position of the first manipulator assembly and a second installation position of the second manipulator assembly.

25. The robotic system of claim 24, wherein the processing unit is further configured to: determine a suggested installation position based on a procedure to be performed by the first manipulator assembly and the second manipulator assembly; and provide the suggested installation position to a user of the robotic system; wherein the first installation position and the second installation position are determined based on the suggested installation position.

26. The robotic system of any of claims 1-6, further comprising: a plurality of second sensors disposed on the second manipulator assembly or a third component different from the first manipulator assembly and the second manipulator assembly, wherein the processing unit is further configured to: receive second sensor data from the plurality of second sensors, wherein the second sensor data provides spatial information about the operating environment external to the second manipulator assembly or external to the third component, and wherein the processing unit is configured to determine the first spatial relationship of the second manipulator assembly relative to the first manipulator assembly using data including the first sensor data by: in response to the first sensor data being sufficient to establish the first spatial relationship, using the first sensor data instead of the second sensor data, and in response to the first sensor data being insufficient to establish the first spatial relationship, using the first sensor data and the second sensor data.

27. A method of operating a robotic system before or after a medical operation, the robotic system comprising a first manipulator assembly in an operating environment, the first manipulator assembly configured to support a first tool, the method comprising: receiving first sensor data from a plurality of first sensors disposed on the first manipulator assembly and not on the first tool, the first sensor data providing spatial information about the operating environment external to the first manipulator assembly, wherein the first manipulator assembly includes a plurality of first links physically coupled to a first base, and wherein the operating environment includes a second manipulator assembly including a plurality of second links physically coupled to a second base, the second base being separately movable relative to the first base; determining a first spatial relationship between the first manipulator assembly and the second manipulator assembly using data including the first sensor data; establishing a first alignment relationship between the first manipulator assembly and the second manipulator assembly based on the first spatial relationship; and based on the first alignment relationship, commanding motion of the second manipulator assembly in response to commands from a first input device operated by an operator.

28. The method of claim 27, wherein the plurality of first sensors includes one or more image sensors, and wherein the first sensor data includes one or more images detected by the one or more image sensors.

29. The method of claim 27, wherein the second manipulator assembly is configured to support a second tool, and wherein the first sensor data includes data based on signals transmitted between the plurality of first sensors and a plurality of second sensors disposed on the second manipulator assembly and not on the second tool.

30. The method of claim 27 or 28, wherein the second manipulator assembly is configured to support a second tool, and wherein the method further comprises: receiving second sensor data from a plurality of second sensors disposed on the second manipulator assembly and not on the second tool, wherein the second sensor data provides spatial information about the operating environment external to the second manipulator assembly, wherein the data used to determine the first spatial relationship further includes the second sensor data.

31. The method of any of claims 27 to 29, further comprising: determining a second alignment relationship between the first manipulator assembly and a third component based on the first sensor data, wherein the third component includes a movable base different from the first manipulator assembly and the second manipulator assembly; determining a third alignment relationship between the second manipulator assembly and the third component based on second sensor data from a plurality of second sensors disposed on the second manipulator assembly; and determining the first alignment relationship using the second alignment relationship and the third alignment relationship.

32. The method of any of claims 27 to 29, further comprising: receiving third sensor data from a plurality of third sensors disposed on a third component, the third component having a movable base different from the first manipulator assembly and the second manipulator assembly; and further determining the first alignment relationship using the third sensor data.

33. The method of claim 32, wherein determining the first alignment relationship further using the third sensor data comprises: determining a second alignment relationship between the second manipulator assembly and the third component based on the third sensor data; determining a third alignment relationship of the first manipulator assembly relative to the third component based on the first sensor data or the third sensor data; and determining the first alignment relationship using the second alignment relationship and the third alignment relationship.

34. The method of any one of claims 27-29, wherein the second manipulator assembly is configured to support a second tool, the method further comprising: receiving third sensor data from a plurality of third sensors disposed on a third component, the third component having a movable base different from the first manipulator assembly and the second manipulator assembly; receiving second sensor data from a plurality of second sensors disposed on the second manipulator assembly and not disposed on the second tool, wherein the second sensor data provides spatial information about the operating environment external to the second manipulator assembly; determining a second alignment relationship between the second manipulator assembly and the third component based on the second sensor data or the third sensor data; determining a third alignment relationship between the first manipulator assembly and the third component based on the third sensor data; and determining the first alignment relationship using the second alignment relationship and the third alignment relationship.

35. The method of claim 32, wherein determining the first alignment relationship further using the third sensor data comprises: in response to determining that the data without the third sensor data is insufficient to establish the first alignment relationship, determining the first alignment relationship further using the third sensor data.

36. The method of claim 32, wherein the third component comprises a third manipulator assembly, the method further comprising: establishing a second alignment relationship between the first manipulator assembly and the third manipulator assembly, and in response to a command from a second input device operable by the operator, commanding motion of the third manipulator assembly based on the second alignment relationship.

37. The method of any one of claims 27-29, further comprising: determining whether the data is insufficient to establish the first alignment relationship; and in response to determining that the data is insufficient to establish the first alignment relationship, performing an adjustment operation.

38. The method of claim 37, wherein performing the adjustment operation comprises: providing an indication to an operator to add a sensor intermediary to the operating environment.

39. The method of claim 37, wherein performing the adjustment operation comprises: providing an indication to an operator to move the first manipulator assembly or the second manipulator assembly to improve sufficiency of the data.

40. The method of claim 37, wherein performing the adjustment operation comprises: ​ ​ ​ commanding movement of the first manipulator assembly or the second manipulator assembly to improve sufficiency of the data.

41. The method of claim 37, wherein determining whether the data is sufficient to establish the first alignment relationship comprises: determining that the data is insufficient to establish the first alignment relationship in response to determining that the plurality of first sensors are occluded from detecting the second manipulator assembly.

42. A non-transitory machine-readable medium comprising a plurality of machine- readable instructions that, when executed by one or more processors, are adapted to cause the one or more processors to perform a method comprising: receiving first sensor data from a plurality of first sensors disposed on a first manipulator assembly in an operating environment, the first sensor data providing spatial information about the operating environment external to the first manipulator assembly, wherein the first manipulator assembly comprises a plurality of first links physically coupled to a first base, wherein the operating environment comprises a second manipulator assembly comprising a plurality of second links physically coupled to a second base, the second base being separately movable relative to the first base; determining a first spatial relationship between the first manipulator assembly and the second manipulator assembly using data comprising the first sensor data; establishing a first alignment relationship between the first manipulator assembly and the second manipulator assembly based on the first spatial relationship; and in response to a command from a first input device operated by an operator, commanding movement of the second manipulator assembly based on the first alignment relationship.

43. The non-transitory machine-readable medium of claim 42, wherein the method further comprises: determining a second alignment relationship between the first manipulator assembly and a third component based on the first sensor data, wherein the third component comprises a movable base different from the first manipulator assembly and the second manipulator assembly; determining a third alignment relationship between the second manipulator assembly and the third component based on second sensor data from a plurality of second sensors disposed on the second manipulator assembly; and determining the first alignment relationship using the second alignment relationship and the third alignment relationship.

44. The non-transitory machine-readable medium of claim 43, wherein the third component comprises a third manipulator assembly, and wherein the method further comprises: establishing a second alignment relationship between the first manipulator assembly and the third manipulator assembly; and in response to a command from a second input device operable by the operator, commanding movement of the third manipulator assembly based on the second alignment relationship.

45. The non-transitory machine-readable medium of claim 42 or 43, wherein the method further comprises: determining whether the data is insufficient to establish the first alignment relationship; and in response to determining that the data is insufficient to establish the first alignment relationship, performing an adjustment operation.

46. The non-transitory machine-readable medium of claim 45, wherein performing the adjustment operation comprises: providing an indication to an operator to add a sensor mediator to the operating environment.

47. The non-transitory machine-readable medium of claim 45, wherein performing the adjustment operation comprises: providing an indication to an operator to move the first manipulator assembly or the second manipulator assembly to improve sufficiency of the data.

48. The non-transitory machine-readable medium of claim 45, wherein performing the adjustment operation comprises: commanding movement of the first manipulator assembly or the second manipulator assembly to improve sufficiency of the data.

49. The non-transitory machine-readable medium of claim 45, wherein determining whether the data is sufficient to establish the first alignment relationship comprises: determining that the data is insufficient to establish the first alignment relationship in response to determining that the plurality of first sensors are occluded from detecting the second manipulator assembly.

Images