Techniques for updating a shared extended reality anchor based on operator interactions with a computer-assisted device

Abstract

Techniques for updating an anchor within an extended reality (XR) environment include the following. A control system communicably coupled to a computer-assisted device and to a sensor system is configured to: determine (based on sensor data) pose information of an operator portion of a first operator during one or more interactions between the operator portion and a device portion of the computer-assisted device, determine pose information of the device portion during the one or more interactions, update (based on the pose information of the operator portion and the pose information of the device portion) an anchor that is shared by a first XR system and a second XR system, and cause the second XR system to render an image viewable by the second operator using the updated anchor.

Description

TECHNIQUES FOR UPDATING A SHARED EXTENDED REALITY ANCHOR BASED ON OPERATOR INTERACTIONS WITH A COMPUTER-ASSISTED DEVICECROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 620,586, filed January 12, 2024, and entitled “Techniques for Updating a Shared Extended Reality Anchor based on Operator Interactions with a Computer-Assisted Device.” The subject matter of this application is incorporated by reference herein.TECHNICAL FIELD

[0002] The present disclosure relates generally to electronic devices and more particularly relates to techniques for updating a shared extended reality anchor based on operator interactions with a computer-assisted device.BACKGROUND

[0003] Computer-assisted electronic systems are being used more and more often. This is especially true in industrial, entertainment, educational, and other settings. As a medical example, the medical facilities of today have large arrays of computer-assisted devices being found in operating rooms, interventional suites, intensive care wards, emergency rooms, and / or the like. Many of these computer-assisted devices are moveable, and may be capable of autonomous or semi-autonomous motion. It is also known for personnel to control the motion and / or operation of moveable computer-assisted devices using one or more input devices located at a user control system. As a specific example of electronic systems comprising computer-assisted devices, minimally invasive, robotic telesurgical devices permit surgeons to operate on patients from bedside or remote locations. Telesurgery refers generally to surgery performed using surgical devices where the surgeon uses some form of remote control, such as a servomechanism, to manipulate surgical instrument movements rather than directly holding and moving the instruments by hand.

[0004] Extended reality (XR) systems are oftentimes used in conjunction with computer- assisted devices to perform tasks at worksites. Examples of XR systems include augmented reality (AR) devices and virtual reality (VR) devices. As used herein, AR refers to a view of the physical environment with an overlay of one or more computer-generated graphical elements, including mixed reality (MR) environments in which physical objects and computer- Attorney Docket No. P06828-WO:0138PCgenerated elements can interact. As used herein, VR refers to a virtual environment that includes computer-generated elements. For example, an XR system can present data about an operating environment of a computer-assisted device or guidance during operation of the computer-assisted device, among other things.

[0005] Conventional methods for determining registrations for XR systems can be less accurate initially and / or drift over time. As a result, the XR systems may display XR content at less accurate positions and / or orientations within XR environments.

[0006] Accordingly, improved techniques for registering XR systems with anchors in XR environments are desirable.SUMMARY

[0007] Consistent with some embodiments, an electronic system includes a computer- assisted device, and a control system communicably coupled to the computer-assisted device and to a sensor system, the sensor system configured to capture sensor data about an environment external to the computer-assisted device. The control system is configured to: determine, based on the sensor data, pose information of an operator portion of a first operator during one or more interactions between the operator portion and a device portion of the computer-assisted device, the pose information of the operator portion comprising at least one parameter selected from the group consisting of: a position of the operator portion during the one or more interactions and an orientation of the operator portion during the one or more interactions, determine pose information of the device portion during the one or more interactions, the pose information of the device portion comprising at least one parameter selected from the group consisting of: a position of the device portion during the one or more interactions and an orientation of the device portion during the one or more interactions, update, based on the pose information of the operator portion and the pose information of the device portion, an anchor that is shared by a first extended reality (XR) system and a second XR system, the first XR system configured to render images for the first operator and the second XR system configured to render images for a second operator, and cause the second XR system to render an image viewable by the second operator using the updated anchor.

[0008] Consistent with some embodiments, a method of operating an electronic system including a computer-assisted device and a control system communicably coupled to the computer-assisted device and to a sensor system, the sensor system configured to capture sensor data about an environment external to the computer-assisted device, and the methodAttorney Docket No. P06828-WO:0138PCincludes: determining, based on the sensor data, pose information of an operator portion of a first operator during one or more interactions between the operator portion and a device portion of the computer-assisted device, the pose information of the operator portion comprising at least one parameter selected from the group consisting of: a position of the operator portion during the one or more interactions and an orientation of the operator portion during the one or more interactions, determining pose information of the device portion during the one or more interactions, the pose information of the device portion comprising at least one parameter selected from the group consisting of: a position of the device portion during the one or more interactions and an orientation of the device portion during the one or more interactions, updating, based on the pose information of the operator portion and the pose information of the device portion, an anchor that is shared by a first extended reality (XR) system and a second XR system, the first XR system configured to render images for the first operator and the second XR system configured to render images for a second operator, and causing the second XR system to render an image viewable by the second operator using the updated anchor.

[0009] Other embodiments include, without limitation, one or more non-transitory machine-readable media including a plurality of machine-readable instructions, which when executed by one or more processors, are adapted to cause the one or more processors to perform any of the methods disclosed herein.

[00010] The foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a simplified diagram including an example of a computer-assisted device and extended reality (XR) systems, according to various embodiments.

[0011] Figure 2 is a perspective view illustrating an XR system, according to various embodiments.

[0012] Figure 3 illustrates in greater detail the control module of Figure 1, according to various embodiments.Attorney Docket No. P06828-WO:0138PC

[0013] Figure 4 illustrates a simplified diagram of a method for updating an anchor based on one or more operator interactions, according to various embodiments.

[0014] Figure 5 illustrates in greater detail the process of updating the anchor in the method of Figure 4, according to various embodiments.

[0015] Figure 6 illustrates a simplified diagram of a method for updating anchors associated with different computer-assisted devices, according to other embodiments.

[0016] Figure 7 illustrates a simplified diagram of a method for updating anchors associated with different computer-assisted devices, according to various other embodiments.

[0017] Figure 8 illustrates an example of updating a shared anchor, according to various embodiments.

[0018] Figure 9 illustrates an example of updating anchors associated with different computer-assisted devices, according to various embodiments.

[0019] Figure 10 illustrates an example of updating a registration transform between anchors associated with different computer-assisted devices, according to various embodiments.DETAILED DESCRIPTION

[0020] This description and the accompanying drawings that illustrate inventive aspects, embodiments, embodiments, or modules should not be taken as limiting—the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure the invention. Like numbers in two or more figures represent the same or similar elements.

[0021] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid Attorney Docket No. P06828-WO:0138PCunnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

[0022] Further, the terminology in this description is not intended to limit the invention. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like-may be used to describe one element’s or feature’s relationship to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. And, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0023] Elements described in detail with reference to one embodiment, embodiment, or module may, whenever practical, be included in other embodiments, embodiments, or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, embodiment, or application may be incorporated into other embodiments, embodiments, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or embodiment non-functional, or unless two or more of the elements provide conflicting functions.

[0024] In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.Attorney Docket No. P06828-WO:0138PC

[0025] This disclosure describes various elements (such as systems and devices, and portions of systems and devices) with examples in three-dimensional space. In such examples, the term “position” refers to the location of an element or a portion of an element in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates). Also in such examples, the term “orientation” refers to the rotational placement of an element or a portion of an element (three degrees of rotational freedom – e.g., roll, pitch, and yaw). Other examples may encompass other dimensional spaces, such as two-dimensional spaces. As used herein, the term “pose” refers to the position, the orientation, or the position and the orientation combined, of an element or a portion of an element. As used herein, and for an element or portion of an element, e.g. a device (e.g., a computer-assisted device or a repositionable arm), the term “proximal” for elements in a kinematic chain refers to a direction toward the base of the kinematic chain, and the term “distal” refers to a direction away from the base along the kinematic chain.

[0026] Aspects of this disclosure are described in reference to electronic systems and computer-assisted devices, which may include systems and devices that are teleoperated, remote-controlled, autonomous, semiautonomous, robotic, and / or the like. Further, aspects of this disclosure are described in terms of an embodiment using a medical system, such as the da Vinci® Surgical System commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including robotic and, if applicable, non-robotic embodiments. Embodiments described for da Vinci® Surgical Systems are merely exemplary, and are not to be considered as limiting the scope of the inventive aspects disclosed herein. For example, techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperational systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like. Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can alsoAttorney Docket No. P06828-WO:0138PCbe used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.System Overview

[0027] Figure 1 is a simplified diagram of an electronic system 100, according to various embodiments. As shown, electronic system 100 includes, without limitation, a computer-assisted device 110 and extended reality (XR) systems 130 and 140. Computer-assisted device 110 includes a repositionable structure, the repositionable structure having a repositionable arm 120, such as a manipulator arm, configured to support an instrument 122. Although one repositionable arm 120 supporting one instrument 122 is shown for illustrative purposes, the repositionable structure of a computer-assisted device can include any suitable number of repositionable arms, and each repositionable arm can be configured to support one instrument, or a plurality of instruments.

[0028] In some examples, instrument 122 is an imaging instrument, a manipulation instrument such as graspers or scissors, a fastening instrument such as a stapler, an irrigation instrument, a suction instrument, an energy application instrument, or any other appropriate instrument. In some examples, instrument 122 is an imaging instrument such as a monoscopic or stereoscopic camera, a still or video camera, an endoscope, a hyperspectral device, an infrared or ultrasonic device, an ultrasonic device, a fluoroscopic device, and / or the like. In some examples, instrument 122 is a medical instrument, such as a medical endoscope, forceps, clip appliers, a gripper, a retractor, a cautery instrument, a suction instrument, a suturing device, a stapling device, a cutting device, and / or the like. In some examples, instrument 122 includes an end effector capable of performing one or multiple tasks, such as grasping a material (e.g., tissue of a patient in a medical example) located in a workspace and delivering energy to the grasped material. In some examples, the energy includes ultrasonic, radio frequency, electrical, magnetic, thermal, light, and / or other types of energy. In some examples, the repositionable arm 120 includes one or more joints and the instrument 122. In some examples, instrument 122 is, during use, inserted into a workspace (e.g., anatomy of a patient or cadaver, a veterinary subject, an anatomical model, and / or the like in some medical examples) through a cannula, access port, and / or the like to perform a procedure.

[0029] In some examples, computer-assisted device 110 is a teleoperated device. In some medical examples, the teleoperated device is a teleoperated medical device, such as a telesurgical device, that can be found in an operating room and / or an interventional suite. In Attorney Docket No. P06828-WO:0138PCsome examples, computer-assisted device 110 is a follower device that is teleoperated by being controlled by one or more leader devices (not shown), such as one or more input devices designed to be contacted and manipulated by an operator (not shown). The one or more input devices may be mechanically grounded (kinematically grounded by mechanical structures) or mechanically ungrounded (not kinematically grounded by mechanical structures). Systems that include a leader device and a follower device are referred to as leader-follower systems, and also sometimes referred to as master-slave systems. When computer-assisted device 110 is a teleoperated follower device comprising the repositionable arm 120 and / or an instrument 122 supported by the repositionable arm and is controlled to move and articulate in response to manipulation of leader device(s) by an operator, and computer-assisted device 110 “follows” through teleoperation the leader input device(s). The operator is thus able to perform tasks at a worksite using the repositionable arm 120 and / or instrument 122.

[0030] In some embodiments, each of XR systems 130 and 140 is an augmented reality (AR) device or a virtual reality (VR) device. XR system 130 is described in greater detail below in conjunction with Figure 2. XR system 140 can include similar components as XR system 130. Although two XR systems 130 and 140 are shown for illustrative purposes, any number of XR systems can be used in some embodiments. In some examples, XR systems 130 and 140 are used in conjunction with computer-assisted device 110 to perform tasks at a worksite. For example, XR systems 130 and 140 could be used to present instructional content on how to operate computer-assisted device 110. As another example, XR systems 130 and 140 could be used to present content that provides guidance during operation of computer- assisted device 110. Illustratively, XR system 130 has displayed AR content 136 next to repositionable arm 120 of computer-assisted device 110. AR content 136 can be provided in any manner appropriate for visual AR content, such as a visual overlay. In order for XR content (e.g., AR content 136) that is presented by XR systems 130 and 140 to be displayed at appropriate positions and orientations relative to computer-assisted device 110, the XR content can be attached to an anchor 124. For clarity, this disclosure describes XR objects, reference frames, and other virtual elements defined in a static or quasi-static way relative to an anchor as “attached” to that anchor. Anchor 124 is defined by a pose that includes a position and an orientation relative to a world coordinate frame. Illustratively, anchor 124 is attached to a base of computer-assisted device 110. In addition, a reference frame 126 is defined for anchor 124. In some examples, reference frame 126 has an origin located at the position of anchor 124 and is oriented based on the orientation of anchor 124 in any technically feasible manner. OnceAttorney Docket No. P06828-WO:0138PCcreated, anchor 124 can be continuously tracked in any technically feasible manner, such as using simultaneous localization and mapping (SLAM). In some embodiments, anchor 124 remains static within the XR environment, and each XR system 130 or 140 stores metadata related to anchor 124 locally. In such cases, when XR system 130 or 140 leaves and then returns back to the same XR environment, XR system 130 or 140 can pick up the world anchor based on the metadata and recognizing that XR system 130 or 140 has been in the XR environment before.

[0031] XR objects that are attached to the anchor 124 appear to stay in place within the XR environment. Registration establishes a geometric relationship between part or all of an XR system and anchor 124. Illustratively, a registration transform 134 between a reference frame 132 associated with XR system 130 and reference frame 126 of anchor 124 is established. In addition, a registration transform 144 between a reference frame 142 associated with XR system 140 and reference frame 126 of anchor 124 is established. Using registration transforms 134 and 144, XR content to be displayed relative to a portion of computer-assisted device 110 is displayed at or near positions in reference frame 132 of XR system 130 and reference frame 142 of XR system 140, respectively, that are determined from a position of the portion of computer-assisted device 110 in reference frame 126 of anchor 124. For example, to display AR content 136, registration transform 134 is used to map a position (and optionally an orientation) of the portion of computer-assisted device 110 in reference frame 126 of anchor 124 to a corresponding position (and optionally orientation) in reference frame 132 of XR system 130. AR content 136 can be displayed near the corresponding position of the portion of the repositionable arm 120 in reference frame 132. The position of repositionable arm 120 in reference frame 126 of anchor 124 is determined using kinematics or in any other technically feasible manner (e.g., using sensors such as a gyroscope, an IMU, shape sensors, etc. that track the position of repositionable arm 120).

[0032] In some examples, when anchor 124 is positioned at a base of computer-assisted device 110, reference frame 126 of anchor 124 has an origin located at a central point on the base of computer-assisted device 110 and is aligned with one or more major axes of computer-assisted device 110. In some examples, a central point on the base of computer-assisted device 110 is on a level, planar surface on which computer-assisted device 110 is wheeled, slid, and / or otherwise repositioned. In some examples, a z-axis of reference frame 126 corresponds to a vertical up direction. As shown, reference frame 132 is a base reference frame of XR system 130. In some examples, reference frame 132 has an origin located at a point within XR Attorney Docket No. P06828-WO:0138PCsystem 130 and is oriented in any technically feasible manner. In some examples, a z-axis of reference frame 132 corresponds to a direction of view of XR system 130, a y-axis of reference frame 132 corresponds to the view up direction, and an x-axis of reference frame 132 is orthogonal to both the z- and y-axes of reference frame 132. Similarly, reference frame 142 is a base reference frame of XR system 140. In some examples, reference frame 142 has an origin located at a point within XR system 140 and is oriented in any technically feasible manner. In some examples, a z-axis of reference frame 142 corresponds to a direction of view of XR system 140, a y-axis of reference frame 142 corresponds to the view up direction, and an x-axis of reference frame 142 is orthogonal to both the z- and y-axes of reference frame 142.

[0033] In some examples, each of registration transforms 134 and 144 is a 6 degrees of freedom (6 DOF) registration transform. For example, the registration transforms 134 and 144 can each include a 3 degrees of freedom (DOF) rotation portion and a 3 DOF translation portion. In such cases, the rotation portion can be represented with a 3x3 matrix describing the 3-dimensional rotational difference between reference frame 126 and reference frame 132 or 142. In some examples, the rotation portion describes rotation about the three axes of reference frame 126. In some examples, the three axes correspond to x, y, and z axes; roll, pitch, and yaw axes, and / or the like. In some examples, the rotation portion is represented using angle-axis, quaternion, and / or similar equivalent notations. In some examples, the translation portion can be represented by a 3x1 vector describing the 3-dimensional displacement between reference frame 126 and reference frame 132 or 142.

[0034] As shown, a control system 150 is provided external to computer-assisted device 110 and XR system 130, and control system 150 communicates with both computer-assisted device 110 and XR system 130. In other embodiments, control system 150 is part of a computer-assisted device 110 or in XR system 130 and / or XR system 140. In some embodiments, control system 150 generates XR content that is presented by XR system 130 and / or XR system 140. In some embodiments, control system 150 also determines or provides control signals to computer-assisted device 110 to control movement of repositionable arm 120 and / or instrument 122 based on the received information and operator input. In some embodiments, control system 150 supports one or more wired communication protocols, (e.g., Ethernet, USB, and / or the like) and / or one or more wireless communication protocols (e.g., Bluetooth, IrDA, HomeRF, IEEE 1002.11, DECT, Wireless Telemetry, and / or the like).Attorney Docket No. P06828-WO:0138PC

[0035] Control system 150 is implemented on one or more computing systems. As shown, control system 150 includes processor(s) 160 and a memory 170 storing a control module 180. In some embodiments, control system 150 includes one or more processors 160 that execute the control module 180, non-persistent storage (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage (e.g., a hard disk, an optical drive such as a compact disk (CD) drive or digital versatile disk (DVD) drive, a flash memory, etc.), a communication interface (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and numerous other elements and functionalities. Although described herein primarily with respect to control module 180 that runs on control system 150, the functionality of control module 180 can be implemented in any technically feasible software and / or hardware, including partially or entirely within XR system 130, within another computing system, within a cloud computing system, etc.

[0036] Each of processor(s) 160 of control system 150 is an integrated circuit for processing instructions. For example, the one or more processors can be one or more cores or micro-cores of a processor, a central processing unit (CPU), a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), and / or the like. In some examples, control system 150 also includes one or more input devices (not shown), such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, or any other type of input device.

[0037] A communication interface of control system 150 can include an integrated circuit for connecting the computing system to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) and / or to another device, such as another computing system.

[0038] In some examples, control system 150 further includes one or more output devices (not shown), such as a display device (e.g., a liquid crystal display (LCD), a plasma display, touchscreen, organic LED display (OLED), projector, or other display device), a printer, a speaker, external storage, or any other output device. One or more of the output devices can be the same or different from the input device(s). Many different types of computing systems exist, and the aforementioned input and output device(s) can take other forms.

[0039] In some embodiments, control system 150 is connected to or is part of a network. The network can include multiple nodes. Control system 150 is implemented on one node orAttorney Docket No. P06828-WO:0138PCon a group of nodes. By way of example, control system 150 is implemented on a node of a distributed system that is connected to other nodes. By way of another example, control system 150 is implemented on a distributed computing system having multiple nodes, where different functions and / or components of control system 150 are located on a different node within the distributed computing system. Further, one or more elements of the aforementioned control system 150 can be located at a remote location and connected to the other elements over a network.

[0040] Software instructions in the form of computer readable program code to perform embodiments of the disclosure is stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable medium such as a CD, DVD, storage device, a diskette, a tape, flash memory, physical memory, or any other computer readable storage medium.Specifically, the software instructions can correspond to computer readable program code that, when executed by a processor(s) (e.g., processor 160), is configured to perform some embodiments of the methods described herein.

[0041] Some embodiments include one or more components of a teleoperated medical system such as a da Vinci® Surgical System, commercialized by Intuitive Surgical, Inc. of Sunnyvale, California, U.S.A. Embodiments on da Vinci® Surgical Systems are merely examples and are not to be considered as limiting the scope of the features disclosed herein. For example, different types of teleoperated systems having computer-assisted devices comprising follower devices configured to be placed at worksites can use the features described herein. Further, non-teleoperated systems can also make use of features described herein.

[0042] Figure 2 is a perspective view illustrating head-mounted XR system 130 in greater detail, according to various embodiments. In some embodiments, XR system 140 includes similar components as XR system 140. As shown in Figure 2, XR system 130 includes a body 205 and a head mount 210. Body 205 includes one or more electronic display elements of an electronic display 230. Body 205 also includes a sensor system 240 that acquires sensor data associated with the physical environment external to XR system 130, which may also be external to any objects (such as computer-assisted devices) in the physical environment.Sensor system 240 can include any technically feasible sensor or sensors, such monoscopic and stereoscopic optical systems, ultrasonic systems, depth cameras such as cameras using time-of-flight sensors, LIDAR (Light Detection and Ranging) sensors, stereo RGB (red, green, Attorney Docket No. P06828-WO:0138PCblue) sensors, RGB-D depth-sensors, etc. Although Figure 2 shows a head-mounted XR system, other XR systems may be used in other embodiments. Examples of other types of XR systems include appropriately configured tablets, smart-phones, projectors, etc. Although sensor system 240 is shown in Figure 2 as included in XR system 130, a sensor system used to provide sensor data associated with the physical environment external to the XR system 130 (which can also include data external to one or more objects in the physical environment) can be provided in any appropriate location. For example, part or all of such a sensor system can be alternatively or additionally located elsewhere in the physical environment, including mounted on walls, ceilings, or stands, or coupled to a computer-assisted device.

[0043] In addition, body 205 includes electronic display 230 and an optics block 235 that together provide image light to a target location of body 205 where an eye 202 of an operator may be positioned. In some examples, body 205 also includes one or more other sensors, such as one or more imaging devices (e.g., one or more imaging sensors for tracking eye 220), accelerometers, and / or angular velocity sensors (which may be part of inertial measurement units (IMUs)), position sensors, and / or other sensors.

[0044] Electronic display 230 is configured to display images viewable by the operator. In various embodiments, electronic display 230 includes a single electronic display or multiple electronic displays (e.g., a display for each eye of an operator). Examples of the electronic display 230 include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a QOLED, a QLED, some other display, or some combination thereof. Optics block 235 includes optical elements that can be used to adjust an orientation of image light emitted from electronic display 230 such that electronic display 230 appears at particular virtual image distances from the operator.

[0045] In some embodiments, XR system 130 operates as an AR device that presents computer-generated media to an operator using electronic display 230 that augments views of a physical, real-world environment visible to the operator through electronic display 230. Examples of computer-generated media presented by XR system 130 include one or more images, video, audio, or some combination thereof. Alternatively or additionally, in some embodiments, XR system 130 operates as a VR device, or some combination of an AR device and a VR device, such as a device that permits switching between AR and VR environments. As described, MR content provides an AR environment in which physical objects andAttorney Docket No. P06828-WO:0138PCcomputer-generated elements can interact, and VR content provides a virtual environment that includes computer-generated elements. In the case of a VR device, sensor system 240 can capture images of the physical environment and display the captured images along with computer-generated elements , such as AR content 136, which is also sometimes referred to as video see through. Examples of commercially available AR devices include Microsoft HoloLens®, Google Glass®, and Meta 2®. Examples of commercially available MR devices include Microsoft HoloLens 2®, Samsung Odyssey+®, HP Reverb®, and Oculus Quest 2®. Examples of commercially available VR devices include Oculus Rift®, Samsung Gear VR®, HTC Vive®, and Google Daydream View®.

[0046] It should be noted that Figure 2 merely shows an example configuration of an XR system. In some embodiments, the techniques for updating a registration transform between an XR system and a computer-assisted device that are disclosed herein are usable with other configurations and / or types of XR systems. The other configurations and / or types of XR systems include head-mounted XR systems, hand-held XR systems, and XR systems that are placed in the environment, among other things. Examples of alternative configurations and / or types of XR systems include optical head-mounted displays (HMDs), mobile devices (e.g., mobile phones, tablet computers, etc.), fully immersive projection systems, etc.Updating an Anchor Shared by Multiple Extended Reality Systems

[0047] An anchor shared by multiple XR systems can be updated based on a determined relationship between pose information of a portion of an operator and pose information of a portion of a computer-assisted device during an interaction between the portion of the operator, who is using one of the multiple XR systems, and the portion of the computer-assisted device.

[0048] Figure 3 illustrates in greater detail the control module 180 of Figure 1, according to various embodiments. As shown, control module 180 includes, without limitation, a sensor data processing module 306, a kinematics evaluation module 308, a registration module 310, and an overlay module 312. During operation, sensor data processing module 306 receives sensor data 302 that is acquired by a sensor system (e.g., sensor system 240) and determines pose information of a portion of an operator that interacts with a portion of a computer-assisted device (e.g., computer-assisted device 110) during the operator interaction. In some embodiments, the pose information of the operator includes one or more position parameters, or one or more position parameters and one or more orientation parameters. In someAttorney Docket No. P06828-WO:0138PCembodiments, the pose information of the portion of the operator is determined using a human pose estimation technique. Alternatively or additionally, sensor data processing module 306 can determine pose information of an extension of a portion of the operator, such as an object of known dimension (e.g., a cable, tool, etc.) that the operator is holding.

[0049] During operation, pose evaluation module 308 receives kinematic data 304 associated with the joints and / or links of a repositionable structure of a computer-assisted device, such as computer-assisted device 110, and pose evaluation module 308 determines pose information of a portion of the computer-assisted. In some embodiments, the pose information of the portion of the computer-assisted device includes one or more position parameters, or one or more position parameters and one or more orientation parameters. This pose information of the portion of the repositionable structure is during the interaction between the portion of the operator and the portion of the repositionable structure. In some embodiments, kinematic data 304 can be acquired by joint sensors that transmit positions and orientations of joints of the repositionable structure to record movements thereof, shape sensors that monitor the shape of an optical fiber to determine the pose of the repositionable structure located at one end of the optical fiber relative to a frame of reference located at the other end of the fiber, and / or in any other technically feasible manner. When a portion of the operator (an “operator portion”) interacts with a portion of a repositionable structure of computer-assisted device 110 (a “device portion”), pose evaluation module 308 uses kinematic data 304 and one or more kinematic models and / or three-dimensional (3D) models of the repositionable structure to determine pose information of the portion of the repositionable structure. In some embodiments, kinematic data 304 is synchronized with sensor data 302 so that comparisons can be made between pose information that is determined using the sensor data 302 and pose information that is determined using the kinematic data 304 and corresponds to the same point(s) in time. Although discussed herein primarily with respect to examples in which the portion of the computer-assisted device is a portion of a repositionable structure, in some embodiments, techniques disclosed herein can be applied to cases where the portion of the computer-assisted device is another portion of the computer-assisted device or is a portion of a computer-assisted device that does not include a repositionable structure. When a portion of the operator interacts with a portion of a computer-assisted device other than a repositionable structure (e.g., a helm, handlebars, a base, etc. of a computer-assisted device), pose information of the portion of the computer-assisted device can Attorney Docket No. P06828-WO:0138PCbe determined based on a pose of the overall computer-assisted device and a known position and orientation of the portion relative to the overall computer-assisted device.

[0050] During operation, registration module 310 determines registration transforms between reference frames associated with XR systems (e.g., reference frames 132 and 142) and reference frame 126 of anchor 124. Internal transforms between portions of the computer-assisted device 110 can also be determined using the known or sensed geometries of the computer-assisted device 110. Thus, when a registration transform is between XR system 130 or 140 and anchor 124 that is fixed to a part of computer-assisted device 110 is known, transforms defining the relative positions and / or orientations can be determined between XR system 130 or 140 and other part(s) of computer-assisted device 110 using these internal transforms. In some embodiments, registration module 310 updates current registration transforms between reference frames associated with XR systems (e.g., XR system 130 and 140) that share anchor 124 and reference frame 126 of anchor 124 based on operator interactions with computer-assisted device 110. In some embodiments, registration module 310 (i) determines initial registration transforms according to appropriate technique(s), and (ii) updates the current registration transforms (which would initially be the initial registration transforms) based on one or more operator interactions with computer-assisted device 110, as discussed in greater detail below in conjunction with Figures 4-7.

[0051] In some examples, determining the initial registration transform between a reference frame associated with an XR system (e.g., XR system 130 or 140) and reference frame 126 of anchor 124 includes prompting an operator to look at, touch, and / or move one or more portions of computer-assisted device 110. As used herein, a prompt can include any suitable visual and / or audio indication of operator action(s) to perform, such as a graphic, textual message, flashing light, audio instruction, etc. For example, the operator can be prompted to look at and / or interact with a particular portion of computer-assisted device 110. As another example, the operator can be prompted to perform a set of interactions and / or a complex operator interaction with a portion of computer-assisted device 110, such as moving a manipulator through a range of movements, while looking at the portion of computer-assisted device 110. Imaging devices in a sensor system of the XR system capture images of the one or more portions of computer-assisted device 110 as the operator looks, touches, and / or moves the one or more portions. The initial registration transform is determined by computing initial value(s) for the registration transform between the reference frame associated with the XR system and reference frame 126 of anchor 124 based on the position and orientation ofAttorney Docket No. P06828-WO:0138PCcomputer-assisted device 110, which can be determined using a model target-based technique in some examples. For example, the model target-based technique can use a photogrammetry technique to reconstruct a 3D model of computer-assisted device 110 from image data, and then match the reconstructed model to a predefined model of computer-assisted device 110 in order to determine the position and orientation of computer-assisted device 110. In some other examples, determining the initial registration transform includes capturing images of one or more portions of the computer-assisted device on which fiducial markers (e.g., QR codes, images, markers that encode data, etc.) have been placed. In such cases, the initial registration transform can be computed partially or entirely based on the positions and / or orientations of the fiducial markers in the captured images. In some examples, the computer vision-based and marker-based registration techniques described above can be combined to determine an initial registration transform. In some examples, any other technically feasible technique can be used to determine an initial registration transform.

[0052] An initial registration transform determined using model target-based, marker- based, and / or other registration techniques can be inaccurate. What is considered inaccurate can depend on the type of XR content to be displayed, the operational context of the computer-assisted device, and / or the operational context of the XR system. For example, for typical human interactions, an operator may perceive XR content that is overlaid to be aligned linearly or angularly particular features (such as when an overlay depicting a portion of a computer-assisted device is overlaid directly on that portion of the computer-assisted device) to be displayed with inaccurate registration when the XR content is offset from a desired position by approximately 1 centimeter or more and / or 5 degrees or more in any axis of rotation at an effective distance of 1 meter away from the eyes of the operator. As a further example, XR content overlaid for eye surgery or fine crafting can be considered to be displayed with inaccurate registration at much smaller deviations. As another example, an operator may perceive “floating” XR content that is not overlaid to be aligned over any specific object or feature of the object (such as offset on the side of a computer-assisted device) to be displayed with accurate registration even if the XR content is misaligned angularly or linearly by more than a few centimeters at an effective distance of 1 meter away from the operator (assuming that the XR content does not impinge into the visual space of the computer-assisted device). As another example, operators can consider XR content to be inaccurately registered at smaller deviations for head-mounted devices than handheld devices (e.g., handheld screens or tablets). Further, initial registrations that are accurate initially can become more inaccurate Attorney Docket No. P06828-WO:0138PCover time due to drift, movement, sensor inaccuracies, etc. When the initial registration transform is inaccurate, the XR system can display XR content at positions and / or orientations relative to anchor 124 that are perceived as incorrect. To improve the registration accuracy, registration module 310 updates the initial registration transform, as well as anchor 124 and other registration transforms between reference frames of other XR systems and anchor 124, based on operator interactions with computer-assisted device 110.

[0053] When one or more portions of the operator interact with one or more portions of a repositionable structure of computer-assisted device 110, one or more points and / or regions of interaction are determined in the reference frame of the XR system (e.g., reference frame 132 or 142). Each point or region of interaction can be an actual point or region of contact between a portion of the operator or extension thereof (gloves, shoes, cable, pointer stick, etc.) and a portion of the computer-assisted device, or an estimate thereof. Each point or region of interaction is determined in both the reference frame of the XR system and reference frame 126 of anchor 124. A point or region of interaction can be determined in the reference frame of the XR system using a human pose estimation technique, described in greater detail below, and sensor data that is acquired by a sensor system (e.g., sensor system 240) of the XR system. A point or region of interaction can be determined in reference frame 126 of anchor 124 either (1) using kinematics and known joint locations of a repositionable structure of computer-assisted device 110 when the operator interacts with a portion of the repositionable structure, or (2) based on a pose of some other portion of the computer-assisted device or of the overall computer-assisted device 110, and a known position and orientation of the portion relative to the overall computer-assisted device 110, when the operators interacts with a portion of the computer-assisted device 110 other than the repositionable structure. Alternatively, point(s) and / or region(s) of interaction with the repositionable structure can be determined using any technically feasible techniques, including non-kinematics techniques (e.g., sensors that directly track motion of the portion(s) of the repositionable structure). The determined point(s) and / or region(s) of interaction are then mapped to a common reference frame (e.g., reference frame 126 of anchor 124, the reference frame of the XR system, etc.) using the registration transform. An error is determined between the point(s) and / or region(s) of interaction in the common reference frame. Then, an update to the initial registration transform is computed based on the error, as discussed in greater detail below in conjunction with Figures 4-5. For example, assume the interaction involves the operator 108 pressing a mechanical button on the repositionable structure of computer-assisted device 110. In such a case, the position of a tip of Attorney Docket No. P06828-WO:0138PCthe finger of the operator that contacts the button in reference frame 132 of XR system 130, corresponding to a position of the button, is detected using a human pose estimation technique based on images acquired by sensor system 240. Then, the registration transform is applied to map the position (e.g., pose information) of the finger / button in reference frame 132 of XR system 130 to a position in reference frame 126 of anchor 124. Specifically, an inverse of the registration transform can be used to map the position of the finger / button in reference frame 132 of XR system 130 to the position in reference frame 126 of anchor 124. Thereafter, the position in reference frame 126 of anchor 124 is compared with a position (e.g., pose information) of the button in reference frame 126 that is determined using kinematics, or in any other technically feasible manner (e.g., using sensors such as a gyroscope, an IMU, shape sensors, etc.). An error between the positions is then computed and used to determine a gradient descent update to the initial registration. Alternatively, the registration transform is applied to map the position in reference frame 126 of anchor 124 to a position in reference frame of the XR system, and the error is computed in the reference frame of the XR system to determine the gradient descent update.

[0054] In some embodiments, in addition to updating the initial registration transform between the reference frame of the XR system associated with the operator who interacts with computer-assisted device 110 and the reference frame of anchor 124 based on the operator interaction(s), registration module 310 further updates other registration transforms between reference frames of other XR systems that share anchor 124 and the reference frame of anchor 124. In such cases, the updated registration transform between the reference frame of the XR system and the reference frame of anchor 124 that is determined based on operator interaction(s) can be used to compute an updated position and orientation of anchor 124 in a world reference frame, which can in turn be used to update the other registration transforms when the positions and orientations of the other XR systems in the world reference frame are known via, e.g., SLAM. In some embodiments, when multiple operators interact with portions of the computer-assisted system, one of the interactions by a corresponding operator can be selected to update a registration transform between a reference frame of an XR system associated with the corresponding operator and the reference frame of anchor 124, and the updated registration transform can in turn be used to update anchor 124 and the registration transforms between reference frames of other XR systems and the reference frame of anchor 124. In some examples, the interaction can be chosen according to various criteria, such as priorities associated with the interactions, a confidence associated with registrationsAttorney Docket No. P06828-WO:0138PCdetermined using the interactions, the duration of each interaction, the type of interactions, the timing of the interactions, the portions of the computer-assisted device that are being interacted with, a proximity of the users and / or portions of the computer-assisted device being interacted with, an identity or role of the operators, the XR systems, the context of an XR application, a combination of the interactions, etc. In some embodiments, when multiple computer-assisted devices are associated with different anchors, the update to an anchor associated with one computer-assisted device can be used to update another anchor associated with another computer-assisted device based on a fixed registration transform between reference frames of the two anchors. Alternatively, in some embodiments, the anchor associated with the other computer-assisted device can remain fixed, and the registration transform between reference frames associated with the two anchors can be updated based on the updated anchor. In such cases, the world anchors are mathematically, but not kinematically, coupled via the registration transform.

[0055] During operation, overlay module 312 generates XR content. As described, XR content includes AR, MR, and / or VR content in some embodiments. Any suitable XR content, such as the AR content 136 described above in conjunction with Figure 1, is generated. For example, the XR content can include instructional content on how to operate a computer-assisted device. As another example, the XR content can include content that provides data related to a procedure being performed by the computer-assisted device, such as previously captured images, models, real-time captured images, data about the functioning of the computer-assisted device, communications from others, tutorials or videos, guidance during operation of the computer-assisted device, etc. For XR system 130 to display XR content spatially relative to a portion of computer-assisted device 110, overlay module 312 applies registration transform 134 to map a known position of the portion of computer-assisted device 110 in reference frame 126 of anchor 124 to a corresponding position in reference frame 132 of XR system 130. Then, overlay module 312 generates XR content for display at or near the corresponding position in reference frame 132. The position of the portion of computer-assisted device 110 in reference frame 126 is either known (e.g., a known position of a helm, handle bar, base, or other portion that is not part of the repositionable structure of computer-assisted device 110), determined using kinematics, or determined in any other technically feasible manner (e.g., using sensors such as a gyroscope, an IMU, shape sensors, etc. that track the position of the portion of computer-assisted device 110). Thereafter, overlay module 312 Attorney Docket No. P06828-WO:0138PCoutputs to XR system 130 a display signal 320 that is used to display the XR content. A similar technique can be used to generate and display XR content using XR system 140.

[0056] In the AR case, overlay module 312 generates content for display to an operator to enhance a view of the physical environment. In the VR case, the content generated by overlay module 312 is combined with image data depicting the physical environment to generate a composite image for display to the operator. The image data is captured by one or more imaging devices in sensor system 240, or elsewhere. In addition, display signal 320 is generated based on the composite image.

[0057] Figure 4 illustrates a simplified diagram of a method 400 for updating an anchor based on one or more operator interactions, according to various embodiments. One or more of the processes 402-412 can be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine readable media that when executed by one or more processors (e.g., the processor 160 in control system 150) cause the one or more processors to perform one or more of the processes 402-412. In some embodiments, processes 402-412 can be performed by one or more modules, such as control module 180. In some embodiments, additional processes, which are not shown, can be performed.

[0058] As shown, at process 402, registration transforms are initialized between reference frames of XR devices (e.g., reference frames 132 and 142 of XR systems 130 and 140, respectively) and a reference frame of an anchor associated with a computer-assisted device (e.g., anchor 124 at base of computer-assisted device 110). As used herein, an anchor being associated with a computer-assisted device means the anchor is defined relative to the computer-assisted device, such as the anchor being fixed relative to a portion (e.g., a base) of the computer-assisted device. In some embodiments, the initial registration transforms can be determined according to any technically feasible technique. In some examples, the initial registration transforms are determined using the model target-based, marker-based, and / or other registration techniques described above in conjunction with Figure 3.

[0059] At process 404, one or more interactions are detected between one or more operators (e.g., operators 108 and / or 112) and one or more portions of the computer-assisted device (e.g., computer-assisted device 110). The one or more operator interactions can occur at any suitable time after the initial registration transform is determined. In some examples, the one or more operator interactions are detected during the setup and / or ordinary course of use of the computer-assisted device by the operator. In some examples, the one or more operator Attorney Docket No. P06828-WO:0138PCinteractions are detected when one or more operators are being trained on operating the computer-assisted device. In some examples, the one or more interactions are detected when one or more operators perform actions specifically to update the registration transform. For example, during the setup, ordinary course of use, training, and / or registration transform update, one or more prompts can instruct an operator to perform the one or more operator interactions. As another example, an operational state of the computer-assisted device at the time of the one or more interactions or a physical configuration of the computer-assisted device at the time of the one or more interactions (e.g., an initial state or configuration of the computer-assisted device) can be used to determine whether the one or more interactions are being performed by one or more operators during setup of the computer-assisted device. In such cases, interactions determined to be performed during the setup can be used to update the registration transform. As another example, an operational state of the computer-assisted device at the time of the one or more interactions or an operational state of the XR system at the time of the one or more interactions (e.g., a state of presenting training or other information via the XR system) can be used to determine whether the one or more interactions are being performed by the one or more operators during training to use the computer-assisted device or the ordinary use of the computer-assisted device. In such cases, interactions determined to be performed during the training or ordinary use can be used to update registration transforms (e.g., registration transform 134 and / or 144).

[0060] The one or more operator interactions at process 404 can include a single interaction, multiple separate interactions that are separated in time, or a compound interaction in which multiple interactions occur at the same time or in time periods that overlap. The one or more operator interactions can include one or more interactions by a single operator or interactions by multiple operators. Further, each interaction can be a discrete interaction that occurs within a shorter period of time, or be a continuous interaction that spans a longer period of time or involves motion of a link or a joint of the computer-assisted device that is more than the depression of a switch. When an interaction spans a period of time, the interaction can remain the same or change (e.g., from contacting a portion of the computer-assisted device to contacting a different portion or more than one portion of the computer-assisted device) over that period of time. An example of a single interaction is an operator pressing a button on the computer-assisted device. Examples of buttons include a physical button (e.g. a power-on button, an emergency stop button), and a virtual button (a menu selection button, a device control button, etc. displayed on a touchscreen of the computer-assisted device). AnotherAttorney Docket No. P06828-WO:0138PCexample of a single interaction is tapping one or more finger tips to a link of the computer-assisted device. Yet another example of a single interaction is an operator looking at a portion of the computer-assisted device. An example of multiple interactions is a sequence of single operator interactions separated in time that occur within a defined period of time, such as a finger tapping on a link of the computer-assisted device multiple times before a timeout.Multiple interactions can also be separate in space, such as a same or different finger pressing different buttons of the computer-assisted device at different times. An example of a compound interaction is multiple portions of an operator contacting different portions of the computer-assisted device at the same time, such as different feet of an operator stepping on different pedals of a computer-assisted device at a same time or in an time-overlapped manner. Another example of a compound interaction is an operator grabbing a first link of a computer-assisted device (such as to hold the link stationary in space) while applying force to a second link of the computer-assisted device (such as to move the second link). Yet another example of a compound interaction is an operator looking at a portion of the computer-assisted device while pressing a button. An example of a continuous interaction is an operator grabbing a portion of the repositionable structure that includes a hand-input sensor and moving the portion of the repositionable structure over a period of time. In such a case, the continuous interaction ends when another predefined event is detected, such as when the operator releases a device control button or is no longer in contact with the computer-assisted device. As shown in the above examples, an operator interaction can comprise the operator interacting with (e.g., touching, grabbing, tapping, moving, pressing, swiping, looking at, etc.) a portion of the computer-assisted device that comprises a mechanical input device of the computer-assisted device, or with a virtual input device displayed on or proximate to a portion of the computer-assisted device. Yet another example of an operator interaction is the operator attaching / coupling or detaching / decoupling an object, such as an instrument, a cannula, a drape, etc. to / from the computer-assisted device.

[0061] In some embodiments, the one or more operator interactions are detected at process 404 by hand-input sensors, foot-input sensors, cameras, inertial sensors (e.g., gyroscopes), and / or other sensors that are included in the computer-assisted device and / or elsewhere in the physical environment. The hand-input and / or foot-input sensors can be buttons, knobs, finger detectors, joysticks, recessed, pedals, among other things. The hand-input and / or foot-input sensors can include touch sensors (e.g., inductive or capacitive touch sensors), contact sensors, Attorney Docket No. P06828-WO:0138PCpressure sensors, strain gauges, springs, etc. for sensing hand input and / or foot input, respectively.

[0062] At process 406, one of the interactions is selected if multiple operator interactions occur concurrently. In some embodiments, one of the interactions can be selected based on priorities associated with the interactions. In such cases, a first interaction between a first portion of an operator and a first portion of a computer-assisted device can be selected over a second interaction between a second portion of that operator (or of another operator) and a second portion of the computer-assisted device, based on priorities associated with the first and second interactions. Selection among more than two interactions can involve consideration of additional priorities associated with additional interactions. Any suitable priorities can be predefined or determined by control module 180, such as priorities based on the type of interactions, portions of the computer-assisted device that are interacted with, operator preference, a combination thereof, etc.

[0063] In some embodiments, one of the interactions is selected based on the type of interactions. In some examples, certain types of interactions can have priority over other types of interactions. For example, a button press interaction could have priority over a touch interaction, and a touch interaction could have priority over a looking interaction.

[0064] In some embodiments, one of the interactions is selected based on how long in time each interaction is. In some examples, an interaction that is longest in time is selected because registration transforms determined based on longer interactions are, as a general matter, more accurate than registration transforms determined based on shorter interactions.

[0065] In some embodiments, one of the interactions is selected based on a confidence in the accuracy of candidate registration transforms determined from the interactions. In some examples, certain types of interactions can be associated with a higher accuracy of registrations than other types of interactions, and longer interactions can also be associated with higher accuracy of registrations than shorter interactions. In some examples, an interaction with a portion of the computer-assisted device that has been calibrated most recently, and is therefore expected to provide the most accurate registrations, is selected. In some embodiments, control module 180 computes a confidence score for each interaction, such as a score that is based on the type of the interaction, the temporal length of the interaction, the calibration time of the portion of the computer-assisted device being interacted Attorney Docket No. P06828-WO:0138PCwith, a combination thereof, etc., and then control module 180 selects an interaction associated with the highest confidence score.

[0066] In some embodiments, one of the interactions is selected based on an identity or role of the operator who performed the interaction. In some examples, one operator is the primary operator whose interactions have the highest priority for updating the anchor. For example, one operator could be an instructor having a higher priority than other operators that are trainees. In some examples, interactions of one or more operators with the computer-assisted device do not update the anchor. Returning to the example of one operator being an instructor and other operators being trainees, interactions of the trainees with the computer-assisted device may not update the anchor.

[0067] In some embodiments, one of the interactions is selected based on the portions of the computer-assisted device that are being interacted with. In some examples, an interaction with a portion of the computer-assisted device that an operator is working on is selected. In some examples, different portions of the computer-assisted device are associated with different kinematic models, some of which can be used to determine more accurate registration transforms. In such a case, control module 180 selects an interaction with a portion of the computer-assisted device that is associated with the most accurate kinematic model. In some examples, when the anchor is associated with a base of the computer-assisted device, control module 180 selects an interaction with a portion of the computer-assisted device that is most proximal to the base of the computer-assisted device. In some examples, control module 180 selects an interaction with a portion of the computer-assisted device that is rigidly attached to a base of the computer-assisted device where an anchor is positioned, as opposed to a portion of the computer-assisted device that can move dynamically with respect to the base of the computer-assisted device, because registration transforms can be determined more accurately using interactions with rigidly attached portions of the computer-assisted device that cannot move to cause inaccuracies in the registration transforms. In some examples, an interaction with a portion of the computer-assisted device that has been calibrated most recently is selected.

[0068] In some embodiments, one of the interactions is selected based on the XR systems associated with the operators who interact with portions of the computer-assisted device. In some examples, certain XR systems can have higher priority than other XR systems, and an interaction associated with a higher priority XR system can be selected over interactionsAttorney Docket No. P06828-WO:0138PCassociated with lower priority XR systems. For example, an XR system for an instructor could have higher priority than an XR system for a trainee. In some examples, certain XR systems can have sensors and / or other capabilities that other XR systems lack, and an interaction by a user associated with an XR system that has the sensors and / or other capabilities can be selected. For example, when an XR system lacks qualified sensors and / or object detection capability, interactions by a user wearing such an XR system are not selected.

[0069] In some embodiments, one of the interactions is selected based on timings of the interactions. In some examples, an interaction that is first in time is selected. In some examples, an interaction that is last in time is selected.

[0070] In some embodiments, one of the interactions is selected based on a combination of the interactions. In some examples, a weighted combination of the interactions can be used. For example, the weights used in the weighted combination could be based on confidences in the accuracy of candidate registration transforms determined using the interactions.

[0071] In some embodiments, one of the interactions is selected based on the context of the application. For example, for a training application on the use of an instrument arm of a computer-assisted device, an operator interacting with the instrument arm can be more relevant than other operators. In such a case, the interaction of a portion of the operator with the instrument arm can be selected.

[0072] In some embodiments, one of the interactions is randomly selected.

[0073] In some embodiments, one of the interactions is selected based on any combination of the foregoing criteria for selecting an interaction.

[0074] At process 408, pose information is determined for a portion of an operator associated with the selected interaction based on sensor data. The pose information of the portion of the operator includes a position of the portion of the operator, an orientation of the portion of the operator, or both during the operator interaction. The pose information can be associated with a point or region of interaction between the portion of the operator and the portion of the computer-assisted device, as described above in conjunction with Figure 3. In some embodiments, the pose information of the portion of the operator is determined in a reference frame of the XR system (e.g., reference frame 132 or 142). In some examples, the portion of the operator is a hand or finger. Additional examples of portions of the operator include multiple fingers, arm, feet, and / or extensions of the foregoing (gloves, shoes, cable, Attorney Docket No. P06828-WO:0138PCpointer stick), etc. In some embodiments, the pose information of the portion of the operator is determined in up to six DOFs in three-space, including any one or more of x, y, or z position and / or roll, pitch, or yaw orientations. In some embodiments, the pose information includes one or more position (e.g., x, y, and / or z) parameters. In some embodiments, the pose information includes one or more position (e.g., x, y, and / or z) parameters and one or more orientation (e.g., roll, pitch, and / or yaw) parameters.

[0075] In some embodiments, the pose information is determined using any technically feasible human pose estimation technique. In some examples, the human pose estimation technique can be a machine learning and / or other computer vision technique that is employed to detect the portion(s) of the operator, or extension(s) thereof, in captured images of the selected interaction between the portion of the operator and the computer-assisted device. In such cases, the portion of the operator can be modeled / identified in various ways. For example, object segmentation and / or part segmentation techniques can be employed to estimate one or more points and / or regions corresponding to specific parts of a hand of the operator, such as the fingers, finger tips, carpels, meta carpels, palm, and / or wrist in images acquired by image sensors. As another example, the center of a palm position can be estimated by fitting a circle formed by the wrist, and two or more of the most proximal joints of the fingers (e.g., thumb proximal joint and the most proximal joints of the index finger, middle finger, ring finger, and pinky). In some examples, an entire region of contact, represented as a point cloud or mesh, can be estimated, such as the contact between hand(s) of an operator and the computer-assisted device. When a hand pose cannot be estimated or only a partial hand is visible, one or more points and / or regions representative of a fingertip and / or joint, carpel, meta carpel, palm, and / or wrist, foot, etc. can be estimated by fitting a pre-posed full hand to partial data using a partial shape matching technique. In some examples, multiple poses or partial poses can be inferred for multiple portions of the operator that are in view of imaging device(s). Additional computer vision-based human pose estimation techniques are described in Sorriento et al., “Optical and Electromagnetic Tracking Systems for Biomedical Applications: A Critical Review on Potentialities and Limitations,” IEEE Rev. Biomed. Eng.2020;13:212-232 (Sept. 2019), and Oudah et al., “Hand Gesture Recognition Based on Computer Vision: A Review of Techniques,” Journal of Imaging 6, no. 8: 73 (2020), which are hereby incorporated by reference herein.

[0076] In some examples, the human pose estimation technique can employ sensor data acquired by or with the aid of devices that the operator contacts, such as sensor clothing that Attorney Docket No. P06828-WO:0138PCthe operator wears, objects including sensors that the operator holds, etc. For example, the pose of a hand can be determined using sensor data acquired by a sensory glove. As another example, the pose of a hand can be determined based on how an operator holds a controller device that includes sensors. As another example, a body tracking suit can be employed in conjunction with an optical camera motion capture technique to determine poses of an entire body of an operator or specific portions of the body, such as arms, legs, etc.

[0077] In some examples, the human pose estimation technique also outputs a confidence score associated with the determined pose of the portion of the operator. For example, the confidence score can indicate a probability that the determined portion and / or pose is correct.

[0078] At process 410, pose information is determined for the portion of the computer- assisted device associated with the selected interaction. The pose information of the portion of the computer-assisted device includes a position of the portion of the computer-assisted device, an orientation of the portion of the computer-assisted device, or both during the selected interaction. In some embodiments, the pose information of the portion of the computer-assisted device is determined in a reference frame of an anchor (e.g., anchor 124) associated with the computer-assisted device (e.g., reference frame 126) and corresponds in time with the operator interaction and with the pose information of the portion of the operator, determined at process 408. In some embodiments, the pose information of the portion of the computer-assisted device is determined at the time of the operator interaction in up to six DOFs in three-space for a rigid body (e.g., a button, switch, portion of screen, part of a rigid link), including any one or more of x, y, and z position and / or roll, pitch, and yaw orientations. When the operator interacts with a portion of a repositionable structure of the computer-assisted device, forward and / or reverse kinematics are used to determine the pose information of the portion of the repositionable structure given known joint locations at the time of the operator interaction. In some embodiments, alternative techniques are used to track the pose information of the portion of a repositionable structure without requiring kinematics. For example, sensors (e.g., a gyroscope) within the portion of the repositionable structure can directly track motion of the portion of the repositionable structure. As additional examples, IMUs, shape sensors, and the like can be used to directly track motion of the portion of the repositionable structure. When the operator interacts with a portion of the computer-assisted device that is not included in a repositionable structure of the computer-assisted device or with a portion of a computer-assisted device that does not include a repositionable structure, pose information of the portion of the computer-assisted device is determined based on a pose ofAttorney Docket No. P06828-WO:0138PCthe overall computer-assisted device 110 and a known position and orientation of the portion relative to the overall computer-assisted device 110.

[0079] At process 412, an anchor (e.g., anchor 124) associated with the computer-assisted device (e.g., computer-assisted device 110) and shared by multiple XR devices (e.g., XR systems 130 and 140) is updated based on the pose information of the portion of the operator and the pose information of the portion of computer-assisted device. Updating of the anchor at process 412 is described in greater detail below in conjunction with Figure 5. In some embodiments, updating the anchor includes updating the registration transform(s) relating reference frames of the multiple XR systems and a reference frame of the anchor. The registration transform(s) can be updated by moving the anchor in a reference frame (e.g., a world reference frame) while not moving the XR system(s) in the reference frame, moving the XR system(s) in a reference frame while not moving the anchor in the reference frame (e.g., by updating the registration transforms of the XR systems relative to the anchor), or moving both the anchor and the XR system(s) in a reference frame by different amounts.

[0080] The updated anchor can be used to render an image including XR content that is displayed by one or more of the XR systems and viewable by operator(s) until a new update to the anchor is determined. In this manner, the XR systems use the latest update of the anchor when rendering images including XR content. In some embodiments, control module 180 causes the XR systems to render such an image, either in response to the one or more operator interactions at process 404 or at a later time. For example, the XR content in the rendered image can provide information for setting up the computer-assisted device, training to use the computer-assisted device, or ordinary operation of the computer-assisted device.

[0081] Although process 412 is described with respect to updating an anchor shared by multiple XR systems (“shared anchor”), in some embodiments, other XR systems may not share the shared anchor that is updated. In some examples, operators of XR systems can, either by choice or forcibly, opt into sharing the shared anchor. For example, an operator of an XR system could select to share or not to share the shared anchor via a user interface (UI), gesture, voice command, etc. A separate anchor can be created for each XR system that does not share the shared anchor, and the separate anchor can be updated independently based on interactions of the operator with the computer-assisted device. Subsequently, the operator of an XR system that is not sharing the shared anchor can select to share the shared anchor, in which case the separate anchor for the XR system is updated to the shared anchor, such as by replacing theAttorney Docket No. P06828-WO:0138PCseparate anchor with the shared anchor. Prior to updating the separate anchor to the shared anchor, the separate anchor can be saved for future use. Alternatively, the separate anchor can be deleted. In some examples, other XR systems can be present that do not share the shared anchor as a result of, e.g., operator preference, a lack of qualified sensors and / or object detection capability, etc. For example, an XR system could lack depth sensors to acquire data for registration, and such an XR system may not share the shared anchor.

[0082] In some embodiments, one or more operators wearing XR systems at one or more other locations (or at the same location) can view a virtual reproduction of the room in which the shared anchor is updated. For example, the one or more other locations can be one or more other rooms. In such cases, one or more corresponding anchors can be placed within the one or more other locations (or the same location), such as at arbitrary positions and orientations within the one or more other locations (or the same location). In addition, the corresponding anchors can be updated based on the update to the shared anchor at process 412, such as by a same amount that the shared anchor is updated at process 412.

[0083] Figure 5 illustrates process 412 for updating the anchor in the method 400 of Figure 4 in greater detail, according to various embodiments. One or more of the processes 502-510 can be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine readable media that when executed by one or more processors (e.g., the processor 160 in control system 150) cause the one or more processors to perform one or more of the processes 502-510. In some embodiments, processes 502-510 can be performed by one or more modules, such as control module 180. In some embodiments, additional processes, which are not shown, can be performed.

[0084] As shown, at process 502, pose information of the selected portion of the operator and / or pose information of the portion of the computer-assisted device is transformed into a common reference frame using a first registration transform (e.g., registration transform 134 or 144) between a first XR system (e.g., XR system 130 or 140) and the anchor (e.g., anchor 124) that is to be updated. In some embodiments, the common reference frame is a reference frame of the anchor, a reference frame of the XR system, or any other reference frame to which the pose information of the portion of the operator and / or the pose information of the portion of the computer-assisted device are transformed into. For example, when the common reference frame is the reference frame of the XR system, the pose information of the portion of the computer-assisted device that is determined at process 410 is mapped to pose information in Attorney Docket No. P06828-WO:0138PCthe reference frame of the XR system using the registration transform. As another example, when the common reference frame is the reference frame of the anchor, the pose information of the portion of the operator determined at process 408 is mapped to pose information in the reference frame of the anchor using an inverse of the registration transform. In some examples the common reference frame can be chosen based on a procedure being performed, a type of the instrument, a type of the imaging device, operator preference, and / or the like.

[0085] At process 504, an error is determined between the pose information of the portion of the operator and the pose information of the computer-assisted device in the common reference frame. In some embodiments, the error is a difference between the pose information of the portion of the operator and the pose information of the portion of the computer-assisted device in the common reference frame. For example, assume that the pose information of the portion of the operator is mapped to pose information in a reference frame of the anchor. In such a case, the error is the difference between the mapped pose information of the portion of the operator and the pose information of the portion of the computer-assisted device in the reference frame of the anchor, determined at process 410. As another example, assume that the pose information of a portion of the computer-assisted device is mapped to pose information in a reference frame of the XR system. In such a case, the error is the difference between the mapped pose information of the computer-assisted device and the pose information of the portion of the operator, determined at process 408.

[0086] At process 506, the first registration transform between the first XR system and the anchor is updated based on the error. In some embodiments, the first registration transform can be updated by transforming the pose information of the portion of the operator and / or the pose information of the portion of the computer-assisted device into a common reference frame using a registration transform between the first XR system and the anchor, determining a difference between the pose information of the portion of the operator and the pose information of the portion of the computer-assisted device in the common reference frame, and updating the registration transform between the first XR system and the anchor based on the error. In some embodiments, the common reference frame is a reference frame of the anchor, a reference frame of the first XR system, or any other reference frame to which the pose information of the portion of the operator and / or the pose information of the portion of the computer-assisted device are transformed into. In some embodiments, the registration transform between the first XR system and the anchor is updated using a gradient descent or any other suitable technique.Attorney Docket No. P06828-WO:0138PC

[0087] At process 508, the anchor is updated based on the updated first registration transform. In some embodiments, an updated position and orientation of the anchor is determined in a world reference frame. In some embodiments, updating the anchor includes computing an updated reference frame of the anchor from a reference frame of the first XR system and the updated first registration transform. In such cases, the updated reference frame of the anchor can be computed as a registration transform between a world reference frame and the updated reference frame. In some embodiments, registration transforms between XR systems and the anchor are updated, which is equivalent to updating the anchor in a world reference frame.

[0088] At process 510, a second registration transform (e.g., registration transform 144 or 134) between a second XR system (e.g., XR system 140 or 130) and the anchor (e.g., anchor 124) is updated based on the updated anchor. The second registration transform provides a geometric relationship between the second XR system and the anchor. The second registration transform can be updated by recomputing the second registration transform to relate a reference frame of the second XR system to a reference frame of the updated anchor.

[0089] Alternatively, in some embodiments, an anchor can be “shared” by a first XR system that maintains a first anchor and a second XR system that maintains a second anchor as follows. The first and second anchors are initialized with the same location and orientation using another system, such as a set of cameras. Subsequently, the first XR system can make updates to the first anchor based on interactions between one or more portions of a first operator with one or more portions of a computer-assisted device according to processes 502-508, described above. The first XR system transmits the updated first anchor to the second XR system, and the second XR system uses the updated first anchor to update the second anchor to have the same location and orientation as the first anchor. Registration transforms between a reference frame of the first XR system and a reference frame of the first anchor, as well as between a reference frame of the second XR system and a reference frame of the second anchor, can also be updated based on the updated first and second anchors, respectively.

[0090] Figure 6 illustrates a simplified diagram of a method 600 for updating anchors associated with different computer-assisted devices, according to other embodiments. One or more of the processes 602-612 can be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine readable media that when executed by one or more processors (e.g., the processor 160 in control system 150) cause the one or moreAttorney Docket No. P06828-WO:0138PCprocessors to perform one or more of the processes 602-612. In some embodiments, processes 602-612 can be performed by one or more modules, such as control module 180. In some embodiments, additional processes, which are not shown, can be performed.

[0091] As shown, at process 602, a first registration transform is initialized between a first XR device and a first anchor associated with a first computer-assisted device, and a second registration transform is initialized between a second XR device and a second anchor associated with a second computer-assisted device. In some embodiments, each of the first registration transform and the second registration transform can be initialized in a manner similar to the initialization of the registration transforms between the XR devices and the anchor associated with a computer-assisted device in process 402 of method 400, described above in conjunction with Figure 4.

[0092] At process 604, one or more interactions are detected between an operator and a portion of the first computer-assisted device. In some embodiments, the one or more interactions can be detected in a manner similar to the detection of the one or more interactions between the operator(s) and the portion(s) of the computer-assisted device at process 404 of method 400, described above in conjunction with Figure 4. In some embodiments, when multiple interactions between different operators and the first computer- assisted device are detected, one of the multiple interactions can be selected according to criteria similar to the criteria described above in conjunction with Figure 4.

[0093] At process 606, pose information of a portion of the operator is determined based on sensor data. In some embodiments, the pose information of the portion of the operator can be determined in a manner similar to the determination of the pose information of the portion of the operator associated with a selected interaction based on sensor data at process 408 of method 400, described above in conjunction with Figure 4.

[0094] At process 608, pose information of the portion of the first computer-assisted device is determined. In some embodiments, the pose information of the portion of the first computer-assisted device can be determined in a manner similar to the determination of the pose information of the portion of the computer-assisted device at process 410 of method 400, described above in conjunction with Figure 4.

[0095] At process 610, the first anchor is updated based on the pose information of the portion of the operator and the pose information of the portion of the first computer-assisted Attorney Docket No. P06828-WO:0138PCdevice. In some embodiments, the first anchor can be updated in a manner similar to the updating of the anchor at process 412 of method 400, described above in conjunction with Figures 4-5. In some embodiments, one or more rregistration transforms between one or more other XR systems and the first anchor can be updated based on the updated first anchor, similar to process 510 described above in conjunction with Figure 5.

[0096] At process 612, the second anchor is updated based on the updated first anchor and a registration transform between a reference frame of the first anchor and a reference frame of the second anchor. The registration transform between the reference frame of the first anchor and the reference frame of the second anchor, which is not updated, can be used to compute the second anchor given the updated first anchor. In some examples, the first computer-assisted device, to which the first anchor is associated, is kinematically coupled to the second computer-assisted device, to which the second anchor is associated. When the kinematic coupling results in a fixed geometric relationship between the first computer-assisted device and the second computer-assisted device, the registration transform between the reference frame of the first anchor and the reference frame of the second anchor that are associated with the first computer-assisted device and the second computer-assisted device, respectively, may not change when the first or the second computer-assisted device is moved. In such cases, the registration transform between the reference frame of the first anchor and the reference frame of the second anchor is not updated. Instead, at process 612, the second anchor is updated using the fixed registration transform when the first anchor is updated.

[0097] Figure 7 illustrates a simplified diagram of a method 700 for updating anchors associated with different computer-assisted devices, according to various other embodiments. One or more of the processes 702-712 can be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine readable media that when executed by one or more processors (e.g., the processor 160 in control system 150) cause the one or more processors to perform one or more of the processes 702-712. In some embodiments, processes 702-712 can be performed by one or more modules, such as control module 180. In some embodiments, additional processes, which are not shown, can be performed.

[0098] As shown, at process 702, a first registration transform is initialized between first XR device and first anchor associated with first computer-assisted device and second registration transform between second XR device and second anchor associated with secondAttorney Docket No. P06828-WO:0138PCcomputer-assisted device. In some embodiments, each of the first registration transform and the second registration transform can be initialized in a manner similar to the initialization of the registration transforms between the XR devices and the anchor associated with a computer-assisted device in process 402 of method 400, described above in conjunction with Figure 4.

[0099] At process 704, one or more interactions are detected between an operator and a portion of the first computer-assisted device. In some embodiments, the one or more interactions can be detected in a manner similar to the detection of the one or more interactions between the operator(s) and the portion(s) of the computer-assisted device at process 404 of method 400, described above in conjunction with Figure 4. In some embodiments, when multiple interactions between different operators and the first computer-assisted device are detected, one of the multiple interactions can be selected according to criteria similar to the criteria described above in conjunction with Figure 4.

[0100] At process 706, pose information of a portion of the operator is determined based on sensor data. In some embodiments, the pose information of the portion of the operator can be determined in a manner similar to the determination of the pose information of the portion of the operator associated with a selected interaction based on sensor data at process 408 of method 400, described above in conjunction with Figure 4.

[0101] At process 708, pose information of the portion of the first computer-assisted device is determined. In some embodiments, the pose information of the portion of the first computer-assisted device can be determined in a manner similar to the determination of the pose information of the portion of the computer-assisted device at process 410 of method 400, described above in conjunction with Figure 4.

[0102] At process 710, the first anchor is updated based on the pose information of the portion of the operator and the pose information of the portion of the first computer-assisted device. In some embodiments, the first anchor can be updated in a manner similar to the updating of the anchor at process 412 of method 400, described above in conjunction with Figure 4. In some embodiments, one or more rregistration transforms between one or more other XR systems and the first anchor can be updated based on the updated first anchor, similar to process 510 described above in conjunction with Figure 5.Attorney Docket No. P06828-WO:0138PC

[0103] At process 712, a registration transform between a reference frame of the first anchor and a reference frame of the second anchor is updated based on the updated first anchor. In some examples, the first computer-assisted device, to which the first anchor is associated, is not kinematically coupled to the second computer-assisted device, to which the second anchor is associated. As a result, the registration transform between the reference frame of the first anchor and the reference frame of the second anchor does not need to be fixed when the first anchor is updated. In some examples, the registration transform between the reference frame of the first anchor and the reference frame of the second anchor is associated with a low confidence in the accuracy of the registration transform. Because of the low confidence in the registration transform, the registration transform is updated when the first anchor is updated.

[0104] Figure 8 illustrates an example of updating a shared anchor, according to various embodiments. As shown, a reference frame 804 of an anchor associated with a computer-assisted device is geometrically related to a world reference frame 802 according to a registration transform 810, a reference frame 806 of a first XR device is geometrically related to reference frame 804 of the anchor according to a registration transform 820, and a reference frame 808 of a second XR device is geometrically related to reference frame 804 of the anchor according to a registration transform 822. In addition, reference frame 806 of the first XR device is geometrically related to world reference frame 802 according to a registration transform 812, and reference frame 808 of the second XR device is geometrically related to world reference frame 802 according to a registration transform 814.

[0105] When reference frame 804 of the anchor is moved to a different position and / or orientation, shown as reference frame 804’ that is geometrically related to world reference frame 802 according to a registration transform 830, the geometric relationships between reference frames 806 and 808 of the XR devices and reference frame 804’ of the anchor is different than the geometric relationships between reference frames 806 and 808, respectively, of the XR devices and reference frame 804 of the anchor. To account for the changed geometric relationships, registration transforms 820 and 822 need to be updated to registration transforms 832 and 834, respectively. Assume a portion of an operator wearing the first XR device interacts with a portion of the computer-assisted device. In such a case, registration transform 820 between reference frame 808 of the first XR device and reference frame 804 of the anchor is updated to registration transform 832 based on pose information of the portion of the operator and pose information of the portion of the computer-assisted device that the Attorney Docket No. P06828-WO:0138PCoperator interacts with, as described above in conjunction with Figures 4-5. Then, updated reference frame 804’ of the anchor is computed from reference frame 806 of the first XR device and updated registration transform 832. Given updated reference frame 804’ of the anchor and the known reference frame 808 of the second XR device, registration transform 834 between reference frame 808 of the second XR device and updated reference frame 804’ of the anchor can be computed.

[0106] Figure 9 illustrates an example of updating anchors associated with different computer-assisted devices, according to various embodiments. As shown, a reference frame 908 of a first XR device is geometrically related to a reference frame 902 of a first anchor associated with a first computer-assisted device according to a registration transform 920, and a reference frame 910 of a second XR device is geometrically related to a reference frame 904 of a second anchor associated with a second computer-assisted device according to a registration transform 940. In addition, reference frame 902 of the first anchor is geometrically related to reference frame 904 of the second anchor according to a registration transform 906.

[0107] When reference frame 902 of the first anchor is moved to a different position and / or orientation, shown as reference frame 902’, the geometric relationship between reference frame 908 of the first XR device and reference frame 902’ of the first anchor is different than the geometric relationship between reference frame 908 of the first XR device and reference frame 902 of the first anchor. To account for the changed geometric relationship, registration transform 920 needs to be updated to a registration transform 930. Assume a portion of an operator wearing the first XR device interacts with a portion of the first computer-assisted device. In such a case, registration transform 920 between reference frame 908 of the first XR device and reference frame 902 of the first anchor is updated to registration transform 920’ based on pose information of the portion of the operator and pose information of a portion of the computer-assisted device that the operator interacts with. Then, updated reference frame 902’ of the first anchor is computed from reference frame 908 of the first XR device and updated registration transform 930, as described above in conjunction with Figure 6.

[0108] In addition, reference frame 904 of the second anchor is updated to reference frame 904’ of the second anchor based on updated reference frame 902’ of the first anchor and registration transform 906 between reference frame 902 of the first anchor and reference frame 904 of the second anchor. To update reference frame 904 of the second anchor, registration transform 906 is used to compute updated reference frame 904’ of the second anchor givenAttorney Docket No. P06828-WO:0138PCupdated reference frame 902’ of the first anchor. As described, the first computer-assisted device, to which the first anchor and corresponding reference frame 902 (and 902’) are associated, can be kinematically coupled to the second computer-assisted device, to which the second anchor and corresponding reference frame 904 (and 904’) are associated, resulting in a fixed geometric relationship between the first computer-assisted device and the second computer-assisted device that means registration transform 906 remains unchanged. Instead, reference frame 904 of the second anchor is updated to updated reference frame 904’ of the second anchor when reference frame 902 of the first anchor is updated to updated reference frame 902’ of the first anchor. Illustratively, reference frame 910 of the second XR device is also geometrically related to updated reference frame 904’ of the second anchor according to a registration transform 950.

[0109] Figure 10 illustrates an example of updating a registration transform between anchors associated with different computer-assisted devices, according to various embodiments. As shown, a reference frame 1008 of a first XR device is geometrically related to a reference frame 1002 of a first anchor associated with a first computer-assisted device according to a registration transform 1020, and a reference frame 1010 of a second XR device is geometrically related to a reference frame 1004 of a second anchor associated with a second computer-assisted device according to a registration transform 1040. In addition, reference frame 1002 of the first anchor is geometrically related to reference frame 1004 of the second anchor according to a registration transform 1006.

[0110] Similar to the description above in conjunction with Figure 9, when reference frame 1002 of the first anchor is moved to a different position and / or orientation, shown as reference frame 1002’, the geometric relationship between reference frame 1008 of the first XR device and reference frame 1002’ of the first anchor is different than the geometric relationship between reference frame 1008 of the first XR device and reference frame 1002 of the anchor. To account for the changed geometric relationship, registration transform 1020 needs to be updated to a registration transform 1030. Assume a portion of an operator wearing the first XR device interacts with a portion of the first computer-assisted device. In such a case, registration transform 1020 between reference frame 1008 of the first XR device and reference frame 1002 of the first anchor is updated to registration transform 1030 based on pose information of the portion of the operator and pose information of a portion of the computer-assisted device that the operator interacts with. Then, updated reference frame 1002’ of the anchor is computedAttorney Docket No. P06828-WO:0138PCfrom reference frame 1008 of the first XR device and updated registration transform 1030, as described above in conjunction with Figure 7.

[0111] Illustratively, reference frame 1004 of the second anchor is not updated based on updated reference frame 1002’ of the first anchor. Instead, registration transform 1006 between reference frame 1002 of the first anchor and reference frame 1004 of the second anchor is updated to a registration transform 1060 between updated reference frame 1002’ of the first anchor and reference frame 1004 of the second anchor. For example, the first computer-assisted device, to which the first anchor and corresponding reference frame 1002 (and 1002’) are associated, may not be kinematically coupled to the second computer-assisted device, to which the second anchor and corresponding reference frame 1004 are associated. As another example, control module 180 could have low confidence in registration transform 1006 between reference frame 1002 of the first anchor and reference frame 1004 of the second anchor. As a result, registration transform 1006 between reference frame 1002 of the first anchor and reference frame 1004 of the second anchor is updated to registration transform 1060.

[0112] The disclosed techniques can update an anchor within an XR environment that is shared by multiple XR systems. As a result of the updated anchor, the XR systems can display XR content at more accurate positions and / or orientations within the XR environment than if the anchor were not updated. The displayed XR content can include instructive content on how to operate the computer-assisted device, content that provides guidance during operation of the computer-assisted device, and the like.

[0113] Some examples of control systems, such as control system 150 may include non- transitory, tangible, machine readable media that include executable code that when executed by one or more processors (e.g., processor 160) may cause the one or more processors to perform the processes of methods 400, 600, and / or 700 and / or the processes of Figures 4, 5, 6, and / or 7. Some common forms of machine readable media that may include the processes of methods 400, 600, and / or 700 and / or the processes of Figures 4, 5, 6, and / or 7 are, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium from which a processor or computer is adapted to read.Attorney Docket No. P06828-WO:0138PC

[0114] Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the invention should be limited only by the following claims, and it is appropriate that the claims be construed broadly and, in a manner, consistent with the scope of the embodiments disclosed herein.

Claims

WHAT IS CLAIMED IS:

1. An electronic system comprising:a computer-assisted device; anda control system communicably coupled to the computer-assisted device and to a sensor system, the sensor system configured to capture sensor data about an environment external to the computer-assisted device,wherein the control system is configured to:determine, based on the sensor data, pose information of an operator portion of a first operator during one or more interactions between the operator portion and a device portion of the computer-assisted device, the pose information of the operator portion comprising at least one parameter selected from the group consisting of: a position of the operator portion during the one or more interactions and an orientation of the operator portion during the one or more interactions,determine pose information of the device portion during the one or more interactions, the pose information of the device portion comprising at least one parameter selected from the group consisting of: a position of the device portion during the one or more interactions and an orientation of the device portion during the one or more interactions,update, based on the pose information of the operator portion and the pose information of the device portion, an anchor that is shared by a first extended reality (XR) system and a second XR system, the first XR system configured to render images for the first operator and the second XR system configured to render images for a second operator, andcause the second XR system to render an image viewable by the second operator using the updated anchor.

2. The electronic system of claim 1, wherein to update the anchor based on the pose information of the operator portion and the pose information of the device portion, the control system is configured to:update a first registration transform between a first reference frame of the first XR system and a second reference frame of the anchor based on the pose information of the operator portion and the pose information of the device portion; andupdate the anchor based on the updated first registration transform; andAttorney Docket No. P06828-WO:0138PCupdate a second registration transform between a third reference frame of the second XR system and the second reference frame of the anchor based on the updated anchor.

3. The electronic system of claim 1, wherein the control system is further configured to select the one or more interactions based on a priority associated with the one or more interactions and a second priority associated with a second interaction, the second interaction being between a second operator portion and a second device portion of the computer-assisted device.

4. The electronic system of claim 1, wherein the control system is further configured to select the one or more interactions based on a confidence score associated with the one or more interactions and a second confidence score associated with a second interaction between a second operator portion and a second device portion of the computer-assisted device.

5. The electronic system of claim 4, wherein the confidence score associated with the one or more interactions indicates a predicted accuracy of a candidate registration transform between a first reference frame of the first XR system and a second reference frame of the anchor, and wherein the candidate registration transform is determined based on the one or more interactions.

6. The electronic system of claim 1, wherein the control system is further configured to select the one or more interactions based on a temporal length of the one or more interactions.

7. The electronic system of claim 1, wherein the control system is further configured to select the one or more interactions based on a type of the one or more interactions.

8. The electronic system of claim 1, wherein the control system is further configured to select the one or more interactions based on a timing of the one or more interactions relative to a second interaction between a second operator portion and a second device portion of the computer-assisted device.

9. The electronic system of claim 1, wherein the control system is further configured to select the one or more interactions based on the device portion of the computer-assisted device.Attorney Docket No. P06828-WO:0138PC10. The electronic system of claim 1, wherein the control system is further configured to select the one or more interactions based on an identity or a role of the first operator.

11. The electronic system of claim 1, wherein the control system is further configured to select the one or more interactions based on the first XR system.

12. The electronic system of claim 1, wherein the control system is further configured to select the one or more interactions based on an application for which the image is rendered.

13. The electronic system of any of claims 1 to 12, wherein the control system is further configured to cause the first XR system to render an image viewable by the first operator using the updated anchor.

14. The electronic system of any of claims 1 to 12, wherein the one or more interactions occur while the first XR system is rendering an image viewable by the first operator.

15. The electronic system of any of claims 1 to 12 wherein the anchor is defined relative to a second portion of the computer-assisted device.

16. The electronic system of any of claims 1 to 12, wherein the anchor is a first anchor, and wherein the first anchor is not shared by a third XR system configured to render images for a third operator using a second anchor.

17. The electronic system of claim 16, wherein the control system is further configured to, in response to a selection to use the updated anchor with the third XR system, update the second anchor based on the updated anchor.

18. The electronic system of any of claims 1 to 12, wherein the control system is further configured to update, based on the updated anchor, a second anchor that is associated with a third XR system.

19. The electronic system of any of claims 1 to 12, wherein the anchor is a first anchor, wherein the control system is further configured to update a second anchor based on theAttorney Docket No. P06828-WO:0138PCupdated anchor and a registration transform between a reference frame of the first anchor and a reference frame of the second anchor, and wherein the second anchor is associated with a second computer-assisted device.

20. The electronic system of any of claims 1 to 12, wherein the anchor is a first anchor, wherein the control system is further configured to update a registration transform between a reference frame of the first anchor and a reference frame of a second anchor based on the updated anchor and the second anchor, and wherein the second anchor is associated with a second computer-assisted device.

21. A method of operating an electronic system comprising a computer-assisted device and a control system communicably coupled to the computer-assisted device and to a sensor system, the sensor system configured to capture sensor data about an environment external to the computer-assisted device, the method comprising:determining, based on the sensor data, pose information of an operator portion of a first operator during one or more interactions between the operator portion and a device portion of the computer-assisted device, the pose information of the operator portion comprising at least one parameter selected from the group consisting of: a position of the operator portion during the one or more interactions and an orientation of the operator portion during the one or more interactions;determining pose information of the device portion during the one or more interactions, the pose information of the device portion comprising at least one parameter selected from the group consisting of: a position of the device portion during the one or more interactions and an orientation of the device portion during the one or more interactions;updating, based on the pose information of the operator portion and the pose information of the device portion, an anchor that is shared by a first extended reality (XR) system and a second XR system, the first XR system configured to render images for the first operator and the second XR system configured to render images for a second operator; and causing the second XR system to render an image viewable by the second operator using the updated anchor.

22. The method of claim 21, wherein updating the anchor comprises:Attorney Docket No. P06828-WO:0138PCupdating a first registration transform between a first reference frame of the first XR system and a second reference frame of the anchor based on the pose information of the operator portion and the pose information of the device portion; andupdating the anchor based on the updated first registration transform; and updating a second registration transform between a third reference frame of the second XR system and the second reference frame of the anchor based on the updated anchor.

23. The method of claim 21, further comprising selecting the one or more interactions based on a priority associated with the one or more interactions and a second priority associated with a second interaction, the second interaction being between a second operator portion and a second device portion of the computer-assisted device.

24. The method of claim 21, further comprising selecting the one or more interactions based on a confidence score associated with the one or more interactions and a second confidence score associated with a second interaction between a second operator portion and a second device portion of the computer-assisted device.

25. The method of claim 24, wherein the confidence score associated with the one or more interactions indicates a predicted accuracy of a candidate registration transform between a first reference frame of the first XR system and a second reference frame of the anchor, and wherein the candidate registration transform is determined based on the one or more interactions.

26. The method of claim 21, further comprising selecting the one or more interactions based on a temporal length of the one or more interactions.

27. The method of claim 21, further comprising selecting the one or more interactions based on a type of the one or more interactions.

28. The method of claim 21, further comprising selecting the one or more interactions based on a timing of the one or more interactions relative to a second interaction between a second operator portion and a second device portion of the computer-assisted device.Attorney Docket No. P06828-WO:0138PC29. The method of claim 21, further comprising selecting the one or more interactions based on the device portion of the computer-assisted device.

30. The method of claim 21, further comprising selecting the one or more interactions based on an identity or a role of the first operator.

31. The method of claim 21, further comprising selecting the one or more interactions based on the first XR system.

32. The method of claim 21, further comprising selecting the one or more interactions based on an application for which the image is rendered.

33. The method of any one of claims 21 to 32, further comprising causing the first XR system to render an image viewable by the first operator using the updated anchor.

34. The method of any one of claims 21 to 32, wherein the anchor is a first anchor, and wherein the first anchor is not shared by a third XR system configured to render images for a third operator using a second anchor.

35. The method of claim 34, further comprising, in response to a selection to use the updated anchor with the third XR system, update the second anchor based on the updated anchor.

36. The method of any one of claims 21 to 32, wherein the anchor is a first anchor, and the method further comprising updating a second anchor based on the updated anchor and a registration transform between a reference frame of the first anchor and a reference frame of the second anchor, wherein the second anchor is associated with a second computer-assisted device.

37. The method of any one of claims 21 to 32, wherein the anchor is a first anchor, and the method further comprises updating a registration transform between a reference frame of the first anchor and a reference frame of a second anchor based on the updated anchor and the second anchor, wherein the second anchor is associated with a second computer-assisted device.Attorney Docket No. P06828-WO:0138PC38. One or more non-transitory machine-readable media comprising a plurality of machine-readable instructions which when executed by one or more processors are adapted to cause the one or more processors to perform the method of any one of claims 21 to 37.

Images