Techniques for updating registration transformation between augmented reality system and computer-aided device

Abstract

Technologies for updating registration of an augmented reality (XR) system with a computer-aided device include the following. A control system communicatively coupled to the computer-aided device and the sensor system is configured to: determine gesture information of an operator portion of an operator based on sensor data during one or more interactions between the operator portion and a device portion of the computer-aided device; determining gesture information for the device portion during the one or more interactions; updating a registration transformation between the XR system and the computer-aided device based on the gesture information of the operator portion and the gesture information of the device portion; and causing the XR system to present an image viewable by the operator using the updated registration transformation.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 403,573, filed on September 2, 2022, entitled “Techniques For Updating A Registration Transform Between An Extended-Reality System And A Computer-Assisted Device,” which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to electronic devices and, more particularly, to techniques for updating a registration transform between an augmented reality system and a computer-assisted device. Background Art

[0004] Computer-assisted electronic systems are used more and more frequently. This is especially true in industrial, entertainment, educational and other environments. As a medical example, today's medical facilities are found to have a large number of computer-assisted devices in operating rooms, interventional suites, intensive care units, emergency rooms, etc. Many of these computer-assisted devices are movable and can be capable of autonomous or semi-autonomous movement. It is also known that staff use one or more input devices located at the user control system to control the movement and / or operation of movable computer-assisted devices. As a specific example of an electronic system including a computer-assisted device, a minimally invasive robotic telesurgery device allows a surgeon to operate on a patient from a bedside or remote site. Telesurgery generally refers to a surgical operation performed using a surgical device, in which the surgeon uses some form of remote control (such as a servo mechanism) to manipulate the movement of the surgical instrument, rather than directly holding and moving the instrument by hand.

[0005] Extended reality (XR) systems are often used in conjunction with computer-assisted devices to perform tasks at work sites. Examples of XR systems include augmented reality (AR) devices and virtual reality (VR) devices. As used herein, AR refers to a view of a physical environment with an overlay of one or more computer-generated graphical elements, including a mixed reality (MR) environment in which physical objects and computer-generated elements can interact. As used herein, VR refers to a virtual environment including computer-generated elements. For example, an XR system may present data about the operating environment of a computer-assisted device or guidance during the operation of a computer-assisted device, etc. In order to display XR content in an appropriate position and / or orientation relative to the computer-assisted device, a registration between the XR system and the computer-assisted device is determined. Registration establishes a geometric relationship between part or all of the XR system and part or all of the computer-assisted device. When an XR system is used to provide XR content relative to the position and / or orientation of a computer-assisted device in space, such registration allows the position and orientation of the computer-assisted device to be determined relative to the XR system.

[0006] Conventional methods for determining registration between an XR system and an object in an environment (such as a computer-assisted device) may be less accurate. As a result, the XR system may display XR content in an incorrect position and / or orientation relative to the computer-assisted device.

[0007] Therefore, there is a need for improved techniques for registering XR systems with objects in the environment, such as computer-aided devices. Summary of the invention

[0008] According to some embodiments, an electronic system includes a computer-assisted device and a control system communicatively coupled to the computer-assisted device and a sensor system configured to capture sensor data about an environment external to the computer-assisted device. The control system is configured to: during one or more interactions between an operator portion of an operator and a device portion of the computer-assisted device, determine gesture information of the operator portion based on the sensor data, the gesture information of the operator portion including at least one parameter selected from the group consisting of: a position of the operator portion during the one or more interactions and an orientation of the operator portion during the one or more interactions, determine gesture information of the device portion during the one or more interactions, the gesture information of the device portion including at least one parameter selected from the group consisting of: a position of the device portion during the one or more interactions and an orientation of the device portion during the one or more interactions, update a registration transformation between an extended reality (XR) system and the computer-assisted device based on the gesture information of the operator portion and the gesture information of the device portion, and cause the XR system to present an image viewable by the operator using the updated registration transformation.

[0009] According to some embodiments, a method of operating an electronic system includes a computer-assisted device and a control system communicatively coupled to the computer-assisted device and a sensor system, the sensor system being configured to capture sensor data about an environment external to the computer-assisted device, and the method includes: determining, during one or more interactions between an operator portion of an operator and a device portion of the computer-assisted device, gesture information of the operator portion based on the sensor data, the gesture information of the operator portion including at least one parameter selected from the group consisting of: a position of the operator portion during the one or more interactions and an orientation of the operator portion during the one or more interactions; determining, during the one or more interactions, gesture information of the device portion, the gesture information of the device portion including at least one parameter selected from the group consisting of: a position of the device portion during the one or more interactions and an orientation of the device portion during the one or more interactions; updating a registration transformation between an extended reality (XR) system and the computer-assisted device based on the gesture information of the operator portion and the gesture information of the device portion; and causing the XR system to present an image viewable by the operator using the updated registration transformation.

[0010] Other embodiments include, but are not limited to, one or more non-transitory machine-readable media comprising a plurality of machine-readable instructions that, when executed by one or more processors, are adapted to cause the one or more processors to perform any of the methods disclosed herein.

[0011] The above general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art through the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a simplified diagram of an example including a computer-assisted device and an XR system, according to various embodiments.

[0013] Figure 2 is a perspective view illustrating an extended reality (XR) system according to various embodiments.

[0014] Figure 3 In more detail, the various embodiments are shown Figure 1 control module.

[0015] Figure 4 A simplified diagram illustrating a method for updating a registration transform between an XR system and a computer-assisted device based on one or more operator interactions in accordance with various embodiments.

[0016] Figure 5 A simplified diagram illustrating a method for updating a registration transform between an XR system and a computer-assisted device in accordance with various embodiments.

[0017] Figure 6 Examples of operator interaction for updating a registration transform between an XR system and a computer-assisted device are shown in accordance with various embodiments.

[0018] Figure 7 Another example of operator interaction for updating a registration transform between an XR system and a computer-assisted device in accordance with various embodiments is shown. DETAILED DESCRIPTION

[0019] The present specification and drawings illustrating aspects, embodiments, embodiments or modules of the invention should not be construed as limiting - the claims define the invention to be protected. Various mechanical, compositional, structural, electrical and operational changes may be made without departing from the spirit and scope of the present specification and claims. In some cases, well-known circuits, structures or techniques are not shown or described in detail in order not to obscure the present invention. The same numbers in two or more figures represent the same or similar elements.

[0020] In this specification, the specific details of some embodiments consistent with the present disclosure are described. In order to provide a thorough understanding of the embodiments, many specific details are described. However, it is obvious to those skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative rather than restrictive. Those skilled in the art may implement other elements, although not specifically described here, within the scope and spirit of the present disclosure. In addition, in order to avoid unnecessary repetition, one or more features shown and described in association with an embodiment may be incorporated into other embodiments, unless otherwise specifically described, or if one or more features will make the embodiment inoperative.

[0021] In addition, the terms in this specification are not intended to limit the present invention. For example, as shown in the figure, spatial relative terms (such as "below", "below", "lower", "above", "upper", "near side", "distal side", etc.) can be used to describe the relationship between an element or feature and another element or feature. In addition to the position and orientation shown in the figure, these spatial relative terms are intended to cover the different positions (i.e., sites) and orientations (i.e., rotational arrangements) of elements or their operations. For example, if the content of one of the figures is flipped, the elements described as "below" or "below" other elements or features will be "above" or "on" other elements or features. Similarly, the description of movement along and around various axes includes various special element positions and orientations. In addition, unless the context otherwise indicates, the singular forms "one", "one" and "the" also include plural forms. Moreover, the terms "include", "comprise", "contain", etc. specify the presence of the features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts and / or groups. The parts described as coupled can be directly electrically coupled or mechanically coupled, or can be indirectly coupled via one or more intermediate parts.

[0022] Whenever feasible, elements described in detail with reference to one embodiment, embodiment or module may be included in other embodiments, embodiments or modules that do not specifically show or describe these elements. For example, if an element is described in detail with reference to one embodiment, but not described with reference to a second embodiment, the element may still be required to be included in the second embodiment. Therefore, in order to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, embodiment or application may be incorporated into other embodiments, embodiments or aspects, unless otherwise specifically stated, unless one or more elements will make an embodiment or embodiment inoperative, or unless two or more of the elements provide conflicting functions.

[0023] In some instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0024] The present disclosure describes various elements (such as systems and devices, and parts of systems and devices) by examples in three-dimensional space. In such examples, the term "position" refers to the position of an element or a part of an element in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). Also in such examples, the term "orientation" refers to the rotational arrangement of an element or a part of an element (three rotational degrees of freedom, such as roll, pitch, and yaw). Other examples may include other dimensional spaces, such as two-dimensional space. As used herein, the term "posture" refers to the position, orientation, or combination of position and orientation of an element or a part of an element. As used herein, and for an element or a part of an element, such as a device (e.g., a computer-aided device or a repositionable arm), the term "proximal" for an element in a kinematic chain refers to the direction toward the base of the kinematic chain, and the term "distal" refers to the direction away from the base along the kinematic chain.

[0025] Some aspects of the present disclosure are described with reference to electronic systems and computer-assisted devices, which may include systems and devices that are teleoperated, telecontrolled, autonomous, semi-autonomous, robotic, etc. In addition, aspects of the present disclosure are described in terms of embodiments using a medical system, such as the da However, those skilled in the art will appreciate that the inventive aspects disclosed herein may be embodied and implemented in a variety of ways, including robotic and non-robotic (if applicable) embodiments. The embodiments described in the surgical system are merely exemplary and should not be considered to limit the scope of the invention disclosed herein. For example, the techniques described with reference to surgical instruments and surgical methods can be used in other contexts. Therefore, the instruments, systems and methods described herein can be used for humans, animals, parts of human or animal anatomical structures, industrial systems, general robots or remote operating systems. As a further example, the instruments, systems and methods described herein can be used for non-medical purposes, including industrial uses, general robotic uses, sensing or manipulating non-tissue artifacts, cosmetic improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, setting up or removing systems, training medical or non-medical personnel, etc. Other example applications include performing procedures on tissues removed from human or animal anatomical structures (whether or not returned to human or animal anatomical structures), and performing procedures on human or animal corpses. In addition, these technologies can also be used for medical treatment or diagnostic procedures that include or do not include surgical aspects.

[0026] System Overview

[0027] Figure 1 1 is a simplified diagram of an electronic system 100 according to various embodiments. As shown, the electronic system 100 includes, but is not limited to, a computer-assisted device 110 and an extended reality (XR) system 130. The computer-assisted device 110 includes a repositionable structure having a repositionable arm 120 (such as a manipulator arm) configured to support an instrument 122. Although one repositionable arm 120 supporting one instrument 122 is shown for illustrative purposes, the repositionable structure of the computer-assisted device may include any suitable number of repositionable arms, and each repositionable arm may be configured to support one instrument or multiple instruments.

[0028] In some examples, the instrument 122 is an imaging instrument, a manipulation instrument (such as a grasper or scissors), a fastening instrument (such as a stapler), an irrigation instrument, a suction instrument, an energy application instrument, or any other suitable instrument. In some examples, the instrument 122 is an imaging instrument, such as a monoscopic or stereoscopic camera, a still or video camera, an endoscope, a hyperspectral device, an infrared or ultrasonic device, an ultrasound device, a fluoroscope device, etc. In some examples, the instrument 122 is a medical instrument, such as a medical endoscope, forceps, a clip applier, a gripper, a retractor, a cauterization instrument, a suction instrument, a suturing device, an anastomosis device, a cutting device, etc. In some examples, the instrument 122 includes an end effector capable of performing one or more tasks, such as grasping a material located in a workspace (e.g., tissue of a patient in a medical example) and delivering energy to the grasped material. In some examples, the energy includes ultrasonic energy, radio frequency energy, electrical energy, magnetic energy, thermal energy, light energy, and / or other types of energy. In some examples, the repositionable arm 120 includes one or more joints and the instrument 122. In some examples, instrument 122 is inserted during use through a cannula, access port, etc. into a working space (eg, in some medical examples, the anatomy of a patient or cadaver, a veterinary subject, an anatomical model, etc.) to perform a procedure.

[0029] In some examples, the computer-assisted device 110 is a remotely operated device. In some medical examples, the remotely operated device is a remotely operated medical device, such as a remote surgical device, which can be found in an operating room and / or an interventional room. In some examples, the computer-assisted device 110 is a follower device that is remotely operated by controlling one or more guide devices (not shown) (such as one or more input devices (not shown) designed to be contacted and manipulated by an operator). One or more input devices can be mechanically grounded (kinematically grounded through a mechanical structure) or mechanically ungrounded (kinematically grounded without a mechanical structure). A system including a guide device and a follower device is called a guide-follower system, and is sometimes also referred to as a master-slave system. When the computer-assisted device 110 is a remotely operated follower device including a repositionable arm 120 and / or an instrument 122 supported by a repositionable arm, and is controlled to move and perform joint movements in response to the operator's manipulation of (one or more) guide devices, the computer-assisted device 110 "follows" (one or more) guide input devices through remote operation. Thus, an operator is able to use the repositionable arm 120 and / or instrument 122 to perform tasks at a work site.

[0030] In some embodiments, XR system 130 is an augmented reality (AR) device or a virtual reality (VR) device. Figure 2The XR system 130 is described in more detail. In some examples, the XR system 130 is used in conjunction with the computer-assisted device 110 to perform a task at a work site. For example, the XR system 130 can be used to present instructional content on how to operate the computer-assisted device 110. As another example, the XR system 130 can be used to present content that provides guidance during the operation of the computer-assisted device 110. Exemplarily, the XR system 130 displays AR content 136 next to the repositionable arm 120 of the computer-assisted device 110. The AR content 136 can be provided in any manner suitable for visual AR content, such as a visual overlay. In order to display the XR content (e.g., AR content 136) presented by the XR system 130 in an appropriate position and orientation relative to the computer-assisted device 110, a registration transformation 134 between the XR system 130 and the computer-assisted device 110 is established. The registration transformation 134 provides a geometric relationship between the XR system 130 and the computer-assisted device 110. By using registration transform 134, XR content displayed relative to a portion of computer-assisted device 110 is displayed at or near a location in reference system 132 of XR system 130 that is determined based on the location of the portion of computer-assisted device 110 in reference system 124 of computer-assisted device 110. For example, to display AR content 136, the location (and optional orientation) of the portion of computer-assisted device 110 in reference system 124 of computer-assisted device 110 is mapped to a corresponding location (and optional orientation) in reference system 132 of XR system 130 using registration transform 134. AR content 136 may be displayed near the corresponding location of the portion of repositionable arm 120 in reference system 132. The location of repositionable arm 120 in reference system 124 of computer-assisted device 110 is determined using kinematics or in any other technically feasible manner (e.g., using sensors that track the location of repositionable arm 120, such as gyroscopes, IMUs, shape sensors, etc.).

[0031] Exemplarily, reference frame 124 is a base reference frame of computer-assisted device 110. In some examples, reference frame 124 has an origin located at a center point on the base of computer-assisted device 110 and is aligned with one or more major axes of computer-assisted device 110. In some examples, the center point on the base of computer-assisted device 110 is located on a horizontal planar surface on which computer-assisted device 110 is pushed, slid, and / or otherwise repositioned. In some examples, the z-axis of reference frame 124 corresponds to a vertical upward direction. As shown, reference frame 132 is a base reference frame of XR system 130. In some examples, reference frame 132 has an origin located at a point within XR system 130 and is oriented in any technically feasible manner. In some examples, the z-axis of reference frame 124 corresponds to the viewing direction of XR system 130, the y-axis of reference frame 124 corresponds to the view upward direction, and the x-axis in reference frame 124 is orthogonal to both the z-axis and the y-axis of reference frame 124.

[0032] In some examples, a six-degree-of-freedom (6DOF) registration transformation 134 is used to characterize the geometric relationship between reference system 124 and reference system 132. For example, the registration transformation 134 may include a three-degree-of-freedom (DOF) rotation portion and a 3DOF translation portion. In this case, the rotation portion can be represented by a 3x3 matrix describing the three-dimensional rotation difference between reference system 124 and reference system 132. In some examples, the rotation portion describes the rotation around three axes of reference system 124. In some examples, the three axes correspond to the x-axis, y-axis, and z-axis; roll axis, pitch axis, and yaw axis, etc. In some examples, the rotation portion is represented using angle-axis, quaternion, and / or similar equivalent symbols. In some examples, the translation portion can be represented by a 3x1 vector describing the three-dimensional displacement between reference system 124 and reference system 132.

[0033] As shown, the control system 140 is provided external to the computer-assisted device 110 and the XR system 130, and the control system 140 communicates with both the computer-assisted device 110 and the XR system 130. In other embodiments, the control system 140 is part of the computer-assisted device 110 or part of the XR system 130. In some embodiments, the control system 140 generates XR content presented by the XR system 130. In some embodiments, the control system 140 also determines or provides control signals to the computer-assisted device 110 based on the received information and operator input to control the movement of the repositionable arm 120 and / or the instrument 122. In some embodiments, the control system 140 supports one or more wired communication protocols (e.g., Ethernet, USB, etc.) and / or one or more wireless communication protocols (e.g., Bluetooth, IrDA, HomeRF, IEEE 1002.11, DECT, wireless telemetry, etc.).

[0034] The control system 140 is implemented on one or more computing systems. As shown, the control system 140 includes a processor 150 and a memory 160 storing a control module 170. In some embodiments, the control system 140 includes one or more processors that execute the control module 170, non-persistent storage devices (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage devices (e.g., hard disk, optical drive, such as compact disk (CD) drive or digital versatile disk (DVD) drive, flash memory, etc.), communication interfaces (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), and many other elements and functions. Although this article is mainly described with respect to the control module 170 running on the control system 140, the functions of the control module 170 can be implemented in any technically feasible software and / or hardware, including partially or completely within the XR system 130, within another computing system, within a cloud computing system, etc.

[0035] Each of the one or more processors of the control system 140 is an integrated circuit for processing instructions. For example, the one or more processors may be one or more cores or micro-cores of a processor, a central processing unit (CPU), a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), a tensor processing unit (TPU), etc. In some examples, the control system 140 also includes one or more input devices (not shown), such as a touch screen, a keyboard, a mouse, a microphone, a touchpad, an electronic pen, or any other type of input device.

[0036] The communication interface of the control system 140 may include an integrated circuit for connecting the computing system to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or another device (such as another computing system).

[0037] In some examples, control system 140 also includes one or more output devices (not shown), such as a display device (e.g., a liquid crystal display (LCD), a plasma display, a touch screen, an organic LED display (OLED), a projector, or other display device), a printer, a speaker, an external storage device, or any other output device. One or more of the output devices may be the same as or different from the input device(s). There are many different types of computing systems, and the input and output devices described above may take other forms.

[0038] In some embodiments, control system 140 is connected to a network or is a part of a network. The network may include multiple nodes. Control system 140 is implemented on a node or a group of nodes. As an example, control system 140 is implemented on a node of a distributed system connected to other nodes. As another example, control system 140 is implemented on a distributed computing system with multiple nodes, wherein different functions and / or components of control system 140 are located on different nodes in the distributed computing system. In addition, one or more elements of the above-mentioned control system 140 can be located at a remote site and connected to other elements via a network.

[0039] The software instructions in the form of computer-readable program code for executing the embodiments of the present disclosure are stored in whole or in part, temporarily or permanently on a non-transient computer-readable medium, such as a CD, DVD, storage device, disk, tape, flash memory, physical memory, or any other computer-readable storage medium. Specifically, the software instructions may correspond to a computer-readable program code that, when executed by (one or more) processors (e.g., processor 150), is configured to perform some embodiments of the methods described herein.

[0040] Some embodiments include remotely operating one or more components of a medical system, such as the da Surgical systems. About da The embodiments of the surgical system are merely examples and should not be considered to limit the scope of the features disclosed herein. For example, different types of teleoperation systems having a computer-assisted device including a follower device configured to be placed at a work site can use the features described herein. In addition, non-teleoperation systems can also utilize the features described herein.

[0041] Figure 2 is a perspective view showing head mounted XR system 130 in more detail according to various embodiments. Figure 2 As shown, the XR system 130 includes a body 205 and a head mount 210. The body 205 includes one or more electronic display elements of an electronic display 230. The body 205 also includes a sensor system 240 that acquires sensor data associated with a physical environment external to the XR system 130, which may also be external to any object in the physical environment (such as a computer-aided device). The sensor system 240 may include any technically feasible one or more sensors, such as monoscopic and stereoscopic optical systems, ultrasound systems, depth cameras (such as cameras using time-of-flight sensors), LIDAR (light detection and ranging) sensors, stereo RGB (red, green, blue) sensors, RGB-D depth sensors, etc. Although Figure 2 A head mounted XR system is shown, but other XR systems may be used in other embodiments. Examples of other types of XR systems include appropriately configured tablets, smartphones, projectors, etc. Although the sensor system 240 is Figure 2 Although shown as being included in XR system 130, a sensor system for providing sensor data associated with a physical environment external to XR system 130 (which may also include data external to one or more objects in the physical environment) may be provided at any suitable location. For example, some or all of such a sensor system may alternatively or additionally be located elsewhere in the physical environment, including being mounted on a wall, ceiling, or bracket, or coupled to a computer-aided device.

[0042] In addition, the body 205 includes an electronic display 230 and an optical module 235, which together provide image light to a target site of the body 205, where the operator's eye 202 can be located. In some examples, the body 205 also includes one or more other sensors, such as one or more imaging devices (e.g., one or more imaging sensors for tracking the eye 220), accelerometers and / or angular velocity sensors (which can be part of an inertial measurement unit (IMU)), position sensors, and / or other sensors.

[0043] The electronic display 230 is configured to display an image viewable by the operator. In various embodiments, the electronic display 230 includes a single electronic display or multiple electronic displays (e.g., a display for each eye of the operator). Examples of the electronic display 230 include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light emitting diode display (AMOLED), QOLED, QLED, some other display, or some combination thereof. The optical module 235 includes an optical element that can be used to adjust the orientation of the image light emitted from the electronic display 230 so that the electronic display 230 appears to be at a specific virtual image distance from the operator.

[0044] In some embodiments, the XR system 130 operates as an AR device that uses an electronic display 230 to present computer-generated media to an operator that augments the operator's view of a physical real-world environment visible through the electronic display 230. Examples of computer-generated media presented by the XR system 130 include one or more images, videos, audio, or some combination thereof. Alternatively or additionally, in some embodiments, the XR system 130 operates as a VR device or some combination of an AR device and a VR device, such as a device that allows switching between an AR environment and a VR environment. As described above, MR content provides an AR environment in which physical objects and computer-generated elements can interact, and VR content provides a virtual environment that includes computer-generated elements. In the case of a VR device, the sensor system 240 can capture images of the physical environment and display the captured images together with computer-generated elements (such as AR content 136), which is sometimes also referred to as video see through. Examples of commercial AR devices include Microsoft Google and Meta Examples of commercial MR devices include Microsoft HoloLens Samsung HP and Oculus Quest Examples of commercial VR devices include Oculus Samsung Gear HTC and Google Daydream

[0045] It should be noted that Figure 2Only example configurations of XR systems are shown. In some embodiments, the techniques disclosed herein for updating the registration transformation between an XR system and a computer-aided device can be used for other configurations and / or types of XR systems. Other configurations and / or types of XR systems include head-mounted XR systems, handheld XR systems, and XR systems placed in an environment, etc. Examples of alternative configurations and / or types of XR systems include optical head-mounted displays (HMDs), mobile devices (e.g., mobile phones, tablet computers, etc.), fully immersive projection systems, etc.

[0046] Update registration transformation between extended reality system and computer-aided device

[0047] The registration between the XR system and the computer-assisted device may be updated based on a relationship between gesture information of a portion of the operator and gesture information of a portion of the computer-assisted device determined during interaction between the operator using the XR system and the computer-assisted device.

[0048] Figure 3 In more detail, the various embodiments are shown Figure 1 control module 170. As shown, the control module 170 includes, but is not limited to, a sensor data processing module 306, a kinematic evaluation module 308, a registration module 310, an overlay module 312, and a synthesis module 316. During operation, the sensor data processing module 306 receives sensor data 302 acquired by a sensor system (e.g., sensor system 240) and determines posture information of a portion of an operator interacting with a portion of a computer-assisted device (e.g., computer-assisted device 110) during operator interaction. In some embodiments, the posture information of the operator includes one or more position parameters, or one or more position parameters and one or more orientation parameters. In some embodiments, a human posture estimation technique is used to determine the posture information of a portion of the operator, as described below in combination with Figure 4 Alternatively or additionally, the sensor data processing module 306 may determine posture information of an extension of a portion of the operator, such as an object of known size held by the operator (eg, a cable, a tool, etc.).

[0049] During operation, the posture assessment module 308 receives kinematic data 304 associated with joints and / or links of a repositionable structure of a computer-assisted device (such as the computer-assisted device 110) and determines posture information of a portion of the computer-assisted device. In some embodiments, the posture information of a portion of the computer-assisted device includes one or more position parameters, or one or more position parameters and one or more orientation parameters. The posture information of a portion of the repositionable structure is during an interaction between a portion of an operator and a portion of the repositionable structure. In some embodiments, the kinematic data 304 can be acquired by a joint sensor, a shape sensor, and / or in any other technically feasible manner, wherein the joint sensor transmits the position and orientation of the joints of the repositionable structure to record its movement, and the shape sensor monitors the shape of an optical fiber to determine the posture of the repositionable structure at one end of the optical fiber relative to a reference system at the other end of the fiber. When a portion of the operator ("operator portion") interacts with a portion of the repositionable structure of the computer-assisted device 110 ("device portion"), the posture assessment module 308 uses the kinematic data 304 and one or more kinematic models and / or three-dimensional (3D) models of the repositionable structure to determine the posture information of the portion of the repositionable structure. In some embodiments, kinematic data 304 is synchronized with sensor data 302 so that a comparison can be made between posture information determined using sensor data 302 and posture information determined using kinematic data 304 and corresponding to the same point in time. Although this document is primarily discussed with respect to examples where a portion of a computer-assisted device is a portion of a repositionable structure, in some embodiments, the techniques disclosed herein can be applied to situations where a portion of a computer-assisted device is another portion of a computer-assisted device or is a portion of a computer-assisted device that does not include a repositionable structure. When a portion of an operator interacts with a portion of a computer-assisted device other than a repositionable structure (e.g., a rudder, a handle, a base, etc. of a computer-assisted device), posture information of a portion of the computer-assisted device can be determined based on the posture of the entire computer-assisted device and the known position and orientation of the portion relative to the entire computer-assisted device.

[0050] During operation, registration module 310 determines a registration transform between XR system 130 and computer-assisted device 110. The registration transform may be between any portion of XR system 130 and computer-assisted device 110. Internal transforms between portions of computer-assisted device 110 may be determined using known or sensed geometry of computer-assisted device 110. Thus, when the registration transform between a portion of XR system 130 and computer-assisted device 110 is known, these internal transforms may be used to determine a transform defining a relative position and / or orientation between other portion(s) of XR system 130 and computer-assisted device 110. In some embodiments, registration module 310 updates a current registration transform based on operator interaction with computer-assisted device 110. In some embodiments, registration module 310 (i) determines an initial registration transform according to appropriate technique(s), and (ii) updates the current registration transform (which will initially be the initial registration transform) based on one or more operator interactions with computer-assisted device 110. In some examples, determining the initial registration transform includes prompting the operator to view, touch, and / or move one or more portions of the computer-assisted device 110. As used herein, a prompt may include any suitable visual and / or audio indication of (one or more) operator actions to be performed, such as a graphic, a text message, a flashing light, an audio instruction, etc. For example, the operator may be prompted to view and / or interact with a particular portion of the computer-assisted device 110. As another example, the operator may be prompted to perform a set of interactions and / or complex operator interactions with a portion of the computer-assisted device 110, such as moving a manipulator through a series of movements while viewing the portion of the computer-assisted device 110. As the operator views, touches, and / or moves one or more portions of the computer-assisted device 110, an imaging device in the sensor system 240 of the XR system 130 captures images of the one or more portions of the computer-assisted device 110. The initial registration transformation is determined by calculating (one or more) initial values ​​of the registration transformation 134 between the reference system 124 of the computer-assisted device 110 and the reference system 132 of the XR system 130 based on the position and orientation of the computer-assisted device 110, which in some examples can be determined using a model-target-based technique. For example, the model-target-based technique can reconstruct a 3D model of the computer-assisted device 110 from the image data using photogrammetry techniques, and then match the reconstructed model with a predefined model of the computer-assisted device 110 to determine the position and orientation of the computer-assisted device 110. In some other examples, determining the initial registration transformation includes capturing images of one or more portions of the computer-assisted device on which fiducial markers (e.g., QR codes, images, markers encoding data, etc.) have been placed.In this case, the initial registration transform may be calculated based in part or in whole on the position and / or orientation of the fiducial markers in the captured images. In some examples, the above-described computer vision-based registration techniques and marker-based registration techniques may be combined to determine the initial registration transform. In some examples, any other technically feasible technique may be used to determine the initial registration transform.

[0051] The initial registration transform determined using model-target-based, marker-based, and / or other registration techniques may be inaccurate. What is considered inaccurate may depend on the type of XR content to be displayed, the operating context of the computer-assisted device, and / or the operating context of the XR system. For example, for typical human interactions, an operator may perceive XR content superimposed to a particular feature in linear or angular alignment (such as when an overlay depicting a portion of a computer-assisted device is superimposed directly on a portion of the computer-assisted device) to be displayed with inaccurate registration when the XR content deviates from the desired position by approximately 1 centimeter or more at an effective distance of 1 meter from the operator's eyes and / or deviates from the desired position by 5 degrees or more on any rotational axis. As a further example, XR content superimposed for eye surgery or fine crafting may be considered to be displayed with inaccurate registration at much smaller deviations. As another example, an operator may perceive “floating” XR content that is not aligned overlaid on any particular object or feature of an object (e.g., offset on one side of a computer-assisted device) as being displayed with accurate registration, even though the XR content appears angularly or linearly misaligned by more than a few centimeters at an effective distance of 1 meter from the operator (assuming the XR content does not intrude into the visual space of the computer-assisted device). As another example, an operator may perceive XR content to be inaccurately registered at smaller deviations for a head-mounted device compared to a handheld device (e.g., a handheld screen or tablet). Furthermore, an initially accurate initial registration may become more inaccurate over time due to drift, movement, sensor inaccuracies, etc. When the initial registration transform is inaccurate, the XR system 130 may display the XR content at a position and / or orientation that is perceived to be incorrect relative to the computer-assisted device 110. To improve the registration accuracy, the registration module 310 updates the initial registration transform based on the operator's interaction with the computer-assisted device 110.

[0052] When one or more parts of the operator interact with one or more parts of the repositionable structure of the computer-assisted device 110, one or more interaction points and / or interaction areas are determined in the reference frame 132 of the XR system 130. Each interaction point or interaction area can be an actual contact point or contact area between a part of the operator or an extension thereof (glove, shoe, cable, pointing stick, etc.) and a part of the computer-assisted device, or an estimate thereof. Each interaction point or interaction area is determined in both the reference frame 132 of the XR system 130 and the reference frame 124 of the computer-assisted device 110. The interaction points or interaction areas can be determined in the reference frame 132 using human body pose estimation techniques described in more detail below and sensor data acquired by the sensor system 240 of the XR system 130. The interaction points or interaction areas may be determined in the reference frame 124 of the computer-assisted device 110 by: (1) using the kinematics and known joint locations of the repositionable structure of the computer-assisted device 110 when the operator interacts with a portion of the repositionable structure, or (2) based on the pose of some other portion of the computer-assisted device or the entire computer-assisted device 110 and the known position and orientation of that portion relative to the entire computer-assisted device 110 when the operator interacts with a portion of the computer-assisted device 110 other than the repositionable structure. Alternatively, any technically feasible technique, including non-kinematic techniques (e.g., sensors that directly track the motion of the portion(s) of the repositionable structure) may be used to determine the interaction points(s) and / or interaction areas with the repositionable structure. The determined interaction points(s) and / or interaction areas are then mapped to a common reference frame (e.g., the reference frame 124 of the computer-assisted device 110, the reference frame 132 of the XR system 130, etc.) using a registration transform. An error is determined between the interaction point(s) and / or interaction region in the common reference frame. An update to the initial registration transformation is then calculated based on the error, as described below in conjunction with Figure 4-Figure 5As discussed in more detail herein. For example, assume that the interaction involves an operator pressing a mechanical button on a repositionable structure of the computer-assisted device 110. In this case, a human pose estimation technique is used to detect the position of the operator's finger tip touching the button in the reference frame 132 of the XR system 130 based on the image acquired by the sensor system 240, which corresponds to the position of the button. A registration transformation is then applied to map the position of the finger / button in the reference frame 132 of the XR system 130 (e.g., pose information) to the position in the reference frame 124 of the computer-assisted device 110. Specifically, the inverse of the registration transformation can be used to map the position of the finger / button in the reference frame 132 of the XR system 130 to the position in the reference frame 124 of the computer-assisted device 110. Thereafter, the position in the reference frame 124 of the computer-assisted device 110 is compared with the position of the button in the reference frame 124 determined using kinematics or in any other technically feasible manner (e.g., using a sensor such as a gyroscope, an IMU, a shape sensor, etc.). The error between the positions is then calculated and used to determine the gradient descent update to the initial registration. Alternatively, the registration transformation is applied to map the positions in the reference frame 124 of the computer-assisted device 110 to the positions in the reference frame 132 of the XR system 130, and the error is calculated in the reference frame 132 to determine the gradient descent update.

[0053] During operation, the overlay module 312 generates XR content. As described above, in some embodiments, the XR content includes AR, MR, and / or VR content. Any suitable XR content is generated, such as the above combined Figure 1AR content 136 described. For example, the XR content may include instructional content about how to operate a computer-assisted device. As another example, the XR content may include content that provides data related to a program being executed by the computer-assisted device, such as previously captured images, models, images captured in real time, data about the functionality of the computer-assisted device, communications from others, tutorials or videos, guidance during operation of the computer-assisted device, etc. In order to display the XR content spatially relative to a portion of the computer-assisted device 110, the overlay module 312 applies the registration transformation 134 to map the known position of the portion of the computer-assisted device 110 in the reference system 124 of the computer-assisted device 110 to a corresponding position in the reference system 132 of the XR system 130. The overlay module 312 then generates the XR content for display at or near the corresponding position in the reference system 132. The position of the portion of the computer-assisted device 110 in the reference frame 124 is known (e.g., the known position of a rudder, handle, base, or other portion that is not part of the repositionable structure of the computer-assisted device 110), determined using kinematics, or determined in any other technically feasible manner (e.g., using sensors that track the position of the portion of the computer-assisted device 110, such as gyroscopes, IMUs, shape sensors, etc.). Thereafter, the overlay module 312 outputs a display signal 320 for displaying the XR content.

[0054] In the case of AR, the overlay module 312 generates content for display to the operator to enhance the view of the physical environment. In the case of VR, the content generated by the overlay module 312 is combined with image data depicting the physical environment to generate a composite image for display to the operator. The image data is captured by one or more imaging devices in the sensor system 240 or elsewhere. In addition, a display signal 320 is generated based on the composite image.

[0055] Figure 4 A simplified diagram of a method for updating a registration transform between an XR system and a computer-assisted device based on one or more operator interactions in accordance with various embodiments is shown. One or more of the processes 402-410 of the method 400 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors (e.g., the processor 150 in the control system 140), causes the one or more processors to perform one or more of the processes 402-440. In some embodiments, the method 400 may be performed by one or more modules, such as the control module 170. In some embodiments, the method 400 may include additional processes not shown. In some embodiments, one or more of the processes 402-410 may be performed at least in part by one or more of the modules of the control system 140.

[0056] As shown, method 400 begins at process 402, in which a registration transformation between the XR system and the computer-assisted device is initially determined. In some embodiments, the initial registration transformation is determined according to any technically feasible technique. In some examples, determining the initial registration transformation includes prompting an operator to look at, touch, and / or move one or more parts of the computer-assisted device. In some examples, initial sensor data that is not associated with interaction with the computer-assisted device is used to determine the initial registration transformation. For example, an imaging device in the sensor system 240 of the XR system 130 captures an image of one or more parts of the computer-assisted device, and a model-target-based technique is used to determine the reference frame 124 of the computer-assisted device 110 and the reference frame 132 of the XR system 130 based on the image data. As described above, the model-target-based technique can reconstruct a 3D model of the computer-assisted device 110 from the image data using photogrammetry techniques, and then match the reconstructed model with a predefined model of the computer-assisted device 110 to determine the position and orientation of the computer-assisted device 110 relative to the XR system 130. In some other examples, determining the initial registration transformation includes capturing an image of one or more portions of the computer-assisted device on which fiducial markers have been placed. In this case, the initial registration transformation can be calculated by comparing the markers in the captured image with the known positions and orientations of the markers on the computer-assisted device 110 to determine the overall position and orientation of the computer-assisted device 110 relative to the XR system 130.

[0057] In process 404, one or more interactions between an operator and a portion of a computer-assisted device (e.g., computer-assisted device 110) are detected. The one or more operator interactions may occur at any suitable time after determining the initial registration transformation. In some examples, the one or more operator interactions are detected during the operator's setup and / or normal use of the computer-assisted device. In some examples, the one or more operator interactions are detected when the operator is being trained to operate the computer-assisted device. In some examples, the one or more interactions are detected when the operator is performing an action specifically for updating the registration transformation. For example, during setup, normal use, training, and / or registration transformation update, one or more prompts may command the operator to perform one or more operator interactions. As another example, the operating state of the computer-assisted device at the time of one or more interactions or the physical configuration of the computer-assisted device at the time of one or more interactions (e.g., the initial state or configuration of the computer-assisted device) may be used to determine whether the operator is performing one or more interactions during the setup of the computer-assisted device. In this case, the interaction determined to be performed during setup may be used to update the registration transformation. As another example, the operating state of a computer-assisted device at the time of one or more interactions or the operating state of an XR system at the time of one or more interactions (e.g., the state of presenting training or other information via the XR system) can be used to determine whether an operator is performing one or more interactions during training to use the computer-assisted device or during normal use of the computer-assisted device. In this case, the interactions determined to be performed during training or normal use can be used to update the registration transformation. In some examples, the interactions that the operator is prompted to perform can change based on the registration transformation errors associated with different portions of the computer-assisted device (assuming that different registration transformations have been determined between each such portion and the XR system). For example, when the registration transformation error associated with a first portion of the computer-assisted device is greater than the registration transformation error associated with a second portion of the computer-assisted device, the operator can be prompted to perform one or more predefined interactions with the first portion of the computer-assisted device, and vice versa. In this case, the registration transformation errors associated with the first portion and the second portion of the computer-assisted device can be determined in a common reference frame, as described above in conjunction with Figure 3 As another example, when the registration transformation error is greater than a threshold for the entire computer-assisted device and / or its part(s) (e.g., due to drift, movement, sensor inaccuracy, etc.), the operator may be prompted to perform one or more predefined interactions to update the registration transformation. Again, the registration transformation error of the computer-assisted device and / or its part(s) may be determined in a common reference frame, as described above in conjunction with Figure 3As a further example, based on the registration transformation error in different parts of the computer-assisted device being greater than an error threshold, one or more operator interactions may be skipped, repeated, and / or added, or the order of the operator interactions may be changed.

[0058] The one or more operator interactions at process 404 may include a single interaction, multiple independent interactions separated in time, or a composite interaction in which multiple interactions occur simultaneously or in overlapping time periods. In addition, the interaction may be a discrete interaction occurring in a short time period, or a continuous interaction spanning a longer time period or involving the movement of a link or joint of a computer-assisted device rather than just pressing a switch. When an interaction spans a time period, the interaction may remain unchanged or change during the time period (e.g., from contacting a portion of a computer-assisted device to contacting a different portion or more than a portion of the computer-assisted device). An example of a single interaction is an operator pressing a button on a computer-assisted device. Examples of buttons include physical buttons (e.g., a power-on button, an emergency stop button) and virtual buttons (a menu selection button displayed on a touch screen of a computer-assisted device, a device control button, etc.). Another example of a single interaction is tapping one or more fingertips on a link of a computer-assisted device. An example of multiple interactions is a series of single operator interactions separated in time occurring within a defined time period, such as a finger tapping a link of a computer-assisted device multiple times before a timeout. Multiple interactions may also be separated in space, such as the same or different fingers pressing different buttons of a computer-assisted device at different times. An example of a composite interaction is multiple parts of an operator contacting different parts of a computer-assisted device simultaneously, such as different feet of an operator stepping on different pedals of a computer-assisted device simultaneously or in a temporally overlapping manner. Another example of a composite interaction is an operator grasping a first link of a computer-assisted device (such as holding the link stationary in space) while applying a force to a second link of the computer-assisted device (such as moving the second link). An example of a continuous interaction is an operator grasping a portion of a repositionable structure including a hand input sensor and moving a portion of the repositionable structure within a period of time. In this case, the continuous interaction ends when another predefined event is detected, such as when the operator releases a device control button or is no longer in contact with the computer-assisted device. As shown in the above examples, the operator interaction can include an operator interacting with a portion of a computer-assisted device or a virtual input device (e.g., touching, grasping, tapping, moving, pressing, sliding, etc.), a portion of the computer-assisted device includes a mechanical input device of the computer-assisted device, and the virtual input device is displayed on or near a portion of the computer-assisted device. Yet another example of operator interaction is an operator attaching / coupling or detaching / decoupling an object (such as an instrument, cannula, drape, etc.) to or from a computer-assisted device.

[0059] In some embodiments, in process 404, one or more operator interactions are detected by hand input sensors, foot input sensors, and / or other sensors included in the computer-assisted device and / or elsewhere in the physical environment. The hand input and / or foot input sensors may be buttons, knobs, finger detectors, joysticks, notches, pedals, etc. The hand input and / or foot input sensors may include touch sensors (e.g., inductive or capacitive touch sensors), contact sensors, pressure sensors, strain gauges, springs, etc., respectively, for sensing hand input and / or foot input.

[0060] In process 406, gesture information of a portion of an operator interacting with a portion of a computer-assisted device is determined based on sensor data acquired by a sensor system (e.g., sensor system 240). The gesture information of the portion of the operator includes a position of the portion of the operator, an orientation of the portion of the operator, or both during the operator interaction. The gesture information may be associated with an interaction point or an interaction area between the portion of the operator and the portion of the computer-assisted device, as described above in conjunction with Figure 3 In some embodiments, posture information of a portion of an operator is determined in a reference frame of the XR system (e.g., reference frame 132). In some examples, the portion of the operator is a hand or a finger. Additional examples of a portion of an operator include multiple fingers, arms, feet, and / or extensions of the above (gloves, shoes, cables, indicator sticks), etc. In some embodiments, posture information of a portion of an operator is determined in up to six DOF in three-dimensional space, the six DOF including any one or more of x, y, or z position and / or roll, pitch, or yaw orientation. In some embodiments, the posture information includes one or more position (e.g., x, y, and / or z) parameters. In some embodiments, the posture information includes one or more position (e.g., x, y, and / or z) parameters and one or more orientation (e.g., roll, pitch, and / or yaw) parameters.

[0061] In some embodiments, posture information is determined using any technically feasible human posture estimation technology. In some examples, the human posture estimation technology can be machine learning and / or other computer vision technology, which is used to detect (one or more) parts of the operator or (one or more) extensions thereof in the captured image of the interaction between the part of the operator and the computer-assisted device. In this case, a part of the operator can be modeled / recognized in various ways. For example, object segmentation and / or part segmentation techniques can be used to estimate one or more points and / or areas corresponding to a specific part of the operator's hand (such as a finger, finger tip, carpal bone, metacarpal bone, palm and / or wrist in an image acquired by an image sensor). As another example, the center of the palm position can be estimated by fitting a circle formed by two or more of the proximal joints of the fingers (e.g., the proximal joint of the thumb and the proximal joints of the index finger, middle finger, ring finger and little finger) and the wrist. In some examples, the entire contact area represented as a point cloud or mesh, such as the contact between the operator's (one or more) hands and the computer-assisted device, can be estimated. When the hand pose cannot be estimated or only a partial hand is visible, one or more points and / or regions representing finger tips and / or joints, carpal bones, metacarpal bones, palm and / or wrist, feet, etc. may be estimated by fitting a pre-posed complete hand to the partial data using partial shape matching techniques. In some examples, multiple poses or partial poses may be inferred for multiple portions of the operator within the field of view of the imaging device(s). Additional computer vision-based human pose estimation techniques are described in Sorriento et al., “Optical and Electromagnetic Tracking Systems for Biomedical Applications: A Critical Review on Potentialities and Limitations,” IEEE Rev. Biomed. Eng. 2020; 13: 212-232 (Sept. 2019) and Oudah et al., “Hand Gesture Recognition Based on Computer Vision: A Review of Techniques,” Journal of Imaging 6, no. 8: 73 (2020), which are incorporated herein by reference.

[0062] In some examples, human pose estimation techniques may employ sensor data acquired through or with the aid of a device that an operator is in contact with, such as a sensor garment worn by the operator, an object including a sensor held by the operator, etc. For example, the pose of a hand may be determined using sensor data acquired by a sensory glove. As another example, the pose of a hand may be determined based on how the operator holds a controller device including a sensor. As another example, a body tracking suit may be used in conjunction with optical camera motion capture techniques to determine the pose of an operator's entire body or a specific part of the body, such as an arm, leg, etc.

[0063] In some embodiments, gesture information of a portion of an operator interacting with a portion of a computer-assisted device is determined based on the user's eye gaze direction and / or the operator's eye convergence point, and / or the eye gaze direction and / or eye convergence point are used to supplement the gesture information of a portion of an operator determined using other techniques described herein. For example, the operator may view a portion of a computer-assisted device with which the operator interacts. As another example, the operator may view and interact with a portion of a virtual input device displayed on a portion of a computer-assisted device. Therefore, the operator's gaze direction or the operator's eye convergence point generally coincides with a portion of the computer-assisted device with which the operator is interacting. In some examples, the eye gaze direction is determined based on the gesture of one or both eyes of the operator, such as by using one or more imaging sensors or other sensors in the XR system 130, such as any one of the one or more sensors 240. In some examples, the eye convergence point is determined based on the location where the eye gaze direction of the operator's eyes converges. In some examples, when the eye gaze direction and / or eye convergence point are not focused on a portion of the computer-assisted device operator, the eye gaze direction and / or eye convergence point are not considered when determining the gesture information of a portion of the operator interacting with a portion of the computer-assisted device. In some examples, eye gaze direction and / or eye vergence point are used to determine gesture information for a portion of an operator interacting with a portion of a computer-assisted device to perform some type of interaction with the computer-assisted device (such as interaction with a virtual input device displayed on a portion of the computer-assisted device, interaction involving attaching an object to a portion of the computer-assisted device, interaction involving detaching an object from a portion of the computer-assisted device, etc.).

[0064] In some examples, the human pose estimation technique also outputs a confidence score associated with the determined pose of a portion of the operator. For example, the confidence score may indicate a probability that the determined portion and / or pose is correct.

[0065] In process 408, posture information of a portion of a computer-assisted device with which an operator interacts is determined. The posture information of the portion of the computer-assisted device includes the position of the portion of the computer-assisted device during the operator interaction, the orientation of the portion of the computer-assisted device, or both. In some embodiments, the posture information of the portion of the computer-assisted device is determined in a reference frame of the computer-assisted device (e.g., reference frame 124), and the posture information corresponds in time to the operator interaction and the posture information of the portion of the operator determined in process 406. In some embodiments, the posture information of the portion of the computer-assisted device is determined in up to six degrees of freedom in a three-dimensional space of a rigid body (e.g., a button, a switch, a portion of a screen, a portion of a rigid link) when the operator interacts, the six degrees of freedom including any one or more of x, y, and z positions and / or roll, pitch, and yaw orientations. When the operator interacts with a portion of a repositionable structure of the computer-assisted device, the posture information of the portion of the repositionable structure is determined using forward and / or inverse kinematics assuming known joint locations when the operator interacts. In some embodiments, an alternative technique is used to track the posture information of the portion of the repositionable structure without kinematics. For example, a sensor (e.g., a gyroscope) within a portion of the repositionable structure can directly track the movement of the portion of the repositionable structure. As additional examples, an IMU, a shape sensor, etc. can be used to directly track the movement of a portion of the repositionable structure. When the operator interacts with a portion of the computer-assisted device that is not included in the repositionable structure of the computer-assisted device or interacts with a portion of the computer-assisted device that does not include the repositionable structure, posture information for the portion of the computer-assisted device is determined based on the posture of the entire computer-assisted device 110 and the known position and orientation of the portion relative to the entire computer-assisted device 110.

[0066] In process 410, the registration transformation between the XR system and the computer-assisted device is updated based on the posture information of the portion of the operator determined during process 406 and the posture information of the portion of the computer-assisted device determined during process 408. The registration transformation to be updated can be an initial registration transformation, or a previously updated registration transformation. Thereafter, the updated registration transformation can be used to present an image including XR content (the XR content is displayed by the XR system and viewable by the operator) until a new update to the registration transformation is determined. In this way, the XR system uses the most recent update to the registration transformation when presenting an image including XR content. In some embodiments, the control module 170 causes the XR system to present such an image in response to one or more operator interactions at or later in process 404. For example, the XR content in the presented image can provide information for setting up a computer-assisted device, training the use of a computer-assisted device, or normal operation of a computer-assisted device.

[0067] Figure 5 Process 410 for updating a registration transform between an XR system and a computer-assisted device is shown in more detail in accordance with various embodiments. One or more of processes 502-504 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium that, when executed by one or more processors (e.g., processor 150 in control system 140), causes the one or more processors to perform one or more of processes 502-504. In some embodiments, processes 502-504 may be performed by one or more modules, such as control module 170. In some embodiments, additional processes not shown may be performed.

[0068] As shown, in process 502, the posture information of a portion of the operator and / or the posture information of a portion of the computer-assisted device is transformed into a common reference system using the registration transformation to be updated. In some embodiments, the common reference system is a reference system of the computer-assisted device (e.g., reference system 124), a reference system of the XR system (e.g., reference system 132), or any other reference system to which the posture information of a portion of the operator and / or the posture information of a portion of the computer-assisted device is transformed. For example, when the common reference system is a reference system of the XR system, the posture information of a portion of the computer-assisted device determined in process 408 is mapped to the posture information in the reference system of the XR system using the registration transformation. As another example, when the common reference system is a reference system of the computer-assisted device, the posture information of a portion of the operator determined in process 406 is mapped to the posture information in the reference system of the computer-assisted device using the inverse of the registration transformation. In some examples, the common reference system can be selected based on the procedure being executed, the type of instrument, the type of imaging device, the operator's preferences, etc.

[0069] In process 504, an error is determined between posture information of a portion of an operator and posture information of a portion of a computer-assisted device in a common reference system. In some embodiments, the error is a difference between posture information of a portion of an operator and posture information of a portion of a computer-assisted device in a common reference system. For example, assume that posture information of a portion of an operator is mapped to posture information in a reference system of a computer-assisted device (e.g., reference system 124). In this case, the error is a difference between the mapped posture information of a portion of an operator and the posture information of a portion of a computer-assisted device in the reference system of the computer-assisted device determined in process 408. As another example, assume that posture information of a portion of a computer-assisted device is mapped to posture information in a reference system of an XR system (e.g., reference system 132). In this case, the error is a difference between the mapped posture information of a portion of a computer-assisted device and the posture information of a portion of an operator determined in process 406.

[0070] In process 506, the registration transformation is updated based on the error. In some embodiments, the registration transformation is updated using gradient descent or any other suitable technique. In some examples, a gradient descent technique is applied to calculate a step size (step) for updating the registration transformation based on the error gradient expressed as a vector. In this case, the registration transformation between the reference system of the XR system and the reference system of the computer-aided device is updated based on the gradient descent step size. In multiple iterations, the gradient descent updates the registration transformation so that if the posture information is mapped to the common reference system again, the error will be reduced or eliminated. In some examples, the error determined in process 504 is filtered by a filtering operation. For example, in some cases, the error determined at process 504 is passed through a frequency domain filter (such as a Kalman filter) to smooth the transition between the previous and current registration transformation updates. In some examples, independent filters (e.g., independent Kalman filters) can be used for the rotational portion and the translational portion of the error, which depends specifically on the baseline noise level when determining the expected and / or observed geometric characteristics. In some examples, the filters used to process different degrees of freedom in the translational portion or the rotational portion may also be different.

[0071] In some embodiments, the registration transformation update is weighted based on the confidence assigned to the posture information of a portion of the operator. In some examples, the size of the above-mentioned gradient descent step is weighted by such confidence. In some other examples, the registration transformation update is performed only when such confidence has a value that satisfies a predefined threshold (which indicates sufficient confidence in the posture information of a portion of the operator). In some examples, as described above in conjunction with process 406, the confidence is based on the confidence score output by the same technology that determines the posture information of a portion of the operator. For example, in this case, the confidence is based on the ability to infer the posture information of the interaction point or interaction area when the interaction point or interaction area is facing away from the sensor system, completely or partially occluded, etc. In some examples, the confidence is based on the degree of consistency between different technologies, such as the consistency between the computer vision technology used to determine the posture information of a portion of the operator and the direction in which the operator is looking based on the position and orientation of the XR system worn by the operator. In some examples, the confidence is based on the amount of calculated error in determining the posture information of a portion of the computer-assisted device.

[0072] After updating the registration transformation between the XR system and the computer-assisted device in process 410, method 400 returns to process 402. In some examples, when the operator moves a portion of the computer-assisted device from one position (in some cases, and orientation) to another position (in some cases, and orientation) over time through multiple interactions and / or continuous interactions, (one or more) additional operator interactions are detected between a portion of the computer-assisted device. In this case, a time sequence of posture information of a portion of the computer-assisted device and a portion of the operator is determined over multiple time steps. The time sequence is then used to iteratively update the registration transformation between the XR system and the computer-assisted device according to processes 402-410. In some examples, the operator is prompted to interact with one or more portions of the computer-assisted device again. For example, if the error determined in process 504 exceeds a predefined threshold, indicating that the registration transformation is particularly inaccurate, the operator can be prompted to press a button again, grasp and move the arm of the repositionable structure, etc.

[0073] Figure 6An example of an operator interaction that can be detected in process 404 according to various embodiments is shown. As shown, the operator 108 wearing the XR system 130 interacts with a button 602 on a repositionable structure of the computer-assisted device 110. When the button 602 interaction is detected (process 402), the acquired sensor data is used to determine the pose information of the finger of the operator 108 pressing the button 602 (process 406). For example, when the sensor data includes image data, human pose estimation techniques can be used to identify the position (and optional orientation) of the finger in the image data. In addition, based on the known joint locations of the repositionable structure of the computer-assisted device 110, kinematic or non-kinematic techniques are used to determine the pose information of the button 602 when the button 602 interacts (process 408). If the button 602 is on a portion of the computer-assisted device 110 other than the repositionable structure, the pose of the entire computer-assisted device 110 and the known position of the button 602 relative to the rest of the computer-assisted device 110 can be used to determine the pose information of the button 602. The registration transformation between the XR system 130 and the computer-assisted device 110 is updated based on the pose information of the operator's 108 finger and the pose information of the button 602 (process 410). As described above, updating the registration transformation includes mapping the pose information of the finger and / or the pose information of the button 602 to a common reference system using an initial (or previously updated) registration transformation (process 502). Then, an error is calculated, which indicates the difference between the pose information of the finger and the pose information of the button 602 in the common reference system (process 504). Thereafter, the registration transformation is updated based on the error (process 506).

[0074] Figure 7Another example of an operator interaction that can be detected at process 404 according to various embodiments is shown. As shown, the operator 108 supporting the XR system 130 interacts with the repositionable arm 120 of the computer-assisted device 110 by pressing the touch / contact sensor 702 and grasping and moving the repositionable arm 120. The movement of the repositionable arm 120 is detected by joint sensors, shape sensors, etc. (process 404). At each time step during the movement of the repositionable arm 120, sensor data is acquired and used to determine the pose information of the operator's 108 hand (process 406). For example, the sensor data includes image data, and human pose estimation techniques are used to identify the position (and optional orientation) of the hand or individual fingers, and the positions (and optional orientations) of these fingers are averaged to obtain the position (and optional orientation) of the hand. In addition, based on the known joint locations of the repositionable structure of the computer-assisted device 110, kinematics is used to determine the pose information of the repositionable arm 120 at each time step (408). The registration transform between the XR system 130 and the computer-assisted device 110 is iteratively updated at these time steps. Iteratively updating the registration transform includes mapping the hand pose information and / or the repositionable arm 120 pose information to a common reference frame using an initial (or previously updated) registration transform (process 502). Then, an error is calculated that indicates the difference between the hand pose information and the repositionable arm 120 pose information in the common reference frame (process 504). Thereafter, the registration transform is updated based on the error (process 506).

[0075] The disclosed technology can update the registration transform between the XR system and the computer-assisted device. Due to the updated registration transform, the XR system can display XR content at a more accurate position and / or orientation relative to the computer-assisted device than without updating the registration transform. The displayed XR content can include instructional content on how to operate the computer-assisted device, content that provides guidance during operation of the computer-assisted device, etc.

[0076] Some examples of control systems (such as control system 140) may include a non-transitory tangible machine-readable medium that includes executable code that, when executed by one or more processors (e.g., processor 150), may cause the one or more processors to perform the processes of methods 400 and / or 500 and / or Figure 4 and / or Figure 5 The process of method 400 and / or 500 may be included. Figure 4 and / or Figure 5Some common forms of machine-readable media for the process are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media from which a processor or computer is suitable for reading.

[0077] Although illustrative embodiments have been shown and described, extensive modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without the corresponding use of other features. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications. Therefore, the scope of the present invention should be limited only by the appended claims, and it is appropriate that the claims be broadly construed and interpreted in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. An electronic system comprising: Computer-aided equipment; as well as a control system communicatively coupled to the computer-assisted device and to a sensor system configured to capture sensor data regarding an environment external to the computer-assisted device, The control system is configured as follows: determining, during one or more interactions between an operator portion of an operator and a device portion of the computer-assisted device, gesture information of the operator portion based on the sensor data, the gesture information of the operator portion comprising at least one parameter selected from the group consisting of: a position of the operator portion during the one or more interactions and an orientation of the operator portion during the one or more interactions, determining posture information of the device portion during the one or more interactions, the posture information of the device portion comprising at least one parameter selected from the group consisting of: the position of the device part during the one or more interactions and the orientation of the device part during the one or more interactions, updating a registration transformation between an extended reality system (XR system) and the computer-aided device based on the posture information of the operator and the posture information of the device, and The XR system is caused to render an image viewable by the operator using the updated registration transform.

2. The electronic system according to claim 1, wherein: The computer-assisted device includes a repositionable structure including a plurality of links coupled by a plurality of joints; The device portion includes a portion of the repositionable structure; and To determine the posture information of the device part, the control system is configured to calculate the posture information based on kinematic data of the part of the repositionable structure.

3. The electronic system of claim 1, wherein the one or more interactions include at least one interaction selected from the group consisting of: the operator portion interacting with a mechanical input device disposed on the device portion; the operator portion interacting with a virtual input device displayed on the device portion; The operator portion contacts the device portion; said operator portion moving said equipment portion; The operator portion attaches an object to the device portion; And the operator portion separates the object from the device portion.

4. The electronic system according to claim 1, wherein: The one or more interactions include physically coupling an object to the device portion or decoupling the object from the device portion; and The control system is also configured to identify the equipment part based on the sensor data. The electronic system of claim 1 , wherein the one or more interactions are a single interaction.

6. The electronic system of claim 5, wherein: The operator portion includes one or more fingers of the operator; The device portion includes a mechanical button disposed on the computer-aided device or a touch screen disposed on the computer-aided device; The single interaction includes the one or more fingers activating the mechanical button, or deactivating the mechanical button, or providing input to the touch screen; and The control system is further configured to identify the one or more fingers based on the sensor data. The electronic system of claim 5 , wherein the single interaction comprises a continuous interaction.

8. The electronic system of claim 7, wherein: The computer-assisted device includes a repositionable structure including a manipulator arm having a plurality of links coupled by a plurality of joints; The operator portion includes the operator's hand; The device portion includes the manipulator arm; The continuous interaction includes the hand of the operator moving the manipulator arm; and The control system is also configured to identify the hand based on the sensor data.

9. The electronic system of claim 1, wherein the one or more interactions include a plurality of interactions.

10. The electronic system of claim 9, wherein the plurality of interactions comprises: A composite interaction or multiple independent interactions.

11. The electronic system of claim 9, wherein: The one or more interactions include a series of interactions; and To update the registration transformation between the XR system and the computer-assisted device, the control system is configured to iteratively update the registration transformation based on the posture information of the operator part and the posture information of the device part using each interaction included in the series of interactions.

12. The electronic system of claim 1 , wherein to determine the posture information of the operator portion, the control system is configured to determine: the operator's eye vergence point; or The operator's gaze direction.

13. An electronic system according to claim 1, wherein the posture information of the operator part includes the operator's eye convergence point or the operator's gaze direction when the one or more interactions include at least one interaction selected from the group consisting of: the operator interacting with a virtual input device displayed on the device part; the operator part attaching an object to the device part; and the operator part detaching an object from the device part.

14. The electronic system of claim 1, wherein the control system is further configured to cause the XR system to present an image including prompts for the operator to perform the one or more interactions.

15. The electronic system of claim 1 , wherein the control system is further configured to determine an initial transformation of the registration transformation between the XR system and the computer-assisted device based on initial sensor data captured by the sensor system and not associated with the one or more interactions.

16. The electronic system according to any one of claims 1 to 15, wherein to update the registration transformation, the control system is configured to: transforming the pose information of the operator portion or the pose information of the device portion into a common reference frame using the registration transformation; determining an error between the posture information on the operator's part and the posture information on the device's part in the common reference frame; and The registration transform is updated based on the error.

17. The electronic system of claim 16, wherein: The common reference frame is a reference frame of the XR system, and the pose information of the device part is transformed into the common reference frame using the registration transformation; or The common reference frame is a reference frame of the computer-aided device, and the pose information of the operator part is transformed into the common reference frame using the inverse of the registration transformation.

18. The electronic system of claim 16, wherein to update the registration transform based on the error, the control system is configured to: One or more operations selected from the group consisting of: a gradient descent operation and a filtering operation are performed.

19. The electronic system of claim 1, wherein to update the registration transformation, the control system is configured to weight the posture information of the operator portion based on a confidence level.

20. The electronic system of claim 19, wherein: To determine the posture information of the operator's part, the control system is configured to use a plurality of techniques for posture determination; and The control system is further configured to determine the confidence level based on a degree of agreement between the plurality of techniques.

21. The electronic system of any one of claims 1 to 15, wherein to determine the posture information of the operator part, the control system is configured to perform one or more object detection operations using human posture estimation techniques.

22. The electronic system of any one of claims 1 to 15, wherein the XR system comprises a head-mounted component, and the sensor system comprises an imaging device disposed on the head-mounted component.

23. The electronic system according to any one of claims 1 to 15, wherein the control system is further configured to: Interactions are identified as the one or more interactions based on the operator being performing a particular interaction to set up the computer-assisted device.

24. The electronic system of claim 23, wherein in order to identify the interaction as being performed by the operator for setting up the computer-assisted device, the control system is configured to: The operational state of the computer-assisted device at the time of the interaction or the physical configuration of the computer-assisted device at the time of the interaction is used.

25. The electronic system of any one of claims 1 to 15, wherein the control system is further configured to identify the interaction as the one or more interactions based on the operator performing the interaction during training for using the computer-assisted device.

26. The electronic system of claim 25, wherein to identify the interaction as being performed by the operator during training for use of the computer-assisted device, the control system is configured to: The operating state of the computer-assisted device at the time of the interaction or the operating state of the XR system at the time of the interaction is used.

27. The electronic system of any one of claims 1 to 15, wherein the one or more interactions include a set of predefined interactions, the set of predefined interactions having a sequence, and wherein the control system is further configured to: prompting the operator to perform the set of predefined interactions according to the sequence; determining a registration transformation error associated with the device portion; and The set of predefined interactions is adjusted based on the registration transformation error.

28. The electronic system of claim 27, wherein to adjust the set of predefined interactions based on the registration transformation error, the control system is configured to: In response to the registration transformation error being greater than an error threshold, at least one interaction is added to the set of predefined interactions.

29. The electronic system of claim 27, wherein to adjust the set of predefined interactions based on the registration transformation error, the control system is configured to: In response to the registration transformation error being less than an error threshold, at least one interaction is removed from the set of predefined interactions.

30. The electronic system of claim 27, wherein to adjust the set of predefined interactions based on the registration transformation error, the control system is configured to: The order of the set of predefined interactions is changed.

31. The electronic system of claim 27, wherein to adjust the set of predefined interactions based on the registration transformation error, the control system is configured to: skipping prompting the operator to perform at least one interaction from the set of predefined interactions; or Prompting the operator to perform at least one interaction from the set of predefined interactions is repeated.

32. A method of operating an electronic system, the electronic system comprising a computer-assisted device and a control system communicatively coupled to the computer-assisted device and a sensor system configured to capture sensor data regarding an environment external to the computer-assisted device, the method comprising: determining, during one or more interactions between an operator portion of an operator and a device portion of the computer-assisted device, posture information of the operator portion based on the sensor data, the posture information of the operator portion comprising at least one parameter selected from the group consisting of: a position of the operator portion during the one or more interactions and an orientation of the operator portion during the one or more interactions; determining posture information of the device portion during the one or more interactions, the posture information of the device portion comprising at least one parameter selected from the group consisting of: a position of the device portion during the one or more interactions and an orientation of the device portion during the one or more interactions; Based on the posture information of the operator and the posture information of the device, updating the registration transformation between an extended reality system (XR system) and the computer-aided device; as well as The XR system is caused to render an image viewable by the operator using the updated registration transform.

33. The method of claim 32, wherein: The computer-assisted device includes a repositionable structure including a plurality of links coupled by a plurality of joints; The device portion includes a portion of the repositionable structure; and Determining pose information of the device portion includes calculating the pose information based on kinematic data of the portion of the repositionable structure.

34. The method of claim 32, wherein: The one or more interactions include physically coupling an object to the device portion or decoupling the object from the device portion; and The method also includes identifying the equipment part based on the sensor data.

35. The method of claim 32, wherein the one or more interactions is a single interaction.

36. The method of claim 35, wherein: The operator portion includes one or more fingers of the operator; The device portion includes a mechanical button disposed on the computer-aided device or a touch screen disposed on the computer-aided device; The single interaction includes the one or more fingers activating the mechanical button, or deactivating the mechanical button, or providing input to the touch screen; and The method also includes identifying the one or more fingers based on the sensor data.

37. The method of claim 35, wherein the single interaction comprises a continuous interaction.

38. The method of claim 37, wherein: The computer-assisted device includes a repositionable structure including a manipulator arm having a plurality of links coupled by a plurality of joints; The operator portion includes the operator's hand; said portion of said computer-assisted device comprising said manipulator arm; The continuous interaction includes the hand of the operator moving the manipulator arm; and The method also includes identifying the hand based on the sensor data.

39. The method of claim 32, wherein the one or more interactions include a plurality of interactions.

40. The method of claim 39, wherein the plurality of interactions comprises: A composite interaction or multiple independent interactions.

41. The method of claim 39, wherein: The one or more interactions include a series of interactions; and Updating the registration transform between the XR system and the computer-assisted device includes iteratively updating the registration transform based on the pose information on the operator part and the pose information on the device part with each interaction included in the series of interactions.

42. The method of claim 32, wherein determining the posture information of the operator portion comprises determining: the operator's eye vergence point; or The operator's gaze direction.

43. A method according to claim 32, wherein the posture information of the operator part includes the operator's eye convergence point or the operator's gaze direction when the one or more interactions include at least one interaction selected from the group consisting of: the operator interacting with a virtual input device displayed on the device part; the operator part attaching an object to the device part; and the operator part detaching an object from the device part.

44. The method of claim 32, further comprising causing the XR system to present an image including prompts for the operator to perform the one or more interactions.

45. The method of claim 32, further comprising determining an initial transformation of the registration transformation between the XR system and the computer-assisted device based on initial sensor data, the initial sensor data being captured by the sensor system and not associated with the one or more interactions.

46. ​​The method of claim 32, wherein updating the registration transform comprises: transforming the pose information of the operator portion or the pose information of the device portion into a common reference frame using the registration transformation; determining an error between the posture information on the operator's part and the posture information on the device's part in the common reference system; as well as The registration transform is updated based on the error.

47. The method of claim 46, wherein updating the registration transform based on the error comprises at least one of: performing one or more operations selected from the group consisting of: a gradient descent operation and a filtering operation; or The posture information of the operator portion is weighted based on confidence.

48. The method of claim 32, wherein determining the posture information of the operator portion comprises: Perform one or more object detection operations using human pose estimation techniques.

49. The method of claim 32, further comprising: Based on the operator being performing a specific interaction for setting up the computer-assisted device, the specific interaction is identified as the one or more interactions.

50. The method of claim 49, wherein identifying the particular interaction as being performed by the operator for setting up the computer-assisted device comprises: The operational state of the computer-assisted device at the time of the particular interaction or the physical configuration of the computer-assisted device at the time of the interaction is used.

51. The method of claim 32, further comprising: The specific interactions are identified as the one or more interactions based on the operator being performing the specific interactions during training for using the computer-assisted device.

52. The method of claim 51 , wherein identifying the interaction as being performed by the operator during training for use of the computer-assisted device comprises: The operating state of the computer-assisted device at the time of the interaction or the operating state of the XR system at the time of the interaction is used.

53. The method of claim 32, wherein the one or more interactions include a set of predefined interactions, the set of predefined interactions having an order, the method further comprising: prompting the operator to perform the set of predefined interactions according to the sequence; determining a registration transformation error associated with the device portion; as well as The set of predefined interactions is adjusted based on the registration transformation error.

54. The method of claim 53, wherein adjusting the set of predefined interactions based on the registration transformation error comprises: In response to the registration transformation error being greater than an error threshold, at least one interaction is added to the set of predefined interactions.

55. The method of claim 53, wherein adjusting the set of predefined interactions based on the registration transformation error comprises: In response to the registration transformation error being less than an error threshold, at least one interaction is removed from the set of predefined interactions.

56. The method of claim 53, wherein adjusting the set of predefined interactions based on the registration transformation error comprises: The order of the set of predefined interactions is changed.

57. The method of claim 53, wherein adjusting the set of predefined interactions based on the registration transformation error comprises: skipping prompting the operator to perform at least one interaction from the set of predefined interactions; or Prompting the operator to perform at least one interaction from the set of predefined interactions is repeated.

58. One or more non-transitory machine-readable media comprising a plurality of machine-readable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform the method of any one of claims 32 to 57.

Images