Extended reality controller pose tracking

Abstract

An illustrative method is performed by an extended reality system that includes a head-mounted device and a controller device. The method includes determining a relative pose of the controller device with respect to the head-mounted device, the determining based on first pose data produced by a first inertial measurement unit (IMU) integrated in the head-mounted device and second pose data produced by a second IMU integrated in the controller device. The method further includes determining a three-dimensional position of the controller device relative to the head-mounted device using a radio frequency (RF) signal transmitted from the head-mounted device and received by the controller device. The method further includes compensating for drift of the relative pose of the controller device with respect to the head-mounted device based on the three-dimensional position determined using the RF signal. Corresponding methods and systems are also disclosed.

Description

EXTENDED REALITY CONTROLLER POSETRACKINGTECHNICAL FIELD

[0001] This description relates to tracking a controller in an extended reality system.BACKGROUND

[0002] Extended reality is an umbrella term referring to various technologies that serve to augment, virtualize, or otherwise extend a user's experience of reality in a variety of ways. For example, augmented reality, virtual reality, and other similar technologies refer to different types of extended reality that have been developed and deployed for use with entertainment, educational, vocational, and other types of applications. In certain cases, extended reality experiences may be presented on head-mounted displays to free up the user’s hands for other tasks such as to hold and manipulate an extended reality controller configured to facilitate user interaction with the extended reality presentation.SUMMARY

[0003] System and methods for extended reality controller pose tracking are described herein. Conventional approaches to tracking the pose of controllers used with headmounted devices in extended reality systems have generally relied on cameras and image processing (e.g., of know n arrangements of LEDs or markers on the controller, etc.). While these approaches may also incorporate data from inertial measurement units (IMUs) in the controllers and / or head-mounted devices, the image processing tends to be overly processorintensive and adds to the size, weight, power, cost and so forth in a way that presents challenges for consumer hardware. To address these challenges, extended reality systems described herein avoid image processing in their tracking of controller pose, relying instead on a combination of 1) IMU-derived pose information from the head-mounted device and controller device (which is precise and dynamic but tends to drift over time from being accurately registered in a shared world space) and 2) relative pose information generated based on exchange of an RF signal (e.g., a millimeter- wave radar signal) between the headmounted device and the controller device (which is readily accessible and can be used not only to initialize the IMU pose but also to continuously constrain it from drifting).

[0004] In one implementation, a method comprises steps including: 1) determining (e.g., estimating) a relative pose of a controller device with respect to a head-mounted device, wherein the determining of the relative pose is based on first pose data produced by (and received from) a first inertial measurement unit (IMU) integrated in the head-mounted device, and second pose data produced by (and received from) a second IMU integrated in the controller device; 2) determining a three-dimensional (3D) position of the controller device relative to the head-mounted device using a radio frequency (RF) signal transmitted from the head-mounted device and received by the controller device (or transmitted from the controller device and received by the head-mounted device); and 3) compensating for drift of the relative pose of the controller device with respect to the head-mounted device based on the 3D position determined using the RF signal. In some examples, this method may be implemented as instructions stored on a non-transitory computer-readable medium that, when executed, cause a processor of a computing device to perform the method.

[0005] In another implementation, an extended reality system comprises: 1) a headmounted device including a first inertial measurement unit (IMU) and a radio frequency (RF) transmitter; 2) a controller device including a second IMU and an RF receiver; and 3) a processor. Based on instructions such as those mentioned above, the processor may be configured to perform a process comprising: determining (e.g., estimating), based on first pose data determined by the first IMU in the head-mounted device and second pose data determined by the second IMU in the controller device, a relative pose of the controller device with respect to the head-mounted device; determining, using an RF signal transmitted by the RF transmitter and received by the RF receiver, a 3D position of the controller device relative to the head-mounted device; and compensating, based on the 3D position determined using the RF signal, for drift of the relative pose of the controller device with respect to the head-mounted device.

[0006] While the implementations described above include transmission of the RF signal from the head-mounted device to the controller device, other implementations may operate similarly, but with the RF signal being transmitted from a transmitter in the controller device to be received by a receiver in the head-mounted device. For example, in one implementation, a method comprises steps including: 1) determining (e.g., estimating) a relative pose of a controller device with respect to ahead-mounted device, the relative pose being based on first pose data produced by a first inertial measurement unit (IMU) integrated in the head-mounted device and second pose data produced by a second IMU integrated in the controller device; 2) determining a 3D position of the controller device relative to thehead-mounted device using a radio frequency (RF) signal transmitted from the controller device and received by the head-mounted device (or transmitted from the head-mounted device and received by the controller device); and 3) compensating for drift of the relative pose of the controller device with respect to the head-mounted device based on the 3D position determined using the RF signal. This method may be implemented as instructions stored on a non-transitory computer-readable medium that, when executed, cause a processor of a computing device to perform the method.

[0007] An extended reality system that uses an RF transmission in this direction may include: 1) a head-mounted device including a first inertial measurement unit (IMU) and a radio frequency (RF) receiver; 2) a controller device including a second IMU and an RF transmitter; and 3) a processor. Based on instructions such as mentioned above, the processor may be configured to perform a process comprising: determining (e.g., estimating), based on first pose data determined by the first IMU in the head-mounted device and second pose data determined by the second IMU in the controller device, a relative pose of the controller device with respect to the head-mounted device; determining, using an RF signal transmitted by the RF transmitter and received by the RF receiver, a 3D position of the controller device relative to the head-mounted device; and compensating, based on the 3D position determined using the RF signal, for drift of the relative pose of the controller device with respect to the head-mounted device.

[0008] The details of these and other implementations are set forth in the accompanying drawings and the description below. Other features will also be made apparent from the following description, drawings, and claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG. 1 A shows an illustrative extended reality system configured to perform controller pose tracking in accordance with principles described herein.

[0010] FIG. IB shows an illustrative method for extended reality controller pose tracking in accordance with principles described herein.

[0011] FIG. 2A shows another illustrative extended reality system configured to perform controller pose tracking in accordance with principles described herein.

[0012] FIG. 2B show s another illustrative method for extended reality controller pose tracking in accordance with principles described herein.

[0013] FIG. 3 shows an illustrative world coordinate system with which the poses of a head-mounted device and a controller device may be registered in accordance with principlesdescribed herein.

[0014] FIG. 4 A shows illustrative aspects of how a relative position and orientation of a controller device with respect to a head-mounted device may be observed in accordance with principles described herein.

[0015] FIG. 4B shows illustrative aspects of how relative positions and orientations of a plurality of controller devices with respect to a head-mounted device may be observed in accordance with principles described herein.

[0016] FIG. 5 shows illustrative aspects of a sensor fusion technique that may be used to derive a drift-compensated relative pose of a controller device with respect to a headmounted device in accordance with principles described herein.

[0017] FIG. 6 shows an illustrative computing system that may be used to implement various devices and / or systems described herein.DETAILED DESCRIPTION

[0018] Systems and methods for extended reality controller pose tracking are described herein. Extended reality systems described herein include a head-mounted device and a controller device that are both used by a user during an extended reality experience (e.g., wearing the head-mounted device on the head and manipulating the controller device by hand). As such, these two devices may move independently from one another and it is desirable for respective poses (e.g., six-degree-of-freedom (6dof) poses) of both devices to be registered with a shared world coordinate system.

[0019] Both the head-mounted and controller devices may include specialized equipment to accurately detect movement, position, orientation and / or other aspects of their respective poses. For example, respective inertial measurement units (IMUs) included in the head-mounted device and the controller device may produce pose data that may be used to determine a pose (e.g., a 6dof pose indicative of orientation and position displacement) of the device in which the IMU is housed with relatively low cost, power consumption, and so forth. IMUs included in devices such as a head-mounted device or controller device of an extended reality system may include various sensors such as, for instance, a 3 -axis gyroscope (to measure a rotation vector), a 3-axis accelerometer (to measure translation acceleration plus gravity) and / or other integrated sensors (e.g., a compass, etc.) that collectively produce pose data from which an accurate 6dof pose of the device housing the IMU may be derived, even in dynamic conditions (e.g., where the device is in motion and the pose is dynamically changing).

[0020] Unfortunately, however, a 6dof pose derived exclusively from such IMU sensor data may tend to drift over time for various reasons. One reason is that every real- world sensor is non-ideal and has at least some amount of intrinsic error and imperfection. Moreover, any error that may be associated with sensors of a given IMU may be compounded by the integration operations performed to convert the detected data into the desired quantities. For example, if an accelerometer detects acceleration of a device but the desired quantity is a position of the device, two integrals (from acceleration to velocity, and then from velocity to position) must be computed. Such computations tend to compound even very small errors that may be present in the sensor itself such that, over time, the position being computed from the data may drift from a ground truth that would be observed from another frame of reference (e.g., from another device). A similar issue as described above for the accelerometer may also be observed for the gyroscope and / or for other IMU sensors that may be relied on to detect a dynamic pose of a given device.

[0021] One way that conventional systems have tried to compensate for such drift and keep independently posed devices registered to a common reference frame (e.g., a shared world coordinate system) relies on image processing. For example, a controller device may incorporate an arrangement of LEDs (visible LEDs, infrared LEDs, etc.) or other such markers that may be readily detectable by an image sensor integrated with the head-mounted device. By analyzing images captured by the image sensor, ground truth indicative of the relative poses of the controller device and the head-mounted device may be determined so that drift in the respective IMU-derived poses for both devices may be corrected or otherwise compensated for. However, such image processing requires significant computing resources, thereby presenting a technical challenge for certain systems. Specifically, any device that is to be worn by a user on his or her person (e.g., a head-mounted device worn on the head, etc.) may have strict design limits on how large a device can be, how much heat can be generated, how much weight can be supported, and so forth. Power consumption of such devices may also be important for battery life parameters. Accordingly, an approach described herein that avoids image processing and the relatively powerful processors that image processing requires (along with the processors’ corresponding power, heat, cost, etc.) therefore provides a helpful technical solution to the technical problems of controller tracking and drift management in a way that is as efficient as possible (i.e., low power, low heat, lightweight, small, low cost, etc.).

[0022] As will be described in detail herein, a radio-frequency (RF) signal such as a millimeter-wave (mm-wave) radar signal may be exchanged between a head-mounted deviceand a controller device in an extended reality system to facilitate real-time motion tracking observations of the controller device with respect to the head-mounted device. These observations may be used to correct and constrain IMU-derived drift that has been described. In this way, various challenges introduced by image-based motion tracking techniques (including extensive resource usage that has been described as well as other challenges such as that image processing approaches require a controller device to stay within a field of view of the image capture) may be mitigated and / or resolved.

[0023] In some cases, the RF signal may be implemented as a mm-wave radar signal, which may require minimal resource usage (e.g., by being relatively small, light, low power, low cost, etc.). Not only can such a radar signal provide movement and position observations for the controller, but, in examples in which a radar transmitter transmits a signal that is received by at least three non-coplanar receivers, an angle of arrival indicative of a real-time orientation may also be observed and used to constrain and compensate for orientation error components of a 6dof pose drift. For example, as will be described in more detail, doppler measurements, elevation angle measurements, azimuth angle measurements, and so forth may be performed based on receipt of the RF signal (e.g., the mm-wave radar signal) by the at least three receivers and these measurements may be merged with pose data produced by the IMUs using a Kalman filter or other sensor fusion technique.

[0024] At least one technical effect produced by solving these technical problems using these technical solutions is that accurate, self-correcting methods of extended reality controller pose tracking may be implemented using components and techniques that are suitable for consumer products and other extended reality systems with strict design constraints along the various dimensions (e.g., heat, power, size, weight, cost, etc.) that have been described.

[0025] Various implementations will now be described in more detail with reference to the figures. It will be understood that the particular implementations described below are provided as non-limiting examples and may be applied in various situations. Additionally, it will be understood that other implementations not explicitly described herein may also fall within the scope of the claims set forth below. Systems and methods described herein for extended reality controller pose tracking may result in any or all of the technical benefits mentioned above, as well as various additional technical benefits that will be described and / or made apparent below.

[0026] FIG. 1A shows an illustrative extended reality system 100 configured to perform controller pose tracking in accordance with principles described herein. Specifically,as shown, extended reality system 100 includes a head-mounted device 102, a controller device 104, and a processor 106. Alongside the block diagram of extended reality system 100, an example implementation of head-mounted device 102 is depicted as a head-mounted extended reality presentation device (including an electronic unit with equipment for audiovisual presentation to a user, a strap to mount the unit on the user's head, etc.). Similarly, an example implementation of controller device 104 is depicted as a handheld extended reality controller (including various buttons that the user may press as he or she moves and aims the controller in accordance with an extended reality experience).

[0027] In the block diagram, head-mounted device 102 is show n to include a first inertial measurement unit (IMU) 108-1 and a radio frequency (RF) transmitter 110, while controller device 104 is shown to include a second IMU 108-2 and an RF receiver 112. While it may not be relied on for determining the controller pose relative to the head-mounted device, head-mounted device 102 also is show n to include a camera 113 that may be used to determine the pose of head-mounted device 102 with respect to the environment (e.g., using visual inertial odometry (VIO), simultaneous localization and mapping (SLAM), or other such techniques), as will be described in more detail below. Processor 106 is shown to be associated with a process 114 that the processor may perform (e g., based on instructions stored in a memory (not explicitly shown) of extended reality system 100). The block representing processor 106 is drawn to overlap both head-mounted device 102 and controller device 104 to illustrate that processing resources used to perform process 114 may be implemented entirely within head-mounted device 102 (e g., such that process 114 is performed by one or more processors integrated within head-mounted device 102), entirely within controller device 104 (i.e., such that process 114 is performed by one or more processors integrated within controller device 104), or distributed between both devices and / or other elements of extended reality system 100 (e.g., such that different parts of process 114 are performed by different processors integrated within different components of the system).

[0028] While the depictions in FIG. 1A illustrate general examples of devices that may implement these components of the system (i.e., head-mounted device 102 and controller device 104), it will be understood that each of these components may be implemented in various ways as may serve a particular implementation. For example, headmounted device 102 may be implemented using any ty pe or form factor of extended reality or mixed reality headset (e.g., virtual reality googles, augmented reality glasses, etc.). Similarly, controller device 104 may be implemented using any type of extended reality’ controller ofany form factor. For instance, controller device 104 may be implemented as a gamecontroller-like device with buttons and / or triggers as shown, as a motion detector controller (e.g., with or without buttons) configured to be worn (e.g., as a ring or bracelet, etc.) so as to free up the user’s hand for other tasks (by not requiring the controller to be held), as a 6dof annotation tool (e.g., a pen controller), as a controller associated with another part of the user's body other than the hands (e.g., the feet, the arms, etc.), or as any other controller type or configuration as may serve a particular implementation.

[0029] The type of RF signal exchanged between RF transmitter 110 and RF receiver 112 may vary in different implementations. As one example, RF transmitter 110 and RF receiver 112 may exchange a radar signal, such as a mm-wave radar signal. With the advent and rapid deployment of 5G cellular technologies, electronic components configured to transmit and receive such mm-wave radar signals may be especially straightforward and / or efficient to implement or include in a design. In other implementations, however, it will be understood that other ty pes of radar or RF signaling (e.g., ultra- wide band (UWB) signaling, LIDAR signaling, etc.) may similarly be implemented to the same effect. Whatever the signaling technology, however, RF transmitter 110 and RF receiver 1 12 are shown to be implemented in opposite devices (e g., RF transmitter 1 10 in head-mounted device 102 and RF receiver 112 in controller device 104), rather than being integrated in the same device (as might be a typical configuration for certain technologies such as LIDAR). It will be made apparent below how various advantages may arise by separating the transmitter and receiver in this way, particularly in implementations where multiple receivers are arranged non- collinearly on a plane to allow for an angle of arrival to be detected.

[0030] Process 114 involves various steps such as will be described below in relation to FIG. IB. Specifically, for example, process 114 may begin with processor 106 determining, based on first pose data produced by IMU 108-1 (in head-mounted device 102) and second pose data produced by IMU 108-2 (in controller device 104), a relative pose of controller device 104 with respect to head-mounted device 102. Due to drift and sensor imperfections, it will be understood that this determination may be an estimation that is to be corrected or compensated for with other data (as will be described below). Additionally, as mentioned above, the pose of the head-mounted device with respect to the environment may also be determined as part of process 114, based on information received from camera 113 (e.g., using techniques such as VIO and / or SLAM). Process 114 may continue by processor 106 determining, using an RF signal 116 transmitted by RF transmitter 110 and received by RF receiver 112, a 3D position of controller device 104 relative to head-mounted device 102.For example, this RF signal may be a mm- wave radar signal, a radar signal associated with another suitable wavelength, or another suitable RF signal that may comport with various design targets (e.g., power, cost, etc.) as may be called for by a particular implementation. Based on this 3D position determined using RF signal 116, processor 106 may compensate for drift of the (estimated) relative pose of controller device 104 with respect to headmounted device 102 in any of the ways described herein. For example, a sensor fusion technique (e.g., involving a Kalman filter or the like) may be used to merge the pose data being produced by the independent IMUs 108-1 and 108-2 with relative pose observations being made based on RF signal 116 (e.g., using mm- wave radar or another suitable technology).

[0031] In some implementations, the relative pose of controller device 104 with respect to head-mounted device 102 may refer to a six-degrees-of-freedom (6dof) pose of controller device 104 with respect to head-mounted device 102. More particularly, the 3-axis position (translation) of controller device 104 with respect to head-mounted device 102 maybe represented by values corresponding to of x-, y-, and z-axes or surge, heave, and sway translation parameters of the 6dof pose. Additionally, the 3-axis orientation (rotation) of controller device 104 with respect to head-mounted device 102 may be represented by values corresponding to normal, transverse, and longitudinal axes of yaw, pitch, and roll rotation parameters of the 6dof pose.

[0032] FIG. IB shows an illustrative flowchart for a method 120. which will be understood to be an example implementation of process 1 14 described above. As shown, method 120 implements extended reality controller pose tracking in accordance with principles described herein. While FIG. IB shows illustrative operations 122-126 according to one implementation, other implementations of method 120 (or of process 114) may omit, add to, reorder, and / or modify any of the operations 122-126 shown in FIG. IB. In some examples, multiple operations shown in FIG. IB or described in relation to FIG. IB may be performed concurrently (e.g., in parallel) with one another, rather than being performed sequentially as illustrated and / or described. Each of operations 122-126 of method 120 will now be described in more detail as the operations may be performed by an implementation of extended reality system 100. More particularly, method 120 may be performed by the processor 106 that is included in extended reality system 100, whether it be incorporated entirely within head-mounted device 102, entirely within controller device 104, or distributed across both of these devices and / or other system components.

[0033] At operation 122, extended reality system 100 may determine (or estimate) arelative pose of controller device 104 with respect to head-mounted device 102 based on pose data produced by the respective IMUs 108-1 and 108-2. More particularly, the relative pose estimated at operation 122 may be based on: 1) first pose data produced by the first IMU 108- 1 integrated in head-mounted device 102, and 2) second pose data produced by the second IMU 108-2 integrated in controller device 104. As has been mentioned, the IMUs 108-1 and 108-2 may be highly efficient and accurate for determining dynamic pose data as the devices 102 and 104 move with respect to one another. However, for a variety of reasons (including that it may not be the direct measurements taken by the IMUs but one or more integrals of these measurements that is of interest for the poses), the pose data from the IMUs will suffer from drift that results in the estimate of the relative pose between the devices becoming less and less accurate over time if it is not corrected (i.e., if the drift is not compensated for).

[0034] Accordingly, at operation 124, extended reality system 100 may determine a 3D position of controller device 104 relative to head-mounted device 102 using an RF signal transmitted from head-mounted device 102 (e.g., using RF transmitter 110) and received by controller device 104 (e.g., using RF receiver 112). This observation of an actual 3D position is independent from the measurements being made by the IMUs and represented in the pose data described above, such that the 3D position observation may be used to correct or compensate for the drift in the relative pose estimated at operation 122. It will be understood that various observations may be made at this stage using this RF (e.g., mm-wave radar) signal. For instance, as will be described and illustrated below, if the RF signal is received by three non-collinear antennas of an RF receiver, the azimuth and elevation angles (i.e., collectively forming an angle of arrival) of the RF signal with respect to the plane on which the three antennas are arranged may be observed and accounted for along with a distance observation for the device as a whole. However, just as the pose data produced by the IMUs. by itself, may be of limited usefulness in tracking the relative pose over time (due to the drift), any relative 3D position detected on the sole basis of the RF signal at operation 124 may be of limited usefulness without more context. This is because the radar observations, while not prone to drift in the same way the IMU measurements are, are less dynamic and more prone to noise and inaccuracy for devices whose relative poses are in flux (i.e., changing quickly and dynamically).

[0035] For these reasons, method 120 continues with operation 126, in which extended reality system 100 may compensate for drift of the relative pose of the controller device with respect to the head-mounted device (estimated at operation 122) based on the 3D position determined using the RF signal (at operation 124). In other words, the IMU-derivedrelative pose may be utilized and relied on generally, but may need to be corrected, constrained, recalibrated, and / or otherwise compensated on a periodic basis using the RF (radar)-based signaling. Put another way, method 120 may involve determining an estimated relative pose of head-mounted device 102 and controller device 104 by subtracting the absolute pose of head-mounted device 102 (based on IMU 108-1 and / or VIO / SLAM or other techniques described in more detail below) from the absolute pose of controller device 104 (based on second IMU 108-2). Method 120 may further involve determining an observed relative pose of head-mounted device 102 and controller device 104 using the RF signal exchange between the devices (e.g., using mm-wave radar or the like). A reliable, dynamic, non-drifting (or at least with a drift that is suitably constrained) relative pose may then be determined based on these estimated and observed relative poses using a sensor fusion technique such as will be described in more detail below (e.g., a Kalman filtering technique or the like).

[0036] FIG. 1 A shows how process 114 may be performed by hardware processing resources of an extended reality system 100, while FIG. IB shows a flowchart for the method 120 that may be performed by such a system. It will be understood that method 120 as well as any variations thereof that are described or suggested in the following disclosure (e.g., which may add additional details or steps, which may modify or reorder certain steps, etc.), may be implemented as instructions stored on a non-transitory computer-readable medium that, when executed, cause a processor of a computing device (e.g., processor 106) to perform the method. Such a medium (e.g., a memory of the computing device, a storage facility of the computing device, etc.) is not explicitly shown in FIGS. 1 A or IB, but will be understood to be present and communicatively coupled to processors such as processor 106 in computing devices configured to perform the method.

[0037] The example implementation of FIGS. 1A and IB relates to an extended reality system 100 in which the RF transmitter (i.e., RF transmitter 110 in this example) is incorporated into the head-mounted device (i.e., head-mounted device 102) and in which one or more antennas of an RF receiver (i.e.. RF receiver 112 in this example) are incorporated into the controller device (i.e., controller device 104). For convenience, this type of implementation (with the RF signal transmitted from the head-mounted device to the controller device, rather than vice versa) will also be used for various detailed examples set forth below-. In spite of the convenience of directing most of the disclosed examples to this type of implementation, however, it will be understood that similar or identical principles as will be described also apply to examples in which the RF signaling is configured to go in theother direction, from the controller device to the head-mounted device.

[0038] To illustrate, FIG. 2 A shows another illustrative extended reality system 200 that, like extended reality system 100, is configured to perform controller pose tracking in accordance with principles described herein. As shown using a similar numbering scheme as employed in FIG. 1A, extended reality system 200 includes a head-mounted device 202 that includes a first IMU 208-1 and a controller device 204 that includes a second IMU 208-2. However, whereas RF transmitter 110 was included in head-mounted device 102 and RF receiver 112 was included in controller device 104 in extended reality system 100, extended reality system 200 shows that an RF receiver 212 is included in head-mounted device 202, while an RF transmitter 210 is included in controller device 204. Additionally, a camera 213 is again included within the head-mounted device for use in determining the pose of headmounted device 202 with respect to the environment using technologies such as VIO and / or SLAM. Consequently, when a processor 206 (analogous to processor 106) performs a process 214 (analogous to process 114), an RF signal 216 (analogous to RF signal 116) is show n to be transmitted in an opposite direction of its analog (i.e., RF signal 116), from controller device 204 to head-mounted device 202. More particularly, processor 206 may be configured to perform process 214 by: determining (e.g., based on first pose data determined by first IMU 208-1 in head-mounted device 202 and second pose data determined by second IMU 208-2 in controller device 204), a relative pose of controller device 204 with respect to headmounted device 202; determining (e.g., using RF signal 216 transmitted by RF transmitter 210 and received by RF receiver 212) a 3D position of controller device 204 relative to headmounted device 202; and compensating (e.g., based on the 3D position determined using RF signal 216) for drift of the relative pose of controller device 204 with respect to headmounted device 202.

[0039] In FIG. 2B, another illustrative flowchart for a method 220 for extended reality controller pose tracking in accordance wdth principles described herein (analogous to method 120) is shown to represent a particular implementation of process 214. Similar to FIG. IB described above, FIG. 2B shows illustrative operations 222-226 according to one implementation with the understanding that other implementations of method 220 (or of process 214) may omit, add to, reorder, and / or modily any of the operations 222-226 in any way as may sen e a particular implementation. In some examples, multiple operations shown in FIG. 2B or described in relation to FIG. 2B may be performed concurrently (e.g., in parallel) with one another, rather than being performed sequentially as illustrated and / or described. Additionally, it will be understood that, as with method 120, method 220 and anyvariations thereof that are described or suggested in the following disclosure, may be implemented as instructions stored on a iion-transilory computer-readable medium that, when executed, cause a processor of a computing device (e.g., processor 206) to perform the method.

[0040] Each of operations 222-226 of method 220 will now be described in more detail as the operations may be performed by an implementation of extended reality system 200. More particularly, method 220 may be performed by the processor 206 that is included in extended reality system 200, whether it be incorporated entirely within head-mounted device 202, entirely within controller device 204, or distributed across both of these devices and / or other system components. Specifically, at operation 222, extended reality system 200 may determine a relative pose of controller device 204 with respect to head-mounted device 202, the relative pose being based on: 1) first pose data produced by first IMU 208-1 integrated in head-mounted device 202, and 2) second pose data produced by second IMU 208-2 integrated in controller device 204. At operation 224, extended reality system 200 may determine a 3D position of controller device 204 relative to head-mounted device 202 using the RF signal 216 transmitted from controller device 204 and received by head-mounted device 202. At operation 226, extended reality system 200 may compensate for drift of the relative pose of controller device 204 with respect to head-mounted device 202 (determined at operation 222) based on the 3D position determined using RF signal 216 (at operation 224).

[0041] FIG. 3 shows an illustrative world coordinate system 302 with which the poses of a head-mounted device (head-mounted device 102 in this example) and a controller device (controller device 104 in this example) may be registered in accordance with principles described herein. For example, an extended reality’ system (e.g., extended reality system 100, extended reality system 200, etc.) may be configured to account for certain aspects of an environment 304 in which an extended reality experience is being presented. For example, objects in environment 304 may be passed through an opaque display of the head-mounted device (e g., for certain virtual reality experiences) or may be visible through a partially - transparent display (e.g., for certain augmented reality experiences). In other examples, such objects may be accounted for not by being displayed or presented to the user but, rather, by being associated with a boundary that the user is not to cross (e.g., for safety reasons, etc.). In any case, a registration 306 of a pose 308 of head-mounted device 102 may be performed with respect to illustrative world coordinate system 302 so that augmented content may be placed and aligned properly, safety boundaries may be implemented accurately, externalobjects passed through may align with reality precisely, and so forth.

[0042] As shown in FIG. 3, the pose 308 of head-mounted device 102 may be derived based on pose data from IMU 108-1 and based on images captures by camera 113 (e.g., to register the pose to the world coordinate system) in the ways that have been described. For example, when an accelerometer included in IMU 108-1 detects acceleration data (one type of pose data), a first integral of the detected acceleration may be determined to indicate a velocity associated with pose 308, a second integral of the detected acceleration may be determined to indicate a position associated with pose 308, and so forth. Other pose data (e.g., data detected by a gyroscope sensor in IMU 108-1, etc.) may similarly be used to determine orientation-related or other aspects of pose 308 in a similar way. While IMU 108-1 may not have any inherent connection to environment 304 to produce pose data that inherently is tied to illustrative world coordinate system 302, pose 308 may be registered or calibrated to correlate to world coordinate system 302 based on the images captured by camera 113 using a variety of techniques represented by registration 306. As one example, the registering of pose 308 of head-mounted device 102 with world coordinate system 302 (i. e. , registration 306) may be performed based on a visual-inertial odometry (VIO) technique. As another example, the registering of pose 308 of head-mounted device 102 with world coordinate system 302 (i.e., registration 306) may be performed based on a simultaneous localization and mapping (SLAM) technique. In some examples, a combination of VIO and SLAM techniques, as well as other such techniques, may be employed to initialize and / or maintain registration 306 such that pose 308 may accurately and continuously correlate with world coordinate system 302.

[0043] Along with determining pose 308 of head-mounted device 102 based on the first pose data from IMU 108-1 and registering this pose 308 of head-mounted device 102 with the world coordinate system 302 corresponding to environment 304, a calibration 310 of a pose 312 of a controller device (e.g., controller device 104 in this example) may also be initialized and maintained with respect to pose 308 (e.g., so as to avoid drift as has been described). Specifically, the compensating for the drift of the relative pose of controller device 104 with respect to head-mounted device 102 (described above, for example, in relation to operations 126 and 226) may include 1) initially calibrating and 2) periodically recalibrating, at a fixed rate (e.g., 0.5 Hz, 1 Hz, 10 Hz, etc.), the relative pose with respect to the world coordinate system. To illustrate, pose 312 of controller device 104 is shown to be derived based on pose data generated by IMU 108-2, as has been described. This pose 312 may be determined and based on IMU-derived pose data from IMU 108-2 in similar ways asdescribed above for pose 308 and IMU 108-1. To prevent pose 312 from drifting, over time, from pose 308 and its registration 306 with illustrative world coordinate system 302. however, calibration 310 (which may include both an initial calibration and a periodic recalibration, as mentioned above) may be performed to compensate for the drift and / or otherwise to constrain or correct the drift to which IMU-derived pose 312 is prone.

[0044] As has been described, compensation for the drift of the relationship between poses 308 and 312 (e.g., the initial calibration and periodic recalibration represented by calibration 310 described above) may be performed based on a 3D position observation between the controller device and the head-mounted device that is observed using an RF signal (e.g., RF signal 116) such as a mm-wave radar signal exchanged between the devices. While a distance observation alone might allow for certain compensation of certain aspects of the relative pose between the devices, for implementations involving 6dof poses, the distance observation alone would not be informative with regard to other aspects of the relative pose. Accordingly, in some implementations, additional RF-based (e.g., radar-based) observations may be made along with the distance observation to further inform the system on these other aspects of the relative pose that may be relevant or of interest.

[0045] Such observations may be achieved in any suitable manner. For example, in certain implementations, an RF signal (e.g., RF signal 116) transmitted by an RF transmitter (e.g., RF transmitter 110) integrated in head-mounted device 102 may be received by at least three antennas of an RF receiver (also referred to informally as three RF receivers in some cases) integrated in the controller device. The at least three antennas of the RF receiver may be arranged non-collinearly on a plane associated with the controller device, for example, so as to allow for an angle of arrival to be determined with respect to the plane on which the RF receiver antennas are non-collinearly arranged. In such examples, extended reality system 100 may determine (e.g., as an additional step in process 114 or method 120) an angle of arrival of the RF signal with respect to the plane associated with the controller device. For instance, this angle of arrival may be based on a receiving of the RF signal at the at least three RF receivers (i.e., the at least three antennas) integrated in the controller device. The compensating for the drift of the relative pose of the controller device with respect to the head-mounted device may then further be based on this angle of arrival that has been determined for the RF signal with respect to the plane. For example, position-related aspects of a 6dof relative pose may be compensated and constrained based on a radar-derived distance observation between the devices, while orientation-related aspects of the 6dof relative pose may be compensated and constrained based on this radar-derived angle ofarrival observation.

[0046] To illustrate, FIG. 4A shows example aspects of how a relative position and orientation of a controller device (controller device 104 in this example) with respect to a head-mounted device (head-mounted device 102 in this example) may be observed in accordance with principles described herein. Specifically, as shown, an implementation of RF transmitter 110 included within head-mounted device 102 is shown to transmit an implementation of RF signal 1 16 to controller device 104. Within controller device 104, three separate antennas 112-1, 112-2, and 112-3 (also referred to as RF receivers 112-1 through 112-3) of the RF receiver 112 described above are shown to be arranged non-collinearly (i.e., such that no line can be drawn to connect all three of the receivers) on a plane 402 associated with controller device 104.

[0047] When RF signal 116 is transmitted by RF transmitter 110, the relative timing of when RF signal 116 is received at each of antennas 112-1, 112-2, and 112-3 reveals the not only the range of each antenna 112 (which, averaged or otherwise combined together, may be constitute a range or distance observation for controller device 104 as a unit), but also an elevation angle measurement and an azimuth angle measurement that can be combined into an overall angle of arrival measurement of plane 402 with respect to RF transmitter 110. As the arrangement of plane 402 may be known for a given design of controller device 104, this angle of arrival of RF signal 116 to plane 402 may be used to determine an orientation observation for controller device 104 with respect to head-mounted device 102. Even if this observation is insufficient, by itself, to reliably indicate the orientation -related aspects of a 6dof relative pose of controller device 104 (e.g., due to dynamic movements of the device, symmetry' of the angle arrival from multiple possible transmission locations, etc.), the observation may, when accounted for together with an IMU-derived 6dof pose of controller device 104 (e.g., pose 312), provide for the accurate and efficient compensation and recalibration of the relative controller device pose that has been described.

[0048] In some implementations, a head-mounted device such as head-mounted device 102 may be configured for use with more than one controller device such as controller device 104. As such, for example, along with the operations of method 120 associated with controller device 104 (i.e., a first controller device), similar operations associated with another controller device (i.e., a second controller device) may also be performed concurrently (e.g., so as to allow the user to have controllers for both hands, as well as, in some implementations, controllers for use with other body parts, controllers operated by other entities such as other users, etc.). More specifically, the operations of method 120 may besupplemented by additional operations such as the following. The extended reality system may be configured to estimate a second relative pose of a second controller device with respect to the head-mounted device, the second relative pose being based on the first pose data (produced by the IMU in the head-mounted device) and third pose data produced by a third IMU integrated in the second controller device. The extended reality' system may then be configured to determine a second 3D position of the second controller device relative to the head-mounted device using the RF signal transmitted from the head-mounted device (and received by not only the controller device but also the second controller device), and to compensate for second drift of the second relative pose of the second controller device with respect to the head-mounted device based on the second 3D position determined using the RF signal.

[0049] To illustrate, FIG. 4B shows illustrative aspects of how relative positions and orientations of a plurality of controller devices with respect to a head-mounted device may be observed in accordance with principles described herein. As with FIG. 4A, FIG. 4B shows an implementation of head-mounted device 102 that features an RF transmitter 110 that transmits an RF signal 116. Rather than a singular controller device 104 such as shown in FIG. 4A, however, FIG. 4B shows a pair of controller devices 104-1 and 104-2 that each include respective RF receivers 112 arranged non-collinearly on respective planes 402. More particularly , controller device 104-1 is shown to include three antennas 112-1, 112-2, and 112-3 of a first RF receiver non-collinearly arranged on a plane 402-1 associated with controller device 104-1 , while controller device 104-2 is shown to include three antennas 112-4, 112-5, and 112-6 of a second RF receiver non-collinearly arranged on a plane 402-2 associated with controller device 104-2.

[0050] It will be understood that additional controller devices, each with one or more respective RF receiver antennas (e.g., including three or more respective antennas arranged non-collinearly on a plane similar to those explicitly shown in FIG. 4B), may further be included with head-mounted device 102 in the extended reality system shown in FIG. 4B. These further controller devices would not necessarily need to have the same number or arrangement of RF receiver antennas as shown for controller devices 104-1 and 104-2, nor do controller devices 104-1 and 104-2 necessarily need to have the same number and arrangement of RF receiver antennas (though they are shown to in this example for purposes of illustration).

[0051] While the configuration shown in FIGS. 4 A and 4B align with the type of extended reality system implementation described and illustrated in relation to FIG. 1 A (i.e.,extended reality system 100 with its RF signal 116 being transmitted from head-mounted device 102 to controller device 104), it will be understood that similar extended reality system configurations, including multi-controller configurations such as shown in FIG. 4B, could also be implemented with the type of extended reality' system implementation described and illustrated in relation to FIG. 2A (i.e., extended reality system 200 with its RF signal 216 being transmitted from controller device 204 to head-mounted device 202). For example, a head-mounted device (e.g., head-mounted device 202) could include three or more RF receiver antennas arranged non-collinearly with respect to a plane associated with the headmounted device (e.g., analogous to antennas 112-1 through 112-3 or antennas 112-4 through 112-6 in FIG. 4B), then each of a plurality of controller devices (e.g., analogous to controller devices 104-1 and 104-2) may include respective RF transmitters that transmit RF signals (e.g., on different frequencies or in other distinguishable ways so as to be differentiable by the head-mounted device) to be received by the head-mounted device and used to determine an angle of arrival indicative of the position of the controllers in space with respect to the head-mounted device.

[0052] RF-based measurements such as mm-wave radar measurements have been described as being used in conjunction with IMU-derived pose data to correct and compensate for the IMU-derived pose data as the estimated pose of a controller device tends to drift relative to a pose of a head-mounted device. As has been mentioned, this compensation may involve using a sensor fusion technique to synthesize the pose data derived from the IMUs with the measurements made using the RF signal. For instance, the compensating for the drift of the relative pose of the controller device with respect to the head-mounted device may be performed using a Kalman fdter to predict a future state of a state vector based on observation of an observation vector when the RF signal is received by the controller device. In this example, the state vector may include a representation of the relative pose of the controller device with respect to the head-mounted device, while the observation vector may include a representation of the 3D position (e.g., the distance and angle of arrival in certain implementations) determined using the RF signal. In this way, the Kalman fdter may be used to predict the state vector and correct for errors inherent in the IMU sensors that lead to the undesirable drift that has been described.

[0053] To illustrate, FIG. 5 show s illustrative aspects of a sensor fusion technique 500 that may be used to derive a drift-compensated relative pose of a controller device with respect to a head-mounted device in accordance with principles described herein. As shown, sensor fusion technique 500 includes three sources of information that are fused or combinedusing a Kalman filter. Specifically, radar data 502 may represent a relative position and / or speed of the controller device with respect to the head-mounted device as observed using the RF signal (e.g., mm-wave radar in one example). IMU data 504 may represent motion, rotation, and acceleration of the controller device as measured by the IMU of the controller device. VIO / SLAM data 506 may represent a pose (e.g., a 6dof pose) of the head-mounted device with respect to the world coordinate system. As shown, data from all of these sources may be processed by a Kalman filter 508 to ultimately determine the compensated relative controller pose of the controller (i.e., analogous to the calibrated and recalibrated pose 312 described and illustrated above for extended reality system 100).

[0054] Kalman filter 508 may be implemented as any suitable type of Kalman filter configured to perform sensor fusion of the data from the data sources shown. In some implementations, for instance, Kalman filter 508 may be implemented as a discrete-time extended Kalman filter (EKF). In a dashed box under Kalman filter 508, equations (i.e., a “System Equation” and a “Observation Equation”) are illustrated for this type of filter. Due to the discrete time nature of this filter, the variable k represents discrete time units processed by the filter, and the future state of the state vector, X, is shown in the system equation as: Xk+i. Xk+i corresponds to a subsequent time unit (k+1) after a current time unit (k) to which the observation, zk (shown in the observation equation) corresponds. It will be understood that operations and computations associated with Kalman filter 508 may be performed byprocessing resources located either on the controller device, the head-mounted device, or distributed between the two.

[0055] The system equation associated with Kalman filter 508 will be understood to describe IMU data 504, which may incorporate some amount of error (e.g., gyroscope and / or accelerometer bias, scale, cross-axis sensitivities, etc.) that are compensated for based on the observations represented in radar data 502 and described by the observation equation. More particularly, X may represent the state vector (with Xk+i being a prediction of the future of the state vector at the next time unit, as mentioned above), which may correspond to the compensated relative controller pose that comes out of the Kalman filter. Z may then represent the observation vector. The wk and nk terms in these equations represent noise and / or error in the system and the system dynamics. The k term represents how the state vector X relates at one time unit (k) to the next discrete time unit (k+1) and is based on the IMU data 504 being measured. The Hk term similarly represents the relationship between Xk and the observations zkthat are represented by radar data 502.

[0056] As show n, the state vector (“State Vector (X)’") may incorporate various aspects of IMU-measured pose data and IMU error. For example, as shown, the representation of the relative pose of the controller device with respect to the head-mounted device (represented by the state vector, X) may include at least: a three-dimensional position of the controller device (“Position (x, y, z)”), a three-dimensional velocity7of the controller device (“Velocity7(x. y, z)”). a three-dimensional orientation of the controller device (“Orientation (r, p, y),” for roll, pitch, and yaw), and a set of error values associated with the IMU included in the controller device. In an implementation in which this IMU includes a gyroscope sensor and an acceleration sensor, the set of error values associated with this IMU may include, as shown: a three-dimensional bias for the gyroscope sensor (“Gyro Bias (x, y, z)”), a three-dimensional bias for the acceleration sensor (“Accel. Bias (x, y, z)”), a three- dimensional scale factor for the gyroscope sensor (“Gyro Scale (x, y, z)”), a three- dimensional scale factor for the acceleration sensor (“Accel. Scale (x, y, z)”), a three- dimensional misalignment error (i.e., a cross-axis sensitivity7error) for the gyroscope sensor (“Gyro Misalign (x, y, z)”), and a three-dimensional misalignment error for the acceleration sensor (“Accel. Misalign, (x, y, z)”).

[0057] The relative speed of the controller device w ith respect to the head-mounted device may be based on a doppler speed detected when the RF signal is received by the controller device, and the angle of arrival of the RF signal with respect to the plane may be based on an elevation angle and an azimuth angle detected when the RF signal is received by7at least three RF receivers arranged non-collinearly on the plane. As such, the observation vector may include, as shown, a distance (i.e., range) of the controller device with respect to the head-mounted device (“Distance”), a representation of a relative speed of the controller device with respect to the head-mounted device (“Doppler”), an angle of arrival of the RF signal with respect to a plane associated with the controller device (“Azimuth” and “Elevation”), a three-dimensional position of the head-mounted device (“HMD Position (x, y, z)”), and a velocity of the head-mounted device (“HMD Velocity (x, y, z)”).

[0058] As has been mentioned, various methods and processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer- readable medium and executable by one or more computing devices. In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory7computer-readable medium (e.g., a memory, etc.), and executes those instructions, thereby performing one or more operations such as the operations described herein. Such instructions may be stored and / or transmitted using any of a variety7of known computer-readable media.

[0059] A computer-readable medium (also referred to as a processor-readable medium) includes any non-transitory medium that participates in providing data (e.g., instructions) that may be read by a computer (e.g., by a processor of a computer). Such a medium may take many forms, including, but not limited to, non-volatile media, and / or volatile media. Non-volatile media may include, for example, optical or magnetic disks and other persistent memory. Volatile media may include, for example, dynamic random-access memory (DRAM), which typically constitutes a main memory. Common forms of computer- readable media include, for example, a disk, hard disk, magnetic tape, any other magnetic medium, a compact disc read-only memory (CD-ROM), a digital video disc (DVD), any other optical medium, random access memory (RAM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EPROM), FLASH- EEPROM, any other memory chip or cartridge, or any other tangible medium from which a computer can read.

[0060] FIG. 6 shows an illustrative computing system 600 that may be used to implement various devices and / or systems described herein. For example, computing system 600 may include or implement (or partially implement) extended reality systems such as extended reality system 100 or extended reality system 200 and / or any components thereof, such as head-mounted devices 102 or 202, controller devices 104 or 204.

[0061] As shown in FIG. 6, computing system 600 may include a communication interface 602, a processor 604. a storage device 606, and an input / output (I / O) module 608 communicatively connected via a communication infrastructure 610. While an illustrative computing system 600 is shown in FIG. 6, the components illustrated in FIG. 6 are not intended to be limiting. Additional or alternative components may be used in other embodiments. Components of computing system 600 shown in FIG. 6 will now be described in additional detail.

[0062] Communication interface 602 may be configured to communicate with one or more computing devices. Examples of communication interface 602 include, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0063] Processor 604 generally represents any type or form of processing unit capable of processing data or interpreting, executing, and / or directing execution of one or more of the instructions, processes, and / or operations described herein. Processor 604 may direct execution of operations in accordance with one or more applications 612 or other computer-executable instructions such as may be stored in storage device 606 or another computer- readable medium.

[0064] Storage device 606 may include one or more data storage media, devices, or configurations and may employ any type, form, and combination of data storage media and / or device. For example, storage device 606 may include, but is not limited to, a hard drive, network drive, flash drive, magnetic disc, optical disc, RAM. dynamic RAM, other non-volatile and / or volatile data storage units, or a combination or sub-combination thereof. Electronic data, including data described herein, may be temporarily and / or permanently stored in storage device 606. For example, data representative of one or more executable applications 612 configured to direct processor 604 to perform any of the operations described herein may be stored within storage device 606. In some examples, data may be arranged in one or more databases residing within storage device 606.

[0065] I / O module 608 may include one or more I / O modules configured to receive user input and provide user output. One or more I / O modules may be used to receive input for a single virtual experience. I / O module 608 may include any hardware, firmware, software, or combination thereof supportive of input and output capabilities. For example, I / O module 608 may include hardware and / or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., touchscreen display), a receiver (e.g., an RF or infrared receiver), motion sensors, and / or one or more input buttons.

[0066] I / O module 608 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, I / O module 608 is configured to provide graphical data to a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and / or any other graphical content as may sen e a particular implementation.

[0067] In some examples, any of the facilities described herein may be implemented by or within one or more components of computing system 600. For example, one or more applications 612 residing within storage device 606 may be configured to direct processor 604 to perform one or more processes or functions associated with processor 106 of extended reality system 100 or processor 206 of extended reality system 200. Likewise, methods 120 and / or 220 (as well as other implementations of processes 114 and / or 214) described above may be incorporated as applications 612 that are implemented as instructions within storage device 606.

[0068] The following examples describe systems and methods for extended reality controller pose tracking in accordance with principles described herein:

[0069] 1. A method comprising: determining a relative pose of a controller device with respect to a head-mounted device, wherein the determining the relative pose is based on: first pose data produced by a first inertial measurement unit (IMU) integrated in the headmounted device, and second pose data produced by a second IMU integrated in the controller device; determining a three-dimensional position of the controller device relative to the headmounted device using a radio frequency (RF) signal transmitted from the head-mounted device and received by the controller device; and compensating for drift of the relative pose of the controller device with respect to the head-mounted device based on the three- dimensional position determined using the RF signal.

[0070] 2. The method of any of the preceding examples, wherein the RF signal is a millimeter-wave radar signal.

[0071] 3. The method of any one of any of the preceding examples, wherein: the RF signal is transmitted by an RF transmitter integrated in the head-mounted device and received by at least three antennas of an RF receiver integrated in the controller device, the at least three antennas being arranged non-collinearly on a plane associated with the controller device; the determining the three-dimensional position includes determining an angle of arrival of the RF signal with respect to the plane associated with the controller device based on a receiving of the RF signal at the at least three antennas of the RF receiver integrated in the controller device; and the compensating for the drift of the relative pose of the controller device w ith respect to the head-mounted device is further based on the angle of arrival of the RF signal with respect to the plane.

[0072] 4. The method of any one of any of the preceding examples, wherein the relative pose of the controller device with respect to the head-mounted device is a six- degrees-of-freedom (6dof) pose of the controller device with respect to the head-mounted device.

[0073] 5. The method of any one of any of the preceding examples, further comprising: determining a pose of the head-mounted device based on the first pose data; and registering the pose of the head-mounted device with a world coordinate system corresponding to an environment in which the head-mounted device is located; wherein the compensating for the drift of the relative pose of the controller device with respect to the head-mounted device includes initially calibrating and periodically recalibrating, at a fixed rate, the relative pose with respect to the world coordinate system.

[0074] 6. The method of any of the preceding examples, wherein the registering of the pose of the head-mounted device with the world coordinate system is performed based on at least one of a visual-inertial odometry (VIO) technique or a simultaneous localization and mapping (SLAM) technique.

[0075] 7. The method of any one of any of the preceding examples, wherein: the compensating for the drift of the relative pose of the controller device with respect to the head-mounted device includes using a Kalman filter to predict a future state of a state vector based on observation of an observation vector when the RF signal is received by the controller device; the state vector includes a representation of the relative pose of the controller device with respect to the head-mounted device; and the observation vector includes a representation of the three-dimensional position determined using the RF signal.

[0076] 8. The method of any of the preceding examples, wherein the Kalman filter is a discrete-time extended Kalman filter and the future state of the state vector corresponds to a subsequent time unit after a current time unit to which the observation corresponds.

[0077] 9. The method of any of the preceding examples, wherein: the representation of the relative pose of the controller device with respect to the head-mounted device includes a three-dimensional position of the controller device, a three-dimensional velocity of the controller device, a three-dimensional orientation of the controller device, and a set of error values associated with the second IMU; and the observation vector includes a representation of a relative speed of the controller device with respect to the head-mounted device, an angle of arrival of the RF signal with respect to a plane associated with the controller device, a three-dimensional position of the head-mounted device, and a velocity of the head-mounted device.

[0078] 10. The method of any of the preceding examples, wherein: the relative speed of the controller device with respect to the head-mounted device is based on a doppler speed detected when the RF signal is received by the controller device; and the angle of arrival of the RF signal with respect to the plane is based on an elevation angle and an azimuth angle detected when the RF signal is received by at least three antennas of an RF receiver, the at least three antennas being arranged non-collinearly on the plane.

[0079] 11. The method of any of the preceding examples, wherein: the second IMU includes a gyroscope sensor and an acceleration sensor; and the set of error values associated with the second IMU includes: a three-dimensional bias for the gyroscope sensor, a three- dimensional bias for the acceleration sensor, a three-dimensional scale factor for the gyroscope sensor, a three-dimensional scale factor for the acceleration sensor, a three-dimensional misalignment error for the gyroscope sensor, and a three-dimensional misalignment error for the acceleration sensor.

[0080] 12. The method of any one of any of the preceding examples, further comprising: determining a second relative pose of a second controller device with respect to the head-mounted device, the second relative pose being based on the first pose data and third pose data produced by a third IMU integrated in the second controller device; determining a second three-dimensional position of the second controller device relative to the headmounted device using the RF signal transmitted from the head-mounted device and received by the controller device and the second controller device; and compensating for second drift of the second relative pose of the second controller device with respect to the head-mounted device based on the second three-dimensional position determined using the RF signal.

[0081] 13. The method of any one of any of the preceding examples, implemented as instructions stored on a non-transilory computer-readable medium that, when executed, cause a processor of a computing device to perform the method.

[0082] 14. A method comprising: determining a relative pose of a controller device with respect to a head-mounted device, wherein the determining the relative pose is based on: first pose data produced by a first inertial measurement unit (IMU) integrated in the headmounted device, and second pose data produced by a second IMU integrated in the controller device; determining a three-dimensional position of the controller device relative to the headmounted device using a radio frequency (RF) signal transmitted from the controller device and received by the head-mounted device; and compensating for drift of the relative pose of the controller device with respect to the head-mounted device based on the three-dimensional position determined using the RF signal.

[0083] 15. The method of any one of the preceding examples, wherein the RF signal is a millimeter-wave radar signal and the relative pose of the controller device with respect to the head-mounted device is a six-degrees-of-freedom (6dol) pose of the controller device with respect to the head-mounted device.

[0084] 16. An extended reality system comprising: a head-mounted device including a first inertial measurement unit (IMU) and a radio frequency (RF) transmitter; a controller device including a second IMU and an RF receiver; and a processor configured to perform a process comprising: determining, based on first pose data produced by the first IMU in the head-mounted device and second pose data produced by the second IMU in the controller device, a relative pose of the controller device with respect to the head-mounted device, determining, using an RF signal transmitted by the RF transmitter and received by the RFreceiver, a three-dimensional position of the controller device relative to the head-mounted device, and compensating, based on the three-dimensional position determined using the RF signal, for drift of the relative pose of the controller device with respect to the head-mounted device.

[0085] 17. The extended reality system of any of the preceding examples, wherein: the RF receiver is a millimeter-wave radar receiver, the RF transmitter is a millimeter-wave radar transmitter, and the RF signal is a millimeter-wave radar signal; and the relative pose of the controller device with respect to the head-mounted device is a six-degrees-of-freedom (6dof) pose of the controller device with respect to the head-mounted device.

[0086] 18. The extended reality system of any one of any of the preceding examples, wherein: the RF receiver includes at least three antennas arranged non-collinearly on a plane associated with the controller device; the process further comprises determining an angle of arrival of the RF signal with respect to the plane associated with the controller device based on a receiving of the RF signal at the at least three antennas of the RF receiver included in the controller device; and the compensating for the drift of the relative pose of the controller device with respect to the head-mounted device is further based on the angle of arrival of the RF signal with respect to the plane.

[0087] 19. The extended reality system of any one of any of the preceding examples, wherein: the process further comprises: determining a pose of the head-mounted device based on the first pose data; and registering the pose of the head-mounted device with a world coordinate system corresponding to an environment in which the head-mounted device is located, the registering of the pose being performed based on at least one of a visual-inertial odometry (VIO) technique or a simultaneous localization and mapping (SLAM) technique; and the compensating for the drift of the relative pose of the controller device with respect to the head-mounted device includes initially calibrating and periodically recalibrating, at a fixed rate, the relative pose with respect to the world coordinate system.

[0088] 20. The extended reality system of any one of any of the preceding examples, wherein: the compensating for the drift of the relative pose of the controller device with respect to the head-mounted device includes using a Kalman filter to predict a future state of a state vector based on observation of an observation vector when the RF signal is received by the controller device; the state vector includes a representation of the relative pose of the controller device with respect to the head-mounted device; and the observation vector includes a representation of the three-dimensional position determined using the RF signal.

[0089] 21. An extended reality system comprising: a head-mounted device includinga first inertial measurement unit (IMU) and a radio frequency (RF) receiver; a controller device including a second IMU and an RF transmitter; and a processor configured to perform a process comprising: determining, based on first pose data produced by the first IMU in the head-mounted device and second pose data produced by the second IMU in the controller device, a relative pose of the controller device with respect to the head-mounted device, determining, using an RF signal transmitted by the RF transmitter and received by the RF receiver, a three-dimensional position of the controller device relative to the head-mounted device, and compensating, based on the three-dimensional position determined using the RF signal, for drift of the relative pose of the controller device with respect to the head-mounted device.

[0090] 22. The extended reality system of any one of any of the preceding examples, wherein: the RF receiver is a millimeter-wave radar receiver, the RF transmitter is a millimeter-wave radar transmitter, and the RF signal is a millimeter-wave radar signal; and the relative pose of the controller device with respect to the head-mounted device is a six- degrees-of-freedom (6dof) pose of the controller device with respect to the head-mounted device.

[0091] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry. specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0092] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the description and claims. In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.

[0093] Specific structural and functional details disclosed herein are merely representative for purposes of describing example implementations. Example implementations, however, may be embodied in many alternate forms and should not beconstrued as limited to only the implementations set forth herein.

[0094] It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. A first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the implementations of the disclosure. As used herein, the term and / or includes any and all combinations of one or more of the associated listed items.

[0095] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the implementations. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used in this specification, specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0096] It will be understood that when an element is referred to as being “coupled,” “connected,” or “responsive” to, or “on,” another element, it can be directly coupled, connected, or responsive to, or on, the other element, or intervening elements may also be present. In contrast, when an element is referred to as being “directly coupled,” “directly connected,” or “directly responsive” to, or “directly on,” another element, there are no intervening elements present. As used herein the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0097] Spatially relative terms, such as “beneath,” “below,” “lower,” “above,” “upper,” and the like, may be used herein for ease of description to describe one element or feature in relationship to another element(s) or feature(s) as illustrated in the figures. It wil 1 be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as “below” or “beneath” other elements or features would then be oriented “above” the other elements or features. Thus, the term “below” can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 130 degrees or at other orientations) and the spatially relative descriptors used herein may be interpreted accordingly.

[0098] It will be understood that although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, a “first” element could be termed a “second” element without departing from the teachings of the present implementations.

[0099] Unless otherwise defined, the terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which these concepts belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and / or the present specification and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0100] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes, and equivalents may occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover such modifications and changes as fall within the scope of the implementations. It will be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and / or sub-combinations of the functions, components, and / or features of the different implementations described. As such, the scope of the present disclosure is not limited to the particular combinations hereafter claimed, but instead extends to encompass any combination of features or example implementations described herein irrespective of whether or not that particular combination has been specifically enumerated in the accompanying claims at this time.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: determining a relative pose of a controller device with respect to a head-mounted device, wherein the determining the relative pose is based on: first pose data produced by a first inertial measurement unit (IMU) integrated in the head-mounted device, and second pose data produced by a second IMU integrated in the controller device; determining a three-dimensional position of the controller device relative to the headmounted device using a radio frequency (RF) signal transmitted from the head-mounted device and received by the controller device; and compensating for drift of the relative pose of the controller device with respect to the head-mounted device based on the three-dimensional position determined using the RF signal.

2. The method of claim 1, wherein the RF signal is a millimeter-wave radar signal.

3. The method of any one of claims 1-2, wherein: the RF signal is transmitted by an RF transmitter integrated in the head-mounted device and received by at least three antennas of an RF receiver integrated in the controller device, the at least three antennas being arranged non-collinearly on a plane associated with the controller device; the determining the three-dimensional position includes determining an angle of arrival of the RF signal with respect to the plane associated with the controller device based on a receiving of the RF signal at the at least three antennas of the RF receiver integrated in the controller device; and the compensating for the drift of the relative pose of the controller device with respect to the head-mounted device is further based on the angle of arrival of the RF signal with respect to the plane.

4. The method of any one of claims 1-3, wherein the relative pose of the controller device with respect to the head-mounted device is a six-degrees-of-freedom (6dof) pose of the controller device with respect to the head-mounted device.

5. The method of any one of claims 1-4, further comprising: determining a pose of the head-mounted device based on the first pose data; and registering the pose of the head-mounted device with a world coordinate system corresponding to an environment in which the head-mounted device is located; wherein the compensating for the drift of the relative pose of the controller device with respect to the head-mounted device includes initially calibrating and periodically recalibrating, at a fixed rate, the relative pose with respect to the world coordinate system.

6. The method of claim 5, wherein the registering of the pose of the headmounted device with the world coordinate system is performed based on at least one of a visual-inertial odometry (VIO) technique or a simultaneous localization and mapping (SLAM) technique.

7. The method of any one of claims 1-6, wherein: the compensating for the drift of the relative pose of the controller device with respect to the head-mounted device includes using a Kalman filter to predict a future state of a state vector based on observation of an observation vector when the RF signal is received by the controller device; the state vector includes a representation of the relative pose of the controller device with respect to the head-mounted device; and the observation vector includes a representation of the three-dimensional position determined using the RF signal.

8. The method of claim 7. wherein the Kalman filter is a discrete-time extended Kalman filter and the future state of the state vector corresponds to a subsequent time unit after a current time unit to which the observation corresponds.

9. The method of any one of claims 7-8, wherein: the representation of the relative pose of the controller device with respect to the head-mounted device includes a three-dimensional position of the controller device, a three- dimensional velocity of the controller device, a three-dimensional orientation of the controller device, and a set of error values associated with the second IMU; and the observation vector includes a representation of a relative speed of the controller device with respect to the head-mounted device, an angle of arrival of the RF signal with respect to a plane associated with the controller device, a three-dimensional position of the head-mounted device, and a velocity7of the head-mounted device.

10. The method of claim 9. wherein: the relative speed of the controller device with respect to the head-mounted device is based on a doppler speed detected when the RF signal is received by the controller device; and the angle of arrival of the RF signal with respect to the plane is based on an elevation angle and an azimuth angle detected when the RF signal is received by at least three antennas of an RF receiver, the at least three antennas being arranged non-collinearly on the plane.

11. The method of any one of claims 9-10, wherein: the second IMU includes a gyroscope sensor and an acceleration sensor; and the set of error values associated with the second IMU includes: a three-dimensional bias for the gyroscope sensor, a three-dimensional bias for the acceleration sensor, a three-dimensional scale factor for the gyroscope sensor, a three-dimensional scale factor for the acceleration sensor, a three-dimensional misalignment error for the gyroscope sensor, and a three-dimensional misalignment error for the acceleration sensor.

12. The method of any one of claims 1-11, further comprising: determining a second relative pose of a second controller device with respect to the head-mounted device, the second relative pose being based on the first pose data and third pose data produced by a third IMU integrated in the second controller device; determining a second three-dimensional position of the second controller device relative to the head-mounted device using the RF signal transmitted from the head-mounted device and received by the controller device and the second controller device; and compensating for second drift of the second relative pose of the second controller device with respect to the head-mounted device based on the second three-dimensional position determined using the RF signal.

13. The method of any one of claims 1-12, implemented as instructions stored on anon-transitory computer-readable medium that, when executed, cause a processor of a computing device to perform the method.

14. A method comprising: determining a relative pose of a controller device with respect to a head-mounted device, wherein the determining the relative pose is based on: first pose data produced by a first inertial measurement unit (IMU) integrated in the head-mounted device, and second pose data produced by a second IMU integrated in the controller device; determining a three-dimensional position of the controller device relative to the headmounted device using a radio frequency (RF) signal transmitted from the controller device and received by the head-mounted device; and compensating for drift of the relative pose of the controller device with respect to the head-mounted device based on the three-dimensional position determined using the RF signal.

15. The method of claim 14, wherein the RF signal is a millimeter- wave radar signal and the relative pose of the controller device with respect to the head-mounted device is a six-degrees-of-freedom (6dof) pose of the controller device with respect to the headmounted device.

16. An extended reality system comprising: a head-mounted device including a first inertial measurement unit (IMU) and a radio frequency (RF) transmitter; a controller device including a second IMU and an RF receiver; and a processor configured to perform a process comprising: determining, based on first pose data produced by the first IMU in the headmounted device and second pose data produced by the second IMU in the controller device, a relative pose of the controller device with respect to the head-mounted device, determining, using an RF signal transmitted by the RF transmitter and received by the RF receiver, a three-dimensional position of the controller device relative to the head-mounted device, and compensating, based on the three-dimensional position determined using the RF signal, for drift of the relative pose of the controller device with respect to the headmounted device.

17. The extended reality system of claim 16, wherein: the RF receiver is a millimeter-wave radar receiver, the RF transmitter is a millimeterwave radar transmitter, and the RF signal is a millimeter-wave radar signal; and the relative pose of the controller device with respect to the head-mounted device is a six-degrees-of-freedom (6dof) pose of the controller device with respect to the head-mounted device.

18. The extended reality system of any one of claims 16-17, wherein: the RF receiver includes at least three antennas arranged non-collinearly on a plane associated with the controller device; the process further comprises determining an angle of arrival of the RF signal with respect to the plane associated with the controller device based on a receiving of the RF signal at the at least three antennas of the RF receiver included in the controller device; and the compensating for the drift of the relative pose of the controller device with respect to the head-mounted device is further based on the angle of arrival of the RF signal with respect to the plane.

19. The extended reality system of any one of claims 16-18, wherein: the process further comprises: determining a pose of the head-mounted device based on the first pose data; and registering the pose of the head-mounted device with a world coordinate system corresponding to an environment in which the head-mounted device is located, the registering of the pose being performed based on at least one of a visual-inertial odometry (VIO) technique or a simultaneous localization and mapping (SLAM) technique; and the compensating for the drift of the relative pose of the controller device with respect to the head-mounted device includes initially calibrating and periodically recalibrating, at a fixed rate, the relative pose with respect to the world coordinate system.

20. The extended reality system of any one of claims 16-19, wherein: the compensating for the drift of the relative pose of the controller device with respect to the head-mounted device includes using a Kalman filter to predict a future state of a state vector based on observation of an observation vector when the RF signal is received by the controller device; the state vector includes a representation of the relative pose of the controller device with respect to the head-mounted device; and the observation vector includes a representation of the three-dimensional position determined using the RF signal.

21. An extended reality system comprising:A head-mounted device including a first inertial measurement unit (IMU) and a radio frequency (RF) receiver; a controller device including a second IMU and an RF transmitter; and a processor configured to perform a process comprising: determining, based on first pose data produced by the first IMU in the headmounted device and second pose data produced by the second IMU in the controller device, a relative pose of the controller device with respect to the head-mounted device, determining, using an RF signal transmitted by the RF transmitter and received by the RF receiver, a three-dimensional position of the controller device relative to the head-mounted device, andcompensating, based on the three-dimensional position determined using the RF signal, for drift of the relative pose of the controller device with respect to the headmounted device.

22. The extended reality7system of claim 21, wherein: the RF receiver is a millimeter-wave radar receiver, the RF transmitter is a millimeterwave radar transmitter, and the RF signal is a millimeter-wave radar signal; and the relative pose of the controller device with respect to the head-mounted device is a six-degrees-of-freedom (6dof) pose of the controller device with respect to the head-mounted device.

Images