Systems and methods for object tracking using fused data

Abstract

In one embodiment, a method includes capturing a first image depicting at least a portion of a user's hand holding a controller in an environment using one or more cameras implemented in a wearable device worn by a user, identifying one or more features from the first image to estimate a pose of the user's hand, estimating a first pose of the controller based on the pose of the user's hand and the estimated grip that defines a relative pose between the user's hand and the controller, receiving IMU data of the controller, and estimating a second pose of the controller by updating the first pose of the controller using the IMU data of the controller. The method utilizes multiple data sources to track the controller under varying conditions in an environment and provide consistently accurate controller tracking.

Description

[Technical Field]

[0001] The present disclosure relates generally to object tracking, and more particularly to methods, apparatus, and systems for object tracking based on fusion of feature estimation and sensor data. [Background technology]

[0002] Input commands provided to AR / VR devices are typically based on controller tracking or hand tracking. The controller is tracked using a known pattern formed by infrared (IR) light-emitting diodes (LEDs) on the controller, allowing commands to be input to specific locations in the environment via buttons on the controller. Input commands can also be made through hand gestures by tracking hand features. For example, a user can turn pages in a virtual book by tracking hand swipe gestures. However, controller tracking is more costly due to the additional hardware required, such as an IR camera and IR LED lights on the controller, which can be interfered with by occlusions or other light sources, and hand tracking is less accurate. Summary of the Invention

[0003] To address the above-mentioned problems, a method, apparatus, and system for tracking a controller by estimating hand grip and adjusting hand grip based on inertial measurement unit (IMU) data from the controller are disclosed. The present disclosure provides a method for tracking a controller without implementing LEDs in the controller (e.g., without tracking LED light patterns), thereby providing a cost-effective and accurate method for tracking a controller. The method disclosed herein may estimate a user's hand grip based on tracking features identified from captured images of the user's hand, and then estimate a pose of the controller using the estimated grip of the user's hand. Furthermore, the method may receive IMU data from the controller to adjust the estimated pose of the controller and provide a final pose of the controller at a faster frequency.

[0004] It should be noted that the embodiments disclosed herein are merely examples and do not limit the scope of the present disclosure. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein.

[0005] According to one aspect of the present invention, there is provided a method by a computing system that includes capturing a first image depicting at least a portion of a user's hand holding a controller in an environment using one or more cameras implemented in a wearable device worn by a user; identifying one or more features from the first image to estimate a pose of the user's hand; estimating a first pose of the controller based on the pose of the user's hand and an estimated grip that defines a relative pose between the user's hand and the controller; receiving IMU data of the controller; and estimating a second pose of the controller by updating the first pose of the controller using the IMU data of the controller.

[0006] A first image may be captured at a first frequency and controller IMU data is received at a second frequency, the second frequency being higher than the first frequency.

[0007] The method may further include receiving IMU data of the wearable device to estimate a pose of the wearable device, and updating a pose of the user's hand based on the pose of the wearable device.

[0008] The pose of the wearable device may be estimated based on IMU data of the wearable device and the first image of the user.

[0009] The method may further include estimating an IMU predicted pose of the hand based on the updated first pose of the controller and the IMU data of the controller, and estimating a second pose of the hand based on the IMU predicted pose of the hand.

[0010] The method may further include capturing a second image of the user using one or more cameras, the second image of the user depicting at least a portion of the user's hands holding the controller in the environment, identifying one or more features from the second image of the user, and estimating a third pose of the hands based on the one or more features identified from the second image of the user.

[0011] The frequency for estimating the second pose of the hand may be higher than the frequency for estimating the third pose of the hand.

[0012] The wearable device may include one or more cameras configured to capture images of the user, a hand tracking unit configured to estimate a pose of the user's hands, and a controller tracking unit configured to estimate a second pose of the controller.

[0013] Estimating a second pose of the controller may include estimating a pose of the controller relative to the environment based on the estimated grip and an estimated pose of the user's hands relative to the environment, adjusting the pose of the controller relative to the environment based on IMU data of the controller, estimating a pose of the controller relative to the hands based on the adjusted pose of the controller relative to the environment and the IMU of the controller, and estimating a second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hands.

[0014] According to one aspect of the invention, there are provided one or more computer-readable non-transitory storage media embodying software that, when executed, is operable to: capture a first image using one or more cameras implemented in a wearable device worn by a user, the first image depicting at least a portion of a user's hand holding a controller in an environment; identify one or more features from the first image to estimate a pose of the user's hand; estimate a first pose of the controller based on the pose of the user's hand and the estimated grip that defines a relative pose between the user's hand and the controller; receive inertial measurement unit (IMU) data of the controller; and estimate a second pose of the controller by updating the first pose of the controller using the controller IMU data.

[0015] A first image may be captured at a first frequency and controller IMU data is received at a second frequency, the second frequency being higher than the first frequency.

[0016] When executed, the software may further be operable to receive IMU data of the wearable device to estimate a pose of the wearable device, and to update the pose of the user's hand based on the pose of the wearable device.

[0017] The pose of the wearable device may be estimated based on IMU data of the wearable device and the first image of the user.

[0018] When executed, the software may be further operable to estimate an IMU predicted pose of the hand based on the updated first pose of the controller and IMU data of the controller, and to estimate a second pose of the hand based on the IMU predicted pose of the hand.

[0019] When executed, the software may be further operable to capture a second image of the user using the one or more cameras, where the second image of the user depicts at least a portion of the user's hands holding the controller in the environment; identify one or more features from the second image of the user; and estimate a third pose of the hands based on the one or more features identified from the second image of the user.

[0020] The frequency for estimating the second pose of the hand may be higher than the frequency for estimating the third pose of the hand.

[0021] The wearable device may include one or more cameras configured to capture images of the user, a hand tracking unit configured to estimate a pose of the user's hands, and a controller tracking unit configured to estimate a second pose of the controller.

[0022] Estimating a second pose of the controller may include estimating a pose of the controller relative to the environment based on the estimated grip and an estimated pose of the user's hands relative to the environment, adjusting the pose of the controller relative to the environment based on IMU data of the controller, estimating a pose of the controller relative to the hands based on the adjusted pose of the controller relative to the environment and the IMU of the controller, and estimating a second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hands.

[0023] According to one aspect of the present invention, there is provided a system comprising one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors and including instructions operable when executed by the one or more of the processors, the instructions causing the system to: capture a first image depicting at least a portion of a user's hand holding a controller in an environment using one or more cameras implemented in a wearable device worn by a user; identify one or more features from the first image to estimate a pose of the user's hand; estimate a first pose of the controller based on the pose of the user's hand and the estimated grip that defines a relative pose between the user's hand and the controller; receive inertial measurement unit (IMU) data of the controller; and estimate a second pose of the controller by updating the first pose of the controller using the IMU data of the controller.

[0024] The instructions, when executed, may further operate to receive IMU data of the wearable device to estimate a pose of the wearable device, and update a pose of the user's hand based on the pose of the wearable device.

[0025] Embodiments in accordance with the present invention are disclosed in the accompanying claims, which are directed, inter alia, to methods, storage media, systems, and computer program products, and any feature recited in one claim category, e.g., a method, may equally be claimed in another claim category, e.g., a system. Dependencies or references in the accompanying claims are selected for formality reasons only. However, any subject matter (e.g., multiple dependencies) resulting from an intentional reference to any preceding claim may also be claimed, and as a result, any combination of claims and their features is disclosed and may be claimed without regard to the dependencies selected in the accompanying claims. Subject matter that may be claimed includes not only combinations of features recited in the accompanying claims, but also any other combinations of claim features, in which each feature recited in a claim may be combined with any other feature or combination of features in the claim. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or illustrated herein or with any of the features of the accompanying claims.

[0026] Certain aspects of the present disclosure and their embodiments may provide solutions to these and other problems. Various embodiments are proposed herein that address one or more of the problems disclosed herein. The methods disclosed herein may provide a tracking method for a controller that estimates and adjusts the pose of the controller based on grip estimation and IMU data of the controller. Furthermore, based on the pose of the controller relative to the environment and the user's hand, the methods disclosed herein may also provide an IMU-predicted pose of the user's hand to reduce the search range of the user's hand in the next frame. Therefore, certain embodiments disclosed herein may cost-effectively track the controller (e.g., no need to install LEDs) and improve the process time for performing tracking tasks.

[0027] Certain embodiments of the present disclosure may include or be implemented in conjunction with a virtual reality system. A virtual reality is a form of realism that is conditioned in some way before being presented to a user and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or any combination and / or derivative thereof. A virtual reality content may include fully generated content or generated content combined with captured content (e.g., photographs of the real world). A virtual reality content may include video, audio, haptic feedback, or any combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereo video to create a three-dimensional effect for the viewer). Furthermore, in some embodiments, a virtual reality may be associated with, for example, applications, products, accessories, services, or some combination thereof, used to create content in the virtual reality and / or used in the virtual reality (e.g., performing activities within the virtual reality). A virtual reality system that provides virtual reality content may be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing virtual reality content to one or more viewers.

[0028] It should be noted that the embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited thereto. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the above-disclosed embodiments. Embodiments in accordance with the present invention are disclosed in the appended claims, which are directed, inter alia, to methods, storage media, systems, and computer program products. Any feature recited in one claim category, e.g., a method, may also be claimed in another claim category, e.g., a system. Dependencies or references in the appended claims are selected for formality reasons only. However, any subject matter (e.g., multiple dependencies) resulting from intentional reference to any preceding claim may also be claimed, and consequently, any combination of claims and their features is disclosed and may be claimed without regard to the dependencies selected in the appended claims. Subject matter that may be claimed includes not only combinations of features set forth in the appended claims, but also any other combinations of the features of the claims, in which each feature recited in a claim may be combined with any other feature or combination of features in the claim. Furthermore, any of the embodiments and features described or depicted in this specification may be claimed in a separate claim and / or in any combination with any of the embodiments or features described or depicted in this specification or with any of the features of the accompanying claims.

[0029] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate several aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure. [Brief explanation of the drawings]

[0030] [Figure 1A] 1 shows an exemplary diagram of a tracking system of a controller. [Figure 1B] 1 shows an exemplary diagram of a tracking system of a controller. [Figure 2] 1 shows an example diagram of a tracking system architecture using fused sensor data. [Figure 3] 1 shows an exemplary diagram of a tracking system comprising a central module and a controller module for tracking a controller. [Figure 4] 1 shows an exemplary diagram of a tracking system comprising a central module, a controller module, and a remote server for tracking the controller locally or globally. [Figure 5] 1 illustrates one embodiment of a method for tracking controller pose adjustments by fusing user hand feature tracking with controller IMU data. [Figure 6] 1 illustrates an exemplary computer system. DETAILED DESCRIPTION OF THE INVENTION

[0031] Current AR / VR devices are generally paired with a portable / wearable device (e.g., a controller) to provide users with an easy and intuitive way to input commands for the AR / VR device. The controller is typically equipped with at least one inertial measurement unit (IMU) and infrared (IR) light-emitting diodes (LEDs) for the AR / VR device to estimate the pose of the controller and / or track the position of the controller so that a user can perform certain functions via the controller. For example, a user may use the controller to display a visual object in the corner of a room. However, equipping the controller with LEDs increases the manufacturing cost of the controller, and tracking the controller by determining the LED light pattern may be subject to interference under certain environmental conditions. Furthermore, relying entirely on feature tracking to track the controller may result in inaccuracies. Certain embodiments disclosed in the present disclosure provide a method for estimating the pose of the controller by fusing feature tracking data of a user's hand with IMU data of the controller.

[0032] Additionally, certain embodiments disclosed in the present disclosure may provide an IMU predicted pose of the user's hand based on a fusion of the hand's estimated grip with the controller's IMU data to facilitate hand tracking in the next frame. By utilizing the controller's IMU data to adjust the hand's grip, the controller's pose can be updated more frequently to maintain efficient and accurate tracking. Certain embodiments disclosed in the present disclosure may be efficiently and low-costly applied to any type of tracking system, such as a visual inertial odometry (VIO)-based simultaneous localization and mapping (SLAM) tracking system.

[0033] 1A-1B illustrate an exemplary tracking system for tracking a controller, according to a specific embodiment. In FIG. 1A, tracking system 100 includes a central module (not shown) and a controller module 110 (e.g., a controller). The central module includes a camera and at least one processor for tracking controller module 110 within an environment. In a specific embodiment, the central module may be implemented in a wearable device, such as a head-mounted device, to capture images of the tracked object (e.g., the controller implemented by controller module 110). For example, a wearable device with a camera may perform inside-out tracking (e.g., SLAM) of the object. In a specific embodiment, the tracked object may also be tracked by one or more cameras mounted / fixed within the environment, e.g., for outside-in tracking.

[0034] The camera of the central module may capture a first frame 120 depicting at least a portion of a user's hand. More specifically, the first frame 120 depicts at least a portion of the user's hand holding the controller module 110. The central module may identify one or more features 122, 124, 126 of at least a portion of the user's hand from the first frame 120. In particular embodiments, the first frame 120 may include one or more features that at least depict the user's hand holding the controller module 110. In FIG. 1B , the controller module 110 includes a handle 112 for the user to hold. The central module identifies features 122, 124, 126 of the user's hand that can be used to estimate the pose of the user's hand. For example, a region 122 where the space enclosed by the hand's index finger and thumb overlaps the controller 110, an ulnar edge of the hand 124 representing the user's hand holding the controller 110, and a region 126 including the fingertips and the controller 110. The identified features 122, 124, 126 from the first frame 120 may be used to estimate the pose / position of the user's hand. Additionally, the pose of the user's hand may be used to estimate the grip of the user's hand. For example, the pose of the user's hand may be a skeleton / basic geometric shape of the user's hand that represents the user's hand gesture. The estimated grip of the user's hand may be utilized to estimate the pose of the controller module 110 based on the estimated grip of the user's hand, which defines the relative pose between the user's hand and the controller module 110.

[0035] The controller module 110 includes at least one IMU so that the controller module 110 can provide IMU data to the central module to update / adjust the estimated pose of the controller module 110. The controller module 110 can provide IMU data at a frequency faster than the frequency at which the central module captures frames of the user and the controller module 110. For example, the central module can capture a second frame 130 of the user holding the controller module 110 and identify features 122, 124, 126, or any other potential features that can be used to estimate the pose of the user's hands from the second frame 130. Before the central module estimates an updated pose of the user's hands based on the identified features in the second frame 130, the central module can use the received IMU data of the controller module 110 to adjust the estimated pose of the controller module 110 that was estimated based on the hand grip estimated from the first frame 120. In certain embodiments, the central module may provide / update the user's hand pose at a frequency of 30 Hz (e.g., based on captured frames) to estimate the pose of the controller module 110, and the controller module 110 may provide IMU data at a frequency of 500 Hz to the central module to update the estimated pose of the controller module 110 so that the pose of the controller module 110 can be tracked / adjusted at a faster frequency based on the IMU data of the controller module 110 to maintain accuracy and efficiency in tracking the controller module 110. In certain embodiments, the central module may output the controller pose based on either tracking result (e.g., feature tracking or IMU tracking) as needed.

[0036] In certain embodiments, the captured frames may be visible light images identified as comprising at least one feature that can be used to estimate a user's hand pose. The visible light images may be RGB images, CMYK images, grayscale images, or any suitable image for estimating a user's hand pose. In certain embodiments, the identified features 122, 124, 126 from the captured frames 120, 130 are configured to be precisely tracked by a camera of a central module to determine the movement, orientation, and / or spatial position of the controller module 110 (e.g., corresponding data of the controller module 110) for playback in a virtual / augmented environment. In certain embodiments, the estimated pose of the controller module 110 may be adjusted by spatial translation (XYZ positioning translation) determined based on the identified features 122, 124, 126 between frames (e.g., the first frame 120 and the second frame 130). For example, the central module may determine an updated spatial position of the user's hand in frame k+1, e.g., a frame captured during movement, and compare it to the previous spatial position of the user's hand in frame k, e.g., a frame previously captured or stored in storage, to readjust the pose of the user's hand. Detailed operations and actions performed by the central module to track the controller module may be further described in Figures 2-5.

[0037] 2 illustrates an exemplary tracking system 200 including a central module and a controller module, according to a specific embodiment. The tracking system 200 includes a central module implemented in a headset worn by a user and a controller module implemented in a controller held by the user. In a specific embodiment, a user may have two controllers paired with a headset, one for each hand. The headset includes at least one camera, at least one IMU, and at least one processor configured to process instructions for tracking the controller. Furthermore, the controller includes at least one IMU configured to provide IMU data of the controller to the headset central module and at least one processor configured to process instructions / calibrations sent from the headset.

[0038] The headset camera captures one or more images of the user and controller 202 in the environment and identifies one or more features of the user's hands from the images 202 for hand tracking 204 via machine learning or deep learning. Based on the identified features that can be used to estimate / determine the pose of the user's hands, the headset processor may estimate the user's hand pose and / or the user's hand position based on the identified features. In certain embodiments, the user's hand pose may be estimated based on recurring features identified across a series of images. In certain embodiments, the headset processor may estimate the user's hand pose relative to the environment 206 based on the results of the hand tracking 204.

[0039] In certain embodiments, the headset 208 IMU may also provide headset IMU data to the headset processor, which may estimate the headset pose relative to the environment 212 via inside-out tracking 210 based on the headset IMU data. In certain embodiments, the headset processor may estimate the headset pose relative to the environment 212 via inside-out tracking 210 based on the headset IMU data and the camera image 202. For example, the headset IMU data may provide angular velocity, acceleration, and movement information of the headset to calculate the pose of the headset within the environment. Furthermore, the headset processor may utilize the headset pose relative to the environment 212 to facilitate hand tracking 204. For example, the headset pose relative to the environment 212 may be provided to facilitate hand tracking 204 by comparing the headset pose / position relative to the environment 212 with images of the user and controller 202 within the environment to adjust / estimate the pose of the user's hands.

[0040] The headset processor may then estimate the grip of the user's hand 214 based on the estimated pose of the user's hand 206, and estimate the pose of the controller relative to the environment 216 based on the estimated grip of the user's hand 214. For example, the headset processor may use the user's hand pose (including identified features from the user's hand) to estimate the user's hand expressing a gesture holding the controller, such that the headset processor may generate a pose of the controller based on the inverse of the user's hand gesture / pose.

[0041] Additionally, the controller IMU provides the controller 220 IMU data to the headset for data fusion 218 to adjust the estimated controller pose based on the user's hand grip. The data fusion unit 218 may utilize the IMU data to calculate an IMU predicted pose of the controller unit 222. The controller unit 222 IMU predicted pose may be utilized by the grip estimator unit 214 to adjust the controller pose relative to the environment and estimate an inverse grip of the user's hand 214, which inverse grip infers the pose of the user's hand 214 based on the pose of the adjusted pose of the controller. In certain embodiments, the final pose of the controller 224 may be provided based on the headset's operation / needs. For example, the final pose of the controller 224 may be estimated between two captured frames (e.g., before the next estimation of the grip). Alternatively, the final pose of the controller 224 may be estimated based on an IMU-adjusted grip, e.g., an estimated grip adjusted by the controller's received IMU data. The headset processor may estimate the final pose of the controller 224 at a particular frequency based on power saving needs or desires.

[0042] Additionally, based on the data provided by data fusion 218, the headset processor may provide an IMU predicted pose of the hand 226 based on the IMU predicted pose of the controller 222 and use the IMU predicted pose of the hand 226 to facilitate hand tracking 204. For example, the IMU predicted pose of the controller 222 may be provided at a faster frequency (e.g., 500 Hz to 1 kHz) to bridge the gap between two frames. By applying the inverse grip estimation to the IMU predicted pose of the controller 222, the headset can generate an IMU predicted pose of the hand 226. The IMU predicted pose of the hand 226 can be used to reduce the search range for the hand in the next frame to improve processing time for hand tracking 204.

[0043] FIG. 3 shows an exemplary diagram of a tracking system 300 including a central module 310 and a controller module 340, according to a specific embodiment. The central module 310 includes a camera 312, an IMU 314, a hand and headset tracking unit 316, and a controller tracking unit 318 for tracking and coordinating the controller module 340 within the environment. The central module 310 is paired with the controller module 340 to perform certain functions via the controller module 340. The controller module 340 includes at least one IMU 342 configured to provide the central module 310 with IMU data 344 for tracking the controller module 340. In a specific embodiment, the controller module 340 transmits the IMU data 344 to the controller tracking unit 318 to calculate predictions for corresponding modules, e.g., correspondence data for the controller module 340. In a specific embodiment, the central module 340 measures the pose of the controller module 340 at a frequency between 500 Hz and 1 kHz based on the IMU data 344 of the controller module 340.

[0044] To generate / estimate the pose of the controller module 340 during operation, the camera 312 of the central module 310 may capture an image or a series of images 320 when the controller module 340 is within the field of view (FOV) of the camera for tracking the controller module 340. In particular embodiments, the image 320 shows at least a portion of a user's hand holding the controller module 340. The camera 312 of the central module 310 transmits the image 320 to the hand / headset tracking unit 316 for estimating the pose of the user's hand based on features identified from the image 320.

[0045] The hand / headset tracking unit 316 identifies one or more features of the user's hand from the image 320 via machine learning, deep learning, or any suitable computational method. Based on the identified features that can be used to estimate / determine the pose of the user's hand 324, the hand / headset tracking unit 316 of the central module 310 estimates the pose of the user's hand 324 and / or the position of the user's hand within the environment based on the identified features of the user's hand. In particular embodiments, the pose of the user's hand 324 may be estimated based on recurring features identified across a series of images. The hand / headset tracking unit 316 of the central module 310 estimates the pose of the user's hand 324 at a frequency based on processing capabilities or requirements. In particular embodiments, the hand / headset tracking unit 316 of the central module 310 estimates the pose of the user's hand 324 at a frequency of 30 Hz.

[0046] In particular embodiments, the IMU 314 of the central module 310 also transmits IMU data 322 to the hand / headset tracking unit 316 to facilitate estimation of the headset pose. For example, the hand / headset tracking unit 316 may perform inside-out tracking to estimate the pose of the central module 310. Based on the images 320 (including the controller module 340 in the environment) and the IMU data 322 of the central module 316, the hand / headset tracking unit 316 of the central module 310 may estimate the pose of the central module 310, such that the estimated pose of the user's hands 324 (estimated based on the images 320) may be adjusted by the pose of the central module 310 (e.g., the position of the central module 310 relative to the user's hands in the environment).

[0047] The hand / headset tracking unit 316 of the central unit 310 sends the pose of the user's hands 324 to the controller tracking unit 318 for controller tracking. The controller tracking unit 318 includes a grip estimation unit 326 configured to estimate the grip of the user's hands, and a data fusion unit 328 configured to fuse / integrate data sent from the grip estimation unit 326 with data sent from the controller module 340.

[0048] The grip estimation unit 326 of the controller tracking unit 318 receives the pose of the user's hand 324 from the hand / headset tracking unit 316 and estimates the grip of the user's hand based on the pose of the user's hand 324. Further, the grip estimation unit 326 estimates the pose of the controller module 340 based on the grip of the user's hand. For example, the pose of the user's hand 324 may reveal the gesture of the user holding the controller module 340. Thus, based on the pose of the user's hand 324, the grip estimation unit 326 may estimate the grip of the user's hand and then estimate the pose of the controller module relative to the environment 330 based on the grip of the user's hand, which defines the relative pose between the user's hand and the controller module 340. Further, the grip estimation unit 326 sends the pose of the controller relative to the environment 330 to the data fusion unit 328.

[0049] The data fusion unit 328 of the controller tracking unit 318 receives the pose of the controller relative to the environment 330 from the grip estimation unit 326 of the controller tracking module 318 in the central module 310, and further receives IMU data 344 from the controller module 340. The data fusion unit 328 may integrate the pose of the controller module relative to the environment 330 with the IMU data 344 of the controller module 340 to output an adjusted / final pose of the controller module for the central module 310 to accurately execute corresponding instructions via the controller module 340. In certain embodiments, the data fusion unit 328 may output the adjusted pose of the controller module at a frequency based on the requirements or processing speed of the central module 310. In certain embodiments, the data fusion unit 328 may output the adjusted pose of the controller module at a frequency faster than the frequency for estimating the pose of the user's hands, such as 30 Hz, because the data fusion unit 328 can update the pose of the controller module 340 sent from the grip estimation unit 326 when it receives IMU data 344 from the controller module 330.

[0050] Additionally, the data fusion unit 328 may also provide an IMU predicted pose of the controller unit 332 based on the IMU data 344 of the controller module 340 to the grip estimation unit 326, so that the grip estimation unit 326 may adjust the pose of the controller module 340 estimated based on the captured frame. The grip estimation unit 326 may provide an IMU predicted pose of the user's hand 346 based on the IMU data 344 of the controller module 340 to the hand tracking unit 316 to facilitate the process of hand tracking. The IMU predicted pose of the user's hand 346 allows the hand tracking unit 316 to identify features of the user's hand within a predicted range in the next captured frame, so that the hand tracking unit 316 may complete hand tracking with less processing time.

[0051] Additionally, the central module 310 may also utilize these captured images 320 containing identified features to perform a wide range of services and functions, such as generating states for the user / controller modules 340, positioning the user / controller modules 340 locally or globally, and / or rendering virtual tags / objects in the environment via the controller modules 340. In particular embodiments, the central module 310 may also use IMU data 322 to assist in generating the user's state. In particular embodiments, the central module 310 may use the user's state information for the controller modules 340 in the environment based on the captured images 320 to project virtual objects in the environment or set virtual tags in a map via the controller modules 340.

[0052] In certain embodiments, tracking system 300 may be implemented in any suitable computing device, such as, for example, a personal computer, a laptop computer, a cellular phone, a smartphone, a tablet computer, an augmented / virtual reality device, a head-mounted device, a portable smart device, a wearable smart device, or any suitable device compatible with tracking system 300. For purposes of this disclosure, a user being tracked and located by a tracking device may refer to a device mounted on a movable object, such as a vehicle, or a device attached to a person. For purposes of this disclosure, a user may be an individual (a human user), an entity (e.g., a business, company, or third-party application), or a group (e.g., of individuals or entities) that interacts or communicates with tracking system 300. In certain embodiments, central module 310 may be implemented in a head-mounted device, and controller module 340 may be implemented in a remote controller separate from the head-mounted device. The head-mounted device comprises one or more processors configured to implement the camera 312, IMU 314, hand / headset tracking unit 316, and controller unit 318 of central module 310. In one embodiment, each processor is configured to separately implement the camera 312, the IMU 314, the hand / headset tracking unit 316, and the controller unit 318. The remote controller includes one or more processors configured to implement the IMU 342 of the controller module 340. In one embodiment, each processor is configured to separately implement the IMU 342.

[0053] This disclosure contemplates any suitable network for connecting elements in tracking system 300 or for connecting tracking system 300 with other systems. By way of example and not limitation, one or more portions of a network may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more of these. A network may include one or more networks.

[0054] 4 shows an exemplary diagram of a tracking system 400 with a mapping service, according to a specific embodiment. The tracking system 400 includes a controller module 410, a central module 420, and a cloud 430. The controller module 410 includes at least one IMU 412 and a processor 414. The controller module 410 receives one or more instructions 442 from the central module 420 to perform specific functions. The controller module 410 is configured to send the IMU data 440 to the central module 420 for pose estimation during operation, so that the central module 420 can accurately execute the instructions 442 within a map or environment via the controller module 410.

[0055] The central module 420 includes a camera 422, at least one IMU 424, a hand tracking unit 426, and a controller tracking unit 428. The central module 420 is configured to track the controller module 410 based on various methods, such as those disclosed in FIGS. 1A-3. The camera 422 of the central module 420 may capture one or more frames of the controller module 410 being held by a user, and the IMU 424 of the central module 420 may provide the IMU data of the central module 420 to the hand tracking unit 426. The hand tracking unit 426 may identify features from the captured frames via machine learning to estimate the pose of the user's hands and adjust the pose of the user's hands based on the IMU data of the central module 420. Furthermore, the hand tracking unit 426 transmits the pose of the user's hands to the controller tracking unit 428 to estimate the pose of the controller module 410. The controller tracking unit 428 receives the user's hand pose and the controller module 410 IMU data 440 and estimates the pose of the controller module 410 by fusing the received data.

[0056] In particular embodiments, the controller tracking unit 428 may determine correspondence data based on features identified in different frames. The correspondence data may include observations and measurements of features, such as the location of features of the controller module 410 in the environment. Furthermore, the controller tracking unit 428 may also perform stereo calculations collected near certain features to provide additional information for the central module 420 to track the controller module 410. In addition, the controller tracking unit 428 of the central module 420 may request a live map from the cloud 430 corresponding to the correspondence data. In particular embodiments, the live map may include map data 444. The controller tracking unit 428 of the central module 420 may also request a remote relocation service 444 so that the controller module 410 can be located locally or globally within the live map. In particular embodiments, the pose of the controller module 410 relative to the environment may be constructed based on frames captured by the camera 422, e.g., a locally constructed map. In certain embodiments, the controller tracking unit 428 of the central module 420 may also transmit corresponding data of the controller module 410 to the cloud 430 for updating maps stored in the cloud 430 (e.g., in a locally built environment).

[0057] FIG. 5 illustrates an exemplary method 500 for tracking a controller, according to a specific embodiment. The controller module of the tracking system may be implemented in a portable device (e.g., a remote controller with input buttons, a smart pack with a touchpad, etc.). The central module of the tracking system may be implemented in a wearable device (e.g., a head-mounted device, etc.), or may be provided or displayed on any computing system (e.g., an end-user device such as a smartphone, a virtual reality system, a gaming system, etc.) and paired with the controller module. Method 500 may begin in step 510 by using a camera to capture a first image depicting at least a portion of a user's hand holding a controller in an environment. In certain embodiments, the camera may be one or more cameras implemented in a wearable device worn by the user. In certain embodiments, the wearable device may be a controller. In certain embodiments, the wearable device may be equipped with one or more IMUs.

[0058] In step 520, method 500 may identify one or more features from the first image to estimate a pose of the user's hand. In particular embodiments, method 500 may further receive IMU data of the wearable device to estimate a pose of the wearable device and update the pose of the user's hand based on the pose of the wearable device. Further, the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image of the user.

[0059] In step 530, the method 500 may estimate a first pose of the controller based on the pose of the user's hands and the estimated grip that defines a relative pose between the user's hands and the controller.

[0060] At step 540, method 500 may receive controller IMU data. In certain embodiments, the controller IMU data may be received at a frequency faster than the frequency at which the first image was captured. For example, the first image may be captured at a first frequency and the controller IMU data may be captured at a second frequency. The second frequency (e.g., 500 Hz) may be higher than the first frequency (e.g., 30 Hz).

[0061] In step 550, method 500 may estimate a second pose of the controller by updating the first pose of the controller using the controller IMU data. In particular embodiments, method 500 may estimate an IMU-predicted pose of the hand based on the updated first pose of the controller and the controller IMU data, and estimate a second pose of the hand based on the IMU-predicted pose of the hand. In particular embodiments, method 500 may estimate the second pose of the controller by estimating the pose of the controller relative to the environment based on the estimated grip, adjusting the pose of the controller relative to the environment based on the controller IMU data, estimating the pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the controller IMU, and estimating the second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.

[0062] In particular embodiments, method 500 may further include using a camera to capture a second image of the user depicting at least a portion of the user's hand holding the controller in the environment, identifying one or more features from the second image of the user, and estimating a third pose of the hand based on the one or more features identified from the second image of the user. Further, the frequency for estimating the second pose of the hand (e.g., 500 Hz) is higher than the frequency for estimating the third pose of the hand (e.g., 30 Hz).

[0063] In particular embodiments, the wearable device may include a camera configured to capture an image of a user, a hand tracking unit configured to estimate a pose of the user's hands, and a controller tracking unit configured to estimate a second pose of the controller.

[0064] Particular embodiments may repeat one or more steps of the method of Figure 5 as needed. Although this disclosure describes and illustrates certain steps of the method of Figure 5 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of Figure 5 occurring in any suitable order. Further, although this disclosure describes and illustrates an exemplary method for local location determination that includes certain steps of the method of Figure 5, this disclosure contemplates any suitable method for local location determination that includes any suitable steps, which may include all, some, or none of the steps of the method of Figure 5, as needed. Further, although this disclosure describes and illustrates certain components, devices, or systems that perform certain steps of the method of Figure 5, this disclosure contemplates any suitable combination of any suitable components, devices, or systems that perform any suitable steps of the method of Figure 5.

[0065] 6 illustrates an exemplary computer system 600. In particular embodiments, one or more computer systems 600 perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems 600 provide functionality described or illustrated herein. In particular embodiments, software executing on one or more computer systems 600 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 600. As used herein, references to a computer system may, where appropriate, encompass computing devices, and vice versa. Furthermore, references to a computer system may, where appropriate, encompass one or more computer systems.

[0066] The present disclosure contemplates any suitable number of computer systems 600. The present disclosure contemplates computer system 600 taking any suitable physical form. By way of example and not limitation, computer system 600 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these. Where appropriate, computer system 600 may include one or more computer systems 600, whether singular or distributed, spanning multiple locations, spanning multiple machines, spanning multiple data centers, or residing in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 600 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 600 may perform one or more steps of one or more methods described or illustrated herein in real time or batch mode. One or more computer systems 600 may perform one or more steps of one or more methods described or illustrated herein at different times or in different locations, as desired.

[0067] In a particular embodiment, computer system 600 includes a processor 602, memory 604, storage 606, an input / output (I / O) interface 608, a communication interface 610, and a bus 612. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular configuration, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable configuration.

[0068] In particular embodiments, processor 602 includes hardware for executing instructions, such as those comprising a computer program. By way of example and not limitation, to execute instructions, processor 602 may retrieve (or fetch) instructions from an internal register, an internal cache, memory 604, or storage 606, decode and execute them, and then write one or more results to an internal register, an internal cache, memory 604, or storage 606. In particular embodiments, processor 602 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor 602 including any suitable number of any suitable internal caches, as appropriate. By way of example and not limitation, processor 602 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in an instruction cache may be copies of instructions in memory 604 or storage 606, and the instruction cache may speed up retrieval of those instructions by processor 602. Data in the data cache may be a copy of data in memory 604 or storage 606, or other suitable data, for instructions executing on processor 602 to access by subsequent instructions executing on processor 602, or to operate on the results of previous instructions executed by processor 602, for writing to memory 604 or storage 606. The data cache may speed up read or write operations by processor 602. The TLB may speed up virtual address translation for processor 602. In particular embodiments, processor 602 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor 602 including any suitable number of any suitable internal registers, as appropriate. Where appropriate, processor 602 may include one or more arithmetic logic units (ALUs), may be a multi-core processor, or may include one or more processors 602. While this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0069] In particular embodiments, memory 604 includes main memory for storing instructions for processor 602 to execute or data for processor 602 to operate on. By way of example and not limitation, computer system 600 may load instructions into memory 604 from storage 606 or another source (e.g., another computer system 600). Processor 602 may then load the instructions from memory 604 into an internal register or internal cache. To execute the instructions, processor 602 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor 602 may write one or more results (which may be intermediate or final results) to an internal register or internal cache. Processor 602 may then write one or more of those results to memory 604. In particular embodiments, processor 602 executes instructions only from one or more internal registers or caches or memory 604 (as opposed to storage 606 or elsewhere) and operates only on data in one or more internal registers or caches or memory 604 (as opposed to storage 606 or elsewhere). One or more memory buses (each of which may include an address bus and a data bus) may couple processor 602 to memory 604. Bus 612, as described below, may include one or more memory buses. In particular embodiments, one or more memory management units (MMUs) reside between processor 602 and memory 604 to facilitate accesses to memory 604 requested by processor 602. In particular embodiments, memory 604 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. This RAM may be dynamic RAM (DRAM) or static RAM (SRAM), where appropriate. Furthermore, this RAM may be single-ported or multi-ported RAM, where appropriate. This disclosure contemplates any suitable RAM. Memory 604 may, where appropriate, include one or more memories 604. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

[0070] In particular embodiments, storage 606 includes mass storage for data or instructions. By way of example and not limitation, storage 606 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more thereof. Storage 606 may include removable or non-removable (i.e., fixed) media, as appropriate. Storage 606 may be internal or external to computer system 600, as appropriate. In particular embodiments, storage 606 is non-volatile solid-state memory. In particular embodiments, storage 606 includes read-only memory (ROM). As appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more thereof. The present disclosure contemplates mass storage 606 taking any suitable physical form. Storage 606 may, where appropriate, include one or more storage control units that facilitate communication between processor 602 and storage 606. Where appropriate, storage 606 may include one or more storages 606. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0071] In particular embodiments, I / O interface 608 includes hardware, software, or both that provide one or more interfaces for communication between computer system 600 and one or more I / O devices. Computer system 600 may include one or more of these I / O devices, as appropriate. One or more of these I / O devices may enable communication between a person and computer system 600. By way of example and not limitation, an I / O device may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I / O device, or a combination of two or more of these. An I / O device may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interface 608 therefor. As appropriate, I / O interface 608 may include one or more device or software drivers that enable processor 602 to drive one or more of these I / O devices. I / O interface 608 may, where appropriate, include one or more I / O interfaces 608. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0072] In particular embodiments, communication interface 610 includes hardware, software, or both that provide one or more interfaces for communications (e.g., packet-based communications) between computer system 600 and one or more other computer systems 600 or one or more networks. By way of example and not limitation, communication interface 610 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a Wi-Fi network. This disclosure contemplates any suitable network and any suitable communication interface 610 therefor. By way of example and not limitation, computer system 600 may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet, or a combination of two or more of these. One or more of these networks, one or more portions of which may be wired or wireless. As an example, computer system 600 may communicate with a wireless PAN (WPAN) (e.g., a BLUETOOTH WPAN, etc.), a Wi-Fi network, a Wi-MAX network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these. Computer system 600 may include any suitable communication interface 610 for any of these networks, as desired. Communication interface 610 may include one or more communication interfaces 610, as desired. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

[0073] In particular embodiments, bus 612 includes hardware, software, or both that couple together components of computer system 600. By way of example and not limitation, bus 612 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) Interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus, or a combination of two or more of these. Bus 612 may include one or more buses 612, as appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0074] As used herein, one or more computer-readable non-transitory storage media may, where appropriate, comprise one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. A computer-readable non-transitory storage medium may, where appropriate, be volatile, non-volatile, or a combination of volatile and non-volatile.

[0075] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated otherwise by context. Furthermore, "and" is both joint and several, unless expressly indicated otherwise or indicated otherwise by context. Thus, as used herein, "A and B" means "A and B, together or separately," unless expressly indicated otherwise or indicated otherwise by context.

[0076] The scope of the present disclosure encompasses all changes, substitutions, variations, changes, and modifications to the exemplary embodiments described or illustrated herein that would be understood by a person skilled in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or illustrated herein. Furthermore, although the present disclosure describes and illustrates each embodiment herein as including particular components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that would be understood by a person skilled in the art. Furthermore, references in the appended claims to a device or system or a component of a device or system that is adapted, arranged, capable, configured, enabled, operative, or functional to perform a particular function encompass that device, system, or component, so long as the device, system, or component is so adapted, arranged, capable, configured, enabled, operative, or functional, regardless of whether it or that particular function is activated, turned on, or unlocked. Furthermore, although this disclosure describes or illustrates particular embodiments as providing certain advantages, the particular embodiment may provide none, some, or all of these advantages.

[0077] According to various embodiments, an advantage of the features described herein is that the present application can provide a tracking method that remains accurate and cost-effective without requiring the paired controller to be equipped with an LED. The tracking method estimates the pose of the user's hand based on features identified from the captured image, and then estimates the grip of the user's hand based on the pose of the user's hand, so that the tracking method can estimate the pose of the controller based on the grip. Furthermore, the tracking method can adjust / calibrate the pose of the controller based on the IMU data of the controller. In addition, the processing time of the tracking method can also be improved by predictions provided by the IMU data. Certain embodiments of the present disclosure also enable tracking of a controller without an LED or when the LED disposed on the controller fails. Therefore, certain embodiments disclosed in the present disclosure can provide an improved, cost-effective tracking method for a controller.

[0078] Although the processes in the figures may indicate a particular order of operations performed by certain embodiments of the present invention, it should be understood that such order is exemplary (e.g., alternative embodiments may perform operations in a different order, combine certain operations, overlap certain operations, etc.).

[0079] While the present invention has been described in terms of several embodiments, those skilled in the art will recognize that the invention is not limited to the described embodiments, but can be practiced with modification and alteration within the scope of the appended claims. Accordingly, the description is to be regarded as illustrative rather than limiting.

Claims

1. By the computing system, capturing, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a portion of the user's hand holding a controller within an environment; identifying one or more features from the first image to estimate a pose of the hand of the user; estimating a first pose of the controller based on the pose of the hand of the user and an estimated grip defining a relative pose between the hand of the user and the controller; receiving first inertial measurement unit (IMU) data of the controller; estimating a second pose of the controller by updating the first pose of the controller using the first IMU data of the controller; receiving second IMU data from the controller; calculating an IMU predicted pose of the controller based on the second IMU data; estimating an IMU predicted pose of the hand by applying an inverse of the estimated grip to the IMU predicted pose of the controller; determining a search range for the hand based on the estimated IMU predicted pose of the hand when a second image depicting at least a portion of the hand of the user is captured; and A method comprising:

2. 2. The method of claim 1, wherein the first image is captured at a first frequency and the IMU data of the controller is received at a second frequency, the second frequency being higher than the first frequency.

3. receiving IMU data of the wearable device to estimate a pose of the wearable device; The method of claim 1 or 2, further comprising: updating the pose of the hand of the user based on the pose of the wearable device.

4. The method described in claim 3, wherein the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image of the user.

5. A method described in any one of claims 1 to 4, further comprising estimating a second pose of the hand based on the IMU predicted pose of the hand.

6. Capturing the second image of the user using the one or more cameras, wherein the second image of the user depicts at least a portion of the user's hand holding the controller within the environment; and identifying the one or more features from the second image of the user; and The method of claim 5 , further comprising: estimating a third pose of the hand based on the one or more features identified from the second image of the user.

7. The method described in claim 6, wherein the frequency at which the second pose of the hand is estimated is higher than the frequency at which the third pose of the hand is estimated.

8. The wearable device is the one or more cameras configured to capture images of the user; a hand tracking unit configured to estimate the pose of the hand of the user; A controller tracking unit configured to estimate the second pose of the controller.

9. estimating the second pose of the controller estimating a pose of the controller relative to the environment based on the estimated grip and the estimated pose of the user's hands relative to the environment; adjusting the pose of the controller relative to the environment based on the first IMU data of the controller; estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the first IMU data of the controller; and estimating the second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.

10. One or more computer-readable non-transitory storage media storing instructions that, when executed by a computing system, cause the computing system to: capturing, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a portion of the user's hand holding a controller within an environment; identifying one or more features from the first image to estimate a pose of the hand of the user; estimating a first pose of the controller based on the pose of the hand of the user and an estimated grip defining a relative pose between the hand of the user and the controller; receiving first inertial measurement unit (IMU) data of the controller; estimating a second pose of the controller by updating the first pose of the controller using the first IMU data of the controller; receiving second IMU data from the controller; calculating an IMU predicted pose of the controller based on the second IMU data; estimating an IMU predicted pose of the hand by applying an inverse of the estimated grip to the IMU predicted pose of the controller; determining a search range for the hand based on the estimated IMU predicted pose of the hand when a second image depicting at least a portion of the hand of the user is captured; and One or more computer-readable non-transitory storage media that cause

11. 11. The medium of claim 10, wherein the first image is captured at a first frequency and the IMU data of the controller is received at a second frequency, the second frequency being higher than the first frequency.

12. The software, when executed by a computing system, causes the computing system to: receiving IMU data of the wearable device to estimate a pose of the wearable device; and updating the pose of the hand of the user based on the pose of the wearable device.

13. The medium described in claim 12, wherein the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image of the user.

14. The software, when executed by the computing system, causes the computing system to: estimating a second pose of the hand based on the IMU-predicted pose of the hand; and capturing the second image of the user using the one or more cameras, the second image of the user depicting at least a portion of the user's hand holding the controller in the environment; and identifying the one or more features from the second image of the user.

15. The medium described in claim 14, wherein the software, when executed by the computing system, is further operable to cause the computing system to estimate a third pose of the hand based on the one or more features identified from the second image of the user.

16. 16. The medium of claim 15, wherein the frequency at which the second pose of the hand is estimated is higher than the frequency at which the third pose of the hand is estimated.

17. The wearable device is the one or more cameras configured to capture images of the user; a hand tracking unit configured to estimate the pose of the hand of the user; A controller tracking unit configured to estimate the second pose of the controller.

18. estimating the second pose of the controller estimating a pose of the controller relative to the environment based on the estimated grip and the estimated pose of the user's hands relative to the environment; adjusting the pose of the controller relative to the environment based on the first IMU data of the controller; estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the first IMU data of the controller; estimating the second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hand; 17. The medium of any one of claims 10 to 16, comprising:

19. 1. A system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions that, when executed by the one or more of the processors, cause the system to: capturing, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a portion of the user's hand holding a controller within an environment; identifying one or more features from the first image to estimate a pose of the hand of the user; estimating a first pose of the controller based on the pose of the hand of the user and an estimated grip defining a relative pose between the hand of the user and the controller; receiving first inertial measurement unit (IMU) data of the controller; estimating a second pose of the controller by updating the first pose of the controller using the first IMU data of the controller; receiving second IMU data from the controller; calculating an IMU predicted pose of the controller based on the second IMU data; estimating an IMU predicted pose of the hand by applying an inverse of the estimated grip to the IMU predicted pose of the controller; determining a search range for the hand based on the estimated IMU predicted pose of the hand when a second image depicting at least a portion of the hand of the user is captured; and The system is operable to:

20. The instructions, when executed, receiving IMU data of the wearable device to estimate a pose of the wearable device; updating the pose of the user's hand based on the pose of the wearable device; 20. The system of claim 19, further operable to:

Images