Hand tracking system using smartwatch and adaptive low frame rate camera

Abstract

Hand tracking is performed using a head-mounted device and a wearable accessory device. The head-mounted device is equipped with an outward facing camera and captures an image of the environment. When the wearable accessory device is in the field of view of the camera, hand tracking may be initialized by establishing a common reference between the head-mounted device and the wearable accessory device. The hand is then tracked using non-image data captured by the wearable accessory device. The neural gesture network processes the non-image data to predict hand gesture information. Occasionally, the camera of the head-mounted device is energized, and additional image data is captured and used to perform drift correction.

Description

Background Technology

[0001] Modern electronic devices offer users new ways to interact with the world around them. For example, devices can be equipped with sensors that can track a user's hand movements. Users can use gestures to select content or initiate activities. Typically, image data is used to perform hand tracking. A camera captures image data of the user's hand and determines its position and location. Therefore, the image data can be analyzed to detect user input actions.

[0002] One drawback of image-based hand tracking methods is that the image data they rely on is provided by a camera, which can consume significant power during both image capture and processing, potentially impacting other system processes. Furthermore, image-based hand tracking techniques typically rely on two or more cameras to determine the hand's depth information. Therefore, the power requirements of image-based hand tracking can be quite substantial.

[0003] An improved technology is needed to provide a low-power solution for hand tracking. Attached Figure Description

[0004] Figures 1A to 1B An example diagram is shown illustrating a user performing an input action with their hands according to one or more implementation schemes.

[0005] Figure 2 A flowchart is shown for a technique using a combination of an accessory device and a camera for hand tracking, according to some implementation schemes.

[0006] Figure 3 A flowchart is shown of a technique for activating an accessory device as a controller according to some implementation schemes.

[0007] Figure 4 A flowchart of a technique for monitoring drift according to some implementation schemes is shown.

[0008] Figure 5 A flowchart of a technique for performing drift correction according to one or more embodiments is shown.

[0009] Figure 6 A system diagram of an electronic device and a wearable accessory device for hand tracking according to one or more embodiments is shown.

[0010] Figure 7 An exemplary system for various hand tracking technologies is shown. Detailed Implementation

[0011] This disclosure relates to systems, methods, and computer-readable media for performing hand tracking using low-power techniques. Specifically, this disclosure relates to techniques for performing hand tracking in a low-power mode using a head-worn device that selectively incorporates a wearable accessory device. The techniques include synchronizing the wearable accessory device with a camera-enabled head-worn device to determine initial pose information. Thus, the wearable accessory device can be used to track the hand even without image data from the head-worn device until the head-worn device re-energizes the camera for drift correction.

[0012] Hand tracking technology comprises three phases. In the first phase, camera-based initialization is performed. For example, a head-mounted device or other camera system can capture images of the environment in front of the user. The positioning of a wearable accessory device (separate from the head-mounted device), such as one worn on the user's arm or hand, can be determined based on the image data, and the image data can be combined with sensor data from the wearable accessory device to establish a common reference system between the head-mounted device and the wearable accessory device. Although the accessory device is referred to as a "wearable accessory device" in the following description, it should be understood that in some embodiments, the wearable accessory device can be a handheld device not worn by the user, such as a controller. In some embodiments, a deep learning-based network is used to infer the fundamental transformation, thereby aligning the coordinates of the wearable accessory device with those of the head-mounted device.

[0013] In the second phase, hand tracking is performed using non-camera sensor data from the wearable accessory device. By relying on non-camera data, a camera from the head-worn device is no longer needed for hand tracking during this phase. The tracking phase may include using sensors on the device to track the position and orientation of the hand. For example, the wearable accessory device may be equipped with an inertial motion unit (IMU), accelerometer, or gyroscope. In some embodiments, the wearable accessory device uses a neural odometry system to apply sensor data to a neural network to predict pose information and reduce drift. In one or more embodiments, the position and / or orientation of the hand determined by the tracking phase can be used to perform user input actions. In some embodiments, the wearable accessory device may include additional sensors for collecting data that can be used for tracking. For example, the wearable accessory device may include an image sensor, enabling VIO / SLAM to be performed for more accurate 3D tracking.

[0014] In the third phase, drift correction is performed. According to one or more embodiments, wearable accessory device tracking and head-worn device tracking can be resynchronized. This can occur, for example, periodically or under triggered conditions. In some embodiments, a neural odometry network can generate a prediction confidence value for the pose. If the confidence value drops below a threshold, drift correction can be performed. Drift correction may involve employing an adaptive fusion algorithm that combines data from sensors on the wearable accessory device with camera data, for example, from a head-worn device or other systems. Thus, during drift correction, the camera can be powered on to capture more image data or otherwise used to reinitialize the tracking data.

[0015] The embodiments described herein provide an efficient method for performing hand tracking with limited use of image data, thereby offering a less resource-intensive technique for determining the position and / or orientation of the hand. Furthermore, the embodiments described herein provide a technological improvement over non-image-based hand tracking by selectively reinitializing the tracking process using camera data.

[0016] In the following disclosure, a physical environment refers to the physical world that people can perceive and / or interact with without the assistance of electronic devices. A physical environment may include physical features, such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with a physical environment through senses such as sight, touch, hearing, taste, and smell. In contrast, an XR environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, and / or virtual reality (VR) content, etc. In the case of an XR system, a subset or representation of a person's physical motion is tracked, and in response, one or more properties of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with at least one physical law. As an example, an XR system may detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds would change in a physical environment. For example, an XR system can detect movement of electronic devices (e.g., mobile phones, tablets, or laptops) that present the XR environment and adjust the graphical content and sound field presented to the user in a manner similar to how such views and sounds would change in the physical environment. In some cases (e.g., for accessibility reasons), an XR system can adjust the characteristics of the graphical content in the XR environment in response to representations of physical movement (e.g., voice commands).

[0017] Many different types of electronic systems enable people to sense and / or interact with various XR environments. Examples include: head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have an integrated opaque display and one or more speakers. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may combine one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some implementations, transparent or translucent displays can be configured to selectively become opaque. Projection-based systems can employ retinal projection techniques that project graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0018] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the accompanying drawings of this disclosure are block diagrams representing structures and devices to avoid obscuring the novel aspects of the disclosed concepts. For clarity, not all features of actual specific embodiments may be described. Additionally, some of the drawings of this disclosure are provided in the form of flowcharts as part of this specification. The blocks in any particular flowchart may be presented in a specific order. However, it should be understood that the specific order of any given flowchart is only for illustrative purposes of one embodiment. In other embodiments, any of the various elements depicted in the flowcharts may be omitted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. Furthermore, other embodiments may include additional steps not depicted as part of the flowcharts. Moreover, the language used in this disclosure has been primarily chosen for readability and instructional purposes and may not have been chosen to define or limit the subject matter of the invention, or to invoke the necessary claims to determine such inventive subject matter. In this disclosure, reference to “an implementation” or “implementation” means that a particular feature, structure or characteristic described in connection with the implementation is included in at least one implementation of the disclosed subject matter, and the repeated references to “an implementation” or “implementation” should not be construed as necessarily involving all of the same implementation.

[0019] It should be understood that in any actual implementation of development (as in any software and / or hardware development project), numerous decisions must be made to achieve the developer's specific goals (e.g., compliance with system and business-related constraints), and these goals may differ between different implementations. It should also be understood that such development work can be complex and time-consuming, but nevertheless, it remains routine work for those of ordinary skill in the art who design and implement graphical modeling systems in benefit from this disclosure.

[0020] For the purposes of this application, the term "input gesture" refers to the hand gesture used to detect input gestures when recognized by a gesture-based input system.

[0021] For the purposes of this application, the term "input gesture" refers to an input posture or set of input postures used for user input when recognized by a gesture-based input system.

[0022] Figures 1A to 1B Example diagrams are shown illustrating users performing input actions with their hands according to one or more implementation schemes. Specifically, Figure 1AThe illustration shows user 105 pointing to image A 115 in physical environment 100A using a hand gesture. According to some embodiments, electronic device 110 may be a head-mounted device that may include one or more outward-facing cameras having a camera field of view 135. For example, electronic device 110 may include outward-facing sensors, such as cameras and depth sensors, capable of capturing one or more parts of the user, such as the hand, arm, and shoulder. Furthermore, in some embodiments, electronic device 110 may include inward-facing sensors (such as eye-tracking cameras) that can be used in conjunction with outward-facing sensors to determine whether a user input gesture has been performed. In some embodiments, electronic device 110 may include a see-through or transmissive display, making components of physical environment 100A visible. However, in some embodiments, electronic device 110 may not include a display. Electronic device 110 may also include various sensors and electronic components required for processing and communication.

[0023] The user may also be wearing a wearable accessory device 125. According to one or more embodiments, the wearable accessory device 125 may be an assistive device worn by the user and equipped with one or more sensors, based on which motion and / or positioning data can be determined. For example, Figure 1A A wearable accessory device 125, worn as a watch on a user's wrist, is shown. Other examples may include rings or bracelets. The wearable accessory device may include an IMU or other motion sensor, based on which characteristics of the user's arm can be determined. In some embodiments, the tracking process can be initialized by combining image data captured by electronic device 110 with wearable accessory device 125 to begin the tracking process. In the example figures shown, the image data captured by electronic device 110 may include a view of wearable accessory device 125. Electronic device 110 can then register the wearable accessory device in its reference frame. This can be achieved by bridging the coordinates of the electronic device with the coordinates of the wearable accessory device. The result is a device head-mounted transformation 145 that represents the relationship between the two coordinate systems. Therefore, when sensor data 130A is received by wearable accessory device 125, the sensor data can be tracked in a coordinate system shared with electronic device 110.

[0024] In some implementations, certain hand gestures or movements, or a series of gestures or movements (such as snapping fingers or double-tapping), can be used to trigger user input actions. In some implementations, the wearable accessory device 125 can be configured to detect user input movements, such as specific gestures of the hand, forearm, or wrist. Figure 1AIn the example, the user is pointing at image A115. The target of the hand gesture can be determined in several ways. In this example, the target is determined based on the hand vector 140A extending from the user's hand to image A115. Because... Figure 1A In this device, the camera of the electronic device 110 is active, so the target of the hand can be identified through image data.

[0025] Go to Figure 1B The image shows user 105 performing an input gesture. The tracking mode no longer relies on camera data captured from electronic device 110. Specifically, hand tracking is now performed using sensor data 130B from wearable accessory device 125, which does not include image data. Therefore, tracking is performed on whether the wearable accessory device is in the camera's field of view or whether the camera is actively capturing image data. In some embodiments, the sensor data may be applied to a neural pose model to determine the movement of the user's arm and the corresponding confidence level of the user's movement. The movement can then be transformed based on device head-mount transformation 145 to determine the arm's pose in a coordinate system common to electronic device 110. In some embodiments, sensor data 130B may be sent to electronic device 110, enabling electronic device 110 to perform neural tracking. Alternatively, wearable accessory device 125 may perform neural tracking and send the results to electronic device 110. Electronic device 110 may determine the corresponding hand vector 140B. Furthermore, in some implementations, if the location of image B 120 is available in a local environment map or otherwise known to electronic device 110, electronic device 110 can determine that the user is pointing towards image B 120. For example, the relationship between the posture of the wearable accessory device and the hand can be known, for example, from registration data or other user-specific data. Alternatively, the relationship between the posture of the wearable accessory device and the hand can be inferred. The vector can be determined based on the approximate direction the hand is pointing, which is based on the position of the hand and / or the wearable accessory device 125. Additionally, the action of performing a watch gesture can be a signal to the head-mounted device to power on the camera and, after the gesture, infer the depth and semantic representation of the scene via the camera to determine the intersection of the vector with objects or components of the environment, thereby determining the target of the pointing direction.

[0026] As will be described in more detail below, the techniques described herein allow for drift correction. Specifically, various parameters being tracked can be monitored to determine whether correction is needed. This can occur, for example, periodically or in response to a drop in confidence values ​​for determining positional information below a predetermined level. According to some embodiments, drift correction involves acquiring image data of the wearable accessory device when drift correction is triggered or when the wearable accessory device is within the camera's field of view. In some embodiments, the user may be prompted to place the wearable accessory device within the camera's field of view. The determined position of the wearable device based on the image data can then be used to re-anchor the wearable accessory device to a common coordinate system.

[0027] Figure 2 A flowchart illustrating a technique for using a combination of an accessory device and a camera for hand tracking, according to some embodiments, is shown. Specifically, Figure 2 This invention relates to a technique for selectively using camera data to support hand tracking using non-image sensor data. For illustrative purposes, the following steps will be described as being performed by a specific component. However, it should be understood that various actions may be performed by alternative components. Various actions may be performed in different orders. In addition, some actions may be performed simultaneously, and some actions may be unnecessary, or additional actions may be added.

[0028] Flowchart 200 begins at block 205, where the wearable accessory device is registered with a reference frame to the camera. According to one or more embodiments, the registration process includes capturing the reference frame at block 210. The reference frame may be captured by a camera on a head-worn device, such as electronic device 110. As described above, the camera may be an outward-facing camera pointing towards the environment, configured to capture scene data in which the user's hand or arm is visible, or more specifically, in which the wearable accessory device 125 is visible.

[0029] Flowchart 200 proceeds to box 215, where the transformation between the wearable accessory device and the head-worn device is determined based on a reference frame. The position of the wearable accessory device, visible in the reference frame, is compared with the wearable accessory device sensor data to determine the transformation. In some embodiments, image data may be used as input to a network trained to predict device position information based on the image data. Then, at box 220, wearable accessory device tracking is initialized. Alternatively, a rule-based deterministic process may be used to map the image data to device position information. In some embodiments, initializing wearable accessory device tracking activates the wearable accessory device as a controller. For example, the wearable accessory device may be tracked to determine position information, which can then be used to determine user input parameters, such as hand or arm pose or orientation. Optionally, as shown in decision box 225, the camera may be powered off. That is, when the wearable accessory device is activated as a controller, camera data is not used to track the wearable accessory device. However, in some embodiments, for example, if the camera is used for other functions of the head-worn device, the camera may remain powered on or in a high-power mode. However, those camera frames are not used for tracking in wearable accessory devices. The following will discuss... Figure 3 The process for registering wearable accessory devices is described in more detail.

[0030] Flowchart 200 proceeds to block 230, where wearable accessory device tracking is performed. As described above, wearable accessory device tracking may involve determining the position and / or orientation of a user's hand or arm based on sensor data captured by the wearable accessory device. At block 235, wearable accessory device sensor data is obtained. In some embodiments, the wearable accessory device sensor data may be obtained from the wearable accessory device or from an IMU or other motion sensor within the wearable accessory device. Therefore, the wearable accessory device sensor data may be non-camera sensor data.

[0031] Flowchart 200 proceeds to box 240, where the position of the wearable accessory device is determined based on sensor data and transformations determined during registration. Specifically, an offset from a previously known position, such as translation and / or rotation, is determined based on motion data from the wearable accessory device's sensor data. Furthermore, this transformation can be used to translate the wearable accessory device's position into a coordinate system shared with the head-mounted device. At box 245, the wearable accessory device's position is used to drive user input analysis. For example, the orientation or posture of a hand or arm can be used to detect where to drive user input actions. This could include, for example, interacting with virtual content, referencing physical objects to the wearable accessory device or head-mounted device, etc. The following will discuss... Figure 4 A more detailed description of the process used to perform tracking of wearable accessory devices.

[0032] At box 250, a determination is made regarding whether a calibration criterion is met. The calibration criterion may indicate that drift correction should be performed to ensure the accuracy of wearable accessory device tracking. In some embodiments, the calibration criterion may be met occasionally or periodically (e.g., after a predefined amount of time). Alternatively, the calibration criterion may be met based on a confidence value for wearable accessory device tracking. For example, a learned network or rule-based deterministic model used to determine the wearable accessory device's motion at box 240 may provide a confidence value that can be used to determine whether the calibration criterion is met. In some embodiments, the calibration criterion may include a combination of factors that may be based on a weighted combination of factors such as a specific user, device, environment, or application. Alternatively, the calibration criterion may include a determination regarding whether the wearable accessory device is visible to a camera. The head-mounted device may determine visibility based on positioning information conveyed from the wearable accessory device or based on the wearable accessory device's known positioning of the head-mounted device. If it is determined that the calibration criterion is not met, the flowchart returns to box 230, and wearable accessory device tracking continues regardless of camera data.

[0033] Returning to box 250, if the calibration criteria are determined to be met, the flowchart proceeds to box 255, and drift correction is performed. In some implementations, performing drift correction includes, at box 260, powering on the camera of the head-mounted device or other electronic equipment in the environment. At box 265, one or more image frames are captured.

[0034] Flowchart 200 proceeds to box 270, where the wearable accessory device position is optimized based on image data and wearable accessory device sensor data. For example, the actual position of the wearable accessory device can be determined from captured image frames and compared with the position calculated from wearable accessory device tracking. In doing so, the wearable accessory device can be reinitialized for wearable accessory device tracking. At optional box 275, the camera can be powered off because image data is not used for wearable accessory device tracking. The flowchart then proceeds to box 230, where wearable accessory device tracking is performed.

[0035] Figure 3 A flowchart illustrating a technique for registering wearable accessory devices for wearable accessory device tracking is shown. Specifically, Figure 3 A common coordinate system for determining the head-mounted device and wearable accessory devices is described to enable wearable accessory device tracking (as described above). Figure 2 Example techniques (described in box 205). For illustrative purposes, the following steps will be described by specific components (such as those described above regarding...). Figures 1A to 1BThe components described herein are executed. However, it should be understood that various actions can be performed by alternative components. Actions can be executed in different orders. Furthermore, some actions can be executed simultaneously, and some actions may be unnecessary, or additional actions may be added.

[0036] Flowchart 300 begins at block 305, wherein in some embodiments, image data of the environment in which the wearable accessory device is present is captured, and the imaging may be performed by an electronic device (such as by...) Figure 1A The camera of the head-worn device (as shown in electronic device 110) captures images.

[0037] Flowchart 300 proceeds to box 310, where camera coordinates are obtained. Camera coordinates indicate the position of the camera capturing the image. Additionally or alternatively, the coordinates may correspond to an electronic device, such as a head-mounted device, where the camera is housed.

[0038] At box 315, the coordinates of the wearable accessory device are determined based on the image data. The coordinates of the wearable accessory device can be determined in several ways. For example, as shown at box 320, the image data can be applied to a basis transformation network to obtain a transformation between the coordinates of the camera capturing the image and the position of the wearable accessory device. The basis transformation network can be a neural network trained to predict pose information of an object based on the image data. For example, the relative fixed point and pose of the wearable accessory device can be predicted based on the image data. In some implementations, the basis transformation network can use one or more image frames, and can use a single frame or a stereo frame. That is, depth can be predicted based on a single image frame, without relying on stereo frames, depth sensor data, or other sensor data captured by the head-mounted device. Therefore, motion data collected at the wearable accessory device can then be used to compare with the basis transformation to determine the updated position of the wearable accessory device, and thus the updated position of the hand. In other words, the basis transformation bridges the coordinates of the head-mounted device with the coordinates of the wearable accessory device to determine user input actions based on the hand. Alternatively, the coordinates of the wearable accessory device can be determined using a formulaic method based on trial and error and the characteristics of the image data.

[0039] Flowchart 300 ends at box 325, where the wearable accessory device is activated as a controller. In some embodiments, activating the wearable accessory device as a controller involves modifying the operation of the image sensor of the head-worn device. For example, the camera may be disabled. Alternatively, the camera may enter a low-power mode. Thus, motion sensor data from the wearable accessory device is fused with the camera data from the head-worn device.

[0040] Go to Figure 4 The document presents a flowchart of a technique for performing hand tracking using motion data from wearable accessory devices without using camera data. Specifically, Figure 4A method for estimating hand position based on motion data captured by wearable accessory devices (as described above) is described. Figure 2 Example techniques (described in box 230). For illustrative purposes, the following steps will be described by specific components (such as those described above regarding...). Figures 1A to 1B The components described herein are executed. However, it should be understood that various actions can be performed by alternative components. Actions can be executed in different orders. Furthermore, some actions can be executed simultaneously, and some actions may be unnecessary, or additional actions may be added.

[0041] The flowchart formula begins at box 405, where the exact location of the wearable accessory device is obtained. The current location of the wearable accessory device at 405 corresponds to when the wearable accessory device is activated as a controller (e.g., from...). Figure 3 The location of the wearable accessory device (at frame 325) is determined. This location can be considered accurate because it is determined not only using sensor data from the wearable accessory device, but also using camera data and / or other sensor data captured from another device (such as a head-mounted device) through a registration process.

[0042] Flowchart 400 proceeds to block 410, where sensor data is acquired at the wearable accessory device. As described above, the sensor data may include motion data, such as data captured from an IMU, accelerometer, gyroscope, or magnetometer. Therefore, the sensor data can indicate changes in the position and / or fixation of the wearable accessory device. For example, the sensor data may indicate translational and / or rotational characteristics of the detected motion.

[0043] At box 415, sensor data is applied to a neural pose model. According to one or more embodiments, the neural pose model can be a deep learning-based network configured to process motion data (such as IMU data) to estimate the device's pose. For example, the neural pose model can be configured to predict translation and rotation based on IMU data. Because sensor data can lose accuracy over time, the neural pose model provides more accurate motion data than relying directly on sensor data. Furthermore, the neural pose model provides confidence values ​​for the predicted motion data. Therefore, at box 420, flowchart 400 includes obtaining motion data and confidence values ​​for the wearable accessory device from the sensor data.

[0044] Flowchart 400 proceeds to box 425, where a determination is made regarding whether the confidence value meets the calibration criteria. The calibration criteria may be a confidence value indicating the accuracy of the predicted positional information. In some embodiments, this determination may also consider whether the wearable accessory device is within the field of view of the head-worn device. If the confidence value is sufficient to prevent the calibration criteria from being met, flowchart 400 proceeds to box 430. At box 430, the position of the wearable accessory device is determined. Specifically, an offset from a previously known position, such as translation and / or rotation, is determined based on motion data from sensor data of the wearable accessory device. Furthermore, this transformation may be used, for example, to translate the position of the wearable accessory device into a coordinate system common to the head-worn device by applying an offset to a previously known position, such as the position obtained at box 405.

[0045] At frame 435, hand posture is determined based on the posture of the wearable accessory device. For example, as... Figure 1B As described, a hand vector 140B can be determined based on the posture of the wearable accessory device 125, according to a predefined or inferred spatial relationship between the hand and the wearable accessory device 125. The hand vector can be user-oriented, or it can typically be determined based on the position and / or orientation of the wearable accessory device.

[0046] Flowchart 400 proceeds to block 440, using hand gestures to drive user input analysis. For example, hand gestures can be analyzed to determine if the hand is pointing at a registered object in the physical environment that the user can interact with upon selection. Therefore, in some implementations, hand gestures may be sufficient to trigger a user input action. Alternatively, hand gestures can be combined with other user input triggers (such as user input detected from voice, haptic input, visual input, or eye tracking) to determine whether a user input action should be triggered. Flowchart 400 then returns to block 410 and receives additional sensor data.

[0047] Returning to box 425, if the confidence value is determined to meet the calibration criteria, such as if the parameters of the defined criteria are met, then flowchart 400 ends at box 445, and the calibration process is initiated. Specifically, the calibration process may correspond to a drift calibration process, in which the wearable accessory device and the head-mounted device are re-anchored, as will be discussed below. Figure 5 To describe in more detail.

[0048] Figure 5 A flowchart illustrating a technique for performing drift correction according to one or more embodiments is shown. Specifically, Figure 5 An example is illustrated of a process for re-anchoring a wearable accessory to a head-mounted device to improve the accuracy of the wearable accessory's tracking, as described above. Figure 2Box 255 describes the steps described. For illustrative purposes, the following steps will be described by specific components (such as those mentioned above). Figures 1A to 1B The components described herein are executed. However, it should be understood that various actions can be performed by alternative components. Actions can be executed in different orders. Furthermore, some actions can be executed simultaneously, and some actions may be unnecessary, or additional actions may be added.

[0049] Flowchart 500 begins at box 505, where current wearable accessory device sensor data is obtained. For example, current wearable accessory device sensor data may correspond to wearable accessory device sensor data that is... Figure 4 At box 425, the confidence value meets the calibration criteria. Alternatively, the next or subsequent frame of wearable accessory device sensor data can be obtained concurrently with the additional sensor data collected at box 510.

[0050] According to one or more embodiments, additional sensor data collected at block 510 can be obtained from an additional device (such as shown in the figure). Figure 1A The head-worn device (electronic device 110) collects the data. In some embodiments, other forms of sensor data may be additionally or alternatively used to re-anchor the wearable accessory device to the head-worn device. This may include, for example, magnetometer data, ultrasonic data, or ultra-wideband signals. Obtaining additional sensor data at block 510 may include activating or powering additional sensors within the wearable accessory device, the head-worn device, or another electronic device.

[0051] In some implementations, additional sensor data may be captured by the camera of the head-worn device. Therefore, at box 515, the camera of the head-worn device is powered on. This step is optional because in some implementations, the camera may already be powered on, but the camera data may not be used for tracking by the wearable accessory device, as described above. Figure 4 As described. In some implementations, at block 515, the camera can switch from a low-power mode to a high-power mode. Then, at block 520, the camera acquires additional image data.

[0052] The flowchart proceeds to box 525, where drift correction is performed based on current sensor data and additional sensor data, such as camera data. As shown in box 530, in some embodiments, a fusion algorithm may be applied. In some embodiments, the fusion algorithm may be configured to anchor the wearable accessory device to the head-mounted device to achieve a specific balance between smoothness and accuracy. For example, a smoother algorithm may feel more natural to the user but may be less accurate when performing corrections. In contrast, a more accurate transition may feel jittery to the user but may result in more accurate tracking. In some embodiments, additional sensor data, such as the additional sensor data described above at box 510, may be used. Once drift correction is performed, the wearable accessory device is re-anchored to the head-mounted device, and wearable accessory device tracking can be performed as described above regarding... Figure 4 It continues as described. Furthermore, although not shown, the camera can subsequently be powered off, disabled, or placed in low-power mode while performing wearable accessory tracking.

[0053] refer to Figure 6 A simplified block diagram of electronic device 600 is depicted. Electronic device 600 may be part of a multifunctional device, such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable accessory device, head-mounted system, projection-based system, base station, laptop computer, desktop computer, network device, or any other electronic system such as those described herein. In some embodiments, electronic device 600 may be a head-mounted device. Electronic device 600 may include one or more additional devices, within which various functionalities may be included or distributed across the one or more additional devices, such as server devices, base stations, accessory devices, etc. Furthermore, electronic device 600 may be communicatively coupled to additional devices, such as wearable accessory device 680, across network 675. Exemplary networks include, but are not limited to, local area networks (such as Universal Serial Bus (USB) networks), organizational LANs, and wide area networks (such as the Internet). In some embodiments, various devices may connect to each other more directly (such as via Bluetooth or other short-range wireless connections).

[0054] Electronic device 600 may include one or more processors 615, such as a central processing unit (CPU) or a graphics processing unit (GPU). Electronic device 600 may also include memory 605. Memory 605 may include one or more different types of memory that can be used in conjunction with processor 615 to perform device functions. For example, memory 605 may include cache, ROM, RAM, or any kind of transient or non-transitory computer-readable storage medium capable of storing computer-readable code. Memory 605 may store various programming modules for execution by processor 615. Programming modules may include, for example, a registration module 630 configured to register wearable accessory device 680 with electronic device 600, as described above. Figure 3 As described above. According to some embodiments, the registration module 630 may be configured to bridge the coordinate system of the electronic device 600, for example, determined by one or more sensors 625, with the coordinate system of the accessory device 680. The programming module may also include a neural tracking module 635, which is configured to determine position information and / or positioning information based on sensor data received from the wearable accessory device 680, as described above. Figure 4 As described above. The neural tracking module can be configured to determine positional information without using camera data or image data. The programming module may also include a drift correction module 640, which is configured to re-anchor the wearable accessory device 680 to the electronic device 600 when the output of the neural tracking module 635 includes a confidence value that drops below a threshold, as described above. Figure 5 As described. The drift correction module 640 can use image data captured from the camera 620 to determine the position information of the wearable accessory device 680 based on the captured image data, and use the position information to correct the position information predicted by the neural tracking module.

[0055] Electronic device 600 may also include storage device 610. Storage device 610 may include one or more non-transitory computer-readable media, including, for example, magnetic disks (fixed disks, floppy disks, and removable disks) and magnetic tapes, optical media (such as CD-ROMs and digital video discs (DVDs)), and semiconductor memory devices (such as electrically programmable read-only memory (EPROMs) and electrically erasable programmable read-only memory (EEPROMs)). Storage device 610 may be used to store various data and structures that may be used to store data related to devices and / or hand tracking for user input. For example, storage device 610 may include registration data 650 that may be used to determine hand vectors, such as hand models, skeletons, or other information related to the user's hand, which may be used in conjunction with position information of wearable accessory device 680 to determine user input actions. Storage device 610 may also include a transformation storage area 655. The transformation storage area 655 can be used to store transformations applied between the wearable accessory device 680 and the electronic device 600, enabling the electronic device to determine the position and / or orientation of the wearable accessory device 680 based on sensor data collected from the accessory device. Furthermore, the storage device 610 may include a tracking model storage area 660, which may include data for performing tracking of the wearable accessory device 680 using sensor data.

[0056] Wearable accessory device 680 can be an accessory device worn by a user. Examples of wearable accessory devices include watches, rings, or bracelets equipped with computing structures. For example, a wearable accessory device may include one or more memories 690 and one or more processors 685. Memory 690 may be configured to store computing modules that can be executed by processor 685. According to one or more embodiments, electronic device 600 is equipped with one or more sensors 695 that can capture stationary and / or motion data of wearable accessory device 680. The sensor data can then be provided to electronic device 600 for detecting user input actions. In some embodiments, memory 690 and processor 685 may cause wearable accessory device 680 to perform at least some of the functionalities described regarding the computing modules of electronic device 600. For example, the wearable accessory device may perform some processing locally instead of transmitting IMU or other sensor data from sensor 695 to electronic device 600. For example, wearable accessory device 680 may run a neural velocity model and transmit velocity estimates to electronic device 600.

[0057] Although the electronic device 600 and the wearable accessory device 680 are depicted as including the numerous components described above, in one or more embodiments, the various components may be distributed differently across the device or across the accessory device. Therefore, although certain calls and transmissions are described herein with respect to the particular system depicted, in one or more embodiments, various calls and transmissions may be performed differently or may be directed differently based on the different distributions of functionality. Additionally, accessory components may be used, and certain combinations of the functionality of any components may be combined.

[0058] Now for reference Figure 7 This document illustrates a simplified functional block diagram of an exemplary multifunction electronic device 700 according to one embodiment. Each of the electronic devices may be a multifunction electronic device, or may have some or all of the components described herein. The multifunction electronic device 700 may include a processor 705, a display 710, a user interface 715, graphics hardware 720, device sensors 725 (e.g., proximity / ambient light sensors, accelerometers and / or gyroscopes), a microphone 730, an audio codec 735, a speaker 740, communication circuitry 745, digital image capture circuitry 750 (e.g., including a camera system), a video codec 755 (e.g., supporting a digital image capture unit), a memory 760, a storage device 765, and a communication bus 770. The multifunction electronic device 700 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0059] Processor 705 executes instructions necessary for the operation of many functions performed by device 700 (e.g., the generation and / or processing of images as disclosed herein). Processor 705 may, for example, drive display 710 and may receive user input from user interface 715. User interface 715 allows a user to interact with device 700. For example, user interface 715 may take various forms, such as buttons, keypad, dial pad, click wheel, keyboard, display screen, touchscreen, gaze, and / or gestures. Processor 705 may also be, for example, a system-on-a-chip, such as those present in mobile devices, and may include a dedicated GPU. Processor 705 may be based on a Reduced Instruction Set Computer (RISC) or Complex Instruction Set Computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. Graphics hardware 720 may be dedicated computing hardware for processing graphics and / or assisting processor 705 in processing graphics information. In one embodiment, graphics hardware 720 may include a programmable GPU.

[0060] Image capture circuit 750 may include two (or more) lens assemblies 780A and 780B, each lens assembly having a separate focal length. For example, lens assembly 780A may have a shorter focal length relative to the focal length of lens assembly 780B. Each lens assembly may have a separate associated sensor element 790A and sensor element 790B. Alternatively, two or more lens assemblies may share a common sensor element. Image capture circuit 750 may capture still images and / or video images. Output from image capture circuit 750 may be processed by video codec 755 and / or processor 705 and / or graphics hardware 720 and / or a dedicated image processing unit or pipeline incorporated within circuit 750. Images thus captured may be stored in memory 760 and / or storage device 765.

[0061] The sensor and camera circuitry 750 can capture still images and video images that can be processed at least in part by the following devices according to this disclosure: video codec 755 and / or processor 705 and / or graphics hardware 720, and / or a dedicated image processing unit incorporated within the circuitry 750. The captured images can be stored in memory 760 and / or storage device 765. Memory 760 may include one or more different types of media used by processor 705 and graphics hardware 720 to perform device functions. For example, memory 760 may include memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage device 765 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device configuration file information, and any other suitable data. Storage device 765 may include one or more non-transitory computer-readable storage media, including, for example, magnetic disks (fixed disks, floppy disks, and removable disks) and magnetic tapes, optical media (such as CD-ROMs and DVDs), and semiconductor memory devices (such as EPROMs and EEPROMs). Memory 760 and storage device 765 can be used to tangibly hold computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 705, such computer program code can implement one or more of the methods described herein.

[0062] The various processes defined in this document take into account options for obtaining and utilizing a user's identifying information. For example, such personal information may be used to track a user's posture and / or movement. However, with regard to the collection of such personal information, it should be obtained with the user's informed consent, and the user should be aware of and control the use of their personal information.

[0063] Personal information will be used by the appropriate parties only for lawful and reasonable purposes. Parties using such information will comply with privacy policies and practices that are at least in accordance with applicable laws and regulations. Furthermore, such policies should be comprehensive and meet or exceed government / industry standards. In addition, parties may not distribute, sell, or otherwise share such information except for any reasonable and lawful purpose.

[0064] Furthermore, the intent of this disclosure is that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by restricting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Where appropriate, deidentification can be facilitated by removing specific identifiers (e.g., date of birth), controlling the amount or characteristics of stored data (e.g., collecting location data at the city level rather than address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods.

[0065] It should be understood that the above description is intended to be illustrative and not restrictive. Material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims and to provide the material in the context of specific embodiments, variations of which will be readily apparent to those skilled in the art (e.g., some embodiments of the disclosed embodiments may be used in combination with each other). Therefore, Figures 3 to 4 The specific arrangement of the steps or actions shown in Figures 1 to 19 is as follows. Figure 2 and Figures 5 to 7 The arrangement of elements shown should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents. In the appended claims, the terms “including” and “in which” are used as common English equivalents to the corresponding terms “comprising” and “wherein”.

Claims

1. A method comprising: obtaining image data of a wearable accessory device at a head-worn device; obtaining sensor data of the wearable accessory device; determining a wearable accessory device transform based on the image data and the sensor data; identifying a position of the wearable accessory device using the wearable accessory device transform and additional sensor data from the wearable accessory device; obtaining additional image data of the wearable accessory device at the head-worn device; and adjusting a tracked position of the wearable accessory device.

2. The method of claim 1, wherein the tracked position of the wearable accessory device is adjusted in response to a confidence value for the position satisfying a correction criterion.

3. The method of claim 2, further comprising: identifying an additional position of the wearable accessory device using the additional sensor data.

4. The method of claim 3, further comprising: re-anchoring the wearable accessory device to the head-worn device to obtain updated position information, wherein tracking is resumed using the additional sensor data and the updated position information.

5. The method of any one of claims 1 to 4, further comprising: in response to determining that the wearable accessory device is within a field of view of a camera that captured the image data: powering off the camera, wherein the determining is made while the camera is operating in a powered on mode.

6. The method of any of claims 1-4, wherein tracking the wearable accessory device comprises: determining a hand position based on tracking the wearable accessory device.

7. The method of claim 6, further comprising: detecting a user input action, and determining a user input based on the hand position and the user input action.

8. A non-transitory computer-readable medium comprising computer-readable code executable by one or more processors to: obtain image data of a wearable accessory device at a head-worn device; obtain sensor data of the wearable accessory device; determine a wearable accessory device transform based on the image data and the sensor data; identify a position of the wearable accessory device using the wearable accessory device transform and additional sensor data from the wearable accessory device; obtain additional image data of the wearable accessory device at the head-worn device; and adjust a tracked position of the wearable accessory device.

9. The non-transitory computer-readable medium of claim 8, wherein the tracked position of the wearable accessory device is adjusted in response to a confidence value for the position satisfying a correction criterion.

10. The non-transitory computer-readable medium of claim 9, further comprising computer-readable code to: identify an additional position of the wearable accessory device using the additional sensor data.

11. The non-transitory computer-readable medium of claim 10, further comprising computer-readable code for: re-anchoring the wearable accessory device to the head-worn device to obtain updated position information, wherein tracking is resumed using the additional sensor data and the updated position information.

12. The non-transitory computer-readable medium of any of claims 8-11, further comprising computer-readable code for, in response to determining that the wearable accessory device is within a field of view of a camera that captured the image data: powering down the camera, wherein the determining is made while the camera is operating in a powered on mode.

13. The non-transitory computer-readable medium of any of claims 8-11, wherein the computer-readable code for tracking the wearable accessory device comprises computer-readable code for: determining a hand position based on the position of the wearable accessory device.

14. The non-transitory computer-readable medium of claim 13, further comprising computer-readable code for: detecting a user input action, and determining a user input based on the hand position and the user input action.

15. A system comprising: one or more processors; and one or more computer-readable media comprising computer-readable code executable by the one or more processors for: obtaining, at a head-worn device, image data of a wearable accessory device; obtaining sensor data of the wearable accessory device; determining a wearable accessory device transform based on the image data and the sensor data; identifying a position of the wearable accessory device using the wearable accessory device transform and additional sensor data from the wearable accessory device; obtaining, at the head-worn device, additional image data of the wearable accessory device; and adjusting a tracked position of the wearable accessory device.

16. The system of claim 15, wherein the tracked position of the wearable accessory device is adjusted in response to a confidence value for the position satisfying a correction criterion.

17. The system of claim 16, further comprising computer-readable code for: identifying an additional position of the wearable accessory device using the additional sensor data.

18. The system of claim 17, further comprising computer-readable code for: re-anchoring the wearable accessory device to the head-worn device to obtain updated position information, wherein tracking is resumed using the additional sensor data and the updated position information. ​ 19. The system of any one of claims 15-18, further comprising computer- readable code to, in response to determining that the wearable accessory device is within a field of view of a camera that captured the image data: powering off the camera, wherein the determining is made while the camera is operating in a powered on mode.

20. The system of any one of claims 15-18, wherein the computer-readable code to track the wearable accessory device comprises computer-readable code to: determine a hand position based on the position of the wearable accessory device.

Images