Hand tracking system with smartwatch and adaptive low frame rate camera

Abstract

Hand tracking is performed using a head-mounted device and a wearable accessory. The head-mounted device is equipped with an outward-facing camera and captures images of the surroundings. When the wearable accessory is within the camera's field of view, hand tracking can be initiated by establishing a common reference between the head-mounted device and the wearable accessory. The hand is then tracked using non-image data captured by the wearable accessory. A posture neural network processes this non-image data to predict hand posture information. Occasionally, the head-mounted device's camera is activated, capturing additional image data that is used for drift correction.

Description

BACKGROUND

[0001] Modern electronic devices offer users new ways to interact with the world around them. For example, devices can be equipped with sensors that track a user's hand movements. Users can then use gestures to select content, start activities, and so on. Hand tracking is typically performed using image data. A camera can capture images of a user's hand and determine its position and location. This image data can then be analyzed to recognize user input actions.

[0002] One of the disadvantages of hand tracking methods is that the image data on which they are based is provided by cameras, which can consume a lot of power during both image acquisition and processing, potentially impacting other system processes. Furthermore, image-based hand tracking techniques often rely on two or more cameras to determine depth information for the hand. Therefore, the power consumption of image-based hand tracking can be quite high.

[0003] An improved hand tracking technology is needed that offers a power-saving solution. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figures 1A-1B show exemplary representations of a user performing an input action with their hand according to one or more embodiments. Fig. Figure 2 shows a flowchart of a technique for using a combination of an accessory device and a camera for hand tracking according to some embodiments. Fig. Figure 3 shows a flowchart of a technique for activating an accessory device according to some embodiments. Fig. Figure 4 shows a flowchart of a technique for monitoring drift according to some embodiments. Fig. Figure 5 shows a flowchart of a technique for correcting drift according to one or more embodiments. Fig. Figure 6 shows a system diagram of an electronic device and a portable accessory device that can be used for hand tracking according to one or more embodiments. Fig. Figure 7 shows an exemplary system for use with various hand tracking technologies. DETAILED DESCRIPTION

[0004] This disclosure relates to systems, methods, and computer-readable media for performing hand tracking using low-power techniques. In particular, this disclosure relates to techniques for the selective use of a head-mounted device in combination with a portable accessory device to perform hand tracking in a low-power mode. The techniques include synchronizing a portable accessory device with a head-mounted device equipped with a camera to determine initial posture information. From there, the hand can be tracked using the portable accessory device without the image data from the head-mounted device until the head-mounted device reactivates the camera for drift correction.

[0005] Hand tracking techniques comprise three phases. In the first phase, camera-based initialization is performed. For example, a head-mounted device or other camera system can capture an image of the environment in front of a user. From the image data, the location of the wearable accessory (separate from the head-mounted device), worn, for example, on the user's arm or hand, can be determined. The image data can be combined with sensor data from the wearable accessory to establish a common reference system between the head-mounted device and the wearable accessory. Although the following description refers to the accessory as a "wearable accessory," it is understood that in some embodiments, the wearable accessory may be a handheld device, such as a controller, not worn by the user.In some embodiments, a deep learning-based network is used to derive the basis transformation and align the coordinates of the portable accessory device with the coordinates of the head-worn device.

[0006] In a second phase, hand tracking is performed using non-camera sensor data from the wearable accessory device. By using non-camera data, the camera of the head-worn device is no longer needed for hand tracking in this phase. The tracking phase may include the use of sensors on the device to track the hand's position and orientation. For example, the wearable accessory device may be equipped with an inertial motion unit (IMU), an accelerometer, a gyroscope, or similar sensors. In some embodiments, the wearable accessory device uses neural odometry and applies the sensor data to a neural network to predict posture information and reduce drift.The hand's position and / or orientation determined during the tracking phase can be used to execute user input actions in one or more embodiments. In some embodiments, the wearable accessory device can include additional sensors for collecting data that can be used for tracking. For example, the wearable accessory device can include image sensors, enabling VIO / SLAM to be performed for more accurate 3D positioning tracking.

[0007] In a third phase, drift correction is performed. According to one or more embodiments, the tracking of the wearable accessory device and the tracking of the head-worn device can be resynchronized. This can occur, for example, periodically or due to a trigger condition. In some embodiments, the neural odometry network can generate a predicted confidence value for a posture. If the confidence value falls below a threshold, drift correction can be performed. An adaptive fusion algorithm can be used for drift correction, combining data from the wearable accessory device's sensor with camera data, for example, from the head-worn device or another system. Therefore, the camera can be switched on during drift correction to acquire additional image data or otherwise used to reinitialize the tracking data.

[0008] The embodiments described herein offer an efficient method for performing hand tracking with limited use of image data, thus providing a less resource-intensive technique for determining the position and / or orientation of a hand. Furthermore, the embodiments described herein offer a technical improvement in image-free hand tracking by selectively reinitializing the tracking process using camera data.

[0009] In the following revelation, a physical environment refers to a physical world that humans can perceive and / or interact with without the aid of electronic devices. The physical environment may include physical features, such as a physical surface or a physical object. For example, the physical environment corresponds to a physical park, which includes physical trees, physical buildings, and physical people. Humans can directly perceive and / or interact with the physical environment, such as through sight, touch, hearing, taste, and smell. In contrast, an XR environment refers to a wholly or partially simulated environment that humans perceive and / or interact with through an electronic device. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and / or the like.An XR system tracks a subset of a person's physical movements or their representations, and in response, adjusts one or more characteristics of one or more virtual objects simulated in the XR environment in a manner consistent with at least one physical law. For example, an XR system can detect head movement and, in response, adjust the graphical content and acoustic field presented to the person in a way similar to how such views and sounds would change in a physical environment. Another example is that the XR system can detect movement of the electronic device representing the XR environment (e.g., a camera, a smartphone, etc.).a mobile phone, tablet, laptop, or similar device) and adapt the graphical content and acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), the XR system can adapt a characteristic of graphical content in the XR environment in response to representations of physical movements (e.g., voice commands).

[0010] There are many different types of electronic systems that allow a person to perceive and / or interact with various XR environments. Examples include: head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capability, windows with integrated display capability, displays designed as lenses intended to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., body-worn or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to accommodate an external opaque display (e.g., a smartphone).The head-mounted system may include one or more imaging sensors to capture images or video recordings of the physical environment and / or one or more microphones to capture audio recordings of the physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. The transparent or translucent display may have a medium through which light, representative of images, is directed toward a person's eyes. The display may utilize digital light projection, OLEDs, LEDs, uLEDs, liquid crystals on silicon, a laser scanning light source, or any combination of these technologies. The medium may be an optical fiber, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to become selectively opaque.Projection-based systems can employ retinal projection technology, which projects graphic images onto a person's retina. Projection systems can also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface.

[0011] The following description presents numerous specific details for explanatory purposes, in order to provide a thorough understanding of the disclosed concepts. As part of this description, some drawings in this disclosure depict structures and devices in block diagram form to avoid obscuring the novel aspects of the disclosed concepts. For the sake of clarity, not all features of an actual implementation may be described. Furthermore, as part of this description, some of the drawings in this disclosure may be provided in the form of flowcharts. The boxes in a particular flowchart may be presented in a specific order. However, it should be understood that the specific sequence of any given flowchart is used only to illustrate one embodiment.In other embodiments, any one of the various elements shown in the flowchart may be deleted, or the illustrated sequence of operations may be performed in a different order or even simultaneously. Furthermore, other embodiments may include additional steps not shown as part of the flowchart. Moreover, the language used in this disclosure has been chosen primarily for readability and instructional purposes, and may not have been chosen to outline or delimit the subject matter of the invention or to refer to the claims necessary to define that subject matter. A reference in this disclosure to “an embodiment” means that a particular feature, structure, or property that is included in the invention may be included in the invention.which is described in connection with the embodiment, is included in at least one embodiment of the disclosed subject matter, and multiple references to “an embodiment” should not be understood to mean that they all necessarily refer to the same embodiment.

[0012] It is understood that developing an actual implementation (as with any software or hardware development project) requires numerous decisions to be made to achieve a developer's specific goals (e.g., compliance with system and business constraints), and that these goals may vary from one implementation to another. Furthermore, it is understood that such development efforts can be complex and time-consuming, but would nevertheless constitute a routine project for the average professional in the development and implementation of graphics modeling systems who benefits from this revelation.

[0013] For the purposes of this application, the term “input posture” refers to a hand posture which, when detected by a gesture-based input system, is used to recognize input gestures.

[0014] For the purposes of this application, the term “input gesture” refers to an input posture or a series of input postures which, when recognized by a gesture-based input system, are used for user input.

[0015] Fig. Figures 1A-1B show exemplary representations of a user performing an input action with their hand according to one or more embodiments. In particular, they show Fig. 1A a user 105, who is pointing with hand gestures to image A 115 in a physical environment 100A. According to some embodiments, the electronic device 110 can be a head-worn device that may have one or more outward-facing cameras with a camera field of view 135. For example, the electronic device 110 may include outward-facing sensors such as cameras, depth sensors, and the like, which can detect one or more parts of the user, such as hands, arms, shoulders, and the like. Furthermore, in some embodiments, the electronic device 110 may include inward-facing sensors such as eye-tracking cameras, which can be used together with the outward-facing sensors to determine whether a user input gesture is being performed.In some embodiments, the electronic device may include a pass-through or see-through display, allowing components of the physical environment 100A to be visible. However, in some embodiments, the electronic device 110 may not have a display. The electronic device 110 may also include various sensors and electronic components required for processing and communication.

[0016] The user may also wear a portable accessory device 125. According to one or more embodiments, the portable accessory device 125 can be a secondary device worn by the user and equipped with one or more sensors from which movement and / or location data can be determined. For example, Fig. 1A The wearable accessory 125 is a watch that the user wears on their wrist in the form of a wristwatch. Other examples include a ring, a bracelet, or the like. The wearable accessory may include an IMU or other motion sensor, which can be used to determine characteristics of the user's arm. In some embodiments, a tracking process can be initiated by combining the image data acquired by the electronic device 110 with that of the wearable accessory 125 to start a tracking process. In the example figure shown, the image data acquired by the electronic device 110 may include a view of the wearable accessory 125. The electronic device 110 can then register the wearable accessory in its reference frame.This can be achieved by linking the coordinates of the electronic device with the coordinates of the portable accessory device. The result is the device-headset transformation 145, which represents a relationship between the two coordinate systems. When sensor data 130A is received from the portable accessory device 125, it can be tracked in a common coordinate system with the electronic device 110.

[0017] In some embodiments, certain hand positions or movements, or sequences of positions or movements (such as snapping or double-tapping), can be used to trigger user input actions. In some embodiments, the wearable accessory device 125 can be configured to detect a user input movement, such as a specific position of the hand, forearm, wrist, or the like. In the example of Fig. In 1A, the user points to image A 115. The target of the hand position can be determined in various ways. In this example, the target is determined based on a hand vector 140A that extends from the user's hand to image A 115. Since in Fig. 1A When the camera of the electronic device 110 is active, the target of the hand can be confirmed by image data.

[0018] In Fig. Figure 1B shows the user 105 performing an input gesture. The tracking mode no longer relies on camera data acquired by the electronic device 110. Specifically, hand tracking is now performed using sensor data 130B from the wearable accessory device 125, which does not include image data. Thus, tracking occurs regardless of whether the wearable accessory device is within the camera's field of view or whether the camera is actively acquiring image data. In some embodiments, the sensor data can be applied to a neural posture model to determine a movement of the user's arm, along with an appropriate confidence level for the user's movement. The movement can then be transformed based on the device-headset transformation 145 to determine an arm posture in a common coordinate system with the electronic device 110.In some embodiments, the sensor data 130B can be transmitted to the electronic device 110, enabling the electronic device 110 to perform neural tracking. Alternatively, the wearable accessory device 125 can perform neural tracking and transmit the result to the electronic device 110. The electronic device 110 can then determine the corresponding hand vector 140B. Furthermore, in some embodiments, the electronic device 110 can determine that the user is pointing at image B 120 if the location of image B 120 is available in a local environment map or is otherwise known to the electronic device 110. For example, a relationship between the posture of the wearable accessory device and the hand may be known, for instance, from registration data or other user-specific data.Alternatively, a relationship can be inferred between the posture of the portable accessory and the hand. The vector can be determined based on the general direction in which the hand is pointing, based on the position of the hand and / or the portable accessory 125. Furthermore, performing an observation gesture can signal the headset to activate the camera and determine the target of the pointing direction by deriving a depth and semantic representation of the scene from the camera after the gesture, in order to determine an intersection point of the vector with an object or component of the environment.

[0019] As described in more detail below, the techniques described here enable drift correction. In particular, various tracking parameters can be monitored to determine whether correction is required. This can be done, for example, periodically in response to the detection that a confidence value of the position information falls below a predetermined value, or similar situations. According to some embodiments, drift correction involves acquiring image data from the portable accessory, either at the time the drift correction is triggered or when the portable accessory is within the camera's field of view. In some embodiments, the user may be prompted to place the portable accessory within the camera's field of view.The position of the portable device determined from the image data can then be used to re-anchor the portable accessory device in the common coordinate system.

[0020] Fig. Figure 2 shows a flowchart of a technique for using a combination of an accessory device and a camera for hand tracking according to some embodiments. In particular, it relates to Fig. 2. A technique for the selective use of camera data to support hand tracking using non-image sensor data. For the sake of clarity, the following steps are described as being performed by specific components. However, it is understood that the various actions can be performed by alternative components. The various actions can be performed in a different order. Furthermore, some actions can be performed simultaneously, and some may not be necessary, or others may be added.

[0021] Flowchart 200 begins at block 205, where the wearable accessory device is registered with the camera's reference frame. According to one or more embodiments, the registration process at block 210 includes capturing a reference frame. The reference frame can be captured by a camera on the head-worn device, for example, the electronic device 110. As described above, the camera can be an outward-facing camera directed toward an environment and configured to capture scene data in which a user's hand or arm is visible, or more precisely, in which a wearable accessory device 125 is visible.

[0022] Flowchart 200 continues with block 215, where the transformation between the wearable accessory and the head-worn device is determined based on the reference frame. The position of the wearable accessory visible in the reference frame is compared with the wearable accessory's sensor data to determine the transformation. In some embodiments, the image data can be used as input to a network trained to predict position information for the device based on image data. Subsequently, at block 220, the tracking of a wearable accessory is initialized. Alternatively, a rule-based deterministic process can be used to convert the image data into position information for the device. In some embodiments, initializing the wearable accessory tracking activates the wearable accessory as a controller.For example, the wearable accessory device can be tracked to determine positional information, which can then be used to determine user input parameters, such as the posture or orientation of the hand or arm. Optionally, the camera can be turned off, as shown in Decision Block 225. That is, while the wearable accessory device is enabled as a controller, camera data is not used to track the wearable accessory device. However, in some embodiments, the camera may remain turned on or be in a high-performance mode, for example, when the camera is used for other functions of the head-worn device. These camera images are not used to track wearable accessories. The process for registering the wearable accessory device is described below. Fig. 3 described in more detail.

[0023] Flowchart 200 continues with Block 230, where wearable accessory tracking is performed. As described above, wearable accessory tracking may involve determining the position and / or orientation of a user's hand or arm based on sensor data acquired by the wearable accessory. Block 235 retrieves sensor data from the wearable accessory. In some embodiments, wearable accessory sensor data may be retrieved from an IMU or other motion sensor on or in the wearable accessory. Therefore, the wearable accessory sensor data may not be camera sensor data.

[0024] Flowchart 200 continues with block 240, where the position of the wearable accessory is determined using sensor data and the transformation determined during registration. Specifically, the motion data from the wearable accessory's sensor data is used to determine any deviation from a previously known position, such as translation and / or rotation. The transformation can also be used to translate the wearable accessory's position into a common coordinate system with the head-worn device. In block 245, the wearable accessory's position is used to analyze user input. For example, the direction or posture of a hand or arm can be used to determine where a user input action is intended to be directed.This can include, for example, interaction with virtual content, referencing physical objects to the wearable accessory or head-worn device, or similar actions. The procedure for tracking wearable accessories is described below with reference to [reference to relevant section]. Fig. 4 described in more detail.

[0025] At block 250, it is determined whether a correction criterion is met. The correction criterion might indicate that drift correction should be performed to ensure the accuracy of tracking the portable accessory. In some embodiments, the correction criterion can be met occasionally or regularly, for example, after a predefined time interval. As another example, the correction criterion can be met based on confidence values ​​for tracking portable accessories. For instance, a trained network or a rule-based deterministic model used to determine the movement of the portable accessory at block 240 can provide the confidence value that can be used to determine whether a correction criterion is met.In some embodiments, the correction criterion may include a combination of factors that can be weighted based on a specific user, device, environment, application, or the like. Alternatively, the correction criterion may include a determination of whether the wearable accessory is visible to the camera. The head-worn device may determine visibility based on location information transmitted from the wearable accessory or based on the known location of the wearable accessory relative to the head-worn device. If the correction criterion is found not to be met, the flowchart returns to block 230, and tracking of the wearable accessory continues without considering the camera data.

[0026] Returning to block 250: If a correction criterion is found to be met, the flowchart continues to block 255 and the drift correction is performed. In some embodiments, performing the drift correction at block 260 includes switching on the camera of the head-worn device or another electronic device in the vicinity. At block 265, one or more image frames are acquired.

[0027] Flowchart 200 continues with block 270, where the position of the portable accessory is refined based on the image data and the portable accessory's sensor data. For example, the actual position of the portable accessory can be determined from the captured single image and compared with a calculated position from the portable accessory tracking. The portable accessory can then be reinitialized for portable accessory tracking. In the optional block 275, the camera can be turned off, as the image data will not be used for portable accessory tracking. The flowchart then continues with block 230, where the portable accessory tracking is performed.

[0028] Fig. Figure 3 shows a flowchart of a technique for registering a portable accessory device for tracking portable accessories. In particular, it describes Fig. 3. An example technique for determining a common coordinate system between the head-worn device and the portable accessory device, so that tracking of the portable accessory device can be carried out as above with reference to Block 205 of Fig. 2 described. For clarification, the following steps are described as being carried out by specific components, such as those described above in relation to Fig. As described in sections 1A-1B, the various actions can be performed by alternative components. These actions can also be performed in a different order. Furthermore, some actions can be performed simultaneously, some may be unnecessary, and others may be added.

[0029] Flowchart 300 begins at block 305, where image data of the environment in which the portable accessory device is located is acquired. In some embodiments, the images can be acquired by a camera of an electronic device, such as a head-worn device, as described by electronic device 110 in Fig. 1A is shown.

[0030] Flowchart 300 continues with block 310, where the camera coordinates are determined. The camera coordinates specify the position of the camera capturing the image. Additionally or alternatively, the coordinates can correspond to the electronic device housing the camera, for example, a head-mounted device.

[0031] In Block 315, the coordinates of the portable accessory are determined from the image data. The coordinates for the portable accessory can be determined in various ways. As shown, for example, in Block 320, the image data can be applied to a basic transformation network to obtain a transformation between the coordinates of the camera capturing the image and the position of the portable accessory. The basic transformation network can be a neural network trained to predict posture information of an object based on image data. For example, the relative position and posture of the portable accessory can be predicted from the image data. In some embodiments, the basic transformation network can use one or more image frames and can employ a single frame or stereo frames.This means that depth can be predicted based on a single image frame, without relying on stereo frames, depth sensor data, or other sensor data acquired by the head-worn device. Accordingly, the motion data collected by the wearable accessory can then be used, via the basis transformation, to determine an updated position of the wearable accessory and, consequently, the hand. In other words, the basis transformation bridges link the coordinates of the head-worn device with the coordinates of the wearable accessory to determine hand-based user input actions. As another example, a formulaic approach based on heuristics and properties of the image data can be used to determine the coordinates of wearable accessories.

[0032] Flowchart 300 ends at block 325, where the wearable accessory is activated as a controller. In some embodiments, activating the wearable accessory as a controller involves changing the operation of an image sensor in the head-worn device. For example, the camera may be disabled. Another example: The camera may enter a power-saving mode, thereby combining the motion sensor data from the wearable accessory with that of the camera in the head-worn device.

[0033] In Fig. Figure 4 presents a flowchart of a technique for performing hand tracking using motion data from the portable accessory device without camera data. In particular, it describes Fig. 4. An example technique for estimating hand position based on motion data acquired by the portable accessory device, as above in reference to block 230 of Fig. 2 described. For clarification, the following steps are described as being carried out by specific components, such as those described above in relation to Fig. As described in sections 1A-1B, the various actions can be performed by alternative components. These actions can also be performed in a different order. Furthermore, some actions can be performed simultaneously, some may be unnecessary, and others may be added.

[0034] The flowchart formula begins at block 405, where the exact position of the portable accessory is obtained. The current position of the portable accessory at 405 can correspond to the position of the portable accessory at the time the portable accessory is activated as a controller, for example, at block 325. Fig. 3. The position can be considered accurate because it was determined not only from the sensor data of the wearable accessory device, but also from camera data and / or other sensor data acquired during the registration process by another device, such as the head-worn device.

[0035] Flowchart 400 continues with block 410, where sensor data from the wearable accessory is obtained. As described above, sensor data can include motion data, such as data acquired by an IMU, accelerometer, gyroscope, magnetometer, or similar sensors. The sensor data can therefore indicate a change in the position and / or location of the wearable accessory. For example, the sensor data can show translational and / or rotational characteristics of the detected motion.

[0036] In Block 415, the sensor data are applied to a neural posture model. According to one or more embodiments, the neural posture model can be a deep-learning-based network configured to process motion data, such as IMU data, to estimate the posture of the device. For example, the neural posture model can be configured to predict translation and rotation based on the IMU data. Because sensor data can lose accuracy over time, the neural posture model can provide more accurate motion data than relying directly on the sensor data. Furthermore, the neural posture model can provide a confidence score for the predicted motion data. Thus, Flowchart 400 in Block 420 includes obtaining the motion data and confidence score of the wearable accessory device from the sensor data.

[0037] Flowchart 400 continues to block 425, where it is determined whether the confidence value satisfies a correction criterion. The correction criterion can be a value of the confidence value that indicates the accuracy of the predicted position information. In some embodiments, the determination may also take into account whether the portable accessory device is within the field of view of the head-worn device. If the confidence value is insufficient to satisfy the correction criterion, flowchart 400 continues to block 430. At block 430, the position of a portable accessory device is determined. In particular, a deviation from a previously known position, such as a translation and / or rotation, is determined from the motion data of the sensor data of the portable accessory device.Furthermore, the transformation can be used to translate the position of a portable accessory device into a common coordinate system with the head-worn device, for example by applying the offset to the previously known position, such as the position determined at Block 405.

[0038] In block 435, a hand position is determined based on the position of the portable accessory device. For example, as in Fig. As described in Figure 1B, a hand vector 140B is determined from a posture of the portable accessory device 125 based on a predefined or derived spatial relationship between the hand and the portable accessory device 125. The hand vector can be user-specific or generally determined based on the position and / or orientation of the portable accessory device.

[0039] Flowchart 400 continues to block 440. Hand position is used to analyze user input. For example, hand position can be analyzed to determine whether the hand is pointing at a registered object in the physical environment with which the user can interact after selection. For this purpose, in some embodiments, hand position alone may be sufficient to trigger a user input action. As another example, hand position can be used in conjunction with other triggers of user input, such as detected user input via speech, tactile input, visual input, eye tracking, or the like, to determine whether a user input action should be triggered. Flowchart 400 then returns to block 410, and additional sensor data is received.

[0040] Returning to Block 425: If it is determined that the confidence score satisfies the correction criterion, such as when the parameters defining the criterion are met, Flowchart 400 ends at Block 445 and the correction process is initiated. Specifically, the correction process may be a drift correction process involving re-anchoring the wearable accessory device and the head-worn device, as described below with reference to Fig. 5 is described in more detail.

[0041] Fig. Figure 5 shows a flowchart of a technique for correcting drift according to one or more embodiments. In particular, it illustrates Fig. 5 the process of re-anchoring the portable accessory device to the head-worn device to improve the accuracy of tracking the portable accessory device, as above in reference to block 255 of Fig. 2 described. For clarification, the following steps are described as being carried out by specific components, such as those described above in relation to Fig. As described in sections 1A-1B, the various actions can be performed by alternative components. These actions can also be performed in a different order. Furthermore, some actions can be performed simultaneously, some may be unnecessary, and others may be added.

[0042] Flowchart 500 begins at block 505, where current sensor data from the portable accessory device is obtained. For example, the current sensor data from the portable accessory device may correspond to the sensor data from the portable accessory device that resulted in the confidence value meeting the correction criterion at block 425. Fig. 4 is fulfilled. Alternatively, a next frame or a later frame of sensor data from the portable accessory device can be obtained simultaneously with the additional sensor data collected at block 510.

[0043] According to one or more embodiments, the additional sensor data collected at block 510 can be collected by an additional device, such as the head-worn device referred to as electronic device 110 in Fig. Figure 1A illustrates this. In some embodiments, additional or alternative forms of sensor data may be used to re-anchor the portable accessory device to the head-worn device. This may include, for example, magnetometer data, ultrasound data, ultra-wideband signals, or the like. Obtaining additional sensor data at block 510 may involve activating or turning on additional sensors, either within the portable accessory device, the head-worn device, or another electronic device.

[0044] In some embodiments, the additional sensor data can be acquired by a camera on the head-worn device. Therefore, in Block 515, the camera of the head-worn device is switched on. This step is optional because, in some embodiments, the camera may already be powered, but the camera data may not be used for tracking wearable accessory devices, as described above in relation to Fig. 4 described. In some embodiments, the camera can be switched from a low-energy mode to a high-performance mode at block 515. Subsequently, additional image data is acquired by the camera in block 520.

[0045] The flowchart continues with Block 525, where drift correction is performed based on the current sensor data and additional sensor data, such as camera data. As shown in Block 530, a fusion algorithm can be applied in some embodiments. In some embodiments, the fusion algorithm can be configured to anchor the wearable accessory device to the head-worn device to achieve a specific balance between smoothness and accuracy. For example, a smoother algorithm might feel more natural to a user but could be less accurate during correction. Conversely, a more accurate transition might feel distorted to a user but result in more precise tracking. In some embodiments, additional sensor data can be used, such as the additional sensor data described above in Block 510.Once the drift correction has been performed, the portable accessory device is reattached to the head-worn device and the tracking of the portable accessory device can be carried out as above with reference to . Fig. Section 4 will be continued as described. Furthermore, although not shown, the camera can be switched off, deactivated, or placed in a power-saving mode while tracking portable accessories.

[0046] Referring to Fig. Figure 6 shows a simplified block diagram of an electronic device 600. The electronic device 600 can be part of a multifunctional device, such as a mobile phone, a tablet computer, a personal digital assistant, a portable music / video playback device, a body-worn device, head-mounted systems, projection-based systems, a base station, a laptop computer, a desktop computer, a network device, or any other electronic systems as described herein. In some embodiments, the electronic device 600 can be a head-worn device. The electronic device 600 can include one or more additional devices, in which the various functions may be contained or distributed, such as server devices, base stations, accessory devices, etc.Furthermore, the electronic device can be communicatively connected via a network 675 to additional devices, such as the portable accessory device 680. Illustrative networks include, but are not limited to, a local area network such as a Universal Serial Bus (USB) network, an organization's local area network, and a wide area network such as the Internet. In some embodiments, the various devices can be connected more directly to one another, such as via Bluetooth or another short-range wireless connection.

[0047] The electronic device 600 can include one or more processors 615, such as a central processing unit (CPU) or a graphics processing unit (GPU). The electronic device 600 can also include a memory 605. The memory 605 can include one or more different types of memory that can be used to perform device functions in conjunction with the processor(s) 615. For example, the memory 605 can include a cache, ROM, RAM, or any type of transient or non-transient computer-readable storage medium capable of storing computer-readable code. The memory 605 can store various programming modules for execution by the processor(s) 615.The programming modules may, for example, include the registration module 630, which is configured to register the portable accessory device 680 with the electronic device 600, as described above. Fig. 3 described. According to some embodiments, the registration module 630 can be configured to link a coordinate system of the electronic device 600, determined, for example, by one or more sensors 625, with a coordinate system of the accessory device 680. The programming modules can also include a neural tracking module 635 configured to determine position and / or location information from sensor data received by the portable accessory device 680, as described above in relation to Fig. 4 described. The neural tracking module can be configured to determine position information without using camera or image data. The programming modules can also include a drift correction module 640, which is configured to re-anchor the portable accessory device 680 to the electronic device 600 when the output of the neural tracking module 635 includes a confidence value that falls below a threshold, as described above with respect to Fig. 5 described. The drift correction module 640 can use image data acquired by the cameras 620 to determine position information for the portable accessory device 680 from the acquired image data and use the position information to correct position information predicted by the neural tracking module.

[0048] The electronic device 600 may also include a storage device 610. The storage device 610 may include one or more non-transient computer-readable media, including, for example, magnetic disks (hard disks, floppy disks, and removable disks) and tape, optical media such as CD-ROMs and digital video discs (DVDs), and semiconductor storage devices such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM). The storage device 610 may be used to store various data and structures that can be used to store data related to device tracking and / or hand tracking for user input.For example, the memory 610 can include registration data 650, which can be used to determine a hand vector, such as a hand model, a skeleton, or other information about the user's hand, which, in conjunction with the position information of the wearable accessory device 680, can be used to determine user input actions. The memory 610 can also include a transformation memory 655. The transformation memory 655 can be used to store the transformation applied between the position information for the wearable accessory device 680 and the electronic device 600, so that the electronic device can determine a position and / or orientation of the wearable accessory device 680 based on the sensor data collected by the accessory device.Furthermore, the storage 610 can include a tracking model storage 660, which can include data for performing tracking of the portable accessory device 680 using sensor data.

[0049] The wearable accessory device 680 can be a user-worn accessory. Examples of wearable accessories include watches, rings, bracelets, or similar items equipped with a computing structure. For example, a wearable accessory device can include one or more memories 690 and one or more processors 685. The memory 690 can be configured to store computing modules that can be executed by the processor(s) 685. According to one or more embodiments, the electronic device 600 is equipped with one or more sensors 695 that can acquire location and / or motion data for the wearable accessory device 680. The sensor data can then be made available to the electronic device 600 for use in detecting user input actions.In some embodiments, the memory 690 and the processor(s) 685 can enable the portable accessory device 680 to perform at least some of the functions described with respect to the computing modules of the electronic device 600. For example, instead of sending IMU or other sensor data from sensor(s) 695 to the electronic device 600, the portable accessory device can perform some processing operations locally. For example, the portable accessory device 680 can execute a neural velocity model and send the velocity estimates to the electronic device 600.

[0050] Although the electronic device 600 and the portable accessory device 680 are depicted as comprising the numerous components described above, the various components may be distributed differently across the device or across additional devices in one or more embodiments. Accordingly, although certain calls and transmissions are described herein with respect to the respective systems as depicted, in one or more embodiments the various calls and transmissions may be performed differently or based on differently distributed functionality with a different objective. Furthermore, additional components may be used, and a certain combination of the functionality of one of the components may be combined.

[0051] Referring now to Fig. Figure 7 shows a simplified functional block diagram of an illustrative electronic multifunction device 700 according to one embodiment. Each of the electronic devices can be a multifunctional electronic device or include some or all of the components of a multifunctional electronic device described herein. The electronic multifunction device 700 can include a processor 705, a display 710, a user interface 715, graphics hardware 720, device sensors 725 (e.g., a proximity / ambient light sensor, an accelerometer, and / or a gyroscope), a microphone 730, audio codec(s) 735, one or more loudspeakers 740, communication switching logic 745, digital image acquisition switching logic 750 (e.g., including a camera system), video codec(s) 755 (e.g., a video camera), and a video video controller 755.to support a digital image capture unit), a memory 760, a storage device 765, and a communication bus 770. The electronic multifunction device 700 can be, for example, a digital camera or a personal electronic device, such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0052] The processor 705 can execute instructions necessary to perform or control the operation of many functions carried out by the device 700 (such as generating and / or processing images, as disclosed herein). For example, the processor 705 can drive the display 710 and receive user input from the user interface 715. The user interface 715 can enable a user to interact with the device 700. For example, the user interface 715 can take a variety of forms, such as a button, a keypad, a rotary control, a click wheel, a keyboard, a display screen, a touchscreen, a gaze, and / or gestures. The processor 705 can also, for example, be a system-on-a-chip, as found in mobile devices, and include a dedicated GPU.The 705 processor can be based on Reduced Instruction Set (RISC) or Complex Instruction Set (CISC) computer architectures, or any other suitable architecture, and can include one or more processing cores. The 720 graphics hardware can be specialized computing hardware for processing graphics and / or assisting the 705 processor in processing graphics information. In one embodiment, the 720 graphics hardware can include a programmable GPU.

[0053] The image acquisition switching logic 750 can include two (or more) lens assemblies 780A and 780B, each lens assembly having its own focal length. For example, lens assembly 780A can have a shorter focal length compared to lens assembly 780B. Each lens assembly can have its own associated sensor element 790A and sensor element 790B. Alternatively, two or more lens assemblies can share a common sensor element. The image acquisition switching logic 750 can acquire still and / or video images. The output from the image acquisition switching logic 750 can be processed by the video codec(s) 755 and / or the processor 705 and / or the graphics hardware 720 and / or a dedicated image processing unit or pipeline contained within the switching logic 750. Images captured in this way can be stored in memory 760 and / or memory 765.

[0054] The sensor and camera switching logic 750 can capture still and video images, which are processed, at least in part, by the video codec(s) 755 and / or the processor 705 and / or the graphics hardware 720 and / or a dedicated image processing unit contained within the switching logic 750, as described in this disclosure. Captured images can be stored in the memory 760 and / or the storage 765. The memory 760 can include one or more different types of media used by the processor 705 and the graphics hardware 720 to perform device functions. For example, the memory 760 can include memory cache, read-only memory (ROM), and / or random-access memory (RAM). The storage 765 can store media (e.g., audio, image, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data.Storage 765 can include one or more non-transient, computer-readable storage media, including, for example, magnetic disks (hard disks, floppy disks, and removable disks) and tape, optical media such as CD-ROMs and DVDs, and semiconductor storage devices such as EPROM and EEPROM. Storage 760 and Storage 765 can be used to store computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed, for example, by Processor 705, such computer program code can implement one or more of the procedures described herein.

[0055] Several processes defined herein consider the possibility of obtaining and using a user's personal information. For example, such personal information may be used to track the user's attitude and / or movement. However, to the extent that such personal information is collected, it should be obtained with the user's consent, and the user should have knowledge of and control over the use of their personal information.

[0056] Personal information will be used by appropriate parties only for lawful and reasonable purposes. Parties using such information will adhere to privacy policies and practices that meet at least the requirements of applicable laws and regulations. Furthermore, such policies must be well-established and recognized as meeting or exceeding governmental / industry standards. In addition, these parties will refrain from disseminating, selling, or otherwise sharing such information outside of appropriate and lawful purposes.

[0057] Furthermore, the intention of this disclosure is to ensure that personal data is managed and handled in a manner that minimizes the risks of unintentional or unauthorized access or use. This risk can be minimized by limiting data collection and deleting data once it is no longer needed. Additionally, and where necessary, including in certain health-related applications, data anonymization may be used to protect a user's privacy. Anonymization may be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth), controlling the amount or specificity of the data stored (e.g., collecting location data at the city level rather than the address level), and the manner in which data is stored (e.g.,Aggregation of data across users), is controlled and / or by other methods.

[0058] It should be clarified that the foregoing description is intended to be illustrative and not limiting. The material has been provided to enable the person skilled in the art to manufacture and use the disclosed subject matter as claimed, and it is provided in the context of certain embodiments, variations of which are readily apparent to the person skilled in the art (e.g., some of the disclosed embodiments can be used in combination with one another). Accordingly, the specific arrangement of steps or actions described in Fig. 3-4 are shown, or the arrangement of elements that are in Fig.The figures shown in Figures 1-2 and 5-7 should not be interpreted as limiting the scope of protection of the disclosed subject matter. The scope of protection of the invention should therefore be determined with reference to the appended claims together with the full scope of protection of equivalents covered by those claims. In the appended claims, the terms "including" and "in which" are used as the simple equivalents of the respective terms "comprising" and "whereby".

Claims

[1] Procedure, encompassing: Receiving image data from a portable accessory device on a head-worn device; Receiving sensor data from the portable accessory device; Determining a transformation of a portable accessory device based on image data and sensor data; Identifying the position of the portable accessory device using the transformation of the portable accessory device and additional sensor data from the portable accessory device; Obtaining additional image data from the portable accessory device on a head-worn device; and Adjusting a tracked position of the portable accessory device. [2] Method according to claim 1, wherein the tracked position of the portable accessory device is adjusted in response to a confidence value for the position that satisfies a correction criterion. [3] The method of claim 2, further comprising: Identifying an additional position of the portable accessory device using the additional sensor data. [4] The method of claim 3, further comprising: Re-anchoring the portable accessory device to a head-worn device to obtain updated position information, Tracking is resumed using the additional sensor data and updated position information. [5] A method according to any one of claims 1 to 4, further comprising, in response to the finding that the portable accessory device is in the field of view of a camera capturing the image data, the following: Turning off the camera, the determination is carried out while the camera is operating in a switched-on mode. [6] Method according to any one of claims 1 to 4, wherein the tracking of the portable accessory device comprises: Determining a hand position based on tracking the portable accessory device. [7] Method according to claim 6, further comprising: Detecting a user input action and Determining user input based on hand position and user input action. [8] Non-transitory computer-readable medium comprising a computer-readable code for carrying out the method by one or more processors according to any one of claims 1 to 7. [9] System, comprehensive: one or more processors; and one or more computer-readable media comprising computer-readable code executable by one or more processors for: Receiving image data from a portable accessory device on a head-worn device; Receiving sensor data from the portable accessory device; Determining a transformation for a portable accessory device based on image data and sensor data; Identifying the position of the portable accessory device using the transformation of the portable accessory device and additional sensor data from the portable accessory device; Receive additional image data from the portable accessory device on a head-worn device; and Adjusting a tracked position of the portable accessory device. [10] System according to claim 9, wherein the tracked position of the portable accessory device is adjusted in response to a confidence value for the position that satisfies a correction criterion. [11] System according to claim 10, further comprising computer-readable code for: Identifying an additional position of the portable accessory device using the additional sensor data. [12] System according to claim 11, further comprising a computer-readable code for: Re-anchoring the portable accessory device to the head-worn device to obtain updated position information, Tracking is resumed using the additional sensor data and updated position information. [13] System according to any one of claims 9 to 12, further comprising a computer-readable code to perform the following in response to the detection that the portable accessory device is in the field of view of a camera which is capturing the image data: Turning off the camera, the determination is carried out while the camera is operating in a switched-on mode. [14] System according to claims 9 to 12, wherein the computer-readable code for tracking the portable accessory device comprises a computer-readable code for: Determining a hand position based on the position of the portable accessory device. [15] System according to claim 14, further comprising computer-readable code for: Detecting a user input action and Determining user input based on hand position and user input action.

Images