Hand and totem input fusion for wearable systems

Abstract

A technique for fusing hand posture data associated with a user's hands and handheld device posture data associated with a handheld device. In some embodiments, both hands of a user are utilized in the embodiments described herein. Hand posture data including the position of the hand within a reference frame is captured using a hand posture sensor. Handheld device posture data including the position of the handheld device within the reference frame is captured using a handheld device posture sensor. A posture discrepancy between the hand posture data and the handheld device posture data is determined, the posture discrepancy including at least the position discrepancy between the position of the hand and the position of the handheld device. A fusion operation is performed based on the posture discrepancy.

Description

[Background technology]

[0001] (Cross-reference of related applications) This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 981,930, filed on 26 February 2020, titled "HAND AND TOTEM INPUT FUSION FOR WEARABLE SYSTEMS," the entire contents of which are incorporated herein by reference for any purpose.

[0002] Modern computing and display technologies are driving the development of systems for so-called "virtual reality" or "augmented reality" experiences, where digitally reproduced images or parts thereof are presented to the user in a manner that appears, or can be perceived, as real. Virtual reality, or "VR," scenarios typically involve the presentation of digital or virtual imagery without transparency to other real-world visual inputs, while augmented reality, or "AR," scenarios typically involve the presentation of digital or virtual imagery as an extension of the user's visualization of the real world around them.

[0003] Despite the advancements made in these display technologies, there remains a need in this field for improved methods, systems, and devices related to augmented reality systems, particularly display systems. [Overview of the Initiative] [Means for solving the problem]

[0004] This disclosure relates, in general terms, to techniques for improving the performance and user experience of optical systems. More specifically, embodiments of this disclosure provide methods for operating augmented reality (AR), virtual reality (VR), or mixed reality (MR) wearable systems in which a handheld device is employed to assist the operation of the wearable system. An overview of this disclosure is provided below as a list of embodiments. Any reference to a set of embodiments, as used below, is understood to be a disjunct reference to each of those embodiments (for example, “Embodiment 1-4” is understood to be “Embodiment 1, 2, 3, or 4”).

[0005] Example 1 is a method for fusing hand posture data and handheld device posture data, comprising the steps of: capturing hand posture data, including the position of the hand in a reference frame, using a hand posture sensor; capturing a handheld device, including the position of the handheld device in a reference frame, using a handheld device posture sensor; determining the posture difference between the hand posture data and the handheld device posture data, wherein the posture difference includes a positional difference between the position of the hand and the position of the handheld device; and performing a fusing operation based on the posture difference.

[0006] Example 2 is the same as in Example 1, wherein the positional difference is the distance between the position of the hand and the position of the handheld device.

[0007] Example 3 is the method of Example 2, further comprising the step of determining whether the distance is above or below a distance threshold.

[0008] Example 4 is the method of Example 3, wherein if it is determined that the distance exceeds a distance threshold, the step of performing a fusion operation includes determining that the handheld device is not being held by a hand.

[0009] Example 5 is the method of Example 3, wherein if it is determined that the distance is less than or equal to the distance threshold, the step of performing a fusion operation includes the step of determining that the handheld device is being held by a hand.

[0010] Example 6 is the method of Example 3, wherein if it is determined that the distance exceeds a distance threshold, the step of performing a fusion operation includes a step of suppressing hand posture data or a step of suppressing handheld device posture data.

[0011] Example 7 is the method of Example 3, wherein, when it is determined that the distance is less than or equal to the distance threshold, the steps of performing a fusion operation include a step of using hand posture data to increase handheld device posture data, a step of using handheld device posture data to increase hand posture data, or a step of using hand posture data and handheld device posture data to generate combined posture data.

[0012] Example 8 is the method of Example 1, wherein the step of capturing hand posture data using a hand posture sensor includes the steps of identifying a plurality of feature points associated with the hand and determining the position of the hand based on the plurality of feature points.

[0013] Example 9 is the method described in Example 1, wherein the hand posture data includes both the hand position and the hand orientation.

[0014] Example 10 is the method described in Example 1, wherein the hand posture data includes the position of the hand but does not include the orientation of the hand.

[0015] Example 11 is the method according to Example 1, wherein the handheld device attitude data includes both the position and orientation of the handheld device.

[0016] Example 12 is the method described in Example 1, where the handheld device posture data includes the position of the handheld device but does not include the orientation of the handheld device.

[0017] Example 13 is the method described in Example 1, where the hand gesture data includes the orientation of the hand, and the handheld device posture data includes the orientation of the handheld device.

[0018] Example 14 is the method described in Example 13, where the posture difference includes the orientation difference between the orientation of the hand and the orientation of the handheld device.

[0019] Example 15 is the method described in Example 14, where the orientation difference is the angular offset between the orientation of the hand and the orientation of the handheld device.

[0020] Example 16 is the method described in Example 15, further including the step of determining whether the angular offset is greater than or less than an angular threshold.

[0021] Example 17 is the method described in Example 13, further including the step of correcting the position of the handheld device based on the three-dimensional (3D) model of the handheld device and the orientation of the handheld device.

[0022] Example 18 is the method described in Example 1, where the hand gesture sensor is different from the handheld device posture sensor.

[0023] Example 19 is the method described in Example 1, where the hand gesture sensor and the handheld device posture sensor are the same sensor.

[0024] Example 20 is the method described in Example 1, where the hand gesture sensor is an image capture device.

[0025] Example 21 is the method described in Example 1, where the hand gesture sensor is a depth sensor.

[0026] Example 22 is the same as in Example 1, wherein the hand posture sensor is mounted on a headset.

[0027] Example 23 is the method according to Example 1, wherein the handheld device attitude sensor includes an electromagnetic (EM) transmitter and an EM receiver.

[0028] Example 24 is the same as in Example 23, wherein the EM transmitter is mounted on a handheld device and the EM receiver is mounted on a headset.

[0029] Embodiment 25 is a system comprising: a hand posture sensor configured to capture hand posture data, including the position of a hand within a reference frame; a handheld device posture sensor configured to capture handheld device posture data, including the position of a handheld device within a reference frame; and one or more processors communicating with the hand posture sensor and the handheld device posture sensor, configured to perform an action, which includes the steps of: determining a posture difference between the hand posture data and the handheld device posture data, wherein the posture difference includes a position difference between the position of the hand and the position of the handheld device; and performing a fusion action based on the posture difference.

[0030] Example 26 is the system described in Example 25, wherein the positional difference is the distance between the position of the hand and the position of the handheld device.

[0031] Example 27 is the system according to Example 26, further comprising the step of determining whether the distance is above a distance threshold, below a distance threshold, or equal to a distance threshold.

[0032] Example 28 is the system described in Example 27, wherein if it is determined that the distance exceeds a distance threshold, the step of performing a fusion operation includes determining that the handheld device is not being held by a hand.

[0033] Example 29 is the system described in Example 27, wherein if it is determined that the distance is less than or equal to the distance threshold, the step of performing a fusion operation includes determining that the handheld device is being held by a hand.

[0034] Example 30 is the system described in Example 27, wherein if it is determined that the distance is less than a distance threshold, the step of performing a fusion operation includes a step of suppressing hand posture data or a step of suppressing handheld device posture data.

[0035] Example 31 is the system described in Example 27, wherein if it is determined that the distance exceeds a distance threshold, the step of performing a fusion operation includes a step of suppressing hand posture data or a step of suppressing handheld device posture data.

[0036] Example 32 is the system described in Example 27, wherein, when it is determined that the distance is less than or equal to the distance threshold, the steps of performing a fusion operation include a step of augmenting hand-held device posture data using hand posture data, a step of augmenting hand posture data using hand-held device posture data, or a step of generating combined posture data using hand posture data and hand-held device posture data.

[0037] Example 33 is the system described in Example 25, wherein the step of capturing hand posture data using a hand posture sensor includes the steps of identifying a plurality of feature points associated with the hand and determining the position of the hand based on the plurality of feature points.

[0038] Example 34 is the system described in Example 25, wherein the hand posture data includes both hand position and hand orientation.

[0039] Example 35 is the system described in Example 25, wherein the hand posture data includes the position of the hand but does not include the orientation of the hand.

[0040] Example 36 is the system described in Example 25, wherein the handheld device attitude data includes both the position and orientation of the handheld device.

[0041] Example 37 is the system described in Example 25, wherein the handheld device attitude data includes the position of the handheld device but does not include the orientation of the handheld device.

[0042] Example 38 is the system described in Example 25, wherein the hand posture data includes hand orientation, and the handheld device posture data includes hand orientation.

[0043] Example 39 is the system described in Example 38, wherein the orientation difference includes an orientation difference between the orientation of the hand and the orientation of the handheld device.

[0044] Example 40 is the system according to Example 38, further comprising the step of correcting the position of the handheld device based on a three-dimensional (3D) model of the handheld device and the orientation of the handheld device.

[0045] Example 41 is the system described in Example 39, wherein the orientation difference is an angular offset between the orientation of the hand and the orientation of the handheld device.

[0046] Example 42 is the system according to Example 41, further comprising the step of determining whether the angular offset is above or below an angular threshold.

[0047] Example 43 is the system according to Example 38, further comprising the step of correcting the position of the handheld device based on a three-dimensional (3D) model of the handheld device and the orientation of the handheld device.

[0048] Example 44 is the system described in Example 25, in which the hand posture sensor is different from the handheld device posture sensor.

[0049] Example 45 is the same system as in Example 25, in which the hand posture sensor and the handheld device posture sensor are the same sensor.

[0050] Example 46 is the system described in Example 25, wherein the hand posture sensor is an image acquisition device.

[0051] Example 47 is the system described in Example 25, wherein the hand posture sensor is a depth sensor.

[0052] Example 48 is the system described in Example 25, in which the hand posture sensor is mounted on a headset.

[0053] Example 49 is the system described in Example 25, in which the handheld device attitude sensor includes an electromagnetic (EM) transmitter and an EM receiver.

[0054] Example 50 is the system described in Example 49, wherein the EM transmitter is mounted on a handheld device and the EM receiver is mounted on a headset.

[0055] Embodiment 51 is a system comprising a plurality of electronic hardware components, at least one wearable component configured to be worn by a user; a plurality of electronic hardware components, at least a portion of which are contained within or attached to the at least one wearable component, configured to present content to the user; a plurality of sensors, at least one of which includes an image capture device; and at least one processor that communicates with the display, the plurality of sensors, and a portable input device configured to be held in the user's hand, and which receives image data from the image capture device, receives data from one or more of the plurality of sensors different from the image capture device, and at least partially determines whether the portable input device is positioned below at least one threshold distance away from the user's hand based on the image data received from the image capture device and the data received from one or more of the plurality of sensors different from the image capture device, and at least partially controls the operation of the display based on whether it is determined that the portable input device is positioned below at least one threshold distance away from the user's hand.

[0056] Embodiment 52 is a system according to Embodiment 50, wherein, in order to determine whether a portable input device is positioned less than one threshold distance away from the user's hand and to control the operation of the display, at least one processor is configured to determine, at least in part, whether the portable input device is positioned within a volume defined relative to the user's hand, based on image data received from an image capture device and data received from one or more of a plurality of sensors different from the image capture device, and at least in part to control the operation of the display based on whether it has been determined that the portable input device is positioned within a volume defined relative to the user's hand.

[0057] Example 53 is the system described in Example 51, wherein the volume defined relative to the user's hand is spherical or ellipsoidal in shape.

[0058] Embodiment 54 is the system described in Embodiment 51, wherein at least one processor is further configured to determine, at least partially, one or both of the position and orientation of the user's hand based on image data received from an image capture device, and at least partially, to determine one or both of the position and orientation of a portable input device based on data received from one or more of a plurality of sensors different from the image capture device, and at least one processor is configured to define the volume relative to the user's hand based at least partially on the position of the user's hand, the orientation of the user's hand, the position of the portable input device, the orientation of the portable input device, or a combination thereof.

[0059] Embodiment 55 is a system as described in Embodiment 53, wherein at least one processor is configured to adjust one or more parameters of a volume based at least in part on one or more factors, the one or more parameters of a volume being the position of the volume, the orientation of the volume, the size of the volume, the shape of the volume, or a combination thereof, the one or more factors being the distance between at least one wearable component and a portable input device, the distance between at least one wearable component and a user's hand, whether it is determined that the portable input device or the user's hand is positioned closer to the at least one wearable component, data received from some or all of a plurality of sensors, or data received from a portable input device.

[0060] Embodiment 56 is the system according to Embodiment 51, wherein the volume defined relative to the user's hand comprises a first volume, and at least one processor is further configured to determine, at least in part, whether the portable input device is located within a second volume defined relative to the user or at least one wearable component, based on image data received from an image capture device and data received from one or more of a plurality of sensors different from the image capture device, and at least in part, whether the portable input device is determined to be located within the volume defined relative to the user's hand, and to control the operation of the display based on whether the portable input device is determined to be located within the volume defined relative to the user's hand, at least one processor is configured to control the operation of the display based on whether (i) the portable input device is determined to be located within the first volume defined relative to the user's hand, or (ii) whether the portable input device is determined to be located within a second volume defined relative to the user or at least one wearable component.

[0061] Example 57 is the system described in Example 55, wherein the second volume is defined with respect to one or more parts of the user that are different from the user's hands, and the one or more parts of the user that are different from the user's chest, shoulders, head, arms, eyes, or any combination thereof, with respect to one or more parts of the user's chest, shoulders, head, arms, eyes, or any combination thereof.

[0062] Example 58 is the system described in Example 56, wherein the second volume corresponds to a range of locations in three-dimensional space relative to the user, located within the user's reach.

[0063] Embodiment 59 is the system according to Embodiment 50, wherein, in order to control the operation of the display based at least in part on whether it is determined that a portable input device is located less than one threshold distance away from the user's hand, at least one processor is configured to control the operation of the display according to a first operating mode in which, in response to a determination that the portable input device is located less than one threshold distance away from the user's hand, user interaction with at least some of the content presented by the display via the portable input device is enabled and hand-based user interaction with at least some of the content presented by the display is limited or disabled, and in response to a determination that the portable input device is not located less than one threshold distance away from the user's hand, user interaction with at least some of the content presented by the hand-based display is enabled and user interaction with at least some of the content presented by the display via the portable input device is limited or disabled. The present invention provides, for example, the following: (Item 1) A method for fusing hand posture data and handheld device posture data, wherein the method is: Using a hand posture sensor, capture the hand posture data, including the position of the hand within a reference frame, Using a handheld device attitude sensor, the handheld device is captured, including its position within the reference frame. Determining the posture difference between the hand posture data and the handheld device posture data, wherein the posture difference includes the positional difference between the hand position and the handheld device position. Based on the aforementioned difference in posture, a fusion movement is performed. Methods that include... (Item 2) The method according to item 1, wherein the positional difference is the distance between the position of the hand and the position of the handheld device. (Item 3) The method according to item 2, further comprising determining whether the distance is above or below a distance threshold. (Item 4) If it is determined that the distance exceeds the distance threshold, the fusion operation is performed. Determining that the handheld device is not being held by the hand. The method described in item 3, including the method described in item 3. (Item 5) If it is determined that the distance is less than or equal to the distance threshold, the fusion operation is performed. Determining that the handheld device is being held by the hand. The method described in item 3, including the method described in item 3. (Item 6) If it is determined that the distance exceeds the distance threshold, the fusion operation is performed. Suppressing the aforementioned hand posture data, or To suppress the posture data of the aforementioned handheld device. The method described in item 3, including the method described in item 3. (Item 7) If it is determined that the distance is less than or equal to the distance threshold, the fusion operation is performed. Using the aforementioned hand posture data, the handheld device posture data is increased. Using the handheld device posture data, the hand posture data can be increased, or Using the aforementioned hand posture data and the aforementioned handheld device posture data, combined posture data is generated. The method described in item 3, including the method described in item 3. (Item 8) Using the hand posture sensor to capture the hand posture data is Identifying multiple feature points associated with the aforementioned hand, The position of the hand is determined based on the aforementioned multiple feature points. The method described in item 1, including the method described in item 1. (Item 9) It is a system, A hand posture sensor configured to capture hand posture data, including the position of the hand within a reference frame, A handheld device attitude sensor configured to capture handheld device attitude data, including the position of the handheld device within the reference frame, One or more processors that communicate with the hand posture sensor and the handheld device posture sensor, wherein the one or more processors Determining the posture difference between the hand posture data and the handheld device posture data, wherein the posture difference includes the positional difference between the hand position and the handheld device position. Based on the aforementioned difference in posture, a fusion movement is performed. One or more processors configured to perform operations including A system equipped with these features. (Item 10) The system according to item 9, wherein the positional difference is the distance between the position of the hand and the position of the handheld device. (Item 11) The system according to item 10, further comprising determining whether the distance is above a distance threshold, below a distance threshold, or equal to a distance threshold. (Item 12) If it is determined that the distance exceeds the distance threshold, the fusion operation is performed. Determining that the handheld device is not being held by the hand. The system described in item 11, including the system described in item 11. (Item 13) If it is determined that the distance is less than or equal to the distance threshold, the fusion operation is performed. Determining that the handheld device is being held by the hand. The system described in item 11, including the system described in item 11. (Item 14) If it is determined that the distance is less than the distance threshold, the fusion operation is performed. Suppressing the aforementioned hand posture data, or To suppress the posture data of the aforementioned handheld device. The system described in item 11, including the system described in item 11. (Item 15) If it is determined that the distance exceeds the distance threshold, the fusion operation is performed. Suppressing the aforementioned hand posture data, or To suppress the posture data of the aforementioned handheld device. The system described in item 11, including the system described in item 11. (Item 16) If it is determined that the distance is less than or equal to the distance threshold, the fusion operation is performed. Using the aforementioned hand posture data, the handheld device posture data is increased. Using the handheld device posture data, the hand posture data can be increased, or Using the aforementioned hand posture data and the aforementioned handheld device posture data, combined posture data is generated. The system described in item 11, including the system described in item 11. (Item 17) Using the hand posture sensor to capture the hand posture data is Identifying multiple feature points associated with the aforementioned hand, The position of the hand is determined based on the aforementioned multiple feature points. The system described in item 9, including the system described in item 9. (Item 18) A non-transient computer-readable medium comprising instructions, which, when executed by one or more processors, cause the one or more processors to perform an operation to fuse hand pose data and handheld device pose data, the operation being Using a hand posture sensor, capture the hand posture data, including the position of the hand within a reference frame, Using a handheld device attitude sensor, the handheld device is captured, including its position within the reference frame. Determining the posture difference between the hand posture data and the handheld device posture data, wherein the posture difference includes the positional difference between the hand position and the handheld device position. Based on the aforementioned difference in posture, a fusion movement is performed. Non-transient computer-readable media, including [specific examples of such media]. (Item 19) The positional difference is the distance between the position of the hand and the position of the handheld device, as described in item 18 for a non-transient computer-readable medium. (Item 20) A non-transient computer-readable medium as described in item 19, further comprising determining whether the distance is above or below a distance threshold. [Brief explanation of the drawing]

[0064] The accompanying drawings, included to provide a further understanding of this disclosure, are incorporated herein and form part thereof, illustrating embodiments of this disclosure and serving to illustrate the principles of this disclosure together with a detailed description. No attempt is made to provide structural details of this disclosure in more detail than may be necessary with respect to a basic understanding of this disclosure and the various ways in which it may be put into practice.

[0065] [Figure 1A] Figures 1A and 1B illustrate examples of the orientations of various components of a wearable system. [Figure 1B] Figures 1A and 1B illustrate examples of the orientations of various components of a wearable system.

[0066] [Figure 2] Figures 2A and 2B illustrate an embodiment for determining the pose difference between a handheld device and the user's hand.

[0067] [Figure 3] Figures 3A-3C illustrate various embodiments for determining whether a handheld device is being held in the user's hand based on the determined posture difference.

[0068] [Figure 4] Figures 4A-4C illustrate various embodiments for determining whether a handheld device is being held in the user's hand based on the determined posture difference.

[0069] [Figure 5] Figure 5 illustrates various possible components for wearable AR / VR / MR systems.

[0070] [Figure 6] Figure 6 illustrates an embodiment of an electromagnetic tracking system that may be employed in wearable AR / VR / MR systems.

[0071] [Figure 7] Figure 7 illustrates an embodiment of how a visual tracking system and / or an electromagnetic tracking system may be incorporated into a wearable AR / VR / MR system.

[0072] [Figure 8A] Figures 8A–8E illustrate various exemplary configurations of wearable AR / VR / MR systems for tracking wearable devices, handheld devices, and / or user hand postures. [Figure 8B] Figures 8A–8E illustrate various exemplary configurations of wearable AR / VR / MR systems for tracking wearable devices, handheld devices, and / or user hand postures. [Figure 8C] Figures 8A–8E illustrate various exemplary configurations of wearable AR / VR / MR systems for tracking wearable devices, handheld devices, and / or user hand postures. [Figure 8D] Figures 8A–8E illustrate various exemplary configurations of wearable AR / VR / MR systems for tracking wearable devices, handheld devices, and / or user hand postures. [Figure 8E] Figures 8A–8E illustrate various exemplary configurations of wearable AR / VR / MR systems for tracking wearable devices, handheld devices, and / or user hand postures.

[0073] [Figure 9] Figure 9 illustrates an example of a set of hand feature points that can be identified based on captured images or videos of a user's hand.

[0074] [Figure 10] Figure 10 illustrates an example of a set of handheld device feature points that can be identified based on a determined orientation of the handheld device and a known 3D model of the handheld device.

[0075] [Figure 11] Figure 11 illustrates an example of a set of hand feature points and a set of handheld device feature points that can be identified.

[0076] [Figure 12A] Figures 12A-12F illustrate methods for implementing hand and totem input fusion for wearable systems. [Figure 12B] Figures 12A-12F illustrate methods for implementing hand and totem input fusion for wearable systems. [Figure 12C] Figures 12A-12F illustrate methods for implementing hand and totem input fusion for wearable systems. [Figure 12D] Figures 12A-12F illustrate methods for implementing hand and totem input fusion for wearable systems. [Figure 12E] Figures 12A-12F illustrate methods for implementing hand and totem input fusion for wearable systems. [Figure 12F] Figures 12A-12F illustrate methods for implementing hand and totem input fusion for wearable systems.

[0077] [Figure 13] Figure 13 illustrates the computer system. [Modes for carrying out the invention]

[0078] Detailed description of specific embodiments Wearable systems can present interactive augmented reality (AR), virtual reality (VR), and / or mixed reality (MR) environments that may contain data elements that can be interacted with by the user through various inputs. Modern computing systems are typically engineered to generate a given output based on a single direct input in data-rich and dynamic interactive environments such as AR / VR / MR environments (for example, a keyboard can relay text input such as receiving it from the user's finger strokes, a speech recognition application can create an executable data string based on the user's voice as direct input, and a computer mouse can guide the cursor in response to the user's direct operation, etc.), but a high degree of specificity may be desirable to perform a particular task.

[0079] Otherwise, in the absence of precise input, computing systems may suffer from high error rates and produce incorrect computer operations that should be performed. For example, when a user intends to move an object in 3D space using a touchpad, the computing system may be unable to correctly interpret the move command if the user has not specified a destination or an object using the touchpad. In another embodiment, using a virtual keyboard as the sole input mode to enter a string of text (e.g., using a user input device or by gestures) can be slow and physically tiring because it requires long-term, fine motor control to type keys that the virtual keyboard renders on an airy or physical surface (e.g., a desk).

[0080] In many instances, specific sensor inputs, used at any given time, can change dynamically as the user interacts with the 3D environment. Input modes may be dynamically added (or "fused" as further described herein) when the device determines that an input mode provides additional information and assists in targeting a virtual object, and input modes may be dynamically removed when that input mode no longer provides relevant information. For example, a wearable system may determine that the user's head pose and gaze direction are directed towards a target object. The device can use these two input modes to assist in selecting the target object. If the device determines that the user is also pointing a totem at the target object, the device can dynamically add a totem input to the head pose and gaze direction inputs, which can provide further certainty that the user intends to select the target object. This can be described as the totem input "converging" with the head pose and gaze direction inputs. Continuing with this embodiment, if the user moves their eyes away from the target object so that their gaze is no longer directed at the target object, the device may discontinue using the eye-gaze input while continuing to use the totem input and head pose input. In this case, it can be said that the eye-gaze input "diverges" from the totem input and head pose input.

[0081] While many techniques have been developed to determine convergence and divergence events between several input modes, one difficulty may arise in detecting convergence between totem input (e.g., movement of a handheld device) and gesture input (e.g., movement of the user's hand) when the totem is near the user's hand but not necessarily being held by the user. For example, the user may place the totem on a surface near their hand, put it in a pocket on their shirt or trousers, or move the totem from one hand to the other. In many applications, knowing whether the totem is actually in the user's hand and which hand it is in (or whether the totem is being held by both hands) can affect the virtual objects the user interacts with. For example, in a game where multiple users are playing with a virtual ball and a virtual bat, the user hitting the ball may be provided with a totem representing the virtual bat. Whether the virtual bat should appear to be held by the user's virtual hand would depend on whether the totem is actually in the user's hand. In another embodiment, the decision of whether or not the totem is in the user's hand may influence how the pose data regarding the totem and the user's hand is used by the wearable system. For example, if the totem is in the user's hand, the pose data regarding the hand may be used to enhance the pose data regarding the totem (e.g., to increase its accuracy), and vice versa.

[0082] In some instances, when the totem is in the user's hand, it may be advantageous to suppress pose data about the user's hand and instead rely substantially on pose data about the totem. For example, an electromagnetic tracking system employed to determine pose data about the totem may yield a more accurate estimate than a visual tracking system employed to determine pose data about the user's hand. Therefore, any attempt to combine or amplify pose data about the totem using pose data about the user's hand may decrease its accuracy.

[0083] Accurate pose information is crucial for many tasks performed by wearable systems. For example, detecting head pose can facilitate the display system's rendering of virtual objects so that they appear to occupy space in the real world in a manner that makes sense to the user. In addition, detecting the pose of real objects, such as handheld devices (which may also be referred herein to as totems, portable input devices, or controllers), haptic devices, or other actual physical objects, associated with the user's head or the wearable system, may also facilitate the display system in the step of presenting display information to the user and enabling the user to efficiently interact with certain aspects of the AR / VR / MR system. At least with respect to AR applications, the placement of virtual objects in spatial relationships with physical objects (e.g., presented to appear spatially close to physical objects in two or three dimensions) can be a matter of considerable importance.

[0084] For example, head movement can significantly complicate the placement of virtual objects within the view of the surrounding environment. This is true whether the view is captured as an image of the surrounding environment and then projected or displayed to the end user, or whether the end user directly perceives the view of the surrounding environment. For instance, head movement is likely to change the end user's field of view, which would likely require updates to the locations where various virtual objects appear within the end user's field of view.

[0085] In addition, head movement can occur within a wide range of speeds and velocities. Head movement speed can fluctuate not only between different head movements, but also within the range of a single head movement, or even across it. For example, head movement speed may initially increase from the starting point (e.g., linearly or non-linearly), decrease as it approaches the endpoint, and reach its maximum speed somewhere between the start and end points of the head movement. Rapid head movement may further exceed the capabilities of certain display or projection techniques for rendering images that appear to the end user as uniform and / or smooth motion.

[0086] For the purpose of accurate pose estimation, current approaches to fusing totem and hand gesture inputs have not been implemented at satisfactory speed or accuracy standards. Therefore, in the context of AR / VR / MR devices, there is a need for better systems, methods, and other techniques. Various embodiments will be described in the following description. For explanatory purposes, specific configurations and details are described to provide a thorough understanding of the embodiments. However, it will also be apparent to those skilled in the art that embodiments can be practiced without specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the embodiments described.

[0087] As used herein, the term "pose" may refer to the position and / or orientation of a component, but does not necessarily imply both position and orientation unless otherwise indicated. For example, "determining the pose" of a component as used herein may generally refer to determining the position of the component and / or determining the orientation of the component. As another example, "determining the pose" of a component may generally refer to determining at least one of the position or orientation of the component. As another example, "determining the pose, including the position" of a component may generally refer to determining the position of the component, optionally, the orientation. As another example, "determining the pose, including the orientation" of a component may generally refer to determining the orientation of the component, optionally, the position.

[0088] Figures 1A and 1B illustrate examples of the poses of various components of a wearable system, including wearable device 102, handheld device 104, and user's hand 106. Referring to FIG. 1A, wearable device WP Y WP Z WP ) and an orientation (a "wearable orientation") defined as (X WO Y WO Z WO ) with respect to the reference frame. The position of wearable device 102 may be represented, among other possibilities, by X, Y, and Z Cartesian values, or longitude, latitude, and altitude values. The orientation of wearable device 102 may be represented, among other possibilities, by pitch angle, yaw angle, and roll angle values.

[0089] Similarly, handheld device 104 has a position (a "handheld position") defined as (X DP Y DP Z DP ) with respect to the reference frame and an orientation defined as (X DO YDO , Z DO The handheld device 104 may have an orientation ("handheld orientation") defined as (X, Y, and Z Cartesian values, or longitude, latitude, and altitude values, among other possibilities. The orientation of the handheld device 104 may be expressed as pitch angle, yaw angle, and roll angle values, among other possibilities. Similarly, the hand 106 may have an orientation (X) relative to the reference frame. HP , Y HP , Z HP The position ("hand position") is defined as (X) relative to the reference frame. HO , Y HO , Z HO The hand may have an orientation ("hand orientation") defined as ). The position of the hand 106 may be expressed in terms of X, Y, and Z Cartesian values, or in terms of longitude, latitude, and altitude values, among other possibilities. The hand orientation 106 may be expressed in terms of pitch angle, yaw angle, and roll angle values, among other possibilities.

[0090] In some embodiments, the reference frame for each position and orientation may be a global reference frame. In some embodiments, the position and orientation of the wearable device 102 may be used as a reference frame such that the position of the wearable device 102 may be set as (0, 0, 0) and the orientation of the wearable device 102 may be set as (0°, 0°, 0°). Thus, the position and orientation of the handheld device 104 and the hand 106 may be determined relative to the position and orientation of the wearable device 102.

[0091] Figure 1B shows an embodiment of a user of a wearable system in which the handheld device 104 is held in the hand and the positions of the handheld device 104 and the hand 106 are brought close to each other. Thus, in some embodiments of the present disclosure, the similarity (e.g., proximity) between the positions of the handheld device 104 and the hand 106 may indicate that the handheld device 104 is in the user's hand. Furthermore, since the user's hand is generally oriented in a particular direction when engaged with the handheld device 104, the similarity between the orientations of the handheld device 104 and the hand 106 may also indicate that the handheld device 104 is in the user's hand.

[0092] The combination of the positional and orientation similarity (or, according to the same logic, positional and orientation differences or differences) between the handheld device 104 and the hand 106 and their relationship to the wearable device 102 can inform the wearable system whether the user is holding the handheld device 104, which may influence decisions regarding input mode selection and handling.

[0093] Figures 2A and 2B illustrate embodiments for determining the orientation difference between the handheld device 104 and the hand 106. The orientation difference may be calculated as a combination of a position difference 202, shown with reference to Figure 2A, and an orientation difference 204, shown with reference to Figure 2B. In some embodiments, the position difference 202 is calculated as the distance between the positions of the handheld device 104 and the hand 106, and therefore may be expressed in any unit of length, such as feet or meters. In some embodiments, the orientation difference 204 is calculated as the angle between the orientations of the handheld device 104 and the hand 106, and therefore may be expressed in any unit for measuring angles, such as radians or degrees.

[0094] Figures 3A-3C illustrate various embodiments for determining whether the handheld device 104 is being held by the hand 106 based on the determined orientation difference. In some embodiments, the distance threshold 302 may be used as the maximum distance between the positions of the handheld device 104 and the hand 106 in which the handheld device 104 is expected to be being held by the hand 106. The distance threshold 302 may correspond to the distance between one or more locations along the outer surface of a volume (e.g., a sphere, ellipsoid, or equivalent) and locations within the volume that could coincide with the position of the handheld device 104 or the position of the hand 106. For example, the distance threshold 302 may be used as a radius with respect to a sphere that can be centered at either of the positions. Referring to Figure 3A, a sphere with a radius equal to the distance threshold 302 is centered at the position of the handheld device 104. Since the position of the hand 106 is outside the sphere, it is determined that the handheld device 104 is not being held by the hand 106.

[0095] Referring to Figure 3B, a sphere with a radius equal to the distance threshold 302 is centered at the position of hand 106. Since the position of the handheld device 104 is outside the sphere, it is determined that the handheld device 104 is not being held by hand 106. Referring to Figure 3C, the sphere is centered at the position of handheld device 104, where the position of hand 106 is inside the sphere, and therefore it is determined that the handheld device 104 is being held by hand 106.

[0096] Figures 4A–4C illustrate various embodiments for determining whether the handheld device 104 is being held by the hand 106 based on determined pose differences. Referring to Figure 4A, the poses of the wearable device 102, the handheld device 104, and the hand 106 may be determined, and a sphere with a radius equal to a distance threshold 302 is centered at the position of the hand 106. In Figure 4A, the position of the handheld device 104 is within the sphere, but in Figure 4B, the sphere is deformed into an ellipsoid based on the relationship between the positions of the wearable device 102 and the hand 106. In the illustrated embodiments, the ellipsoid extends further toward the position of the wearable device 102 than away from the position of the wearable device 102, taking into account the tendency for the handheld device 104 to be further away from the user (e.g., on a table or desk) when not being held by the user. When the position of the handheld device 104 is compared with the ellipsoid, it can be determined that the position is outside the ellipsoid and therefore the handheld device 104 is not being held by the hand 106.

[0097] In some embodiments, the first volume (e.g., a sphere, ellipsoid, or equivalent) may be defined relative to the hand 106 (e.g., aligned to the position of the hand 106), and the second volume may be defined relative to the wearable device 102 (e.g., aligned to the position of the wearable device 102) or one or more parts of the user other than the hand 106 (e.g., aligned to an estimated position of the user's chest or shoulder). In some of these embodiments, it may be determined that the handheld device 104 is being held by the hand 106 once it is determined that the position of the handheld device 104 lies within both the first and second volumes (e.g., within the overlapping region between the first and second volumes in three-dimensional space). However, in at least some of these embodiments, if the position of the handheld device 104 is determined to be outside one or both of the first and second volumes (for example, outside the overlapping region between the first and second volumes in three-dimensional space), then it may be determined that the handheld device 104 is not being held by the hand 106. In such embodiments, the position, orientation, size, and / or shape of the overlapping three-dimensional region between the first and second volumes may behave similarly to that of the ellipsoidal volume described above with reference to Figure 4B.

[0098] In some embodiments, the second volume may correspond to a range of locations in three-dimensional space that are within the user's reach. That is, in some embodiments, one or more of the outer surfaces of the second volume may be defined, at least in part, based on the estimated or assumed length, span, or maximum reach of one or both of the user's arms. In these embodiments, one or more such estimated or assumed parameters may be determined at least in part based on image data obtained by one or more sensors of the wearable device 102 (e.g., one or more outward-facing cameras, one or more electromagnetic (EM) sensors, one or more depth cameras or sensors, etc.), data indicating user input (e.g., user input defining values ​​for one or more parameters) provided through the handheld device 104 and / or through one or more user interface components of the wearable device 102, data obtained or output by one or more sensors of the handheld device 104 (e.g., one or more cameras, one or more inertial measurement units (IMUs), or equivalents), data received over one or more communication networks, and equivalents.

[0099] In some implementations, the first volume does not have to be defined relative to the hand 106, but instead is defined relative to the handheld device 104 (e.g., aligned to the position of the handheld device 104). In such implementations, the handheld device 104 may be determined to be held by the hand 106 if the position of the hand 106 is determined to be within both the first and second volumes (e.g., within the overlapping region between the first and second volumes in three-dimensional space), and the handheld device 104 may be determined not to be held by the hand 106 if the position of the hand 106 is determined to be outside one or both of the first and second volumes (e.g., outside the overlapping region between the first and second volumes in three-dimensional space). In some embodiments, the position of the handheld device 104 or the hand 106 may be compared to the overlapping region between two or more (e.g., three, four, five, etc.) different volumes in three-dimensional space. Other configurations are also possible.

[0100] Referring to Figure 4C, in some embodiments, the distance between the position of the wearable device 102 and the positions of the handheld device 104 and the hand 106 may be determined and used to assist in determining whether the handheld device 104 is being held by the hand 106. For example, in some embodiments, the two distances may be compared to determine whether the handheld device 104 or the hand 106 is further away from the wearable device 102. In some instances, it may be determined that the handheld device 104 is not being held by the hand 106 only if the handheld device 104 is further away from the wearable device 102 than the hand 106 (i.e., the distance between the positions of the wearable device 102 and the hand 106 is less than the distance between the positions of the wearable device 102 and the handheld device 104). In some embodiments, a second distance threshold may be used to determine whether the handheld device 104 is at least the second distance threshold, further away from the wearable device 102 than the hand 106 (i.e., the distance between the positions of the wearable device 102 and the hand 106 is at least the second distance threshold, less than the distance between the positions of the wearable device 102 and the handheld device 104).

[0101] Figure 5 illustrates possible components for various wearable AR / VR / MR systems 500. In the illustrated embodiment, a wearable system user 560 is depicted wearing a head-mounted component 558, which features a frame 564 structure coupled to a display system 562 positioned in front of the user's eyes. A speaker 566 is coupled to the frame 564 in the depicted configuration and positioned adjacent to the user's ear canal (in one embodiment, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / adjustable sound control). The display system 562 is operably coupled (as shown by 568) to the local processing and data module 570 by wired conductors or wireless connectivity, and may be mounted in various configurations, such as being fixedly attached to the frame 564, fixedly attached to a helmet or hat, detachably attached to the user 560's torso in a backpack configuration, or detachably attached to the user 560's waist in a belt-connected configuration.

[0102] The local processing and data module 570 may include a power-efficient processor or controller and digital memory such as flash memory, both of which may be used to assist in processing, caching, and storing data acquired from sensors, such as (1) image acquisition devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes, which can be operably coupled to the frame 564, and / or (2) possibly, the remote processing module 572 and / or remote data repository 574, for passage to the display system 562 after processing or reading.

[0103] The local processing and data module 570 may be operably coupled to the remote processing module 572 and the remote data repository 574 (as indicated by 576, 578) via one or more wired or wireless link, etc., so that these remote modules 572, 574 are operably coupled to each other and available as resources to the local processing and data module 570. In one embodiment, the remote processing module 572 may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. In one embodiment, the remote data repository 574 may include relatively large-capacity digital data storage equipment, which may be available through the internet or other networking configurations in a “cloud” resource configuration. In one embodiment, all data is stored in the local processing and data module, all calculations are performed there, and fully autonomous use from any remote module is enabled.

[0104] Figure 6 illustrates an embodiment of an electromagnetic tracking system 600 that may be employed by the wearable system 500. The electromagnetic tracking system 600 may be similar to those developed by organizations such as the Biosense (RTM) division of Johnson & Johnson Corporation, Polhemus (RTM), Inc. (Colchester, Vermont) and manufactured by Sixense (RTM) Entertainment, Inc. (Los Gatos, California) and other imitators. In one or more embodiments, the electromagnetic tracking system 600 includes an electromagnetic field emitter 602 configured to emit a known magnetic field. As shown in Figure 6, the electromagnetic field emitter 602 may be coupled to a power source 610 (e.g., an electric current, a battery, etc.) to provide power to the electromagnetic field emitter 602.

[0105] In one or more embodiments, the electromagnetic field emitter 602 includes several coils (e.g., at least three coils positioned perpendicular to each other and producing fields in the X, Y, and Z directions) that generate a magnetic field. These magnetic fields are used to establish a coordinate space that allows the system to map the position of sensors relative to known magnetic fields and to help determine the position and / or orientation of the sensors. In one or more embodiments, electromagnetic sensors 604A, 604B, etc., may be attached to one or more real objects. Electromagnetic sensors 604 may include smaller coils that can be induced through an electromagnetic field from which an electric current is emitted.

[0106] Generally, the components of the electromagnetic field sensor 604 may include small coils or loops, such as a set of three coils oriented in different ways (i.e., oriented orthogonally to each other, etc.), coupled together in a small structure such as a cube or other container, which are positioned / oriented to capture the magnetic flux incident from the magnetic field emitted by the electromagnetic field emitter 602, and the relative position and orientation of the sensor with respect to the emitter may be calculated by comparing the currents induced through these coils and determining the relative position and orientation of the coils with respect to each other.

[0107] As will be further illustrated with reference to Figure 7, one or more motion sensors, such as an IMU, may be operably coupled to each of the electromagnetic field emitter 602 and electromagnetic field sensor 604 to detect the position and orientation of each component relative to each other and / or to the coordinate system. In one or more embodiments, multiple sensors (possibly including an IMU) may be used in conjunction with the electromagnetic field emitter 602 and electromagnetic field sensor 604 to detect the position and orientation of each component. In some instances, the electromagnetic tracking system may provide position in three directions (i.e., X, Y, and Z directions) and further in two or three orientation angles. In some embodiments, IMU measurements may be compared with coil measurements to determine the position and orientation of the sensor. In one or more embodiments, both EM data and motion data may be combined, along with various other sources of data such as cameras, depth sensors, and other sensors, to determine the position and orientation. This information may be transmitted to the processing unit 606 (e.g., via wireless communication, Bluetooth®, etc.). In some embodiments, the posture may be reported at a relatively high refresh rate in a conventional system.

[0108] Conventionally, electromagnetic emitters are coupled to relatively stable, large objects such as tables, operating tables, walls, or ceilings, while one or more sensors are coupled to smaller objects such as medical devices, handheld game components, or equivalents. Alternatively, various features of electromagnetic tracking systems may be employed to produce configurations in which changes or differences in position and / or orientation between two objects moving in space relative to a more stable global coordinate system can be tracked. For example, a variation of the electromagnetic tracking system can be used to track the difference in position and orientation between a head-mounted component and a handheld component, while the head orientation relative to a global coordinate system (e.g., the user's local room environment) is determined separately, for example, by simultaneous localization and mapping (SLAM) techniques using an outward-facing camera coupled to the head-mounted component of the system, as shown in Figure 7.

[0109] The processing unit 606 may control the electromagnetic field emitter 602 and may also capture data from various electromagnetic field sensors 604. It should be understood that the various components of this system may be coupled to one another through any electromechanical or wireless means. The processing unit 606 may also include data on known magnetic fields and the coordinate space associated with those magnetic fields. This information is then used to detect the position and orientation of the sensors in relation to the coordinate space corresponding to the known electromagnetic field.

[0110] One advantage of electromagnetic tracking systems is that they produce highly accurate tracking results with minimal latency and high resolution. In addition, electromagnetic tracking systems do not necessarily rely on optical trackers, and sensors / objects not within the user's line of sight can be easily tracked. It should be understood that the strength of the electromagnetic field decreases as a cubic function of the distance from the coil transmitter (e.g., electromagnetic field emitter 602). Therefore, the processing unit 606 may be configured to execute a function, such as an algorithm that predicts distance based on the measured strength, to determine the position and orientation of the sensor / object at a variable distance from the electromagnetic field emitter 602.

[0111] Assuming a rapid decrease in electromagnetic field strength as movement moves further away from the electromagnetic emitter, the best results in terms of accuracy, efficiency, and low latency can be achieved at closer distances. In a typical electromagnetic tracking system, the electromagnetic field emitter has a sensor powered by an electric current (e.g., a plug-in power source) and located within a 20-foot radius of the electromagnetic field emitter. A shorter radius between the sensor and the field emitter may be more desirable in many applications, including AR applications.

[0112] Figure 7 illustrates an embodiment of how a visual tracking system and / or an electromagnetic tracking system may be incorporated into a wearable AR / VR / MR system having a wearable device 702 (e.g., a headset) and a handheld device 704 (e.g., a controller). In some embodiments, the handheld device 704 may be a handheld controller that allows the user to provide input to the wearable system. For example, the handheld device 704 may be a totem for use in a game scenario. The handheld device 704 may also be a tactile device and may include one or more tactile surfaces that utilize various sensor types. During operation of the wearable system, the user may hold the handheld device 704 in their left or right hand by actively grasping it and / or by securing a mounting mechanism to the user's hand (e.g., by wrapping a strap around it).

[0113] The handheld device 704 may include one or more base points (referred to herein as handheld base points 722) positioned along one or more external surfaces of the handheld device 704 such that the base points may be within the field of view of an imaging device outside the handheld device 704. The handheld base points 722 may have known relationships with one another so that the imaging device can determine its position and / or orientation relative to the handheld device 704 by capturing an image of one or more of the handheld base points 722. The handheld base points 722 may be dynamic, static, electrically powered, or unpowered, and in some embodiments may be distinguishable from one another. For example, the first base point may be a light-emitting diode (LED) having a first wavelength, and the second base point may be an LED having a second wavelength. Alternatively, or in addition, different base points may have different brightness levels and / or be pulsed at different frequencies (for example, the first base point may be pulsed at 100 Hz and the second base point at 150 Hz).

[0114] The handheld device 704 may include one or more imaging devices (hereinafter referred to as handheld imaging devices 726) positioned such that when the handheld device 704 is held by the user, several features around the wearable device 702 and / or the handheld device 704 are within the field of view of the imaging devices. For example, a front handheld imaging device 726A may be positioned so that its field of view is oriented toward one or more features around the handheld device 704 so that it is away from the user, and a rear handheld imaging device 726B may be positioned so that its field of view is oriented toward the wearable device 702. The handheld imaging devices 726 may include one or more front-facing imaging devices and / or one or more rear-facing imaging devices to create a desired cumulative field of view. In some embodiments, the handheld imaging devices 726 may be optical devices such as cameras, which may capture still or moving images.

[0115] The handheld device 704 may include an IMU (hereinafter referred to as the handheld IMU 724) which is firmly fixed within the handheld device 704 so that the rotation and linear motion of the handheld device 704 are similarly covered by the handheld IMU 724. In some instances, the handheld IMU 724 may include one or more accelerometers (e.g., three), one or more gyroscopes (e.g., three), one or more magnetometers (e.g., three), and / or digital signal processing hardware and software for converting raw measurements into processed data. For example, the handheld IMU 724 may include an accelerometer, a gyroscope, and a magnetometer for each of the three axes. For each axis, the handheld IMU 724 may output one or more of the following: linear position, linear velocity, linear acceleration, rotational position, rotational velocity, and / or rotational acceleration. Alternatively, or in addition, the handheld IMU724 may output raw data from which any of the processed data in the forms described above may be calculated.

[0116] The handheld device 704 may include a rechargeable and / or replaceable battery 728 or other power source to power the handheld base point 722, the handheld imaging device 726, the handheld IMU 724, the electromagnetic field emitter 732, and any other components of the handheld device 704. Although not shown in Figure 7, the handheld device 704 may include a network for enabling wireless communication with the wearable device 702 and / or beltpack 740. For example, using the handheld imaging device 726 and the handheld IMU 724, the handheld device 704 may transmit raw or processed data to the wearable device 702 and / or beltpack 740 in response to data detection or capture.

[0117] The wearable device 702 may include one or more base points (referred to herein as wearable base points 706) positioned along one or more external surfaces of the wearable device 702 such that the base points may be within the field of view of the rear handheld imaging device 726B. The wearable base points 706 may have known relationships with one another so that the imaging device can determine its position and / or orientation relative to the wearable device 702 by capturing an image of one or more of the wearable base points 706. The wearable base points 706 may be dynamic, static, electrically powered, or unpowered, and in some embodiments may be distinguishable from one another. For example, the first base point may be an LED having a first wavelength, and the second base point may be an LED having a second wavelength. Alternatively, or in addition, the different base points may have different brightness and / or be pulsed at different frequencies.

[0118] The wearable device 702 may include one or more imaging devices (hereinafter referred to as wearable imaging devices 710) positioned such that the handheld device 704 (specifically, the handheld base point 722) is within the field of view of the imaging device when the handheld device 704 is held by the user. For example, one or more wearable imaging devices 710 may be positioned facing forward on the wearable device 702 above, below, and / or to the optical see-through component of the wearable device 702. In one embodiment, two wearable imaging devices 710 may be positioned on the opposite side of the optical see-through component of the wearable device 702. In some embodiments, the wearable imaging devices 710 may be optical devices such as cameras, which may capture still or moving images.

[0119] The wearable device 702 may include an IMU (hereinafter referred to as the wearable IMU 708) which is firmly fixed within the wearable device 702 so that the rotation and linear motion of the wearable device 702 are similarly covered by the wearable IMU 708. In some instances, the wearable IMU 708 may include one or more accelerometers (e.g., three), one or more gyroscopes (e.g., three), one or more magnetometers (e.g., three), and / or digital signal processing hardware and software for converting raw measurements into processed data. For example, the wearable IMU 708 may include an accelerometer, a gyroscope, and a magnetometer for each of the three axes. For each axis, the wearable IMU 708 may output one or more of the following: linear position, linear velocity, linear acceleration, rotational position, rotational velocity, and / or rotational acceleration. Alternatively, or in addition, the wearable IMU708 may output raw data from which any of the processed data in the forms described above may be calculated.

[0120] In some embodiments, the wearable system may include a belt pack 740, which may include a computing device (e.g., one or more processors and associated memory) for performing the localization of the handheld device 704 relative to the wearable device 702. Alternatively, or in addition, the computing device may reside in the wearable device 702 itself or further in the handheld device 704. The computing device may receive raw or processed data from each of the wearable IMU 708, wearable imaging device 710, handheld IMU 724, and handheld imaging device 726 (via wired and / or wireless connections) and calculate the geospatial position of the handheld device 704 (relative to the geospatial position of the wearable device 702) and the orientation of the handheld device 704 (relative to the orientation of the wearable device 702). The computing device may, in turn, include a mapping database 742 (e.g., a passable world model, coordinate space, etc.), detect orientation, determine the coordinates of real and virtual objects, and in one or more embodiments, connect to cloud resources and the passable world model. In some embodiments, images captured using a wearable imaging device 710 and / or a handheld imaging device 726 may be used to construct the passable world model. For example, features may be detected in the captured images, and the collected data (e.g., sparse points) may otherwise be used to construct the passable world model or environment map.

[0121] The wearable system may include an electromagnetic field emitter 732 incorporated as part of a handheld device 704. In some embodiments, the electromagnetic field emitter 732 may be incorporated as part of a beltpack 470 or a wearable device 702. In some embodiments, data from an IMU or other motion sensor may help determine the position and / or orientation of the electromagnetic field emitter 732. This may be beneficial if both the electromagnetic field emitter 732 and the electromagnetic field sensor 734 are mobile. Placing the electromagnetic field emitter 732 in a handheld controller rather than a beltpack ensures that the electromagnetic field emitter does not compete for resources in the beltpack, but instead uses the battery 728 in the handheld device 704, as shown in the embodiment of Figure 7.

[0122] In some embodiments, the electromagnetic sensor 734 may be positioned relative to the wearable device 702, such as being mounted in one or more locations on the wearable device 702 and / or in one or more locations on the belt pack 740. Since the electromagnetic sensor 734 can be engineered to be small (and therefore, in some cases, not very sensitive), having multiple sensors can improve efficiency and accuracy.

[0123] In many instances, conventional electromagnetic emitters can be too bulky for wearable devices. Therefore, electromagnetic field emitters may be engineered to be smaller by using smaller coils compared to conventional systems. However, assuming that the strength of the electromagnetic field decreases as a cubic function of the distance from the field emitter, a shorter radius (e.g., about 3–3.5 feet) between the electromagnetic field sensor 734 and the electromagnetic field emitter 732 can reduce power consumption compared to conventional systems. This aspect may also be used to extend the life of the battery 728 that can power the handheld controller 704 and the electromagnetic field emitter 732. In some embodiments, this aspect may be used to reduce the size of the coil that generates the magnetic field in the electromagnetic field emitter 732. However, to obtain the same strength of the magnetic field, the power may need to be increased. This allows for a smaller electromagnetic field emitter 732 that can fit compactly into the handheld device 704.

[0124] When using electromagnetic tracking systems for AR devices, several other modifications may be made, which may benefit from more efficient attitude reporting rates than in other applications. For example, motion-based or IMU-based attitude tracking may be employed. In many cases, improved stability of the IMU can lead to improved efficiency in the attitude detection process. IMUs may be engineered to remain stable for up to 50–100 milliseconds. It should be understood that some embodiments may utilize an external attitude estimator module, which may allow attitude updates to be reported at a rate of 10–20 Hz (e.g., the IMU may drift over time). By keeping the IMU stable at a reasonable rate, the attitude update rate can be reduced to 10–20 Hz (compared to higher frequencies in conventional systems).

[0125] If the electromagnetic tracking system can be activated on a 10% duty cycle (e.g., pinging only for ground truth every 100 milliseconds), this would be an additional way to conserve power in a wearable system. This would mean the electromagnetic tracking system wakes up every 10 milliseconds to generate an attitude estimate. This directly leads to power consumption savings, which in turn can affect the size, battery life, and cost of the wearable system. In one or more embodiments, this duty cycle reduction can be strategically utilized by providing not just one, but two handheld devices (not shown). For example, a user might play a game that requires two totems, etc. Or, in a multi-user game, two users might each have their own totem / handheld controllers to play the game. When two controllers (e.g., symmetrical controllers for each hand) are used instead of one, the controllers may operate on an offset duty cycle. The same concept can also be applied to controllers used, for example, by two different users playing a multiplayer game.

[0126] Figures 8A–8D illustrate various exemplary configurations of a wearable AR / VR / MR system for tracking the posture of a wearable device 802, a handheld device 804, and / or a hand 830. Figure 8A illustrates an exemplary configuration of wearable system 800A, in which wearable device 802 includes one or more wearable imaging devices 810 having a field of view, at least partially and at least temporarily, including a handheld base point 822, and handheld device 804 includes one or more handheld base points 822. Wearable system 800 may include additional sensors mounted on handheld device 804, such as a handheld IMU 824. One advantage of such a configuration may be the simplicity and low power consumption of handheld device 804. Wearable system 800 may also include additional sensors mounted on wearable device 802, such as a wearable IMU 808.

[0127] Figure 8B illustrates an exemplary configuration of the wearable system 800B, in which the wearable device 802 includes one or more wearable base points 806, and the handheld device 804 includes one or more rear-facing handheld imaging devices 826 having a field of view that includes at least partially and at least temporarily the wearable base points 806. The wearable system 800B may also include additional sensors mounted on the handheld device 804, such as a handheld IMU 824. One advantage of such a configuration may be that the handheld device 804 has all the data necessary to perform its own localization relative to the wearable device 802, thereby reducing the processing load on the wearable device 802. The wearable system 800B may also include additional sensors mounted on the wearable device 802, such as a wearable IMU 808.

[0128] Figure 8C illustrates an exemplary configuration of the wearable system 800C, in which the handheld device 826 includes a front handheld imaging device 826A having a field of view that includes at least partially and at least temporarily one or more ambient features 844 while the handheld device 804 is held by the user, and a rear handheld imaging device 826B having a field of view that includes at least partially and at least temporarily one or more wearable base points 806. In the exemplary configuration, multiple wearable base points 822 are attached to the wearable device 802. The wearable system 800C may include additional sensors mounted on the handheld device 804, such as a handheld IMU 824. One advantage of such a configuration may be the increased accuracy provided by the multiple imaging devices. The wearable system 800C may also include additional sensors mounted on the wearable device 802, such as a wearable IMU 808.

[0129] Figure 8D illustrates an exemplary configuration of wearable system 800D, in which the wearable device 802 includes an electromagnetic sensor 834, and the handheld device 826 includes an electromagnetic field emitter 832, enabling the determination of the orientation of the wearable device and / or the handheld device 804. Figure 8E illustrates an exemplary configuration of wearable system 800E, in which the wearable device 802 includes one or more wearable imaging devices 810 that have a field of view, including a hand 830, at least partially and at least temporarily, and enable the determination of the orientation of the hand 830.

[0130] Figure 9 illustrates an embodiment of a set of hand feature points 904 that can be identified based on captured images or videos of a user's hand 902. The hand feature points 904 may be identified using various image analysis techniques, such as training a machine learning model (e.g., a neural network) using a set of labeled images. After training, the machine learning model may receive images or streams of images as input and produce hand feature points 904. In some embodiments, the hand feature points 904 may be used to determine the position of the hand 902. For example, the position of the hand 902 may be set as one of the hand feature points 904, or the hand feature points 904 may be used to calculate different positions along the hand, such as the center of mass.

[0131] Figure 10 illustrates an embodiment of a set of handheld device feature points 1004 that can be identified based on a determined orientation of the handheld device 1002 and a known 3D model of the handheld device 1002. For example, the determined position of the handheld device 1002 may be the position of the electromagnetic field emitter 1008 or emitter position 1006, from which the handheld device feature points 1004 can be determined. In some embodiments, the handheld device feature point 1004 may correspond to a location along the handheld device 1002 that is closest to its corresponding hand feature point, such as the hand feature point 904 shown in Figure 9.

[0132] Figure 11 illustrates an embodiment of an identifiable set of hand feature points 1102 and a set of handheld device feature points 1104. In some embodiments, a position difference 1106 may be calculated as the distance between two corresponding feature points (e.g., one feature point from hand feature points 1102 and one feature point from handheld device feature points 1104). In some embodiments, multiple position differences 1106 may be calculated between corresponding feature points, which may be combined into a total position difference. In some embodiments, the position difference 1106 may be calculated as the maximum, minimum, or average distance between corresponding feature points. Other possibilities are also considered.

[0133] Figures 12A-12F illustrate Method 1200 for implementing hand and totem input fusion for wearable systems such as AR / VR / MR systems. One or more steps of Method 1200 may be omitted during implementation of Method 1200, and the steps of Method 1200 do not need to be performed in the order shown. One or more steps of Method 1200 may be performed by one or more processors, such as those contained within the local processing and data module 570 or the teleprocessing module 572.

[0134] Referring to Figure 12A, in step 1202 of method 1200, hand posture data is captured. In some embodiments, the hand posture data includes the position of the hand within a reference frame. In some embodiments, the hand posture data includes the orientation of the hand within a reference frame. In some embodiments, the hand posture data is captured using a hand posture sensor. In some embodiments, the step of capturing hand posture data may include the step of capturing sensor data using a hand posture sensor and the step of generating hand posture data based on the captured sensor data. For example, referring to Figure 12B, in substep 1202-1, sensor data is captured using a hand posture sensor. In substep 1202-2, the position of the hand is determined based on the sensor data, for example, by processing and analyzing the sensor data. In substep 1202-3, the orientation of the hand is determined based on the sensor data, by processing and analyzing the sensor data.

[0135] Referring to Figure 12A, in step 1204, handheld device attitude data is captured. In some embodiments, the handheld device attitude data includes the position of the handheld device within a reference frame. In some embodiments, the handheld device attitude data includes the orientation of the handheld device within a reference frame. In some embodiments, the handheld device attitude data is captured using a handheld device attitude sensor. In some embodiments, the step of capturing handheld device attitude data may include the step of capturing sensor data using a handheld device attitude sensor, and the step of generating handheld device attitude data based on the captured sensor data. For example, referring to Figure 12C, in substep 1204-1, sensor data is captured using a handheld device attitude sensor. In substep 1204-2, the position of the handheld device is determined based on the sensor data by processing and analyzing the sensor data. In substep 1204-3, the orientation of the handheld device is determined based on the sensor data by processing and analyzing the sensor data.

[0136] In some embodiments, the hand posture sensor and the handheld device posture sensor are the same sensor. For example, an image acquisition device mounted on a headset may collect information about the posture of both devices and therefore may be both a hand posture sensor and a handheld device posture sensor. In some embodiments, the hand posture sensor and the handheld device posture sensor are different from each other. For example, the hand posture sensor may be an image acquisition device, and the handheld device posture sensor may be an electromagnetic transmitter and an electromagnetic receiver.

[0137] Referring to Figure 12A, step 1206 determines the posture difference between the hand posture data and the handheld device posture data. In some embodiments, the posture difference includes a positional difference between the positions of the hand and the handheld device. For example, referring to Figure 12D, substep 1206-1 determines the positional difference between the positions of the hand and the handheld device. The positional difference may be the distance between the two positions or some other distance metric based on the two positions. In some embodiments, the posture difference includes an orientation difference between the orientations of the hand and the handheld device. For example, referring to Figure 12D, substep 1206-2 determines the orientation difference between the orientations of the hand and the handheld device. The orientation difference may be an angular offset between the two orientations or some other measure with an angular offset based on the two orientations. In some embodiments, the angular offset is not limited to a single angle, and the angular offset may include more than one angle.

[0138] Referring to Figure 12A, step 1208 is performed on the basis of the fusion operation. In some embodiments, the fusion operation may also be based on whether the distance between the positions of the hand and the handheld device is above or below a distance threshold. For example, referring to Figure 12E, substep 1208-1 determines whether the distance is above or below a distance threshold. If it is determined that the distance is above the distance threshold, method 1200 proceeds to substep 1208-2. If it is determined that the distance is below the distance threshold, method 1200 proceeds to substep 1208-4. In substep 1208-2, it is determined that the handheld device is not held by the hand. In substep 1208-3, the hand pose data is suppressed so that the step of performing the fusion operation includes a step of suppressing the hand pose data. In some embodiments, the step of suppressing the hand pose data includes a step of not using and / or considering the hand pose data as a direct input to the processing module. In some embodiments, the step of suppressing hand posture data includes the step of not using and / or considering hand posture data with respect to any determination of the position and / or orientation of the handheld device.

[0139] In substep 1208-4, it is determined that the handheld device is held by a hand. In substep 1208-5, the hand pose data is used to augment the handheld device pose data such that the step of performing a fusion motion includes a step of augmenting the handheld device pose data using the hand pose data. In some embodiments, the step of augmenting the handheld device pose data using the hand pose data includes a step of using the hand pose data in any determination of the position and / or orientation of the handheld device.

[0140] As another embodiment, referring to Figure 12F, substeps 1208-1, 1208-2, and 1208-4 are performed in the same manner as described with reference to Figure 12E. In substep 1208-6, the handheld device pose data is suppressed such that the step of performing the fusion operation includes a step of suppressing the handheld device pose data. In some embodiments, the step of suppressing the handheld device pose data includes a step of not using and / or considering the handheld device pose data as a direct input to the processing module. In some embodiments, the step of suppressing the handheld device pose data includes a step of not using and / or considering the handheld device pose data with respect to any determination of hand position and / or orientation.

[0141] In substep 1208-7, handheld device posture data is used to augment hand posture data such that the step of performing a fusion operation includes a step of augmenting hand posture data using the handheld device posture data. In some embodiments, the step of augmenting hand posture data using handheld device posture data includes a step of using handheld device posture data in any determination of hand position and / or orientation.

[0142] Figure 13 illustrates a simplified computer system 1300 according to embodiments described herein. The computer system 1300 as illustrated in Figure 13 may be incorporated into the devices described herein. Figure 13 provides a schematic diagram of one embodiment of the computer system 1300 that can carry out some or all of the steps of the method provided by various embodiments. It should be noted that Figure 13 is intended only to provide a generalized illustration of various components, any or all of which may be used as appropriate. Figure 13 therefore illustrates, in a broad sense, how individual system elements may be implemented in a relatively separate or relatively more integrated manner.

[0143] The computer system 1300 is shown to include hardware elements that may, as appropriate, be electrically coupled via bus 1305 or otherwise communicate. The hardware elements may include one or more processors 1310, including, but not limited to, one or more general-purpose processors and / or one or more special-purpose processors, such as a digital signal processing chip, a graphics accelerator, and / or equivalent; one or more input devices 1315, which may, but not limited to, a mouse, a keyboard, a camera, and / or equivalent; and one or more output devices 1320, which may, but not limited to, a display device, a printer, and / or equivalent.

[0144] The computer system 1300 further includes and / or communicates with one or more non-transient storage devices 1325, which may include, but are not limited to, local and / or network-accessible storage devices, and / or, but are not limited to, solid-state storage devices such as disk drives, drive arrays, optical storage devices, random access memory ("RAM"), and / or read-only memory ("ROM"), which may be programmable, flash-updatable, and / or equivalent. Such storage devices may be configured to implement any suitable data store, including, but not limited to, various file systems, database structures, and / or equivalents.

[0145] The computer system 1300 may also include a communication subsystem 1319, which may include, but is not limited to, modems, network cards (wireless or wired), infrared communication devices, wireless communication devices, and / or chipsets, such as Bluetooth® devices, 802.11 devices, WiFi devices, WiMAX devices, cellular communication equipment, and / or equivalents. The communication subsystem 1319 may include one or more input and / or output communication interfaces, which may enable data to be exchanged with networks, such as those described below to give as one embodiment, i.e., networks of other computer systems, televisions, and / or any other devices described herein. Depending on the desired functionality and / or other implementation concerns, a portable electronic device or similar device may communicate images and / or other information via the communication subsystem 1319. In other embodiments, a portable electronic device, e.g., a first electronic device, may be incorporated into the computer system 1300, e.g., an electronic device, as an input device 1315. In some embodiments, the computer system 1300 may further include working memory 1335, which may include RAM or ROM devices as described above.

[0146] The computer system 1300 may also include software elements, shown as currently located in working memory 1335, including other code such as an operating system 1340, device drivers, executable libraries, and / or one or more application programs 1345, which may include computer programs provided by various embodiments and / or implement methods provided by other embodiments as described herein and / or be designed to constitute a system. Simply put, as an embodiment, one or more procedures described with respect to the methods discussed above may be implemented as code and / or instructions executable by a computer and / or a processor within a computer, and in some aspects such code and / or instructions may then be used to configure and / or adapt a general-purpose computer or other device to perform one or more operations in accordance with the methods described.

[0147] These instructions and / or sets of code may be stored on a non-transient computer-readable storage medium, such as the storage device 1325 described above. In some cases, the storage medium may be incorporated into a computer system, such as computer system 1300. In other embodiments, the storage medium may be a removable medium, such as a compact disk, separate from the computer system, and / or the storage medium may be provided in an installation package so that it can be used to program, configure, and / or adapt a general-purpose computer using the instructions / code stored thereon. These instructions may take the form of executable code that is executable by computer system 1300, and / or take the form of source and / or installable code that then takes the form of executable code, depending on compilation and / or installation on computer system 1300 using, for example, one of various generally available compilers, installation programs, compression / decompression utilities, etc.

[0148] It will be apparent to those skilled in the art that substantial variations can be fabricated according to specific requirements. For example, customized hardware may also be used, and / or certain elements may be implemented in hardware, software including portable software such as applets, or both. Furthermore, connectivity to other computing devices, such as network input / output devices, may also be employed.

[0149] As described above, in one aspect, some embodiments may employ a computer system such as computer system 1300 to carry out the methods according to various embodiments of the Art. According to one embodiment, some or all of the procedures of such a method are carried out by computer system 1300 in response to processor 1310 executing one or more instructions, or a sequence of instructions, and / or other code, such as application program 1345, which may be incorporated into operating system 1340. Such instructions may be read into working memory 1335 from another computer-readable medium, such as one or more of the storage devices 1325. Simply as an example, the execution of a sequence of instructions contained in working memory 1335 may cause processor 1310 to carry out one or more procedures of the methods described herein. In addition, or alternatively, some of the methods described herein may be carried out through specialized hardware.

[0150] As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any medium involved in providing data that causes a machine to operate in a specific manner. In some embodiments implemented using computer system 1300, various computer-readable media may be involved in providing instructions / code to processor 1310 for execution and / or may be used to store and / or carry such instructions / code. In many implementations, computer-readable media are physical and / or tangible storage media. Such media may take the form of non-volatile or volatile media. Non-volatile media include, for example, optical and / or magnetic disks such as storage device 1325. Volatile media include, but are not limited to, dynamic memory such as working memory 1335.

[0151] Physical and / or tangible computer-readable media in general forms include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with perforation patterns, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, or any other media from which a computer can read instructions and / or code.

[0152] Various forms of computer-readable media may be involved in transporting one or more instructions or sequences thereof for execution to the processor 1310. Simply put, as an embodiment, the instructions may first be transported on a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit the instructions as signals via a transmission medium to be received and / or executed by the computer system 1300.

[0153] The communication subsystem 1339 and / or its components generally receive signals, and the bus 1305 then transports the signals and / or the data, instructions, etc. carried by the signals to the working memory 1335, from which the processor 1310 reads and executes the instructions. Instructions received by the working memory 1335 may optionally be stored on the non-transient storage device 1325 either before or after execution by the processor 1310.

[0154] The methods, systems, and devices discussed above are embodiments. Various configurations may omit, substitute, or add various procedures or components as appropriate. For example, in alternative configurations, the method may be performed in a different order than described, and / or various steps may be added, omitted, and / or combined. Also, features described in relation to one configuration may be combined in various other configurations. Different aspects and elements of a configuration may be combined in a similar manner. Furthermore, the art is progressive, and therefore many of the elements are embodiments and do not limit the scope of this disclosure or claims.

[0155] Specific details are given in the description to provide a thorough understanding of the exemplary configurations, including their implementation. However, the configurations may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary details to avoid obscuring the configurations. This description provides only exemplary configurations and does not limit the scope, availability, or configurations of the claims. Rather, the foregoing description of the configurations will provide a useful description for implementing the techniques described to those skilled in the art. Various modifications may be made to the function and arrangement of the elements without departing from the spirit or scope of this disclosure.

[0156] Furthermore, the configuration may be described as a process, depicted as a schematic flowchart or block diagram. Each may be described as a sequential process, but many operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. The process may have additional steps not included in the diagram. Furthermore, embodiments of this method may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the required tasks may be stored in a non-transient computer-readable medium such as a storage medium. The processor may perform the tasks described.

[0157] While several exemplary configurations have been described, various modifications, alternative structures, and equivalents may be used without departing from the spirit of this disclosure. For example, the elements described above may be components of a larger system, and other rules may take precedence over or modify the use of the art in a different way. Also, some steps may be taken before, during, or after the elements described above are considered. Therefore, the above description is not restrictive to the scope of the claims.

[0158] As used herein and in the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly indicates otherwise. Thus, for example, “user” includes one or more such users, and “processor” includes one or more processors and their equivalents known to those skilled in the art.

[0159] Furthermore, the words “comprise,” “comprising,” “contains,” “containing,” “include,” “including,” and “includes,” when used herein and in the following claims, are intended to specify the presence of the described feature, integer, component, or step, but they do not preclude the presence or addition of one or more other features, integers, components, steps, actions, or groups.

[0160] Furthermore, it should be understood that the examples and embodiments described herein are for illustrative purposes only, and that various modifications or changes in light thereof are suggested to those skilled in the art and are included within the spirit, authority and scope of the appended claims.

Claims

1. A method for fusing hand posture data and handheld device posture data, wherein the method is: Using a hand posture sensor, capture hand posture data including multiple positions in a set of hand feature points within a reference frame and the orientation of the hand, Capturing handheld device orientation data, including the orientation of the handheld device within the reference frame, using a handheld device orientation sensor, wherein the handheld device orientation sensor is different from the hand orientation sensor. Based on the orientation of the captured handheld device and the three-dimensional model of the handheld device, the position of the set of feature points of the handheld device is determined. Determining the posture difference between the hand posture data and the handheld device posture data, wherein determining the posture difference is Determining positional differences based on a set of distances, wherein each distance in the set of distances corresponds to the distance between the position of a first feature point in the set of feature points of the handheld device and the position of a second feature point in the set of feature points of the hand that coincides with the first feature point. To determine the orientation difference between the orientation of the hand and the orientation of the handheld device, The difference in orientation is determined based on the difference in position and the difference in orientation, This includes, Based on the aforementioned difference in posture, a fusion movement is performed. Methods that include...

2. The method according to claim 1, wherein the positional difference is the maximum, minimum, or average distance of the set of distances.

3. The method according to claim 1, further comprising determining whether the posture difference is above or below a threshold.

4. If it is determined that the aforementioned posture difference exceeds the threshold, the fusion operation is performed. Determining that the handheld device is not being held by the hand. The method according to claim 3, including the method described in claim 3.

5. If it is determined that the posture difference is less than or equal to the threshold, the fusion operation is performed. Determining that the handheld device is being held by the hand. The method according to claim 3, including the method described in claim 3.

6. If it is determined that the aforementioned posture difference exceeds the threshold, the fusion operation is performed. Not using the aforementioned hand posture data, or The aforementioned handheld device posture data will not be used. The method according to claim 3, including the method described in claim 3.

7. If it is determined that the posture difference is less than or equal to the threshold, the fusion operation is performed. Using the aforementioned hand posture data, the position or orientation of the handheld device is determined. Using the handheld device posture data, determine the position or orientation of the hand, or Using the aforementioned hand posture data and the aforementioned handheld device posture data, combined posture data is generated. The method according to claim 3, including the method described in claim 3.

8. It is a system, A hand posture sensor configured to capture hand posture data including multiple positions in a set of hand feature points within a reference frame and the orientation of the hand, A handheld device attitude sensor configured to capture handheld device attitude data including the orientation of a handheld device within the reference frame, wherein the handheld device attitude sensor is different from the hand attitude sensor, One or more processors that communicate with the hand posture sensor and the handheld device posture sensor, wherein the one or more processors Based on the orientation of the captured handheld device and the three-dimensional model of the handheld device, the position of the set of feature points of the handheld device is determined. Determining the posture difference between the hand posture data and the handheld device posture data, wherein determining the posture difference is Determining positional differences based on a set of distances, wherein each distance in the set of distances corresponds to the distance between the position of a first feature point in the set of feature points of the handheld device and the position of a second feature point in the set of feature points of the hand that coincides with the first feature point. To determine the orientation difference between the orientation of the hand and the orientation of the handheld device, The difference in orientation is determined based on the difference in position and the difference in orientation, This includes, Based on the aforementioned difference in posture, a fusion movement is performed. One or more processors configured to perform operations including A system equipped with these features.

9. The system according to claim 8, wherein the positional difference is the maximum, minimum, or average distance of the set of distances.

10. The system according to claim 8, further comprising determining whether the posture difference is above or below a threshold.

11. If it is determined that the aforementioned posture difference exceeds the threshold, the fusion operation is performed. Determining that the handheld device is not being held by the hand. The system according to claim 10, including the following:

12. If it is determined that the aforementioned difference in posture is less than or equal to the threshold, the fusion operation is performed. Determining that the handheld device is being held by the hand. The system according to claim 10, including the following:

13. If it is determined that the aforementioned posture difference exceeds the threshold, the fusion operation is performed. Not using the aforementioned hand posture data, or The aforementioned handheld device posture data will not be used. The system according to claim 10, including the following:

14. If it is determined that the posture difference is less than or equal to the threshold, the fusion operation is performed. Using the aforementioned hand posture data, the position or orientation of the handheld device is determined. Using the handheld device posture data, determine the position or orientation of the hand, or Using the aforementioned hand posture data and the aforementioned handheld device posture data, combined posture data is generated. The system according to claim 10, including the following:

15. A non-transient computer-readable medium comprising instructions, which, when executed by one or more processors, cause the one or more processors to perform an operation to fuse hand pose data and handheld device pose data, the operation being Using a hand posture sensor, capture hand posture data including multiple positions in a set of hand feature points within a reference frame and the orientation of the hand, Capturing handheld device orientation data, including the orientation of the handheld device within the reference frame, using a handheld device orientation sensor, wherein the handheld device orientation sensor is different from the hand orientation sensor. Based on the orientation of the captured handheld device and the three-dimensional model of the handheld device, the position of the set of feature points of the handheld device is determined. Determining the posture difference between the hand posture data and the handheld device posture data, wherein determining the posture difference is Determining positional differences based on a set of distances, wherein each distance in the set of distances corresponds to the distance between the position of a first feature point in the set of feature points of the handheld device and the position of a second feature point in the set of feature points of the hand that coincides with the first feature point. To determine the orientation difference between the orientation of the hand and the orientation of the handheld device, The difference in orientation is determined based on the difference in position and the difference in orientation, This includes, Based on the aforementioned difference in posture, a fusion movement is performed. Non-transient computer-readable media, including [specific examples of such media].

16. The non-transient computer-readable medium according to claim 15, wherein the positional difference is the maximum, minimum, or average distance of the set of distances.

17. The non-transient computer-readable medium according to claim 15, further comprising determining whether the orientation difference is above or below a threshold.

Images