State machine and rejection reference for invoking UI gesture

Abstract

To provide a method and a system for a state machine and rejection reference for invoking an UI gesture.SOLUTION: An input gesture having a specific direction of a palm is detected based on geometric property of a hand relative to a head. Line-of-sight information is used to determine a hand gesture state. The gesture state indicates a palm-up gesture or a palm flip gesture. A state machine of a palm direction is used to determine a state of a palm direction based on the geometric property. A gesture detection state machine is used to determine a hand gesture based on the palm direction state and the line-of-sight vector. An action is invoked based on the hand gesture state.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] Some devices are capable of generating and presenting an augmented reality (XR) environment. An XR environment may include a fully or partially simulated environment that people sense and / or interact with via electronic systems. In XR, a subset of a person's body movements or a representation thereof is tracked, and one or more properties of one or more simulated virtual objects within the XR environment are adjusted accordingly to behave with realistic properties. In some embodiments, a user may use gestures to interact with virtual content. For example, a user can use gestures to select content, initiate an activity, etc. However, what is needed are improved techniques for improving hand pose determination. [Brief explanation of the drawings]

[0002] [Figure 1A] 1 shows an exemplary illustration of a user using hand poses as input poses, in accordance with one or more embodiments. [Figure 1B] 1 shows an exemplary illustration of a user using hand poses as input poses, in accordance with one or more embodiments.

[0003] [Figure 2A] FIG. 10 is an illustrative diagram of a user using an alternative hand pose as an input pose, according to one or more embodiments, in accordance with some embodiments. [Figure 2B] FIG. 10 is an illustrative diagram of a user using an alternative hand pose as an input pose, according to one or more embodiments, in accordance with some embodiments.

[0004] [Figure 3] 1 shows a flowchart of a technique for determining whether a hand is in an input pose, according to some embodiments.

[0005] [Figure 4]1 shows a flowchart of a technique for determining relative properties of hands and head, according to some embodiments.

[0006] [Figure 5] 1 illustrates a hand orientation state machine for determining palm position states, according to one or more embodiments.

[0007] [Figure 6A] 1 illustrates a flowchart of a technique for determining whether gaze criteria are met, according to one or more embodiments.

[0008] [Figure 6B] 1 shows a diagram of a gaze target in accordance with one or more embodiments.

[0009] [Figure 7] 1 illustrates a gesture detection state machine for determining hand gesture states, according to one or more embodiments.

[0010] [Figure 8] 1 illustrates a state machine for activating and inhibiting hand gestures, according to one or more embodiments.

[0011] [Figure 9] FIG. 1 illustrates a system diagram of an electronic device that can be used for gesture input, according to one or more embodiments.

[0012] [Figure 10] 1 illustrates an exemplary system for use in various augmented reality technologies. DETAILED DESCRIPTION OF THE INVENTION

[0013] The present disclosure relates to systems, methods, and computer-readable media for enabling gesture recognition and input. In some augmented reality contexts, specific hand poses may be used as user input poses. For example, detection of a specific hand pose may trigger a specific user input action or otherwise be used to allow a user to interact with an electronic device or content generated by the electronic device. One classification of hand poses that may be used as user input poses may include a hand detected in a palm-up position. Another classification is a palm-flip gesture, in which a hand is flipped from a palm-up position to a palm-down position. For example, a user may initiate the presentation of an icon or other virtual content by holding a hand in a palm-up position. From this position, the user can flip their hand to activate additional or alternative virtual content by flipping their hand to a palm-down position.

[0014] According to one or more embodiments, determining whether the hands are in an input pose includes tracking not only the hands but also additional joint location information of the user, such as head position. In some embodiments, the location information may be determined based on sensor data from sensors capturing various joints. Additionally or alternatively, the location information of the various joints may be inferred or otherwise derived from sensor data or a wearable device, such as a head-worn device. For example, head position may be determined based on an offset distance and / or orientation from a headset position, or may use the headset position as the head position and / or orientation, according to one or more embodiments.

[0015] In some embodiments, a hand may be determined to be in a palm-up position if the palm is mostly facing the head. This may be determined, for example, from camera data captured by a head-worn device or otherwise from the user's perspective toward the user's hand. For example, a determination may be made as to whether the hand is mostly facing one or more cameras. To that end, a spatial relationship between the hand and the head may be determined based on sensor data or otherwise based on location information. If the hand is determined to be sufficiently facing the user's head, the hand pose is classified as a palm-up input pose. Similarly, if the hand is determined to be sufficiently facing away from the head or camera from the palm-up position, the hand may be determined to be in a palm-flip pose. Also, if the hand is determined to be bent, upside down, etc., the hand may be determined to be in an invalid position.

[0016] In some embodiments, when a user is interacting with a user interface using a controller, the palm-up position and / or palm-flip pose may be defined based on the orientation of the controller relative to the user's head. To that end, the spatial relationship between the hands and head may be determined based on the spatial relationship between data derived from hand tracking and the location and / or orientation of the head. Alternatively, the spatial relationship between the hands and head may be based on the spatial relationship between the orientation of the controller and the location and / or orientation of the head. Thus, in some embodiments, the hand pose may be determined without regard to hand tracking data.

[0017] A hand gesture state may be determined by refining the determination of the hand pose based on the gaze. For example, a hand may be determined to be in a palm-up gesture or a palm-flip gesture only if the user's gaze is determined to meet a gaze criterion. The gaze criterion may be met, for example, if the gaze target is within a threshold distance of a virtual object or within a threshold distance of the hand. Alternatively, if a user is interacting with a user interface using a handheld controller, the gaze criterion may be met if the gaze target is within a bounding box or other predefined geometric shape around the location of the controller based on controller tracking data. Thus, if a hand pose transitions from an invalid pose to a palm-up pose, a palm-up gesture may be determined only if the gaze criterion is met. Thus, by considering the hand pose along with the gaze, a gesture state may be determined.

[0018] According to some embodiments, gesture determination may be modified based on one or more rejection reasons. For example, certain criteria may indicate that presentation of a virtual object should be prevented or that the current presentation of a virtual object should be rejected. Additionally, several criteria may be used to determine whether to cancel an action associated with a potentially initiated gesture.

[0019] The embodiments described herein provide an efficient method for determining whether a user is performing an input gesture using only standard joint position and other location information and without the need for additional specialized computer vision algorithms, thereby providing a less resource-intensive technique for determining palm orientation. Additionally, the embodiments described herein improve input gesture detection techniques by considering hand pose along with gaze to further infer whether a detected gesture is intentional, thereby improving the usability and accuracy of gesture-based input systems.

[0020] In the following disclosure, a physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. The physical environment may include physical features such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through sight, touch, hearing, taste, smell, and the like. In contrast, an XR environment refers to a fully or partially simulated environment that people sense and / or interact with through electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In an XR system, a subset or representation of a person's physical movements is tracked, and one or more properties of one or more virtual objects simulated within the XR environment are adjusted accordingly to behave according to at least one law of physics. As one example, an XR system can detect movement of a person's head and, in response, adjust the graphical content and sound fields presented to that person in a manner similar to how such views and sounds would change in a physical environment. As another example, an XR system can detect movement of an electronic device presenting the XR environment (e.g., a mobile phone, tablet, laptop, etc.) and, in response, adjust the graphical content and sound fields presented to that person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), the XR system can adjust a characteristic(s) of the graphical content within the XR environment in response to expressions of body movement (e.g., voice commands).

[0021] A wide variety of electronic systems exist that enable people to sense and / or interact with various XR environments. Examples include head-mountable systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mountable system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head-mountable system may be configured to accept an external opaque display (e.g., a smartphone). A head-mountable system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mountable system may have a transparent or translucent display rather than an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser-scanned light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, the transparent or translucent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology that projects a graphical image onto a person's retina. A projection system may also be configured to project virtual objects into a physical environment, for example, as a hologram or onto a physical surface.

[0022] In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the disclosed concepts. As part of this description, some of the drawings of the present disclosure represent structures and devices in block diagram form to avoid obscuring the novel aspects of the disclosed concepts. Also, for purposes of clarity, not all features of an actual implementation are described herein. Furthermore, as part of this description, some of the drawings of the present disclosure may be provided in flowchart form. The boxes in any particular flowchart may be presented in a particular order. However, it should be understood that the particular sequence of any flowchart is merely used to illustrate one embodiment. In other embodiments, any of the various components shown in the flowcharts may be omitted, or the sequence of actions shown may be performed in a different order or simultaneously. Additionally, other embodiments may include additional steps not shown as part of the flowcharts. Furthermore, the language used in this disclosure has been chosen primarily for purposes of readability and explanation, and not to limit or restrict the inventive subject matter or to rely on the scope of the claims necessary to determine such inventive subject matter. In this disclosure, reference to "one embodiment" or "one embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the disclosed subject matter, and multiple references to "one embodiment" or "one embodiment" should not be understood as necessarily all referring to the same embodiment.

[0023] It should be understood that in developing an actual implementation (such as a software and / or hardware development project), numerous decisions must be made to achieve the developer's particular goals (e.g., compliance with system and business-related constraints), and that these goals may vary from implementation to implementation. It should also be understood that such a development effort may be complex and time-consuming, but would nevertheless be a routine undertaking for one of ordinary skill in the art of designing and implementing graphical modeling systems having the benefit of this disclosure.

[0024] For the purposes of this application, the term "hand pose" refers to the position and / or orientation of the hand.

[0025] For purposes of this application, the term "input gesture" refers to a hand pose or movement that, when detected, triggers a user input action. Exemplary hand poses

[0026] 1A-1B show exemplary diagrams of a user performing a first input gesture, according to one or more embodiments. In particular, FIG. 1A shows a user 105 using an electronic device 115 within a physical environment 100. According to some embodiments, the electronic device 115 may include a pass-through or see-through display so that components of the physical environment 100 are visible. In some embodiments, the electronic device 115 may include one or more sensors configured to track the user to determine whether the user's pose should be processed as user input. For example, the electronic device 115 may include outward-facing sensors, such as a camera, depth sensor, etc., that may capture one or more parts of the user, such as a hand, arm, or shoulder. Additionally, in some embodiments, the electronic device 115 may include inward-facing sensors, such as an eye-tracking camera, that may be used in conjunction with the outward-facing sensors to determine whether a user input gesture has been performed.

[0027] A particular hand position or gesture may be associated with a user input action. In the example shown, the user 105 has their hand in hand pose 110 with their hand in a palm-up position. For purposes of this example, the palm-up position may be associated with a user input action that causes a user interface (UI) component A 120 to be presented. According to one or more embodiments, the UI component A 120 does not actually exist in the physical environment 100, but may be virtual content presented by the electronic device 115 and an augmented reality context such that the UI component A 120 appears in the physical environment 100 from the perspective of the user 105. The virtual content may include, for example, graphical content, image data, or other content for presentation to the user. In some embodiments, the hand pose 110 may be determined to be a palm-up input pose based on the relative position of the hand with respect to the head. For example, the hand may be determined to be in a palm-up position if the hand is facing toward the head rather than facing away from the head. As will be described in more detail below with respect to Figures 3-9, various techniques may be used to determine whether the hand is in a palm-up position.

[0028] FIG. 2A illustrates an alternative example of a user input component. In particular, in FIG. 2A , user 105 has repositioned their hand so that the palm faces down. In particular, hand pose 210 shows the palm facing the floor of the physical environment. According to some embodiments, determining that the hand is in a palm-down position may be associated with a different user input action than the palm-up pose shown in FIG. 1A . Furthermore, detecting a palm-down position may indicate a palm-flip gesture. For example, as shown in FIG. 1A , when a user is in a palm-up input gesture position and flips their hand so that the palm is in a palm-down position, the gesture may be associated with a particular user input action. Here, hand pose 210 is associated with the presentation of UI component B 220. According to one or more embodiments, UI component B 220 may be virtual content that does not actually exist within the physical environment 100 but is presented by the electronic device 115 and an augmented reality context such that UI component A 120 appears within the physical environment 100 from the perspective of the user 105. User Interface Gesture Invocation Overview

[0029] Referring to FIG. 3, a flowchart of a technique for determining whether a hand is in an input pose is shown, according to some embodiments. For purposes of explanation, the following steps are described as being performed by specific components. However, it should be understood that various actions may be performed by alternative components. Various actions may be performed in a different order. Furthermore, some actions may be performed simultaneously, some may not be required, or other actions may be added.

[0030] Flowchart 300 begins at block 305, where tracking data of a user is captured. According to some embodiments, the tracking data is obtained from sensors on the electronic device, such as a camera, a depth sensor, etc. The tracking data may include, for example, image data, depth data, etc., from which pose, position, and / or movement may be estimated. For example, location information of one or more joints of a hand may be determined from the tracking data and used to estimate the pose of the hand. According to one or more embodiments, the tracking data may include position, orientation, and / or movement information for different parts of the user.

[0031] In some embodiments, the tracking data may include or be based on additional sensor data, such as captured image data and / or depth data of the user's hands or both hands, in the case of hand tracking data, as shown in optional block 310. In some embodiments, the sensor data may be captured from a sensor on the electronic device, such as an outward-facing camera on the head-worn device or a camera otherwise configured within the electronic device to capture sensor data including the user's hands. Capturing sensor data may also include acquiring head tracking data in block 315. In some embodiments, the sensor data may include position and / or orientation information of the electronic device from which location or movement information of the user can be determined. According to some embodiments, the position and / or orientation of the user's head may be derived from position and / or orientation data of the electronic device when the device is worn on the head, such as with a headset, speculum, or other head-worn device.

[0032] In some embodiments, capturing user tracking data may further include obtaining eye-tracking data, as shown in block 320. Gaze may be detected from sensor data, for example, from an eye-tracking camera or other sensor on the device. For example, the head-worn device may include an inward-facing sensor configured to capture sensor data of one or more of the user's eyes, or an area of ​​the face around the eyes, which may be used to determine gaze. For example, the direction in which the user is looking may be determined in the form of a gaze vector. The gaze vector may be projected onto a scene including physical and virtual content.

[0033] As indicated at optional block 325, flowchart 300 may also include obtaining controller tracking data. In some embodiments, the controller tracking data may include sensor data, such as captured image data and / or depth data, of a controller held by a user. In some embodiments, the controller tracking data may include the location of the controller, which may include one or more representative points in space, representative geometric shapes, etc., that represent the location of the controller. The controller tracking data may optionally include additional information derived from the sensor data, such as the orientation of the controller.

[0034] Flowchart 300 continues at block 325, where hand geometric properties are calculated or otherwise determined for the hands. In some embodiments, the geometric properties may include the relative positions and / or orientations of the hands (or points in space representing the hands and / or controllers) and the head (or points in space representing the head). In some embodiments, the geometric properties may include various vectors determined based on location information of various parts of the user. Exemplary parameters and other metrics for the geometric properties are determined in more detail below with respect to FIG. 4.

[0035] At block 330, a hand orientation state is determined based on the geometric characteristics. According to one or more embodiments, the hand orientation state may indicate the pose and / or position of the hand and / or controller in a particular frame. In some embodiments, the hand pose may be determined using various metrics of the hand's geometric characteristics relative to the hand. For example, palm and head position and / or orientation information, and / or the relative positioning of the palm and head, may be used to determine whether the hand is in a palm-up orientation state because the palm is mostly facing toward the head or camera, or a palm-down orientation state because the palm is mostly facing away from the head. In embodiments in which the user is holding a controller, controller and head position and / or orientation information, and / or the relative positioning of the controller and head may be used to determine whether the controller satisfies a palm-up orientation state, a palm-down orientation state, etc. In some embodiments, a hand orientation state machine (also referred to as a "state machine") may be used to determine the hand orientation state, as described in more detail below with respect to FIG. 5.

[0036] Flowchart 300 proceeds to block 335, where a gesture detection state is determined based on the hand orientation state and gaze information. According to some embodiments, the gesture detection state may differ from the hand orientation state by using geometric characteristics to infer the intention of the hand orientation to indicate a gesture. For example, a hand with a palm-up hand orientation state may not be detected as a palm-up gesture if other geometric characteristics indicate that the hand orientation is not intended to be an input gesture. As an example, a hand orientation corresponding to an input gesture may be ignored when the user's gaze indicates that the hand orientation is not intended to be an input gesture. In some embodiments, a gaze target may be considered to determine whether a gaze criterion is met. The gaze criterion may be met, for example, if the user is looking at a hand performing a pose or at a point in space within a region where virtual content associated with a user input action is currently being presented or will be presented. In embodiments in which the user is using a controller, the gaze criteria may be met, for example, if the user is looking at the controller, which may be determined, for example, if the target of the user's gaze is within a predefined geometric shape surrounding the location of the controller. In some embodiments, a gesture detection state machine may be used to determine the gesture detection state, as described in more detail below with respect to FIG.

[0037] In block 340, suppression and / or rejection rules may be applied to the gesture detection state to obtain a gesture activation state. The gesture activation state may indicate the state of a hand gesture that may trigger a user input action. The gesture activation state may differ from the gesture detection state in that the gesture detection state indicates a detected gesture, whereas the gesture activation state indicates a gesture to be used for user input and is based on the gesture detection state. Example suppression and / or rejection rules or criteria may be based on characteristics of the hand, head, gaze, etc. that, when met, indicate that the hand gesture should be ignored and / or the associated input action should be modified. For example, a UI component or other virtual content may be blocked from being revealed, the UI component or other virtual content may be rejected, or an active input action may be canceled. Example rejection reasons may include, for example, hand movement, wrist movement, occlusion, the relative distance between the hand and head, a predefined hand pose to be rejected, etc. In some embodiments, a gesture activation state machine can be used to determine the gesture activation state, as described in more detail below with respect to FIG.

[0038] The flowchart 300 proceeds to block 345, where a determination is made as to whether a gesture activation state is associated with a user input. For example, the gesture activation state may be selected from one or more valid input gestures and invalid states. Examples of valid input gestures include, for example, a palm-up input gesture and a palm-flip input gesture, as described above with respect to FIGS. 1A and 2A. If it is determined that the gesture activation state is associated with the user input (e.g., if the gesture activation state matches a valid input gesture), the flowchart ends at block 350, and a user input action is invoked based on the gesture activation state. For example, if the hand pose 110 of FIG. 1A is determined to correspond to a valid palm-up gesture activation state based on the palm position and gaze direction, UI component A 120 is presented. Similarly, if the hand pose 210 of FIG. 2A is determined to correspond to a valid palm-flip gesture activation state, UI component B 220 is presented.

[0039] Returning to block 345, if a determination is made that the gesture activation-state is not associated with the user input (e.g., if the gesture activation-state is determined to be invalid), the flowchart ends at block 355 and the user input action is suppressed. For example, a UI component associated with the gesture may not be presented. According to one or more embodiments, one or more corrective actions may be taken. As an example, a previously activated input action may be canceled or a currently presented UI component may be dismissed.

[0040] According to embodiments described herein, an input pose may be identified based on various spatial relationships between a user's hands and head. FIG. 4 shows a flowchart of a technique for determining several relative characteristics of the hands and head, according to some embodiments. For purposes of explanation, the following steps are described as being performed by particular components and with respect to the example shown in FIGS. 1A-1B. However, it should be understood that various actions may be performed by alternative components. Various actions may be performed in a different order. Furthermore, some actions may be performed simultaneously, some may not be required, or other actions may be added.

[0041] Flowchart 400 begins at block 405, where geometric characteristics of the hands and head are determined. The geometric characteristics may include, for example, position and / or orientation information of the hands and head. This may include determining a palm normal, as shown in block 410. According to one or more embodiments, the palm normal may be defined by a vector pointing away from the center representative point of the palm. Referring to FIG. 1A, the palm normal 140 is shown pointing upward. In contrast, referring to FIG. 2A, the palm normal 240 is shown pointing downward.

[0042] Determining the geometric characteristics of the hand and head may further include determining a palm-forward vector in block 415. The palm-forward vector may be a directional vector indicating the direction the palm is pointing. This may be determined, for example, based on a directional vector originating at the wrist and extending through the knuckle of the index finger or other joint or representative location on the top of the palm. As shown in FIG. 1A, the palm-forward vector 145 is shown pointing slightly upward, while in FIG. 2A, the palm-forward vector 245 is shown pointing slightly downward.

[0043] Determining the geometric characteristics of the hand and head may further include determining a palm-to-head vector at block 420. The palm-to-head vector may indicate a direction vector from a palm origin location toward a gaze origin, such as a representative head location, a head-worn device location, an eye location, etc. As shown in FIG. 1A, palm-to-head vector 135 is shown in the palm-to-eye region and is similar to palm-to-head vector 235 in FIG. 2A.

[0044] Determining the geometric characteristics of the hands and head may further include determining a head vector at block 425. The head vector may indicate a direction vector in an upward direction, e.g., a "y" direction, from the perspective of the head and / or headset from a head location or representative head location, such as a head-worn device location, an eye location, etc. That is, the head vector may change direction as the head tilts. As shown in FIG. 1A, a head vector 225 is shown extending from the head of the user 105 and is similar to the head vector 225 of FIG. 2A.

[0045] Various geometric properties can be used to determine other spatial relationships between the hands and head. Accordingly, the flowchart proceeds to block 430, where relative properties of the hands and head are determined. The relative properties of the hands and head may be based on measurements between different geometric properties, as described above with respect to block 405.

[0046] According to one or more embodiments, determining the relative characteristics of the hand and head may include determining a palm-up-to-head angle based on the palm-forward vector and the palm-to-head vector, as shown in block 435. According to one or more embodiments, the palm-up-to-head angle may indicate how far the hand faces the user's eyes, the device's camera, etc. Stated differently, the palm-up-to-head angle may indicate a relative characteristic of the hand and head, indicating the relative orientation of the palm with respect to the head, or a representative location of the head, such as the gaze origin, head-worn device location, etc. Referring to FIG. 1B , palm-up-to-head angle 150 is shown as the angle between palm-to-head vector 135 and palm normal 140. Similarly, as shown in FIG. 2B , palm-up-to-head angle 250 indicates the angle between palm normal 240 and palm-to-head vector 235.

[0047] According to one or more embodiments, determining the relative characteristics of the hand and head may include determining a palm-forward-to-head y angle based on the palm-forward vector and the head vector, as shown in block 440. According to one or more embodiments, the palm-forward-to-head y angle may indicate whether the hand is bent or pointing. In other words, the palm-forward-to-head y angle may indicate when the hand is performing an extreme bending action or other pose that may be used to block or cancel user input. Referring to FIG. 1B , palm-forward-to-head y angle 160 is shown as the angle between head vector 125 (transformed from the determined location resulting from the user's head in FIG. 1A ) and palm-forward vector 145. Similarly, as shown in FIG. 2B , palm-forward-to-head y angle 260 indicates the angle between palm-forward vector 245 and head vector 225.

[0048] According to one or more embodiments, determining the relative characteristics of the hand and head may include determining a palm-up-to-head y-angle based on the palm normal vector and the head vector, as shown in block 445. According to one or more embodiments, the palm-up-to-head y-angle may indicate how far the palm is pointing up. Referring to FIG. 1B, palm-up-to-head y-angle 155 is shown as the angle between head vector 125 (transformed from the determined location resulting from the user's head in FIG. 1A) and palm normal 140. Similarly, as shown in FIG. 2B, palm-up-to-head y-angle 255 indicates the angle between palm normal 240 and head vector 225. Determining hand orientation

[0049] According to one or more embodiments, various parameters related to geometric characteristics may be used in conjunction to determine whether a user input gesture is allowed. In some embodiments, one or more state machines are used to determine whether a user input gesture is allowed. Figure 5 illustrates a hand orientation state machine for determining a palm position state, according to one or more embodiments.

[0050] According to one or more embodiments, the hand orientation state machine 500 is configured to perform a preliminary check of hand orientation states based on various geometric parameters. In some embodiments, the candidate hand orientation states may include a palm-up state 502, a palm-flip state 506, and an invalid state 504, where the hand pose is neither a palm-up state nor a palm-flip state. According to one or more embodiments, the hand orientation states may start with a hand orientation state based on the hand pose.

[0051] According to one or more embodiments, the hand orientation state may transition from an invalid state 504 to a palm-up state 502 based on the palm-up to head angle, as shown at 510. If the palm-up to head angle is less than a first threshold angle and the palm-forward to head y-angle is less than a second threshold angle, the hand orientation state may transition from the invalid state 504 to the palm-up state 502. From the invalid state 504, the hand orientation state may transition to the palm-up state 502, as shown at 510. However, in some embodiments, the hand orientation state may not transition from the invalid state 504 to the palm-flip state 506. In other words, to transition from the invalid state 504 to the palm-up state 502, the palm normal vector, the palm-forward vector, the palm-to-head vector, and / or the head vector may be considered. In some embodiments, the palm-up to head angle is considered and indicates a metric of how far the hand is facing the eyes, head, or camera. Additionally, the palm-up to head y angle may be considered, which indicates how bent or down the hand is. In some embodiments, the palm-up to head angle is compared to a first threshold angle, and the palm-up to head y angle is compared to a second threshold angle (which may be the same as or different from the first threshold angle).

[0052] As an example, referring back to Figure 1B, while the hand is in the palm-up position, palm-up to head angle 150 and palm-forward to head-y angle 160 are both less than 45 degrees. In contrast, referring to Figure 2B, palm-up to head angle 250 and palm-forward to head-y angle 260 are both at least 90 degrees. Thus, the palm position of Figure 1 likely indicates that palm-up to head angle 150 and palm-forward to head-y angle 160 meet the threshold at 510 of hand orientation state machine 500.

[0053] Alternatively, as shown at 515, the hand orientation state may transition from palm-up state 502 to invalid state 504 based on the palm-forward to head-y angle being greater than a threshold. Similarly, as shown at 530, the hand orientation state may transition from palm-flip state 506 to invalid state 504 based on the palm-forward to head-y angle being greater than a threshold. This may occur based on the direction the hand is pointing, for example, when the hand is pointing down because it is either bent or upside down. In other words, to transition to invalid state 504, the palm-forward vector and / or the head vector may be considered.

[0054] As an example, returning to Figure 1B, while the hand is in a palm-up position, the palm-to-head y angle 160 is approximately 45 degrees. In contrast, referring to Figure 2B, the palm-to-head y angle 260 is approximately 80 degrees. Thus, the palm positions in both Figures 1 and 2 are likely to exhibit palm-to-head y angles that meet the threshold at 530 of the hand orientation state machine 500.

[0055] According to one or more embodiments, the hand orientation state can transition from palm-up state 502 to palm-flip state 506 based on the palm-up to head angle and the palm-up to head-y angle, as shown at 525. In other words, to transition from palm-up state 502 to palm-flip state 506, the palm normal vector, the palm-to-head vector, and / or the head vector may be considered. In some embodiments, the palm-up to head angle is considered and indicates a metric of how far the hand is facing the eyes, head, or camera. Additionally, the palm-up to head-y angle, which indicates how far the palm normal is pointing upward, may be considered. In some embodiments, the palm-up to head angle is compared to a first threshold angle, and the palm-forward to head-y angle is compared to a second threshold angle (which may be the same or different value as the first threshold angle). Furthermore, each threshold may be the same or different from the thresholds considered in steps 510, 515, and 530. The hand orientation state may transition from palm-up state 502 to palm-flip state 506 if the palm-up to head angle is greater than a first threshold angle and the palm-up to head y angle is greater than a second threshold angle.

[0056] As an example, returning to Figure 1B, while the hand is in the palm-up position, palm-up to head angle 150 and palm-up to head-y angle 155 are both less than 30 degrees. In contrast, referring to Figure 2B, palm-up to head-y angle 155 and palm-up to head angle 250 are both at least 100 degrees. Thus, the palm position of Figure 2B likely indicates that palm-up to head angle 250 and palm-up to head-y angle 255 meet the threshold at 525 of hand orientation state machine 500.

[0057] Finally, as shown in block 520, the hand orientation state can transition from the palm-flip state 506 to the palm-up state 502 based on the palm-up to head angle. In other words, to transition from the palm-flip state 506 to the palm-up state 502, the palm normal vector and / or the palm-to-head vector may be considered. In some embodiments, the palm-up to head angle is considered and indicates a metric of how far the hand is facing the eyes, head, or camera. In some embodiments, the palm-up to head angle is compared to a threshold angle, which may be the same as or different from other thresholds used in the hand orientation state machine 500. If the palm-up to head angle is less than the threshold angle, the hand orientation state may transition from the palm-flip state 506 to the palm-up state 502.

[0058] As an example, referring back to Figure 1B, while the hand is in the palm-up position, the palm-up to head angle 150 is less than 30 degrees. In contrast, referring to Figure 2B, the palm-up to head angle 250 is at least 100 degrees. Therefore, the palm position of Figure 1 likely exhibits a palm-up to head angle 255 that meets the threshold at 520 of the hand orientation state machine 500. Determining Gesture Detection Status

[0059] According to one or more embodiments, the hand orientation state is determined without regard to gaze, but the gaze vector may be taken into account when determining the gesture detection state. In particular, the gaze vector may be identified and used to determine whether gaze criteria are met. In general, gaze criteria may be met if the gaze target is directed toward an area of ​​interest, such as an area around the hand performing the gesture, or a portion of the environment displaying a virtual component, or a location where a virtual component is displayed.

[0060] 6A shows a flowchart of a technique for determining whether line-of-sight criteria are met, according to one or more embodiments. For purposes of explanation, the following steps are described as being performed by specific components. However, it should be understood that various actions may be performed by alternative components. Various actions may be performed in a different order. Furthermore, some actions may be performed simultaneously, some may not be required, or other actions may be added.

[0061] Flowchart 600 begins at block 605, where eye-tracking data is acquired. For example, the eye-tracking system may include one or more sensors configured to capture image data or other sensor data from which the gaze direction of the eye can be determined. Flowchart 600 proceeds to block 610, where a gaze vector is acquired from the eye-tracking data. According to one or more embodiments, the gaze vector may be acquired from eye-tracking data, such as from an inward-facing camera on a head-worn device or other electronic device facing the user. The eye-tracking system may include one or more sensors configured to capture image data or other sensor data from which the gaze direction of the eye can be determined.

[0062] At block 615, a determination is made as to whether the gaze recently targeted a user interface component or a region reserved for a user interface component. This may occur when the most recent instance of the gaze vector intersecting the UI component region occurred within a threshold period, such as when the user momentarily looked away. A gaze target is determined from the gaze vector. For example, returning to FIG. 1A, gaze vector 130 is shown originating at electronic device 115 or the eye position behind electronic device 115 and pointing toward UI component A 120. Similarly, in FIG. 2A, gaze vector 230 is directed toward UI component B 220. In some embodiments, the gaze target may be a point in space relative to physical and / or virtual content within the augmented reality environment. Within the threshold period, the flowchart proceeds to block 620, where the threshold UI distance is adjusted. For example, if the user is looking away, the UI region may be narrowed to tighten gaze criteria. In the example shown in FIG. 6B, the target region may be associated with UI component 660. The target region may surround UI component 660 and / or may be based on the location at which the UI component is presented, such as the anchor position of the UI component. This may occur, for example, when the UI component is presented based on the location of another component in the environment, such as the fingertip of a hand, a physical object in the environment, etc. The target region around the UI component may shrink from region 670 to region 665 over time.

[0063] After the threshold UI distance is adjusted in block 620, or if a determination is made in block 615 that the gaze has not recently targeted a UI component, flowchart 600 proceeds to block 625, where a determination is made as to whether the gaze target is within the threshold UI distance. As shown in FIG. 6B , the threshold UI distance may be in either region 665 or region 670, depending on whether the threshold UI distance was adjusted in block 620. If it is determined that the gaze target is within the current threshold UI distance of the UI component, the flowchart ends at block 635, and the gaze criteria are considered satisfied.

[0064] Returning to block 625, if it is determined that the gaze target is not within the threshold UI distance, the flowchart 600 proceeds to block 630, where a determination is made as to whether the gaze target is within a threshold hand distance. With respect to the hand 650, the hand region 655 may be determined in several ways. For example, the hand or the geometric shape around the hand may be determined in the image data and compared to the gaze vector. As another example, the hand skeleton may be obtained using hand tracking data, and a determination may be made as to whether the gaze is within threshold locations of skeletal components whose location information is known. As one example, the hand region 655 may be defined as a region consisting of a bone length distance around each joint location, creating a bubble shape. If it is determined that the gaze target is within the threshold hand distance, the flowchart ends at block 635, and the gaze criteria are determined to be met. However, if it is determined in block 630 that the gaze target is not within the threshold hand distance, such as the hand region 655, the flowchart ends at block 640, and the gaze criteria are determined to be not met.

[0065] In some embodiments, additional considerations may be applied to determining whether the gaze criteria are met. In some embodiments, a debounce parameter may be applied. For example, the gaze signal may be required to stabilize before the gaze criteria are considered met or not met. A debounce period may be applied to determining whether the gaze criteria are met. In some embodiments, the debounce period may be different for determining whether the gaze criteria are met than for determining whether the gaze criteria are no longer met. Furthermore, in some embodiments, the debounce period(s) may be adjusted based on the gaze. For example, if the user is significantly looking away (i.e., exceeding a threshold gaze angle or distance), the debounce time may be shortened.

[0066] A hand gesture state may be determined based on gaze status, such as whether gaze criteria are met, and hand orientation state. FIG. 7 illustrates a gesture detection state machine for determining a gesture detection state, according to one or more embodiments. As described above, gaze criteria may be determined to be met if the user is looking at or near a hand or a UI component (or, in some embodiments, an area in which the UI component is presented). To that end, for clarity, gesture detection state machine 700 indicates that the gaze criteria are met by the term “looking” and that the gaze criteria are not met by the term “not looking.” In some embodiments, candidate hand gesture states may include a palm-up state 702, a palm-flip state 706, and an invalid state 704, where the gesture is neither a palm-up state nor a palm-flip state. Thus, in some embodiments, gesture detection states may be considered improved states from hand orientation states, as described above with respect to FIG. 5, once gaze is taken into account. In other words, gesture detection states may be an extension of hand orientation states. To that end, the gesture detection state machine 700 may start from a state determined from the hand orientation state machine 500 of Figure 5. In some embodiments, the hand orientation state may be determined using other techniques, such as when the user is holding a controller, and the orientation of the controller relative to the head determines the hand orientation state.

[0067] According to one or more embodiments, the gesture detection state may transition from the palm-up state 702 to the palm-flip state 706, as shown at 725, based on the pose of the hand determined to be in the palm-flip state, regardless of gaze. Thus, in some embodiments, gaze may not be considered when transitioning a gesture from the palm-up state to the palm-flip state. Similarly, at 720, the palm-flip state 706 may transition to the palm-up state 702 based on the state of the hand orientation, which is in the palm-up state. In other words, the transition between the palm-up state and the palm-flip state may be based on characteristics of the head and hand, such as the palm normal vector, the palm-to-head vector, and / or the head vector, and may not be related to the gaze vector. To that end, the gesture detection state may reflect the state of the hand orientation with respect to the transition between the palm-up state and the palm-flip state. In some embodiments, gaze may be taken into consideration. For example, the gaze may be required to be directed at the hand or a UI component area to determine the state change. When the gaze target moves away from the UI component, the UI may be dismissed and may need to be re-engaged by looking at the hand.

[0068] As shown at 730, the gesture detection state may transition from the palm-flip state 706 to the invalid state 704 based on the gaze and pose orientation states. In some embodiments, the gesture detection state may transition from the palm-flip state 706 to the invalid state 704 if the gaze criteria are not met or the pose is invalid. Similarly, the gesture detection state may transition from the palm-up state 702 to the invalid state 704 if the gaze criteria are not met or the pose is invalid, as shown at 715. In other words, if the hand orientation state indicates an invalid pose, the gesture detection state also becomes invalid. However, in some embodiments, the hand gesture state may also transition from the palm-flip state 706 or the palm-up state 702 to the invalid state if the gaze criteria are not met.

[0069] The gesture detection state can transition from the invalid state 704 to the palm-up state if the hand orientation state is in the palm-up state 702 and the gaze criteria are met, as shown at 710. For example, if the hand orientation state is determined to be in the palm-up state from an invalid state where the hand is upside down or otherwise pointing down, the gesture detection state transitions to the palm-up state 702 only if the gaze criteria are met.

[0070] According to one or more embodiments, the hand gesture state may not support a transition from the invalid state 704 to the palm-flip state 706. However, in some embodiments, the gesture detection state machine 700 may optionally support a transition from the invalid state 704 to the palm-flip state 706. For example, as shown at 735, the hand gesture state may transition from the invalid state 704 to the palm-flip state 706 if the hand orientation state is determined to be in the palm-flip state and the gaze is determined to have recently met gaze criteria. This may occur, for example, if the user looks away from and back onto the UI component within a predefined time window. Determining Gesture Activation State

[0071] According to one or more embodiments, additional considerations may be used when determining whether to activate an input action associated with a hand gesture. These various suppression rules or criteria may block, reject, and / or cancel a user input action. In some embodiments, various parameters may be used to determine whether a user action that would reveal a UI component should be blocked. Examples may include detecting recent hand movement or an otherwise unstable hand position, or certain detected hand poses, such as contact detected between two fingers, the hand being in a pinch pose, or the tips of the index finger and thumb being close to each other, which may indicate the user is holding an item. In these cases, the rejection reason may prevent the user input component from being revealed that would otherwise be revealed based on the hand gesture state.

[0072] As another example, the rejection criteria may indicate parameters that cause an action that would reveal or reject a UI component to be blocked. Examples may include a determination that the hand has recently made another gesture, such as a hover, touch, or pinch; a determination that the hand is close to a headset or otherwise within a predetermined proximity range of the user's head position; a determination that the hand is occluded by an object (which may indicate, for example, that the hand is occupied); a determination that two hands are close to each other; a determination that the hand has recently made an indirect pinch that is not anchored to the hand; or a determination that the index finger is curled (which may indicate, for example, that the user is about to drop the hand). In these cases, satisfaction of the blocking criteria may prevent a user input component from being revealed that would otherwise be revealed based on the hand gesture state.

[0073] As yet another example, a rejection reason may be used to prevent a transition to the palm-flip state and / or cancel an input action associated with the gesture. Examples may include determining that the wrist has recently moved or that the hand is occluded by an object. In these cases, the rejection reason may prevent a transition to the palm-flip state and / or cancel the gesture, thereby requiring a palm-up hand gesture reset.

[0074] The gesture activation state may be determined based on the gesture detection state and the suppression criteria. To that end, the gesture activation state may be considered an extension of the gesture detection state. FIG. 8 illustrates a state machine for activating and suppressing hand gestures, according to one or more embodiments. In some embodiments, candidate gesture activation states may include a palm-up state 802, a palm-flip state 806, and a disabled state 804, where the gesture is neither the palm-up state nor the palm-flip state. The gesture activation state is used to determine whether to allow or suppress a user input action associated with the hand gesture. Thus, the gesture activation state is used to activate the user input action associated with the corresponding state. Thus, in some embodiments, the gesture activation state machine 800 may start from a state determined from the gesture detection state machine 700 of FIG. 7.

[0075] According to one or more embodiments, the gesture activation state can transition from the palm-up state 802 to the palm-flip state 806, as shown at 825, based on the gesture detection state being determined to be the palm-flip state. Thus, in some embodiments, the inhibition criteria may not be considered when transitioning from the palm-up state to the palm-flip state. Similarly, at 820, the palm-flip state 806 can transition to the palm-up state 802 based on the gesture activation state being the palm-up state. In other words, the transition between the palm-up state and the palm-flip state can be based on head and hand characteristics, such as the palm normal vector, palm-to-head vector, and / or head vector, regardless of the inhibition criteria (or gaze information, as described above with respect to the gesture detection state machine 700 of FIG. 7). Additionally, the transition between the palm-up state and the palm-flip state can be based on the determined hand orientation state.

[0076] From the palm-flip state 806, the hand gesture state (as a gesture activation state) may transition to the invalid state 804 based on gaze and hand pose information, as shown at 830. In some embodiments, the hand gesture state may transition from the palm-flip state 806 to the invalid state 804 if any overruling reasons are true or if any rejection reasons are true. As described above, the overruling reasons may indicate parameters that prevent the action of revealing or rejecting a UI component. The rejection reasons may be used to prevent the transition to the palm-flip state and / or to cancel the input action associated with the gesture. Similarly, the hand gesture state may transition from the palm-up state 802 to the invalid state 804 if any of the rejection reasons are true, as shown at 815.

[0077] As shown at 810, if the gesture detection state is palm-up state 802 and the inhibit reason is not true, the gesture activation state may transition from the invalid state 804 to the palm-up state. For example, if the gesture detection state is determined to be in the palm-up state from the invalid state 804, where the hand is upside down or otherwise pointing down, the hand gesture state transitions to the palm-up state 802 (as the gesture activation state) only if the inhibit criteria do not prevent, disallow, or otherwise override the transition.

[0078] According to one or more embodiments, the gesture activation state machine 800 may not support a transition from the invalid state 804 to the palm-flip state 806. However, in some embodiments, the gesture activation state machine 800 may optionally support a transition from the invalid state 804 to the palm-flip state 806. For example, as shown at 835, the gesture activation state may transition from the invalid state 804 to the palm-flip state 806 if the hand gesture state is determined to be the palm-flip state and the rejection reason is not true. This may occur, for example, if the palm-up state is blocked due to a blocking reason such as a moving hand.

[0079] Once the gesture activation state is determined, an action corresponding to the gesture activation state may be invoked. For example, if the gesture activation state is palm-up state 802, a first user input action may be performed, while if the gesture activation state is palm-flip state 806, a second user input action may be performed. Furthermore, if the gesture activation state is disabled state 804, no action may be performed, and optionally, the input action may be canceled according to a corresponding inhibit reason. Exemplary Electronic Devices and Related Components

[0080] Referring to FIG. 9 , a simplified block diagram of an electronic device 900 is shown. The electronic device 900 may be part of a multifunction device such as a mobile phone, a tablet computer, a personal digital assistant, a portable music / video player, a wearable device, a head-mounted system, a projection system, a base station, a laptop computer, a desktop computer, a network device, or any other electronic system such as those described herein. For example, the electronic device 115 of FIG. 1A or the electronic device 215 of FIG. 2A may be an example of the electronic device 900. The electronic device 900 may include one or more additional devices, such as a server device, a base station, an accessory device, or the like, within which various functions may be included or throughout which various functions may be distributed. Exemplary networks include, but are not limited to, local networks such as a Universal Serial Bus (USB) network, an organizational local area network, and a wide area network such as the Internet. According to one or more embodiments, the electronic device 900 is utilized to interact with a user interface of an application 955. According to one or more embodiments, the application 955 may include one or more editing applications or applications that provide editing functionality, such as markup. It should be understood that the various components and functions within electronic device 900 may be distributed differently across modules or components, or even across additional devices.

[0081] The electronic device 900 may include one or more processors 920, such as a central processing unit (CPU) or a graphics processing unit (GPU). The electronic device 900 may also include memory 930. The memory 930 may include one or more various memories that may be used in conjunction with the processor 920 to perform device functions. For example, the memory 930 may include cache, ROM, RAM, or any type of transitory or non-transitory computer-readable medium capable of storing computer-readable code. The memory 930 may store various programming modules for execution by the processor(s) 920, including a tracking module 945 and various other applications 955. The electronic device 900 may also include storage 940. The storage device 940 may include one or more non-transitory computer-readable media, including, for example, magnetic disks and tapes (fixed, floppy, and removable), optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor memory devices such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM). The storage device 940 may be used to store various data and structures that may be used to store data related to hand tracking and UI preferences. The storage device 940 may be configured to store hand tracking network 975 and other data used to determine hand movements, such as enrollment data 985, according to one or more embodiments. The electronic device may further include a network interface that allows the electronic device 900 to communicate over a network.

[0082] The electronic device 900 may also include one or more cameras 905 or other sensors 910, such as depth sensors, that can determine the depth of a scene. In one or more embodiments, each of the one or more cameras 905 may be a traditional RGB camera or a depth camera. Further, the camera 905 may include a stereo camera or other multi-camera system. Additionally, the electronic device 900 may include other sensors that can collect sensor data to track a user's movements, such as a depth camera, an infrared sensor, or orientation sensors, such as one or more gyroscopes, accelerometers, etc.

[0083] According to one or more embodiments, memory 930 may include one or more modules containing computer-readable code executable by processor 920 to perform functions. Memory 930 may include, for example, a tracking module 945 and one or more applications 955. Tracking module 945 may be used to track the location of a user's hands, arms, joints, and other indicators of pose and / or movement in a physical environment. Tracking module 945 may use sensor data, such as data from camera 905 and / or sensors 910. In some embodiments, tracking module 945 may track user movements to determine whether to trigger user input from detected input gestures. In some embodiments described herein, tracking module 945 may be configured to determine a hand input state, a hand gesture state, and / or a gesture activation state based on hand tracking data, head pose information, gaze information, etc. Electronic device 900 may optionally include a display 980, or other device through which a user interface (UI) may be displayed or presented for interaction by a user. The UI may be associated with one or more of the applications 955, for example. The display 980 may be an opaque display, or may be translucent or transparent, such as a pass-through or see-through display. The display 980 may incorporate LEDs, OLEDs, digital light projectors, liquid crystal on silicon, etc.

[0084] While electronic device 900 is shown as including many of the components described above, in one or more embodiments, various components may be distributed across multiple devices. Thus, while certain calls and transmissions are described herein with respect to the particular system shown, in one or more embodiments, various calls and transmissions may be performed differently or directed differently based on differently distributed functionality. Furthermore, additional components may be used, and some combination of the functionality of any of the components may be combined.

[0085] 10 , a simplified functional block diagram of an exemplary multifunction electronic device 1000 is shown, according to one embodiment. Each of the electronic devices may be a multifunction electronic device or may have some or all of the illustrated components of the multifunction electronic devices described herein. Multifunction electronic device 1000 may include a processor 1005, a display 1010, a user interface 1015, graphics hardware 1020, device sensors 1025 (e.g., proximity / ambient light sensors, accelerometers, and / or gyroscopes), a microphone 1030, audio codec(s) 1035, speaker(s) 1040, communications circuitry 1045, digital image capture circuitry 1050 (e.g., including a camera system), video codec(s) 1055 (e.g., supporting a digital image capture unit), memory 1060, storage 1065, and a communications bus 1070. Multifunction electronic device 1000 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0086] The processor 1005 can execute instructions necessary to perform or control the operation of numerous functions performed by the device 1000 (e.g., image generation and / or processing as disclosed herein). The processor 1005 can, for example, drive the display 1010 and receive user input from a user interface 1015. The user interface 1015 can allow a user to interact with the device 1000. For example, the user interface 1015 can take various forms, such as buttons, a keypad, a dial, a click wheel, a keyboard, a display screen, and / or a touch screen, gaze, and / or gestures. The processor 1005 can also be a system-on-chip, such as found in mobile devices, and can include a dedicated GPU. The processor 1005 can be based on a reduced instruction set computer (RISC) or complex instruction set computer (CISC) architecture or any other suitable architecture and can include one or more processing cores. The graphics hardware 1020 can be dedicated computing hardware for processing graphics and / or assisting the processor 1005 in processing graphics information. In one embodiment, graphics hardware 1020 may include a programmable GPU.

[0087] The image capture circuit 1050 may include two (or more) lens assemblies 1080A and 1080B, each having a distinct focal length. For example, the lens assembly 1080A may have a short focal length relative to the focal length of the lens assembly 1080B. Each lens assembly may have a distinct associated sensor element 1090. Alternatively, two or more lens assemblies may share a common sensor element. The image capture circuit 1050 may capture still images and / or video images. Output from the image capture circuit 1050 may be processed by the video codec(s) 1055 and / or the processor 1005 and / or the graphics hardware 1020, and / or a dedicated image processing unit or pipeline incorporated within the circuit 1050. Images so captured may be stored in the memory 1060 and / or the storage device 1065.

[0088] The sensor and camera circuitry 1050 can capture still and video images, which may be processed, at least in part, by the video codec(s) 1055 and / or the processor 1005 and / or the graphics hardware 1020, and / or a dedicated image processing unit incorporated within the circuitry 1050, in accordance with this disclosure. The captured images may be stored in memory 1060 and / or storage 1065. The memory 1060 may include one or more various types of media used by the processor 1005 and the graphics hardware 1020 to perform the functions of the device. For example, the memory 1060 may include a memory cache, read-only memory (ROM), and / or random access memory (RAM). The storage 1065 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage 1065 may include one or more non-transitory computer-readable media including, for example, magnetic disks and tapes (fixed, floppy, and removable), optical media such as CD-ROMs and DVDs, and semiconductor memory devices such as EPROMs and EEPROMs. The memory 1060 and storage 1065 may be used to tangibly store computer program instructions or code organized into one or more modules and written in any desired computer programming language. For example, when executed by the processor 1005, such computer program code may perform one or more of the methods described herein.

[0089] The various processes defined herein contemplate options for obtaining and utilizing identifying information about a user. For example, such personal information may be utilized to track the user's pose and / or movement. However, to the extent such personal information is collected, such information should be obtained with the user's informed consent, and the user should have knowledge and control over the use of that personal information.

[0090] Personal information will be used only for legitimate and reasonable purposes by appropriate parties. Parties using such information will adhere to privacy policies and practices that comply with, at a minimum, applicable laws and regulations. Furthermore, such policies should comply with or exceed well-established government / industry standards. Furthermore, these parties will not distribute, sell, or share such information outside of any reasonable and legitimate purposes.

[0091] Furthermore, it is the intent of this disclosure that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Risk can be minimized by limiting data collection and deleting data when it is no longer needed. Furthermore, where applicable, including in certain health-related applications, data anonymization can be used to protect user privacy. Anonymization can be facilitated where appropriate by removing certain identifiers (e.g., date of birth), controlling the amount or specificity of data stored (e.g., collecting location data at a city level rather than an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.

[0092] It should be understood that the foregoing description is illustrative only, and not limiting. The materials have been presented in the context of specific embodiments to enable those skilled in the art to make and use the disclosed subject matter as claimed, and variations of those embodiments will be readily apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in combination with each other). Therefore, the particular arrangement of steps or actions illustrated in Figures 3-6A and 7-8, or the arrangement of elements illustrated in Figures 1-2, 6B, and 9-10, should not be construed as limiting the scope of the disclosed subject matter. The scope of the invention should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the words "including" and "in which" are used as the plain-English equivalents of the terms "comprising" and "wherein," respectively.

Claims

1. 1. A method comprising: determining geometric characteristics of a user's hand relative to their head performing a gesture; determining a gaze vector of the user; determining a hand gesture state from a plurality of candidate hand gesture states based on the gaze vector and the geometric characteristics of the hand relative to the head of the user; In response to determining that the hand gesture-state corresponds to an input gesture, invoking an action corresponding to the input gesture; A method comprising:

2. Determining geometric characteristics of the user's hand relative to the head includes: acquiring hand tracking data of the hand performing the gesture; and obtaining a head vector of the user.

3. The method of claim 1 , wherein the candidate hand gesture states include a palm-up state, a palm-flip state, and a null state.

4. Determining the hand gesture state includes: determining a hand orientation state from a plurality of candidate hand orientation states based on the geometric characteristics of the hand relative to the head; and determining the hand gesture state based on the hand orientation state and the gaze vector.

5. 5. The method of claim 4, wherein the hand orientation state is determined using a hand orientation state machine based on one or more of the group consisting of: 1) a palm-up to head angle indicating the relative position of the user's palm toward the user's head; 2) a palm-forward to head-y angle indicating the pointing direction of the user's palm relative to the user's head; and 3) a palm-up to head-y angle indicating the relative position of the palm pointing in an upward direction.

6. 5. The method of claim 4, wherein the hand gesture state is determined using a gesture detection state machine based on one or more of the group consisting of: 1) the hand gesture state; and 2) determining whether gaze criteria are met.

7. Determining whether the gaze criteria are met includes: The method of claim 6 , comprising determining whether the target of the gaze vector is within a threshold distance of at least one of the hand, a controller, and a user input component.

8. acquiring controller tracking data; determining an orientation of the controller based on the controller tracking data; Determining the hand gesture state includes: determining a hand orientation state from a plurality of candidate hand orientation states based on the geometric characteristics of an orientation of the controller relative to the head; and determining the hand gesture state based on an orientation state of the controller and the gaze vector.

9. A non-transitory computer readable medium containing computer readable code, the computer readable code comprising: determining geometric characteristics of a user's hand relative to their head performing a gesture; determining a gaze vector of the user; determining a hand gesture-state from a plurality of candidate hand gesture-states based on the gaze vector and the geometric characteristics of the hand relative to the head of the user; a non-transitory computer-readable medium executable by one or more processors to: in response to determining that the hand gesture-state corresponds to an input gesture, invoke an action corresponding to the input gesture;

10. The computer readable code for determining a geometric characteristic of a user's hand relative to a head comprises: obtaining hand tracking data of the hand performing the gesture; The non-transitory computer-readable medium of claim 9 comprising computer-readable code for obtaining a head vector of the user.

11. The computer readable code for determining a geometric characteristic of a user's hand relative to a head comprises: acquiring controller tracking data for a controller held by the hand; The non-transitory computer-readable medium of claim 9 comprising computer-readable code for obtaining a head vector of the user.

12. The non-transitory computer-readable medium of claim 9 , wherein the candidate hand gesture states include a palm-up state, a palm-flip state, and a null state.

13. The computer readable code for determining the hand gesture state further comprises: determining a hand orientation state from a plurality of candidate hand orientation states based on the geometric characteristics of the hand relative to the head; 10. The non-transitory computer-readable medium of claim 9, comprising computer-readable code for determining the hand gesture state based on the hand orientation state and the gaze vector.

14. 14. The non-transitory computer-readable medium of claim 13, wherein the hand orientation state is determined using a hand orientation state machine based on one or more of the group consisting of: 1) a palm-up to head angle indicating a relative position of the user's palm toward the user's head; 2) a palm-forward to head-y angle indicating a pointing direction of the user's palm relative to the user's head; and 3) a palm-up to head-y angle indicating a relative position of the palm pointing in an upward direction.

15. 14. The non-transitory computer-readable medium of claim 13, wherein the hand gesture state is determined using a gesture detection state machine based on one or more of the group consisting of: 1) the hand gesture state; and 2) determining whether a gaze criterion is met.

16. The computer readable code for determining whether the line of sight criteria are met comprises:

16. The non-transitory computer-readable medium of claim 15, comprising computer-readable code for determining whether a target of the gaze vector is within a threshold distance of at least one of the hand, a controller, and a user input component.

17. The computer readable code for invoking an action corresponding to the input gesture comprises:

10. The non-transitory computer-readable medium of claim 9, further comprising computer-readable code for determining a gesture activation state based on the hand gesture state and one or more inhibition criteria.

18. 1. A system comprising: one or more processors; and one or more computer readable media containing computer readable code, said computer readable code comprising: determining geometric characteristics of a user's hand relative to their head performing a gesture; determining a gaze vector of the user; determining a hand gesture-state from a plurality of candidate hand gesture-states based on the gaze vector and the geometric characteristics of the hand relative to the head of the user; The system is executable by one or more processors to, in response to determining that the hand gesture-state corresponds to an input gesture, invoke an action corresponding to the input gesture.

19. The computer readable code for determining the hand gesture state further comprises: determining a hand orientation state from a plurality of candidate hand orientation states based on the geometric characteristics of the hand relative to the head; 20. The system of claim 18, comprising computer readable code for determining the hand gesture state based on the hand orientation state and the gaze vector.

20. The computer readable code for invoking an action corresponding to the input gesture comprises:

20. The system of claim 18, further comprising computer readable code for determining a gesture activation state based on the hand gesture state and one or more inhibition criteria.

Images