State machine and denial criteria for UI gesture calls

Abstract

The invention relates to a state machine and denial criteria for UI gesture calls. An input gesture having a particular palm orientation is detected based on geometric characteristics of the hand relative to the head. The gaze information is used to determine a hand gesture state. The gesture state refers to a palm upward gesture or a palm turning gesture. The hand orientation state machine is used for determining a hand orientation state based on the geometric characteristics. A gesture detection state machine is used to determine a hand gesture based on the hand orientation state and the gaze vector. An action is invoked based on the hand gesture state.

Description

Background Technology

[0001] Some devices can generate and present extended reality (XR) environments. An XR environment can include a fully or partially simulated environment that people perceive and / or interact with via electronic systems. In XR, a subset of a person's physical motion, or a representation thereof, is tracked, and in response, one or more properties of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with real-world attributes. In some implementations, users can use gestures to interact with virtual content. For example, users can use gestures to select content, initiate activities, etc. However, an improved technique is needed to refine the determination of hand gestures. Attached Figure Description

[0002] Figures 1A to 1B An example diagram is shown illustrating a user's use of hand gestures as input gestures according to one or more implementation schemes.

[0003] Figures 2A to 2B An example diagram illustrating how a user uses another facial gesture as input gesture according to one or more implementation schemes.

[0004] Figure 3 A flowchart is shown for a technique for determining whether a hand is in an input posture, according to some implementation schemes.

[0005] Figure 4 A flowchart is shown of a technique for determining the relative characteristics of the hand and head according to some embodiments.

[0006] Figure 5 A hand orientation state machine for determining the palm position state is shown according to one or more embodiments.

[0007] Figure 6A A flowchart is shown of a technique for determining whether a gaze criterion is met, according to one or more implementation schemes.

[0008] Figure 6B A diagram illustrating the gaze target according to one or more implementation schemes is shown.

[0009] Figure 7 A gesture detection state machine for determining the state of a hand gesture is shown according to one or more embodiments.

[0010] Figure 8 A state machine for activating and inhibiting gestures according to one or more embodiments is shown.

[0011] Figure 9 A system diagram of an electronic device that can be used for gesture input according to one or more embodiments is shown.

[0012] Figure 10 An exemplary system for various extended reality technologies is shown. Detailed Implementation

[0013] This disclosure relates to systems, methods, and computer-readable media for implementing gesture recognition and input. In some augmented reality environments, certain hand gestures can be used as user input gestures. For example, the detection of a particular hand gesture can trigger a specific user input action, or otherwise allow a user to interact with an electronic device or content generated by the electronic device. One category of hand gestures that can be used as user input gestures involves the detection of a hand in a palm-up position. Another category is palm-flipping gestures, where the hand flips from a palm-up position to a palm-down position. For example, a user can initiate the presentation of an icon or other virtual content by keeping their hand in a palm-up position. From that position, the user can flip their hand to a palm-down position to activate additional or alternative virtual content.

[0014] According to one or more embodiments, determining whether a hand is in an input posture includes tracking not only the hand but also additional joint position information of the user, such as head position. In some embodiments, position information can be determined based on sensor data from sensors capturing various joints. Additionally or alternatively, position information of various joints can be inferred or otherwise derived from sensor data or wearable devices (such as head-mounted devices). For example, according to one or more embodiments, head position can be determined based on offset distance and / or orientation relative to the head-mounted device position, or the head-mounted device position can be used as head position and / or orientation.

[0015] In some implementations, a palm-up position can be determined if the palm of the hand primarily faces the head. This can be determined, for example, based on camera data captured by the head-mounted device or otherwise from the user's perspective of their hand. For instance, it can be determined whether the hand is primarily facing the camera. To do this, the spatial relationship between the hand and head can be determined based on sensor data or otherwise based on positional information. If it is determined that the hand is fully facing the user's head, the hand posture is classified as a palm-up input posture. Similarly, if it is determined from the palm-up position that the hand is fully facing away from the head or camera, the hand can be determined to be in a palm-flipped posture. Additionally, if the hand is determined to be bent, inverted, etc., the hand can be determined to be in an invalid position.

[0016] In some implementations, if a user is interacting with the user interface using a controller, the palm-up position and / or palm-flipping posture can be defined based on the controller's orientation relative to the user's head. For this purpose, the spatial relationship between the hand and head can be determined based on the spatial relationship between data derived from hand tracking and the head's position and / or orientation. Alternatively, the spatial relationship between the hand and head can be based on the spatial relationship between the controller's orientation and the head's position and / or orientation. Therefore, in some implementations, hand poses can be determined without considering hand tracking data.

[0017] Hand gesture states can be determined by refining hand pose determination based on gaze. For example, if the user's gaze meets the gaze criteria, it can be determined that the hand is in either a palm-up or palm-flip gesture. For instance, the gaze criterion is met if the gazed target is within a threshold distance of a virtual object or within a threshold distance of the hand. Alternatively, if the user is interacting with the user interface using a handheld controller, the gaze criterion is met if the gazed target is within a bounding box or other predefined geometry around the controller position based on controller tracking data. Therefore, if the hand pose changes from an invalid pose to a palm-up pose, a palm-up gesture can only be determined if the gaze criterion is met. Thus, gesture states can be determined by considering both hand pose and gaze.

[0018] According to some implementation schemes, gesture determination can be modified based on one or more rejection reasons. For example, certain criteria can indicate that the rendering of a virtual object should be prevented, or that the current rendering of a virtual object should be deactivated. Furthermore, criteria can be used to determine whether to cancel an action associated with a gesture that may have already been initiated.

[0019] The implementation described herein provides an efficient method for determining whether a user is performing an input gesture using only standard joint positions and other positional information, without requiring any additional specialized computer vision algorithms, thus providing a low-resource-intensive technique for determining hand orientation. Furthermore, the implementation described herein improves input gesture detection technology by considering hand posture and gaze to further infer whether the detected gesture is intentional, thereby improving the usefulness and accuracy of gesture-based input systems.

[0020] In the following disclosure, a physical environment refers to the physical world that people can perceive and / or interact with without the aid of electronic devices. A physical environment may include physical features, such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with a physical environment through senses such as sight, touch, hearing, taste, and smell. In contrast, an XR environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In the case of an XR system, a subset or representation of a person's physical motion is tracked, and in response, one or more properties of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with at least one physical law. As an example, an XR system may detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. As another example, an XR system can detect movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) that present the XR environment and adjust the graphical content and sound field presented to the user in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust the characteristics of the graphical content in the XR environment in response to representations of physical movement (e.g., voice commands).

[0021] Many different types of electronic systems enable people to sense and / or interact with various XR environments. Examples include: head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have an integrated opaque display and one or more speakers. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may combine one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some implementations, transparent or translucent displays can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0022] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the accompanying drawings of this disclosure are block diagrams representing structures and devices to avoid obscuring the novel aspects of the disclosed concepts. For clarity, not all features of actual specific embodiments may be described. Additionally, some of the drawings of this disclosure are provided in the form of flowcharts as part of this specification. The blocks in any particular flowchart may be presented in a specific order. However, it should be understood that the specific order of any given flowchart is only for illustrative purposes of one embodiment. In other embodiments, any of the various elements depicted in the flowcharts may be omitted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. Furthermore, other embodiments may include additional steps not shown as part of the flowcharts. Moreover, the language used in this disclosure has been primarily chosen for readability and instructional purposes and may not have been chosen to define or limit the subject matter of the invention, or to invoke the necessary claims to determine such inventive subject matter. In this disclosure, reference to “an implementation” or “implementation” means that a particular feature, structure or characteristic described in connection with the implementation is included in at least one implementation of the disclosed subject matter, and the repeated references to “an implementation” or “implementation” should not be construed as necessarily involving all of the same implementation.

[0023] It should be understood that in any actual implementation of development (as in any software and / or hardware development project), numerous decisions must be made to achieve the developer's specific goals (e.g., compliance with system and business-related constraints), and these goals may differ between different implementations. It should also be understood that such development work can be complex and time-consuming, but nevertheless, it remains routine work for those of ordinary skill in the art who design and implement graphical modeling systems in benefit from this disclosure.

[0024] For the purposes of this application, the term "hand posture" refers to the positioning and / or orientation of the hand.

[0025] For the purposes of this application, the term "input gesture" refers to a hand gesture or movement that triggers a user input action when detected.

[0026] Example hand gestures

[0027] Figures 1A to 1B An example diagram is shown illustrating a user performing a first input gesture according to one or more implementation schemes. Specifically, Figure 1AA user 105 is shown using electronic device 115 within a physical environment 100. According to some embodiments, electronic device 115 may include a see-through or transmissive display, making components of the physical environment 100 visible. In some embodiments, electronic device 115 may include one or more sensors configured to track the user to determine whether the user's gesture should be processed as user input. For example, electronic device 115 may include outward-facing sensors, such as cameras, depth sensors, etc., capable of capturing one or more parts of the user (such as hands, arms, shoulders, etc.). Furthermore, in some embodiments, electronic device 115 may include inward-facing sensors (such as eye-tracking cameras) that can be used in conjunction with outward-facing sensors to determine whether to execute a user input gesture.

[0028] Certain hand positions or gestures can be associated with user input actions. In the example shown, user 105 positions their hand in a palm-up position, hand gesture 110. For illustrative purposes, the palm-up position can be associated with a user input action to cause a user interface (UI) component A 120 to be presented. According to one or more embodiments, UI component A 120 can be virtual content that does not actually exist in the physical environment 100, but is presented by the electronic device 115 and the extended reality environment such that UI component A 120 appears within the physical environment 100 from the user 105's perspective. Virtual content may include, for example, graphical content, image data, or other content for presentation to the user. In some embodiments, hand gesture 110 can be determined as a palm-up input gesture based on the relative position of the hand and head. For example, if the hand faces the head more than it faces away from the head, the hand can be determined to be in a palm-up position. Various techniques can be used to determine whether a hand is in a palm-up position, as will be discussed below. Figures 3 to 9 To describe in more detail.

[0029] Figure 2A Alternative examples of user input components are depicted. Specifically, in Figure 2A In this example, user 105 has changed their hand position so that their palms are now facing down. Specifically, hand posture 210 shows the palms facing the floor of the physical environment. According to some implementations, determining that the hand is in a palm-down position can be different from... Figure 1A The user input action, as shown, is associated with a palm-up posture. Furthermore, detecting a palm-down position can indicate a palm-flipping gesture. For example, when the user is in a position such as... Figure 1AThe hand gesture position shown is palm-up, and when the user flips their hand so that the palm is palm-down, the gesture can be associated with a specific user input action. Here, hand gesture 210 is associated with the presentation of UI component B 220. According to one or more embodiments, UI component B 220 can be virtual content that does not actually exist in the physical environment 100, but is presented by the electronic device 115 and the extended reality environment, so that UI component A 120 appears to appear within the physical environment 100 from the user 105's perspective.

[0030] User Interface Gesture Call Overview

[0031] Go to Figure 3 The diagram illustrates a flowchart of a technique for determining whether a hand is in an input posture, according to some embodiments. For illustrative purposes, the following steps will be described as being performed by a specific component. However, it should be understood that various actions can be performed by alternative components. Various actions can be performed in different orders. Furthermore, some actions can be performed simultaneously, and some actions may be unnecessary, or additional actions may be added.

[0032] Flowchart 300 begins at box 305, capturing user tracking data. According to some embodiments, tracking data is obtained from sensors on an electronic device (such as a camera, depth sensor, etc.). Tracking data may include, for example, image data, depth data, etc., from which pose, positioning, and / or movement can be estimated. For example, positional information of one or more joints of the hand can be determined from the tracking data and used to estimate hand pose. According to one or more embodiments, tracking data may include positioning information, orientation information, and / or movement information of different parts of the user.

[0033] In some embodiments, the tracking data may include or be based on additional sensor data, such as captured image data and / or depth data of one or more of the user's hands (in the case of hand tracking data), as shown in optional box 310. In some embodiments, the sensor data may be captured from sensors on the electronic device, such as an outward-facing camera on a head-mounted device, or a camera otherwise configured in the electronic device to capture sensor data including the user's hands. Capturing sensor data may also include obtaining head tracking data in box 315. In some embodiments, the sensor data may include positioning and / or orientation information of the electronic device, based on which the user's position or motion information can be determined. According to some embodiments, when the device is worn on the head (such as with headphones, glasses, or other head-mounted devices), the positioning and / or orientation of the user's head may be derived from the positioning and / or orientation data of the electronic device.

[0034] In some implementations, capturing user tracking data may additionally include obtaining gaze tracking data, as shown in box 320. For example, gaze may be detected based on sensor data from an eye-tracking camera or other sensors on the device. For instance, a head-mounted device may include inward-facing sensors configured to capture sensor data from one or both of the user's eyes or facial regions around the eyes, which can be used to determine gaze. For example, the direction the user is looking may be determined in the form of a gaze vector. The gaze vector may be projected onto a scene that includes both physical and virtual content.

[0035] As shown in optional box 325, flowchart 300 may also include obtaining controller tracking data. In some embodiments, controller tracking data may include sensor data, such as captured image data and / or depth data of the controller held by the user. In some embodiments, controller tracking data may include the position of the controller, which may include one or more representative points in space representing the position of the controller, representative geometry, etc. Controller tracking data may optionally include additional information derived from the sensor data, such as the orientation of the controller, etc.

[0036] Flowchart 300 proceeds to block 325, where the geometric properties of the hand relative to the hand are calculated or otherwise determined. In some embodiments, the geometric properties may include the relative positioning and / or orientation of the hand (or a point in space representing the hand and / or controller) and the head (or a point in space representing the head). In some embodiments, the geometric properties may include various vectors determined based on positional information of various parts of the user. The following will discuss... Figure 4 To determine in more detail the example parameters and other metrics related to geometric properties.

[0037] In box 330, a hand orientation state is determined based on geometric characteristics. According to one or more embodiments, the hand orientation state can indicate the pose and / or position of the hand and / or controller in a particular frame. In some embodiments, various measures of the hand's geometric characteristics relative to the hand can be used to determine the hand pose. For example, the position and / or orientation information of the palm and head, and / or the relative positioning of the palm and head, can be used to determine whether the palm is primarily facing the head or camera, thus in a palm-up orientation state, or whether the palm is primarily facing away from the head, thus in a palm-down orientation state. In embodiments where the user is holding the controller, the position and / or orientation information of the controller and head, and / or the relative positioning of the controller and head, can be used to determine whether the controller satisfies a palm-up orientation state, a palm-down orientation state, etc. In some embodiments, as will be discussed below... Figure 5 In more detail, a hand orientation state machine can be used to determine the hand orientation state.

[0038] Flowchart 300 proceeds to block 335, where a gesture detection state is determined based on hand orientation state and gaze information. According to some embodiments, the gesture detection state may differ from the hand orientation state by using geometric properties to infer the intentionality of the hand orientation to indicate a gesture. For example, a hand with a palm-up orientation state may not be detected as a palm-up gesture if other geometric properties indicate that the hand orientation is not intended as an input gesture. As an example, the hand orientation corresponding to the input gesture may be ignored when the user's gaze indicates that the hand orientation is not intended as an input gesture. In some embodiments, the gaze target may be considered to determine whether the gaze criterion is met. For example, the gaze criterion is met if the user is looking at a hand performing the gesture, or a point in space within an area currently presenting virtual content associated with the user's input action, or a point in space within an area where virtual content will be presented. In embodiments where the user is using a controller, the gaze criterion is met, for example, if the user is looking at the controller, or if the target the user is looking at is within a predefined geometry around the controller's location. In some embodiments, a gesture detection state machine may be used to determine the gesture detection state, as will be discussed below. Figure 7 To describe in more detail.

[0039] In box 340, suppression and / or rejection rules can be applied to the gesture detection state to obtain a gesture activation state. A gesture activation state can indicate the state of a hand gesture that can trigger a user input action. The difference between a gesture activation state and a gesture detection state is that the gesture detection state indicates a detected gesture, while the gesture activation state indicates a gesture that should be used for user input and is based on the gesture detection state. Examples of suppression and / or rejection rules or criteria can be based on characteristics of the hand, head, gaze, etc., indicating that the hand gesture should be ignored and / or the associated input action should be modified when these rules or criteria are met. For example, the appearance of UI components or other virtual content can be blocked, UI components or other virtual content can be deactivated, or an activated input action can be canceled. Examples of rejection reasons can include, for example, hand movement, wrist movement, occlusion, relative distance between hand and head, predefined hand gestures that should be rejected, etc. In some implementations, a gesture activation state machine can be used to determine the gesture activation state, as will be discussed below. Figure 8 To describe in more detail.

[0040] Flowchart 300 proceeds to block 345, where it is determined whether a gesture activation state is associated with user input. For example, a gesture activation state can be selected from one or more valid input gestures and invalid states. Examples of valid input gestures include, for example, a palm-up input gesture and a palm-flip input gesture, as described above. Figure 1A and Figure 2ADescription. If it is determined that the gesture activation state is associated with user input (e.g., if the gesture activation state matches a valid input gesture), the flowchart ends at box 350, and the user input action is invoked based on the gesture activation state. For example, if determined based on palm position and gaze direction... Figure 1A If hand gesture 110 corresponds to a valid palm-up gesture activation state, then UI component A 120 will be rendered. Similarly, if Figure 2A If the hand gesture 210 is determined to correspond to a valid palm flip gesture activation state, then UI component B 220 will be rendered.

[0041] Returning to box 345, if it is determined that the gesture activation state is not associated with user input (e.g., if the gesture activation state is determined to be invalid), the flowchart ends at box 355, and the user input action is suppressed. For example, the UI component associated with the gesture may not be presented. Depending on one or more implementations, one or more corrective actions may be taken. As an example, a previously activated input action may be canceled, or the currently presented UI component may be de-presented.

[0042] According to the implementation scheme described herein, input gestures can be identified based on various spatial relationships between the user's hands and head. Figure 4 A flowchart illustrating a technique for determining some relative characteristics of the hand and head according to some embodiments is shown. For illustrative purposes, the following steps will be described as being performed by specific components and for... Figures 1A to 1B The example shown is executed as described. However, it should be understood that various actions can be performed by alternative components. Various actions can be performed in different orders. Furthermore, some actions can be performed simultaneously, and some actions may be unnecessary, or additional actions may be added.

[0043] Flowchart 400 begins at box 405, where the geometric properties of the hand and head are determined. These geometric properties may include, for example, position and / or orientation information of the hand and head. As shown at box 410, this may include determining the palm normal. According to one or more embodiments, the palm normal may be defined by a vector originating from a representative point at the center of the palm and directed away from the palm. [Go to...] Figure 1A The palm normal 140 is shown as pointing upwards. In contrast, turning to... Figure 2A The palm normal 240 is shown as being in a downward direction.

[0044] Determining the geometry of the hand and head may additionally include determining the palm-forward vector at box 415. The palm-forward vector can be a direction vector indicating the pointing direction of the palm. This can be determined, for example, based on a direction vector originating from the wrist and extending through the index finger joint or other joints or representative locations on the upper part of the palm. Figure 1AAs shown, the forward vector 145 of the palm is depicted as pointing slightly upwards, while... Figure 2A In the middle, the palm points forward at a vector of 245, slightly downward.

[0045] Determining the geometry of the hand and head may additionally include defining the palm-to-head vector at frame 420. The palm-to-head vector can indicate the direction vector from the origin of the palm towards the gaze origin (such as a representative head position, head-mounted device position, eye position, etc.). Figure 1A As shown, the palm-to-head vector 135 is represented as the region from the palm to the eye, and is similar to... Figure 2A The vector from the palm to the head is 235.

[0046] Determining the geometry of the hands and head may additionally include determining the head vector at frame 425. The head vector may indicate a direction vector originating from the head position or a representative head position (such as the head-mounted device position, eye position, etc.) and positioned upwards (e.g., in the "y" direction) from the viewpoint of the head and / or head-mounted device. That is, the head vector may change direction as the head tilts. Figure 1A As shown, head vector 225 is depicted as extending from the head of user 105, and is similar to... Figure 2A The head vector is 225.

[0047] Various geometric properties can be used to determine other spatial relationships between the hand and head. Therefore, the flowchart proceeds to box 430, where the relative properties of the hand and head are determined. These relative properties can be based on measurements between different geometric properties, as described above for box 405.

[0048] According to one or more embodiments, determining the relative characteristics of the hand and head may include, as shown in box 435, determining the palm-up-to-head angle based on the palm-forward vector and the palm-to-head vector. According to one or more embodiments, the palm-up-to-head angle may indicate the degree to which the hand is facing the user's eyes, the device's camera, etc. In other words, the palm-up-to-head angle may indicate the relative characteristics of the hand and head, which indicate the relative orientation of the palm to the head, or a representative position of the head, such as the gaze origin, the position of the head-mounted device, etc. Go to Figure 1B The angle 150° from palm upward to head is shown as the angle between the palm-to-head vector 135° and the palm normal 140°. Similarly, as... Figure 2B As shown, the angle 250 from palm up to head indicates the angle between the palm normal 240 and the palm-to-head vector 235.

[0049] According to one or more embodiments, determining the relative characteristics of the hand and head may include, as shown in box 440, determining the palm-forward to head y-angle based on the palm-forward vector and the head vector. According to one or more embodiments, the palm-forward to head y-angle may indicate whether the hand is bent or pointing. In other words, the palm-forward to head y-angle may indicate when the hand is performing an extreme bending motion or other posture that can be used to block or cancel user input. Go to Figure 1B The palm facing forward to the head at an angle of 160° is shown as the head vector 125° (which has been derived from the source). Figure 1A The angle between the user's head position (transposed) and the forward vector of the palm (145°). Similarly, as... Figure 2B As shown, the y-angle 260° from the palm forward to the head indicates the angle between the palm forward vector 245° and the head vector 225°.

[0050] According to one or more embodiments, determining the relative characteristics of the hand and head may include, as shown in box 445, determining the palm-up to head y-angle based on the palm normal vector and the head vector. According to one or more embodiments, the palm-up to head y-angle may indicate the degree to which the palm faces upward. Go to Figure 1B The angle y from palm up to head 155 is shown as head vector 125 (which has been derived from the source). Figure 1A The angle between the user's head position (transposed) and the palm normal at 140 degrees. Similarly, as... Figure 2B As shown, the angle 255 from the palm upward to the head indicates the angle between the palm normal 240 and the head vector 225.

[0051] Hand orientation status determined

[0052] According to one or more implementations, various parameters related to geometric characteristics can be used in combination to determine whether a user is allowed to input a gesture. In some implementations, one or more state machines are used to determine whether a user is allowed to input a gesture. Figure 5 A hand orientation state machine for determining the palm position state is shown according to one or more embodiments.

[0053] According to one or more embodiments, the hand orientation state machine 500 is configured to perform a preliminary check of the hand posture state based on various parameters. In some embodiments, candidate hand orientation states may include a palm-up state 502, a palm-flipped state 506, and an invalid state 504 (where the hand posture is neither palm-up nor palm-flipped). According to one or more embodiments, the hand orientation state may begin with a hand orientation state based on the hand posture.

[0054] According to one or more embodiments, as shown at 510, the hand orientation state can transition from an invalid state 504 to a palm-up state 502 based on the palm-up to head angle. If the palm-up to head angle is less than a first threshold angle, and the palm-forward to head y-angle is less than a second threshold angle, the hand orientation state can transition from an invalid state 504 to a palm-up state 502. As shown at 510, the hand orientation state can transition from an invalid state 504 to a palm-up state 502. However, in some embodiments, the hand orientation state may not transition from an invalid state 504 to a palm-flipped state 506. In other words, to transition from an invalid state 504 to a palm-up state 502, the palm normal vector, the palm-forward vector, the palm-to-head vector, and / or the head vector can be considered. In some embodiments, the palm-up to head angle is considered, which is a measure indicating the degree to which the hand faces the eyes, head, or camera. Additionally, the palm-forward to head y-angle can be considered, which indicates the degree to which the hand is bent or pointing downwards. In some implementations, the angle from palm up to head is compared with a first threshold angle, and the angle from palm forward to head (y) is compared with a second threshold angle (which may be the same as or different from the first threshold angle).

[0055] As an example, return to Figure 1B When the hand is in a palm-up position, both the 150° angle from palm-up to head and the 160° angle from palm-forward to head are less than 45 degrees. In contrast, when turned... Figure 2B The palm-up to head angle 250° and the palm-up to head y-angle 260° are both at least 90 degrees. Therefore, the palm position in Figure 1 may indicate a palm-up to head angle 150° and a palm-forward to head y-angle 160° that satisfy the threshold at 510 of the hand orientation state machine 500.

[0056] Alternatively, as shown at 515, the hand orientation state can transition from a palm-up state 502 to an invalid state 504 based on the palm-forward to head y-angle being greater than a threshold. Similarly, as shown at 530, the hand orientation state can transition from a palm-flipped state 506 to an invalid state 504 based on the palm-forward to head y-angle being greater than a threshold. This can occur, for example, based on the pointing direction of the hand, such as when the hand is pointing downwards, because the hand is bent or because the hand is inverted. In other words, to transition to invalid state 504, the palm-forward vector and / or the head vector can be considered.

[0057] As an example, return to Figure 1B When the hand is in a palm-up position, the y-angle from the palm forward to the head is approximately 45 degrees (160 degrees). In contrast, turning... Figure 2BThe y-angle from the palm forward to the head is approximately 80 degrees (260°). Therefore, the palm positions in Figures 1 and 2 may indicate the palm-forward-to-head y-angle that satisfies the threshold at 530 of the hand orientation state machine 500.

[0058] According to one or more embodiments, as shown at 525, the hand orientation state can transition from a palm-up state 502 to a palm-flipped state 506 based on the palm-up-to-head angle and the palm-up-to-head y-angle. In other words, to transition from a palm-up state 502 to a palm-flipped state 506, the palm normal vector, the palm-to-head vector, and / or the head vector can be considered. In some embodiments, the palm-up-to-head angle is considered, which is a measure of the degree to which the hand faces the eyes, head, or camera. Additionally, the palm-up-to-head y-angle can be considered, which indicates the degree to which the palm normal plane is facing upwards. In some embodiments, the palm-up-to-head angle is compared to a first threshold angle, and the palm-forward-to-head y-angle is compared to a second threshold angle (which may be the same as or different from the first threshold angle). Furthermore, each threshold may be the same as or different from the thresholds considered at steps 510, 515, and 530. If the angle from palm up to head is greater than the first threshold angle, and the y-angle from palm up to head is greater than the second threshold angle, then the hand orientation state can change from palm up state 502 to palm flipped state 506.

[0059] As an example, return to Figure 1B When the hand is in a palm-up position, both the 150° angle from palm-up to head and the 155° angle from palm-up to head are less than 30 degrees. In contrast, when turned... Figure 2B The y-angle from palm up to head is 155 degrees, and the y-angle from palm up to head is 250 degrees, both of which are at least 100 degrees. Therefore, Figure 2B The palm position may indicate the palm-up to head angle 250 and palm-up to head y-angle 255 that satisfy the threshold at 525 of the hand orientation state machine 500.

[0060] Finally, as shown in box 520, the hand orientation state can transition from palm-flipped state 506 to palm-up state 502 based on the palm-up to head angle. In other words, to transition from palm-flipped state 506 to palm-up state 502, the palm normal vector and / or palm-to-head vector can be considered. In some embodiments, the palm-up to head angle, a measure indicating the degree to which the hand faces the eyes, head, or camera, is considered. In some embodiments, the palm-up to head angle is compared to a threshold angle, which may be the same as or different from other thresholds used in the hand orientation state machine 500. If the palm-up to head angle is less than the threshold angle, the hand orientation state can transition from palm-flipped state 506 to palm-up state 502.

[0061] As an example, return to Figure 1B When the hand is in a palm-up position, the angle from the palm up to the head is less than 30 degrees (150 degrees). In contrast, when turning... Figure 2B The palm-up to head angle 250° is at least 100 degrees. Therefore, the palm position in Figure 1 may indicate a palm-up to head angle 255° that satisfies the threshold at 520 of the hand orientation state machine 500.

[0062] Gesture detection status confirmed

[0063] According to one or more implementations, although the hand orientation state is determined without considering gaze, gaze vectors can be considered when determining the gesture detection state. Specifically, gaze vectors can be identified and used to determine whether gaze criteria are met. Typically, gaze criteria are met if the gaze is directed towards a region of interest, such as the area around the hand performing the gesture, a part of the environment where the virtual part is displayed, or the location where the virtual part is to be displayed.

[0064] Figure 6A A flowchart illustrating a technique for determining whether a gaze criterion is met, according to one or more embodiments, is shown. For illustrative purposes, the following steps will be described as being performed by a specific component. However, it should be understood that various actions may be performed by alternative components. Various actions may be performed in different orders. Furthermore, some actions may be performed simultaneously, and some actions may be unnecessary, or additional actions may be added.

[0065] Flowchart 600 begins at block 605, where gaze tracking data is obtained. For example, an eye-tracking system may include one or more sensors configured to capture image data or other sensor data, based on which the eye's viewing direction can be determined. Flowchart 600 proceeds to block 610, where a gaze vector is obtained from the gaze tracking data. According to one or more embodiments, the gaze vector can be obtained from the gaze tracking data, such as an inward-facing camera on a head-mounted device or other user-facing electronic device. The gaze tracking system may include one or more sensors configured to capture image data or other sensor data, based on which the eye's viewing direction can be determined.

[0066] In box 615, determine whether the gaze was recently directed at a user interface component or an area reserved for a user interface component. This might happen, for example, when the most recent instance of a gaze vector intersecting with a UI component area occurs within a threshold time period, such as if the user temporarily looks away. Determine the gaze target based on the gaze vector. For example, return to... Figure 1AThe gaze vector 130 is shown as originating from the eye position behind the electronic device 115 and pointing towards the UI component A120. Similarly, in Figure 2A In this context, gaze vector 230 points to UI component B 220. In some implementations, the gaze target can be a point in space relative to physical and / or virtual content in the extended reality environment. Within a threshold time period, the flowchart proceeds to box 620, and the threshold UI distance is adjusted. For example, if the user looks away, the UI area can be narrowed, making the gaze criterion more stringent. Figure 6B In the example shown, the target area may be associated with UI component 660. The target area may surround UI component 660 and / or may be based on the location where the UI component is to be presented, such as the anchor positioning of the UI component. This may happen, for example, if the UI component is to be presented based on the location of another component in the environment (such as the fingertips of a hand, a physical object in the environment, etc.). During a time period, the target area around the UI component may shrink from area 670 to area 665.

[0067] After adjusting the threshold UI distance in box 620, or if it is determined in box 615 that the gaze has not recently been targeting a UI component, flowchart 600 proceeds to box 625 and determines whether the gaze target is within the threshold UI distance. Figure 6B As shown, the threshold UI distance can be within region 665 or region 670, depending on whether the threshold UI distance is adjusted at box 620. If it is determined that the gaze target is within the current threshold UI distance of the UI component, the flowchart ends at box 635, and the gaze criterion is considered met.

[0068] Returning to box 625, if it is determined that the gaze target is not within the threshold UI distance, flowchart 600 proceeds to box 630, where it is determined whether the gaze target is within the threshold hand distance. For the hand 650, the hand region 655 can be determined in several ways. For example, the geometry of the hand or its surroundings can be determined in image data and compared with the gaze vector. As another example, hand tracking data can be used to obtain the skeleton of the hand, and it can be determined whether the gaze falls within a threshold location for the skeletal component for which its positional information is known. As an example, hand region 655 can be defined as a region consisting of bone length distances around each joint location, creating a bubble shape. If it is determined that the gaze target is within the threshold hand distance, the flowchart ends at box 635, and the gaze criterion is satisfied. However, if it is determined at box 630 that the gaze target is not within the threshold hand distance, such as hand region 655, the flowchart ends at box 640, and the gaze criterion is not satisfied.

[0069] In some implementations, additional considerations may be applied to determine whether gaze criteria are met. In some implementations, de-jitter parameters may be applied. For example, gaze signal stabilization may be required before determining whether gaze criteria are met or not. A de-jitter duration may be applied to determine whether gaze criteria are met. In some implementations, the de-jitter duration used to determine whether gaze criteria are met may differ from the de-jitter duration used to determine when gaze criteria are no longer met. Furthermore, in some implementations, the de-jitter duration may be adjusted based on gaze. For example, if the user significantly averts their gaze (i.e., beyond a threshold gaze angle or distance), the de-jitter time may be reduced.

[0070] The hand gesture state can be determined based on the gaze state (such as whether the gaze criteria are met) and the hand orientation state. Figure 7 A gesture detection state machine for determining a gesture detection state is illustrated according to one or more embodiments. As described above, a gaze criterion can be determined to be met if the user is looking at or near a hand or UI component (or, in some embodiments, an area where the UI component is to be displayed). For clarity, gesture detection state machine 700 indicates that the gaze criterion is met by the term "LOOKING" and indicates that the gaze criterion is not met by the term "NOT LOOKING". In some embodiments, candidate gesture states may include a palm-up state 702, a palm-flipped state 706, and an invalid state 704, wherein the gesture is neither in a palm-up nor palm-flipped state. Therefore, in some embodiments, once gaze is taken into account, the gesture detection state can be considered relative to the state described above. Figure 5 The gesture detection state is a refined state describing the hand orientation state. In other words, the gesture detection state can be an extension of the hand orientation state. Therefore, the gesture detection state machine 700 can be derived from... Figure 5 The hand orientation state machine 500 determines the initial state. In some implementations, other techniques may be used to determine the hand orientation state, such as whether the user is holding the controller, and the orientation of the controller relative to the head.

[0071] According to one or more embodiments, the gesture detection state can transition from palm-up state 702 to palm-flipped state 706 based on the hand posture being determined to be in a palm-flipped state (as shown at 725), regardless of gaze. Therefore, in some embodiments, gaze may not be considered when transitioning the gesture from palm-up to palm-flipped state. Similarly, at 720, palm-flipped state 706 can transition to palm-up state 702 based on the hand orientation state being palm-up. In other words, the transition between palm-up and palm-flipped states can be based on head and hand characteristics, such as the palm normal vector, the palm-to-head vector, and / or the head vector, regardless of the gaze vector. For this purpose, the gesture detection state can mirror the hand orientation state relative to the transition between palm-up and palm-flipped. In some embodiments, gaze may be considered. For example, it may be necessary to direct gaze towards the hand or UI component area to determine the state change. If the gaze target moves away from the UI component, the UI may be deactivated, and it may be necessary to re-engage the UI by looking at the hand.

[0072] The gesture detection state can transition from palm-flipped state 706 to invalid state 704 based on gaze and posture orientation, as shown at 730. In some embodiments, the gesture detection state can transition from palm-flipped state 706 to invalid state 704 if the gaze criterion is not met or if the posture is invalid. Similarly, the gesture detection state can transition from palm-up state 702 to invalid state 704 if the gaze criterion is not met or if the posture is invalid, as shown at 715. In other words, if the hand orientation state indicates an invalid posture, the gesture detection state will also be invalid. However, in some embodiments, the gesture state can also transition from palm-flipped state 706 or palm-up state 702 to invalid if the gaze criterion is not met.

[0073] If the hand orientation is palm-up and the gaze criterion is met, the gesture detection state can transition from invalid state 704 to palm-up state 702, as shown at 710. For example, if the hand orientation is determined to be palm-up from an invalid state (where the hand is upside down or otherwise pointing downwards), the gesture detection state will only transition to palm-up state 702 if the gaze criterion is met.

[0074] According to one or more embodiments, the gesture state may not support the transition from invalid state 704 to palm-flipped state 706. However, in some embodiments, the gesture detection state machine 700 can optionally support the transition from invalid state 704 to palm-flipped state 706. For example, as shown at 735, the gesture state can transition from invalid state 704 to palm-flipped state 706 if the hand orientation state is determined to be palm-flipped and the gaze is determined to have recently met the gaze criterion. This may occur, for example, if the user scans back and forth across UI components within a predefined time window.

[0075] Gesture activation status confirmed

[0076] According to one or more implementations, additional considerations may be used in determining whether to activate an input action associated with a hand gesture. These various inhibition rules or criteria can prevent, reject, and / or cancel user input actions. In some implementations, various parameters may be used to determine whether a user action that would cause a UI component to appear should be blocked. Examples may include detecting that a hand has recently moved or its position is otherwise unstable, or certain detected hand postures, such as whether the hand is in a pinching posture, where contact between two fingers is detected, or whether the tips of the index finger and thumb are close together, which may indicate that the user is holding an object. In these cases, a rejection reason can prevent the user input component from appearing, which would otherwise appear based on the hand gesture state.

[0077] As another example, the disengagement criteria can indicate parameters that prevent actions that would otherwise reveal or deactivate UI components. Examples could include determining that a hand has recently been in another gesture, such as hovering, touching, or pinching; determining that a hand is near the headset or otherwise within a predefined proximity of the user's head position; determining that a hand is obscured by an object (which could indicate, for example, that a hand is occupied); determining that two hands are close together; determining that a hand has recently been in a non-hand-anchored indirect pinch; or determining that the index finger is curled (which could indicate, for example, that the user is about to put their hand down). In these cases, satisfying the blocking criteria prevents the user input component from being revealed, which would otherwise be revealed based on the hand gesture state.

[0078] As another example, a rejection reason can be used to prevent a transition to a palm-flipped state and / or cancel an input action associated with a gesture. Examples may include determining that the wrist has recently moved or that the hand is obscured by an object. In these cases, a rejection reason can prevent the transition to a palm-flipped state and / or cancel the gesture, thus requiring a palm-up hand gesture reset.

[0079] The gesture activation state can be determined based on the gesture detection state and the inhibition criterion. Therefore, the gesture activation state can be considered an extension of the gesture detection state. Figure 8 A state machine for activating and suppressing gestures is illustrated according to one or more embodiments. In some embodiments, candidate gesture activation states may include a palm-up state 802, a palm-flipped state 806, and an invalid state 804, wherein the gesture is neither in a palm-up nor a palm-flipped state. The gesture activation state is used to determine whether to allow or suppress the user input action associated with the hand gesture. Therefore, the gesture activation state is used to activate the user input action associated with the corresponding state. Thus, in some embodiments, the gesture activation state machine 800 can be determined from... Figure 7 The gesture detection state machine 700 starts in the determined state.

[0080] According to one or more embodiments, the gesture activation state can transition from palm-up state 802 to palm-up state 806 based on the gesture detection state being determined to be in a palm-flipped state, as shown at 825. Therefore, in some embodiments, the suppression criterion may not be considered in the transition from the palm-up state to the palm-flipped state. Similarly, at 820, based on the gesture activation state being palm-up, palm-flipped state 806 can transition to palm-up state 802. In other words, without considering the suppression criterion (or, as mentioned above regarding...), the palm-flipped state 806 can transition to palm-up state 802. Figure 7 As described in the gesture detection state machine 700, in the case of gaze information, the transition between the palm-up state and the palm-flipped state can be based on head and hand characteristics, such as the palm normal vector, the palm-to-head vector, and / or the head vector. Furthermore, the transition between palm-up and palm-flipped can be based on the determined hand orientation state.

[0081] As shown at 830, the hand gesture state (in the gesture active state) can transition from palm-flipped state 806 to invalid state 804 based on gaze and hand posture information. In some embodiments, the hand gesture state can transition from palm-flipped state 806 to invalid state 804 if any release reason is true or any rejection reason is true. As described above, release reasons can indicate parameters that prevent these actions from displaying or deactivating UI components. Rejection reasons can be used to prevent transitions to the palm-flipped state and / or cancel input actions associated with the gesture. Similarly, as shown at 815, the hand gesture state can transition from palm-up state 802 to invalid state 804 if any release reason is true.

[0082] If the gesture detection state is palm-up and no inhibition reason is true, the gesture activation state can transition from invalid state 804 to palm-up state 802, as shown at 810. For example, if the gesture detection state is determined to be palm-up from invalid state 804 (where the hand is inverted or otherwise pointing downwards), the hand gesture state will transition to palm-up state 802 (i.e., the gesture activation state) only if no inhibition criterion prevents, rejects, or otherwise denies the transition.

[0083] According to one or more embodiments, the gesture activation state machine 800 may not support the transition from invalid state 804 to palm-flipped state 806. However, in some embodiments, the gesture activation state machine 800 may optionally support the transition from invalid state 804 to palm-flipped state 806. For example, as shown at 835, if the gesture state is determined to be palm-flipped and no rejection reason is true, the gesture activation state may transition from invalid state 804 to palm-flipped state 806. This may happen, for example, if a palm-up state is blocked due to a blocking reason such as a moving hand.

[0084] Once the gesture activation state is determined, the action corresponding to that state can be invoked. For example, if the gesture activation state is palm up (state 802), a first user input action can be executed, while if the gesture activation state is palm down (state 806), a second user input action can be executed. Furthermore, if the gesture activation state is invalid (state 804), no action can be executed, and optionally, the input action can be canceled based on the corresponding suppression reason.

[0085] Example electronic devices and related components

[0086] refer to Figure 9 A simplified block diagram of an electronic device 900 is depicted. The electronic device 900 may be part of a multifunctional device, such as a mobile phone, tablet computer, personal digital assistant, portable music / video player, wearable device, head-mounted system, projection-based system, base station, laptop computer, desktop computer, network device, or any other electronic system such as those described herein. For example, Figure 1A Electronic devices 115 or Figure 2AElectronic device 215 can be an example of electronic device 900. Electronic device 900 may include one or more additional devices, such as server equipment, base stations, auxiliary devices, etc., and various functions may be contained within or distributed across the one or more additional devices. Exemplary networks include, but are not limited to, local area networks (such as Universal Serial Bus (USB) networks), organizational LANs, and wide area networks (such as the Internet). According to one or more embodiments, electronic device 900 is used to interact with the user interface of application 955. According to one or more embodiments, application 955 may include one or more editing applications, or applications that otherwise provide editing functions such as tagging. It should be understood that various components and functions within electronic device 900 may be distributed differently across modules or components or even additional devices.

[0087] Electronic device 900 may include one or more processors 920, such as a central processing unit (CPU) or a graphics processing unit (GPU). Electronic device 900 may also include memory 930. Memory 930 may include one or more different types of memory that can be used in conjunction with processor 920 to perform device functions. For example, memory 930 may include cache, ROM, RAM, or any kind of transient or non-transitory computer-readable storage medium capable of storing computer-readable code. Memory 930 may store various programming modules for execution by processor 920, including a tracking module 945 and various other application programs 955. Electronic device 900 may also include storage device 940. Storage device 940 may include one or more non-transitory computer-readable media, including, for example, magnetic disks (fixed disks, floppy disks, and removable disks) and magnetic tape, optical media (such as CD-ROMs and digital video discs (DVDs)), and semiconductor memory devices (such as electrically programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM)). Storage device 940 may be used to store various data and structures that can be used to store data related to hand tracking and UI preferences. According to one or more embodiments, storage device 940 may be configured to store hand tracking network 975 and other data for determining hand movements, such as registration data 985. The electronic device may additionally include a network interface from which electronic device 900 can communicate over a network.

[0088] The electronic device 900 may also include one or more cameras 905 or other sensors 910, such as depth sensors that can determine the depth of a scene. In one or more embodiments, each of the one or more cameras 905 may be a conventional RGB camera or a depth camera. Furthermore, the cameras 905 may include stereo cameras or other multi-camera systems. Additionally, the electronic device 900 may include other sensors capable of collecting sensor data for tracking user movement, such as depth cameras, infrared sensors, or orientation sensors, such as one or more gyroscopes, accelerometers, etc.

[0089] According to one or more embodiments, memory 930 may include one or more modules comprising computer-readable code executable by processor 920 to perform functions. Memory 930 may include, for example, a tracking module 945 and one or more applications 955. Tracking module 945 may be used to track the position of the hand, arm, joints, and other indicators of user posture and / or movement in the physical environment. Tracking module 945 may use sensor data, such as data from camera 905 and / or sensor 910. In some embodiments, tracking module 945 may track user movement to determine whether user input is triggered based on a detected input gesture. In some embodiments described herein, tracking module 945 may be configured to determine hand input state, hand gesture state, and / or gesture activation state based on hand tracking data, head posture information, gaze information, etc. Electronic device 900 may optionally include display 980 or other device through which a user interface (UI) may be displayed or presented for user interaction. For example, the UI may be associated with one or more of applications 955. The display 980 can be an opaque display, or it can be semi-transparent or transparent, such as a transmissive or see-through display. The display 980 can incorporate LED, OLED, digital light projector, silicon-based liquid crystal, etc.

[0090] Although the electronic device 900 is depicted as including the numerous components described above, in one or more embodiments, the various components may be distributed across multiple devices. Therefore, although certain calls and transmissions are described herein with respect to the specific system depicted, in one or more embodiments, various calls and transmissions may be performed differently, or may be directed differently based on the functions of different distributions. Additionally, additional components may be used, or certain combinations of the functions of any components may be combined.

[0091] Now for reference Figure 10This document illustrates a simplified functional block diagram of an exemplary multi-functional electronic device 1000 according to one embodiment. Each of the electronic devices may be a multi-functional electronic device, or may have some or all of the components described herein. The multi-functional electronic device 1000 may include a processor 1005, a display 1010, a user interface 1015, graphics hardware 1020, device sensors 1025 (e.g., proximity sensor / ambient light sensor, accelerometer and / or gyroscope), a microphone 1030, an audio codec 1035, a speaker 1040, communication circuitry 1045, digital image capture circuitry 1050 (e.g., including a camera system), a video codec 1055 (e.g., supporting a digital image capture unit), a memory 1060, a storage device 1065, and a communication bus 1070. The multi-functional electronic device 1000 may be, for example, a digital camera or a personal electronic device, such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0092] Processor 1005 executes instructions necessary for the operation of implementing or controlling many functions performed by device 1000 (e.g., the generation and / or processing of images as disclosed herein). Processor 1005 may, for example, drive display 1010 and may receive user input from user interface 1015. User interface 1015 allows a user to interact with device 1000. For example, user interface 1015 may take various forms, such as buttons, keypad, dial pad, click wheel, keyboard, display screen, touchscreen, gaze, and / or gestures. Processor 1005 may also be, for example, a system-on-a-chip, such as those found in mobile devices, and may include a dedicated GPU. Processor 1005 may be based on a Reduced Instruction Set Computer (RISC) or Complex Instruction Set Computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. Graphics hardware 1020 may be dedicated computing hardware for processing graphics and / or assisting processor 1005 in processing graphics information. In one embodiment, graphics hardware 1020 may include a programmable GPU.

[0093] Image capture circuit 1050 may include two (or more) lens assemblies 1080A and 1080B, each lens assembly having a separate focal length. For example, lens assembly 1080A may have a shorter focal length relative to the focal length of lens assembly 1080B. Each lens assembly may have a separate associated sensor element 1090. Alternatively, two or more lens assemblies may share a common sensor element. Image capture circuit 1050 may capture still images and / or video images. Output from image capture circuit 1050 may be processed by video codec 1055 and / or processor 1005 and / or graphics hardware 1020 and / or dedicated image processing units or pipelines incorporated within circuit 1050. Images thus captured may be stored in memory 1060 and / or storage device 1065.

[0094] Sensor and camera circuitry 1050 can capture still images and video images that can be processed at least in part by devices according to this disclosure: video codec 1055 and / or processor 1005 and / or graphics hardware 1020, and / or a dedicated image processing unit incorporated within circuitry 1050. Captured images can be stored in memory 1060 and / or storage device 1065. Memory 1060 may include one or more different types of media used by processor 1005 and graphics hardware 1020 to perform device functions. For example, memory 1060 may include memory cache, read-only memory (ROM), and / or random access memory (RAM). Storage device 1065 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device configuration file information, and any other suitable data. Storage device 1065 may include one or more non-transitory computer-readable storage media, including, for example, magnetic disks (fixed disks, floppy disks, and removable disks) and magnetic tapes, optical media (such as CD-ROMs and DVDs), and semiconductor memory devices (such as EPROMs and EEPROMs). Memory 1060 and storage device 1065 can be used to tangibly hold computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed, for example, by processor 1005, such computer program code can implement one or more of the methods described herein.

[0095] The various processes defined in this document consider options for obtaining and utilizing a user's identifying information. For example, such personal information may be used to track a user's posture and / or movement. However, with regard to the collection of such personal information, it should be obtained with the user's informed consent, and the user should be aware of and control the use of their personal information.

[0096] Personal information will be used by the appropriate parties only for lawful and reasonable purposes. Parties using such information will comply with privacy policies and practices that are at least in accordance with applicable laws and regulations. Furthermore, such policies should be comprehensive and meet or exceed government / industry standards. In addition, parties may not distribute, sell, or otherwise share such information except for any reasonable and lawful purpose.

[0097] Furthermore, the intent of this disclosure is that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by restricting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Where appropriate, deidentification can be facilitated by removing specific identifiers (e.g., date of birth), controlling the amount or characteristics of stored data (e.g., collecting location data at the city level rather than address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods.

[0098] It should be understood that the above description is intended to be illustrative and not restrictive. Material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims and to provide that material in the context of a particular embodiment, variations of which will be readily apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in combination with each other). Therefore, Figures 3 to 6A and Figures 7 to 8 The specific arrangement of the steps or actions shown, or Figures 1 to 2, Figure 6B and Figures 9 to 10 The arrangement of the elements shown should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents. In the appended claims, the terms “comprising” and “therein” are used as common English equivalents of the corresponding terms “including” and “wherein”.

Claims

1. A method, the method comprising: Determine the geometric characteristics of the user's hand relative to their head when performing the gesture; Determine the user's gaze vector; The hand gesture state is determined from multiple candidate hand gesture states based on the user's gaze vector and the geometric characteristics of the hand relative to the head; as well as In response to determining that the hand gesture state corresponds to an input gesture, an action corresponding to the input gesture is invoked.

2. The method of claim 1, wherein determining the geometric characteristics of the user's hand relative to the head comprises: Obtain hand tracking data of the hand performing the gesture; as well as Obtain the user's head vector.

3. The method according to claim 1, wherein the candidate hand gesture state includes a palm-up state, a palm-flipped state, and an invalid state.

4. The method according to claim 1, wherein determining the hand gesture state includes: A hand orientation state is determined from multiple candidate hand orientation states based on the geometric characteristics of the hand relative to the head. as well as The hand gesture state is determined based on the hand orientation state and the gaze vector.

5. The method of claim 4, wherein the hand orientation state is determined using a hand orientation state machine based on one or more of the following: 1) a palm-up-to-head angle indicating the relative position of the user's palm toward the user's head, 2) a palm-forward-to-head y-angle indicating the direction in which the user's palm points relative to the user's head, and 3) a palm-up-to-head y-angle indicating the relative position of the palm toward the upward direction.

6. The method of claim 4, wherein the hand gesture state is determined using a gesture detection state machine and based on one or more of the following: 1) the hand gesture state; and 2) determining whether a gaze criterion is met.

7. The method of claim 6, wherein determining whether the gaze criterion is met comprises: Determine whether the target of the gaze vector is within a threshold distance of at least one of the hand, controller, and user input device.

8. The method according to claim 4, further comprising: Obtain controller tracking data; as well as The controller orientation is determined based on the controller tracking data. Determining the state of the hand gesture includes: The hand orientation state is determined from multiple candidate hand orientation states based on the geometric characteristics of the controller orientation relative to the head. as well as The hand gesture state is determined based on the controller orientation state and the gaze vector.

9. The method according to any one of claims 1 to 8, wherein invoking the action corresponding to the input gesture further comprises: The gesture activation state is determined based on the hand gesture state and one or more inhibition criteria.

10. A non-transitory computer-readable medium comprising computer-readable code, said computer-readable code being executable by one or more processors to: Determine the geometric characteristics of the user's hand relative to their head when performing the gesture; Determine the user's gaze vector; The hand gesture state is determined from multiple candidate hand gesture states based on the user's gaze vector and the geometric characteristics of the hand relative to the head; as well as In response to determining that the hand gesture state corresponds to an input gesture, an action corresponding to the input gesture is invoked.

11. The non-transitory computer-readable medium of claim 10, wherein the computer-readable code for determining the geometric characteristics of a user's hand relative to the head comprises computer-readable code for: Obtain hand tracking data of the hand performing the gesture; and Obtain the user's head vector.

12. The non-transitory computer-readable medium of claim 10, wherein the computer-readable code for determining the geometric characteristics of a user's hand relative to the head comprises computer-readable code for: Obtain controller tracking data from the controller held by the hand; and Obtain the user's head vector.

13. The non-transitory computer-readable medium of claim 10, wherein the candidate hand gesture state includes a palm-up state, a palm-flipped state, and an invalid state.

14. The non-transitory computer-readable medium of claim 10, wherein the computer-readable code for determining the hand gesture state comprises computer-readable code for: A hand orientation state is determined from multiple candidate hand orientation states based on the geometric characteristics of the hand relative to the head; and The hand gesture state is determined based on the hand orientation state and the gaze vector.

15. The non-transitory computer-readable medium of claim 14, wherein the hand orientation state is determined using a hand orientation state machine based on one or more of the following: 1) a palm-up to head angle indicating the relative position of the user's palm toward the user's head, 2) a palm-forward to head y-angle indicating the direction in which the user's palm points relative to the user's head, and 3) a palm-up to head y-angle indicating the relative position of the palm toward the upward direction.

16. The non-transitory computer-readable medium of claim 14, wherein the hand gesture state is determined using a gesture detection state machine and based on one or more of the following: 1) the hand gesture state; and 2) determining whether a gaze criterion is met.

17. The non-transitory computer-readable medium of claim 16, wherein the computer-readable code for determining whether the gaze criterion is satisfied comprises computer-readable code for: Determine whether the target of the gaze vector is within a threshold distance of at least one of the hand, controller, and user input device.

18. A system comprising: One or more processors; and One or more computer-readable media, the one or more computer-readable media including computer-readable code, the computer-readable code being executable by the one or more processors to: Determine the geometric characteristics of the user's hand relative to their head when performing the gesture; Determine the user's gaze vector; The hand gesture state is determined from multiple candidate hand gesture states based on the user's gaze vector and the geometric characteristics of the hand relative to the head; as well as In response to determining that the hand gesture state corresponds to an input gesture, an action corresponding to the input gesture is invoked.

19. The system of claim 18, wherein the computer-readable code for determining the geometric characteristics of a user's hand relative to the head comprises computer-readable code for: Obtain hand tracking data of the hand performing the gesture; and Obtain the user's head vector.

20. The system of claim 18, wherein the computer-readable code for determining the hand gesture state comprises computer-readable code for: A hand orientation state is determined from multiple candidate hand orientation states based on the geometric characteristics of the hand relative to the head; and The hand gesture state is determined based on the hand orientation state and the gaze vector.

Images