Active and inactive mode transitions for user input
By marking the hand as active or inactive mode in the XR environment and identifying gesture input using different parameters, the problem of unexpected input gestures is solved, and the responsiveness and experience of user interaction is improved.
Patent Information
- Application Number
- CN202410698230.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-28
- Filing Date
- 2024-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
In the existing Extended Reality (XR) environment, gesture recognition systems are difficult to effectively manage unexpected input gestures, resulting in reduced responsiveness and poor user experience.
By marking the hand as in active or inactive mode and identifying gesture input using parameters associated with different modes, reducing the number of gestures entered by accident and improving responsiveness to user input gestures.
By reducing unexpected input actions when inactive, the responsiveness of user interaction is improved, the incidence of unexpected input gestures is reduced, and the user experience is improved.
Smart Images

Figure CN119065575A_ABST
Abstract
Description
Background Art
[0001] Some devices can generate and present extended reality (XR) environments. XR environments can include fully or partially simulated environments that people perceive and / or interact with via electronic systems. In XR, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with real-world properties. Some XR environments allow multiple users to interact with virtual objects or with each other in the XR environment. For example, a user can use gestures to interact with components of the XR environment. However, an improved technique is needed to manage the tracking of hands performing gestures. BRIEF DESCRIPTION OF THE DRAWINGS
[0002] Figure 1A to Figure 1B An example diagram illustrating processing gesture-based user input using active and inactive modes according to one or more embodiments.
[0003] Figure 2 A flowchart of a technique for detecting input gestures according to some embodiments is shown.
[0004] Figure 3 A flowchart of a technique for determining the intentionality of a gesture according to some embodiments is shown.
[0005] Figure 4 A flow diagram of an action network is shown according to some embodiments.
[0006] Figure 5 A flowchart of a technique for processing gesture-based user input using active and inactive modes is shown according to some embodiments.
[0007] Figure 6 A flow diagram of a technique for transitioning between an active mode and an inactive mode is shown according to one or more embodiments.
[0008] Figure 7 A system diagram of an electronic device that can be used for gesture recognition according to one or more embodiments is shown.
[0009] Figure 8 Exemplary systems for use in various extended reality techniques are shown. DETAILED DESCRIPTION
[0010] The present disclosure relates to systems, methods, and computer-readable media for implementing gesture recognition and input. In some augmented reality scenarios, image data and / or other sensor data may be used to detect gestures by tracking hand data. However, a user using gesture-based input may inadvertently perform gestures that may be recognized as user input. To reduce the number of accidental input gestures, hands may be marked as being in an active mode or an inactive mode. Different modes may be associated with different parameters by which gesture input is recognized.
[0011] In some embodiments, each hand can be marked as active or inactive based on various factors. These factors can include specific detected trigger events or conditions that cause the hand to transition from an inactive state to an active state. Similarly, certain detected trigger events or conditions can cause the hand to transition from an active state to an inactive state. The active state and the inactive state can be associated with different gesture recognition parameters. That is, the parameters by which a gesture is recognized by a hand in an active state can be more easily met than the parameters by which a gesture is recognized by a hand in an inactive state.
[0012] The embodiments described herein provide an improved gesture-based user interface by reducing the likelihood of initiating input actions when the hand is in an inactive state compared to an active state. By doing so, responsiveness to user input gestures is improved when the user is most likely to interact with the gesture-based user interface. At the same time, accidental input is reduced by limiting the actions that produce gesture-based input during the inactive state.
[0013] In the following disclosure, a physical environment refers to the physical world that people can perceive and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense the physical environment and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, an XR environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset or representation of a person's physical motion is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. As an example, the XR system can detect head movement, and in response, adjust the graphics content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting an XR environment and adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust characteristics of graphical content in an XR environment in response to representations of physical movement (e.g., voice commands).
[0014] There are many different types of electronic systems that enable a person to sense and / or interact with various XR environments. Examples include: head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing the image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some specific implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.
[0015] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the disclosed concept. As part of this description, some of the drawings in the drawings of the present disclosure represent structures and devices in the form of block diagrams to avoid blurring the novel aspects of the disclosed concept. For the sake of clarity, all features of the actual specific implementation may not be described. In addition, as part of this specification, some drawings of the present disclosure may be provided in the form of flow charts. The blocks in any particular flow chart may be presented in a specific order. However, it should be understood that the specific order of any given flow chart is only used to illustrate an embodiment. In other embodiments, any element in the various elements depicted in the flow chart may be deleted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. In addition, other embodiments may include additional steps that are not shown as a part of the flow chart. In addition, the language used in the present disclosure has been selected primarily for readability and guidance purposes, and may not be selected to delimit or limit the subject matter of the present invention, or to resort to the necessary claims to determine such subject matter of the invention. Reference in this disclosure to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter, and multiple references to "one embodiment" or "an embodiment" should not be construed as necessarily all referring to the same embodiment.
[0016] It should be understood that in the development of any actual implementation (as in any software and / or hardware development project), many decisions must be made to achieve the developer's specific goals (e.g., to meet system and business-related constraints), and that these goals may vary from one implementation to another. It should also be understood that such development work may be complex and time-consuming, but is nonetheless a routine task for those of ordinary skill in designing and implementing a graphical modeling system having the benefit of this disclosure.
[0017] Figure 1AA diagram of a technique for processing gesture-based user input using an active mode and an inactive mode according to one or more embodiments is shown. In particular, in a first view 100A, a user 105 is shown attempting to select a gesture. Device 115 can obtain sensor data of the user's hand 110A. It should be understood that, although user interface 120 is depicted as being separated from device 115, in some embodiments, user interface 120 can be presented on a display of device 115. In some embodiments, device 115 can use hand tracking or other vision-based tracking networks or processes. That is, electronic device 115 can have one or more cameras or other sensors configured on or in the device in a manner that captures an image of a hand or other body tracking data (e.g., position depth on a hand). Electronic device 115 can be a mobile device, such as a wearable device with a camera and / or other sensors facing the user's hand.
[0018] According to some embodiments, the device can use sensor data to detect user input gestures and process user input accordingly. In some embodiments, the device 115 can determine the input action associated with the detected user input gesture based on hand posture and / or other tracking data. In some embodiments, the device 115 can determine whether the hand posture and / or other tracking data should trigger the input action based on the activity mode or inactivity mode of a specific hand. That is, the device 115 can determine the mode for the hand (or in some embodiments, for each hand). Whether the hand is in an active state or in an inactive state can make the device detect or determine whether the user input action should be triggered based on the corresponding criterion set. As described above, the device can use different parameters to determine whether the hand posture that triggers the input action is detected when the hand is in an inactive state and when the hand is in an active state. However, the parameters for triggering the input action may be more stringent in the inactive mode than in the active state. In other words, when in an active state or an inactive state, the user input action can be triggered by a hand posture. However, in an inactive state, the parameters by which the user input action is triggered are more difficult to meet than when the hand is in an active state.
[0019] As will be described below, various acceptor and rejector actions may be considered when determining whether a hand is in an active mode or an inactive mode. For the purposes of this example, hand 110A is in an active state. In this example, hand 110A is performing a pinch gesture, which is determined to be a valid input gesture for triggering a user input action, as shown at 140A. Thus, user input component 125A is shown as being selected.
[0020] Go to Figure 1B, which illustrates an alternative diagram of a technique for processing gesture-based user input using active and inactive modes according to one or more embodiments. In particular, in the second view 100B, the user 105 is shown attempting to select a gesture. The device 115 may capture the user's hand 110B in contact with Figure 1A Sensor data at different positions of the hand 110A. Figure 1A In the embodiment, it should be understood that although the user interface 120 is depicted as being separate from the device 115, in some embodiments, the user interface 120 can be presented on the display of the device 115. In some embodiments, the device 115 can be a device that can use hand tracking or other vision-based tracking networks or processes. That is, the electronic device 115 can have one or more cameras or other sensors configured on or in the device in a manner that allows the capture of images of the hands or other body tracking data. The electronic device 115 can be a mobile device, such as a wearable device with a camera and / or other sensors facing the user's hand.
[0021] According to some embodiments, device 115 can use sensor data to detect user input gestures and process user input accordingly. In some embodiments, device 115 can determine the input action associated with the detected user input gesture based on hand posture and / or other tracking data. As described above, the device can use a parameter set to determine whether a hand posture that triggers an input action is detected when the hand is in an inactive state, and the parameter set is more difficult to meet than the parameter set used to determine whether the hand posture triggers the input action when the hand is in an active state. In other words, when in an active state or an inactive state, the user input action can be triggered by a hand posture. However, in an inactive state, the parameters by which the user input action is triggered are more stringent than when the hand is in an active state. In other words, the hand tracking data makes the probability of the input action being triggered greater in an active state than in an inactive state.
[0022] As will be described below, various acceptor and rejector actions may be considered when determining whether a hand is in active mode or in inactive mode. For the purposes of this example, hand 110B is in inactive mode. Examples of rejection actions that cause a hand to be considered in inactive mode include a determination that the user is using a peripheral device (e.g., based on a hand posture or input detected at a peripheral device), a self-interaction type posture, a timeout event from the last valid gesture input action, etc. In addition, in some embodiments, a user may actively enter inactive mode through a specific gesture or other predefined user input action or event. Other examples of rejection include: for a pinch gesture, detecting that the palm is facing down and a peripheral event is received, detecting that one or more hands are palm-down in a peripheral use posture, detecting that the hand has fallen to a static posture, or a predefined rejector action such as when the thumb and index finger slide along each other. Any or some combination of these rejector actions will cause the hand to be placed in inactive mode, and cause corresponding gestures such as pinching to be rejected.
[0023] In this example, hand 110B is performing a pinch gesture, as shown at 140B. For the purposes of this example, the hand's posture is on the user side, which can be considered a posture consistent with the use of a peripheral device, which can cause device 115 to mark hand 110A as inactive. Because the hand is in inactive mode, the pinch gesture is not determined to be a valid input gesture for triggering a user input action. As a result, user input component 125B remains unselected.
[0024] Figure 2 A flow chart of a technique for detecting input gestures according to some embodiments is shown. In particular, Figure 2 A gesture estimation pipeline is shown that recognizes and processes user input gestures. Although the flowcharts show various components that are described as performing a particular process, it should be understood that the flow of the diagrams may be different according to some embodiments, and the functions of the components may be different according to some embodiments.
[0025] Flowchart 200 begins with sensor data 202. In some embodiments, the sensor data may include captured image data and / or depth data of one or both hands of a user. In some embodiments, the sensor data may be captured from a sensor on an electronic device, such as an outward-facing camera on a head-mounted device, or a camera otherwise configured in an electronic device to capture sensor data including the user's hands. According to one or more embodiments, the sensor data may be captured by one or more cameras, which may include one or more sets of stereo cameras. In some embodiments, sensor data 202 may include additional data collected by the electronic device and related to the user. For example, the sensor data may provide position data of the electronic device, such as the position and orientation of the device.
[0026] In some embodiments, sensor data 202 may be applied to a hand tracking network 204. The hand tracking network may be a network trained to estimate the physical state of one or both hands of a user. In some embodiments, hand tracking network 204 predicts hand poses 206. The hand poses may be classified poses of the hand based on the estimated physical state, or some other form of data indicating the pose of the hand may be provided. For example, in some embodiments, hand pose data 206 may include estimates for the joint positions of the hand. Additionally, in some embodiments, hand tracking network 204 may be trained to provide estimates of device (such as a headset) position and / or estimates of simulated world space.
[0027] In some embodiments, the hand tracking network 204 can be further configured to provide touch data. The touch data can include a prediction of whether a touch is occurring between two areas on the hand for one or more given frames. For example, a machine learning model can be trained to predict whether a thumb pad and an index finger are in contact. For the purposes described herein, a touch refers to contact between two surfaces regardless of intent, and a pinch is defined as a touch performed with the intent to produce a corresponding input action. As will be described in more detail below, in some embodiments, hand tracking can predict whether a touch occurs based on the sensor data 202 and / or the hand gesture data 206.
[0028] According to one or more embodiments, the gesture determination framework 210 provides a determination as to whether a particular gesture presented in the sensor data 202 is intentional. That is, a determination is made as to whether a classified gesture of the hand (e.g., based on or provided by the hand gesture data 206) is intentional. When the determined hand gesture includes a touch (such as a pinch), then the gesture determination framework 210 can use the touch signal 208 provided by the hand tracking network 204 in determining whether an intentional gesture is performed.
[0029] In some embodiments, the gesture determination framework 210 can utilize additional data not explicitly depicted in FIG. 1 . For example, the gesture determination framework 110 can receive signals such as user interface (UI) geometry, line of sight estimates, events generated by connected peripherals, user interactions with objects, etc. As will be described in Figure 3 As described in , gesture determination framework 110 can consider various features from the input to make a determination for a particular input gesture whether the gesture is intentional. This determination can be transmitted to UI gesture processing module 114 in the form of gesture signal 112. The gesture signal can indicate whether an intentional input gesture has occurred. In some embodiments, gesture signal 112 can also be used to indicate whether a previous gesture signal should be canceled. For example, if the user moves their position, puts down their hand, etc., this cancellation may occur.
[0030] According to one or more embodiments, hand gesture data 206 and / or touch signal 208 can be determined based on a set of probes, as will be described in more detail below. These probes can be used to determine whether hand gesture 206 is associated with a user input gesture. In some embodiments, multiple parameter sets or parameter probes can be used to determine whether hand gesture is associated with a user input gesture. The specific parameters used to determine whether a hand gesture is associated with a user input gesture can be based on whether the associated hand is in an active mode or in an inactive mode. The device can assign one or both hands to an active mode or an inactive mode based on various acceptor or rejector actions. In general, the device assigns a model to one or both hands based on the determination that the hand is actively engaged with the user interface through gesture-based input. This can be done based on certain predefined events (e.g., acceptor events) or contextual prompts from the user and / or UI.
[0031] Examples of acceptor events include, for example, detecting a nominal gesture tap. This can include a short duration, static pinch. For example, a nominal gesture tap can be detected as occurring within a threshold time frame and including a pinch that is less than a threshold movement. Another example of an acceptor event includes the detection of intentional scrolling. Intentional scrolling can be detected, for example, based on the speed at which the fingers come together to form a pinch and when the hand moves over a threshold distance while the fingers are still pinched. A third example of an acceptor event includes a very fast pinch with a strong pinch style. A strong pinch style can be defined by a motion defined as a typical pinch motion. A fourth example of an acceptor event includes repeated actions, such as repeated pinches. In some embodiments, repeated pinches can be detected when two pinches are detected in a continuous manner within a threshold distance and an associated gaze is optionally guided within a threshold distance. In some embodiments, an acceptor event may not be affected by a hand mode. That is, a gesture that satisfies an acceptor event will trigger a user input action, regardless of whether the hand is in an active or inactive state.
[0032] In some embodiments, each of these example acceptor events can trigger a corresponding user input action when the hand is in an active state and when the hand is in an inactive state. However, the parameters used during the inactive mode in which an event is detected to trigger a user input action may be more difficult to meet than the parameters used during the active mode. As an example, for a nominal gesture tap, the time frame and / or threshold movement may be more difficult to meet in the inactive mode, so that in order to trigger a user input action, the nominal gesture tap can meet a narrower definition of parameters. Similarly, in the case of intentional scrolling, the required speed and / or threshold distance may be different in the active mode and the inactive mode in order to trigger the user input scroll. In addition, for fast pinching, pinching may require a better pinching style to be defined in the inactive mode than in the active mode. In some embodiments, the parameters used in the inactive mode may rely on additional data in addition to the parameters used during the active mode. As an example, a well-defined pinch can trigger a user input action in the active mode, but in the inactive mode, the gaze tracking data must indicate that the user is gazing at the user input component.
[0033] In some embodiments, the heuristics used to determine the hand pose 206 or the touch signal 208 can be dynamically modified based on a given state of the hand. Additionally or alternatively, the gesture determination framework 210 can consider a particular state of the hand for determining whether a gesture triggers a user input action and thereby provide an appropriate gesture signal 212. In particular, in some embodiments, the state of the hand can be considered when determining the intentionality of a gesture.
[0034] The UI gesture processing module 214 can be configured to enable user input actions based on the gesture signal 212. Certain gestures such as pinching can be associated with selection actions of UI components, etc. In some embodiments, if a cancel signal corresponding to the initiated gesture signal 212 is received, the system can process the gesture differently than if the gesture was not canceled. For example, the UI component can be shown as selected but not activated, etc. As another example, a previously initiated tap caused by the user can be truncated or canceled.
[0035] As described above, the gesture determination framework may be configured to generate an intentionality classification of a gesture.The gesture determination framework 210 may be configured to estimate a hand pose or gesture and determine whether the gesture is intended to trigger a user input action. Figure 3 A flow chart of a technique for classifying the intentionality of a gesture according to some embodiments is shown. For purposes of explanation, the following steps will be described as being performed by Figure 2However, it should be understood that various actions can be performed by alternative components. Various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions may be added.
[0036] Flowchart 300 begins at box 305 where a touch is detected based on contextual data from a hand tracking network. The touch may be detected, for example, based on a touch signal 208 received from hand tracking network 204. According to some embodiments, some gestures may require a touch, such as a pinch, etc. In addition, different types of touches may be utilized to identify multiple types of pinches. According to some embodiments, not every gesture requires a touch. Therefore, a touch may not be detected, or a touch signal may indicate that a touch occurred. In some embodiments, a touch signal may not be received or may be otherwise ignored, and the gesture may still be recognized.
[0037] Flowchart 300 continues to box 310, where the touch phase is determined based on the hand tracking data. The touch phase can indicate what stage of the touch action the finger is currently in for a given frame. According to some embodiments, the features of interest in determining intentionality can vary depending on the current state of the gesture. For gestures that include pinching or other touch actions, the stage that the gesture is currently in may affect the ability to enable, cancel or reject the associated input action. Some examples of touch phases include an idle state, an entry state where a touch event is beginning, such as a downward pinch phase. A hold state (where a pinch is currently occurring) and an exit phase (e.g., when an upward pinch occurs so that the pinch ends).
[0038] At box 315, estimate low-level features associated with the touch. The low-level features can be determined based on the hand tracking data, and / or the additional data may include an estimate of what the hand is doing during the frame. For example, other data sources include posture information of the device used to capture the hand tracking data, hand posture, UI geometry, gaze estimation, etc. In some embodiments, the low-level features are determined without considering intent. Examples of low-level features include, for example, pinch speed when pinching down, measurement of wrist flexion, finger curling, proximity of the hand to the head, speed of the hand, etc.
[0039] Flowchart 300 continues to box 320, where high-level low-state features are estimated. High-level low-state features may include modal features that estimate what the user is doing during the touch in order to determine intentionality. In some embodiments, high-level features may be interoperable and individually verifiable features. Examples include estimates of whether the hand is using one or more peripherals, the frequency of repetition of gestures (e.g., in the case where the user is repeatedly pinching), whether the hand is holding an object, whether the hand is in a resting position, a specific pinch or gesture style (i.e., a pinch using the pad of two fingers or using the side of a finger). In some embodiments, high-level features may be based on user activity, such as the user fidgeting, talking, or reading. According to one or more embodiments, high-level features may be determined based on hand tracking data, determined touch phases, and / or estimated basic features. In some embodiments, high-level features may directly determine the intentionality of the action. For example, if the user is using a peripheral device such as a keyboard, the pinch may be rejected, or the gesture may be determined to be unintentional.
[0040] According to one or more embodiments, advanced features can be used to determine various acceptor and rejector events. As described above, examples of acceptor events include, for example, detection of a nominal gesture tap, detection of an intentional scroll, very fast pinching with a strong pinching style, and repetitive actions such as repeated pinching. These acceptor actions indicate that the hand should be in an active mode. Examples of rejector events include determining a self-interaction, which can be detected when the hand is close to another part of the user (such as an arm or other part of the body). By determining that the hand is close to another part of the user, it can be determined that the hand is performing a self-interaction and the hand can be considered to be in an inactive mode. As another example, advanced features can indicate that the user may be in a peripheral use mode based on a user gesture or user input or other signal. The determination that the user is in a peripheral user mode can trigger the hand to be placed in an inactive mode.
[0041] The flowchart ends at box 325, where the gesture determination framework 210 combines the high-level features and the touch stage to classify intentionality. In some embodiments, the gesture determination framework 210 uses a conditional combination of high-level features and the touch stage to classify intentionality. This classification can then be used to signal a gesture to be processed as an input gesture (thereby activating an associated UI input action), cancel an associated action if the gesture is determined to be unintentional (e.g., if a UI action associated with the gesture has already been initiated), or ignore the gesture.
[0042] Figure 2 The touch signal 208 can be determined in a variety of ways. For example, in some embodiments, a heuristic method can be used based on hand tracking data to determine whether a touch has occurred and / or the current touch stage. Figure 4 A flow diagram of an action network providing an example machine learning process for determining whether a touch has occurred is shown according to some embodiments.
[0043] The pipeline 400 begins with a set of frames 402 as input. The frames 402 can be a time sequence of image frames of a hand captured by one or more cameras. The cameras can be single cameras, stereo cameras, cameras whose camera exposures have been synchronized, or a combination thereof. The cameras can be located on a user's electronic device such as a mobile device or a head mounted device. The frames can include a series of one or more frames associated with a predetermined time. For example, the frames 402 can include a series of individual frames captured at consecutive times, or can include multiple frames captured at each time in the consecutive time. The entirety of the frames can represent a sequence of motions of the hand, according to which a touch may or may not be detected at any particular time.
[0044] Frame 402 may be applied to gesture model 404. Gesture model 404 may be a trained neural network configured to predict a 3D pose 408 of a hand based on a given frame (or set of frames, such as in the case of a stereo camera) for a given time. That is, each frame of frame set 402 may be applied to gesture model 404 to generate 3D gesture 408. Thus, the gesture model may predict the pose of the hand at a particular point in time. In some embodiments, geometric features 412 may be derived from 3D gesture 408. The geometric features may indicate relational features between joints of the hand that may be identified by the 3D gesture. That is, in some embodiments, 3D gesture 408 may indicate the positioning and location of joints in the hand, while geometric features 412 may indicate spatial relationships between joints. As an example, geometric features 412 may indicate the distance between two joints, etc.
[0045] In some embodiments, frame 402 may be additionally applied to an encoder 406 that is trained to generate potential values for a given input frame (or multiple given input frames) from a specific time indicating the appearance of the hand. Appearance features 410 may be features that can be identified from frame 402 but are not particularly useful for gestures. Thus, these appearance features may be ignored by gesture model 404, but may be used within pipeline 400 to determine whether a touch has occurred. For example, appearance features 410 may be complementary features of geometric features 412 or 3D gestures 408 to advance the goal of determining a specific action 420 (such as whether a touch has occurred). According to some embodiments, encoder 406 may be part of a network associated with gesture model 404, so that the encoder may use some of the gesture data to predict appearance features. In addition, in some embodiments, 3D gestures 408 and appearance features 410 may be predicted by a single model or two separate, unrelated models. The result of encoder 406 may be, for example, a set of appearance features 410 in the form of a set of potential values.
[0046] The fusion network 414 is configured to receive the geometric features 412, the 3D pose 408, and the appearance features 410 as inputs and generate a set of encodings 416 at a time. The fusion network 414 can combine the geometric features 412, the 3D pose 408, and the appearance features 410 in any number of ways. For example, the various features can be weighted differently in the combination, or otherwise combined in different ways to obtain a set of encodings 416 at a time.
[0047] The encoding is then run through the temporary network 418 to determine the action 420 each time. The action 420 may indicate, for example, whether a touch has occurred or a change in the stage of touch. The temporary network 418 may consider both the frame (or set of frames) at the specific time for determining the action 420 as well as the other frames in the frame set 402.
[0048] Figure 5 A flow chart of a technique for processing gesture-based user input using active and inactive modes according to some embodiments is shown. In particular, Figure 5 The flowchart presented in depicts an example technique for detecting and processing gestures, as described above with respect to FIG. 1. For purposes of explanation, the following steps will be described as being performed by specific components. However, it should be understood that the various actions may be performed by alternative components. Furthermore, the various actions may be performed in a different order. In addition, some actions may be performed simultaneously, and some actions may not be required, or other actions may be added.
[0049] Flowchart 500 begins at box 505, where a gesture is detected. In some embodiments, the device may select a gesture from a set of predefined gesture classifications based on the posture. In some embodiments, the gesture may be based on a hand posture in a single frame (or in some embodiments, a stereo frame pair), over a series of frames, etc. As described above, the gesture may be determined, for example, by a gesture determination framework 210. In some embodiments, gestures may be detected based on hand postures and / or touch signals received from a hand tracking pipeline. The hand tracking pipeline may be configured to obtain hand tracking data from one or more sensors on the device. The hand tracking data may be obtained from one or more cameras (including stereo cameras, etc.). In some embodiments, the hand tracking data may include sensor data captured by an outward-facing camera of the head-mounted device.
[0050] Flowchart 500 continues at block 510 where acceptors and rejectors are calculated. In some embodiments, when a gesture is detected, a determination is made as to whether there is any acceptor action or rejector action. However, the calculation of acceptors and rejectors may be performed at different times. According to some embodiments, the acceptors and rejectors may be the same or may be different for active mode and inactive mode.
[0051] Acceptors can include events that indicate that a particular hand or hands are in an active mode. Acceptor events include, for example, detection of a nominal gesture tap, detection of an intentional scroll, very fast pinches with a strong pinch style, and repetitive actions such as repeated pinches. These acceptor actions indicate that the hand should be in an active mode. Examples of rejector events include determining a self-interaction, which can be detected when a hand is close to another part of the user (such as an arm or other part of the body). By determining that the hand is close to another part of the user, it can be determined that the hand is performing a self-interaction and the hand can be considered to be in an inactive mode. As another example, a high-level feature can indicate that a user may be in a peripheral use mode based on a user gesture or user input or other signal. A determination that the user is in a peripheral user mode can trigger placing the hand in an inactive mode.
[0052] The flowchart proceeds to box 515, where a determination is made as to whether the gesture is rejected. That is, the attempted gesture at box 505 can be rejected based on the detected rejector action. If the gesture is rejected, the flowchart ends at box 520 and the gesture is canceled or rejected. If an input action for the gesture has not yet begun, the gesture is ignored for any associated user input action. In some embodiments, determining that the gesture is rejected causes the specific hand or hands used to perform the gesture to be marked by the device as being in an inactive mode, or can help mark the hand as being in an inactive mode. However, in some embodiments, the specific hand or hands can remain in the current mode regardless of whether the gesture is rejected.
[0053] According to some embodiments, a gesture can be rejected at various times during the gesture. Thus, an input action associated with the gesture may be initiated before the gesture is rejected. For example, the user may be using a peripheral device such as a keyboard. Hand tracking data can detect that the hand is performing an input gesture based on the hand posture when the user is typing. In response, the device can initiate a user input action associated with the detected gesture. The user input action may include, for example, a visual prompt and an associated action for a specific UI component affected by the gesture. When a rejector action is determined, the user input action can be canceled so that the visual prompt is no longer presented and / or the resulting input action is discarded. According to some embodiments, canceling the input action may include rolling back any user input action that has been started. For example, if the user input action includes a dragging motion of a UI component, the UI component may return to its original position in response to the cancellation.
[0054] Returning to block 515, if the gesture is not rejected, the flowchart 500 proceeds to block 525 and a determination is made as to whether the hand is active. If the hand has been marked by the device as being in an active state, the hand is determined to be active. For example, based on a previously detected acceptor event, the hand may be in an active state. If the gesture is not rejected and the hand is active, the flowchart ends at block 545 and the gesture is accepted. According to one or more embodiments, accepting the gesture causes the associated input action to be performed.
[0055] In some embodiments, gestures can be recognized regardless of whether the hand is in an active state or in an inactive mode. However, the parameters by which the gesture causes the associated user input action to be executed can vary. Thus, returning to box 525, if the gesture is not rejected but the hand is not active, flowchart 500 proceeds to box 530 and makes a determination as to whether the activation criteria are met. The activation criteria may include detecting one of the acceptor actions in the acceptor action at box 510. That is, when the hand is inactive, the gesture is accepted based on the detected acceptor action. The acceptor action may be associated with parameters that are more difficult to satisfy the gesture than the parameters used during the active state. That is, when the hand is already in an active state, the acceptor may not make sense, but when the hand is in an inactive state, the acceptor may be useful. In contrast, the rejector may be useful both when the hand is in an active state and when the hand is in an inactive state.
[0056] If a determination is made at block 530 that the activation state is not satisfied, then the user input action is not performed and the hand is maintained in the inactive state for the next frame, as shown at block 535. Thus, the lack of activation from the inactive state does not change the state of the hand.
[0057] Returning to block 530, if the activation criteria are met, the flowchart proceeds to block 540 and the hand is activated. In some embodiments, the hand is activated by marking a particular hand or hands as being in active mode. The flowchart then ends and the gesture is accepted at block 545. In some embodiments, accepting the gesture may include performing a user input action associated with the gesture.
[0058] According to some embodiments, determining whether one or more hands should be placed in an inactive mode can be determined in response to detecting a gesture or attempted gesture of one or more hands. Additionally or alternatively, a determination as to whether to place one or more hands in an inactive mode can be made upon detecting a system event (e.g., input from a peripheral device, body tracking data such as an arm falling rapidly toward the torso or an arm retracting toward the torso), a timeout event, or the like.
[0059] Figure 6 A flow chart of a technique for transitioning between an active mode and an inactive mode according to one or more embodiments is shown. In particular, Figure 6 A state diagram depicts transitions between active and inactive modes according to one or more embodiments.
[0060] The state diagram begins at box 605, where gesture recognition is performed on the hand tracking data using parameters for one or more gestures associated with the active mode. This process assumes that the state diagram begins with the hand in an active state. Alternatively, if the process begins with the hand in an inactive mode, the state diagram can begin at box 620, as will be described below. For the purposes of this description, the state diagram begins at box 605 and uses parameters for one or more gestures associated with the active mode. Performing gesture recognition in the active mode can include, for example, performing gesture recognition using parameters that are easier to satisfy than when the hand is in the inactive mode. That is, because the hand is determined to be actively engaged with the user interface, the gesture is less likely to be unintentional, and therefore the gesture is more likely to trigger a user input action.
[0061] The flowchart continues to box 610, where a determination is made as to whether a gesture is recognized. The gesture may be recognized, for example, using the gesture determination framework 210 described above. In some embodiments, a gesture may be recognized as an attempt gesture, but recognition of the attempt gesture alone is not sufficient to trigger an associated user input action. Instead, once a potential gesture is detected, a determination is made as to whether to accept or reject the gesture. For example, this may be done within the gesture determination framework 210 or in a separate pipeline before generating the gesture signal 212. The gesture may be recognized, for example, based on a general set of parameters applied to the hand tracking data and / or other tracking data collected by the device.
[0062] If a gesture is not recognized at box 610, the figure proceeds to box 615, and a determination is made at box 615 as to whether the inactivity criteria are met. The inactivity criteria can be one or more actions or events that cause the device to determine that a particular hand or hands are not actively interacting with the user interface. That is, the determination can be made on the basis of a single hand or for both hands. The inactivity criteria can be based on unrecognized gestures, or can be based on separate contextual cues. For example, the inactivity criteria can include the detection of a rejector event as described above. In addition to specific actions or contextual cues, rejector events can also include timeout events, etc. That is, in some embodiments, if a gesture is not recognized within a predetermined amount of time, one or more hands can be placed in an inactive mode. If the inactivity criteria are not met, the flowchart proceeds to box 605 and one or more hands remain in active mode. At box 605, gesture recognition is continued using active mode parameters.
[0063] Returning to 610, if a determination is made that a gesture is recognized, the flow also proceeds to block 605 and the hand or hands remain in active mode. At block 605, gesture recognition continues using the active mode parameters. Notably, when a determination is made at block 610, the hand or hands are in active mode. Thus, if a gesture is recognized, the hand remains in active mode. Although not shown, the recognized gesture may trigger a user input action associated with the recognized gesture.
[0064] Returning to block 615, if a determination is made that the inactivity criteria are met while the hand is in the active mode, the flowchart proceeds to block 620 where gesture recognition is performed on the hand tracking data using the parameters associated with the inactivity mode. That is, if the inactivity criteria are met at block 615, the device may mark or otherwise treat the affected hand or hands as being in the inactive mode. From this point on, gesture recognition is performed on any subsequent data using the parameters associated with the inactive mode.
[0065] As described above, in some embodiments, once the hand is in the inactive mode, the gesture will not trigger user input until an acceptor event is detected, thereby triggering the hand to be placed in the active mode (and allowing the triggering gesture to initiate the associated user input action, if applicable). Thus, in some embodiments, performing gesture recognition at block 620 may include monitoring hand tracking data and / or other data to determine whether an acceptor event or action is detected.
[0066] The process proceeds to box 625, where a determination is made as to whether the active mode criteria are met. The active mode criteria can be used to determine whether to transition the hand to the active mode. Examples of active mode criteria can include a recognized gesture that causes an input action. The gesture can be identified, for example, using the gesture determination framework 210 described above. In some embodiments, a gesture can be identified as an attempt gesture, but identifying only the attempt gesture is not enough to trigger the associated user input action. Instead, once a potential gesture is detected, a determination is made as to whether to accept or reject the gesture. For example, this can be done within the gesture determination framework 210 or in a separate pipeline before generating the gesture signal 212. Gestures can be identified, for example, based on a general set of parameters applied to hand tracking data and / or other tracking data collected by the device. In some embodiments, gestures are identified using inactive parameters associated with an inactive mode, which may be more difficult to meet than the parameters associated with the active mode applied to the hand tracking data when the hand is in the active mode. It is worth noting that the active criteria (for transitioning the hand to the active mode) and the inactive criteria (for transitioning the hand to the inactive mode) can be different from each other and may not mirror each other.
[0067] If a determination is made at block 625 that the activity criteria are not met, the flow proceeds to block 620 and gesture recognition continues to be performed on additional received hand tracking data and / or other data using the inactive parameters. Returning to block 625, if a determination is made that the activity criteria are met, the hand or hands are placed in an active mode and the flow returns to block 605 where additional hand tracking data is processed using the active mode parameters.
[0068] refer to Figure 7 , depicting a simplified block diagram of an electronic device 700. The electronic device 700 may be part of a multi-function device, such as a mobile phone, a tablet computer, a personal digital assistant, a portable music / video player, a wearable device, a head-mounted system, a projection-based system, a base station, a laptop computer, a desktop computer, a network device, or any other electronic system such as described herein. The electronic device 700 may include one or more additional devices, such as a server device, a base station, an auxiliary device, etc., and various functions may be included in the one or more additional devices or various functions may be distributed on the one or more additional devices. Illustrative networks include, but are not limited to, local networks (such as universal serial bus (USB) networks), local area networks of organizations, and wide area networks (such as the Internet). According to one or more embodiments, the electronic device 700 is used to interact with the user interface of one or more applications 735. It should be understood that the various components and functions within the electronic device 700 may be distributed differently on modules or components or even additional devices.
[0069] The electronic device 700 may include one or more processors 720, such as a central processing unit (CPU) or a graphics processing unit (GPU). The electronic device 700 may also include a memory 730. The memory 730 may include one or more different types of memories, which may be used to perform device functions in conjunction with the processor 720. For example, the memory 730 may include a cache, a ROM, a RAM, or any type of transient or non-transient computer-readable storage medium capable of storing computer-readable code. The memory 730 may store various programming modules for execution by the processor 720, including a tracking module 745 and various other applications 735. The electronic device 700 may also include a storage device 740. The storage device 740 may include one or more non-transient computer-readable media, including, for example, disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and digital video discs (DVDs)), and semiconductor memory devices (such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs)). The storage device 740 may be used to store various data and structures, which may be used to store data related to hand tracking and UI preferences. According to one or more embodiments, the storage device 740 may be configured to store the hand tracking network 755. The storage device 740 may be further configured to store user specific data to facilitate hand tracking (such as hand size, bone length, user preferences, etc.) as registration data 750. The electronic device may additionally include a network interface from which the electronic device 700 can communicate over a network.
[0070] The electronic device 700 may also include one or more cameras 705 or other sensors 710, such as a depth sensor from which the depth of the scene can be determined. In one or more embodiments, each of the one or more cameras 705 can be a traditional RGB camera or a depth camera. In addition, the camera 705 can include a stereo camera or other multi-camera system. In addition, the electronic device 700 may include other sensors that can collect sensor data for tracking user movement, such as a depth camera, an infrared sensor, or an orientation sensor, such as one or more gyroscopes, accelerometers, etc.
[0071] According to one or more embodiments, the memory 730 may include one or more modules including computer readable codes that can be executed by the processor 720 to perform functions. The memory 730 may include, for example, a tracking module 745 and one or more applications 735. The tracking module 745 may be used to track the position of the hand and other user movements in the physical environment and determine whether the hand is in an active state or in an inactive state. The tracking module 745 may use sensor data, such as data from the camera 705 and / or the sensor 710. In some embodiments, the tracking module 745 may track user movement to determine whether to trigger user input according to the input gesture detected. The electronic device 700 may also include a display 725 that can present a UI for user interaction. For example, the UI may be associated with one or more applications in the application 735. The display 725 may be an opaque display or may be translucent or transparent. The display 725 may incorporate LEDs, OLEDs, digital light projectors, silicon-based liquid crystals, etc.
[0072] Although the electronic device 700 is depicted as including the multiple components described above, in one or more embodiments, various components may be distributed on multiple devices. Therefore, although certain calls and transmissions are described herein with respect to the specific system depicted, in one or more embodiments, various calls and transmissions may be directed differently based on the functions of different distributions. In addition, additional components may be used, and certain combinations of the functions of any components may also be combined. For example, the electronic device 700 may be communicatively connected to an additional electronic device 770 on a network 760. According to some embodiments, the electronic device 770 may be part of a multi-function device, such as a mobile phone, a tablet computer, a personal digital assistant, a portable music / video player, a wearable device, a head-mounted system, a projection-based system, a base station, a laptop computer, a desktop computer, a network device, or any other electronic system such as described herein. In some embodiments, the electronic device 770 may include a processor 775 or multiple processors of one or more types and a memory 780. In some embodiments, the memory 780 may be configured to store instructions executable by the processor 775, such as instructions related to the functions described above for the memory 730. In addition, the electronic device 770 may include an I / O device 785 that may be activated or otherwise used in conjunction with gesture input. For example, the I / O devices may include one or more cameras, displays, and the like that may be configured to receive input using gesture input or to provide output generated in response to gesture input.
[0073] Reference now Figure 8, a simplified functional block diagram of an exemplary multifunctional electronic device 800 according to one embodiment is shown. The electronic device may be a multifunctional electronic device, or may have some or all of the components described herein for the multifunctional electronic device. The multifunctional electronic device 800 may include a processor 805, a display 810, a user interface 815, graphics hardware 820, a device sensor 825 (e.g., a proximity sensor / ambient light sensor, an accelerometer and / or a gyroscope), a microphone 830, an audio codec 835, a speaker 840, a communication circuit 845, a digital image capture circuit 850 (e.g., including a camera system), a video codec 855 (e.g., supporting a digital image capture unit), a memory 860, a storage device 865, and a communication bus 870. The multifunctional electronic device 800 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.
[0074] The processor 805 can execute instructions necessary to implement or control the operation of many functions performed by the device 800 (such as, the generation and / or processing of images as disclosed herein). The processor 805 can, for example, drive the display 810 and can receive user input from the user interface 815. The user interface 815 can allow the user to interact with the device 800. For example, the user interface 815 can take a variety of forms, such as buttons, keypads, dials, click wheels, keyboards, display screens, touch screens, sight lines and / or gestures. The processor 805 can also be, for example, a system on a chip, such as a system on a chip found in a mobile device, and include a dedicated GPU. The processor 805 can be based on a reduced instruction set computer (RISC) or a complex instruction set computer (CISC) architecture or any other suitable architecture, and can include one or more processing cores. The graphics hardware 820 can be a dedicated computing hardware for processing graphics and / or assisting the processor 805 to process graphics information. In one embodiment, the graphics hardware 820 may include a programmable GPU.
[0075] The image capture circuit 850 may include two (or more) lens assemblies 880A and 880B, each of which may have a separate focal length. For example, relative to the focal length of lens assembly 880B, lens assembly 880A may have a shorter focal length. Each lens assembly may have a separate associated sensor element 890A or 890B. Alternatively, two or more lens assemblies may share a common sensor element. The image capture circuit 850 may capture still images and / or video images. The output from the image capture circuit 850 may be processed by the following devices: a video codec 855 and / or a processor 805 and / or graphics hardware 820 and / or a dedicated image processing unit or pipeline incorporated into the circuit 865. Such captured images may be stored in the memory 860 and / or the storage device 865.
[0076] The sensor and camera circuit 850 can capture still images and / or video images that can be processed at least in part by the following devices according to the present disclosure: video codec 855 and / or processor 805 and / or graphics hardware 820 and / or a dedicated image processing unit incorporated into the circuit 850. Such captured images can be stored in memory 860 and / or storage device 865. Memory 860 may include one or more different types of media used by processor 805 and graphics hardware 820 to perform device functions. For example, memory 860 may include memory cache, read-only memory (ROM) and / or random access memory (RAM). Storage device 865 can store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage device 865 may include one or more non-transitory computer-readable storage media, including, for example, disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and DVDs), and semiconductor memory devices (such as EPROMs and EEPROMs). The memory 860 and storage device 865 can be used to tangibly retain computer program instructions or codes organized into one or more modules and written in any desired computer programming language. For example, when executed by the processor 805, such computer program code can implement one or more of the methods described herein.
[0077] The various processes defined herein contemplate the option of obtaining and utilizing identifying information of users. For example, such personal information may be utilized in order to track the user's movements. However, to the extent such personal information is collected, such information should be obtained with the user's informed consent, and the user should be aware of and control the use of their personal information.
[0078] Personal information will be used by appropriate parties only for lawful and reasonable purposes. Parties utilizing such information will adhere to privacy policies and practices that at a minimum comply with applicable laws and regulations. In addition, such policies should be comprehensive and meet or exceed government / industry standards. In addition, parties may not distribute, sell or otherwise share such information except for any reasonable and lawful purpose.
[0079] Furthermore, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. Additionally, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. Where appropriate, de-identification can be facilitated by removing specific identifiers (e.g., date of birth), controlling the amount or characteristics of stored data (e.g., collecting location data at a city level rather than an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.
[0080] It should be understood that the above description is intended to be illustrative and not limiting. The material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims, and the material is provided in the context of specific embodiments, variations of which will be apparent to those skilled in the art (for example, some of the disclosed embodiments may be used in conjunction with each other). Therefore, Figures 2 to 6 The specific arrangement of steps or actions shown in or in FIG. Figures 7 and 8 The arrangement of elements shown in the accompanying drawings should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the present invention should be determined by reference to the appended claims and the full scope of equivalents to which such claims are entitled. In the appended claims, the terms "including" and "wherein" are used as the plain English equivalents of the corresponding terms "comprising" and "wherein".
Claims
1. A method, comprising: obtaining hand tracking data for the first hand based on the sensor data; detecting a first input gesture of the first hand based on the hand tracking data; determining a current state of the first hand according to a group consisting of an active state and an inactive state, wherein the active state is associated with a first set of parameters for the first input gesture, and wherein the inactive state is associated with a second set of parameters for the first input gesture; as well as Determining whether to initiate an input action associated with the first input gesture is based on the first parameter set or the second parameter set according to the current state of the first hand. 2 . The method of claim 1 , wherein the hand tracking data causes a probability that the input action is initiated in the active state to be greater than a probability that the hand tracking data causes the input action to be initiated in the inactive state.
3. The method according to claim 1, further comprising: When the first hand is in the active state: Determining that the inactivity criteria are met; as well as Based on determining that the inactivity criterion is satisfied, it is determined that the first hand is in an inactive state. The method of claim 3 , wherein the inactivity criteria comprises a timeout.
5. The method according to claim 3, further comprising: The first hand is transitioned from the inactive state to the active state in response to initiating the input action associated with the first input gesture based on the second set of parameters.
6. The method according to claim 1, further comprising: While the first hand is in the inactive state: Determine that activity criteria are met; as well as Based on determining that the activity criterion is satisfied, it is determined that the first hand is in the activity state.
7. The method according to claim 1, further comprising: obtaining hand tracking data for a second hand based on the sensor data; detecting a second input gesture of the second hand based on the hand tracking data; determining whether the second hand is active; as well as Based on determining that the first hand is not in the active state, the second input gesture is ignored when performing the input action.
8. A computer program product comprising computer readable code executable by one or more processors to: obtaining hand tracking data for the first hand based on the sensor data; detecting a first input gesture of the first hand based on the hand tracking data; determining whether the first hand is active or inactive; initiating an input action associated with the first input gesture based on a first set of parameters in response to determining that the first hand is in the active state; as well as Based on determining that the first hand is in the inactive state, an input action associated with the first input gesture is initiated based on a second set of parameters different from the first set of parameters.
9. The computer program product of claim 8, wherein the hand tracking data causes a probability that the input action is initiated in the active state to be greater than a probability that the hand tracking data causes the input action to be initiated in the inactive state.
10. The computer program product of claim 8, further comprising computer readable code for performing the following operations when the first hand is in the active state: Determining that the inactivity criteria are met; and Based on determining that the inactivity criterion is satisfied, it is determined that the first hand is in an inactive state.
11. The computer program product of claim 10, further comprising computer readable code for: The first hand is transitioned from the inactive state to the active state in response to initiation of an input action based on the second set of parameters.
12. The computer program product of claim 8, further comprising computer readable code for performing the following operations when the first hand is in the inactive state: Determining that the activity criteria are met; and Based on determining that the activity criterion is satisfied, it is determined that the first hand is in the activity state.
13. The computer program product of claim 8, further comprising computer readable code for: obtaining hand tracking data for a second hand based on the sensor data; detecting a second input gesture of the second hand based on the hand tracking data; determining whether the second hand is active; as well as Based on determining that the first hand is not in the active state, the second input gesture is ignored when performing the input action based on the first input gesture.
14. The computer program product of claim 8, wherein the first input gesture is performed by the first hand and a second hand, and wherein initiating the input action comprises transitioning the second hand to the active state.
15. A system, comprising: one or more processors; as well as One or more computer readable media comprising computer readable code executable by the one or more processors to: obtaining hand tracking data for the first hand based on the sensor data; detecting a first input gesture of the first hand based on the hand tracking data; determining whether the first hand is active or inactive; initiating an input action associated with the first input gesture based on a first set of parameters in response to determining that the first hand is in the active state; as well as Based on determining that the first hand is in the inactive state, the input action associated with the first input gesture is initiated based on a second set of parameters different from the first set of parameters.
16. The system of claim 15, further comprising computer readable code for performing the following operations when the first hand is in the active state: Determining that the inactivity criteria are met; and Based on determining that the inactivity criterion is satisfied, it is determined that the first hand is in an inactive state.
17. The system of claim 16, wherein the inactivity criteria comprises a timeout.
18. The system of claim 16, further comprising computer readable code for: The first hand is transitioned from the inactive state to the active state in response to initiation of an input action based on the second set of parameters.
19. The system of claim 15, further comprising computer readable code for: obtaining hand tracking data for a second hand based on the sensor data; detecting a second input gesture of the second hand based on the hand tracking data; determining whether the second hand is active; as well as Based on determining that the first hand is not in the active state, the second input gesture is ignored when performing the input action.
20. The system of claim 15, wherein the first input gesture is performed by the first hand and a second hand, and wherein initiating the input action comprises transitioning the second hand to the active state.
