Indirect motion mapping pipeline and kneading centroid

By using the combination of pinched centroid and insensitive zone in XR technology to identify and process user gestures, the inaccuracy and artifacts of gesture input are solved, and the interactive experience in the XR environment is improved.

CN119065574APending Publication Date: 2024-12-03APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410698196.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-05-28
Filing Date
2024-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing Extended Reality (XR) technologies are difficult to effectively manage and track user gesture input, especially in the movement and position changes of the hand when performing gestures, resulting in inaccuracy and artifacts of input actions.

Method used

By determining the kneading centroid and combining hand posture and sensor data, the kneading centroid and insensitive area are used to identify and process the user's indirect gestures, reduce the impact of unintentional movements, and convert the gestures into user input actions.

Benefits of technology

Improves the accuracy of gesture recognition and the stability of user input, reduces artifacts caused by jitter and unintentional motion, and enhances the interactive experience in the XR environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065574A_ABST
    Figure CN119065574A_ABST
Patent Text Reader

Abstract

The present disclosure generally relates to an indirect motion mapping pipeline and a kneading centroid. Processing gesture input includes obtaining hand tracking data of a hand from one or more sensors, where the hand performs a kneading in the hand tracking data, determining a hand position and a hand orientation of the hand from the hand tracking data, applying a predefined offset from the hand position based on the hand orientation to determine a kneading centroid, and outputting the kneading centroid. And determining an input location based on the kneading center of mass. The kneading centroid is used in an indirect motion mapping pipeline to determine characteristics of user input based on user motion.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Some devices can generate and present extended reality (XR) environments. XR environments can include fully or partially simulated environments that people perceive and / or interact with via electronic systems. In XR, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with real-world properties. Some XR environments allow multiple users to interact with virtual objects or with each other in the XR environment. For example, a user can use gestures to interact with components of the XR environment. However, an improved technique is needed to manage the tracking of hands performing gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1A to Figure 1B An example diagram illustrating hand tracking in accordance with one or more embodiments is shown.

[0003] Figure 2 A flowchart of a technique for detecting input gestures according to some embodiments is shown.

[0004] Figure 3 A flowchart of a technique for determining the intentionality of a gesture according to some embodiments is shown.

[0005] Figure 4 A schematic diagram of user-centric motion mapping according to some embodiments is shown.

[0006] Figure 5 A flow diagram illustrating techniques for an indirect motion mapping pipeline according to some embodiments.

[0007] Figure 6 A flow diagram illustrating a technique for an indirect motion mapping pipeline for motion tracking in accordance with one or more embodiments.

[0008] Figure 7 A flow chart of a technique for determining a kneading centroid is shown in accordance with one or more embodiments.

[0009] Figure 8 A system diagram of an electronic device that can be used for gesture recognition according to one or more embodiments is shown.

[0010] Fig. 9 Exemplary systems for various extended reality techniques are shown. DETAILED DESCRIPTION

[0011] The present disclosure relates to systems, methods, and computer-readable media for implementing gesture recognition and input. In some augmented reality scenarios, image data and / or other sensor data may be used to detect gestures by tracking hand data. The techniques described herein are directed to detecting indirect user input gestures and processing the gestures as user input actions. Specifically, the techniques described herein are directed to determining features of user input in an action involving pinching, such that the position and motion of the pinch can be converted into user input.

[0012] According to some embodiments, a pinch centroid of a hand may be determined for user input determination. The pinch centroid may be determined based on a particular positional relationship to the hand. For example, the pinch centroid may be at a location where two fingers (such as an index finger and thumb) are in contact, or may be determined in another manner—such as an offset from a hand origin or other location. In some embodiments, the pinch centroid may be associated with other characteristics, such as a dead zone configured to mitigate jitter or other artifacts caused by sensor readings, or unintentional user motion when attempting to perform a pinch.

[0013] In some embodiments, the pinch centroid can be used for indirect gestures to provide user input. An input gesture is a gesture that causes a user input action to be initiated when detected. An indirect gesture is an input gesture in which the user is interacting with a virtual component without attempting to contact the virtual component. For example, indirect gestures may include pinching, rolling, etc. Indirect gestures may be performed by one hand or two hands. For example, an indirect 2D two-hand gesture may require scaling, rotation, drawing, etc., and depending on the manipulator posture, the pinch centroid may be used to determine the user input position. As another example, given an indirect gesture such as rolling or other interaction gestures, the pinch centroid may be used with body estimation to determine user-centered motion, and the contact position may be determined based on the user-centered motion. In some embodiments, determining the contact position may also include applying sensitivity scaling to the user-centered motion. In another example, the pinch centroid may be used for indirect 3D gestures, such as window placement, dragging, and dropping. In these indirect 3D gestures, the pinch centroid may be used with body estimation to determine a 3D motion map, and the world position of the user input may be determined based on the 3D motion map.

[0014] In the following disclosure, a physical environment refers to the physical world that people can perceive and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense the physical environment and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, an XR environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset or representation of a person's physical motion is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. As an example, the XR system can detect head movement, and in response, adjust the graphics content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, tablet device, laptop computer, etc.) presenting an XR environment and adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust characteristics of graphical content in an XR environment in response to representations of physical movement (e.g., voice commands).

[0015] There are many different types of electronic systems that enable a person to sense and / or interact with various XR environments. Examples include: head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet devices, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing the image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some specific implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0016] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the disclosed concept. As part of this description, some of the drawings in the drawings of the present disclosure represent structures and devices in the form of block diagrams to avoid blurring the novel aspects of the disclosed concept. For the sake of clarity, all features of the actual specific implementation may not be described. In addition, as part of this specification, some drawings of the present disclosure may be provided in the form of flow charts. The blocks in any particular flow chart may be presented in a specific order. However, it should be understood that the specific order of any given flow chart is only used to illustrate an embodiment. In other embodiments, any element in the various elements depicted in the flow chart may be deleted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. In addition, other embodiments may include additional steps that are not shown as a part of the flow chart. In addition, the language used in the present disclosure has been selected primarily for readability and guidance purposes, and may not be selected to delimit or limit the subject matter of the present invention, or to resort to the necessary claims to determine such subject matter of the invention. Reference in this disclosure to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter, and multiple references to "one embodiment" or "an embodiment" should not be construed as necessarily all referring to the same embodiment.

[0017] It should be understood that in the development of any actual implementation (as in any software and / or hardware development project), many decisions must be made to achieve the developer's specific goals (e.g., to meet system and business-related constraints), and that these goals may vary from one implementation to another. It should also be understood that such development work may be complex and time-consuming, but is nonetheless a routine task for those of ordinary skill in designing and implementing a graphical modeling system having the benefit of this disclosure.

[0018] Figure 1A A schematic diagram of a technique for using a pinch centroid for user input according to one or more embodiments is shown. Specifically, in a first view 100A, a user's hand 110A is performing a pinch. According to one or more embodiments, the location in the real world where the pinch is determined to be performed is considered to be the pinch centroid 112A. The pinch centroid 112A can be determined in a variety of ways. For example, the pinch centroid 112A can be determined based on a location in the real world where contact is determined between a first surface and a second surface (such as an index finger pad and a thumb pad), as shown at 140. Alternatively, other estimates of the pinch centroid 112A can be used. In this example, the pinch centroid is based on a relative position to the hand when the pinch occurs.

[0019] exist Figure 1AIn the figure, the pinch center of mass 112A is determined based on an estimate of the position as a position offset of the hand. According to some embodiments, the hand posture is determined. The hand posture can be determined based on hand tracking data or other data that provides position information of the hand. In some embodiments, the hand posture can be determined based on the predicted position of one or more joints of the hand. For example, the hand tracking network can provide an estimated position of each joint in a set of joints of the hand. In some embodiments, the hand posture can be determined based on the relative position of a set of these joints or a subset of these joints. In this example, the wrist position 118A is compared with the index finger knuckle 114 and the little finger knuckle 116 to identify the position and orientation of the hand (e.g., hand posture). The index finger knuckle 114 and the little finger knuckle 116 can be the knuckle position of the top of the hand where each finger starts. According to some embodiments, the relative position of the wrist position 118A, the index finger knuckle 114 and the little finger knuckle 116 can provide a geometric plane 164, and the orientation of the hand can be determined based on the geometric plane. The position of the hand in real space can be determined similarly. In some embodiments, the directional position of the hand can be determined based on the same subset of joint positions or a different subset of joint positions. For example, the position of the wrist position 118A and the position of the index finger knuckle 114 can be used. However, a different set of joint positions can be used. In addition, the subset of joint positions used to determine the position and / or orientation of the hand can be dynamically modified based on contextual information, hand tracking information, etc. For example, occlusion determination can be made based on joint positions. If a portion of the hand is occluded, such as the pinky knuckle 116, an alternative joint position can be used to determine the orientation, position, etc.

[0020] In some embodiments, the pinch centroid can be determined based on an offset 130 from the hand pose. This can be a predefined global offset distance for any user, a personalized offset distance for a specific user's anatomy (e.g., bone length information from a user profile repository, enrollment data, etc.), or some other distance. In some embodiments, the offset 130 can be determined in a predefined direction based on the hand pose data. For example, based on a plane 164 derived from the wrist position 118, the index finger knuckle 114, and the pinky knuckle 116, the offset 130 can be determined at a predefined angle 162. The offset can be determined based on a specific point on the hand, such as the index finger knuckle 114 shown, or can be determined based on a central portion or origin of the hand derived from the position or orientation of the hand, etc. The result of the offset 130 of the hand pose provides a pinch centroid 112A. In some embodiments, the pinch centroid 112A can be juxtaposed or not juxtaposed with the actual location where contact occurs between the two fingers, as shown at 140.

[0021] The pinch centroid may have additional characteristics applied to it, such as an insensitive zone 145A. The insensitive zone 145A may include a predefined or dynamic area around the pinch centroid, such that any input determination that starts and ends within the insensitive zone may be reported as a tap. That is, in view of the user input gesture, the movement detected within the insensitive zone 145A when the finger is in the pinch position is ignored. Doing so can reduce the jitter of the sensor sensitivity caused by the user's unintentional movement. For example, if the duration of the pinch lasts for multiple frames, but the position of the pinch remains within the insensitive zone 145A, the resulting output may be considered a tap gesture. Conversely, if during the duration of the pinch (i.e., when the finger is in the pinch position), the pinch centroid leaves the insensitive zone 145A, the insensitive zone will be mitigated (e.g., reduced in size and / or removed), and the movement of the pinch will be converted into a corresponding movement of the user input. In this way, the gesture can be processed as a scroll or drag, rather than a tap.

[0022] In some embodiments, the dead zone 145A may be a dynamic dead zone. The size of the dynamic dead zone may change based on the characteristics of the user's motion. For example, during wrist rolling, when the user rotates the pinch center of mass around the wrist, the dead zone may be disabled. In some embodiments, the dynamic dead zone may shrink after a predetermined amount of time to better distinguish between a long press and a drag. As another example, for fast movements, such as repeated pinching at high speed, the size of the dead zone may increase.

[0023] According to one or more embodiments, the pinch centroid 112A can be used for user input. In this example, the hand 110A is performing a pinch gesture, which is determined to be a valid input gesture for triggering a user input action shown in the user interface 120A. In this way, the user input component 125 is shown as being selected. Although the user interface 120A is depicted as being in front of the hand, it should be understood that in some embodiments, the user input component may be a virtual input component, for example, presented on a display of a head-mounted device such as headphones, so that the user interface 120A is presented in front of the user's hand for the user using the head-mounted device. In this way, the user input component 125 is shown as being selected.

[0024] In accordance with one or more embodiments, in some embodiments, additional signals may be used to utilize the pinch centroid for user input. Figure 1B , a system setting 100B is provided, wherein a user 105 is viewing a user interface 120B. Figure 1B, user 105 is performing a gesture using hand 110B extending from the user's body. As shown, motion 160 caused by hand 110B is provided as an arc motion. In some embodiments, the motion is tracked in a spherical coordinate system. The spherical coordinate system provides the arc length of motion 160 relative to origin 152. As shown, the origin is at 152, which can coincide with the user's body. However, in some embodiments, the origin can be considered to be shoulder joint 150, wrist position 118B, or some other origin determined based on a predicted joint position of user 105. The arc length of motion 160 can be determined based on radius 154, which is determined from the hand origin 152 to the pinch center of mass 112B, which can be described as above with reference to Figure 1A The user motion in 3D space, as shown by motion 160, is mapped to a 2D user interface ("UI") plane, such as user interface 120B. For example, user input component 135 can move in direction 122 across the 2D plane of user interface 120B based on the mapping of user motion 160.

[0025] In some embodiments, an input gesture may be determined based on the motion 160 when the pinch centroid 112B leaves the dead zone 145B. According to one or more embodiments, indirect gestures involving motion, such as dragging or scrolling, may utilize a dead zone 165 around the user input component 135. Additionally or alternatively, the dead zone 165 may surround the pinch centroid 112B. The dead zone 165 may be used to reduce jitter based on unintentional user movement or sensor sensitivity during the user motion 160. The dead zone 165 may include a predefined or dynamic area around the user input location based on the pinch centroid, such that any input determination within the dead zone 165 may be reported from the center point of the dead zone 165. In this way, a user who performs the motion 160 with a slight jitter may not see the jitter reflected back on the user interface 120B in the direction of movement 122. That is, the dead zone 165 may follow the pinch centroid 112B and provide an area so that when the movement of the hand stops, the dead zone 165 will keep the input location on the UI stationary even if there is a small jitter.

[0026] According to some embodiments, additional motion mapping may be applied to motion 160 to affect the resulting movement of the user input component. For example, if user 105 were to move the entire user interface 120B, the resulting movement of user interface 120B may be represented in an arc-shaped manner rotating around a point of the user (such as the user's head). Additionally, in some embodiments, sensitivity scaling and acceleration profiles may be applied to motion 160 in order to maintain low-speed accuracy while allowing greater gains at high speeds.

[0027] Figure 2 A flow chart of a technique for detecting input gestures according to some embodiments is shown. Specifically, Figure 2 A gesture estimation pipeline is shown, where user input gestures are recognized and processed. Although the flowcharts show various components described as performing a particular process, it should be understood that the flow of the diagrams may be different according to some embodiments, and the functions of the components may be different according to some embodiments.

[0028] Flowchart 200 begins with sensor data 202. In some embodiments, the sensor data may include captured image data and / or depth data of one or both hands of a user. In some embodiments, the sensor data may be captured from a sensor on an electronic device, such as an outward-facing camera on a head-mounted device, or a camera otherwise configured in an electronic device to capture sensor data including the user's hands. According to one or more embodiments, the sensor data may be captured by one or more cameras, which may include one or more sets of stereo cameras. In some embodiments, sensor data 202 may include additional data collected by the electronic device and related to the user. For example, the sensor data may provide position data of the electronic device, such as the position and orientation of the device.

[0029] In some embodiments, sensor data 202 may be applied to a hand tracking network 204. The hand tracking network may be a network trained to estimate the physical state of one or both hands of a user. In some embodiments, hand tracking network 204 predicts hand pose data 206. The hand pose may be a classified pose of the hand based on the estimated physical state, or may provide some other form of data indicating the pose of the hand. For example, in some embodiments, hand pose data 206 may include an estimate of the joint position of the hand. Additionally, in some embodiments, hand tracking network 204 may be trained to provide an estimate of the device position and / or an estimate of the simulated world space.

[0030] In some embodiments, the hand tracking network 204 may also be configured to provide contact data. The contact data may include a prediction of whether a contact event occurs between two areas on the hand for a given frame or frames. For example, a machine learning model may be trained to predict whether the thumb pad and index finger are in contact. For the purposes described herein, a contact event refers to contact between two surfaces regardless of whether it is intentional, and a pinch is defined as a contact event performed to generate a corresponding input action. As will be described in more detail below, in some embodiments, hand tracking may predict whether a contact event occurs based on the sensor data 202 and / or the hand gesture data 206.

[0031] According to one or more embodiments, the gesture determination framework 210 provides a determination as to whether a particular gesture presented in the sensor data 202 is intentional. That is, a determination is made as to whether a classified hand gesture (e.g., based on or provided by the hand gesture data 206) is intentional. When the determined hand gesture includes a contact event such as a pinch, then the gesture determination framework 210 can use the contact signal 208 provided by the hand tracking network 204 to determine whether an intentional gesture is performed.

[0032] In some embodiments, the gesture determination framework 210 can utilize Figure 2 2. For example, gesture determination framework 210 can receive gesture signals 212, such as user interface (UI) geometry, gaze estimation, events generated by connected peripherals, user interactions with objects, etc. Gesture signals 212 can indicate whether an intentional input gesture has occurred. In some embodiments, gesture signals 212 can also be used to indicate whether a previous gesture signal should be canceled. For example, if the user moves their position, puts down their hand, etc., the cancellation may occur.

[0033] The UI gesture processing module 214 can be configured to implement user input actions based on the gesture signal 212. Certain gestures such as pinching can be associated with selection actions of UI components, etc. In some embodiments, if a cancel signal corresponding to a gesture signal 212 that has been initiated is received, the system can process the gesture differently than if the gesture was not canceled. For example, the UI component can be shown as selected but not activated, etc. As another example, a previously initiated tap caused by the user can be truncated or canceled.

[0034] As described above, the gesture determination framework 210 can be configured to generate a classification of the intentionality of a gesture.The gesture determination framework 210 can be configured to estimate a hand pose or gesture and determine whether the gesture is intended to trigger a user input action. Figure 3 A flow chart of a technique for classifying the intentionality of a gesture according to some embodiments is shown. For purposes of explanation, the following steps will be described as being performed by Figure 2 However, it should be understood that various actions can be performed by alternative components. Various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions may be added.

[0035] Flowchart 300 begins at block 305, where a contact event is detected based on contact data from a hand tracking network. For example, a contact event may be detected based on a contact signal 208 received from a hand tracking network 204. According to some embodiments, some gestures may require contact, such as a pinch, etc. In addition, different types of contacts may be utilized to identify multiple types of pinches. According to some embodiments, not every gesture requires a contact event. Thus, a contact may not be detected, or a contact signal 208 may indicate that a contact has occurred. In some embodiments, a contact signal 208 may not be received, or may be ignored and a gesture may still be recognized.

[0036] Flowchart 300 continues to block 310, where the contact phase is determined based on the hand tracking data. For a given frame, the contact phase may indicate which phase of the contact event the finger is currently in. According to some embodiments, the features of interest in determining intentionality may vary depending on the current state of the gesture. For gestures that include a pinch or other contact event, the phase that the gesture is currently in may affect the ability to enable, cancel, or reject the associated input action. Some examples of contact phases include an idle state, an entry state where a contact event begins (such as a pinch-in phase), a hold state where a pinch is currently occurring, and an exit phase (e.g., a pinch-out occurs at the end of a pinch).

[0037] At box 315, low-level features are estimated in association with the contact event. The low-level features may be determined based on the hand tracking data, and / or the additional data may include an estimate of what the hand is doing during the frame. For example, other data sources include posture information of the device used to capture the hand tracking data, hand posture, UI geometry, etc. In some embodiments, low-level features are determined without considering intent. Examples of low-level features include, for example, pinch speed when pinching down, measurement of wrist flexion, finger curling, proximity of the hand to the head, hand speed, etc.

[0038] Flowchart 300 continues to box 320, where high-level low-state features are estimated. High-level low-state features may include modal features that estimate what the user is doing during the contact event in order to determine intentionality. In some embodiments, high-level features may be interoperable and individually verifiable features. Examples include estimating whether the hand is using one or more peripheral devices, the repetition frequency of the gesture (e.g., whether the user is pinching quickly), whether the hand is holding an object, whether the hand is in a static position, and / or a specific pinch or gesture style (i.e., using the pulp of two fingers or using the side of the finger). In some embodiments, high-level features may be based on user activities, such as the user fidgeting, talking, or reading. According to one or more embodiments, high-level features may be determined based on hand tracking data, determined contact phases, and / or estimated basic features. In some embodiments, high-level features may directly determine the intentionality of the action. For example, if the user is using a peripheral device such as a keyboard, the pinch may be rejected, or the gesture may be determined to be unintentional.

[0039] The flowchart ends at box 325, where the gesture determination framework 210 combines high-level features and contact stages to classify intentionality. In some embodiments, the gesture determination framework 210 uses a conditional combination of high-level features and contact stages to classify intentionality. This classification can then be used to signal a gesture to be processed as an input gesture (thereby activating an associated UI input action), cancel an associated action if the gesture is determined to be unintentional (e.g., if a UI action associated with the gesture has already been initiated), or ignore the gesture.

[0040] Steering Figure 4 , a diagram of various locations on the body for motion mapping is shown. Specifically, shoulders 408 are shown relative to determined neck position 406 and determined head position 404. In some embodiments, the system providing motion mapping can simultaneously track head position 404 and / or neck position 406, from which the position of shoulders 408 can be determined. According to one or more embodiments, the tracking network can provide a predicted position of a set of joints of user 402, such as head 404, neck 406, shoulders 408, elbows 410, and wrists 412. The pinch center of mass 414 can be as described above with reference to Figure 1A to Figure 1B Determined as described.

[0041] A distance 416 from the origin to the pinch center of mass 414 can be determined. In some embodiments, the distance can be determined based on the horizontal distance between the torso position, shoulder position, wrist position, etc. and the pinch center of mass 414. In diagram 400, an example distance 416 between the position of shoulder 408 and the pinch center of mass 414 is depicted. However, the origin can alternatively be located at the elbow 410 or wrist 412, or some other location, for example, depending on the user's posture. The arc length of the motion is determined. The arc length is determined based on the distance that the user's hand travels along a curved path in 3D space when performing the gesture. In some embodiments, the arc length can be associated with direction information that is also determined.

[0042] Figure 5 A flow chart of a technique for an indirect motion mapping pipeline according to some embodiments is shown. Specifically, Figure 5 A flow chart of a technique for processing indirect 3D gestures is shown. For the purpose of explanation, the following steps will be described as follows from FIG. Figure 2 However, it should be understood that various actions can be performed by alternative components. Various actions can be performed in different orders. In addition, some actions can be performed simultaneously, some actions may not be required, or other actions can be added.

[0043] Flowchart 500 begins at block 505, where hand tracking data is obtained from a camera frame. The hand tracking data may include, for example, image data, depth data, etc. In some embodiments, the hand tracking data may include or be based on additional sensor data, such as image data and / or depth data capturing one or more hands of a user. In some embodiments, the sensor data may be captured from a sensor on an electronic device, such as an outward-facing camera on a head-mounted device, or a camera otherwise configured in an electronic device to capture sensor data including the user's hands. In some embodiments, the sensor data may include additional data collected by the electronic device and related to the user. For example, the sensor data may provide position data for the electronic device, such as the position and orientation of the device.

[0044] Flowchart 500 continues at block 510, where a hand pose is determined based on the hand tracking data. Figure 1A As described, the hand pose can be determined based on the tracking data. In some embodiments, the hand pose can be estimated based on the predicted position of one or more joints of the hand. For example, the hand tracking network can provide an estimated position of each joint in a set of joints of the hand. In some embodiments, the hand pose can be determined based on the relative position of a set of these joints or a subset of these joints.

[0045] At block 515, a pinch centroid is determined based on the hand gesture. According to one or more embodiments, the pinch centroid is a point in space that is tracked to determine user input. Figure 7 As described, the pinch center of mass can be determined based on characteristics of the position and orientation of the hand. For example, based on the hand posture, the pinch center of mass can be determined at a specific offset including a distance and angle from a specific position on the hand. In some embodiments, a dead zone is applied to the pinch center of mass so that the dead zone is tracked around the pinch center of mass or the input location on the UI to reduce the effects of unintentional user movement, artifacts from sensor data, and the like. The dead zone can be an area around the pinch center of mass such that when the pinch center of mass moves within the dead zone, the resulting user input will be based on a center position within the dead zone. Therefore, if the user shakes or trembles when moving the hand, the shaking will not be reflected back to the user in the form of user input on the user interface.

[0046] In some embodiments, the pinch centroid may also be associated with an insensitive zone. The insensitive zone may be an area around the initial position of the pinch centroid in order to distinguish between static input gestures (such as taps or long pinches) and dynamic input gestures (such as dragging, scrolling, etc.). According to one or more embodiments, if the pinch centroid remains within the insensitive zone from the moment the pinch is completed to the moment the pinch is interrupted, the detected pinch motion will be discarded and the input gesture will be processed as a static gesture. In contrast, if the pinch centroid leaves the insensitive zone while the finger is in the pinch posture, the input gesture will be determined as a dynamic input gesture, and the motion associated with the gesture will be used to process the movement of the user input. In some embodiments, the motion associated with the gesture will be additionally processed according to the dead zone as described above.

[0047] The flowchart continues to block 520, where motion mapping is applied to the hand gestures in the set of camera frames. In some embodiments, motion mapping can facilitate 3D movement of UI components. According to some embodiments, motion mapping can include determining the spherical motion of the pinch centroids relative to the origin, as described above with reference to Figure 1B and Figure 4 As described. For the purpose of 3D motion, the spherical motion can be mapped to a different origin, such as the user's head, rather than mapping the spherical motion to a plane. In this way, the 3D movement of the pinch center of mass around the origin can be remapped to a more general user space, such as around the user's head or other location. In some embodiments, the user can interact with the user input through headphones worn by the user, and the spherical motion of the pinch center of mass can be remapped to a 3D area around the headphones. In some embodiments, when the user uses spherical motion for indirect 3D gestures, the spherical coordinates of the pinch center of mass and the mapped motion can converge over time. At box 525, the contact position is determined based on the motion mapping.

[0048] The flow chart ends at block 530, where the input gesture is processed for potential input actions using the contact position. The resulting input gesture can cause the 3D movement of the user input component to move in 3D. In some embodiments, additional processing can be applied to the motion to improve the responsiveness of the user input. For example, an acceleration curve can be applied to maintain the accuracy of slower movements while increasing the activity of larger movements. Scaling can also be performed, for example, based on the distance of the UI component from the user, etc.

[0049] Figure 6 A flow chart of a technique for an indirect motion mapping pipeline for motion tracking according to one or more embodiments is shown. Specifically, Figure 6 A flow chart of a technique for processing indirect 2D gestures, such as a user using gestures to implement input actions on a UI plane, is shown. For purposes of explanation, the following steps will be described as follows from FIG. Figure 2 However, it should be understood that various actions can be performed by alternative components. Various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions may be added.

[0050] The flowchart begins at blocks 505, 510 and 515, as described above with reference to Figure 5 At block 505, hand tracking data is obtained from a camera frame or other sensor of an electronic device such as a headset. At block 510, hand pose information is determined based on the hand tracking data. Then, at block 515, a pinch centroid is determined based on the hand pose. The pinch centroid can be determined in a variety of ways, which will be referred to below. Figure 7 Describe in more detail.

[0051] Flowchart 600 proceeds to box 620, where the user motion is matched with the planar motion. The input motion is mapped from a spherical coordinate system to a 2D plane associated with the UI. According to some embodiments, the arc length of the user's movement is determined according to the spherical coordinate system. The arc length can indicate the distance and direction of the gesture performed by the hand during the input motion. According to some embodiments, the arc length is measured along the yaw and pitch directions and mapped to the X and Y coordinates. The arc length is also mapped to the Z coordinate relative to the radial motion. The coordinates in the spherical coordinate system will be projected onto the UI plane. In some embodiments, a spherical projection will be performed, where the position is proportional to the angle. That is, the amount of distance covered on the user interface plane is based on the angular distance traveled by the hand during the motion. Alternatively, in some embodiments, the distance covered on the plane will be based on the actual distance that the user's hand moves along the arc.

[0052] The flowchart includes, at box 625, allocating a dead zone around the pinch centroid. According to one or more embodiments, indirect gestures involving motion, such as drag gestures or scrolling, can utilize a dead zone around the user input location. The dead zone can be used to reduce jitter based on unintentional user movement or sensor sensitivity during user motion. The dead zone can include a predefined or dynamic area around the user input location based on the pinch centroid, such that any input determination within the dead zone can be reported from the center point of the dead zone. In this way, a user performing motion with a slight jitter may not see the jitter reflected back in the direction of the movement on the user interface.

[0053] At box 630, sensitivity scaling is applied to the motion. In some embodiments, the sensitivity scaling may include an acceleration curve. An acceleration curve may be applied to maintain low speed accuracy by applying a unity gain factor to position changes at low speeds. At higher speeds, the acceleration curve may allow the input to propagate farther by increasing the gain at higher speeds. In some embodiments, equivalently or scaling may be applied to the output motion to compensate for the UI components changing size as the distance of the UI changes relative to the distance of the user. Additionally, in some embodiments, asymmetric gains for sensitivity scaling may be applied based on direction, such as the plane of movement and / or the direction of rotation.

[0054] Flowchart 600 proceeds to box 635, where the user input position is determined based on the scaling. For example, a specific position on the UI plane can be determined as the target position of the UI input based on the scaling. Flowchart 600 ends at box 640, where the input gesture is processed for potential input actions using the user input position. For example, a cursor or other input component can be moved along the UI plane based on the mapping. Since the mapping takes into account distance and direction, the corresponding distance and direction will be obvious based on the change presentation of the UI component. The UI component may include, for example, a cursor, a scroll bar, or other input components based on movement.

[0055] Figure 7 A flow chart of a technique for determining a kneading centroid according to one or more embodiments is shown. Specifically, Figure 7 Depicts an example technique for determining a pinch centroid for processing user input, as described above with reference to Figure 5 and Figure 6 For the purpose of explanation, the following steps will be described as follows: Figure 2 However, it should be understood that various actions can be performed by alternative components. Various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions may be added.

[0056] The flowchart begins at box 705, where, as part of determining the pinch centroid from box 515, a hand pose including hand position and orientation is determined. The hand pose may be determined based on hand tracking data or other data that provides position information of the hand. In some embodiments, the hand pose may be estimated based on the predicted position of one or more joints of the hand. For example, a hand tracking network may provide an estimated position of each joint in a set of joints of the hand. In some embodiments, the hand pose may be determined based on the relative position of a set of these joints or a subset of these joints. For example, the relative position of the wrist position 118, the index finger knuckle 114, and the pinky knuckle 116 may provide a geometric plane from which the orientation of the hand may be determined. The position of the hand in world space may be determined similarly. In some embodiments, the directional position of the hand may be determined based on the same subset of joint positions or different subsets of joint positions. For example, the wrist position and the index finger knuckle position may be used. In addition, the subset of joint positions used to determine the position and / or orientation of the hand may be dynamically modified based on contextual information, hand tracking information, etc. For example, occlusion determination may be performed based on the joint positions. If a portion of the hand is occluded, such as pinky knuckle 116, an alternate joint position may be used to determine orientation, position, etc. In accordance with one or more embodiments, hand tracking information may be upsampled prior to determining a hand pose.

[0057] At box 710, a predefined offset from the hand position is applied based on the orientation to obtain the pinch center of mass. This can be a predefined global offset distance for any user, a personalized offset distance for a specific user (e.g., from a user profile repository, enrollment data, etc.), or some other distance. In some embodiments, the offset can be determined in a predefined direction based on the hand posture data. For example, based on a plane derived from the wrist joint, the index finger knuckle, and the pinky finger knuckle, the offset can be determined at a predefined angle. The result of the hand posture offset provides the pinch center of mass. It should be noted that the pinch center of mass may not be juxtaposed with the actual location where contact occurs between the two fingers.

[0058] According to one or more embodiments, a dynamic dead zone may be used. When conditions are more sensitive to noise, the size of the dynamic dead zone may change. This may be based on environmental conditions, characteristics of UI components, etc. When a dynamic dead zone is implemented, a target dead zone size may be determined. The size of the current dead zone will vary frame by frame over a series of frames, so that the transition from the current dead zone to the target dead zone is smooth over a series of frames.

[0059] The flowchart ends at box 720, where an insensitive zone is assigned relative to the pinch centroid. The insensitive zone can include a predefined or dynamic area around the pinch centroid, so that any input determination within the insensitive zone can be reported as a tap or a static gesture. That is, motion detected within the insensitive zone is discarded from consideration of the user input gesture. Doing so can mitigate jitter in sensor sensitivity caused by unintentional user movement. For example, if the duration of the pinch lasts for multiple frames, but the position of the pinch remains within the insensitive zone, the resulting output can be considered a tap gesture or other static gesture. Conversely, if the pinch centroid leaves the insensitive zone during the duration of the pinch, the motion of the pinch will be considered for the resulting user input.

[0060] In some embodiments, a dynamic insensitive zone may be used. The dynamic insensitive zone may be adapted in size based on the characteristics of the user action or motion. In one example, when the user motion involves less movement and therefore the distinction between a static gesture and a dynamic gesture is less obvious, the insensitive zone may be reduced or reduced, or may not be considered. As another example, if a motion involving rotation about the wrist is detected, such as a static wrist position compared to the motion of the pinch center of mass, the insensitive zone may be reduced or removed. As another example, if the user is performing a gesture in which the user holds a pinch, the insensitive zone may be reduced or eliminated if the user is initially stationary for a threshold amount of time. As another example, a high-speed pinch detected based on the distance between the two fingers involved in the pinch changes at a rate that exceeds a predefined threshold. If a high-speed pinch is detected, the size of the insensitive zone may be increased. By increasing the size of the insensitive zone for fast pinches, the system can accept less precise gestures.

[0061] refer to Figure 8 , depicting a simplified block diagram of an electronic device 800. The electronic device 800 may be part of a multi-function device, such as a mobile phone, a tablet computer, a personal digital assistant, a portable music / video player, a wearable device, a head-mounted system, a projection-based system, a base station, a laptop computer, a desktop computer, a network device, or any other electronic system such as those described herein. The electronic device 800 may include one or more additional devices such as a server device, a base station, an accessory device, and various functions may be included in these additional devices or may be distributed between these additional devices. Exemplary networks include, but are not limited to, local area networks such as universal serial bus (USB) networks, local area networks of organizations, and wide area networks such as the Internet. According to one or more embodiments, the electronic device 800 is used to interact with the user interface of the application 835. It should be understood that the various components and functions within the electronic device 800 may be distributed differently on modules or components, or even on additional devices.

[0062] The electronic device 800 may include one or more processors 820, such as a central processing unit (CPU) or a graphics processing unit (GPU). The electronic device 800 may also include a memory 830. The memory 830 may include one or more different types of memory that can be used to perform device functions in conjunction with the processor 820. For example, the memory 830 may include a cache, a ROM, a RAM, or any kind of transient or non-transient computer-readable storage medium capable of storing computer-readable code. The memory 830 may store various programming modules executed by the processor 820, including a tracking module 845 and various other applications 835. The electronic device 800 may also include a storage device 840. The storage device 840 may include one or more non-transient computer-readable media, including, for example, disks (fixed, floppy, and removable) and tapes, optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor storage devices such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs). The storage device 840 may be used to store various data and structures that can be used to store data related to hand tracking and UI preferences. According to one or more embodiments, storage device 840 can be configured to store hand tracking network 855 and other data used to determine hand motion, such as registration data 850. The electronic device can additionally include a network interface through which electronic device 800 can communicate across a network.

[0063] The electronic device 800 may also include one or more cameras 805 or other sensors 810, such as depth sensors, from which the depth of the scene can be determined. In one or more embodiments, each of the one or more cameras 805 can be a traditional RGB camera or a depth camera. In addition, the camera 805 can include a stereo camera or other multi-camera system. In addition, the electronic device 800 may include other sensors that can collect sensor data for tracking user movement, such as a depth camera, an infrared sensor, or an orientation sensor, such as one or more gyroscopes, accelerometers, etc.

[0064] According to one or more embodiments, the memory 830 may include one or more modules, which include computer-readable codes that can be executed by the processor 820 to perform various functions. The memory 830 may include, for example, a tracking module 845 and one or more applications 835. The tracking module 845 can be used to track the position of the hand and other user movements in the physical environment. The tracking module 845 can use sensor data, such as data from the camera 805 and / or the sensor 810. In some embodiments, the tracking module 845 can track user movement to determine whether to trigger user input according to the input gesture detected. The electronic device 800 may also include a display 825, which can present a UI for user interaction. For example, the UI may be associated with one or more applications 835. The display 825 may be an opaque display, or may be translucent or transparent. The display 825 may incorporate LED, OLED, digital light projector, liquid crystal on silicon, etc.

[0065] Although the electronic device 800 is depicted as including the above-described multiple components, in one or more embodiments, various components may be distributed across multiple devices. Therefore, although certain calls and transmissions are described herein with respect to the specific system depicted, in one or more embodiments, various calls and transmissions may be directed differently based on the differently distributed functionality. In addition, additional components may be used, and certain combinations of the functionality of any component may also be combined.

[0066] Reference now Fig. 9 , a simplified functional block diagram of an exemplary multifunctional electronic device 900 according to one embodiment is shown. Each of the electronic devices may be a multifunctional electronic device, or may have some or all of the components described in the multifunctional electronic device described herein. The multifunctional electronic device 900 may include a processor 905, a display 910, a user interface 915, graphics hardware 920, a device sensor 925 (e.g., a proximity sensor / ambient light sensor, an accelerometer and / or a gyroscope), a microphone 930, an audio codec 935, a speaker 940, a communication circuit 945, a digital image capture circuit 950 (e.g., including a camera system), a video codec 955 (e.g., supporting a digital image capture unit), a memory 960, a storage device 965, and a communication bus 970. The multifunctional electronic device 900 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0067] The processor 905 may execute instructions necessary to perform or control the operation of many functions performed by the device 900 (e.g., generation and / or processing of images such as disclosed herein). The processor 905 may, for example, drive the display 910 and receive user input from the user interface 915. The user interface 915 may allow a user to interact with the device 900. For example, the user interface 915 may take various forms, such as buttons, keypads, dials, click wheels, keyboards, display screens, touch screens, gazes, and / or gestures. For example, the processor 905 may also be a system on a chip, such as those found in mobile devices, and include a dedicated GPU. The processor 905 may be based on a reduced instruction set computer (RISC) or a complex instruction set computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. The graphics hardware 920 may be dedicated computing hardware for processing graphics and / or assisting the processor 905 in processing graphics information. In one embodiment, the graphics hardware 920 may include a programmable GPU.

[0068] The image capture circuit 950 may include two (or more) lens assemblies 980A and 980B, each of which may have a separate focal length. For example, relative to the focal length of lens assembly 980B, lens assembly 980A may have a shorter focal length. Each lens assembly may have a separate associated sensor element 990A and 990B. Alternatively, two or more lens assemblies may share a common sensor element. The image capture circuit 950 may capture still images and / or video images. The output from the image capture circuit 950 may be processed by a video codec 955 and / or processor 905 and / or graphics hardware 920 and / or a dedicated image processing unit or pipeline incorporated into the circuit 950. The image captured in this way may be stored in a memory 960 and / or a storage device 965.

[0069] The sensor and camera circuit 950 can capture still images and video images, which can be processed at least in part by the video codec 955 and / or the processor 905 and / or the graphics hardware 920 and / or a dedicated image processing unit incorporated into the circuit 950 according to the present disclosure. The captured images can be stored in the memory 960 and / or the storage device 965. The memory 960 may include one or more different types of media used by the processor 905 and the graphics hardware 920 to perform device functions. For example, the memory 960 may include a memory cache, a read-only memory (ROM) and / or a random access memory (RAM). The storage device 965 can store media (e.g., audio, image and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage device 965 may include one or more non-transitory computer-readable storage media, including, for example, disks (fixed, floppy and removable) and tapes, optical media such as CD-ROMs and DVDs, and semiconductor storage devices such as EPROMs and EEPROMs. The memory 960 and storage device 965 may be used to tangibly store computer program instructions or codes organized into one or more modules and written in any desired computer programming language. When executed by, for example, the processor 905, such computer program code may implement one or more methods described herein.

[0070] The various processes defined herein contemplate the option of obtaining and utilizing identifying information of users. For example, such personal information may be utilized in order to track the user's movements. However, to the extent such personal information is collected, such information should be obtained with the user's informed consent, and the user should be aware of and control the use of their personal information.

[0071] Personal information will be used by appropriate parties only for lawful and reasonable purposes. Parties utilizing such information will adhere to privacy policies and practices that at a minimum comply with applicable laws and regulations. In addition, such policies should be comprehensive and meet or exceed government / industry standards. In addition, parties may not distribute, sell or otherwise share such information except for any reasonable and lawful purpose.

[0072] Furthermore, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. Additionally, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. Where appropriate, de-identification can be facilitated by removing specific identifiers (e.g., date of birth), controlling the amount or characteristics of stored data (e.g., collecting location data at a city level rather than an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.

[0073] It should be understood that the above description is intended to be illustrative and not limiting. The material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims, and the material is provided in the context of specific embodiments, variations of which will be apparent to those skilled in the art (for example, some of the disclosed embodiments may be used in conjunction with each other). Therefore, Figures 2 to 3 and Figures 5 to 7 The specific arrangement of the steps or actions shown in FIG. Figure 4 and Figures 8 to 9 The arrangement of elements shown in the accompanying drawings should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the present invention should be determined by reference to the appended claims and the full scope of equivalents to which such claims are given. In the appended claims, the terms "including" and "wherein" are used as the plain English equivalents of the corresponding terms "comprising" and "wherein".

Claims

1. A method, the method include: obtaining hand tracking data of a hand from one or more sensors, wherein the hand performs a pinch in the hand tracking data; determining a hand position and a hand orientation of the hand based on the hand tracking data; based on the hand orientation, applying a predefined offset from the hand position to determine a pinch centroid; as well as An input location is determined based on the pinch centroid. 2 . The method of claim 1 , wherein the input position is further determined based on a dead zone applied around the pinch centroid.

3. The method of claim 1 , wherein the hand tracking data is associated with user motion, and wherein the input location is determined based on the pinch centroid. include: determining input motion based on the hand tracking data; as well as applying a sensitivity scaling to the input motion based on one or more characteristics of the input motion; as well as A gain of an output is adapted based on the input motion and the sensitivity scaling.

4. The method of claim 1, wherein the hand position is based on an index finger knuckle position, and wherein the predefined offset is applied to the index finger knuckle position.

5. The method according to claim 1, further comprising: include: In response to detecting that the position of the pinch centroid remains within a predefined insensitivity zone during the duration of the pinch, the pinch is processed as a tap gesture.

6. The method according to claim 1, further comprising: include: In response to detecting that the position of the pinch centroid leaves a predefined dead zone during the duration of the pinch, the pinch is processed as a drag gesture. 7 . The method of claim 6 , wherein the hand tracking data is associated with user motion, and wherein the size of the dead zone is adapted according to the user motion.

8. The method according to claim 7, further comprising: include: Based on determining that the user motion has stopped, the size of the insensitive zone is reduced.

9. A computer program product comprising computer readable code executable by one or more processors to: obtaining hand tracking data of a hand from one or more sensors, wherein the hand performs a pinch in the hand tracking data; determining a hand position and a hand orientation of the hand based on the hand tracking data; based on the hand orientation, applying a predefined offset from the hand position to determine a pinch centroid; as well as An input location is determined based on the pinch centroid.

10. The computer program product of claim 9, wherein the input location is further determined based on a dead zone applied around the pinch centroid.

11. The computer program product of claim 9, wherein the hand tracking data is associated with user motion, and wherein the computer readable code for determining the input location based on the pinch centroid comprises computer readable code for: Determining input motion based on the hand tracking data ; as well as applying a sensitivity scaling to the input motion based on one or more characteristics of the input motion; as well as A gain of an output is adapted based on the input motion and the sensitivity scaling.

12. The computer program product of claim 9, wherein the hand position is based on an index finger knuckle position, and wherein the predefined offset is applied to the index finger knuckle position.

13. The computer program product of claim 9, further comprising computer readable code for: In response to detecting that the position of the pinch centroid remains within a predefined insensitivity zone during the duration of the pinch, processing the pinch as a tap gesture; and In response to detecting that the position of the pinch centroid leaves the predefined dead zone during the duration of the pinch, processing the pinch as a drag gesture.

14. The non-transitory computer-readable medium of claim 13, wherein the hand tracking data is associated with user motion, and wherein the size of the predefined dead zone is adapted according to the user motion.

15. The computer program product of claim 14, further comprising computer readable code for: Based on determining that the user motion has stopped, reducing the size of the predefined insensitive zone.

16. A system, wherein the system include: one or more processors; as well as One or more computer readable media, the one or more computer readable media comprising computer readable code, the computer readable code being executable by the one or more processors to perform the following operations: obtaining hand tracking data of a hand from one or more sensors, wherein the hand performs a pinch in the hand tracking data; determining a hand position and a hand orientation of the hand based on the hand tracking data; based on the hand orientation, applying a predefined offset from the hand position to determine a pinch centroid; as well as An input location is determined based on the pinch centroid.

17. The system of claim 16, wherein the input location is further determined based on a dead zone applied around the pinch centroid.

18. The system of claim 16, wherein the hand tracking data is associated with user motion, and wherein the computer readable code for determining the input location based on the pinch centroid comprises computer readable code for: Determining input motion based on the hand tracking data ; as well as applying a sensitivity scaling to the input motion based on one or more characteristics of the input motion; as well as A gain of an output is adapted based on the input motion and the sensitivity scaling.

19. The system of claim 16, wherein the hand position is based on an index finger knuckle position, and wherein the predefined offset is applied to the index finger knuckle position.

20. The system of claim 19, further comprising computer readable code for: In response to detecting that the position of the pinch centroid remains within a predefined insensitivity zone during the duration of the pinch, processing the pinch as a tap gesture; and In response to detecting that the position of the pinch centroid leaves the predefined dead zone during the duration of the pinch, processing the pinch as a drag gesture.