Kneading identification using finger zones

By defining gesture areas on the user's hands and using hand tracking data and machine learning classifiers, identifying and managing pinch gesture intentions in the extended real-world environment, the problem of gesture input management is solved, and input accuracy and system robustness is improved.

CN119065579APending Publication Date: 2024-12-03APPLE INC
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410700672.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-09-29
Filing Date
2024-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to effectively manage and identify gesture inputs performed by users in extended real-life environments, especially the intentionality of pinch gestures, resulting in waste of computing resources and errors in input actions.

Method used

By defining a specific gesture area on the user's hand, combining hand tracking data and machine learning classifiers, the intentionality of the pinch gesture is determined, and the corresponding user input actions are enabled or cancelled when intentional changes are made.

Benefits of technology

It improves the accuracy and efficiency of gesture input, reduces the occurrence of unintentional input actions, and enhances the robustness of the system and the utilization of computing resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065579A_ABST
    Figure CN119065579A_ABST
Patent Text Reader

Abstract

A pinch identification using a finger region is provided. Enabling gesture input includes: obtaining hand tracking data based on one or more camera frames; detecting a contact event between the first finger and the second finger based on the hand tracking data; and determining a first contact position on the first finger and a second contact position on the second finger. In accordance with a determination that the first contact location and the second contact location are within a first predefined gesture region for the first gesture, an input action corresponding to the first gesture is enabled.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Some devices can generate and present extended reality (XR) environments. XR environments can include fully or partially simulated environments that people perceive and / or interact with via electronic systems. In XR, a subset of a person's physical movements, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with real-world properties. Some XR environments allow multiple users to interact with virtual objects or with each other in the XR environment. For example, a user can use gestures to interact with components of the XR environment. However, an improved technique is needed to manage the tracking of hands performing gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 An example diagram illustrating detecting a pinch based on different user gestures according to one or more embodiments is shown.

[0003] Figure 2 An example diagram of a gesture zone for detecting gestures according to one or more embodiments is shown.

[0004] Figure 3 A flowchart of a technique for detecting input gestures according to some embodiments is shown.

[0005] FIG. 4A to FIG. 4B A flowchart of a technique for enabling gesture input using a gesture area according to some embodiments is shown.

[0006] Figure 5 A flowchart of a technique for classifying the intentionality of gestures according to some embodiments is shown.

[0007] Figure 6 A flow chart of a technique for classifying touch phases according to some embodiments is shown.

[0008] Figure 7 A flow diagram of an action network is shown according to some embodiments.

[0009] Figure 8 An example diagram for canceling gesture input according to some embodiments is shown.

[0010] Fig. 9 A flowchart of a technique for canceling gesture input based on a gesture area according to some embodiments is shown.

[0011] Fig.10 A system diagram of an electronic device that can be used for gesture recognition according to one or more embodiments is shown.

[0012] Fig.11 Exemplary systems for various extended reality techniques are shown. DETAILED DESCRIPTION

[0013] The present disclosure relates to systems, methods, and computer-readable media that enable gesture recognition and input. In particular, the present disclosure relates to a technique for improving detection of intentionality of a pinch-based action for enabling gesture input.

[0014] In some augmented reality scenarios, image data and / or other sensor data may be used to detect gestures by tracking hand data. For some gestures (such as pinching), hand posture information is obtained to determine whether a touch occurs between two fingers or between two parts of a hand. The application framework determines the intentionality of the touch, which may be determined based on the hand tracking data. In some embodiments, other considerations may be used, such as line of sight information, other peripheral object information, user interface (UI) components, or other contextual information. The framework may then determine whether an input action should be enabled based on the intentionality of the gesture.

[0015] According to one or more embodiments, specific areas of the hand can be used to determine the intentionality of a pinch. For example, predefined areas can be determined on one or more hands of a user. These areas can be located, for example, around the fingertips of the thumb and one or more other fingers. A touch between two areas can be determined as an intentional pinch, while a touch that includes two parts of the user's hand that are not both in these areas can be determined as an unintentional touch. In some embodiments, a touch between two parts of the user's hand that are not both within the determined area may not be classified as a touch, thereby reducing calculations by avoiding determining intentionality. In addition, in some embodiments, when the release of the pinch occurs at a different contact point than the touch, the input action can be canceled. In addition, by using touch areas, a machine learning classifier trained for pinch recognition becomes more robust by constraining the problem space.

[0016] In some embodiments, the determined pinch may initiate an action that may then be canceled based on the movement of the user's finger during the touch. That is, a pinch may begin with a touch between two portions of a user's hand within a zone. The user may maintain contact between the two portions of the hand, but one or both of the contact points may be within the zone. That is, the user's finger may "slide" out of the zones while maintaining contact. In this case, the pinch may no longer be considered intentional, and the associated user input action associated with the pinch may be canceled.

[0017] In the following disclosure, a physical environment refers to the physical world that people can perceive and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense the physical environment and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, an XR environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset or representation of a person's physical motion is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. As an example, the XR system can detect head movement, and in response, adjust the graphics content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, tablet device, laptop computer, etc.) presenting an XR environment and adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust characteristics of graphical content in an XR environment in response to representations of physical movement (e.g., voice commands).

[0018] There are many different types of electronic systems that enable a person to sense and / or interact with various XR environments. Examples include: head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet devices, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing the image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0019] In the following description, for the purpose of explanation, a lot of specific details are set forth in order to provide a thorough understanding of the disclosed concept. As part of this description, some of the drawings in the drawings of the present disclosure represent structures and devices in the form of block diagrams to avoid blurring the novel aspects of the disclosed concept. For the sake of clarity, all features of the actual specific implementation may not be described. In addition, as part of this specification, some drawings of the present disclosure may be provided in the form of flow charts. The blocks in any particular flow chart may be presented in a specific order. However, it should be understood that the specific order of any given flow chart is only used to illustrate an embodiment. In other embodiments, any element in the various elements depicted in the flow chart may be deleted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. In addition, other embodiments may include additional steps that are not shown as a part of the flow chart. In addition, the language used in the present disclosure has been selected primarily for readability and guidance purposes, and may not be selected to delimit or limit the subject matter of the present invention, thereby resorting to the necessary claims to determine such inventive subject matter. References in this disclosure to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter, and multiple references to "one embodiment" or "an embodiment" should not be construed as necessarily all referring to the same embodiment.

[0020] It should be understood that in the development of any actual implementation (as in any software and / or hardware development project), many decisions must be made to achieve the developer's specific goals (e.g., to meet system and business-related constraints), and that these goals may vary from one implementation to another. It should also be understood that such development work may be complex and time-consuming, but is nonetheless a routine task for those of ordinary skill in the design and implementation of a graphical modeling system having the benefit of this disclosure.

[0021] For the purposes of the following description, the term "contact event" refers to a user action associated with a touch-based gesture, such as a pinch. A contact event includes a touch, which is detected when two parts of a user's hand are in contact. A contact event also includes a release, which is detected when two parts of a user's hand cease to be in contact.

[0022] For the purposes of the following description, the term "gesture-based input event" refers to a process triggered in response to a gesture-based input. A gesture-based input event includes visual feedback of a UI element affected by a gesture-based input, such as highlighting, magnification, etc. A gesture-based input event also includes an input action, which includes the selection of a UI element or other processes performed on a UI element.

[0023] Figure 1A diagram of a technique for enabling user input actions using gesture zones according to one or more embodiments is shown. Two example hand poses are shown. A first pose 100A shows a hand 105A having a first gesture zone area 110A and a second gesture zone area 115A. A gesture zone may be an area or a set of areas on a hand that, when the area or the set of areas includes a contact point for a pinch or other such gesture, causes the gesture to be recognized as a user input gesture, or is otherwise used to determine whether the input gesture is intended. The contact point is detected at 120. Because the contact point 120 includes the surface of two portions of the hand that are within the area of ​​the gesture zone (i.e., the first gesture zone area 110A and the second gesture zone area 115A), the pose of the hand 105A is classified as a touch. Thus, in some embodiments, a user input action may be triggered in association with the detected touch. For example, the touch may be detected as part of a computer application that utilizes a detected pinch gesture for user input. Thus, an associated gesture-based input event may be performed based on the detected touch. In some embodiments, a gesture-based input event may include a visual feedback component and an input action component. A detected touch may trigger a visual feedback component, and a release of the detected touch may trigger execution of an associated input action.

[0024] As will be described in more detail below, touches may be detected based on sensor data captured by the electronic device. For example, image data and / or other sensor data may be collected and analyzed by the device to determine a hand gesture. This may occur, for example, using hand tracking or other vision-based tracking networks or processes. That is, the electronic device may have one or more cameras or other sensors configured on or in the device in a manner that captures an image of the hand. The electronic device may be a mobile device, such as a wearable device with a camera and / or other sensors facing the user's hand.

[0025] Compared to the gesture shown at 100A, second gesture 100B shows hand 105B having first gesture zone area 110B and second gesture zone area 115B. A contact point is detected at 130. Because at least one surface location at the contact point is not within the gesture zone (i.e., first gesture zone area 110B and second gesture zone area 115B), the gesture of hand 105B is not classified as a touch.

[0026] According to one or more embodiments, gesture zones may be located at various locations on the hand. In one example, a gesture zone associated with pinching may include an area at the tip of the index finger and the tip of the thumb. In some embodiments, the specific size of one or more of the zones may be a predetermined size, or may be proportional to the user's hand. For example, the size of a zone on a particular finger may extend from the fingertip to a first joint, a second joint, the entire finger, and the like. Furthermore, in some embodiments, the size of the zone may be user-specific, such as based on registration information of the user's hand. For example, two users may have different pinching styles, resulting in two different shapes of gesture zones and / or gesture zone regions.

[0027] As will be described in more detail below, transforming a gesture into a user input action may include a multi-step process. In some embodiments, in order to activate a user input action associated with a pinch gesture, a touch must first be detected. For the purposes described herein, a touch refers to contact between two surfaces regardless of intent, and a pinch is defined as a touch performed with the intent to produce a corresponding input action. Therefore, a gesture classified as a touch may indicate that a contact is detected for the gesture. In addition, a touch may be part of a contact event that includes a touch and a release of the touch. In some embodiments, enabling user input according to a gesture may additionally include determining the intentionality of the touch. That is, a determination may be made as to whether the touch is intended to be a user input action in the form of a pinch. Therefore, not all detected touch gestures are processed as user input gestures. According to some embodiments, if a contact point is detected but the contact point does not include two gesture areas, the gesture may not be classified as including a touch, thereby negating the need to perform additional analysis of the sensor data of the frame. Alternatively, a gesture may be classified as including a touch, but the fact that at least one of the points on the hand surface involved in the contact point is not within the gesture area may be considered to reduce the possibility of the intentionality of the touch.

[0028] In some embodiments, multiple gesture zones may be used to determine how to classify a particular hand gesture relative to a detected touch or the intentionality of a detected touch. Figure 2An example diagram is shown in which a diagram of a hand is presented with multiple types of gesture zones. In this example, hand 200 includes two gesture zones consisting of two groups of areas. The first gesture zone includes a first area 210A at the area of ​​the fingertip of the thumb and a second area 210B at the fingertip of the index finger. The second gesture zone includes a first area 205A at the area of ​​the fingertip of the thumb and additional areas at 205B, 205C, 205D, and 205E. The two gesture zones can be used in various ways. For example, the two gesture zones can correspond to different input or gesture types. As another example, the two groups of zones can contribute differently to the determination of the intentionality of a particular gesture. For example, the zone identified by 210 can be associated with a first user input gesture, while the zone identified by 205 can be associated with a second gesture. As another example, a touch between two locations in the zone identified by 210 can more strongly indicate an intentional touch than a touch including a location in the zone identified by 205.

[0029] Figure 3 A flow chart of a technique for detecting input gestures according to some embodiments is shown. Specifically, Figure 3 A gesture estimation pipeline 300 is shown in which user input gestures are recognized and processed. Although the flowchart shows various components described as performing a particular process, it should be understood that the flow of the diagram may be different according to some embodiments, and the functions of the components may be different according to some embodiments.

[0030] Flowchart 300 begins with sensor data 302. In some embodiments, the sensor data may include captured image data and / or depth data of one or both hands of a user. In some embodiments, the sensor data may be captured from a sensor on an electronic device, such as an outward-facing camera on a head-mounted device, or a camera otherwise configured in an electronic device to capture sensor data including the user's hands. According to one or more embodiments, the sensor data may be captured by one or more cameras, which may include one or more sets of stereo cameras. In some embodiments, sensor data 302 may include additional data collected by the electronic device and related to the user. For example, the sensor data may provide position data of the electronic device, such as the position and orientation of the device.

[0031] In some embodiments, sensor data 302 may be applied to a hand tracking network 304. The hand tracking network may be a network trained to estimate the physical state of one or both hands of a user. In some embodiments, hand tracking network 304 predicts hand poses 306. The hand poses may be classified poses of the hand based on the estimated physical state, or may provide some other form of data indicating the pose of the hand. For example, in some embodiments, hand pose data 306 may include estimates of the joint positions of the hand. Additionally, in some embodiments, hand tracking network 104 may be trained to provide estimates of the position of a device (such as a headset) and / or an estimate of a simulated world space.

[0032] In some embodiments, the hand tracking network 304 may be further configured to provide touch data. The touch data may include a prediction of whether a touch is occurring between two areas on the hand for one or more given frames. The determination may be provided in the form of a touch signal 308. For example, a machine learning model may be trained to predict whether two parts of the hand are in contact. In some embodiments, the hand tracking network may additionally determine the location on the hand where contact was made during a touch. As will be described in more detail below, in some embodiments, hand tracking may predict whether a touch is occurring based on the sensor data 302 and / or the hand gesture data 306.

[0033] In some embodiments, hand tracking network 304 may be configured to determine the locations on the hand that were in contact during a touch. If both locations of the hand are within one or more gesture zones, hand tracking 304 may indicate that a touch is occurring in touch signal 308. In some embodiments, if one or both of the locations are not within a gesture zone, hand tracking data 304 may indicate that no touch is occurring via touch signal 308. Alternatively, touch signal 308 may indicate that a touch is occurring regardless of whether the touch occurs within or outside of one or more gesture zones.

[0034] According to one or more embodiments, the gesture determination framework 310 provides a determination as to whether a particular gesture presented in the sensor data 302 is intentional. That is, a determination is made as to whether a classified gesture of the hand (e.g., based on or provided by the hand gesture data 306) is intentional. When the determined hand gesture includes a touch (such as a pinch), then the gesture determination framework 310 can use the touch signal 308 provided by the hand tracking network 104 to determine whether an intentional gesture is being performed.

[0035] In some embodiments, the gesture determination framework 310 can utilize Figure 3 For example, the gesture determination framework 310 may receive signals such as user interface (UI) geometry, line of sight estimates, events generated by connected peripherals, user interactions with objects, etc. As will be described in Figure 5 As described in , gesture determination framework 310 can consider various features from the input to make a determination for a particular input gesture whether the gesture is intentional. This determination can be transmitted to UI gesture processing module 314 in the form of gesture signal 312. The gesture signal can indicate whether an intentional input gesture has occurred. In some embodiments, gesture signal 312 can also be used to indicate whether a previous gesture signal should be canceled. For example, if the user moves their position, puts down their hand, etc., this cancellation may occur.

[0036] According to some embodiments, if at least one of the points on the hand surface involved in the contact point is not within a gesture zone, the gesture determination network 310 can reduce the likelihood of intentionality of the determined touch. In some embodiments, the system can utilize multiple sets of gesture zones, and a particular set of gesture zones detected as contacting on the user's hand can differently contribute to intentionality determinations by the gesture determination network 310. In some embodiments, one or more gesture zones can be monitored so that a touch is detected only if the touch occurs within a gesture zone.

[0037] The UI gesture processing module 314 can be configured to enable user input actions based on the gesture signal 312. Certain gestures such as pinching can be associated with selection actions of UI components, etc. In some embodiments, if a cancel signal is received corresponding to a gesture signal 312 that has been initiated, the system can process the gesture differently than if the gesture was not canceled. For example, the UI component can be shown as selected but not activated, etc. As another example, a previously initiated tap caused by the user can be truncated or canceled.

[0038] Figure 4A A flow chart of a technique for implementing gesture input according to some embodiments is shown. Specifically, the flow chart presented in FIG4 depicts an example technique for detecting and processing gestures based on a gesture zone. For the purpose of explanation, the following steps will be described as being performed by specific components. However, it should be understood that various actions may be performed by alternative components. In addition, various actions may be performed in different orders. In addition, some actions may be performed simultaneously, and some actions may not be required, or other actions may be added.

[0039] Flowchart 400 begins at block 405 where hand tracking data is obtained from one or more camera frames. According to one or more embodiments, the hand tracking data may include image data and / or depth data. The hand tracking data may be obtained from one or more cameras (including stereo cameras, etc.). In some embodiments, the hand tracking data may include sensor data captured by an outward-facing camera of a head mounted device. The hand tracking data may be obtained from the hand tracking network 104 or another source that generates hand tracking data from cameras or other sensor data.

[0040] At box 410, a contact event is detected based on the hand tracking data between two parts of the hand. In some embodiments, the contact event may indicate that the touch was detected by the hand tracking network. As described above, the touch can include a contact point between two areas of the hand. Therefore, detecting the contact event may further include determining the area of ​​the hand that made contact during the contact event. For example, in some embodiments, the device may determine whether the touch location on the hand is within a predefined gesture area. In some embodiments, the device may compare the touch location with a hand model that includes a gesture area to determine whether one or both of the touch locations are within the gesture area.

[0041] At box 415, a gesture zone is selected based on the hand tracking data. In some embodiments, the system may use different gesture zones for different gestures. The gesture determination framework 310 may utilize the hand tracking data 304 to identify the attempted gesture. For example, the attempted gesture may be determined based on a posture and / or movement of one or both hands, which may be classified against predefined gestures. In some embodiments, different gestures may be associated with different gesture zones. In some embodiments, a gesture zone may be selected additionally or alternatively based on a system context (such as one or more applications currently running on the system, one or more available user input components, etc.). That is, a particular gesture may be associated with a first set of gesture zones in one context and associated with a second set of gesture zones in another context.

[0042] Flowchart 400 proceeds to block 420, where a determination is made as to whether both contact locations involved in the contact event are in the selected gesture zone. In some embodiments, the contact location may be determined by a hand tracking pipeline. The contact location may be compared to a model of a hand including the selected gesture zone to determine whether a particular contact location is within the gesture zone. Two contact points at corresponding locations on one or more hands making contact may be identified for the contact event. If at least one of the contact points on the two fingers (or portions of one or more hands) is not within the gesture zone, the flowchart ends at block 425 and the gesture is ignored. User input gestures may be ignored by actively transmitting a signal that the detected gesture is not intended, or by refraining from transmitting a gesture signal, for example, from the gesture determination framework 310 to the UI gesture processing module 314. In other words, the gesture may be classified as a non-touch, or determined to be an unintentional touch, so that no user input is generated. In some embodiments, a particular gesture zone may be associated with a particular gesture, such as an expected gesture based on the current user interface, a component on the user interface, an application context, and the like.

[0043] Returning to block 420, if it is determined that both contact locations are within the selected gesture area, flowchart 400 ends at block 430 and the user input action associated with the gesture is enabled. In some embodiments, enabling the user input action may include triggering the UI gesture processing module to initiate the input action associated with the gesture.

[0044] Figure 4B A flow chart of a technique for handling different stages of touches according to some embodiments is shown. In particular, the flow chart presented in FIG. 4 depicts an example technique for detecting and processing input events on a per-frame basis. For purposes of explanation, the following steps will be described as being performed by specific components. However, it should be understood that various actions may be performed by alternative components. In addition, various actions may be performed in different orders. In addition, some actions may be performed simultaneously, and some actions may not be required, or other actions may be added.

[0045] Flowchart 450 begins at block 455, where a touch is detected within the gesture area. A touch may be detected for a given image frame and / or other sensor data. In some embodiments, a contact event may include a touch and a release of a touch. A touch may be detected for a given frame when contact is detected between two surface locations on one or both hands of a user. Additionally, in some embodiments, a touch may be detected from touch signal 308, as described above with respect to Figure 3 described.

[0046] The flow chart continues at box 460, where visual feedback is generated for the UI component in response to the touch. In some embodiments, a gesture-based input event may include multiple stages, including visual feedback and input actions. Visual feedback may provide a visual indication of the UI component that is the subject of the gesture detected by the contact event. For example, visual feedback may include an amplification or highlighting of the UI component in the user interface, such as a hover effect. In some embodiments, during the visual feedback, an action or process that may be triggered by the UI component is not activated. Instead, the visual feedback serves as an indication to the user of the UI component that the component will be selected or otherwise activated by the gesture.

[0047] The flowchart proceeds to box 465, and one or more additional frames are captured. In some embodiments, the touch determination is made on a per-frame basis, or is sampled from a set of continuous frames. At box 470, a determination is made as to whether a touch release is detected from an additional frame. For a given frame, if a touch is no longer detected, and a touch was previously detected in a previous frame, a touch release can be detected. In other words, if no contact is detected for a given frame and it is determined that the hand is performing a touch in the most recently processed frame, a touch release can be detected. If a touch release is not detected (i.e., if it is determined that the hand continues to perform a touch), the flowchart 450 returns to box 460 and continues to generate visual feedback.

[0048] Returning to block 470, if it is determined that a touch release is detected, the flowchart ends at block 475 and a user input action is initiated. That is, upon release of the touch-based input, a UI component may be selected and an associated process may be executed.

[0049] As described above, the gesture determination framework can be configured to generate an intentionality classification of a gesture.The gesture determination framework 310 can be configured to estimate a hand pose or gesture and determine whether the gesture is intended to trigger a user input action. Figure 5 A flow chart of a technique for classifying the intentionality of a gesture according to some embodiments is shown. For purposes of explanation, the following steps will be described as being performed by Figure 3 However, it should be understood that various actions can be performed by alternative components. Various actions can be performed in different orders. In addition, some actions can be performed simultaneously, and some actions may not be required, or other actions can be added.

[0050] Flowchart 500 begins at box 505, where a touch event is detected based on contextual data from a hand tracking network. The touch may be detected, for example, based on a touch signal 508 received from a hand tracking network 504. According to some embodiments, some gestures may require a touch, such as a pinch, etc. In addition, different types of touches may be used to identify multiple types of pinches. According to some embodiments, not every gesture requires a touch. Therefore, a touch may not be detected, or a touch signal may indicate that a touch occurred. In some embodiments, a touch signal may not be received or may be otherwise ignored, and the gesture may still be recognized.

[0051] Flowchart 500 continues to box 510, where the touch stage is determined based on the hand tracking data. The touch stage may indicate what stage of the touch action the finger is currently in for a given frame. According to some embodiments, the features that are of interest when determining intentionality may vary depending on the current state of the gesture. For gestures that include pinching or other touch actions, the stage that the gesture is currently in may affect the ability to enable, cancel, or reject the associated input action. Some examples of touch stages include an idle state, an entry state where a contact event is beginning, such as a pinch down stage. A hold state (where a pinch is currently occurring), and an exit stage (e.g., when an upward pinch occurs so that the pinch ends). The following will discuss touch stages in detail. Figure 6 The various touch phases are described in more detail.

[0052] At box 515, estimate low-level features associated with the touch. The low-level features can be determined based on the hand tracking data, and / or the additional data may include an estimate of what the hand is doing during the frame. For example, other data sources include posture information of the device used to capture the hand tracking data, hand posture, UI geometry, etc. In some embodiments, the low-level features are determined without considering the intent. Examples of low-level features include, for example, pinch speed when pinching down, measurement of wrist flexion, finger curling, proximity of the hand to the head, hand speed, etc.

[0053] Flowchart 500 continues to box 520, where high-level, low-state features are estimated. High-level low-state features may include modal features that estimate what the user is doing during the touch in order to determine intentionality. In some embodiments, high-level features may be interoperable and individually verifiable features. Examples include estimates of whether the hand is using one or more peripherals, the repetition frequency of gestures (e.g., if the user is pinching quickly), if the hand is holding an object, if the hand is in a resting position, a specific pinch or gesture style (i.e., a pinch using the pad of two fingers or using the side of a finger). In some embodiments, high-level features may be based on user activity, such as the user fidgeting, talking, or reading. According to one or more embodiments, high-level features may be determined based on hand tracking data, the determined touch phase, and / or the estimated basic features. In some embodiments, high-level features may directly determine the intentionality of the action. For example, if the user is using a peripheral device such as a keyboard, the pinch may be rejected, or the gesture may be determined to be unintentional.

[0054] According to one or more embodiments, gesture zones can be considered high-level features. For example, in some embodiments, whether a contact point on a hand is within a gesture zone can be considered a high-level feature. Additionally, in some embodiments, multiple sets of gesture zones, such as for different gesture styles or types, can be considered. Thus, intentionality can be determined directly or indirectly based at least in part on a touch occurring between two contact locations within an area of ​​a gesture zone.

[0055] The flowchart ends at box 525, where the gesture determination framework 310 combines the high-level features and the touch stage to classify intentionality. In some embodiments, the gesture determination framework 310 uses a conditional combination of high-level features and the touch stage to classify intentionality. This classification can then be used to signal a gesture to be processed as an input gesture (thereby activating an associated UI input action), cancel an associated action if the gesture is determined to be unintentional (e.g., if a UI action associated with the gesture has already been initiated), or ignore the gesture.

[0056] Figure 6 A flow chart of a technique for classifying touch phases according to some embodiments is shown. Figure 6 Describes a flow chart for determining the touch stage, as described above with respect to Figure 5 Specifically, Figure 6 can be interpreted as a state machine that determines the state of touches in a given frame. Figure 6 For the purposes described in, a touch indicates contact between two parts of a hand. For example, but not limited to, a contact event between two fingers or a contact event between a finger pad and another part of a hand can be considered a touch.

[0057] In some embodiments, a particular touch phase or movement between touch phases may be determined based on a set of heuristics using image data and / or depth data from a particular frame (e.g., from hand tracking data). Figure 7 As described, in some embodiments, a machine learning model may be used to determine the touch phase.

[0058] Process 600 begins at 605, where the hand is not being tracked. For example, during the idle phase 610, the hand may not be tracked. Because the hand is not being tracked during this phase, the touch phase may actually be in the enter, hold, or exit phase, but the actual phase may not be recognized because the hand is not being tracked. For example, the hand may not be tracked when the hand tracking process is not running, or when the touch detection process is not running. That is, according to one or more embodiments, hand tracking may occur for other purposes rather than for touch detection. As another example, the touch phase may be tracked at a different power level than during an active tracking session. In the idle phase 610, no touch may be detected between two parts of the hand for a predetermined amount of time. For example, two fingers may not have touched within a certain number of previous frames.

[0059] When the tracked hand moves in a manner that would predictably lead to touch, then flow continues to box 615 where the start of a touch is detected. When the start of a touch is detected at 615, then the touch phase enters enter state 620. In enter state 620, a touch is expected but has not yet occurred. This may occur, for example, during a "pinch-down" where two fingers move toward each other, or during another movement where parts of the hand move toward each other. For example, movement may be detected based on the posture of the hand over a set of frames, such as comparing the current posture of the hand in the current frame to a previous posture of the hand in a previous frame. Additionally or alternatively, a machine learning model may be used to detect the start of a touch, as described below with respect to Figure 7 as described.

[0060] Entering state 720 may occur on a single frame in several frames. After entering state 720, a touch may be determined to occur, as shown at 725, or may be determined to no longer occur, as shown in the case of stopping the detected touch at 755. When a touch is determined to occur for a particular frame, a continuous touch may be detected, as shown at 725. This causes the touch phase to enter a hold phase 730. In the hold phase, a touch is currently occurring. That is, two areas of the hand may be determined to be in contact. For example, the index finger and thumb may be determined to be touching in a pinch gesture.

[0061] Returning to the entry state 620, if subsequent frames indicate that a touch is no longer detected, as shown at 655 where stop entry is detected, the touch phase enters the exit phase 640. In the exit phase 640, a touch is no longer detected. This may occur, for example, if the determination that a touch was being initiated in the entry state 620 is incorrect. For example, the user may have initiated movement of two fingers toward each other but never actually touched the two fingers together. For example, the touch phase may exit the entry state 620 after a timeout period or a predetermined number of frames in which no sustained touch is detected.

[0062] In the event that the touch phase enters the hold phase 630, then the hold phase 630 occurs when the touch occurs. For example, the touch phase may be maintained for each frame in which the two parts of the hand are determined to be in contact. For example, this may occur if the pinch is performed very slowly so that contact occurs over multiple frames, or if the pinch is maintained during a dragging motion, etc. For subsequent frames, once it is detected that the two parts of the hand are no longer in contact, the touch phase may no longer be detected, as shown by 635 detecting the cessation of the touch. Therefore, the touch phase enters the exit phase 640. As described above, the exit phase 640 may indicate that the touch never actually occurred, or that the touch is ending when the exit phase 640 is entered from the hold phase 630. For example, this may occur when the parts of the hand that were previously in contact move away from each other (such as in an "up pinch").

[0063] If entry 645 is not detected for one or more subsequent frames, flow 600 can continue from exit stage 640 to idle stage 610. Alternatively, a start touch can be detected from exit stage 640, as shown at 650, and the touch stage can re-enter enter state 620. This can occur, for example, when a user quickly progresses from pinching up to pinching down, etc. Flow 600 can continue while frames of hand tracking data are received.

[0064] A touch signal indicating that a touch event is occurring for a frame may be determined in a variety of ways. For example, in some embodiments, a heuristic method may be used based on hand tracking data to determine whether a touch has occurred and / or the current touch stage. Figure 7 A flow diagram of an action network providing an example machine learning process for determining whether a touch has occurred is shown according to some embodiments.

[0065] The pipeline 700 begins with a set of frames 702 as input. The frames 702 can be a time sequence of image frames of a hand captured by one or more cameras. The cameras can be single cameras, stereo cameras, cameras whose camera exposures have been synchronized, or a combination thereof. The cameras can be located on a user's electronic device such as a mobile device or a head mounted device. The frames can include a series of one or more frames associated with a predetermined time. For example, the frames 702 can include a series of individual frames captured at consecutive times, or can include multiple frames captured at each time in the consecutive time. The entirety of the frames can represent a sequence of motions of the hand, based on which a touch may or may not be detected at any particular time.

[0066] Frame 702 may be applied to gesture model 704. Gesture model 704 may be a trained neural network configured to predict a 3D pose 708 of a hand based on a given frame (or set of frames, such as in the case of a stereo camera) at a given time. That is, each frame of frame set 702 may be applied to gesture model 704 to generate 3D pose 708. Thus, the gesture model may predict the pose of the hand at a particular point in time. In some embodiments, geometric features 712 may be derived from 3D pose 708. The geometric features may indicate relational features between joints of the hand that may be identified by the 3D pose. That is, in some embodiments, 3D pose 708 may indicate the position and location of joints in the hand, while geometric features 712 may indicate spatial relationships between joints. For example, geometric features 712 may indicate the distance between two joints, etc.

[0067] In some embodiments, the frame 702 may be additionally applied to an encoder 706 that is trained to generate potential values ​​for a given input frame (or frames) from a specific time indicating the appearance of the hand. Appearance features 710 may be features that can be identified from the frame 702 but are not particularly useful for the gesture. Therefore, these appearance features may be ignored by the gesture model 704, but may be used within the pipeline 700 to determine whether a touch has occurred. For example, the appearance features 710 may be complementary features of the geometric features 712 or the 3D gesture 708 to advance the goal of determining a specific action 720 (such as whether a touch has occurred). According to some embodiments, the encoder 706 may be part of a network associated with the gesture model 704, so that the encoder may use some of the gesture data to predict the appearance features. In addition, in some embodiments, the 3D gesture 708 and the appearance features 710 may be predicted by a single model or two separate, unrelated models. The result of the encoder 706 may be, for example, a set of appearance features 710 in the form of a set of fingerprints.

[0068] According to one or more embodiments, at least one of 3D gesture 708, geometric features 712, and appearance features 710 may include data related to a pinch zone. That is, in some embodiments, gesture model 704 and / or encoder 706 may be configured to generate data indicating whether the contact point involves one or more regions of the hand that include a pinch zone.

[0069] The fusion network 714 is configured to receive the geometric features 712, the 3D pose 708, and the appearance features 710 as inputs and generate a set of encodings 716 each time. The fusion network 714 can combine the geometric features 712, the 3D pose 708, and the appearance features 710 in any number of ways. For example, the various features can be weighted differently in the combination, or otherwise combined in different ways to obtain a set of encodings 716 each time.

[0070] The encoding is then run through the temporary network 718 to determine the action 720 each time. The action 720 may indicate, for example, whether a touch has occurred or a change in the stage of touch. The temporary network 718 may consider both the frame (or set of frames) at the specific time used to determine the action 720 as well as the other frames in the frame set 702.

[0071] Figure 8 An example diagram is shown in which a slide-out operation is used to cancel a gesture. Specifically, Figure 8 1 shows a system setup for a user to interact with a device using gesture input according to some embodiments. It should be understood that Figure 8 The various features and descriptions are provided for illustrative purposes and are not necessarily intended to limit the scope of the present disclosure.

[0072] Hand 805 is presented over the course of three frames 800A, 800B, and 800C. For the purposes of this example diagram, the three frames 800A, 800B, and 800C are intended to depict views of hand 805 at three points in time. First frame 800A shows hand 805A having a first gesture zone 810A and a second gesture zone 815A. Contact point 820 is detected where the two regions of hand 805A are in contact. Because contact point 820 includes a position on the surface of two portions of the hand located within the regions of the gesture zone (i.e., first region 810A and second region 815A), the gesture of hand 805A is classified as a touch. Thus, in some embodiments, a gesture-based input event may be triggered in association with the detected touch. For example, visual feedback may be generated in association with a UI component that may be selected by a gesture.

[0073] Turning to the second frame 800B, hand 805B is shown in a pose where the user has begun to slide their fingers together. Thus, contact point 830 has moved down along the fingers compared to contact point 820 in frame 800A. However, contact point 830 still includes two portions of the user's hand that make contact within the touch zones in regions 810B and 815B. Thus, the hand continues to be considered to be in touch, and visual feedback can continue to be presented.

[0074] In contrast, turning to third frame 800C, hand 805C is shown in a pose where the user has continued to slide their fingers together. Thus, contact point 840 has moved down along the fingers compared to contact point 820 in frame 800A and contact point 830 in frame 800B. Here, contact point 840 does not include two surface areas in the gesture zone. Because contact point 840 includes at least one surface of two parts of the hand that is not within the gesture zone (i.e., first gesture zone 810C and second gesture zone 815C), the pose of hand 805C is not classified as a touch, or is otherwise determined to be an unintentional gesture.

[0075] In some embodiments, an action initiated based on a previously determined pinch may be canceled based on movement of the user's fingers during the touch so that the two contact points are no longer in the gesture zone. As shown here, the user may maintain contact between the two parts of the hand, but one or both of the contact points may not be within these zones, and the pinch may no longer be considered intentional. As such, the associated user input action for the gesture-based input event may not be performed.

[0076] According to one or more embodiments, upon detecting that the pinch state moves from an entering or holding state to an exiting state, a cancel signal may be transmitted to the UI component to cancel the previously initiated input action. The cancel signal may cause the system to process the associated gesture in a different manner than when it was not canceled. For example, the UI component may be shown as having visual feedback but not activated, etc. As another example, a previously initiated stroke drawn by the user may be truncated or canceled. In some embodiments, the UI gesture processing module 114 may present an indication that the gesture is recognized (e.g., by highlighting the associated UI component), but will refrain from completing the input action.

[0077] In some embodiments, other characteristics of the finger movement can cause a gesture-based input event to be canceled. For example, if a user slides their fingers along one another so that the contact point travels a predetermined distance along the finger or hand, the gesture event can be canceled even if the entire movement of the contact point remains within the gesture zone.

[0078] Steering Fig. 9 , presents a flow chart of a technique for canceling gesture-based user input based on a change in the classification of a hand posture. For purposes of explanation, the following steps will be described as being performed by specific components. However, it should be understood that various actions may be performed by alternative components. Furthermore, various actions may be performed in different orders. Additionally, some actions may be performed simultaneously, and some actions may not be required, or other actions may be added.

[0079] Flowchart 900 begins at box 905, where hand tracking data is obtained from a camera frame. The hand tracking data may include, for example, image data, depth data, etc. In some embodiments, the hand tracking data may include or be based on additional sensor data, such as image data and / or depth data capturing one or more hands of a user. In some embodiments, the sensor data may be captured from a sensor on an electronic device, such as an outward-facing camera on a head-mounted device, or a camera otherwise configured in an electronic device to capture sensor data including the user's hands. In some embodiments, the sensor data may include additional data collected by the electronic device and related to the user. For example, the sensor data may provide position data for the electronic device, such as the position and orientation of the device.

[0080] At block 910, a contact event is detected based on the hand tracking data between two parts of a hand. In some embodiments, a contact event can be detected by identifying a touch between two surfaces of a hand or multiple hands. In some embodiments, a touch can be identified when the hand tracking data indicates that the hand tracking network has detected the touch.

[0081] Flowchart 900 proceeds to block 915, where a determination is made as to whether both contact locations are within a gesture zone. In some embodiments, the contact locations may be determined by a hand tracking pipeline. The contact locations may be compared to a model of a hand that includes a predefined gesture zone to determine whether a particular contact location is within a gesture zone. A gesture zone may be specific to or associated with a particular gesture that may be recognized by a hand tracking network. If it is determined that neither contact point is within a gesture zone, the flowchart ends at block 920, and the gesture is disregarded for user input.

[0082] Returning to block 915, if both areas of the hand that are in contact with each other are within the gesture zone, the flowchart proceeds to block 925, where visual feedback is presented with respect to the UI component. The UI component to which visual feedback is applied may be based on the UI component that is the subject of the detected input gesture. In some embodiments, the UI component may be based on gesture and / or line of sight detection.

[0083] Flowchart 900 proceeds to block 930, where additional sensor data is received in the form of camera frames. That is, in some embodiments, the techniques described in flowchart 900 may be performed on a per-frame basis, or otherwise performed periodically. Flowchart 900 then returns to block 910 to determine whether touch events continue to be detected.

[0084] At block 935, it is determined whether a touch release is detected from an additional frame. For a given frame, if a touch is no longer detected and the touch was previously detected in a previous frame, a touch release may be detected. In other words, if no contact is detected for a given frame and it is determined that the hand is performing a touch in the most recently processed frame, a touch release may be detected. If a touch release is not detected (i.e., if it is determined that the hand continues to perform a touch), flowchart 900 returns to block 925 and continues to present visual feedback.

[0085] Returning to box 935, if a release is detected, the flowchart proceeds to box 940. At box 940, a determination is made as to whether one or more cancellation parameters are met. The cancellation parameters may include characteristics of the hand tracking data that indicate a determination of a change in intentionality such that the associated input action should no longer be performed. For example, the cancellation parameters may include a determination that the contact positions are not all within the gesture zone area at the time of release. As another example, the cancellation parameters may include a satisfying threshold distance along the contact position of the finger or hand in order to qualify as a shift of a "swipe away" gesture. If it is determined at box 940 that one or more cancellation parameters are not met, the flowchart ends at box 945, and an input action associated with a gesture-based input event may be initiated.

[0086] Returning to block 940, if it is determined that one or more cancellation parameters are satisfied, the flowchart ends at block 950 and the input action is canceled. According to some embodiments, a cancel signal may be transmitted to cancel the current input action. The cancel signal may cause the input action associated with the gesture-based input event to be disregarded, or otherwise cause the gesture-based input event to be processed differently than if the gesture-based input event was not canceled or executed.

[0087] refer to Fig.10 , depicting a simplified block diagram of an electronic device 1000. The electronic device 1000 may be part of a multi-function device, such as a mobile phone, a tablet computer, a personal digital assistant, a portable music / video player, a wearable device, a head-mounted system, a projection-based system, a base station, a laptop computer, a desktop computer, a network device, or any other electronic system such as described herein. The electronic device 1000 may include one or more additional devices, such as a server device, a base station, an auxiliary device, etc., and various functions may be included in the one or more additional devices or various functions may be distributed on the one or more additional devices. Exemplary networks include, but are not limited to, local networks (such as universal serial bus (USB) networks), local area networks of organizations, and wide area networks (such as the Internet). According to one or more embodiments, the electronic device 1000 is used to interact with the user interface of the application 1035. It should be understood that the various components and functions within the electronic device 1000 may be distributed differently on modules or components or even additional devices.

[0088] The electronic device 1000 may include one or more processors 1020, such as a central processing unit (CPU) or a graphics processing unit (GPU). The electronic device 1000 may also include a memory 1030. The memory 1030 may include one or more different types of memory, which may be used to perform device functions in conjunction with the processor 1020. For example, the memory 1030 may include a cache, a ROM, a RAM, or any kind of transient or non-transient computer-readable storage medium capable of storing computer-readable code. The memory 1030 may store various programming modules for execution by the processor 1020, including a tracking module 1045 and various other applications 1055. The electronic device 1000 may also include a storage device 1040. The storage device 1040 may include one or more non-transient computer-readable media, including, for example, magnetic disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and digital video discs (DVDs)), and semiconductor memory devices (such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs)). Storage 1030 may be used to store various data and structures that may be used to store data related to hand tracking, gesture zones, and UI preferences. For example, gesture zones may be specific to a user. According to one or more embodiments, storage 1040 may be configured to store hand tracking network 1055. In addition, storage 1040 may include registration data 1050, which may be captured by a user of electronic device 1000 during a registration process and may include image data and other sensor data specific to the user, as well as data for driving a virtual representation of the user, such as geometry, bone length, etc.

[0089] The electronic device 1000 may also include one or more cameras 1005 or other sensors 1010, such as a depth sensor from which the depth of the scene can be determined. In one or more embodiments, each of the one or more cameras 1005 can be a traditional RGB camera or a depth camera. In addition, the camera 1005 may include a stereo camera or other multi-camera system. In addition, the electronic device 1000 may include other sensors that can collect sensor data for tracking user movement, such as a depth camera, an infrared sensor, or an orientation sensor, such as one or more gyroscopes, accelerometers, etc.

[0090] According to one or more embodiments, the memory 1030 may include one or more modules including computer readable code that can be executed by the processor 1020 to perform functions. The memory 1030 may include, for example, a tracking module 1045 and one or more applications 1035. The tracking module 1045 can be used to track the position of the hand and other user movements in the physical environment. The tracking module 1045 can use sensor data, such as data from the camera 1005 and / or the sensor 1010. In some embodiments, the tracking module 1045 can track user movement to determine whether to trigger user input based on the detected input gesture and in particular using the gesture area. The electronic device 1000 may also include a display 1025 that can present a UI for user interaction. For example, the UI may be associated with one or more applications in the application 1035. The display 1025 may be an opaque display, or may be translucent or transparent. The display 1025 may include an LED, an OLED, a digital light projector, a silicon-based liquid crystal, etc.

[0091] Although electronic device 1000 is depicted as including numerous components described above, in one or more embodiments, various components may be distributed across multiple devices. Thus, although certain calls and transmissions are described herein with respect to the particular system depicted, in one or more embodiments, various calls and transmissions may be directed differently based on the differently distributed functionality. Additionally, additional components may be used, and certain combinations of the functionality of any component may be combined.

[0092] According to one or more embodiments, the electronic device 1000 may be communicatively connected to one or more additional devices, such as the electronic device 1070, via the network 1060. The network 1060 may be a local area network, a wide area network, etc. For example, the electronic device 1000 may be communicatively connected to the electronic device 1070 via the Internet. In some embodiments, the electronic device 1000 and the electronic device 1070 may participate in a coexistence session. Additionally or alternatively, the electronic device 1070 may be an auxiliary device of the electronic device 1000 and perform at least some of the functions described above with respect to the electronic device 1000. In this way, the electronic device 1070 may include components similar to those described above with respect to the electronic device 1000. For example, the electronic device 1070 may include one or more processors, such as a processor 1075 and a memory 1080, which may contain instructions executable by the processor 1075. In addition, the electronic device 1070 may include one or more I / O devices 1085, such as a keyboard, a microphone, a speaker, etc.

[0093] Reference now Fig.11, which shows a simplified functional block diagram of an exemplary multifunctional electronic device 1100 according to one embodiment. The electronic device may be a multifunctional electronic device, or may have some or all of the components described herein for the multifunctional electronic device. The multifunctional electronic device 1100 may include a processor 1105, a display 1110, a user interface 1115, graphics hardware 1120, device sensors 1125 (e.g., proximity sensors / ambient light sensors, accelerometers and / or gyroscopes), microphones 1130, audio codecs 1135, speakers 1140, communication circuits 1145, digital image capture circuits 1150 (e.g., including a camera system), video codecs 1155 (e.g., supporting a digital image capture unit), memory 1160, storage devices 1165, and communication buses 1170. The multifunctional electronic device 1100 may be, for example, a digital camera or a personal electronic device such as a personal digital assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

[0094] The processor 1105 may execute instructions necessary to implement or control the operation of many functions performed by the device 1100 (e.g., such as the generation and / or processing of images disclosed herein). The processor 1105 may, for example, drive the display 1110 and may receive user input from the user interface 1115. The user interface 1115 may allow a user to interact with the device 1100. For example, the user interface 1115 may take on a variety of forms, such as buttons, keypads, dials, click wheels, keyboards, display screens, touch screens, sight lines, and / or gestures. The processor 1105 may also, for example, be a system on a chip, such as a system on a chip found in a mobile device, and include a dedicated GPU. The processor 1105 may be based on a reduced instruction set computer (RISC) or a complex instruction set computer (CISC) architecture or any other suitable architecture, and may include one or more processing cores. The graphics hardware 1120 may be dedicated computing hardware for processing graphics and / or assisting the processor 1105 to process graphics information. In one embodiment, the graphics hardware 1120 may include a programmable GPU.

[0095] The image capture circuit 1150 may include two (or more) lens assemblies 1180A and 1180B, wherein each lens assembly may have a separate focal length. For example, relative to the focal length of lens assembly 1180B, lens assembly 1180A may have a short focal length. Each lens assembly may have a separate associated sensor element 1190. Alternatively, two or more lens assemblies may share a common sensor element. The image capture circuit 1150 may capture still images and / or video images. The output from the image capture circuit 1150 may be processed by a video codec 1155 and / or a processor 1105 and / or graphics hardware 1120 and / or a dedicated image processing unit or pipeline incorporated within the circuit 1165. The images captured in this way may be stored in the memory 1160 and / or the storage device 1165.

[0096] The sensor and camera circuit 1150 may capture still images and video images that may be processed at least in part by the following devices according to the present disclosure: the video codec 1155 and / or the processor 1105 and / or the graphics hardware 1120, and / or a dedicated image processing unit incorporated into the circuit 1150. The images captured in this way may be stored in the memory 1160 and / or the storage device 1165. The memory 1160 may include one or more different types of media used by the processor 1105 and the graphics hardware 1120 to perform device functions. For example, the memory 1160 may include a memory cache, a read-only memory (ROM) and / or a random access memory (RAM). The storage device 1165 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. The storage device 1165 may include one or more non-transitory computer-readable storage media, including, for example, disks (fixed hard disks, floppy disks, and removable disks) and tapes, optical media (such as CD-ROMs and DVDs), and semiconductor memory devices (such as EPROMs and EEPROMs). The memory 1160 and storage device 1165 can be used to tangibly hold computer program instructions or codes organized into one or more modules and written in any desired computer programming language. When executed by, for example, the processor 1105, such computer program code can implement one or more of the methods described herein.

[0097] The various processes defined herein contemplate the option of obtaining and utilizing identifying information of users. For example, such personal information may be utilized in order to track the user's movements. However, to the extent such personal information is collected, such information should be obtained with the user's informed consent, and the user should be aware of and control the use of their personal information.

[0098] Personal information will be used by appropriate parties only for lawful and reasonable purposes. Parties utilizing such information will adhere to privacy policies and practices that at a minimum comply with applicable laws and regulations. In addition, such policies should be comprehensive and meet or exceed government / industry standards. In addition, parties may not distribute, sell or otherwise share such information except for any reasonable and lawful purpose.

[0099] Furthermore, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. Additionally, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. Where appropriate, de-identification can be facilitated by removing specific identifiers (e.g., date of birth), controlling the amount or characteristics of stored data (e.g., collecting location data at a city level rather than an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.

[0100] It should be understood that the above description is intended to be illustrative and not limiting. The material has been presented to enable any person skilled in the art to make and use the disclosed subject matter protected by the claims, and the material is provided in the context of specific embodiments, variations of which will be apparent to those skilled in the art (for example, some of the disclosed embodiments may be used in conjunction with each other). Therefore, Figures 3 to 7 and Fig. 9 The specific arrangement of steps or actions shown in Figure 1 to Figure 2 , Figure 8 and Figure 10 to Figure 11 The arrangement of elements shown in the accompanying drawings should not be construed as limiting the scope of the disclosed subject matter. Therefore, the scope of the present invention should be determined by reference to the appended claims and the full scope of equivalents to which such claims are entitled. In the appended claims, the terms "including" and "wherein" are used as the plain English equivalents of the corresponding terms "comprising" and "wherein".

Claims

1. A method comprising: Obtain hand tracking data based on a series of camera frames; detecting a touch between a first finger and a second finger in a first frame based on the hand tracking data; detecting a first contact position of the touch at a first point on the first finger; initiating a gesture-based input event corresponding to a first gesture based on determining that the first contact position is within a predefined gesture area for a first gesture; detecting the touch at a second contact location at a second point on the second finger in a second frame; as well as The gesture-based input event is canceled based on a difference between the first point and the second point on the first finger. 2 . The method of claim 1 , wherein the contact event comprises the touch and the release of the touch, and wherein the gesture-based input event comprises visual feedback and an input action. The method of claim 2 , wherein the touching triggers the visual feedback of the gesture-based input event.

4. The method of claim 2, wherein the first contact location is within a first predefined gesture area for the first gesture in the first frame, and The input action is canceled according to the second contact position being outside the first predefined gesture area for the first gesture in the second frame.

5. The method of claim 2, wherein canceling the gesture-based input event according to a difference between the first point and the second point on the first finger comprises: It is determined that a distance between the first contact location and the second contact location satisfies a predetermined threshold distance within a predefined time period. The method of claim 1 , wherein obtaining the hand tracking data comprises obtaining a series of camera frames, wherein the touch is detected based on the series of camera frames, and wherein contact determination is performed for each frame in the series of camera frames. The method of claim 1 , wherein the first gesture comprises a pinch.

8. A computer program product comprising computer readable code for performing the following operations: obtaining hand tracking data based on one or more camera frames; detecting a touch between a first finger and a second finger in a first frame based on the hand tracking data; detecting a first contact position of the touch at a first point on the first finger; initiating a gesture-based input event corresponding to a first gesture based on determining that the first contact position is within a first predefined gesture area for a first gesture; detecting the touch at a second contact location at a second point on the second finger in a second frame; as well as The gesture-based input event is canceled based on a difference between the first point and the second point on the first finger.

9. The computer program product of claim 8, wherein a contact event comprises the touch and a release of the touch, and wherein the gesture-based input event comprises visual feedback and an input action.

10. The computer program product of claim 9, wherein the touch triggers the visual feedback of the gesture-based input event.

11. The computer program product according to claim 9, wherein the first contact position is within a first predefined gesture area for the first gesture in the first frame, and The input action is canceled according to the second contact position being outside the first predefined gesture area for the first gesture in the second frame.

12. A computer program product according to claim 9, wherein the computer-readable code for canceling the gesture-based input event based on the difference between the first point and the second point on the first finger includes computer-readable code for determining that the distance between the first contact position and the second contact position satisfies a predetermined threshold distance within a predetermined time period.

13. A computer program product according to claim 8, wherein the computer-readable code for obtaining the hand tracking data includes computer-readable code for obtaining a series of camera frames, wherein the touch is detected based on the series of camera frames, and wherein contact determination is performed for each frame in the series of camera frames.

14. The computer program product of claim 8, wherein the first gesture comprises a pinch.

15. A system comprising: one or more processors; and One or more computer-readable media comprising computer-readable code executable by the one or more processors to: obtaining hand tracking data based on one or more camera frames; detecting a touch between a first finger and a second finger in a first frame based on the hand tracking data; detecting a first contact position of the touch at a first point on the first finger; initiating a gesture-based input event corresponding to a first gesture based on determining that the first contact position is within a first predefined gesture area for a first gesture; detecting the touch at a second contact location at a second point on the second finger in a second frame; as well as The gesture-based input event is canceled based on a difference between the first point and the second point on the first finger.

16. The system of claim 15, wherein a contact event comprises the touch and a release of the touch, and wherein the gesture-based input event comprises visual feedback and an input action. The system of claim 16 , wherein the touch triggers the visual feedback of the gesture-based input event.

18. The system according to claim 16, wherein the first contact position is within a predefined gesture area for the first gesture in the first frame, and The input action is canceled according to the second contact position being outside the first predefined gesture area for the first gesture in the second frame.

19. A system according to claim 16, wherein the computer-readable code for canceling the gesture-based input event based on the difference between the first point and the second point on the first finger includes computer-readable code for determining that the distance between the first contact position and the second contact position satisfies a predetermined threshold distance within a predetermined time period.

20. The system of claim 15, wherein the first gesture comprises a pinch.

Citation Information

Cited By

  • Gesture interaction method and device in XR environment, electronic equipment and product

    CN120428869A