Input recognition in 3d environment

By combining sensor data, automatically selecting the interaction mode, accurately explaining the user's direct and indirect interaction in three-dimensional space, solving the problems of accurate and inefficient interaction recognition in the prior art, and achieving a more efficient and natural user interaction experience.

CN119065545APending Publication Date: 2024-12-03APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410708803.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-05-29
Filing Date
2024-06-03
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

When evaluating the interaction between users and electronic devices, existing user interaction systems are difficult to effectively distinguish and process the direct and indirect interaction modes of users in three-dimensional space, resulting in low accuracy and inefficiency in interaction recognition.

Method used

By using sensor data (such as image, depth sensor data, hand motion data) combined with direct and indirect interaction recognition processes, automatically or manually selecting the interaction mode, determine the interaction type of virtual elements based on hand position and gaze direction, and realize user activity accurate explanation.

Benefits of technology

It improves the accuracy and efficiency of user interaction, provides a more flexible and natural user experience, adapts to different interaction scenarios, and enhances the way users interact with virtual elements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065545A_ABST
    Figure CN119065545A_ABST
Patent Text Reader

Abstract

The invention relates to input recognition in a 3D environment. Various implementations disclosed herein include devices, systems, and methods for interpreting user activity when a user interacts with a user interface (UI) element positioned within a three-dimensional (3D) space, such as an augmented reality (XR) environment. Some implementations enable a user to interact with virtual elements displayed in a 3D environment utilizing alternative input modalities, such as an XR environment that interprets user activity as direct or indirect interaction with virtual elements.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 470,811, filed on June 2, 2023, the contents of which are incorporated herein in their entirety. Technical Field

[0003] The present disclosure generally relates to evaluating user interactions with electronic devices involving hand and body gestures, gaze, voice, and / or other user activities. Background Art

[0004] Existing user interaction systems can be improved in terms of facilitating interactions based on user activities. Summary of the Invention

[0005] Various embodiments disclosed herein include devices, systems, and methods for interpreting user activities when the user interacts with virtual elements (e.g., user interface elements) positioned within a three-dimensional (3D) space, such as an extended reality (XR) environment. Some embodiments enable a user to interact with virtual elements displayed in a 3D environment that utilizes an alternative input modality, such as an XR environment that interprets user activities as direct interactions or indirect interactions with the virtual elements. A determination may be made based on sensor data (e.g., images, depth sensor data, motion data of a device held or worn by the user, etc.) and / or using one or more modality distinction criteria as to whether one of multiple input modality schemes (e.g., using a direct interpretation process or an indirect interpretation process) should be used to interpret the user activities.

[0006] In some embodiments, an XR environment may enable a user to provide input using both (a) a direct input modality, in which the user interacts with virtual content by virtually touching the virtual content (and / or being close to the virtual content as determined via one or more proximity criteria), and (b) an indirect input modality, in which the user interacts with the virtual content using gaze, pointing gestures, an input device, etc. to interact with the content without having to virtually touch or be close to the virtual content.

[0007] In some implementations, for direct interaction, a virtual element (e.g., a user interface (UI) element) is identified based on the intersection of the user's hand position and a 3D area associated with the virtual element (or proximity to the 3D area as determined via one or more proximity criteria). Based on determining that a direct input modality is applicable, a direct interaction recognition process is applied to interpret the user activity, for example, to determine whether the gesture / hand motion is a tap, a swipe, etc. During such user activity, the hand (or other body part) virtually interacts with the virtual element by virtually contacting the UI element. For example, the user can move their fingers to directly tap, pinch, swipe, or otherwise interact with the UI element in 3D space.

[0008] In some embodiments, for indirect interaction, the virtual element that is the subject of the user interaction is identified in another manner (e.g., based on the user's gaze direction and associated hand gesture). Based on determining that the indirect input modality is applicable, an indirect interaction recognition process is applied to interpret the user activity (e.g., gaze direction and associated hand pinch) as initiating or performing an action with respect to the gaze-identified virtual element. For example, the user activity can be indirect in the sense that the hand does not need to be in contact with or close to the UI element in 3D space. For example, when the UI is displayed at a distance in 3D space relative to the user's viewpoint (e.g., the user's position in 3D space), a gesture can be performed by the user's hand when the user's hand is resting to the side of the user or otherwise close to the user.

[0009] In some implementations, a processor performs a method by executing instructions stored on a computer-readable medium. The method involves obtaining hand data associated with a position of a hand in 3D space. The hand data can be obtained based on first sensor data (e.g., using an outward-facing image sensor on an HMD), and the hand position can be associated with a hand pose. The method involves obtaining gaze data associated with a direction of gaze in 3D space. The gaze data can be obtained based on second sensor data (e.g., based on an outward-facing image and / or depth sensor on the HMD that captures an image of the hand).

[0010] The method involves selecting a direct interaction mode. The direct interaction mode can be selected manually or automatically. In some implementations, the direct interaction mode is automatically selected based on a criterion (e.g., based on a user hand position and a virtual object position in 3D space satisfying a spatial relationship with each other). Upon selecting the direct interaction mode, the method involves identifying a first interaction with a first virtual object based on identifying that the hand position intersects a first region associated with the position of the first virtual object in an XR environment corresponding to the 3D space.

[0011] The method also involves selecting an indirect interaction mode before or after the direct interaction mode. This mode can be automatically or manually selected. In some implementations, the indirect interaction mode is selected based on automatically determining that criteria for the direct interaction mode are not met. Upon selecting the indirect interaction mode, the method involves identifying a second interaction with the second virtual object based on identifying that the gaze direction intersects a second region associated with the position of the second virtual object in the XR environment.

[0012] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the performance of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium has instructions stored therein that, when executed by one or more processors of the device, cause the device to perform or cause the performance of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the performance of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] So that the present disclosure may be understood by those skilled in the art, a more particular description may be had with reference to aspects of certain illustrative embodiments, some of which are illustrated in the accompanying drawings.

[0014] Figure 1A to Figure 1B Example electronic devices are shown operating in a physical environment according to some implementations.

[0015] Figure 2 Shown is a diagram of a device provided according to some specific implementations. Figure 1A to Figure 1B A view of virtual elements within a 3D physical environment, where the user performs direct interaction.

[0016] Figure 3 Shown is a diagram of a device provided according to some specific implementations. Figure 1A to Figure 1B A view of virtual elements within a 3D physical environment, where the user performs indirect interactions.

[0017] Figure 4 An exemplary determination of a direct interaction mode based on a 3D area surrounding a virtual object is shown according to some implementations.

[0018] Figure 5 An exemplary determination of a direct interaction mode based on a 3D area around a virtual object and a region of interest is shown according to some implementations.

[0019] Figure 6Another exemplary determination of a direct interaction mode based on a 3D area around a virtual object and a region of interest is shown according to some implementations.

[0020] Figure 7 An exemplary determination of a direct interaction mode based on a time threshold according to some implementations is shown.

[0021] Figures 8A to 8B An exemplary determination of an indirect interaction mode according to some implementations is shown.

[0022] Figure 9 Indirect interaction identification according to some implementations is shown.

[0023] Figure 10 A transition from a direct interaction mode to an indirect interaction mode is shown according to some implementations.

[0024] Figure 11 is a process flow diagram illustrating an example process for providing direct and indirect input using hand and eye data according to some implementations.

[0025] Figure 12 is a flow chart illustrating a method for interpreting user activity as user interaction with a virtual element positioned within a 3D space, according to some implementations.

[0026] 13A to 13B Shown are gestures performed relative to user interface elements and corresponding 3D areas on a 2D user interface, according to some implementations.

[0027] Figure 14 Shown are gestures performed relative to user interface elements on a 2D user interface, according to some implementations.

[0028] Figure 15 Shown are gestures performed relative to user interface elements on a 2D user interface, corresponding 3D areas for expected tap gestures, and corresponding dead zone areas for expected tap gestures, according to some implementations.

[0029] Figure 16 Shown are gestures performed relative to user interface elements on a 2D user interface, corresponding non-cylindrical 3D areas for anticipating a tap gesture, and corresponding dead zone areas for anticipating a tap gesture, according to some implementations.

[0030] Figure 17 Shown are gestures performed relative to user interface elements on a 2D user interface, corresponding non-cylindrical 3D areas for anticipating a tap gesture, and corresponding dead zone areas for anticipating a tap gesture, according to some implementations.

[0031] 18A to 18B Example non-cylindrical 3D regions are shown according to some implementations.

[0032] Figure 19 Exemplary 3D regions oriented based on viewpoint direction according to some implementations are shown.

[0033] FIG. 20A to FIG. 20B An exemplary 3D region that is resized based on gesture velocity according to some implementations is shown.

[0034] Figure 21 is a flow chart illustrating a method for interpreting user position data as input based on a gesture path relative to a defined 3D region, according to some implementations.

[0035] Figure 22 A view of an XR environment provided by the device of FIG. 1 based on the physical environment of FIG. 1 is shown, including interpretation of intentional dragging movements in the XR environment, according to some implementations.

[0036] Figure 23 Interpreting the user's intent when moving relative to the actual user interface location is shown.

[0037] Figure 24 Interpreting the user's intent when moving relative to the actual user interface location is shown.

[0038] Figures 25 to 26 Movement is shown having characteristics corresponding to retraction according to some implementations.

[0039] Figure 27 A retraction deadband is shown according to some implementations.

[0040] Figures 28 to 29 Dynamic break volume according to some implementations is shown.

[0041] Figures 30 to 31 Trajectory correction according to some implementations is shown.

[0042] Figure 32 is a flow chart illustrating a method for determining which segments of a movement to associate with user interface content based on characteristics of the movement, according to some implementations.

[0043] Figure 33 is a flow chart illustrating a method for accounting for motion using dynamic blackout volumes, according to some implementations.

[0044] Figure 34A user is shown making a pinch gesture while changing gaze direction relative to content presented in a view of the XR environment provided by device 110 of FIG. 1 , according to some implementations.

[0045] Figure 35A 、 Figure 35B and Figure 35C Hand engagement, indirect selection, and hand- and gaze-based indirect gestures are shown according to some implementations.

[0046] Figure 36A 、 Figure 36B 、 Figure 36C and Figure 36D Various anomalies associated with user gaze direction relative to user interface elements are shown, according to some implementations.

[0047] Figure 37 An exemplary interaction tracking process according to some implementations is shown.

[0048] Figure 38 Associating pinches with gaze events is shown on a graph showing gaze velocity over time, according to some implementations.

[0049] Figure 39 Associating pinches with gaze events is shown on a graph showing gaze velocity over time, according to some implementations.

[0050] Figure 40 Associating a pinch with a gaze hold event rather than a saccade event is shown according to some implementations.

[0051] Figure 41 is a flow chart illustrating a method for triggering a user interface response based on a gaze hold event according to some implementations.

[0052] Figure 42 Use of an exemplary input support framework to generate interaction data based on hand and gaze data, as well as UI target data, according to some implementations is shown.

[0053] Figures 43A to 43B Example hit detection corrections according to some implementations are shown.

[0054] Figures 44A to 44B 2D elements and 3D elements are shown that may be positioned within a 3D environment according to some implementations.

[0055] Figure 45 Exemplary control areas and fuzzy hit test areas around elements of a user interface are shown according to some implementations.

[0056] Figures 46A to 46CExemplary control areas and fuzzy hit test areas around elements of a user interface are shown according to some implementations.

[0057] Figure 47 Determining whether to associate a user activity with a user interface element according to some implementations is shown.

[0058] Figures 48A to 48B The use of sampling in determining to associate user activity with an element is shown according to some implementations.

[0059] Figures 49A to 49D Depicted is the hysteresis in determining how to associate user activity with elements according to some implementations.

[0060] Figures 50A to 50B is a flow chart illustrating a method for supporting application input recognition according to some implementations.

[0061] Figure 51 is a flow chart illustrating another method for supporting application input recognition according to some implementations.

[0062] Figure 52 is a flow chart illustrating a method for supporting application input recognition using sampling, according to some implementations.

[0063] Figure 53 is a flow chart illustrating a method for supporting application input recognition, and thereby application user activity recognition, according to some implementations.

[0064] Figure 54 A flowchart of a technique for detecting input gestures according to some embodiments is shown.

[0065] Figure 55 A flowchart of techniques for implementing gesture input according to some embodiments is shown.

[0066] Figure 56 A flowchart of a technique for classifying the intentionality of a gesture according to some embodiments is shown.

[0067] Figure 57 A flow diagram of a technique for classifying touch phases is shown according to some embodiments.

[0068] Figure 58 A flow diagram of an action network is shown according to some embodiments.

[0069] Figure 59 A flowchart for processing occluded hands for gesture recognition is shown according to some embodiments.

[0070] Figures 60A to 60B An exemplary diagram of a technique for handling occluded gestures is shown in accordance with one or more embodiments.

[0071] Figure 61 A flowchart of a technique for processing occluded hands for gesture recognition is shown, according to some embodiments.

[0072] Figure 62 is a flow chart illustrating a method for user input recognition according to some implementations.

[0073] Figure 63 is a block diagram of an electronic device according to some specific implementations.

[0074] As is common practice, the various features shown in the accompanying drawings may not be drawn to scale. Therefore, the dimensions of various features may be arbitrarily expanded or reduced for clarity. Furthermore, some of the accompanying drawings may not depict all components of a given system, method, or apparatus. Finally, throughout the specification and accompanying drawings, similar reference numerals may be used to denote similar features. DETAILED DESCRIPTION

[0075] Numerous details are described to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings illustrate only some example aspects of the present disclosure and, therefore, should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the example implementations described herein.

[0076] Figure 1A to Figure 1B Exemplary electronic devices 105 and 110 are shown operating in physical environment 100. Figure 1A to Figure 1B In the example of FIG, the physical environment 100 is a room including a table 120. The electronic devices 105 and 110 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 100 and objects therein (e.g., images, sounds, lighting, characteristics, etc.) and information about the user 102 of the electronic devices 105 and 110 and to evaluate the physical environment and objects therein. The information about the physical environment 100 and / or the user 102 can be used to provide visual and audio content and / or to identify the current location of the physical environment 100 (e.g., including the location of objects such as the table 120 in the physical environment 100) and / or the location of the user within the physical environment 100.

[0077] In some implementations, a view of an extended reality (XR) environment can be provided to one or more participants (e.g., user 102 and / or other participants not shown) via electronic devices 105 (e.g., a wearable device such as an HMD, etc.) and / or 110 (e.g., a handheld device such as a mobile device, a tablet computing device, a laptop computer, etc.). Such an XR environment can include a view of a 3D environment generated based on camera images and / or depth camera images of the physical environment 100 and a representation of the user 102 based on the camera images and / or depth camera images of the user 102. Such an XR environment can include virtual content positioned at a 3D position relative to a 3D coordinate system (i.e., 3D space) associated with the XR environment, which can correspond to the 3D coordinate system of the physical environment 100.

[0078] In some implementations, video (e.g., pass-through video depicting a physical environment) is received from an image sensor of a device (e.g., device 105 or device 110). In some implementations, a 3D representation of the virtual environment is aligned with a 3D coordinate system of the physical environment. The size of the 3D representation of the virtual environment can be generated based on, for example, the scale of the physical environment or the positioning of open space, floors, walls, etc., such that the 3D representation is configured to align with corresponding features of the physical environment. In some implementations, a viewpoint (e.g., of user 102) within the 3D coordinate system can be determined based on the position of the electronic device within the physical environment. The viewpoint can be determined based on, for example, image data, depth sensor data, motion sensor data, etc., which can be retrieved via a virtual inertial odometry system (VIO), a simultaneous localization and mapping (SLAM) system, etc.

[0079] Figure 2 Shows the Figure 1A to Figure 1B 102 is a view of virtual elements within a 3D physical environment of an XR device, wherein the user performs direct interaction. In this example, the user 102 makes gestures relative to content presented in views 210a-210b of the XR environment provided by a device (e.g., device 105 or device 110 above). The views 210a-210b of the XR environment include an exemplary user interface 230 of an application (e.g., an example of virtual content) and a drawing 220 of the table 120 (i.e., an example of real content). As an example, in Figure 2 In the embodiment of the present invention, the user interface 230 is a two-dimensional virtual object (e.g., having a flat, front-facing surface). Providing such a view may involve determining the 3D properties of the physical environment 100 described above (e.g., the position of the table 120 in the physical environment 100, the size of the table 120, the size of the physical environment 100, etc.) and positioning the virtual content (e.g., the user interface 230) in a 3D coordinate system corresponding to the physical environment 100.

[0080] exist Figure 2 In the example of FIG, user interface 230 includes various content items, including background portion 235 and icons 242, 244, 246, 248. Icons 242, 244, 246, 248 can be displayed on flat user interface 230. User interface 230 can be the user interface of an application, as shown in this example. For illustrative purposes, user interface 230 is simplified, and the user interface can actually include any level of complexity, any number of content items, and / or a combination of 2D and / or 3D content. User interface 230 can be provided by various types of operating systems and / or applications, including but not limited to messaging applications, web browser applications, content viewing applications, content creation and editing applications, or any other application that can display, present, or otherwise use visual and / or audio content.

[0081] In this example, background portion 235 of user interface 230 is flat. In this example, background portion 235 includes all aspects (e.g., visual characteristics) of user interface 230 being displayed except for icons 242, 244, 246, 248. Displaying the background portion of an operating system or application's user interface as a flat surface can provide various advantages. For example, doing so can provide an easier-to-understand or otherwise easier-to-use portion of the XR environment to access an application's user interface. In some implementations, one or more flat background portions are used to sequentially and / or simultaneously present multiple user interfaces (e.g., corresponding to multiple different applications) within the XR environment, although other configurations are possible.

[0082] In some implementations, the position and / or orientation of one or more user interfaces can be determined to facilitate visibility and / or usage. Within the 3D environment, one or more user interfaces can be in a fixed position and orientation. In such cases, user movement will not affect the position or orientation of the user interface within the 3D environment (e.g., such that the user interfaces remain at their respective positions or orientations and do not move relative to the user's viewpoint).

[0083] The position of the user interface within the 3D environment may be based on determining a distance of the user interface from the user (e.g., from an initial or current user position). The position and / or distance from the user may be determined based on various criteria, including, but not limited to, criteria that consider application type, application functionality, content type, content / text size, environment type, environment size, environment complexity, ambient lighting, the presence of other people in the environment, use of the application or content by multiple users, user preferences, user input, and other factors.

[0084] In some implementations, one or more user interfaces can be body-locked content, e.g., at a distance and orientation offset relative to a portion of the user's body (e.g., their torso). For example, body-locked content of a user interface can be 0.5 meters to the left of and 45 degrees to the forward-facing vector of the user's torso. If the user's head turns while their torso remains stationary, the body-locked user interface will appear to remain stationary in the 3D environment at 0.5 meters to the left of and 45 degrees to the forward-facing vector of the torso. However, if the user does rotate their torso (e.g., by rotating in their chair), the body-locked user interface will follow the torso rotation and reposition within the 3D environment so that it remains 0.5 meters to the left of and 45 degrees to the new forward-facing vector of their torso.

[0085] In other implementations, user interface content is defined at a specific distance from the user, where it remains fixed relative to the user's orientation (e.g., if initially displayed in a cardinal orientation, it will remain in that cardinal orientation regardless of any head or body movement). In this example, the orientation of the body-locked content is not referenced to any part of the user's body. In this different implementation, the body-locked user interface will not reposition itself based on torso rotation. For example, a body-locked user interface may be defined as 2 meters away and, based on the direction the user is currently facing, may initially be displayed to the north of the user. If the user rotates their torso 180 degrees to face south, the body-locked user interface will remain 2 meters away from the user's north, with north now directly behind the user.

[0086] The body-locked user interface can also be configured to always remain aligned with the gravity line or horizon, so that changes in head and / or body orientation in roll will not cause the body-locked user interface to move within the 3D environment. On the other hand, translational movement will cause the body-locked content to reposition within the 3D environment to maintain the distance offset.

[0087] exist Figure 2 In the example of FIG2 , user 102 moves their hand from an initial position, as shown by the position of the drawing 222 of the hand in view 210a. The hand moves along a path 250 to a later position, as shown by the position of the drawing 222 in view 210b. As user 102 moves their hand along this path 250, the finger intersects the user interface 230. Specifically, as the finger moves along the path 250, it virtually pierces the icon 246 in the user interface 230 and thus, from the user's viewpoint, the fingertip portion of the finger (not shown) is optionally obscured by the user interface 230 in view 210b.

[0088] The embodiments disclosed herein explain how users move Figure 1A to Figure 1BThe user 102 moves their hand / finger along a path 250 relative to a user interface element, such as an icon 246 in the user interface 230, to identify user input / interaction. The interpretation of user movement and other user activities can be based on identifying user intent using multiple potentially separate recognition processes corresponding to different input modalities. Using multiple potentially separate recognition processes for different modalities can improve functionality, accuracy, efficiency, and / or provide other device-based or interaction-based benefits.

[0089] exist Figure 2 In an example of , recognizing input may involve determining that the gesture is a direct interaction and then using a direct input recognition process to recognize the gesture. For example, such a gesture may be interpreted as a tap input directed at icon 246. When making such a gesture, the user's actual motion relative to icon 246 may deviate from the ideal motion (e.g., a straight path passing through the center of the user interface element (e.g., icon 246) in a direction completely orthogonal to the plane of the user interface element). The actual path may be curved, jagged, or otherwise nonlinear, and may be at an angle rather than orthogonal to the plane of the user interface element. The path may have properties that make the gesture similar to other types of input gestures (e.g., swipe, drag, flick, etc.). For example, non-orthogonal motion may make the gesture similar to a swipe motion, in which the user provides input by piercing the user interface element and then moving in a direction along the plane of the user interface (e.g., horizontally or vertically).

[0090] Some embodiments disclosed herein determine that a direct interaction mode is applicable and, based on the direct interaction mode, utilize a direct interaction recognition process to distinguish or otherwise interpret user activity corresponding to direct input, such as identifying intended user interactions based on whether and how a gesture path intersects one or more 3D regions of space. Such recognition processes can take into account actual human tendencies associated with direct interaction (e.g., a natural arching of the hand during an intended straight action, a tendency to move based on the shoulder or other pivot position (e.g., an elbow), etc.), human perception issues (e.g., a user cannot see or know the exact position of virtual content relative to their hand), and / or other direct interaction-specific issues.

[0091] Note that the user's movements in the real world (e.g., physical environment 100) correspond to movements within a 3D space (e.g., an XR environment based on the real world and including virtual content such as a user interface positioned relative to real-world objects including the user). Thus, the user moves their hand in the physical environment 100, for example, through empty space, but the hand (i.e., a drawing or representation of the hand) intersects and / or pierces the user interface 230 of the XR environment based on the physical environment. In this way, the user is directly interacting virtually with the virtual content.

[0092] Figure 3 Shows the Figure 1A to Figure 1B 3D physical environment, wherein a user performs indirect interaction. In this example, user 102 performs a gesture while viewing content presented in view 305 of an XR environment provided by a device (e.g., device 105 or device 110) (e.g., while user 102's gaze is directed toward the content). View 305 of the XR environment includes Figure 2 An exemplary user interface 230 of . Figure 3 In the example of FIG, user 102 performs a pinch gesture with their hands as shown in drawing 222 while gazing at user interface icon 246 along gaze direction path 310. In this example, this user activity (e.g., a pinch gesture along with a gaze directed at a UI element) corresponds to a user intent to interact with user interface icon 246.

[0093] Implementations disclosed herein interpret user activity (such as user 102 forming a pinch gesture in conjunction with a gaze directed at a UI element) to identify a user / interaction. For example, such user activity may be interpreted as a tap input directed at icon 246, e.g., selecting icon 246 in user interface 230, which causes the device to activate icon 246 and optionally display content (e.g., a user interface) corresponding to icon 246. However, when performing such actions, the timing between the user's gaze direction and / or gesture and the detection of the gaze at the target with which the user intended to associate the gesture may not be fully executed and / or timed.

[0094] Some embodiments disclosed herein determine that an indirect interaction mode is applicable. Based on the indirect interaction mode, an indirect interaction recognition process is used to identify expected user interactions based on user activities. For example, the process may be based on a gesture path (e.g., similar to Figure 3 The intended user interaction is identified by determining whether and how the path 310 in the image (e.g., path 310 in the image) intersects one or more 3D regions in the space. Such an identification process may take into account actual human tendencies associated with indirect interactions (e.g., eye saccades, eye fixations, and other natural human gaze behaviors, hand arching movements, retractions that do not correspond to the intended insertion direction, etc.), human perception issues (e.g., users cannot see or know the exact position of virtual content relative to their hands), and / or other indirect interaction-specific issues.

[0095] Some implementations determine an interaction mode, e.g., a direct interaction mode or an indirect interaction mode, such that user behavior can be interpreted by a dedicated (or otherwise separate) recognition process for the appropriate interaction type, e.g., using a direct interaction recognition process for direct interactions and an indirect interaction recognition process for indirect interactions. Such dedicated (or otherwise separate) process utilization can be more efficient, more accurate, or provide other benefits relative to using a single recognition process configured to recognize multiple types of interactions (e.g., both direct and indirect).

[0096] Figure 4 An exemplary determination of a direct interaction mode based on a 3D area 430 surrounding a user interface 400 is shown. In this example, a user 102 is using a device 105 (e.g., implemented as an HMD) to view and interact with an XR environment that includes the user interface 400. The 3D area 430 surrounding the user interface 400 is determined by the device 105. Note that in this example, the dotted lines indicating the boundaries of the 3D area 430 are for illustrative purposes and are not visible to the user 102. The 3D area 430 is used to determine whether a direct interaction mode or an indirect interaction mode is applicable at different points in time based on user activity. In this example, the interaction mode is determined based on whether the hand 422 of the user 102 intersects the 3D area 430. In this case, Figure 5 In the example shown in FIG4 , hand 422 does intersect 3D area 430 (e.g., breaks through the boundary of 3D area 430), and thus the interaction mode is determined to be a direct interaction mode. Thus, the user's activity (e.g., hand movement, etc.) is evaluated using, for example, a direct interaction recognition process (such as providing a selection of pointing to user interface 400).

[0097] Figure 5 An exemplary determination of a direct interaction mode based on a 3D area 430 and a focus area 510 around a user interface 400 is shown. In this example, a user 102 is using a device 105 (e.g., implemented as an HMD) to view and interact with an XR environment including the user interface 400. The 3D area 430 around the user interface 400 is determined by the device 105, as described above with reference to FIG. Figure 4Similarly discussed. Note that in this example, the dotted lines indicating the boundaries of the 3D area 430 are for illustrative purposes and are not visible to the user 102. The area of ​​interest 510 is also a 3D area (e.g., a conical area centered on a centerline corresponding to the user's head orientation or the forward-facing orientation of the device 105). The user's head orientation may be determined based on an estimated posture (e.g., the position and / or orientation of the head using sensors on the device 105). For example, the head orientation may be determined based on determining the position of a plane corresponding to the user's cheek and extending a vector from a position on the user's face in a direction orthogonal to the plane. In another example, the head orientation may be determined in part based on the orientation of the HMD worn on the head, for example, based on a device posture tracking technique that determines a vector directly referenced to the device rather than based on the user's face. The user device orientation may be determined via sensor data (e.g., image data, depth data, motion data, VIO process, SLAM process, etc.). The region of interest 510 can be determined based on a field of view provided in the view provided by the device 105 (e.g., corresponding to a central portion of the field of view (e.g., the central 30%, 50%, 70%, etc.)), such that the region of interest 510 has predetermined three-dimensional characteristics (e.g., a fixed volume relative to the centerline discussed above). In other specific implementations, the region of interest 510 corresponds to the field of view provided in the view provided by the device 105.

[0098] 3D area 430 and region of interest 510 are used to determine at different points in time whether direct interaction mode or indirect interaction mode is applicable to the detected user activity. In this example, the interaction mode is determined based on determining that hand 422 of user 102 intersects 3D area 430 at intersection 515 and whether intersection 515 is within the region of interest. In this case, intersection 515 is also within region of interest 510, and therefore, the interaction mode is determined to be direct interaction mode. Thus, for example, the user's activities (e.g., hand movements, etc.) are evaluated using a direct interaction recognition process, as previously discussed herein. The interaction mode may or may not also apply with respect to potential interactions with other (e.g., nearby) user interface elements (such as second user interface 520). For example, in Figure 5 , because the hand 422 intersects with the 3D area 430 surrounding the user interface 400 (e.g., at the intersection 515), but not with the 3D area surrounding the second user interface 520, when the device 105 performs a selection operation pointing to the user interface 400, the device 105 does not perform a selection operation pointing to the second user interface 520 based on the user's activity (e.g., the movement of the hand 422).

[0099] Figure 6Another exemplary determination of a direct interaction mode based on a 3D area 620 and a focus area 610 around a virtual object (e.g., second user interface 520) is shown. In this example, user 102 is using device 105 to view a video that includes Figure 5 4 and the XR environment of the second user interface 520. A 3D area 620 around a virtual object (e.g., the second user interface 520) is determined by the device 105. Note that in this example, the dashed lines indicating the boundaries of the 3D area 620 are for illustrative purposes and are not visible to the user 102. In some implementations, the 3D area 620 has one or more characteristics of the 3D area 520 discussed above. The focus area 610 is also a 3D area (e.g., a conical area centered on a centerline corresponding to the user's head orientation or the forward-facing orientation of the device 105). In some implementations, the focus area 610 has one or more characteristics of the focus area 510 discussed above. The 3D area 620 and the focus area 610 are used to determine at different points in time whether the direct interaction mode or the indirect interaction mode is applicable to the detected user activity. In this example, the interaction mode is dedicated to interaction with the second user interface 520 (and is therefore determined to be not applicable to the user interface 400).

[0100] Based on determining that the hand 422 of the user 102 intersects the 3D area 620 (e.g., similar to Figure 5 The interaction mode for the second user interface 520 is determined by intersecting at the intersection point 515 in the image and whether the intersection point is within the focus area. Figure 6 , the intersection point is within the focus area 610, and therefore, the interaction mode is determined to be a direct interaction mode. Thus, for example, the user's activities (eg, hand movements, etc.) are evaluated using a direct interaction recognition process, as similarly described above.

[0101] It should be noted that the intersection point can correspond to a single location (e.g., an x, y, z location), to a 2D location, or to a 3D area at which the intersection occurs. Whether the intersection point is considered to be within the region of interest can be based on whether there is any intersection, such as whether any point on the hand 422 intersects with any point in the region of interest 610. In another example, whether the intersection point is considered to be within the region of interest is based on other criteria, such as whether at least a portion of a given size of the hand 422 is within the region of interest 610. In another example, a specific point on the hand (e.g., a point on or near the end of a specific finger) is used to determine whether the intersection point is considered to be within the region of interest, such as when the specific point on the hand 422 is within the region of interest 610.

[0102] Figure 7 An exemplary determination of a direct interaction mode based on a time threshold is shown. Figure 6The direct interaction mode is established based on the determination made in . This direct interaction mode with the second user interface 520 is determined to remain active for a period of time (e.g., 0.1, 1, 5, 10, 15, 30, etc. seconds) after the determination is made. Thus, even if the user looks up (changes the focus area 710) to cover the user interface 400 instead of the second user interface 520, as Figure 7 As shown, the direct interaction mode determined for the second user interface 520 remains active. In other words, the direct interaction mode remains applicable to hand 422 interactions with the second user interface 520 (e.g., selection interactions, movement interactions, etc.) for a threshold amount of time. Once the threshold amount of time has expired, the interaction mode can be re-determined, for example, as direct or indirect and / or with respect to the same or different user interface elements. In some implementations, additional or alternative heuristics (e.g., in addition to time-based thresholds) are used to determine when to re-determine the interaction mode, for example, based on a change in content, a change in the environment, or movement of the user with certain characteristics.

[0103] Figure 8A An exemplary determination of an indirect interaction mode based on a 3D area surrounding a virtual object is shown. In this example, user 102 is using device 105 to view a virtual object comprising Figures 4 to 7 400 is used to determine whether a direct interaction mode or an indirect interaction mode is applicable to a detected user activity. In this example, as shown in FIG. Figure 8A As shown, the interaction mode is determined based on whether the hand 422 of the user 102 intersects the 3D area 430. In this case, the hand 422 does not intersect the 3D area 430, and therefore, the interaction mode is determined to be an indirect interaction mode. Thus, the user's activities (e.g., hand movements along with the gaze direction 805, etc.) are evaluated using, for example, an indirect interaction recognition process (such as providing a selection to point at the user interface 400), as similarly described herein.

[0104] In some implementations, a third interaction mode (e.g., a mode that does not allow interaction) can be determined based on one or more criteria being met. For example, such a mode can be triggered by a user providing input to specifically enter a mode in which the user's activities will not interfere with the content (e.g., by providing relative interaction with the content). In another example, such a mode can be triggered manually or automatically for a low-power mode.

[0105] Figure 8BAn exemplary determination of an indirect interaction mode based on a 3D area 620 and a focus area 850 around a virtual object (i.e., a second user interface 520) is shown. In this example, the user 102 is using the device 105 to view and interact with an XR environment including the user interface 400 and the second user interface 520. The 3D area 620 around the second user interface 520 is determined by the device 105, as similarly described above. Note that in this example, the dotted lines indicating the boundaries of the 3D area 620 are for illustrative purposes and are not visible to the user 102. The focus area 850 is also a 3D area (e.g., a conical area centered on a centerline 855 corresponding to the direction of the user's head or the forward-facing direction of the device 105), as similarly described above.

[0106] 3D area 620 and focus area 850 are used to determine whether direct interaction mode or indirect interaction mode is applicable at different points in time. In this example, the interaction mode is determined based on determining that the hand 422 of user 102 intersects the 3D area 620 at an intersection and whether the intersection is also within the focus area 850. In this case, the hand 422 of user 102 does intersect with the 3D area 620, but the intersection is not within the focus area 850, and therefore the interaction mode is determined to be indirect interaction mode. Therefore, for example, the user's activities (e.g., hand movements and gaze direction 805, etc.) are evaluated using an indirect interaction recognition process, as similarly described above. With respect to potential interactions with nearby user interface elements (such as user interface 400), the interaction mode may or may not also be applicable, as similarly described above.

[0107] Figure 9 Indirect interaction recognition is shown. In this example, sensor data on device 105 is used to identify a user interaction by user 102, for example, based on outward-facing image sensor data, depth sensor data, eye sensor data, motion sensor data, etc. Such data can be monitored for engagement conditions corresponding to the start of a user interaction. In some implementations, engagement is triggered based on detecting a hand motion (e.g., raising a hand or raising a hand while performing a specific gesture).

[0108] In this example, at block 910, the process includes detecting that the user 102 has positioned their hand 422 within the field of view of the outward-facing image sensor (e.g., at Figure 9 The process may include detecting a particular hand configuration (e.g., claw, flat hand, stable hand in any configuration, etc.) as an indication of hand engagement, or may simply include detecting the presence of hand 422 within the sensor's field of view.

[0109] At block 920, the process includes identifying an object using the user gaze data. In this example, the process includes identifying the gaze direction 905 of the user 102 at a user interface element 915 of the user interface 400, such as Figure 9 shown.

[0110] At block 930, the process includes displaying feedback based on the object identified by the gaze. In this example, the feedback graphically distinguishes user interface element 915 (e.g., from other user interface elements displayed in user interface 400) to indicate that user interface element 915 now has a different state (e.g., a "hover" state similar to the state of a traditional UI icon when a cursor is over an item without clicking / tapping). In some implementations, displaying user interface element 915 with feedback includes highlighting or bolding user interface element 915, displaying user interface element 915 with a focus ring or glow, increasing the size of user interface element 915, and / or changing the color of user interface element 915.

[0111] At block 940, the process includes identifying a gesture to be associated with the identified object. In this example, the user is gazing at user interface element 915 in gaze direction 905 when device 105 detects a pinch gesture by hand 422 (or at least within a threshold time relative thereto). In this example, the pinch gesture is interpreted as initiating an action on user interface element 915, e.g., resulting in a cursor "click" event similar to a traditional UI icon (during which the cursor is positioned over the icon and a trigger such as a mouse click or trackpad tap is received) or a selection action similar to a touchscreen "tap" event.

[0112] Figure 9 An example of identifying indirect user interactions is shown. Numerous other types of indirect interactions can be identified, for example, based on one or more user actions that identify a user interface element and / or provide input (e.g., no-action / hover type input, selection type input, input with direction, path, speed, acceleration, etc.). Inputs in 3D space similar to inputs on a 2D interface can be identified, such as inputs similar to mouse movements, mouse button clicks, touch screen touch events, trackpad events, joystick events, game controller events, etc.

[0113] Figure 10100b 。 The transition from direct interaction mode to indirect interaction mode is shown. In this example, at a first point in time, direct interaction mode is determined based on the user's hand 422 being within the 3D area 1030 surrounding the user interface 1000a, as similarly discussed herein. In this example, the interaction mode is determined based on whether the user's hand 422 intersects with the 3D area 1030. In this case, the hand 422 does intersect with the 3D area 1030, and therefore the interaction mode is determined to be direct interaction mode. Therefore, for example, a direct interaction recognition process is used to evaluate the user's activities (e.g., hand movements, etc.), as previously discussed above. The direct interaction recognition process may include identifying a direct touch interaction with the UI element 1014. The direct touch may initiate an action corresponding to the UI element 1014, such as triggering the display of the user interface 1000b.

[0114] Then, at a second point in time, the direct interaction mode is exited based on the user hand 422 being outside of the 3D area 1030 surrounding the user interface 1000b.

[0115] At a second point in time, an indirect interaction mode may be determined based on the user's hand 422 being outside of the 3D area 1030 surrounding the user interface 1000b. In this example, the interaction mode is determined based on whether the user's hand 422 intersects the 3D area 1030, as previously discussed herein. In this case, the hand 422 does not intersect the 3D area 1030, and therefore, the interaction mode is determined to be an indirect interaction mode. Thus, for example, an indirect interaction recognition process is used to evaluate the user's activity (e.g., hand movement, gaze direction, etc.). The direct interaction recognition process may include identifying a gaze-pinch interaction with the UI element 1015, for example, identifying a pinch of the hand 422 occurring when the gaze direction 1005 is directed at the UI element 1015. This interaction may initiate an action corresponding to the UI element 1015, for example, triggering a change in the user interface 1000b, such as a change in the appearance of the user interface 1000b or one or more elements displayed in the user interface 1000b.

[0116] During the user experience, the user's interaction mode can transition from direct to indirect, and vice versa, thereby providing the user with significant flexibility in interacting with UI content in a manner that best suits the user and / or the situation. This transition can be accompanied by feedback indicating to the user that a switch in interaction mode has occurred and / or identifying the new / active mode.

[0117] Figure 11is a process flow diagram illustrating an exemplary process for providing direct and indirect input using hand and eye data. In this example, hand data (e.g., "hand data") is input to a hand processing block 1110. An upsampling and prediction block 1112 upsamples the hand data (e.g., adds more frames) and makes predictions about future hand positions, paths, trajectories, velocities, accelerations, etc. In some implementations, in the hand input process, the internal state of the upsampler is updated with new observation data from a computer vision-based hand tracking system. In the interaction detection process, data is received about when the next frame will be presented, which calculates an interpolated / predicted timestamp and queries the upsampler for joint data corresponding to that timestamp. These steps can occur asynchronously and / or at different frame rates. The process may typically query the upsampler for several predictions at different timestamps between updates from the computer vision tracking system, for example, hand tracking may be running at 30 Hz while the display may be rendered at 90 Hz.

[0118] The hand and point of view correction (POVc) blending block 1114 can use corrected and uncorrected hand joint data to mitigate POVc-coupled hand jumps that can occur, for example, due to background depth and camera cuts. The hand and POVc blending block can calculate new hand pose information to account for the discrepancy between the position of the user's hands in the real world (e.g., as tracked by a computer vision system) and where the hands appear to be located from the perspective of the user viewing the display. The algorithm can smoothly blend the real and POV-corrected versions of the hand skeleton to create hand pose information optimized for input recognition in use scenarios where hand registration to the UI is important (e.g., during direct input, where fingers must align with buttons). When the user's hand is outside the user's field of view (FOV) and / or the sensor's field of view, it may be undesirable to expend resources or otherwise provide correction, resulting in jumps in hand position. This process can include, for example, smoothly blending hand data over time based on whether the user is in direct or indirect mode, whether the hand is near a UI element, or other factors.

[0119] Eye data (for example, " eye data ") is input to gaze processing block 1120. Glance detection block 1122 removes glance, blink and / or other gaze loss events (for example, only leaves the gaze fragment corresponding to watching attentively). Exemplary glance detection process can use gaze confidence, tracking state, pupil center, pupil diameter, interpupillary distance (IPD), gaze ray data and speed data to detect glance and blink for removing and / or to identify watching attentively for gaze interaction. It can be distinguished between the gaze event as watching attentively and the gaze event as pan so that more accurately based on the input of gaze. Gaze filtering block 1124 filters and / or smoothes the gaze data signal. This can relate to smoothing gaze data during watching attentively. Filtering can be closely coupled with gaze motion classification. Gaze processing block 1120 can lock / remember the last smooth gaze position for those intermediate time periods, or can utilize additional algorithm / strategy to " fill " the data between watching attentively.

[0120] The processed hand and eye data is provided to the interaction detection block 1130 as Figure 11 As shown. Note that the interaction detection block 1130 may additionally use UI geometry data including data about the user interface of the application (e.g., the appearance of the user interface, the location of user interface elements within the user interface, the behavior of user interface elements within the user interface (e.g., interactivity), etc.). The UI geometry data may be provided at a different rate than the hand and / or eye data. In one example, UI data is received at a first rate (e.g., 90 fps) while hand and gaze data are received asynchronously at a slower average rate (e.g., 60 fps). The interaction detection block 1130 may additionally use frame target times, such as the system's prediction of when the frame will be rendered. The interaction detection block 1130 may, for example, run asynchronously upon receiving UI updates and / or gaze / hand data. In some implementations, as discussed below, the interaction detection block 1130 includes a gaze blur hit test and focus block 1132, an unexpected rejection block 1134, and a direct and indirect mode detection block 1136.

[0121] The gaze blur hit test and focus area block 1132 identifies virtual elements (e.g., UI elements) at which the user's gaze is considered to be associated with (e.g., pointing toward) a focus area, for example, based on head or device orientation / direction. An exemplary gaze blur hit test process may include using gaze ray data, confidence data, gesture data (e.g., hand movement classification), gaze cluster expansion data, etc. to relax / tighten the gaze area / cone (e.g., focus area) based on the accuracy of gaze tracking and / or user behavior. The process may include utilizing UI geometry from a system process based on UI information provided by one or more applications, such as identifying interaction targets (e.g., which UI elements are to be associated with a given user activity) based on a 3D representation of the UI elements. The use of such UI information may facilitate more accurate hit testing. In addition to UI geometry, blur hit testing may query more detailed data, for example, data about remote effects, gestures, or other contexts.

[0122] The accidental rejection block 1134 identifies instances of hand and gaze behavior that are rejected (i.e., associated with subconscious user behavior that is not an intentional interaction) but would still be associated with a virtual element under appropriate conditions. The accidental rejection block 1134 attempts to understand the intention behind the user's movement to better interpret the movement as, for example, an intentional UI element interaction or something else. An exemplary accidental rejection process can use hand joint data, interaction state history, gaze focus area data, occlusion data, etc. to reject (or cancel) unintentional user interactions. This process may include utilizing UI geometry from a 3D simulation system based on UI information provided by the application, for example, to distinguish intentional user activity from accidental activity based on nearby UI elements. The use of such UI information can facilitate more accurate accidental rejection. For example, based on determining (from data declared by the application) that the application's UI is 2D / planar and therefore movement in the Z direction is unlikely to be an intended input directed toward the UI, i.e., it is likely not an accidental movement intended to trigger a UI interaction, and therefore the input support process can have greater confidence in rejecting the user activity as accidental, the user movement in the Z direction can be identified as accidental.

[0123] Some implementations support two interaction modes (e.g., direct interaction mode and indirect interaction mode). Some implementations support other combinations of interaction modes (e.g., direct interaction mode, indirect interaction mode, gaze-only interaction mode, and / or peripheral device interaction). A process can be used to determine which, if any, of several interaction models a user is executing. This can involve identifying and ignoring unexpected behavior, and also disambiguating which interaction model is expected when the user wants to interact intentionally. In this example, the direct vs. indirect mode detection block 1136 uses processed hand and eye data to determine an interaction mode for interpreting the user's activity, e.g., selecting a direct interaction mode or an indirect interaction mode, as previously discussed herein.

[0124] If direct interaction mode is selected, hand and eye data (processed and / or unprocessed) is provided to a direct interaction recognition block 1140, which provides direct input to the user interface. In some implementations, the direct interaction recognition block 1140 includes a trajectory correction block 1142 and a hover, reach, and break block 1144. The trajectory correction block 1142 can adjust the trajectory of the user's motion, such as adjusting the user's hand motion to account for the user's behavior in the 3D environment, for example, so that the motion data better corresponds to the user's intended motion / intended interaction. The hover, reach, and break block 1144 can use hand gestures, motion paths, velocities, accelerations, and the like to perform various recognition processes to identify certain types of direct interactions with user interface elements, such as a "hover" interaction based on detecting a hand / finger approaching but not touching a user interface element, a "reach" interaction based on detecting a point (in 3D space / time) at which the hand / finger has made contact with the user interface element, a "break" interaction based on detecting a point (in 3D space / time) at which the hand / finger has ceased contact with the user interface element, and the like.

[0125] Many types of direct interactions can be recognized and provided as user input. In another example, a direct interaction is recognized by recognizing that a user makes contact with a slider user interface element, moves their hand a distance "X" to the left while in contact with the slider, and then retracts their hand (e.g., removes their hand from the slider user interface element) to break contact with the slider. The slider can be moved to the left, for example, a distance X, based on this input. In another example, a user makes a similar motion to provide input that imparts velocity on a user interface element, such as providing a 3D swipe gesture on a user interface to cause the user interface to begin moving and continue moving after the hand breaks contact, where the continued motion is based on the velocity already imparted on the UI object, such as when a user swipes upward on a multi-touch interface of a mobile phone.

[0126] If indirect interaction mode is selected, the hand and eye data (processed and / or unprocessed) is provided to an indirect interaction recognition block 1150, which provides indirect input to the user interface. In some implementations, the indirect interaction recognition block 1150 includes a user-centric motion block 1152 and a pinch and gaze fusion block 1154. The user-centric motion block 1152 accounts for user-centric motion in the hand data (e.g., to account for the fact that the user may be performing a motion that pivots around a rotation point (e.g., shoulder, elbow, wrist, etc.)). The user-centric motion block 1152 can be used to map the user's hand motion to an object or affordance that is separate from the hand, such as an affordance displayed over application content.

[0127] The pinch and gaze fusion block 1154 determines which gaze data will be associated with a user activity indicating an intention to interact with a UI element (such as a pinch gesture). Various types of indirect interactions can be recognized and provided as user input. In one example, the user pinches their fingers together and moves the pinched hand while gazing at a UI element to provide a movement input pointing to the UI element. In another example, the user quickly pinches and releases their fingers while gazing at a UI element to provide a selection event (e.g., similar to a touch screen tap or mouse click input event).

[0128] like Figure 11 The input support process shown in the example of may perform one or more core functions. In some specific implementations, these functions include, but are not limited to, obtaining gaze data and associating the user's gaze with UI elements associated with one or more applications. Doing so may involve performing a fuzzy hit testing process. Doing so may involve creating a focus region. A focus region may be considered a spatial region / area. Larger objects may be partially inside / partially outside the focus region, such that the system needs to test where the user is interacting on the object to understand whether the interaction point is within the user's focus region. The focus region may include some or all parts of a set of zero or more UI elements that a user may be focusing on at a given time. The focus region may be used to disambiguate between interaction types (e.g., direct or indirect input modality), disambiguate which application the user is interacting with, and / or disambiguate which UI elements are the targets of the user's intent to interact with the UI elements.

[0129] In some implementations, the input support process relies heavily on the simulation system to provide a geometric representation of the user interface for hit testing against hand data and gaze data. The input support process can use the collision world (e.g., a 3D geometry abstraction) provided by the simulation system. The input support process can use user interface metadata information (e.g., UI priority, hierarchical display order, transparency information, etc.) to facilitate appropriate UI target selection. The input support process can utilize a separate hit testing process that generates 3D world hit test results. The input support process can utilize a hit testing process that uses an application-provided hierarchical tree (e.g., declaring UI elements, relationships, and / or rendering, such as information about which areas are transparent, rendered on top, etc.).

[0130] In some implementations, the input supports the execution of a process Figure 11 The functionality shown is used to provide an input support process that generates data for an application. The input support process can generate an interaction state for one or more UI elements, such as identifying whether the interaction is direct or indirect, whether the interaction is a hover / close proximity interaction (e.g., associated with user activity that is not linked to UI interaction behavior) or a gesture / pinch / touch interaction (e.g., associated with user activity that indicates intentional UI behavior).

[0131] In some implementations, the input support process includes providing an interaction gesture, such as a point on a UI element that is configured to drive a touchscreen tap-type input.

[0132] In some implementations, the input support process includes providing a manipulator pose that corresponds to, for example, the position and / or orientation of the hand itself. The manipulator pose can provide a 3D position of a stable hand center or pinch centroid (e.g., based on contact of two fingers of the hand). The manipulator pose can provide the position and / or orientation of the manipulator driving the interaction, for example, for direct touch interactions, index fingertip information (e.g., position), and for indirect pinch interactions, pinch information (e.g., pinch centroid).

[0133] In some implementations, input support processing includes providing an activity target (e.g., a UI element).

[0134] In one example, a user initiates an interaction by pinching while gazing at a UI element. The application receives an interaction gesture on the UI element and recognizes the user's intent to interact with the UI element. The user continues pinching and moves their hand to the left. The application receives more interaction gestures, such as a set of positions on the UI based on the 3D position of the hand as it moves to the left, and responds by moving the associated UI element to the left. In this example, by pinching while gazing at a UI object and then moving the pinched hand to the left, the user provides input that the application recognizes as moving the object to the left. The application can respond accordingly without needing to receive (and in fact does not receive) information about the user's gaze direction and / or specific hand characteristics. In this example, the application only receives interaction gesture data and can therefore be an application that only receives interaction gesture data as a proxy for touch input data. Additionally or alternatively, manipulator gesture data, such as the 3D position of the hand as it moves to the left, can be provided to the application, and the application can determine the motion of the associated object based on the changed manipulator gesture position. Applications can be provided with information based on user changes, for example, providing user-centric acceleration deltas similar to incremental updates for a trackpad, which communicates the amount the user has moved from one frame to another, rather than absolute position. For example, communicating changes via a separate API can help us optimize a set of signals to accurately represent motion (e.g., for scrolling) separately from signals that also require absolute position accuracy (e.g., for drawing).

[0135] In some implementations, the input support process includes receiving a gaze direction that may potentially be associated with different UI elements, for example, because the gaze direction is between UI elements or because UI elements overlap each other. The fuzzy hit testing and other such processes discussed above can help disambiguate the user's intent in identifying the appropriate UI element in such cases.

[0136] In some implementations, the input support process includes recognizing two-handed 3D gestures, such as a two-handed pinch-to-zoom gesture, and providing information to an application corresponding to multi-touch touchscreen input gestures. For example, doing so can involve generating an interaction gesture for each hand that can be interpreted by the application's 2D touch-based gesture recognition process as two touch points moving closer or further away from each other, which can be interpreted as a pinch-to-zoom touchscreen gesture (e.g., zooming in and out of a user interface).

[0137] In some implementations, applications are developed for use on mobile platforms that use single-touch and / or multi-touch input gestures. Applications can be developed by application developers by including calls to a gesture API and then specifying what to do for certain types of touch events, e.g., do "X" for a tap event, "Y" for a swipe event, and so on. It may be desirable to use such applications on new or different platforms that offer input modalities other than (or in addition to) single-touch and / or multi-touch input gestures. Rather than implementing all new 3D gestures and gesture recognition processes within applications, some implementations disclosed herein interpret 3D user activity (e.g., hand position, gaze direction, etc.) and send proxy data to the application corresponding to where the touch would have been if the 3D user activity had been performed as a 2D touch input gesture. The input support process can provide such proxy data to such applications, and in doing so, enable a wide range of existing 2D / touch-based applications to be easily ported and used within new 3D systems that use new and different 3D user activity-based input. The input support process can make additional information available so that newer / modified applications can utilize / use more information about the user's 3D activities. Alternatively, this information can be utilized by older, unmodified applications that link with an XR-compatible version of the UI framework, which can utilize this additional data on the application's behalf behind the scenes (e.g., a binary-compatible application using a standard long press recognizer can benefit from an XR-based OS implementation that uses the z-depth of a touch to more accurately recognize a long press than purely 2D native methods).

[0138] In some implementations, to enable the use of both traditional 2D / touch-based applications and newer applications with additional 3D recognition capabilities, the input support process may provide multiple types of information, such as interaction gesture data for touch-based input recognition to support traditional applications and manipulator gesture data for supporting applications with additional 3D recognition capabilities.

[0139] Interpreting 3D user activity as 2D touch-based activity may require overcoming various challenges. For example, a user's hand movement in 3D space when performing a touch gesture may be equivalent to a touchscreen tap gesture, but may lack the precision expected of a touchscreen tap gesture. A touch on a touchscreen involves contact with the physical surface where the finger rests, whereas the motion of "virtually" touching a UI in a 3D XR environment may involve the user's hand piercing a UI element (e.g., as shown in the figure). Figure 2210b of ). The user may intend to tap the surface of the virtual button, but actually penetrate the surface at one location and then retract the hand / finger at a slightly different location. This user activity involving two different UI element locations (e.g., a first entry location and a second exit location for the corresponding UI element) can be interpreted (by the input support process) as a tap at a single point using various algorithms and / or machine learning processes (e.g., performing trajectory correction). The application receives the single location (from the input support process) and recognizes it as a touch event. Thus, the input support process can classify and / or interpret user activity to account for the unique circumstances of interacting within a 3D environment, and encapsulate the data provided to the application so that the application itself does not need to distinguish between the actions intended by the user and the actions actually performed by the user.

[0140] When providing data to the application, the input support process can take into account sensor inaccuracies / limitations, such as taking into account the fact that the identified gaze direction may be different from the actual gaze direction. The application itself does not need to take such inaccuracies into account.

[0141] In formatting an application's data in a format that the application will understand, the input support process can resolve ambiguities, account for inaccuracies, and / or repackage input provided in input modalities that the application itself does not understand / needs to understand into a format that the application does recognize.

[0142] In short, the input support process can perform various processes to interpret raw 3D data for consumption by applications, so that applications (and their developers) do not need to perform those processes through manually programmed processes. Additionally, having such processes performed by a shared OS process can unify the input experience across all applications, so that users receive consistent UI response behavior.

[0143] Furthermore, a shared input support process (e.g., used by multiple applications) can enable multiple interaction modalities, for example, providing different ways for a user to select a button, and abstracting user activity data associated with those different modalities for the same type of interaction to provide that same data to the application for different types of input. Thus, as long as the input support process can abstract new or changed input modality user activity into a format understood by the application, input modalities can be changed and / or added over time without requiring changes to the application.

[0144] In some implementations, an application utilizes a gesture recognizer running within the application process. The gesture recognizer may include a state machine that categorizes input received in an input data stream, e.g., tap started, tap in progress, or tap completed. The input support process may provide data to the application that the gesture recognizer running within the application process recognizes as a specific type of input (e.g., a tap) that corresponds to a single input modality dedicated to the application (e.g., a touch-based input modality). The input support process may configure the data provided to the application to ensure that the application's process accurately recognizes user activity, e.g., by providing the data in a form expected by the gesture recognizer. In one example, the input support process converts user activity involving user gaze and gestures into data that can be recognized as touch input. In another example, the input support process converts user activity involving a 6DOF controller and gaze direction into data that can be recognized as touch input.

[0145] Application developers do not have to write applications for different input modality platforms. Instead, application developers can write applications for the touch platform (or general 2D input modality), and the application can be imported with little or no effort to work within the 3D user activity platform. A single application can be used on mobile devices, laptop devices, tablet devices, desktop devices, and 3D XR devices. The technology disclosed herein enables binary code compatibility, that is, applications can be executed in different environments that utilize different input modalities.

[0146] In some embodiments, the input support process provides gesture flags and / or gesture classifications to the application, for example, to indicate to the application that the user has raised their hand or that a particular gesture has been recognized as a tap, swipe, scroll, etc., as a hint that the application can use when recognizing gestures.

[0147] In some implementations, the input support process facilitates the application's recognition of gaze-only input without providing raw gaze data to the application. This may involve periodically providing gaze data (i.e., the location on the UI element that the user is gazing at). Gaze-only data may be associated with an interaction state, and data may be provided to the application based on gaze-only data only when the gaze data is associated with a particular interaction state, such as when a gaze hover state applies. Such data may be provided only based on explicit user authorization. For example, when a user stares at a UI element for more than a threshold amount of time (e.g., 0.01, 0.1, 0.5, 1, 2, 3, 4, 5, etc. seconds), the application may be notified and an affordance (e.g., a dot) may be displayed. If the user subsequently views the affordance, the application may be notified that the stared-at UI element has been hit / selected. In another example, gazing at a dot within a UI element for more than a threshold amount of time may trigger hover feedback, which may be handled outside the process or by the application. A gaze-only input modality can be another input modality that is abstracted into data provided to an application so that the application can interpret the input, for example, using a general or touch-based gesture recognition process.

[0148] In some implementations, the input support process determines the interaction state of each hand (e.g., direct vs. indirect) and / or the interaction state associated with the user's gaze (e.g., gaze only or not only gaze).

[0149] Figure 12 1 is a flow chart illustrating a method 1200 for interpreting user activity as user interaction with a virtual element positioned within a 3D space. In some implementations, a device such as electronic device 105 or electronic device 110 performs method 1200. In some implementations, method 1200 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. Method 1200 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, method 1200 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0150] At block 1202, method 1200 includes displaying an XR environment corresponding to a 3D space, wherein the XR environment includes a virtual object depicted at a location in the 3D space. The virtual object can be a user interface element, such as a button, icon, text input field, slider, or menu item, that is rendered as part of a virtual UI displayed at a fixed location or otherwise within the XR environment, for example, a virtual 2D menu with buttons displayed a few feet in front of a user in XR.

[0151] At block 1204, method 1200 includes tracking a user hand position in 3D space based on sensor data. This may involve obtaining user position data corresponding to a 3D position of a portion of the user (e.g., a finger / hand) in 3D space. The user position data may be obtained based on sensor data (e.g., image sensor, depth sensor data, motion sensor data, etc.) and may provide a path of the portion of the user over a period of time. In one example, the user position data may be a skeletal representation of the user generated periodically (e.g., at 30 fps) based on an outward-facing image / depth sensor on the HMD. Such user position data may provide a path of a portion of the skeleton (e.g., a finger skeleton portion). In other words, for example, by tracking the orientation of the skeleton at multiple moments in time, the path of the finger portion of the skeleton relative to the 3D space over time may be determined.

[0152] In some implementations, the portion of the user corresponds to a point on or within a finger of the user. The portion of the user can correspond to a point on or within a hand of the user. The user position data can correspond to a position within a skeletal representation of the user that is generated periodically (e.g., at multiple points in time over a period of time).

[0153] At block 1206, method 1200 includes determining an interaction mode based on the user's hand position and the position of the virtual object in the 3D space. Determining the interaction mode may include selecting whether the interaction mode is direct or indirect based on the user's hand position and the position of the virtual object in the 3D space. Determining the interaction mode may involve selecting a direct interaction mode based on the hand position intersecting a 3D region associated with the virtual object in the 3D space. Determining the interaction mode may involve selecting an indirect interaction mode based on the hand position being outside the 3D region associated with the virtual object in the 3D space.

[0154] At block 1208, method 1200 includes interpreting the hand gesture using an interaction recognition process associated with the determined interaction mode. In some implementations, the method involves tracking the position of the hand gesture in 3D space and identifying one or more interaction points based on the tracking. The method may involve identifying one or more interaction points based on gaze direction.

[0155] In some implementations, determining the interaction mode includes selecting a direct interaction mode in which a virtual element is identified based on an intersection of a hand position and a 3D area associated with a virtual object in 3D space, and the interaction recognition process is a direct interaction recognition process applied to a hand gesture. Determining the interaction mode may involve selecting the direct interaction mode based on: determining a region of interest based on a gaze direction or a head gesture of the user; determining an intersection point where the hand position and the 3D area associated with the virtual object in the 3D space intersect; and determining that the intersection point is within the region of interest. In some implementations, the interaction mode is a direct interaction mode, and method 1200 includes suspending interaction detection for a predetermined period of time (e.g., 0.1, 1, 2, 4, 5, 10, 15, etc. seconds) after recognition of the hand gesture.

[0156] Some implementations initiate selection of a virtual object based on the interaction mode being a direct interaction mode and the gesture being a direct touch with the virtual object. Some implementations initiate hovering of a virtual object based on the interaction mode being a direct interaction mode and the hand position being close to the position of the virtual object and within the focus area. Hovering can be an interaction similar to a mouse or cursor hover in a traditional computing (e.g., desktop computer, laptop computer, etc.) environment, where the cursor is positioned over or on a UI element without activating or selecting a command, during which hover feedback can be provided, such as highlighting or otherwise distinguishing the element. When the position of the virtual object is no longer in the focus area, the hovering of the virtual object can be automatically maintained, for example, for a predetermined period of time, such as 10 seconds, 30 seconds, etc.

[0157] In some implementations, determining the interaction mode includes selecting an indirect interaction mode in which a virtual element is identified based on an intersection of a gaze direction and an area associated with a virtual object, and the interaction recognition process is an indirect interaction recognition process applied to a hand gesture. Determining the interaction mode may involve selecting the indirect interaction mode based on: determining a region of interest based on a gaze direction or a head gesture of the user; determining an intersection point of the hand position and a 3D area associated with the virtual object in the 3D space; and determining that the intersection point is outside the region of interest.

[0158] Some implementations initiate selection of a virtual object based on the interaction mode being an indirect interaction mode and the gesture being a pinch that occurs while gaze is directed toward the virtual object. In some implementations, the interaction mode is an indirect interaction mode, and interpreting the gesture includes: recognizing that the gesture is a pinch; and identifying the virtual object based on the gaze direction occurring during the pinch. In some implementations, the interaction mode is an indirect interaction mode, and interpreting the gesture includes: recognizing that the gesture corresponds to a hand engagement; identifying the virtual object based on the gaze direction occurring during the hand engagement; and initiating hovering of the virtual object based on the hand engagement.

[0159] In some implementations, method 1200 is performed by an electronic device that is a head-mounted device (HMD), and / or the XR environment is a virtual reality environment or an augmented reality environment.

[0160] In some implementations, a system or method detects when a user rests their hand (or arm) on their knee, the armrest of a chair they are sitting in, or another physical object for or in indirect mode. The user's hand or arm supported by another physical object can be used to determine whether to use indirect interaction mode and / or to determine how to interpret gestures performed by the hand.

[0161] In some embodiments, the method is performed by executing instructions (stored in a non-transitory computer-readable medium). The method may obtain a user hand position in a 3D space based on sensor data, wherein the hand position is associated with a gesture. The method determines that an arm or hand associated with the hand position is supported by (e.g., resting on) a physical object (e.g., a user's knee, a user's leg, an armrest, etc.). Based on determining that the arm or hand is supported by the physical object, the method interprets the gesture using an interaction recognition process associated with an indirect interaction mode. The gesture may be interpreted as a user interaction in an extended reality (XR) environment corresponding to the 3D space. Determining to utilize the indirect interaction mode may be based on determining that the arm or hand is supported by a physical object. The method may initiate an action (e.g., selection of a virtual object) based on recognizing the gesture (e.g., based on recognizing a gesture that occurs when gaze is directed toward a virtual object).

[0162] In the indirect interaction mode, a virtual element may be identified based on the intersection of a gaze direction and a region associated with a virtual object.

[0163] The interaction recognition process is an indirect interaction recognition process of the application that identifies hand gestures. In some specific implementations, recognizing gestures involves recognizing pinches.

[0164] In some embodiments, the method determines that the arm or hand associated with the hand position is supported by the physical object based on determining a first position of the arm or hand, determining a second position of the physical object, and determining that a relationship between the first position and the second position meets a criterion (for example, determining that a skeletal representation of the user's hand is within a threshold distance of the surface of the physical object based on image data, depth data, etc.).

[0165] 3D pose recognition using defined 3D regions

[0166] Various embodiments disclosed herein include devices, systems, and methods for interpreting user hand gestures relative to UI elements in 3D space using defined 3D areas. Such gestures can be direct, where the hand virtually interacts with a UI element by virtually touching the UI element. For example, a user can move their fingers to directly tap, pinch, swipe, or otherwise interact with a UI element in 3D space. Conversely, gestures can be indirect, where the hand remains separate from the UI element in 3D space. For example, when the UI is displayed at a distance in 3D space, such gestures can be performed by the user's hand while the user's hand is resting on the user's side or otherwise close to the user. The embodiments disclosed herein are particularly useful for direct gestures, but may also be applied to indirect gestures.

[0167] The specific implementation disclosed herein uses defined 3D areas to interpret hand gestures relative to UI elements in 3D space. The type or 3D location of the input recognized may depend on whether and how the path of the user's hand gesture in 3D space relates to the 3D area. In one example, the 3D area is used as a no-input or "dead zone" area relative to a particular type of input gesture. In a specific example, to determine whether to perform a tap gesture relative to a flat virtual user interface (and where to perform the tap gesture), the path of the user's finger as it moves toward and through the flat virtual user interface may be considered. The 3D area for a "tap" gesture may be defined based on the 3D location of the virtual user interface (e.g., a button on a flat virtual UI). Whether and / or how the path of the user's finger is interpreted as a tap gesture (or another type of gesture) may be based on the path and the defined 3D area. For example, if the path of a finger enters or leaves a particular 3D area (e.g., goes outside a 3D area not expected to be associated with a tap on a particular UI icon and / or enters a 3D area expected to be associated with a tap on that UI icon), then the path may be determined not to correspond to a tap gesture. The path may instead be analyzed as input with respect to other matters (e.g., whether the path corresponds to a non-tap gesture (e.g., a scroll, a swipe, etc.)).

[0168] 3D areas can be configured to provide various features. For example, a 3D area can be positioned and / or configured based on the position, orientation, and / or shape of the UI element to which it corresponds (e.g., based on the position and shape of a button on a flat virtual user interface). In some implementations, the 3D area has a non-cylindrical shape (e.g., a conical shape), or otherwise has a shape with a different contour based on the distance from the virtual user interface surface to which it corresponds. In some implementations, the 3D area has an orientation based on the viewpoint direction. In some implementations, the 3D area has a size determined based on the speed of movement of the portion of the user (e.g., a finger) as the finger moves along the path.

[0169] 13A to 13B Shown are gestures made relative to user interface elements and corresponding 3D areas on a 2D user interface. In this example, Figure 13A An XY view of the user interface 1300 is shown, and Figure 13B A YZ view of user interface 1300 is shown as hand 1302 performs a gesture by moving along path 1315 (i.e., from hand position 1302a to hand position 1302b). In this example, the gesture is intended (by user 102) as a tap gesture on virtual button 1310. The user performs the gesture by moving their hand / finger across virtual button 1310, i.e., the user intends to make a motion with their hand such that path 1315 intersects virtual button 1310 at position 1320.

[0170] Note that the user's movements in the real world (e.g., physical environment 100) correspond to movements within a 3D space (e.g., an XR environment based on the real world and including virtual content such as a user interface positioned relative to real-world objects including the user). Thus, the user moves their hand through empty space in the physical environment 100, but the hand (i.e., the drawing or representation of the hand) intersects and / or pierces the user interface 1300 of the XR environment based on the physical environment.

[0171] The user's device (e.g., device 110 or device 105 of FIG. 1 ) or another device determines whether the user's gesture (e.g., path 1315) corresponds to input to user interface 1300. For example, one or more 3D regions may be defined and used to interpret the user's gesture. Figure 13BIn the example of , a first 3D region 1340 is defined as corresponding to the 3D area in which the intended user's hand / finger is located after piercing the virtual button 1310 through the plane of the user interface 1300 during the motion path associated with the tap gesture. Conversely, a second 3D region 1350 (surrounding the cylindrical 3D region 1340) is defined as corresponding to a different 3D area in which the intended user's hand does not enter during the motion path associated with the tap gesture after piercing the virtual button 1310. If the user's hand path 1315 leaves the first 3D region 1340 and / or enters the second 3D region 1350 after piercing the user interface 1300, the data is further analyzed to determine whether it is a specific non-tap gesture (e.g., using a classification machine learning model focused on non-tap gestures). However, as Figure 13B As shown, if the user's hand path 1315 remains within the first 3D area 1340 or does not enter the second 3D area 1350 after piercing the user interface 1300, the data is further analyzed to determine whether it is a tap gesture (e.g., using a classification machine learning model focused on tap gestures).

[0172] In other embodiments, no additional classification algorithm / machine learning model is used, and the motion path is classified as a tap gesture or a non-tap gesture based solely on evaluating which 3D areas (e.g., the first 3D area 1340 and / or the second 3D area 1350) the path 1315 intersects.

[0173] In some implementations, if path 1315 does not leave the first 3D area 340 and does not enter the second 3D area 1350, the gesture is interpreted as a tap gesture, and if path 1315 does leave the first 3D area or enter the second 3D area 1350, the gesture data (e.g., path 1315) is analyzed to determine which of multiple types of non-tap gestures (if any) the gesture is.

[0174] Figure 14 Shown are gestures made relative to user interface elements on user interface 1300. The lack of tactile feedback may result in imprecise hand placement / inaccurate touch. For example, a user may attempt to provide a tap gesture by piercing the UI surface directly, but actually have an angled trajectory. Figure 14In this example, the user's hand moves along path 1415 from hand position 1402a to hand position 1402b, piercing user interface 1300 at position 1440 and stopping at position 1420. This stopping position 1420 can be used to identify a corresponding position 1430 on user interface 1300, which can be used to determine which user interface element (if any) of user interface 1300 the gesture was intended for. This can lead to inaccuracies, for example, if a different UI element is at piercing position 1440 (e.g., an intended UI element) compared to the corresponding stopping position 1430 (e.g., an interpreted position).

[0175] In some implementations, the puncture location 1440 is used instead of the corresponding stop location 1430 to interpret the gesture, for example, to identify the intended UI element. In some implementations, both the puncture location 1440 and the corresponding stop location 1430 on the user interface 1300 are used to interpret the gesture. If both locations (i.e., the puncture location 1440 and the corresponding stop location 1430) are within the same user interface element, the input may be associated with that user interface element. If the puncture location 1440 and the corresponding stop location 1430 are within different user interface elements (or one of these locations is not within a user interface element), an algorithm or machine learning model may be used to apply criteria to select which user interface element the gesture will be associated with. In some implementations, a separation distance 1460 (e.g., the separation distance between the puncture location 1440 and the stop location 1420 on a virtual plane parallel to the user interface 1300) is used to determine whether and how to interpret the input gesture. For example, if the separation distance is greater than a threshold, the input may be excluded and not considered a tap input.

[0176] In some implementations, the device learns to interpret the expected puncture position based on the actual stop position, e.g., determining the corresponding stop position 1430 as the expected puncture position. In some implementations, the determined expected puncture position is used to identify the expected user interface element. In some implementations, both the determined expected puncture position and the actual puncture position are used to interpret the gesture, e.g., by identifying the user interface element and / or user interface coordinates, e.g., by averaging or weighting. In some implementations, the system learns how to interpret a particular user's input gesture based on feedback from the user (e.g., feedback indicating that the expected puncture position should be used instead of the actual puncture position).

[0177] Figure 15Shown is a gesture made relative to a user interface element on user interface 1300, an inclined 3D area 1540 for an expected tap gesture, and an inclined dead zone area 1550 for an expected tap gesture. In this example, the user moves their hand along path 1515 from initial position 1502a through second position 1502b, and thus moves through user interface 1300 at an angle (i.e., a non-orthogonal direction). A first inclined 3D area 1540 is defined based on path 1515 to correspond to a 3D area in which the user's hand / finger is expected to be after piercing user interface 1300 (e.g., at a specific location or through a specific UI element) during the motion path associated with the tap gesture. Conversely, a second inclined 3D area 1550 is defined to correspond to a different 3D area in which the user's hand is expected to not enter during the motion path associated with the tap gesture. If the user's hand path 1515 leaves the first 3D area 1540 and / or enters the second 3D area 1550 after piercing the user interface 1300, the motion is deemed to correspond to a non-tap gesture and / or is further analyzed to identify the type of non-tap gesture, if any, that was intended for the movement. Figure 15 As shown, if the user's hand path 1515 remains within the first tilted 3D area 1540 or does not enter the second tilted 3D area 1550 after piercing the user interface 1300, the motion is deemed to correspond to a tap gesture and / or is further analyzed to identify the tap gesture or its characteristics (e.g., duration, intended UI target, etc.). Figure 5 It shows how a 3D area (eg, a dead zone) can be centered in the direction of hand movement, such as for identifying a tap gesture and a non-tap gesture.

[0178] Figure 16 16. A gesture is shown relative to a user interface element on user interface 1300, a corresponding non-cylindrical 3D area 1640 for an intended tap gesture, and a corresponding dead zone area 1650 for an intended tap gesture. In this example, the user moves their hand along path 1615 from initial position 1602a through second position 1602b, and thereby through user interface 1300. The hand stops (and / or retracts) when the fingertip reaches position 1635. In this example, the central axis of non-cylindrical 3D area 1640 is aligned with the normal direction of user interface 1300. In alternative implementations, the central axis of non-cylindrical 3D area 1640 can be aligned differently (e.g., based on viewing direction).

[0179] First 3D area 1640 is defined as the 3D area that corresponds to the user's hand / finger, after penetrating user interface 1300 (e.g., at a particular location or through a particular UI element), during the motion path associated with a tap gesture. In this example, first 3D area 1640 has a non-cylindrical shape, e.g., a cone whose diameter increases based on increasing distance from user interface 1300 in 3D space. This shape allows the motion path to deviate more (e.g., from the perfect / expected path) after passing through a user interface element at a greater distance from the user interface, while still being considered a specific gesture type, such as a tap gesture. Consequently, separation distance 1660 (e.g., the distance between fingertip position 1635 when the hand is stopped and corresponding position 1625 on an orthogonal line passing through the user interface element) can be greater than diameter 1670 of first 3D shape 1640 at the plane of user interface 1300. Using a non-cylindrical 3D region shape for the first 3D region 1640 may provide greater flexibility or otherwise enable evaluation of user movements in a manner that allows for deviations from perfect gestural behavior while still ensuring that the movements are accurately interpreted.

[0180] exist Figure 16 , second 3D region 1650 (outside of first 3D region 1640) is defined as corresponding to a distinct 3D region that the user's hand is not expected to enter during the motion path associated with the tap gesture. If the user's hand path 1615 leaves first 3D region 1640 and / or enters second 3D region 1650 after piercing user interface 1300, the motion is deemed to correspond to a non-tap gesture and / or is further analyzed to identify the type of non-tap gesture, if any, that was intended for the movement. However, as Figure 16 As shown, if the user's hand path 1615 remains within the first 3D area 1640 or does not enter the second 3D area 1650 after piercing the user interface 1300, the motion is deemed to correspond to a tap gesture and / or is further analyzed to identify the tap gesture or its characteristics (e.g., duration, intended UI target, etc.).

[0181] Figure 1717. A gesture is shown relative to a user interface element on user interface 1300, a corresponding non-cylindrical 3D area 1740 for an expected tap gesture, and a corresponding dead zone area 1750 for the expected tap gesture. The user moves their hand along path 1715 from initial position 1702a through second position 1702b, and thus through user interface 1300. The hand stops (and / or retracts) when the fingertip reaches position 1735. 3D area 1740 for an expected tap gesture is tilted to be centered on an extended portion of path 1715, e.g., where the user's hand would have traveled if the hand had continued on a straight path, given the trajectory of path 1715.

[0182] First 3D area 1740 is defined as the 3D area that corresponds to the user's hand / finger, after penetrating user interface 1300 (e.g., at a specific location or through a specific UI element), during the motion path associated with a tap gesture. In this example, first 3D area 1740 has a non-cylindrical shape, e.g., a rounded-end pyramidal shape whose diameter increases at a slower rate as it moves farther from user interface 1300 in 3D space. This shape allows the motion path to deviate more (e.g., from the perfect / expected path) after passing through a user interface element at a greater distance from the user interface, while still being considered a specific gesture type, e.g., a tap gesture. Consequently, separation distance 1760 (e.g., the distance between fingertip position 1735 when the hand is stopped and a corresponding position 1725 on a line orthogonal to the user interface element) can be greater than the diameter of first 3D shape 1740 at the plane of user interface 1300. Using a non-cylindrical 3D region shape for the first 3D region 1740 may provide greater flexibility or otherwise enable evaluation of user movements in a manner that allows for deviations from perfect gestural behavior while still ensuring that the movements are accurately interpreted.

[0183] exist Figure 17 , the second 3D area 1750 (outside the first 3D area 1740) is defined as corresponding to a different 3D area that the user's hand is not expected to enter during the motion path associated with the tap gesture. If the user's hand path 1715 leaves the first 3D area 1740 and / or enters the second 3D area 1750 after piercing the user interface 1300, the motion is deemed to correspond to a non-tap gesture and / or is further analyzed to identify the type of non-tap gesture, if any, that was intended for the movement. However, as Figure 17As shown, if the user's hand path 1715 remains within the first 3D area 1740 or does not enter the second 3D area 1750 after piercing the user interface 1300, the motion is deemed to correspond to a tap gesture and / or is further analyzed to identify the tap gesture or its characteristics (e.g., duration, intended UI target, etc.).

[0184] 18A to 18B An exemplary non-cylindrical 3D region is shown. Figure 18A , a first 3D region 1840a is defined as corresponding to the 3D area in which the user's hand / finger is expected to be after penetrating the user interface 1800 (e.g., at a particular location or through a particular UI element) during the motion path associated with the tap gesture. In this example, the first 3D region 1840a has a non-cylindrical shape along at least a portion of its length, e.g., an initial cone shape that increases in diameter before changing to a cylindrical shape for the outer portion when beyond position 1860a. This shape enables the motion path to deviate more (e.g., deviate from the perfect / expected path) after passing through user interface elements at a greater distance from the user interface, while still being considered a particular gesture type, e.g., a tap gesture. Using a non-cylindrical 3D area shape for the first position of the first 3D region 1840a can provide greater flexibility, or otherwise enable evaluation of user movements in a manner that allows for deviations from perfect gesture behavior while still ensuring that the movement is accurately interpreted. In Figure 18A , a second 3D region 1850b (outside the first 3D region 1840a) is defined as corresponding to a different 3D region into which the user's hand is not expected to enter during the motion path associated with the tap gesture.

[0185] exist Figure 18BIn the example, first 3D region 1840b is defined as the 3D region corresponding to the area within which the user's hand / finger is intended to be located after penetrating user interface 1800 (e.g., at a specific location or through a specific UI element) during the motion path associated with the tap gesture. In this example, first 3D region 1840b has a non-cylindrical shape along at least a portion of its length, e.g., an initial rounded-end pyramidal shape with increasing diameter before changing to a cylindrical shape beyond position 1860b. Non-cylindrical 3D shape 1840b can be defined logarithmically or exponentially. 3D shape 1840b is initially steep in portion 1830 (e.g., rapidly changing diameter / height along z), then gradually changes to a less steep slope (e.g., less rapidly changing diameter / height along z) in portion 1870, before becoming flat (cylindrical / constant diameter / height) beyond position 1860b. This shape enables the motion path to deviate more (e.g., deviate from the perfect / expected path) after passing through a user interface element at a greater distance from the user interface, while still being considered a particular gesture type, such as a tap gesture. Using a non-cylindrical 3D region shape for the first position of the first 3D region 1840b may provide greater flexibility or otherwise enable evaluation of user movements in a manner that allows for deviations from perfect gesture behavior while still ensuring that the movement is accurately interpreted. Figure 18B , a second 3D region 1850b (outside the first 3D region 1840b) is defined as corresponding to a different 3D region that the user's hand is not expected to enter during the motion path associated with the tap gesture.

[0186] Figure 19 Example 3D regions 1920a-c are shown oriented relative to user interface 1900 based on viewpoint directions. Viewpoint directions 1910a-c in this example are determined based on the position of the fingertip when it respectively penetrates user interface 1900 and the user's viewpoint 1905. Thus, 3D region 1920a is oriented at an angle (relative to an orthogonal direction 1930) based on viewpoint direction 1910a, which is determined based on viewpoint 1905 and the position of the hand penetrating the user interface when in position 1902a. Similarly, 3D region 1920c is oriented at an angle (relative to an orthogonal direction 1930) based on viewpoint direction 1910c, which is determined based on viewpoint 1905 and the position of the hand penetrating the user interface when in position 1902c.

[0187] In contrast, 3D region 1920b is treated differently. 3D region 1920b is oriented along an orthogonal direction to the plane on which user interface 1900 is located, rather than along a viewpoint direction such as viewpoint direction 1910b. The choice of using an orthogonal orientation (rather than a viewpoint-based orientation) may be based on the circumstances, i.e., based on direction 1910b being similar to an orthogonal direction (e.g., within a threshold similarity metric of orthogonal directions). The system may be configured to orient the 3D region based on whether the viewpoint direction deviates significantly from the orthogonal direction (e.g., exceeding a threshold similarity metric). If the viewpoint direction does not deviate substantially, an orthogonal orientation is used. If the viewpoint direction does deviate substantially, a non-orthogonal orientation determined based on the viewpoint direction is used. In other examples, 3D region 920b may (similar to 3D regions 1920a, 1920c) be oriented based on the viewpoint direction (e.g., based on direction 1910b).

[0188] exist Figure 19 , during a non-tap interaction (e.g., a scroll gesture) the fingertip follows trajectory 1940. This shows an example of how a typical non-tap gesture trajectory may go outside of 3D areas 1920a and 1920c.

[0189] FIG. 20A to FIG. 20B An example 3D region that is resized based on gesture velocity is shown. Figure 20A , a first 3D region 2040b is defined as the 3D region that corresponds to the intended location of the user's hand / finger after penetrating the user interface 1300 (e.g., at a particular location or through a particular UI element) during the motion path associated with the tap gesture. In this example, the first 3D region 2040b has a non-cylindrical shape along at least a portion of its length. The diameter 2010a of the 3D region 2040b is relatively narrow. This is based on the relatively slow velocity of the hand gesture from hand position 2002a to and / or through hand position 2002b. The 3D region can expand at higher velocities to allow for greater flexibility / clutter that can occur when a user taps very quickly. During relatively slow movements, for example, when considering whether a path corresponds to a tap, relatively small inaccuracies in the direction of the gesture's penetration can be tolerated.

[0190] exist Figure 20B , the first 3D area 2042a is further defined as corresponding to the 3D area in which the user's hand / finger is expected to be after penetrating the user interface 1300 (e.g., at a specific location or through a specific UI element) during the motion path associated with the tap gesture. In this example, the first 3D area 2040a has a non-cylindrical shape along at least a portion of its length. However, unlike Figure 20ADiameter 2010b of 3D region 2042a is relatively wider than diameter 2010a of first 3D region 2040a. This is based on the relatively fast speed of the hand gesture from hand position 2002c to and / or through hand position 2002d. During relatively fast movements, such as when considering whether a path corresponds to a tap, relatively large inaccuracies in the direction of the gesture's penetration can be tolerated.

[0191] Figure 21 2 is a flow chart illustrating a method 2100 for interpreting user position data as input based on a gesture path relative to a defined 3D area. In some implementations, a device such as electronic device 110 or device 105 performs method 2100. In some implementations, method 2100 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. Method 2100 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, method 2100 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0192] At block 2102, method 2100 displays a user interface in an XR environment corresponding to a 3D space, wherein the user interface includes user interface elements. The user interface includes user interface elements, such as buttons, icons, text input fields, sliders, or menu items, that are presented as part of a virtual UI displayed at a fixed location or otherwise within the XR environment, for example, a virtual 2D menu with buttons displayed several feet in front of a user in XR.

[0193] At block 2104, method 2100 obtains user position data corresponding to a 3D position of a portion of the user (e.g., a finger / hand) in 3D space. The user position data may be obtained based on sensor data (e.g., image sensor, depth sensor data, motion sensor data, etc.) and may provide a path of the portion of the user over a period of time. In one example, the user position data may be a skeletal representation of the user that is generated periodically (e.g., at 30 fps) based on an outward-facing image / depth sensor on the HMD. Such user position data may provide a path of a portion of the skeleton (e.g., a finger skeleton portion). In other words, for example, by tracking the orientation of the skeleton at multiple moments in time, the path of the finger portion of the skeleton relative to 3D space over time may be determined.

[0194] In some implementations, the portion of the user corresponds to a point on or within a finger of the user. The portion of the user can correspond to a point on or within a hand of the user. The user position data can correspond to a position within a skeletal representation of the user that is generated periodically (e.g., at multiple points in time over a period of time).

[0195] At block 2106, method 2100 determines a 3D region (e.g., a no-input or dead zone region) based on the position of the user interface element within the 3D space. For example, this may involve determining the position, orientation, size, and shape of the 3D region based on how many feet in front of the user's viewpoint a button or other UI element on a flat 2D interface would appear within the XR environment.

[0196] In some implementations, the 3D region has a shape whose width (eg, diameter) varies based on the distance from the 3D element (eg, forming a tapered cylinder, cone, frustum, etc.). Figure 16 、 Figure 17 、 Figure 18A 、 Figure 18B 、 Figure 19 and FIG. 20 illustrate that a 3D region has a shape whose width (eg, diameter) varies based on the distance from the 3D element.

[0197] In some implementations, the 3D region has an orientation determined based on a viewpoint direction. In some implementations, the 3D region has an orientation determined based on a viewpoint direction, which is determined based on a puncture location and a viewpoint location. Figure 19 A 3D region is shown with an orientation determined based on the viewpoint direction.

[0198] In some implementations, the 3D region has size attributes (eg, width / diameter) determined based on a speed associated with the movement of the portion of the user along the path. Figure 20 illustrates a 3D region having size attributes determined based on gesture speed.

[0199] At block 2108, method 2100 interprets the user position data as input corresponding to a user interface element based on the path of the portion of the user relative to the 3D area. For example, the user position data and the path of the portion of the user can be used to determine whether the path corresponds to a tap input gesture (e.g., where the path remains within the first 3D area and / or does not enter the second 3D area) or a non-tap input gesture (e.g., where the path does not remain within the first 3D area and / or enters the second 3D area). In some implementations, if all 3D positions of the finger along the path remain within the first 3D area and / or do not enter the second 3D area, the finger position data can be considered exclusively for tap recognition (e.g., sent to a tap recognizer / classifier). If the finger position along the path does enter the second 3D area, the finger position data can instead be considered for non-tap recognition (e.g., sent to a non-tap classifier). In some implementations, the first 3D area is a dead zone in the sense that in-plane motion is suppressed when the finger is within the area. In the second region, the presence of a finger is considered a non-tap gesture and there is no motion suppression, eg, there is no dead zone effect.

[0200] In some implementations, interpreting the user location data involves determining whether the path of the portion of the user during the time period corresponds to a tap gesture. Interpreting the user location data may involve determining that the path of the portion of the user relative to the 3D area is entirely within the 3D area. Interpreting the location data may be based on determining that the path of the portion of the user relative to the 3D area is entirely within the 3D area. Based on this, method 2100 may determine to use a tap recognizer algorithm or machine learning model to determine that the path of the portion of the user corresponds to a tap. Such a tap recognizer algorithm or machine learning model may use only two-dimensional input data, for example, a position on a plane associated with the user interface determined based on the 3D position of the portion of the user along the path. Using only 2D data may enable faster, real-time, or otherwise more efficient evaluation of gestures. Furthermore, using a 3D area when analyzing user motion as described herein may enable the selection of an appropriate (e.g., gesture-type-specific) algorithm or machine learning model. Such selection of an appropriate processing technique may enable efficient processing; for example, a gesture-type-specific processing technique may be more efficient than a general algorithm or machine learning model that attempts to identify / interpret input for multiple (e.g., all) input gesture types.

[0201] Interpreting the user position data may involve determining that the path of the portion of the user relative to the 3D area extends beyond the 3D area. Interpreting the user position data may involve determining, based on determining that the path of the portion of the user relative to the 3D area extends beyond the 3D area, that the path of the portion of the user corresponds to a non-tap gesture using a non-tap recognizer algorithm or a machine learning model. Different processing modules may thus be used for some types of input (corresponding to tap gestures in which the path does not extend beyond the 3D area) and other types of input (corresponding to non-tap gestures in which the path does extend beyond the 3D area). For example, using such different alternative processing modules may be more efficient than using a single processing module for both tap input and non-tap input. In some embodiments, the path corresponds to a non-tap gesture identified by a non-tap recognizer algorithm or a machine learning model that uses only two-dimensional input data, such as a scroll gesture, a pan gesture, a swipe gesture, or a drag gesture.

[0202] Interpret user movements as direct touch UI interactions

[0203] Various embodiments disclosed herein include devices, systems, and methods for interpreting direct touch-based gestures (such as drag gestures and swipe gestures) made by a user virtually touching one location of a user interface and, while still touching, moving their hand to another location of the user interface (UI). Such gestures can be made relative to a user interface that is presented as virtual content in a 3D space of an extended reality (XR) environment. Ideally, such gestures would be associated with a user interface location based on where the user's hand virtually intersected the user interface (e.g., where the hand made and broke contact with the user interface). However, because the user's perception of when and where they virtually touched the user interface (e.g., overlapped the user interface in XR space) can be inaccurate, unintended gain or loss of motion associated with the user interface (referred to as "hooking") can occur. For example, when the user expected no movement of the segment during the user interface contact, the segment moved by the user may be associated with the user interface contact. Conversely, when the user expected no movement of the segment during the user interface contact, the segment moved by the user may not be associated with the user interface contact.

[0204] Some implementations determine which segments of a movement are to be associated with user interface content based on characteristics of the movement. In a drag (i.e., where a user attempts to touch at a location on the user interface, moves to a second location on the user interface, and releases touch at the second location), a hook may occur when a segment of the movement associated with a retracting hand is associated with a UI contact, contrary to the user's expectation that such a retraction would not occur during the UI contact. This may cause the system to identify an incorrect breakpoint on the user interface, i.e., using the retracted portion of the movement to identify the breakpoint rather than a location on the user interface corresponding to the user's location when the intended UI contact motion stopped. Some implementations avoid such erroneous associations (and thus more accurately interpret movement) by determining whether to associate such segments (e.g., potential retraction segments) based on whether the characteristics of the segment indicate retraction. In other words, if a segment of the movement is likely to be a retraction, some implementations determine that a segment of the movement that would otherwise be associated with user interface content (e.g., based on actual position overlap) should not be associated with the user interface contact. This may involve determining not to associate a segment of the motion with a user interface contact based on determining that the segment is likely to be a retraction based on evaluating the segment's alignment with the retraction axis, the significance of a change in retraction direction, or a cessation of motion.

[0205] In some implementations, user movement is interpreted using a technique that uses dynamic interruption volumes to avoid accidental gain or loss of motion associated with the UI. Some implementations determine that an interruption occurs when the user moves outside an interruption volume, which is dynamically adjusted based on retraction confidence and / or penetration depth. Intentional swipe momentum can be maintained by interrupting at an appropriate time before the motion is lost due to arcing or retraction.

[0206] Figure 22 1 , a device 2210 (e.g., device 110 or device 105 of FIG. 1 ) provides views 2210a-e of an XR environment based on a physical environment, in which user movements are interpreted. The views 2210a-c of the XR environment include an exemplary user interface 2230 of an application (i.e., virtual content) and a drawing 2220 of the table 120 (i.e., real content). Providing such views may involve determining 3D properties of the physical environment 100 and positioning the virtual content, such as the user interface 2230, in a 3D coordinate system corresponding to the physical environment 100.

[0207] exist Figure 22In the example of FIG. 1 , user interface 2230 may include various content and user interface elements, including a scroll bar axis 2240 and its scroll bar handle 2242 (also referred to as a scroll bar thumb). Interaction with scroll bar handle 2242 may be used by user 102 to provide input to user interface 2230, such as by scrolling displayed content or otherwise responding. User interface 2230 may be flat (e.g., planar or curved without depth). Displaying user interface 2230 as a flat surface may provide various advantages. Doing so may provide a user interface that is easy to understand or otherwise easy to use as part of the XR environment to access applications.

[0208] User interface 2230 can be a user interface of an application, as shown in this example. For illustrative purposes, user interface 2230 is simplified, and the user interface can actually include any level of complexity, any number of user interface elements, and / or a combination of 2D and / or 3D content. User interface 2230 can be provided by various types of operating systems and / or applications, including but not limited to messaging applications, web browser applications, content viewing applications, content creation and editing applications, or any other application that can display, present, or otherwise use visual and / or audio content.

[0209] In some implementations, multiple user interfaces (e.g., corresponding to multiple different applications) are presented sequentially and / or simultaneously within the XR environment using one or more flat background portions. In some implementations, the position and / or orientation of such one or more user interfaces can be determined to facilitate visibility and / or usage. Within the 3D environment, one or more user interfaces can be in a fixed position and orientation. In such cases, user movement (e.g., movement of the user moving their head while wearing the HMD) will not affect the position or orientation of the user interface within the 3D environment.

[0210] In other implementations, one or more user interfaces may be body-locked content, e.g., at a distance and orientation offset relative to a portion of the user's body (e.g., their torso). For example, body-locked content for a user interface may be 2 meters away and 45 degrees to the left of the user's torso's forward vector. While wearing the HMD, if the user turns their head and their torso remains stationary, the body-locked user interface will appear to remain stationary in the 3D environment at 2 meters to the left of and 45 degrees to the torso's forward-facing vector. However, if the user does rotate their torso (e.g., by rotating in their chair), the body-locked user interface will follow the torso rotation and reposition within the 3D environment so that it remains 2 meters to the left of and 45 degrees to the new forward vector of their torso.

[0211] In other implementations, user interface content is defined at a specific distance from the user, where it remains fixed relative to the user's orientation (e.g., if initially displayed in a cardinal orientation, it will remain in that cardinal orientation regardless of any head or body movement). In this example, the orientation of the body-locked content is not referenced to any part of the user's body. In this different implementation, the body-locked user interface will not reposition itself based on torso rotation. For example, a body-locked user interface may be defined as 2 meters away and, based on the direction the user is currently facing, may initially be displayed north of the user. If the user rotates their torso 180 degrees to face south, the body-locked user interface will remain 2 meters north of the user, with north now directly behind the user.

[0212] The body-locked user interface can also be configured to always remain aligned with the line of gravity or the horizon, so that changes in head and / or body orientation in roll will not cause the body-locked user interface to move within the 3D environment. Translational movements will cause the body-locked content to reposition within the 3D environment to maintain the distance offset.

[0213] exist Figure 22 , at a first moment corresponding to view 2210a, user 102 has positioned their hand in physical environment 100, and a corresponding drawing 2202 of user 102 shows that the fingertips of user 102 have not yet touched user interface 2230. Device 2110 can track user positioning, e.g., the location of a user's fingers, hand, arm, etc.

[0214] The user 102 moves their hand / fingers forward in the physical environment 100, causing a corresponding movement of the user's 102 drawing 2202. Thus, at a second moment corresponding to view 2210b, the user 102 has positioned their hand in the physical environment 100, and the user's 102 corresponding drawing 2202 shows the user's 102 fingertips touching or extending into the scroll bar handle 2242.

[0215] Device 2110 can determine the position of the user relative to user interface 2230 (e.g., within the XR environment) and identify a user interaction with the user interface based on a positional relationship between the user and the user interface and / or information indicating when the user perceives or anticipates contact of their hand / finger with the user interface. In this example, when a portion of drawing 2202 of the fingertip of user 102 contacts scroll bar handle 2242, device 2110 detects a touch point (e.g., a point in time and / or in 3D space where contact between the user and the user interface occurs or is anticipated to occur).

[0216] Detecting such a touch point may initiate a user interaction. For example, the device 2110 may begin tracking subsequent movement corresponding to a drag-type user interaction, which would be interpreted as moving the scroll bar handle 2242 along or otherwise based on the right / left movement of the portion of the drawing 2202 by the user 102. Movement of the scroll bar handle 2242 (caused by such user motion) may also trigger a corresponding user interface response, such as causing the user interface 2230 to scroll the displayed content according to the amount of movement of the scroll bar handle 2242.

[0217] exist Figure 22 In the example of FIG22 , at a third moment in time corresponding to view 2210 c, user 102 has moved their hand in physical environment 100, and the corresponding drawing 2202 of user 102 has moved to the left relative to user interface 2230, while the hand is still considered in contact with user interface 2230. The movement of the hand may continue to drag scroll bar handle 2242 in this manner until an interruption point (e.g., a point in time and / or 3D space where contact between the user and the user interface occurs or is expected to cease).

[0218] In this example, at a fourth moment in time, corresponding to view 2210d, user 102 continues to move their hand in physical environment 100, and user 102's corresponding drawing 2202 continues to move leftward relative to user interface 2230 because the hand is still considered in contact with the user interface before it reaches interruption point 2250. At a fifth moment in time, corresponding to view 2210e, device 2110 detects that the user has ended the drag-type user interaction and is retracting the hand, as shown by drawing 2202. The segment of the user's movement (e.g., the movement after the user begins to retract the drawing 2202 away from the user interface 2230 at interruption point 2250) is not associated with user interface contact, e.g., it is not interpreted as part of a drag-type user interaction.

[0219] Specific implementations disclosed herein interpret user movements relative to the positioning of a user interface within 3D space such that the user movements are interpreted as direct touches with the user interface in accordance with the user's expectations, e.g., when the user perceives or believes that they are virtually touching the user interface, which may not necessarily correlate precisely with actual contact occurring between the user and a drawing of the user interface in an XR environment.

[0220] Some implementations determine which segments of a movement will be associated with user interface content based on characteristics of the movement. During a drag (i.e., where a user attempts to touch at a location on the user interface, moves to a second location on the user interface, and releases touch at the second location), a hook may occur when a segment of the movement associated with a retracting hand is associated with a UI contact, contrary to the user's expectation that such a retraction would not occur during a UI contact. This may cause the system to identify an incorrect breakpoint on the user interface, i.e., using a retraction to identify the breakpoint rather than a location on the user interface that corresponds to the user's location when the dragging motion stopped.

[0221] Some implementations avoid such erroneous associations (and thus more accurately interpret movement) by determining whether to associate a segment (e.g., a potential retraction segment) based on whether the segment's characteristics indicate retraction. In other words, if a segment of motion is a retraction, some implementations determine that a segment of motion that would otherwise be associated with user interface content (e.g., based on actual position overlap) should not be associated with a user interface contact. This can involve determining not to associate a segment of motion with a user interface contact based on determining that the segment is a retraction based on (a) an assessment of the segment's alignment with the retraction axis, (b) the significance of a change in the retraction direction, or (c) cessation of motion.

[0222] Figure 23 23. The user's intention when moving relative to the actual user interface position is shown. In this example, during a first segment 2301 of the user's movement, the user 2310 moves a part of their body (e.g., their finger, hand, etc.) with the intention of making contact with the user interface. In this example, the first segment 2301 of the movement extends through the actual UI plane 2305 to the perceived UI plane 2304. The user may perceive (or otherwise expect) the UI plane to be in a different position than its actual position for various reasons.

[0223] Based on the user's perception of where the UI plane is (i.e., the perceived location of UI plane 2304), the user continues to move a part of their body (e.g., their finger, hand, etc.) during a second segment 2302 of movement in the drag-type motion, for example, moving their finger across the user interface. The actual path of motion during this second segment 2302 can be linear or non-linear (e.g., an arc as shown). In this example, based on the movement during the first segment 2301 and / or the second segment 2302, the device 2110 determines the location of a touchdown point 2315 on the actual user interface 2305. In one example, a change in direction exceeding a threshold is determined as the time of touchdown point 2315, and the location of touchdown point 2315 is determined based on where the movement intersects the actual UI plane 2305. In another example, the location 2306 where this change occurs is used to determine a corresponding location on the actual UI plane 2305 to be used as the touchdown point.

[0224] After establishing the touch point, the user's movement is monitored and used as user input. The movement is used as input (ie, continues to be associated with contact with the user interface) until a condition is met, for example, a break point is determined.

[0225] In this example, based on the user's perception of where the UI plane is (i.e., the perceived position of UI plane 2304), at the end of the intended drag motion that occurs at the end of the second segment 2302, the user moves a part of their body (e.g., their finger, hand, etc.) back toward themselves in a retracting motion during the third segment 2303 of the motion. During the second segment 2302 and the third segment 2303 of the motion, the motion is evaluated to attempt to identify when and where the user intended the UI contact to end. This evaluation can occur repeatedly (e.g., every frame, every 5 frames, every 0.1ms, etc.) so that once the user ceases intending to make contact with the user interface (or shortly thereafter), the association of the motion with the user interface contact can be determined. This can involve evaluating the path of the motion to determine whether the current segment of the motion has characteristics that meet retract criteria. Such retract criteria can be configured to distinguish a retracting motion from another type of motion (e.g., a subsequent drag motion, a swipe motion, etc.). The characteristic may be, but is not limited to: (a) a measure of alignment between the direction of movement and the direction of retraction; (b) a measure of the change in the direction of retraction; and / or (c) whether the user (eg, finger) has stopped.

[0226] exist Figure 23In the example of , the third segment 2303 is determined to be a retraction motion. Therefore, this third segment 2303 is not considered to be a movement associated with the user interface contact / drag input. Only the second segment 2302 is considered to be a movement associated with the user interface contact / drag input. The evaluation of whether a segment should be associated with the user interface contact can be used to determine an appropriate interruption point for the movement. In this example, the second segment 2302 transitions to the third segment 2303 at point 2307, that is, the association of the movement with the user interface contact is determined to end at this point in time. This is used to determine a corresponding position 2330 on the actual user interface 2305 to be used as an interruption point, rather than the position 2320 where the user's retracted body part (e.g., hand, finger, etc.) passes over the actual user interface 2305. Described below Figures 25 to 27 Additional examples are provided for using movement characteristics to interpret segments of user movement (eg, with respect to determining which segments should be associated with user interface contacts).

[0227] Figure 24 Also shown is the user's intent when making the movement relative to the actual user interface location. In this example, the user 2410 makes a swiping movement with a part of their body (e.g., their finger, hand, etc.). In this example, a first segment 2401 of the movement swipes through the actual UI plane 2405 into the perceived UI plane 2404. Based on the user's perception of where the UI plane is (i.e., the location of the perceived UI plane 2404), the user continues the swiping movement during a second segment 2402 and through a third segment 2403, during which the swiping motion generally arcs back toward the user. The end of a swipe may differ from a drag retraction (e.g., as in Figure 23 ), and in motion, can be used to identify the type of motion (e.g., drag or swipe) and / or handle the end of the motion differently (e.g., third segments 2303, 2403).

[0228] In some specific implementations, Figure 24 The swipe motion shown in FIG is interpreted using a dynamic interruption volume to avoid accidental gain or loss of motion associated with the UI. This may involve determining an interruption event occurs based on determining that the movement leaves an interruption volume, which is dynamically adjusted based on (a) retraction confidence and / or (b) penetration depth. Intended swipe momentum can be maintained by interrupting the motion at an appropriate time before it is lost due to arcing or retraction (e.g., using swipe trajectory correction). The following describes Figures 28 to 31 Additional examples using dynamic break volumes and correction trajectories (eg, swipe trajectories) are provided.

[0229] Figures 25 to 26A segment of a movement having characteristics corresponding to a drag motion followed by a retraction motion is shown. In this example, a user movement (e.g., of user 2510) includes a drag segment 2502 and a retraction segment 2503 relative to an actual user interface 2505. The movement transitions from drag segment 2502 to retraction segment 2503 at point 2503. This transition is detected based on detecting that retraction segment 2503 has one or more characteristics corresponding to retraction. In this example, a retraction direction 2510 is identified based on the current position of user 2510 (e.g., a finger, hand, etc.) and the user's head 2520. In other examples, the retraction direction may be based on another part of the user, for example, the direction between the current position of user 2510 (e.g., a finger, hand, etc.) and the center of the user's torso (not shown).

[0230] Retraction direction 2510 can be used to determine a retraction confidence, e.g., a measure of confidence that the current segment of movement corresponds to a retraction rather than another type of motion. For example, such a retraction confidence can be based on the degree of alignment of the segment with the retraction motion. Movements that are more aligned with retraction direction 2510 are more likely to correspond to a drag-retraction movement than movements that are not aligned with retraction direction 2510 (e.g., perpendicular to the retraction direction, etc.). In this example, retraction segment 2503 of movement is closely aligned with retraction direction 2510, so the segment is determined to be a retraction following a drag.

[0231] In some implementations, movement characteristics are used to detect retraction and / or trigger determination of an early interruption event (ie, before the user actually disconnects from the user interface).

[0232] In some implementations, rather than using the instantaneous direction of movement (e.g., direction 2603) to compare with the retraction direction 2515 to identify a retraction, an average direction of movement (2604) can be determined and compared with the retraction direction 2515 to identify a retraction. This can help ensure that noise or small changes in direction do not inadvertently trigger a retraction detection. For example, using the average direction of movement 2604 to identify a retraction may be more accurate than using the current instantaneous direction of movement 2603.

[0233] In some implementations, a hysteresis position 2504 (e.g., index fingertip hysteresis position) is used to determine an average direction of movement (e.g., movement 2604), and this average direction of movement is used to evaluate retraction confidence. This hysteresis position 2504 can be a lazy follow-up of the user position (e.g., finger position) determined using a delayed moving average filter (50ms, 125ms). The hysteresis position 2504 can be used to determine the average direction of movement from the hysteresis position 2504 to the current position 2508. For example, Retract axis / direction For example Current moving direction and retract axis / direction Can be used based on their dot product: To determine the retraction confidence. In this example, r c = 1.0 indicates a high confidence retraction, r c = -1.0 indicates a high confidence piercing type move, and r c = 0.0 indicates a low confidence retraction (no retraction). In cases where the sensor data providing the trajectory information is uncertain or in other cases where the trajectory of the movement is not trusted, the retraction confidence can be overridden or automatically set to 0.

[0234] Figure 27 A retraction dead zone 2720 is shown. Figures 25 to 26 In an example, a retraction dead zone 2720 is generated based on detecting the occurrence of a motion corresponding to a retraction. Retraction dead zone 2720 is an area or volume of 3D space used to interpret movement. For example, hand movement within retraction dead zone 2720 is considered a retraction. However, if the user movement leaves retraction dead zone 2720 in 3D space, the user movement may no longer be considered a retraction and may therefore be interpreted as an input, for example, as a tap, drag, swipe, etc. The retraction dead zone can be used to distinguish between a motion corresponding to an input and a motion corresponding to a retraction. The retraction dead zone can be shaped, positioned, and otherwise configured so that movement closer to the user interface 2505 is more likely to be outside of retraction dead zone 2720 than movement farther away from the user interface 2505, and therefore more likely to be interpreted as a continuous scroll, drag, etc. The retraction dead zone 2720 can have various shapes, for example, having a straight profile or a curved (e.g., exponential) profile.

[0235] exist Figure 27 , the retraction dead zone 2720 is aligned with (e.g., centered on) the retraction axis / direction 2515 so that any in-plane motion is discarded. Movement during the retraction segment 2503 within the retraction dead zone 2720 will not be associated with user interface contact, e.g., will not continue to affect the drag response. However, if the movement exits the retraction dead zone 2720, the movement may resume being considered as movement associated with the user interface contact. The retraction dead zone 2720 can be configured to time out after a threshold amount of time.

[0236] Figures 28 to 29A dynamic interruption volume is shown. Such an interruption volume is particularly useful with respect to swipe type inputs. Swipes tend to be faster and have more arcing motion than drag interactions. When swiping, a user may expect to maintain the motion / velocity at the time / space point when the UI contact is perceived to be interrupted. For example, a user may swipe and expect the swipe to initiate a scroll that continues after the UI contact is interrupted, based on the movement speed when the UI content ends. However, this perceived interruption may not precisely coincide with the actual interruption of the contact from the user interface. Among other things, some embodiments disclosed herein utilize dynamic interruption volumes to maintain the user's intended swipe momentum, for example, by interrupting the motion early before it is lost due to arcing motion or retraction.

[0237] Figure 28 A user movement 2802 (of user 2810) relative to a user interface 2805 is shown. An interruption volume 2815 is generated and used to determine when to interrupt the swipe motion, i.e., to discontinue associating the movement 2802 with the user interface contact. For example, based on the current position of the user 2810 or a position determined based on the current position of the user 2810 (e.g., a hysteresis position), the shape or position of the interruption volume 2815 may be adjusted over time. In this example, an axis 2830 of the interruption volume 2815 is aligned with a target axis (e.g., the z-axis of the user interface 2805 based on the current hysteresis position 2812). The current hysteresis position 2812 may be determined based on the current user position 2813 (e.g., based on a hysteresis parameter, such as a predetermined hysteresis period, a hysteresis distance, etc.). In this example, the interruption volume 2815 is a tracking hysteresis (e.g., an index finger associated with the position of the index fingertip). 滞后 2820) of the center of mass C xy The interruption volume 2815 can be configured to change shape, position, and / or orientation based on and / or during movement 2802. The interruption volume 2815 can expand and contract in an umbrella-like manner that maintains symmetry about axis 2830 while potentially shifting laterally relative to the user interface (e.g., in Figure 28 The interruption volume 2815 may be shifted based on the retraction confidence and / or based on the penetration direction depth 2825 (e.g., tracking the index finger 滞后 2820) to increase the slope.

[0238] In some implementations, the interruption volume 2815 is asymmetric, e.g., not symmetric about the axis 2830. For example, the interruption volume 2815 can include only a lower portion below the axis 2830. In some implementations, the interruption volume 2815 is symmetric about an axis that is not perpendicular / orthogonal to the user interface 2805. For example, the interruption volume can be symmetric about an axis that is at a predetermined angle relative to the user interface, where the predetermined angle is determined based on user-specific characteristics (e.g., typical motion path characteristics of the user when performing a given type of gesture).

[0239] In alternative implementations, the interruption volume 2815 is determined based on a predicted path (e.g., based on a trajectory, speed, or other characteristic of the user's motion). For example, the interruption volume 2815 can be determined based on a predicted path predicted when the gesture is initially recognized, such as a swipe gesture, and is associated with the speed, direction, path, or other motion characteristic. In some implementations, the interruption volume 2815 can be configured with respect to shape and position based on one or more points along the predicted path. In some implementations, the interruption volume is determined and / or adjusted over time during the course of the user's motion based on both the current user position and the predicted user path.

[0240] Figure 29 Different user movements 2902 (of user 2910) relative to user interface 2905 are shown. An interruption volume 2915 is generated and dynamically changed during movement 2902. Interruption volume 2915 is used to determine when to interrupt the swipe motion, i.e., to discontinue associating movement 902 with the user interface contact. In this example, axis 2930 of interruption volume 2915 is aligned with the target axis (e.g., the z-axis of user interface 2905 based on the current hysteresis position). In this example, interruption volume 2915 is the tracking hysteresis (e.g., the index finger associated with the index fingertip position). 滞后 2920) of the center of mass C xy The interruption volume 915 can be configured to change shape, position, and / or orientation based on and / or during movement 2902. The interruption volume 2915 can expand and contract in an umbrella-like manner, shift based on retraction confidence, and / or based on penetration direction depth 2925 (e.g., tracking index finger). 滞后 2920) to increase the slope.

[0241] Figure 28 and Figure 29 It is shown how different dynamic cutoff volumes 2815, 2915 can be used to account for different movements 2802, 2902. Figure 28 and Figure 29Different movements 2802, 2902 are shown, and corresponding dynamic interruption volumes 2815, 2915 have different shapes, sizes, and positions. Furthermore, during a given movement, the position, shape, and / or orientation of a given interruption volume is dynamically adjusted to correspond to the current state of the movement. The position of the interruption volume moves to adapt to the user's current position, depth, and movement path. Using dynamic (context-specific) interruption volumes can enable the device to better determine interruption events in different situations and ultimately interpret user movements more consistently with user expectations than when using fixed (universal) interruption volumes.

[0242] The shape of the interruption volume 2815, 2915 can be determined using parameters that allow the interruption volume to be customized for a particular implementation. Such parameters may include: β (slope sensitivity), which corresponds to the sensitivity of the slope to changes in penetration depth; and α (penetration depth scalar), which corresponds to the degree to which the interruption / volume center of mass can be displaced. These parameters can be used to determine the characteristics of the center of mass of the interruption volume 2815, 2915. For example, the length D0 2860, 2960 can be determined based on the hysteresis 2820, 2920 and the penetration depth scalar: For example, D0 = index finger 滞后 *α. The slope θ 2850 , 2950 can be determined based on the length D0 2860 , 2960 and the slope sensitivity: for example, θ=90−atan2(D0,β). The axis C of the interruption volume 2815 , 2915 z 2830, 2930 can be based on the retraction confidence r c (eg, as determined via the techniques disclosed herein) and the penetration depth 2825, 2925: for example, C z = mapping(|r c |, depth). The positioning of the interruption volumes 2815, 2915 relative to other dimensions (e.g., x / y) may depend on the hysteresis position, e.g., the index finger 滞后(xy) :For example, C xy =Index finger 滞后(xy) .

[0243] Figures 30 to 31 Shown based on Figure 28 2802. Natural arc motion (e.g., during a swipe) may result in lost motion upon interruption, which may cause UI issues such as "laborious" scrolling. Some implementations maintain the intended swipe speed upon interruption without introducing noticeable hooking of speed changes. Some implementations suppress overly aggressive hooking that is not interrupted early (e.g., based on drag retraction detection) via other techniques.

[0244] Figures 30 to 31The corrected trajectory 3020 is shown to be associated with the movement 2802 instead of the instantaneous trajectory 3120. In this example, the lag (ie, the index finger lag direction ) is used to determine the corrected trajectory 3020. The index finger lag direction may be determined based on the current index finger position and the previous index finger position (eg, the position of the previous frame): for example, The position difference (Δpos) can be determined based on the current index finger position and the previous index finger position: for example, Δpos = index finger gt -index finger prev If the moving segment has not been classified as dragging, the device may predict whether the next frame's delta (e.g., at position 3103) will be outside the interruption volume 2815. If so, the device compares the frame's position delta with the direction Consistent, for example, if the movement is predicted to leave the interruption volume 2815 in the next frame, the device corrects the trajectory. This technique can suppress some of the "bounce" of hooks for failed swipes and should not affect failed drags.

[0245] Figure 32 3200 is a flowchart illustrating a method 3200 for determining which segments of a movement to associate with user interface content based on characteristics of the movement. In some implementations, a device such as electronic device 110 or electronic device 105 performs method 3200. In some implementations, method 3200 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. Method 3200 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, method 3200 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0246] At block 3202, method 3200 displays an XR environment corresponding to a 3D environment, wherein the XR environment includes a user interface and a movement. The movement includes a fragment.

[0247] At block 3204, method 3200 determines the occurrence of an event associated with contact with a user interface in the XR environment (e.g., a make contact event), e.g., based on determining that contact with the UI occurs, is expected to occur, or is perceived by the user. This can involve determining when the user penetrates the user interface. This can involve indicating that a direct touch gesture is in effect, that input criteria (e.g., drag and / or swipe criteria) have been met, and that movement is being tracked relative to input to the user interface.

[0248] At block 3206, method 3200 determines whether each of the segments in the movement has a characteristic that satisfies a drag-retraction criterion. The drag-retraction criterion is configured to distinguish a retraction motion following a drag from another type of motion. The device may use one or more sensors to track a portion of the user (e.g., the user's hand, finger, fingertip, index fingertip, etc.). By way of example, the characteristic may be, but is not limited to: (a) a measure of alignment between the direction of movement and the direction of retraction during the corresponding segment; (b) a measure of how quickly the direction of movement changes; and / or (c) whether the user (e.g., hand / finger) has stopped moving. Figure 23 、 Figure 25 and Figure 26 Properties that can be used to evaluate whether a segment meets the drag-retract criteria are shown.

[0249] In some implementations, the characteristic includes a drag retraction confidence level determined based on an alignment between a movement direction and a retraction direction during the corresponding segment. The retraction direction is a direction from a portion of the user (e.g., a finger, a hand, etc.) used for interaction to a central portion of the user's head (e.g., a head, a torso, etc.). The drag retraction criterion may be whether the drag retraction confidence level exceeds a threshold.

[0250] In some implementations, the drag retraction criterion is whether the change in drag retraction confidence exceeds a threshold (e.g., a kink threshold). A rapid change in drag retraction confidence can correspond to a rapid change in movement direction relative to the retraction axis, which can indicate that the user's intended movement of touching the user interface has ended. Similarly, the drag retraction criterion can include whether a portion of the user has stopped moving (e.g., is currently moving at a rate below a threshold speed of 0.1 m / s). Stopping can indicate that the user's intended movement of touching the user interface has ended or that the user has or is about to begin retraction.

[0251] At block 3208, method 3200 associates a subset of the moved segments (e.g., one, some, or all of the segments) with a user interface contact based on whether characteristics of each of the segments satisfy a drag-retract criteria. In some implementations, association of the selected segments is achieved by implementing a drag-retract dead zone such that movement that occurs during retraction (because such movement is within the drag-retract dead zone) is not recognized as user interface contact motion. Figure 7 An exemplary drag-retract dead zone is shown.

[0252] Figure 333300 is a flow chart illustrating a method 3300 for interpreting movement using dynamic blackout volumes. In some implementations, a device such as electronic device 110 or electronic device 105 performs method 3300. In some implementations, method 3300 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. Method 3300 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, method 3300 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0253] At block 3302 , method 3300 displays an XR environment corresponding to a 3D environment, wherein the XR environment includes a user interface and movement.

[0254] At block 3304, method 3300 determines the occurrence of an event associated with contact with a user interface in the XR environment (e.g., a make contact event), e.g., based on determining that contact with the UI occurs, is expected to occur, or is perceived by the user. This can involve determining when the user penetrates the user interface. This can involve indicating that a direct touch gesture is in effect, that input criteria (e.g., drag and / or swipe criteria) have been met, and that movement is being tracked relative to input to the user interface.

[0255] At block 3306, method 3300 adjusts an interruption volume based on the movement, the interruption volume defining an area of ​​the XR environment where the movement will be associated with user interface contact. Adjusting the interruption volume may involve shifting the interruption volume based on a retraction confidence, where the retraction confidence is based on an alignment between a direction of the movement and a retraction direction. The retraction direction may be a direction from a portion of the user used for interaction (e.g., a hand, a finger, etc.) to a central portion of the user (e.g., a head, a torso, etc.). Adjusting the interruption volume may involve adjusting a slope of the interruption volume based on a penetration depth of the movement. Examples of adjusting the interruption volume are described in Figures 28 to 29 Shown in.

[0256] At block 3308 , the method 3300 determines to discontinue associating the movement with the user interface contact based on the movement crossing a boundary of an interruption volume (eg, determining that an interruption event has occurred).

[0257] In some implementations, trajectory correction is provided. For example, this can involve adjusting the velocity associated with a first time (e.g., correcting the trajectory direction of the current frame) based on a determination that the movement will cross outside the boundaries of the interruption volume at a subsequent time (e.g., the next frame). The velocity associated with a first time can be adjusted based on the velocity at a previous time. Figures 30 to 31 An example of trajectory correction is provided in .

[0258] User interface response based on gaze hold event evaluation

[0259] Various specific implementations disclosed herein include evaluating user interactions to trigger the device, system, and method of user interface responses. In some specific implementations, user interface responses are triggered based on identification to a gaze hold event (i.e., a gaze event similar to gaze that is not associated with a glance behavior). Only a gaze hold event (and therefore not a glance-related behavior in gaze data) (or an acceptable portion of a gaze hold event) can be used to trigger a user interface response. Using a gaze hold event to trigger user interface behavior may be advantageous, for example, because the gaze direction during a gaze hold event is more likely to correspond to the user's perception of what they see and / or intentionally looking at something.

[0260] Using the gaze hold event can be conducive to accurate hover response based on gaze, for example, the user interface can be enabled to highlight the icon when the user intentionally looks at the user interface icon (for example, similar to the mouse hover response on the user interface based on a mouse), and not highlight the icon when the user's gaze happens to move over the icon when the user looks around in the user interface. Similarly, using the gaze hold event can promote accurate input response based on gesture and gaze association. In one example, this involves associating one-hand gestures such as pinching, the gesture of all five fingers on a hand or multi-finger swipe gestures with the user's intentional gaze UI object, without associating such activity with the object that happens to be gazed during the relevant or other unintentional behavior of scanning. In another example, this involves associating multi-hand gestures such as two hands pinching simultaneously or two hands moving away from each other with the user's intentional gaze UI object, without associating such activity with the object that happens to be gazed during the relevant or other unintentional behavior of scanning. In another example, this involves associating head movements such as nodding, shaking the head, or tilting the head with the user's intentional gaze at a UI object, without associating such activity with the object that happened to be gazed at during a glance-related or other unintentional behavior. In some implementations, gaze is associated with one or more of a hand gesture, a head gesture, a torso-based gesture, an arm gesture, a leg gesture, or a whole-body movement, for example, gaze is associated with a combined hand / head gesture. Gaze may additionally or alternatively be associated with input provided via a physical device such as a keyboard, mouse, handheld controller, watch, etc.

[0261] In some implementations, gaze hold events are used to associate non-eye-based user activities such as hand or head gestures with eye-based activities such as a specific user interface component displayed within a view of a user's gaze at a 3D environment. For example, a user's pinching hand gesture may be associated with the user gazing at a specific user interface component such as a button at about the same time as making the pinching hand gesture. These associated behaviors (e.g., pinching and gazing at a button) can then be interpreted as user input, e.g., selecting or otherwise acting on user input of the user interface component. In some implementations, non-eye-based user activities are only associated with certain types of eye-based user activities that may correspond to the user perceiving what they see and / or intentionally looking at something. For example, it may be desirable to associate a user's hand gesture with a gaze hold event corresponding to intentional / perceived user activity. A gaze hold event occurs when the gaze remains on an object while the head is stationary or moving. It may not be desirable to associate a user's hand gesture with a glance eye event, which may occur reflexively rather than based on the user perceiving what they see or intentionally looking at something.

[0262] Gaze data can be checked or interpreted to identify gaze holding events (e.g., non-glance eye events). Non-eye-based user activities can then be associated with an event in those events, rather than being associated with a reflexive glance eye event. For example, during a glance event, a user may move their eyes away from a user interface element they intend to interact with for a brief period. Some specific implementations ensure that non-eye-based activities (e.g., user hand gestures) are not associated with glance events or other gaze events during which a user gaze does not accurately correspond to a user interface or other content that the user intends to interact with.

[0263] In some specific implementations, eye gaze data (e.g., eye velocity data, eye acceleration data, the change of gaze posture, etc.) is used to identify a gaze event subset that only includes gaze hold events and excludes reflexive glance events, blinks, and other eye behaviors that do not correspond to user perception of what they see and / or intentionally look at something. Excluding glance events, blinks, and other eye behaviors that do not correspond to user perception of what they see and / or intentionally look at something can improve the accuracy and / or efficiency of a system that attempts to accurately associate non-eye-based user activities with intentional user gaze (that is, intentional gaze at the user interface component for the purpose of providing user input corresponding to a user interface component). Therefore, in some specific implementations, based on gaze hold events being more likely to correspond to user perception of what they see and / or intentionally look at something than non-gaze hold events, user non-eye-based activities such as hand gestures are only associated with these gaze hold events.

[0264] In some embodiments, a processor performs a method by executing instructions stored on a computer-readable medium. The method obtains gaze motion classification data generated based on sensor data of an eye captured by one or more sensors. The gaze motion classification data distinguishes a gaze period associated with a gaze hold event (e.g., an intentional gaze on a user interface target) from a gaze period associated with a non-gaze hold event (e.g., a gaze shift event, a blink / loss event, etc.). The gaze motion classification data can be provided by a simple gaze motion classifier (e.g., a heuristic algorithm that only evaluates gaze speed, or a more complex algorithm or machine learning model that is not just gaze speed). Using gaze motion classification data can be beneficial for triggering a user interface response only in appropriate circumstances (e.g., based only on the gaze of an intentional gaze on a user interface target and not based on unintentional gaze motion (e.g., a saccade, a blink, etc.). In some embodiments, the gaze classification output (e.g., identifying a gaze hold event) is evaluated to lock gaze during saccades, losses, and rapid gazes and / or stabilize gaze during gaze.

[0265] The method may use gaze classification data generated based on gaze velocities at multiple times. The gaze data may be obtained based on sensor data of the eye captured by a sensor. For example, the gaze data may be based on a signal of live gaze velocity data obtained based on a live image stream of the eye captured by an inward-facing camera of a head-mounted device.

[0266] Gaze hold events may be identified based on gaze velocity. Saccadic gaze events, blinks, and / or other eye events that are unlikely to correspond to the user perceiving what they are seeing and / or intentionally looking at something may be excluded from identified gaze hold events.

[0267] The method detects user activity and triggers a user interface response based on the user activity and gaze movement classification data. The method may trigger the user interface response based on determining that a gaze hold event among the gaze hold events corresponds to a user interface element. For example, based on a gaze hold event having a gaze direction directed toward a specific user interface icon, the method may trigger a hover-type response through the user interface, such as highlighting the icon as a "hover" or "focus" element of the user interface. In some embodiments, the user activity is a gaze in the gaze direction that occurs during the gaze hold event, and the user interface response includes providing an indication of the user's attention to the user interface element based on determining that the gaze hold event corresponds to the user interface element.

[0268] In some implementations, the user activity is a gesture or input device interaction other than gazing, a gaze hold event is associated with the user activity, and a user interface response is triggered based on associating the user activity with the user interface element. For example, based on a gaze hold event directed at a particular user interface icon and the occurrence of a user activity associated with the gaze hold event (e.g., a pinch gesture), the method can trigger a selection-type response through the user interface, e.g., triggering a selection or "click" action on the user interface icon.

[0269] The method can detect that a user activity has occurred, where the activity is different from a gaze hold event, for example, the activity is a non-eye-based eye activity, such as a pinch or other hand gesture. Examples of activities different from a gaze hold event include activities that are distinct from the eyes, including but not limited to single-hand gestures, multiple-hand gestures, head movement, torso movement, arm or leg movement, full-body movement, and / or interaction with other devices.

[0270] The method can associate a gaze hold event with an activity. Thus, in various implementations, a gaze hold event is associated with one or more of the following: a gesture performed by a single hand, a gesture involving one or more fingers, a gesture performed by two hands, a gesture performed by the head, a gesture performed by a hand and a head position / movement performed substantially simultaneously, and / or an input to a device such as a controller, an input device, a wearable device, or a handheld device.

[0271] In some specific implementations, the method determines that gaze keeps (for example, non-glance) event during activity (for example, pinching) / occurs with this activity simultaneously, and based on this determination, gaze is kept event and is associated with activity.Therefore, the pinching that occurs when user's gaze is fixed on the center of button is associated with this gaze and is therefore associated with this button. In some specific implementations, the method determines that gaze keeps (for example, non-glance) event not during activity (for example, pinching) / occurs with this activity simultaneously, and determines whether this activity is effective later stage activity (for example, effective later stage pinching). This can be based on determining whether later stage activity keeps (for example, non-glance) event within the threshold time of previous gaze, and if so, activity is kept event associated with this previous gaze. In some specific implementations, if gaze (for example, non-glance) event not during activity / with this activity simultaneously or before in time threshold, then the method waits to see whether gaze keeps occurring in upcoming time period. In some embodiments, the present invention relates to a method for the present invention that can be used to associate a user interface component and other objects with the user interface component.In some embodiments, the present invention relates to a method for the present invention that can be used to associate a user interface component and other objects with the user interface component.In some embodiments, the present invention relates to a method for the present invention that can be used to associate a user interface component and other objects with the user interface component.In some embodiments, the present invention relates to a method for the present invention that can be used to associate a user interface component and other objects with the user interface component.In some embodiments, the present invention relates to a method for the present invention that can be used to associate a user interface component and other objects with the user interface component.In some embodiments, the present invention relates to a method for the present invention that can be used to associate a user interface component and other objects with the user interface component.In some embodiments, the present invention relates to a method for the present invention that can be used to associate a user interface component and other objects with the user interface component.

[0272] In some specific implementations, user interface response is based on user activity (for example, larger pan), and wherein response ignores / does not use the gaze after the larger pan event to keep event.In one example, user activity is the pan event with the characteristic that exceeds threshold value, and user interface response is based on excluding the potential gaze that occurs during a certain period of time after the pan event to keep event or the gaze that keeps a part for event.In some specific implementations, identification gaze keeps event and comprises: excluding the potential gaze that occurs during a certain period of time after the pan event in speed data to keep event or the gaze that keeps a part for event, wherein exclude potential gaze and keep event based on the following: (a) represents the amplitude of speed change during the pan event; (b) rate of change of speed during the pan event; (c) potential gaze keeps the duration of event; and (d) gaze travel distance.

[0273] Figure 34The user 102 is shown performing a pinch gesture while changing the direction of gaze relative to content presented in views 3410a-c of the XR environment provided by the device 110 (or device 105) of Figure 1. The views 3410a-c include an exemplary user interface 3430 of an application and a depiction 3420 of the table 120. Providing such views may involve determining 3D properties of the physical environment 100 and positioning virtual content, such as the user interface 3430, in a 3D coordinate system corresponding to the physical environment 100.

[0274] exist Figure 34 In the example of FIG, user interface 3430 includes various content items, including a background portion 3435, message bubble portions 3442, 3444, 3446, and a new message input portion 3448 with a button 3450. Message bubble portions 3442, 3444, 3446 may be displayed with a 3D effect on the flat user interface 3430 in the view provided by device 110. User interface 3430 may be the user interface of a text messaging application, as shown in this example. User interface 3430 is simplified for illustrative purposes, and the user interface may actually include any level of complexity, any number of content items, and / or a combination of 2D and / or 3D content. User interface 3430 may be provided by various types of operating systems and / or applications, including but not limited to messaging applications, web browser applications, content viewing applications, content creation and editing applications, or any other application that can display, present, or otherwise use visual and / or audio content.

[0275] In this example, user interface 3430 is presented in a manner that combines a 2D flat portion and a 3D effect to provide functional and aesthetic benefits. In this example, background portion 3435 of user interface 3430 is flat. In this example, background portion 3435 includes all aspects of displayed user interface 3430, except for message bubbles 3442, 3444, 3446 and a new message input portion 3448 with button 3450. Displaying the background portion of the user interface of an operating system or application as a flat surface can provide various advantages. Doing so can provide a user interface that is easy to understand or otherwise easy to use as part of the XR environment to access an application. In some implementations, a flat background portion is used to present multiple user interfaces (e.g., corresponding to multiple different applications) sequentially and / or simultaneously within the XR environment.

[0276] In some implementations, the position and / or orientation of one or more user interfaces are determined to facilitate visibility and / or usage. Within a 3D environment, one or more user interfaces may be in a fixed position and orientation. In such a case, user movement will not affect the position or orientation of the user interface within the 3D environment.

[0277] In other embodiments, one or more user interfaces may be body-locked content, e.g., at a distance and orientation offset relative to a portion of the user's body (e.g., their torso). For example, body-locked content for a user interface may be 2 meters away and 45 degrees to the left of the user's torso's forward vector. If the user's head turns while their torso remains stationary, the body-locked user interface will appear to remain stationary in the 3D environment at 2 meters to the left and 45 degrees to the torso's forward-facing vector. However, if the user does rotate their torso (e.g., by rotating in their chair), the body-locked user interface will follow the torso rotation and reposition within the 3D environment so that it remains 2 meters to the left and 45 degrees to the new forward vector of their torso.

[0278] In other implementations, user interface content is defined at a specific distance from the user, where it remains fixed relative to the user's orientation (e.g., if initially displayed in a cardinal orientation, it will remain in that cardinal orientation regardless of any head or body movement). In this example, the orientation of the body-locked content is not referenced to any part of the user's body. In this different implementation, the body-locked user interface will not reposition itself based on torso rotation. For example, a body-locked user interface may be defined as 2 meters away and, based on the direction the user is currently facing, may initially be displayed north of the user. If the user rotates their torso 180 degrees to face south, the body-locked user interface will remain 2 meters north of the user, with north now directly behind the user.

[0279] The body-locked user interface can also be configured to always remain aligned with the line of gravity or the horizon, so that changes in head and / or body orientation in roll will not cause the body-locked user interface to move within the 3D environment. Translational movements will cause the body-locked content to reposition within the 3D environment to maintain the distance offset.

[0280] Views 3410a-c illustrate a user's gaze 3460 and hand 3470 gestures occurring at consecutive points in time (e.g., view 3410a corresponds to a first moment in time, view 3410b corresponds to a second moment in time after the first moment in time, and view 3410c corresponds to a third moment in time after the second moment in time). In this example, during the time period from the first moment in time to the third moment in time, the user intends to provide user input by gazing at button 3450 (i.e., directing their gaze direction 3460 toward button 3450) while simultaneously making a pinch gesture with hand 3470 to select button 3450. The user should understand that this type of input (e.g., gazing at a user interface object such as button 3450 while making a pinch gesture) will be interpreted as input corresponding to the gazed user interface object.

[0281] However, in attempting to do so, in this example, user 102 experiences an involuntary glance, looking away from button 3450 at the second moment when the pinch occurs. Thus, at the first moment shown in view 3410a, user 102 has not yet pinched and is looking at button 3450. At the second moment shown in view 3410b, user 102 has pinched hands 3470, but an involuntary reflex glance occurs, and thus, gaze 3460 is directed toward depiction 3420 of table 120 rather than button 3450. This gaze direction does not correspond to the user's intention or what the user was perceiving. At the third moment shown in view 3410c, user 102 no longer pinches hands 3470 and the glance has ended, with gaze 3460 returning to the object, e.g., button 3450.

[0282] Some embodiments disclosed herein evaluate user gaze data, such as gaze velocity, to identify types of eye events that should be associated with non-eye-based activities versus types of eye events that should not be associated with non-eye-based activities. Some embodiments attempt to distinguish gaze hold events (i.e., eye gaze events associated with a user intentionally gazing at an object and / or perceiving what they are seeing) from other gaze events (e.g., saccades, blinks, etc.) in which the user is not intentionally gazing at an object and / or perceiving what they are seeing.

[0283] exist Figure 34 In the example, user gaze data, such as gaze velocity data, can be evaluated to determine that there is no gaze hold event at the second moment when the pinch occurs (shown in view 210b), for example, instead the gaze event occurring at the second moment is a saccade gaze event.

[0284] Based on determining that the pinch did not occur during / simultaneously with a gaze hold event, device 110 may attempt to associate the pinch with a previous or future gaze hold event. For example, device 110 may determine that a pinch occurred (at a second moment shown in view 3410b) within a predetermined threshold amount of time after an identified gaze hold event (e.g., occurring at a first moment shown in view 3410a). For example, the threshold may be 1ms, 2ms, 10ms, 50ms, etc. If the first moment (shown in view 3410a) and the second moment (shown in view 3410b) occur within 1ms, 2ms, 10ms, etc. of each other, then the pinch that occurred at the second moment (shown in view 3410b) is associated with the gaze hold event (i.e., the user 102 gazing in the direction of button 3450) at the first moment (shown in view 3410a).

[0285] On the other hand, if the first moment (shown in view 3410a) and the second moment (shown in view 3410b) do not occur within a threshold value (e.g., 1 ms, 2 ms, 10 ms, etc.) of each other, the pinch that occurred at the second moment (shown in view 3410b) is not associated with the gaze hold event that occurred at the first moment (shown in view 3410a). If no previous gaze hold event occurred within the threshold amount of time, the device 110 may wait until new gaze data is received and evaluate such data to determine whether a new gaze event occurred after the pinch that occurred at the second moment (shown in view 3410b). For example, the device 110 may determine that the pinch (at the second moment shown in view 3410b) occurred within a predetermined threshold amount of time before the identified gaze hold event (e.g., occurring at the third moment shown in view 3410c). For example, the threshold value may be 1 ms, 2 ms, 10 ms, 50 ms, etc. If the third moment (shown in view 3410c) and the second moment (shown in view 3410b) occur within 1 ms, 2 ms, 10 ms, etc. of each other, the pinch that occurred at the second moment (shown in view 3410b) is associated with the gaze hold event (i.e., user 102's gaze direction is toward button 3450) at the third moment (shown in view 3410c). The threshold amount of time used to evaluate a previous gaze hold event or to wait for a new gaze hold event may be the same or different from each other.

[0286] If no new gaze hold event occurs within a threshold amount of time, the device 110 may determine that the pinch that occurs at the second moment (shown in view 3410b) should not be associated with any gaze event. In other words, if no valid gaze hold event occurs within a certain time window before and after a given non-eye-based user activity, the device 110 may determine not to associate the activity with any eye-based activity. Non-gaze-based activities (e.g., pinches) may still be interpreted as input, but will not be associated with gaze events / directions. In some implementations, a given input type (e.g., pinches) is interpreted as a first way when associated with an eye-based event and as another way when not associated with an eye-based event. In some implementations, non-eye-based activities such as pinches are not processed as input unless associated with a gaze event. In some implementations, when a non-eye-based activity cannot be associated with a valid gaze hold event / user interface object, the device 110 presents a visual or auditory output requiring the user 102 to clarify or further input.

[0287] Figure 35A 、 Figure 35B and Figure 35C Hand engagement, indirect selection, and hand- and gaze-based indirect gestures are shown.

[0288] exist Figure 35A , illustration 3505a shows a hand engagement user input in which a user provides input by simply lifting / moving a hand 3510. In this example, the hand engagement user input is processed as an input that does not require and / or use any associated gaze or other eye data.

[0289] Such gestures can be recognized by the device using one or more sensors of various types. For example, an image sensor can capture a sequence of images that can be interpreted to identify an object (e.g., a hand), its movement path, configuration (e.g., whether fingers are touching / pinching), etc.

[0290] exist Figure 35B In the example, illustrations 3505b-c illustrate indirect selection. Illustration 3505b shows a user's eye 3515 gazing at user interface object 3525, which in this example is one of two user interface objects 3520, 3525 visible to the user. In illustration 3505c, the user makes a pinch gesture with hand 3510 to identify user interface object 3525. User interface object 3525 is identified based on a pinch associated with the gaze on user interface object 3525 as shown in illustration 3505b. The association of the pinch of illustration 305c with the gaze of illustration 3505b (and its associated user interface object) can be based on various association criteria. For example, a pinch and gaze / UI object can be associated based on the pinch and gaze occurring simultaneously. As another example, a pinch and gaze / UI object can be associated based on the gaze occurring within a predetermined threshold amount of time before or after the pinch.

[0291] exist Figure 35B In the example shown in FIG, user interface object 3525 is graphically changed (e.g., with highlighting, a different color, etc.) to indicate that the object has been selected based on the recognition of indirect selection input based on the user's gaze. In other words, the selected user interface object 3525 is identified as having been selected based on the change in its appearance. In some implementations, a gaze hold event is used to trigger a user interface response similar to a traditional mouse-based hover user interface response. In some implementations, gaze direction alone (i.e., before or without a pinch) is used to trigger a user interface response. A gaze hold event can be identified and used to trigger such a user interface response. For example, a gaze hold event can be identified, a user interface target (e.g., user interface element 3525) toward which the gaze is directed can be identified based on the gaze direction during the gaze hold event, and the user interface can be modified with a response corresponding to the recognition that the user is intentionally looking at the user interface target, e.g., by highlighting user interface element 3525.

[0292] The user's gaze may additionally or alternatively be associated with a gesture, such as Figure 35BAs shown. In some implementations, a gaze hold event is used to trigger a user interface response similar to a traditional mouse-based click / select user interface response. In some implementations, gaze direction is used in conjunction with a user gesture to trigger a user interface response. A gaze hold event can be identified, associated with a user gesture, and a combined gaze-gesture input combination can be used to trigger such a user interface response. For example, a gaze hold event can be identified, a gesture can be associated with the gaze hold event (e.g., based on occurring simultaneously or approximately simultaneously), a user interface target (e.g., user interface element 3525) toward which the gaze is directed can be identified based on the gaze direction during the gaze hold event, and the user interface can be modified with a response corresponding to recognizing that the user was intentionally looking at the user interface target when (or approximately when) the user provided the gesture, e.g., by triggering selection of user interface element 3525 and / or a change in the appearance of user interface element 3525, etc.

[0293] Figure 35B Eye and hand movements can be detected based on one or more sensors. For example, eye movements can be detected based on an inward-facing camera of a head-mounted device (HMD), while hand movements can be detected based on an outward-facing camera on the HMD.

[0294] exist Figure 35C In the example, illustrations 3505d-e illustrate indirect gestures. Illustration 3505d illustrates a gesture in which the user moves hand 3510 to the left while pinching (i.e., maintaining the pinch / touching fingers during the movement). In this example, the pinch shown in illustration 3505c is maintained as shown in illustration 3505d. In other words, when a pinch is made (i.e., the fingers are brought together and touching), the pinch at that moment is associated with the pinch and gaze ( Figure 35B ) is associated with the selected user interface element 3525, and the continued / ongoing pinch in which the finger and thumb remain in contact is then interpreted as additional input, such as Figure 35C 3505d. In illustration 3505d, the hand engagement user input is processed as an input that does not require and / or use any associated gaze or other eye data. The input is simply a movement of the hand 3510 in the left direction.

[0295] Based on the activity, the selected user interface object 3525 is moved. In this example, the direction and distance that the selected user interface object moves are based on the direction and distance of the hand movement. In some implementations, the direction of the user interface object movement is constrained to a direction on a defined 2D plane, for example, a direction on a 2D plane on which user interface elements are displayed, such as on a virtual screen a few feet in front of the user. For example, the direction of the UI object movement can be constrained to a direction that most closely corresponds to the 3D direction of the hand movement. In some implementations, the amount of movement / distance is scaled, for example, 1 inch of hand movement corresponds to 2 inches of UI object movement, 4 inches of UI object movement, 1 foot of UI object movement, etc.

[0296] In illustration 3505e, the user interrupts the pinch made in illustration 3505d and maintained during the movement of illustration 3505d. In illustration 3505e, the hand engagement user input (interrupting the pinch) is processed as an input that does not require and / or use any associated gaze or other eye data. The input is simply the separation of the fingers that were pinched together. In this example, the pinch interruption of illustration 3505e is interpreted as ending the movement of UI object 3525, that is, once the pinch is interrupted, UI object 3525 also stops moving based on the hand movement.

[0297] The gestures shown in Figures 3505d-e can be recognized by the device using one or more sensors of various types. For example, an image sensor can capture a sequence of images that can be interpreted to identify an object (e.g., a hand), its movement path, configuration (e.g., when fingers touch / pinch, when fingers stop touching / pinch), etc.

[0298] Figure 36A 、 Figure 36B 、 Figure 36C and Figure 36D Various anomalies associated with the user's gaze direction relative to user interface elements are shown. Figure 36A , gaze direction changes from gaze point 3605 along path 3607 relative to user interface object 3615 with which the user intends to interact (i.e., pinching while gazing as a pinch plus gaze input). In this example, the user's gaze slews across user interface object 3615 such that a pinch occurs at pinch point 3610, while the user's gaze is outside of user interface object 3615.

[0299] exist Figure 36B, gaze direction changes from gaze point 3605 along path 3607 relative to user interface object 3615 with which the user intends to interact (i.e., pinching while gazing as a pinch plus gaze input). In this example, the user's gaze flickers near target user interface object 3615 so that a pinch occurs at pinch point 3610, while the user's gaze is outside of user interface object 3615.

[0300] exist Figure 36C , gaze direction changes from gaze point 3605 along path 3607 relative to user interface object 3615 with which the user intends to interact (i.e., pinching while gazing as a pinch plus gaze input). In this example, the user's gaze undergoes a saccade such that the pinch occurs at pinch point 3610 while the user is gazing outside of user interface object 3615.

[0301] exist Figure 36D , gaze direction changes from gaze point 3605 along path 3607 relative to user interface object 3615 with which the user intends to interact (i.e., pinching while gazing as a pinch plus gaze input). In this example, a blink / data drop gaze event occurs such that a pinch occurs at pinch point 3610, while the user's gaze direction is unknown.

[0302] Figure 37 Shown is an exemplary interactive tracking process 3700. In this example, at motion classification frame 3710, eye gaze data are classified into the event of different classification types. This may involve processing gaze data (e.g., a series of gaze directions and / or gaze speed values) and determining an algorithm or machine learning model for the gaze events that occur at one or more time points. At frame 3720, posture stabilization and glance removal are applied to gaze data and / or gaze classification. Stability can be adjusted for eye twitches and / or smaller eye movements that do not correspond to intentional / perceived user eye movements. Glance removal may involve identifying eye gaze events corresponding to involuntary / reflex eye glances and removing those events, e.g., changing gaze data to remove gaze data corresponding to gaze events of those types.

[0303] At block 3730 (hit test manager), eye gaze data (e.g., eye gaze hold events identified in stable and glance-removed eye gaze data) is evaluated along with user interface conflict data 3740 to identify eye gazes corresponding to specific user interface elements. For example, a user interface on a virtual 2D surface or within a 3D region may be presented within a view of a 3D environment. The gaze direction of a gaze hold event within the 3D environment may be evaluated relative to user interface elements, e.g., to identify when the gaze direction of a gaze hold event intersects (or approaches) specific user interface elements. For example, this may involve determining that the user is gazing at a specific user interface element at a specific point in time when a gaze hold event occurs.

[0304] At block 3760 (Pinch and Gaze Association), the hand data 3750 is associated with the gaze hold event and associated user interface element identified by the hit test manager 3730. This may involve determining that a gesture occurring at a particular moment or during a particular time period should be associated with a particular gaze hold event and its associated user interface element. As described herein, such association may be based on timing and / or other criteria.

[0305] At block 3770 (Interaction State Manager), the hand data 3750 associated with the gaze hold event and the associated user interface element is used to manage interactions. For example, a user input event can be provided to an application that is providing a user interface so that the application can respond to the user input event, for example, by changing the user interface. The user input event can identify the user interface element with which a given input is associated, for example, identifying that the user has provided a gaze plus pinch input to select element A, that the user has provided a pinch input to move 10 distance units to the left, that the user has released the pinch, etc. The user input is thus recognized and used to trigger an interaction state update.

[0306] Figure 38A graph 3800 illustrating gaze velocity 3805 occurring over time is shown. In this example, during an extended gaze event 3810 between times 3811 and 3816, the user's gaze on a user interface element is interrupted by a brief gaze deviation between times 3812 and 3814. Thus, the user's gaze is directed toward the user interface element between times 3811 and 3812 and between times 3814 and 3816. The user pinches between times 3813 and 3815, with a pinch occurrence 3820 occurring at time 3813 and a pinch interruption (e.g., release of the pinch) occurring at time 3815. The initial portion of the pinch occurs between times 3813 and 3814, during which brief gaze deviations also occur, e.g., during an override and correction or distraction glance event. However, a brief gaze deviation is identified based on the gaze velocity during its occurrence (during the time period between times 3812 and 3814), and only gaze hold events 3804a and 3804b are considered for potential association with pinch occurrence 3820. Thus, in this example, pinch occurrence 3820 is associated with gaze event 3804b at pinch event 3830, i.e., the gaze at time 3815 (and the corresponding gazed user interface item). Thus, pinches that occurred during brief gaze deviations are recovered and associated with the correct / intended gaze direction / user interface element.

[0307] Figure 39 Associating a pinch with a gaze event is shown on a graph 3900 showing gaze velocity 3905 over time. In this example, a pinch 3902a-b occurs off-target during an extended gaze event 3910 between times 3911 and 3917, but is resumed during a gaze hold event. In this example, a gaze event 3906 is detected as occurring during the time period between times 3901 and 3917, and a gaze event 3908 within this gaze hold event 3906 can be associated with a user interface element. For example, between times 3911 and 3914, the user's gaze is relatively fixed but just outside the user interface element; between times 3914 and 3915, the user's gaze is relatively fixed within the user interface element; and between times 3916 and 3917, the user's gaze is relatively fixed and just outside the user interface element. A pinch occurring during a gaze hold event 3906 is associated with a gaze event 3908 for which a corresponding user interface element exists. Therefore, the first pinch 3902a (from time 3912 to time 3913) is associated with the gaze event 3908 (and its corresponding UI element). Similarly, the second pinch 3902b (from time 3916 to time 3917) is also associated with the gaze hold event 3906 (and its corresponding UI element).

[0308] Figure 40Show that kneading 4010 is associated with gaze keeping event rather than glance event.In this example, the position of multiple gaze events 4020a-k place user's gaze is shown relative to user interface element 4015.Based on analyzing gaze data, gaze event 4020a-4020d, 4020g-4020k is determined as gaze keeping event separately.By contrast, based on analyzing gaze data, gaze event 4020e, 4020f is determined as glance moment.In this example, kneading 4010 occurs during glance, and gaze is outside user interface element 4015.But, kneading 4010 is not associated with glance moment.On the contrary, kneading 4010 is associated with last effective gaze target, and in this example, last effective gaze target is the position associated with gaze event 4020d.In this way, unintentional gaze motion (for example, glance, blink etc.) is not considered for association, because they are removed from signal and are not included in effective gaze target.

[0309] Figure 41 4 is a flow chart illustrating a method 4100 for triggering a user interface response based on a gaze hold event. In some implementations, a device such as electronic device 110 performs method 4100. In some implementations, method 4100 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. Method 4100 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, method 4100 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0310] At block 4102, method 4100 obtains gaze data including gaze velocities at a plurality of times, the gaze data being obtained based on sensor data of an eye captured by one or more sensors. For example, the gaze data may be based on a signal of live gaze velocity data, the signal of live gaze velocity data being obtained based on a live image stream of the eye captured by an inward-facing camera of a head-mounted device (HMD).

[0311] At block 4104, method 4100 identifies gaze hold events based on gaze velocity. Identifying gaze hold events may involve motion classification, posture stabilization, and / or blink removal. In some implementations, a gaze velocity signal is used to classify eye / gaze motion. For example, gaze velocity and / or position data may be stabilized to account for eye twitches and micro-eye movements that are not associated with conscious or intentional behavior. In some implementations, an event rejection process is performed to remove gaze events associated with saccades and blinks, as well as other events that are unlikely to be associated with intentional and / or conscious interactions of the user.

[0312] At block 4106, method 4100 triggers a user interface response based on determining that one of the plurality of gaze hold events corresponds to a user interface element. For example, based on a gaze hold event having a gaze direction directed toward a particular user interface icon, the method may trigger a hover-type response through the user interface, e.g., highlighting the icon as a "hover" or "focus" element of the user interface. For another example, based on a gaze hold event directed toward a particular user interface icon and the occurrence of a user activity associated with the gaze hold event (e.g., a pinch gesture), the method may trigger a selection-type response through the user interface, e.g., triggering a selection or "click" action on the user interface icon.

[0313] The method can detect that an activity has occurred, where the activity is different from a gaze hold event. The activity can be a non-eye-based eye activity, such as a pinch or other gesture. Examples of activity different from a gaze hold event include activities that are distinct from the eyes, including but not limited to single-hand gestures, multi-hand gestures, head movement, torso movement, arm or leg movement, full-body movement, and / or interaction with a device.

[0314] One-hand gestures include, but are not limited to, a user forming a shape / configuration and / or performing a particular motion with a single hand, such as by pinching (e.g., touching the index finger or other fingers to the thumb), grasping (e.g., forming the hand into a ball), pointing (e.g., by extending one or more fingers in a particular direction), or performing a multi-finger gesture. One example of a gesture involves a user pinching, where the pinching (e.g., touching the fingers to the thumb and then releasing) provides input (e.g., selection of whatever the user is gazing at). Another example of a gesture involves a user pinching (e.g., to initiate detection of a gesture) and then moving or changing the hand while maintaining the pinch, for example, pinching and then moving the hand to provide a directional input movement based on the direction of the hand's movement.

[0315] One example of a multi-finger gesture is when a user spreads all of their fingers (e.g., configuring their hand so that no finger touches any other finger). Another example of a multi-finger gesture is a multi-finger swipe (e.g., extending two or more fingers and moving those fingers along a specific path or across a specific real or virtual surface). Another example of a multi-finger gesture is when the fingers are held approximately flat, with all fingers touching adjacent fingers. Another example of a multi-finger gesture is when two fingers are extended in a peace sign configuration. Another example of a multi-finger gesture is when all fingers are extended straight out from the palm and then bent at their respective knuckles. Another example of a multi-finger gesture is when the thumb touches two or more of the fingertips in a specific order, e.g., first touching the index finger and then the pinky finger. Another example of a multi-finger gesture is when the fingers are held in a specific configuration, e.g., with the index finger touching the middle finger, the middle finger not touching the ring finger, and the ring finger touching the pinky finger, while the entire hand moves along a specific path (e.g., up and down).

[0316] Multi-hand gestures include, but are not limited to, a user forming a shape / configuration and / or performing a specific motion with both hands simultaneously or within a threshold amount of time of each other (e.g., within a 2-second time window). One example of a multi-hand gesture involves a user pinching both hands together, where the pinching (e.g., touching fingers to thumbs within a threshold amount of time and then releasing both hands) provides input (e.g., a specific interaction with whatever the user is gazing at). Another example of a multi-hand gesture involves a user pinching both hands together within a threshold amount of time of each other (e.g., to initiate detection of the gesture), followed by moving or changing one or both hands while maintaining the pinch, for example, (a) pinching both hands together and then moving the hands toward or away from each other to provide a zoom in or zoom out input, (b) pinching both hands together and then moving the hands together simultaneously left, right, up, down, etc. to provide a translation input in the direction of the movement, or (c) pinching both hands together and moving the hands while maintaining a distance between the hands while changing their relative positioning to provide a rotational input based on the change (e.g., as if holding a string between the hands and rotating the string to provide a corresponding rotational input to a user interface element).

[0317] A multi-hand gesture may involve each hand performing a gesture, such as by pinching (e.g., touching the index finger or other fingers to the thumb), grasping (e.g., forming the hand into a ball), pointing (e.g., by extending one or more fingers in a particular direction), or performing a multi-finger gesture. In one example, a multi-hand gesture is provided (or initiated) by pinching both hands simultaneously (e.g., within a threshold time interval of each other). In one example, a combined (e.g., multi-hand) gesture is based on the timing between two initiating actions (e.g., pinches performed by each hand) and / or hand proximity to each other.

[0318] A head gesture may involve movement of the head about a degree of freedom (eg, translation, rotation, etc.). Head movement may involve, but is not limited to, nodding, shaking, or tilting the head.

[0319] The user activity to be associated with the direction of gaze may involve user input provided via a device (e.g., a device separate from the HMD or other device that senses the direction of the user's gaze. Such a device may be an input device such as a keyboard, a mouse, a VR controller, a ring, a wearable device such as a watch, a handheld device such as a phone, a tablet, or a laptop, or any other type of device capable of interaction or user input.

[0320] User activities can involve the user using their hands to interact with a controller or other input device, pressing a hotkey, nodding, turning their torso, making facial expressions, jumping, sitting down, or any other activity performed by the user that is distinct from the user's eye gaze. Activities can be detected based on sensor data (e.g., from an outward-facing camera) or based on input device data. Activities can be static, e.g., the user holding their hands steady in a particular configuration, or non-static, e.g., the user performing a particular motion, such as moving their hands while maintaining a pinching hand configuration.

[0321] The method can associate a gaze hold event with an activity. Thus, in various implementations, a gaze hold event is associated with one or more of the following: a gesture made with a single hand, a gesture involving one or more fingers, a gesture made with both hands, a gesture made with the head, a gesture made with a hand and a head position / movement made substantially simultaneously, and / or an input to a device such as a controller, an input device, a wearable device, or a handheld device.

[0322] Associating a gaze hold event with an activity may be based on determining that the activity occurred during the gaze hold event. Associating a gaze hold event with an activity may involve determining that the activity did not occur during any gaze hold event and determining that the activity occurred within a threshold time after the gaze hold event. Associating a gaze hold event with an activity may involve determining that the activity did not occur during any gaze hold event, determining that the activity did not occur within a threshold time after any gaze hold event, and determining that the gaze hold event occurred within a threshold time after the activity.

[0323] In some implementations, method 4100 associates a gaze hold event with another user activity (e.g., a pinch) during presentation of content on a device, such as an HMD. In such implementations, a gaze hold event may be associated with a user gazing at a portion of the content being presented, and the association may therefore associate other user activities (e.g., a pinch) with that portion of the content. In some implementations, content is provided within a view of a 3D environment, such as an XR environment. In some implementations, the view includes only virtual content. In some implementations, the view includes mixed reality or augmented reality content. In some implementations, at least a portion of the 3D environment depicted in the view corresponds to a physical environment proximate to the device, for example, via pass-through video or via a see-through portion of the device. In some implementations, no 3D environment is presented. For example, a user's gaze hold event may be associated with input received when a user gazes and provides input to an activity on a device, such as a smartphone or tablet computer (i.e., a device that does not present 3D content or does not use a stereoscopic display to display content at different depths).

[0324] In some embodiments, the method of the present invention can be used to determine the time series of the gaze of the eye of the eye and the time series of the eye.In some embodiments, the method of the present invention can be used to determine the time series of the gaze of the eye of the eye and the time series of the eye.In some embodiments, the method of the present invention can be used to determine the time series of the gaze of the eye of the eye and the time series of the eye.In some embodiments, the method of the present invention can be used to determine the time series of the gaze of the eye of the eye and the time series of the eye.In some embodiments, the method of the present invention can be used to determine the time series of the gaze of the eye of the eye and the time series of the eye.In some embodiments, the method of the present invention can be used to determine the time series of the gaze of the eye of the eye and the time series of the eye.In some embodiments, the method of the present invention can be used to determine the time series of the gaze of the eye of the eye and the time series of the eye. Furthermore, in some additional or alternative implementations, gaze hold events based on gaze progression identification exclude potential gaze hold events in eye tracking data that occur during a certain period of time after a saccade event, where the saccade event has an eye travel distance greater than a threshold distance.

[0325] Gaze hold events or portions thereof may be excluded from consideration for providing user interface responses based on various criteria. In some implementations, this involves excluding gaze holds (e.g., a portion of a gaze hold event) within the period when a gaze hold begins (and accepting the remainder), even when the gaze hold event may take longer than the exclusion period. A user may glance and land on a target next to their intended target, and then slowly drift from there to the intended target. In this case, the time from landing on the adjacent target and drifting on is excluded, but the portion of the gaze hold event that occurs after landing on the intended target is accepted.

[0326] In some embodiments, potential gaze hold events are excluded at least if they occur during a certain time period after a saccade event, where potential gaze hold events are excluded based on: (a) the magnitude of the change in velocity during the saccade event, (b) the rate of change of velocity during the saccade event, (c) the duration of the potential gaze hold event, and / or (d) the distance the eyes traveled during the saccade event.

[0327] In some implementations, potential gaze hold event is excluded based on that the potential gaze hold event occurs between two glance events with one or more specific characteristics such as those mentioned above. For example, in the case where there is a larger glance, a short intermediate gaze hold event and then another larger glance, the intermediate gaze hold event can be excluded.

[0328] In some specific implementations, the less pan after one or more larger pans that may be mistakenly classified as gaze keeps event based on determining that it is to have the pan event of one or more specific characteristics such as above-mentioned those and be correctly characterized (that is, as less pan rather than gaze keeps event).Similarly, gaze classifier can classify the gaze data associated with continuous pans (and therefore the second portion of pan is classified as gaze keeps event) by mistakenly identifying gap.This misclassification can be based on determining that it is to have the pan event of one or more specific characteristics such as above-mentioned those and be correctly characterized (for example, as pan event rather than gaze keeps event).

[0329] Excluding potential gaze events in such cases may be beneficial, because when the user carries out larger eye movement (for example, larger amplitude pan), eyes may not arrive at expected gaze target as quickly as in other cases.When eyes carry out larger eye movement, it can not accurately fall on the place expected by the user usually, for example, fall on expected user interface target.Usually, eyes naturally fall on roughly area surrounding (rather than accurately falling on this area), and then move and adjust to the precise position of expected user interface element after subsequent gaze maintenance event.System can exclude one or more initial gaze maintenance events (for example, due to speed, eyes travel distance, with the time proximity of significant gaze deviation event etc.) after significant gaze deviation event and gaze maintenance event later is interpreted as suitable gaze maintenance event for identifying gaze direction in the user interface response of triggering.Therefore, excluding the gaze maintenance event that occurs soon after such larger eye movement can help to ensure that gaze maintenance event is only used for triggering user interface response when gaze maintenance event may have the gaze direction corresponding to expected gaze target.

[0330] Fuzzy Hit Test

[0331] Various embodiments disclosed herein include devices, systems, and methods for associating user activities with elements within a 3D environment. Some embodiments are performed in a manner that takes into account inaccuracies (i.e., ambiguities) in sensor-based user activity detection, such as inaccuracies in gaze tracking based on sensor data or inaccuracies in hand / joint positions based on sensor data. Some embodiments use sampling techniques to associate user activities in a 3D environment with appropriate elements, such as a portion of a user interface (UI) positioned within the 3D environment. For example, sampling techniques can be used to identify sample positions within the 3D environment (e.g., sample positions around a gaze direction) to evaluate them and ultimately select from them to associate with user activities, such as associating a particular UI button with a gaze direction. In one example, the user activity is a gaze direction, and a ray pattern around the gaze direction is generated to identify candidate UI targets, from which UI elements are selected to be associated with the gaze direction.

[0332] In some embodiments, the processor performs the method by executing instructions stored on a computer-readable medium of an electronic device. The method receives data corresponding to user activity in a 3D coordinate system. The data can provide a directional representation of the user activity (e.g., a ray in a 3D coordinate system). For example, the method can identify a gaze direction associated with the user's gaze at one or more points in time (e.g., a gaze direction associated with an indirect user interaction). In another example, the method can identify the user's hand position or motion and determine a ray from a viewpoint (e.g., an eye position) to the hand or a ray from the viewpoint to a position where the hand intersects the virtual UI (e.g., an eye-to-touch point ray synthesized on the first frame of a direct user interaction).

[0333] The method generates a plurality of sample positions in a 3D coordinate system based on data corresponding to user activity. In some embodiments, given a gaze direction, ray, or point in a 3D coordinate system identified based on user activity, the method can identify a plurality of nearby rays or points in the 3D coordinate system. In one example, the method generates a pattern of 10 to 30 rays in a pattern. The pattern may include rays or points that may be spaced relative to each other to meet predetermined criteria, such as being spaced within the pattern to ensure that UI elements of a certain size and / or shape within the pattern boundary will intersect with at least one of the rays or points. The pattern may change over time, such as from frame to frame. For example, the pattern for each frame may include randomly positioned rays / points (e.g., gaze rays or gaze ray intersection points) generated based on user activity.

[0334] The method identifies a UI target within a 3D coordinate system based on multiple sample positions. In some implementations, the UI target includes a collision volume corresponding to a 2D or 3D element (e.g., corresponding to the 3D position / bounds of a 3D virtual object and / or a 2D UI element of an application defined by a layer tree, etc.). In some implementations, a collision volume corresponding to a UI hierarchy is identified, and then the UI hierarchy is examined to identify which UI element of the UI hierarchy is at the top at the sample position (e.g., the nearest, opaque, unobstructed element).

[0335] The method selects a UI target from among the identified UI targets to be associated with the user activity based on a selection criterion. In some implementations, for each of the identified UI targets, the method calculates a nearest opaque point and a distance (e.g., angular distance) to the user activity location (e.g., to the gaze direction location, etc.), and then classifies and prioritizes the UI targets according to a policy. This may involve, when multiple UI targets are within a threshold distance (e.g., within a 1° angular distance) of the user activity location, the method prioritizing the UI targets according to the policy by ranking the UI targets based on type (e.g., type of effects enabled for different element types), UI element layering, nesting, and / or geometry of the UI targets, proximity to the user activity location (e.g., gaze direction location), and / or based on hysteresis logic that prioritizes previous targets and / or avoids flickering.

[0336] Various embodiments disclosed herein include devices, systems, and methods for associating user activities with elements within a 3D environment. Some embodiments provide output to an application in a manner that facilitates the application's identification of user activities (e.g., based on the association of user activities with elements), such as providing output based on 3D user activities so that the application can use the output to identify the user activities as being associated with appropriate / intended elements using existing touchscreen / 2D input. In some embodiments, this can enable applications configured for a 2D environment (e.g., mobile device applications) to execute within a 3D environment without requiring the application to change its own 2D input recognition process or otherwise account for ambiguities in the underlying 3D user activities, such as inaccuracies in sensor data tracking gaze (indirectly) or hand / joint position (directly).

[0337] In some specific implementations, the processor performs the method by executing instructions stored on a computer-readable medium of an electronic device. The method receives data corresponding to user activity in a 3D coordinate system. The data can provide a directional representation of the user activity (e.g., a ray in a 3D coordinate system). For example, the method can identify a gaze direction associated with the user's gaze at one or more time points (e.g., a gaze direction associated with an indirect user interaction). In another example, the method can identify the user's hand position or motion and determine a ray from a viewpoint (e.g., an eye position) to the hand or a ray from the viewpoint to a position where the hand intersects the virtual UI (e.g., an eye-to-touch point ray synthesized on the first frame of direct user interaction).

[0338] The method receives data corresponding to the positioning of a UI element of an application within a 3D coordinate system (e.g., a geometry collision world). The data corresponding to the positioning of the UI element can be based at least in part on data provided by the application (e.g., the position / shape of a 2D element intended for a 2D window area). In some implementations, the application can provide a hierarchical tree for positioning UI elements on, for example, a 2D area of ​​a 3D coordinate system. Such information can be provided to a simulation process that positions the application element in 3D space, for example, by defining a 3D position in 3D space of one or more collision bodies (e.g., each collision body having a rectangular window area) for the application element.

[0339] The method identifies one or more UI targets within a 3D coordinate system based on data corresponding to a user activity and data corresponding to positioning of UI elements of an application within the 3D coordinate system. The method selects a UI target from the identified UI targets to be associated with the user activity based on selection criteria. Based on selecting the UI target to be associated with the user activity, the method identifies a point within a 2D region to the application so that the application can associate an action (e.g., select / hit / hover / etc.) with the UI target.

[0340] Figure 4242. The use of an exemplary input support framework 4240 is shown to generate interaction data based on hand data 4210, gaze data 4220, and UI target data 4230 to produce interaction data 4250, which can be provided to one or more applications and / or used by system processes to provide a desired user experience. In some implementations, the input support process 4240 is configured to understand the intent of the user interaction, generate input signals and events to create a reliable and consistent user experience across multiple applications, detect out-of-process input, and responsibly route it through the system. The input support process 4240 can arbitrate which application, process, and / or UI element should receive user input, for example, based on identifying which application or UI element is the intended target of the user activity. The input support process 4240 can keep sensitive user data (e.g., gaze, hand / body registration data, etc.) private; only abstracted or high-level information is shared with applications.

[0341] The input support framework 4240 can obtain hand data 4210, gaze data 4220, and UI target data 4230 and determine the user interaction state. In some implementations, it is in a user environment where multiple input modalities are available to the user (e.g., where the user can Figure 2 Direct interaction as shown in Figure 3 For example, the input support process may determine that the user's right hand is performing an intentional pinch and gaze interaction with the user interface element, the left hand is directly tapping the user interface element, or the left hand is shaking and therefore idle / not doing anything related to the user interface.

[0342] Based on the determined user interaction intent, the input support framework 4240 can generate interaction data 4250 (e.g., including interaction gestures, manipulator gestures, and / or interaction states). The input support framework 4240 can generate input signals and events that applications can consume without requiring custom or in-process 3D input recognition algorithms. In some implementations, the input support framework 4240 provides interaction data 4250 in a format that applications can consume as touch events on a touchscreen or as trackpad taps with a 2D cursor at specific locations. Doing so can enable the same application (with little or no additional input recognition processing) to interpret interactions across different environments, including new environments for which the application was not originally created and / or that use new and different input modalities. Furthermore, application responses to input can be more reliable and consistent across applications in a given environment and across different environments, for example, enabling consistent UI responses for 2D interactions with applications on tablets, mobile devices, laptops, etc., as well as for 3D interactions with applications on HMDs and / or other 3D / XR devices.

[0343] The input support framework can also manage user activity data so that different applications are unaware of user activity relative to other applications. For example, one application will not receive user activity information while the user enters a password into another application. This can involve the input support framework 4240 accurately identifying which application the user's activity corresponds to and then routing interaction data 4250 only to the correct application. The input support framework 4240 can use details about the UIs of multiple possible target applications to disambiguate input.

[0344] Fuzzy hit testing process example

[0345] Some implementations associate user activities with elements within a 3D environment. Some implementations do this in a manner that accounts for inaccuracies (i.e., ambiguities) in sensor-based detection of user activities (e.g., gaze tracking based on sensor data or inaccuracies in hand / joint position based on sensor data). Associating user activities with elements may involve "hit testing," which generally refers to identifying user activities corresponding to possible inputs and determining where and how to route information about the user activities to corresponding UI or other elements or otherwise routed into the system. Some implementations use sampling techniques to perform hit testing to associate user activities in a 3D environment with appropriate elements (e.g., portions of a UI) positioned within the 3D environment. For example, sampling techniques can be used to identify sample locations within the 3D environment (e.g., sample locations around a gaze direction / gaze intersection) to evaluate and ultimately select to associate with the user activity, such as associating a particular UI button with a gaze direction. In one example, the user activity is a gaze direction, and a ray pattern around the gaze direction is generated to identify candidate UI targets, from which a UI element is selected to be associated with the gaze direction.

[0346] In a 2D user interface context (e.g., on mobile and laptop devices), hit testing can begin with a touch point (e.g., on a touch screen), and hit testing can determine to which application or process the touch / hit should be provided / associated. In such a context, the application can perform some or all of such hit testing itself (e.g., using an in-application process specified by the application's code or functions embedded in the application's executable code (e.g., from an application development library) or otherwise used by the application, and / or using custom logic developed by the application developer). In such a process, the application itself can effectively perform in-depth hit testing, such as determining which element within the application's user interface should receive / respond to a given touch screen tap or other such user activity.

[0347] In a 2D context, hit testing may involve evaluating a hierarchical UI structure (e.g., a tree structure) that uses a hierarchical structure to define the positions and characteristics of UI elements. Hit testing may involve starting with a high-level element of the UI structure (e.g., starting with the main window) and recursively moving down the hierarchy / tree (e.g., if a child passes the hit test, the window will ask its children for a given touch location, the child will ask its children, and so on, recursively down the hierarchy / tree). The hit test pass logic may ask whether this point is within the bounds / boundaries of each hierarchical element. The hit test logic may provide a visible element area and a hittable area associated with the element, where the hittable area has a different size or shape than the visible element. In one example, hit test logic is utilized so that a small point-like element will have a hittable area corresponding to a large square surrounding the point-like element, such that element hit testing involves identifying that the hit point is within this larger area.

[0348] One area of ​​problems overcome by aspects of the present disclosure is enabling hit testing logic used by 2D applications for 2D interactions on a touch screen via a mouse, etc. for mobile devices, laptops, etc. to work in the context of gaze and / or other 3D user activities. Gaze and / or other 3D activity tracking may not have the same accuracy and / or characteristics as touch screen and / or other 2D user activities. Additionally, aspects of the present disclosure may provide hit testing in a manner that protects user information (e.g., associating user activities with applications and other content), such as by not providing applications with unrestricted access to user gaze direction information, hand size / shape / activity information, etc.

[0349] A touch screen can identify user activity (e.g., identifying a user touching the screen at a particular x, y location) relatively accurately (e.g., to within 100 microns). In contrast, tracking user gaze, hand position / configuration / motion, and / or other 3D user activity can be significantly less accurate. Some embodiments disclosed herein provide hit testing and / or other processes that account for the possibility of this greater "fuzziness" when tracking 3D user activity. For example, a device can track a user's gaze with an accuracy of plus or minus 1 degree. Thus, if a user is looking at a button (e.g., at the edge of a button), the detected gaze ray may appear to be completely outside the button. Some embodiments evaluate such user activity and determine to associate a gaze direction with a button despite the gaze being outside the button to account for the inaccuracies of the system.

[0350] Some implementations perform such determinations outside of the application process, such as using a system-level process to determine which applications, UI elements within an application, and / or non-application elements the user activity will be associated with. Performing some or all aspects of fuzzy hit testing or other such processes outside of an application's process (e.g., via a system process) can provide various advantages. This can extend the capabilities of applications (e.g., existing applications or applications otherwise configured for another / non-3D platform) that are not configured for 3D user activity and / or specific user activity (e.g., gaze and / or gesture-based input). This can protect user privacy, for example, by abstracting the user activity data provided to the application after performing fuzzy hit testing, so that information about the user activity is limited (e.g., avoiding providing unlimited gaze / hand data). This can take into account user activity that can be associated with one of multiple different applications (due to fuzziness). This can ensure that fuzzy hit testing and related processes are executed consistently across multiple different applications. Additionally, this can make it easier to upgrade or change fuzzy hit testing and related processes, such as without having to reconfigure individual applications with sensing hardware, hit testing features, or other aspects of the system that change over time. This can provide a single phase for fuzz hit testing (eg, at the operating system level) that can be more efficient, effective, and / or more accurate than performing at multiple times and / or using multiple phases.

[0351] Making early, accurate decisions at the operating system level and using application-specific information can unify fuzzy hit testing for optimal system performance. For example, using a central layer with detailed sensor data (e.g., low-level gaze / pupil data and hand data) and detailed application user interface data (e.g., information about specified element positions, sizes, properties, transparency, remote effects, etc. from the application's UI hierarchy) can achieve a robust, accurate, and efficient system. In some embodiments, the system-level fuzzy hit testing information has enough information about the application's UI elements that it can determine (e.g., without burdening the system's rendering / display components) which element of the UI is "on top" (e.g., as seen by the user) for a given sample position / sample ray direction. Without such information, figuring out the top UI element can be complex and error-prone, for example, rounded transparent corners of UI elements, transparent holes within a given object, action graphics defined as cubes with transparent areas surrounding the actual shape of the graphic, etc. can affect accuracy. Similarly, UI elements can be clipped and / or prioritized in a way that hinders accurate association of UI elements with user activities without enough information. Some implementations provide sufficiently detailed information to system processes to perform accurate unified fuzzy hit testing using detailed sensor data while accounting for transparency, clipping, prioritization, and / or while protecting user privacy.

[0352] Furthermore, such a system process can provide information about user activity (e.g., based on its own blur hit testing) in a manner such that the user activity will be associated with the appropriate UI element (e.g., based on the application's separate association / blur hit testing process). For example, an application may provide a 2D user interface element positioned in a 3D world, and the system-level process can correct for inaccuracies in 3D user activity sensing by determining that the user activity should be associated with a particular element of the application's UI and provide input data such that the application will recognize input to that element, e.g., provide a 2D interaction point within the bounds of that element.

[0353] Some implementations address issues arising from relatively low accuracy of gaze tracking or other 3D user activity tracking processes, such as eye tracking algorithms having a first-order error such that the gaze hit location may not always be where the user is gazing and can result in gaze aiming failures where the gaze point is slightly off / out of target. Some implementations provide a system or OS-level process for system-wide hit testing "tolerance" or "fuzziness" for aiming UI elements via gaze or other 3D user activity.

[0354] Figures 43A to 43BAn exemplary hit detection correction is shown in the context of a user interface element 4300 provided / specified by an application. In this example, a user gaze direction 4305a is determined based on sensor data. The detected gaze direction 4305a intersects the user interface at a point 4310a outside of the user interface element 4300. A system-level fuzzy hit detection process is performed and a determination is made that a gaze should be associated with the user interface element 4300, e.g., as if the user had gazed in the gaze direction 4305b at a point 4310b within the user interface element 4300. The system-level detection process can provide an output to the application so that the application will recognize input to the UI element 4300, e.g., by identifying the point 4310b as an interaction point with the application.

[0355] In some specific implementations, such as Figures 43A to 43B As shown, the fuzzy hit test process is configured to "snap" (i.e., reposition) points associated with 3D user activity to the bounds of nearby UI or other elements, enabling applications to correctly interpret 3D user activity without requiring configuration to address potential inaccuracies in sensor data or processes used to detect 3D user activity. Furthermore, a system-level process can perform such fuzzy hit testing to select an appropriate element from among elements provided by potentially multiple applications.

[0356] Figures 44A to 44B 2D elements and 3D elements that can be positioned within a 3D environment are shown. In this example, such elements are tracked by a simulation process / engine that positions collision volumes within a 3D coordinate system. Such collision volumes can be used to describe or specify the 2D and 3D geometry of the XR environment. In some implementations, the 3D geometry can include collision volumes that contain UI features specified via multiple layers of complex data structures (e.g., using a hierarchical or tree structure). In some implementations, a UI hierarchy that encodes the appearance and characteristics of the UI (e.g., specifying an application that renders pixel images, vector graphics, and / or textures, and / or provides animations to provide a desired user experience) can be used. Core A UI (Content Encoding) structure is used to specify the contents of a collision volume to an application. A hierarchical layer structure can be used to specify the appearance of, for example, a window, where buttons are within a window, text is within a window, and so on. 3D elements (e.g., a 3D model or a cube) can be specified without a hierarchical or other UI-specific structure.

[0357] exist Figure 44A and Figure 44BIn the example shown in FIG4 , a 3D cube 4440 and a 3D sphere 4440 are positioned within corresponding collision volumes within a 3D coordinate system. A user interface window 4432 having elements (e.g., element 4435) specified by a hierarchical UI definition / animation structure is also provided. The user interface window (and its elements) are positioned within the collision volume 4430 within the 3D coordinate system. For example, the hierarchical UI definition may specify that window 4432 has a base layer, which has a layer for a disc, which has a layer for a button, and the button may have a layer for text and button properties. The hierarchical structure of the UI can be simple or complex, small or large, dense or sparse. The hierarchical structure may include layers that specify remote effects for UI elements. Remote effects can be implemented as special layers with behaviors / actions associated with specific user activities (e.g., hovering when gazed at, etc.). Applications can use remote effects to convey effects intended for specific UI elements to the input recognition process, such as glowing when gazed at, expanding when gazed at, etc.

[0358] In some implementations, the hierarchical layers provide a 2D arrangement of layers displayed on a plane of a 3D coordinate system (e.g., in a 2D area). In some implementations, one or more of such 2D layers are implemented in a distinctive manner (e.g., slightly protruding from the rest of the layers). Such prominence can be considered when displaying content, determining interactions with the content, and / or prioritizing interactions with portions of the content.

[0359] In some implementations, the application framework controls element sizing. Controlling sizing may involve taking into account areas around UI elements that act as visual padding, such as providing an area of ​​a minimum size for interacting with each element. This may involve ensuring that elements include space around visual features to achieve a minimum element size, such as blur hit testing may require a minimum threshold (e.g., 15 points, 18 points, 21 points, etc.). This may involve controlling visual space to be consistent between elements in a layout to ensure fair hit testing between elements, such as not providing some elements with significantly more space / padding than others. This may involve making control sizes static for each element, such as not adapting hit areas to more compact layouts, gaze entry / exit, viewing angle, etc.

[0360] Some implementations provide an input support process that provides system-wide fuzzy hit testing. This can involve providing fuzzy hit testing in a way that supplements hit box sizing by applying additional viewpoint tolerance and prioritization logic. This can involve using a process or algorithm that assesses which visual targets are close to the user's active location (e.g., close to gaze) and snapping that location to the desired target. This can enable dynamic hit areas (i.e., based on layout, viewpoint, gaze entry / exit, etc.) and prioritization of UI elements across contexts.

[0361] Figure 45 Exemplary control areas 4510a-e and fuzzy hit test areas 4520a-e around elements of a user interface are shown. Figure 45 Reasonable visual spacing between elements 4510a-e is shown. The top row buttons (i.e., UI elements 4510a-d) have some spacing built into the buttons, such as the button label / icon and the space around the displayed button edges. Therefore, the control areas 4510a-e correspond to the edges of the buttons. On the other hand, the sliders (i.e., UI element 4510a) have no built-in spacing, such as no space around the slider bar. Therefore, the control area 4510e has increased spacing to achieve the minimum size requirement. Within a single UI or across multiple elements of different UIs, approximately equal spacing can be provided when setting the element size. The control area size of the UI elements is also static, i.e., the control area 4510a-e is not adjusted or changed unless the corresponding element itself changes size.

[0362] The system can apply fuzzy hit testing to Figure 45 This may involve (e.g., algorithmically) defining a control region, detecting (e.g., algorithmically) the UI element closest to the user interaction location (e.g., gaze point), and snapping (e.g., algorithmically) the interaction point to the nearest edge of the UI element. A disambiguation process or logic may be applied to determine which UI element (to which the user interaction location will be snapped) is between the controls. Such a process may determine the UI element closest to the user interaction location and / or consider other criteria. This is in Figure 45 45. The user interaction location within blurry hit test area 4520a-e is shown in FIG. 45, i.e., the user interaction location within blurry hit test area 4520a is associated with control area 4510a, the user interaction location within blurry hit test area 4520b is associated with control area 4510b, the user interaction location within blurry hit test area 4520c is associated with control area 4510c, the user interaction location within blurry hit test area 4520d is associated with control area 4510d, and the user interaction location within blurry hit test area 4520e is associated with control area 4510e. Blurry hit test areas 4520a-e are smaller between controls 4510a-e to avoid overlap, for example, by dividing the distance between adjacent controls. Note that blurry hit testing in 3D can be applied based on angular size requirements (e.g., based on a 1-degree gaze angle imprecision). For example, the size of blurry hit test areas 4520a-e can increase or decrease (while avoiding overlap) as the user moves closer to or further away from the user interface or looks directly at the user interface or from a significantly larger angle.

[0363] Figures 46A to 46CExemplary control areas and blur hit test areas around elements of a user interface are shown. Figure 46A Exemplary control areas 4610a-c and fuzzy hit test areas 4620a-c around elements of a user interface are shown. Figure 46B Exemplary control areas 4630a-d and fuzzy hit test areas 4640a-d around elements of the user interface are shown. Figure 46C Exemplary control areas 4650a-c and fuzzy hit test areas 4640a-d around elements of the user interface are shown.

[0364] Figure 47 Determining to associate a user activity with a UI element, such as an exemplary blur hit test process, is illustrated. In this example, gaze direction 4705 is detected and used to identify UI location 4710, for example, based on determining that gaze direction 4705 intersects the UI at UI location 4710 within a 3D coordinate system. In this example, the UI element has a control area 4740 and a blur hit test area 4750 (e.g., which may be based on a minimum angular distance from the nearest edge of control area 4740 or other criteria or strategy). Based on the UI location being within blur hit test area 4750 (e.g., distance 4730 is less than an angular distance threshold), the process determines to associate the user activity with the user interface element. This may involve treating the gaze as if it intersects the UI at location 4720. The location on the UI associated with the user activity may be the nearest location on the UI element. This location may be the nearest opaque location on the UI element. This location may be the nearest location on the UI element that is opaque and not obscured by, for example, another UI element or virtual object. Location 4720 may be provided to the owner (eg, the application providing the UI element) so that the application can respond appropriately to user activity.

[0365] Determining whether to associate a user activity with a UI element and / or which UI element to associate the user activity with (e.g., fuzzy hit testing) can involve various processes. In some implementations, this involves (1) discovering UI targets, (2) finding the closest point and distance to the user activity location, and (3) classifying and / or prioritizing the targets according to a policy.

[0366] The first step, discovering UI targets, may involve finding all UI targets in an area around the user's active location. For example, the detected gaze direction may be used to determine the user interface location (e.g., the location where the gaze direction intersects the UI). Additional / sample UI locations around the user interface location (e.g., around the gaze direction or around the user's active location identified in other ways) may be determined (e.g., by generating a scattered ray pattern configured to detect all UI targets having at least a minimum size (e.g., at least 1° high) around the UI location). The additional / sample UI locations around the UI location may then be used to identify visible UI targets, for example, by taking into account transparency, focus level, rendering order, etc.

[0367] The second step, finding the nearest point and distance to the user activity location, may involve identifying the nearest point within each identified visible UI target. For example, for each discovered UI target, this may involve calculating the nearest opaque (e.g., non-transparent) point. This may involve identifying the distance (e.g., angular distance) between the nearest opaque point and the user interface location associated with the user activity (e.g., calculating the angular distance based on the viewpoint location).

[0368] The third step, classifying and / or prioritizing targets according to the policy, may involve determining if multiple UI targets are identified. When multiple targets are found (e.g., within a 1° gaze range), the process may select which of the UI targets to associate with the user activity based on the policy. The process / policy may rank targets by type (e.g., prioritizing remote effects, such as hovering over other layers / geometric type elements). For targets of the same type, the process / policy may prioritize nested targets and then prioritize UI targets with the closest / smallest distance (e.g., angle) to the user activity location (e.g., gaze location). The process / policy may apply hysteresis logic to prioritize previously identified / associated UI targets, which may avoid flickering, as described with respect to 13A to 13B More detailed explanation.

[0369] Figure 48A 4 shows the use of sampling in determining to associate a user activity with a user interface element. In this example, a UI including UI elements 4810a-d is positioned within a 3D coordinate system in which the user activity is evaluated. In this example, the user activity is gaze. The system detects a gaze direction 4805, which may have an expected amount or range or error, for example, the user's actual gaze direction may be expected to be within 1° of the detected gaze direction with a certain confidence level. The detected gaze direction 4805 is used to identify a user activity location 4815. In this example, the user activity location 4815 is the point at which the gaze direction 4805 is determined to intersect the UI.

[0370] The user activity location 4815 is used to discover possible UI targets. This may involve finding all UI targets in the area around the user activity location 4815. In this example, additional / sample UI locations 4820a-q are identified around the user interface location 4815. Additional / sample UI locations are determined by generating a pattern configured to detect all UI targets having at least a minimum size (e.g., at least 1° in one or more directions, at least a minimum size on the plane of the UI (e.g., height, width, etc.), etc.) around the UI location. By using rays to generate a pattern, a pattern corresponding to the angle requirement can be generated. For example, a scattered ray pattern can be generated, and the intersection of each ray in the scattered ray pattern is used to identify a pattern of sampled locations.

[0371] Additional / sample UI positions 4820a-q around UI position 4815 may then be used to identify visible UI targets, e.g., by taking into account transparency, focus level, rendering order, etc. A given ray (e.g., of a scattered ray pattern) may intersect multiple (e.g., 2, 3, 4, 5, 10, etc.) elements (e.g., colliders, remote effects, hierarchical UI tree layers, etc.). The process may determine (e.g., via logic) which of the intersecting elements is on top, opaque, unobstructed, etc., e.g., which element intersected by the ray is visible based on ordering, transparency, not being clipped by the rendering system, etc.

[0372] exist Figure 48A 48 , the process determines which element of the UI, if any, is on top for each of the additional / sample locations 4820a-q, e.g., for each of the 17 points of the pattern of additional sample locations. Note that in this example, one of the sample / additional locations (i.e., sample / additional location 4820i) corresponds to user activity location 4815. The elements determined to be on top of each of the additional / sample locations 4820a-q provide a list of (0 or more) candidate UI targets. In the example of FIG. 48 , the candidate UI targets include UI elements 4810a, 4810b, and 4810c (based on each intersecting at least one of the additional / sample locations 4820a-q). UI element 4810d is not identified as a candidate UI target because none of the additional / sample locations 4820a-q intersects with it.

[0373] Figure 48B An alternative pattern of additional / sample locations 4860a-s is shown. Some implementations use an optimized pattern of additional / sample locations (or rays used to generate such a pattern). For example, using a configuration of equally spaced locations (e.g., forming a Figure 48BThe shape shown) can enable the use of a relatively small or minimal number of sampling positions. Patterns can be selected to achieve various purposes. For example, a pattern can be configured to ensure that objects of at least a specified minimum size (e.g., minimum height, minimum width, etc.) will be found. A pattern can be configured to ensure that an area of ​​at least a certain size will be covered by the entire pattern, such as ensuring that an area covering an area associated with an expected error threshold for user activity detection (e.g., corresponding to an expected 1° error in gaze tracking) is covered. In one example, a pattern is configured to ensure that the pattern will capture / detect rectangular UI elements with at least a minimum height and / or width size and / or that the pattern will use a minimum number of sampling positions to cover an area of ​​the UI corresponding to gaze detection accuracy. Using a relatively small or minimal number of sampling positions can reduce power consumption, reduce processing requirements and / or increase processing speed, such as making blur hit testing faster and more efficient. Using a tilted pattern can help ensure that long and thin elements (e.g., text) displayed horizontally or vertically are found.

[0374] Determining which element is on top based on rays within a 3D coordinate system may involve using a physics solving process. For example, each ray may be passed into a physics solver that utilizes a 3D simulation / collision world to determine which collision volume(s) intersect the ray. It may iterate along each ray to determine whether each collision volume corresponds to a 3D virtual object (e.g., a virtual cube, etc.) or hosts UI content (e.g., by hosting a UI hierarchical tree element). In some implementations, the physics solver identifies intersections with collision volumes and then uses a separate process to determine which UI element is on top within the collision volume. For example, if the UI within the collision volume is represented by an application Core (CA) structure is provided, CA hit testing (e.g., a 2D hit testing process) can be performed to identify which UI element is on top at a given location. Determining which element is on top may involve determining whether a given UI element is opaque or transparent (and where). Such testing may require detailed information about the application's user interface. Performing such testing via a system-level process (i.e., as opposed to within the application's own process) may require the application to expose UI information for use by the system. In some embodiments, one or more applications provide UI information (e.g., a hierarchical structure identifying UI element locations, layering, remote effects, etc.) for use by system-level processes (e.g., input support processes) when performing system-level fuzzy hit testing.

[0375] 48 may involve extracting geometric information from the candidate UI target, such as rectangular shape, size, rounded corners, which corners are rounded, where there is transparency, etc. Such geometric information may be used to identify the nearest opaque, unobstructed point on the candidate UI target, such as the point on the candidate UI target that has the smallest distance (e.g., angle) to the gaze 4805 / user activity location 4815.

[0376] The process shown in Figure 48 may involve using information about candidate UI targets to prioritize those targets, such as using a policy to determine which candidate UI target is associated with a gaze. Using a consistent policy across multiple applications or otherwise across the system can provide a consistent user experience, such as ensuring that applications have similar behavior in disambiguating user activities. This can help application developers who can design application UI layouts based on known or otherwise expected interpretations of user activities. If a given UI element includes elements that are difficult to hit, then those elements will always be difficult to hit / interact with, and the developer can change the UI layout to make the elements easier to hit / interact with.

[0377] Some implementations utilize a strategy that ranks candidate UI targets based on target type. In some implementations, remote effect candidate targets are prioritized over other types of candidate targets. Thus, if the gaze direction is 0.9 degrees from a remote effect and 0.2 degrees from a normal UI layer or 3D object, the remote effect will be selected regardless of its greater distance. Such a strategy ensures that gazes within a threshold distance of a remote effect (e.g., within a 1° angular distance) will always reliably hit the remote effect. Such a strategy can encourage developers to utilize remote effects to specify behaviors for important content.

[0378] For targets of the same type, the policy can prioritize nested candidate targets. For example, if a button is inside a support plane (button and support plane are both of the same type), the policy can prioritize the inner button. Such a policy ensures that inner / smaller UI elements have the least ambiguity.

[0379] In some implementations, developers can prioritize UI elements with higher priority by specifying smaller (and potentially less noticeable) 3D positions for UI elements. For example, developers can position most UI elements on a 2D plane and position certain higher-priority UI elements slightly above the 2D plane in an elevated / pop-up position. Strategies for prioritizing candidate targets can prioritize UI elements that are closer / pop-up, such as always selecting a UI target when it pops up.

[0380] Some implementations utilize the following strategies:

[0381] Strategy Step 1 - Categorize different types of targets:

[0382] - Ranged effects outperform all other targets

[0383] ○ Example: Hyperlink with remote effect "steals" gaze from plain text

[0384] Strategy Step 2 - Same Type Target

[0385] - Prioritize nested ranged effects

[0386] ○ Example: Toggle button has higher priority than list cell

[0387] - Closest angular distance wins

[0388] - Angle lag on previous target to prevent target flickering

[0389] Some UI content (such as text) can be relatively dense, e.g., letters / words are much denser than 1° apart from each other from a typical viewing position. Strategies can take such density into account, for example, when prioritizing certain types of dense content over other types of dense content (e.g., prioritizing hyperlinks over plain text). Some implementations enable specific / different gestures to allow a user to select lower-priority content (e.g., text) while still allowing plain text adjacent to a higher-priority hyperlink to be selected.

[0390] Figures 49A to 49D The lag in determining when to associate a user activity with a user interface element is shown. This can involve prioritizing candidate targets if they have been previously identified, such as prioritizing buttons when the user's gaze remains on or near a button. Such prioritization can be accomplished by expanding the fuzzy hit test area of ​​the target. Such prioritization can reduce the appearance of flicker, an effect that can occur when user activity between two UI elements alternates between being associated with the first UI element and not the second UI element, returning to the first UI element, and so on.

[0391] exist Figure 49A , during the initial user activity association, each of the two UI elements 4902a-b has an equally sized fuzzy hit test area 4904a-b, for example, each such area is determined based on a 1° angular distance. Based on these areas and the disambiguation strategy, the user activity position above the decision boundary 4910 will be associated with UI element 4902a, and the user activity position below the decision boundary 4910 will be associated with UI element 4902b. Decision boundaries 4910 are equidistant between UI elements 4902a-b.

[0392] Once UI element 4902a is associated with user activity (e.g., gaze at an initial point in time), the region changes. Figure 49B As shown, after the initial association, the two UI elements 4902a-b have different sized blur hit test areas. Blur hit test area 4906a is determined based on an angular distance of 1.2° and is therefore larger than the original blur hit test area 4902a (including the area used for Figure 49B ). Blur hit test region 4902b is still determined based on a 1° angular distance. Based on these regions and the disambiguation strategy, the user activity position above decision boundary 4920 will be associated with UI element 4902a, and the user activity position below decision boundary 4920 will be associated with UI element 4902b. This decision boundary 4920 is closer to UI element 4902b than to UI element 4902a, and is no longer equidistant between the two.

[0393] exist Figure 49C , during the initial user activity association, each of the two UI elements 4952a-b has an equally sized fuzzy hit test area 4954a-b, for example, each such area is determined based on a 1° angular distance. Based on these areas and the disambiguation strategy, the user activity position above the decision boundary 4960 will be associated with UI element 4952a, and the user activity position below the decision boundary 4960 will be associated with UI element 4952b. Decision boundaries 4960 are equidistant between UI elements 4952a-b.

[0394] Once UI element 4952a is associated with user activity (e.g., gaze at an initial point in time), the region changes. Figure 49D As shown, after the initial association, the two UI elements 4952a-b have different sized blur hit test areas. Blur hit test area 4956a is determined based on an angular distance of 1.2° and is therefore larger than the original blur hit test area 4952a (including the area used for Figure 49D ). Blur hit test region 4952b is still determined based on a 1° angular distance. Based on these regions and the disambiguation strategy, the user activity position above decision boundary 4970 will be associated with UI element 4952a, and the user activity position below decision boundary 4970 will be associated with UI element 4952b. This decision boundary 4970 is closer to UI element 4952b than to UI element 4952a, and is no longer equidistant between the two.

[0395] Direct touch example with fuzzy hit testing

[0396] Fuzzy hit testing can be used to improve aiming accuracy with respect to direct touch user interactions. For example, fuzzy hit testing can be used to improve aiming accuracy for the initial touch point for direct touch interactions. Inaccuracies in detecting the initial touch point location can be due to sensor inaccuracies (e.g., accurately detecting hand joint locations), depth perception issues (e.g., the user intended to pierce at a certain location, but actually pierced at another location), extinction and inclusion errors, parallax errors, and potentially other problems. Fuzzy hit testing can be used to adjust the detected landing location to be close to a nearby UI element (e.g., the target closest to it) or within a nearby UI element.

[0397] In some implementations, a direction-based blur hit process, such as that used for gaze blur hit correction, is additionally or alternatively used to adjust the direct touch interaction location. Doing so can involve synthesizing a simulated gaze direction based on the detected direct touch location, such as determining a ray from the current viewpoint location (real or synthesized) that passes through the detected direct touch location.

[0398] For example, at the beginning of a direct touch (e.g., on the first frame where a direct touch is detected), a fuzzy hit testing process is performed using rays that are synthesized based on the location of the detected direct touch. For direct touches, the user interface or other element with which the user interacts is within arm's reach of the user. Thus, in some implementations, the synthesis of gaze rays can be based on a viewpoint that is a selected distance away from the content (e.g., 0.57 meters away). With such a distance, the tolerance of the ray-based / angular distance-based fuzzy hit testing process can correspond to an expected amount of distance on the plane of the UI, e.g., a 1° angular distance can correspond to a 1 cm radial separation on the surface of a flat UI that is 0.57 meters away.

[0399] In some implementations, the amount of tolerance (e.g., 1° angular distance / 1 cm Euclidean distance) can be static. In other implementations, the amount of tolerance (e.g., 1° angular distance / 1 cm Euclidean distance) is dynamic. For example, the amount of tolerance can be scaled in magnitude based on the approach movement of the hand and / or penetration speed during a direct touch. The hand approach movement and speed can indicate or be associated with user care (e.g., precise or loose / sloppy). More precise user activity can be given less tolerance in the fuzzy hit test (e.g., a smaller radius) or different tilts based on penetration angle, etc. Using a fixed amount of tolerance can simplify debugging and ensure greater consistency.

[0400] A fuzzy hit testing process can then be applied, for example, as shown in FIG. 48 , to associate the user activity (e.g., a direct touch initial location) with a UI element, such as snapping the detected location to a new location within a nearby UI element.

[0401] The system can continue to track user activity, such as when the user's hand continues to move to swipe, retract, etc., where subsequent positions on the UI are adjusted based on the initial adjustment / starting UI touch point position, such as correcting the hand trajectory based on the initial correction.

[0402] Direct touch blur hit testing can ignore hysteresis. For example, it may not make sense to consider previous user activity in some or all direct touch contexts, such as if the user touches and pulls back, then touches again and pulls back, etc., then hysteresis may not make sense.

[0403] In some implementations, the fuzzy hit test is run only when direct touch user activity begins (e.g., only on a single frame at or near the start of the user activity). Doing so can save power, save processing resources, and make the process faster and more efficient than it would otherwise be. This may be appropriate for some types of direct touches, but not for other types of direct touches. For example, it may be appropriate to run the fuzzy hit test only on the first frame of a direct touch initial touch, but it may be appropriate to run the fuzzy hit test to detect hover-type direct touch events (e.g., where a finger is just hovering over one or another UI element).

[0404] Direct touch user activities can involve different types of hover feedback. For example, the system can provide a light that continuously tracks the user's finger (e.g., located anywhere on the UI disk directly below the user's finger) and UI element-specific hover feedback (e.g., causing a button to pop slightly when the user's finger hovers over it). Fuzzy hit testing can be used for one or both of these different types of feedback, but can be particularly useful for the second type, where user activities need to be associated with a specific (and typically relatively small) UI element rather than a specific location on a window, disk, or other relatively large UI area.

[0405] Fuzzy hit testing snaps the touch location to a point on a nearby UI element and can change the target ID associated with the user activity. For example, a user can hover over a button and the system can pop the button out, identifying the button as the new target of the user activity. If the user misses slightly and hits the supporting surface, such as missing the bounds of the button, the system can also change the target so that if the user pokes the supporting surface, they will also poke the popped-out button. This can help solve problems when a touch that pops up a button occurs, such as when the top of the button is touched or when the button is pressed.

[0406] Blur hit testing for indirect user activity (e.g., gaze) can be run frequently, such as on every frame in some implementations, or less frequently, depending on the requirements for detecting interactions based on a given user activity.

[0407] In some implementations, fuzzy hit testing for direct touch user activity corresponding to a hand touch is guided based on the user's gaze direction. For example, direct touch fuzzy hit testing can bias UI element association towards the UI element that the user was gazing at during (or just before) the direct touch. Although gaze is not an obvious part of the user interaction, it still helps disambiguate user intent. In some implementations, on the first frame of a direct touch, the touch location is detected if the fuzzy hit test point of the gaze direction (the captured hit test point) is within 1 cm of the detected touch point, for example, only accepting the detected touch point based on the gaze direction being approximately at the same location. This is based on the insight that users tend to look at UI elements when touching them. Similarly, if a touch fails, users tend to immediately look towards the intended target. In the context of relatively dense UIs (e.g., a virtual keyboard), simply using the gaze point as the touch point can lead to incorrect / inaccurate aiming. However, for relatively isolated targets, this can provide accuracy and efficiency. For example, when the detected touch point is located in a region between two targets, gaze can be effectively used to disambiguate between two direct touch targets.

[0408] Additionally, knowing that people tend to look before they touch, especially for small buttons, can be used to calibrate gaze and / or hand tracking during use of the device. For example, the system can determine that gaze direction is consistently detected to be deviating from the upper left by a certain amount (e.g., 0.5) across multiple user interface interactions and adjust gaze tracking accordingly.

[0409] Example Input Support Process

[0410] Figure 50A An exemplary architecture is shown for receiving user activity data and application UI information at a system process that outputs interaction event data for one or more applications for identifying input.

[0411] In this example, the sensor system 5010 and the AR system 5020 can be considered as the sensing layer, e.g., determining where the user's hand is in the 3D world, where the user is gazing in the 3D world, etc., and the OS process 5030 and the application 5040 can be considered as the interaction layer, e.g., determining what the user is interacting with, how the user is interacting, etc.

[0412] The sensor system 5010 may include various sensors, including but not limited to color / RGB image sensors, grayscale image sensors, depth sensors, dynamic vision sensors, motion sensors, etc. The sensor system may include an image signal processor (ISP) component and / or other components that process sensor data.

[0413] The sensor system 5010 can provide sensor data to the XR system 420 in various forms. In some implementations, the sensor data is sent over time, periodically, and / or at a fixed rate (e.g., at X frames of sensor data per second (fps)). In one example, hand data based on images captured by one or more outward-facing image sensors (e.g., on a device such as an HMD) is sent to the XR system 5020 at a fixed rate (e.g., 10 fps, 20 fps, 30 fps, 60 fps, 120 fps, etc.). In another example, such hand data is sent at a non-fixed rate or otherwise irregularly. In one example, gaze data based on images captured by one or more inward-facing image sensors (e.g., inside a device such as an HMD) is sent to the XR system 5020 at a fixed rate (e.g., 10 fps, 20 fps, 30 fps, 60 fps, 120 fps, etc.). In another example, such gaze data is sent at a non-fixed rate or otherwise irregularly. The hand data and gaze data can be sent to the XR system 5020 at the same or different rates and / or at the same or different times. The sensor data may include additional types of information and / or may provide information about other parts of the user or the physical environment the user is in. Such other sensor data may be provided at the same or different rates and / or at the same or different times as the hand data and / or gaze data.

[0414] The XR system 5020 utilizes the received sensor data to perform user activity monitoring and / or tracking. In one example, the XR system 5020 is configured to provide a relatively low-level tracking algorithm. For example, the hand system 5022 of the XR system 5020 may use the sensor data to execute a hand tracking algorithm to track the position, posture (e.g., position and orientation), configuration (e.g., shape), or other aspects of the hand over time. The hand system 5022 may generate, update, and / or track a 3D model of the hand, such as a model of the hand that uses defined "joints" or nodes (which may or may not correspond to the user's physiological hand joints) to represent the shape of the hand. In one example, a hand model of 20+ joints is maintained over time based on the sensor data, so that the hand data generated by the hand system 5022 represents the current position / posture / configuration of the hand at different points in time, which may enable determination of 3D movements or other changes made by the hand over time. The hand system may alternatively (or additionally) track a subset of points on the surface of the user's hand, for example, tracking the position of one or more fingertips / thumbtips of the user's hand. Such tracking may, but need not, include determining when such parts (e.g., fingertips / thumbtips) touch each other and / or when they touch other parts of the user or the physical environment. The hand system 5022 may output hand position, posture, and / or configuration information and a confidence value corresponding to such hand data.

[0415] The gaze system 5024 of the XR system 5020 can use sensor data to execute a gaze tracking algorithm to track eye characteristics, such as, but not limited to, gaze direction, over time. The gaze system 5024 can use the sensor data to directly determine the gaze direction of one or both eyes (e.g., without modeling the shape of the eyes and / or head). The gaze system 5024 can use the sensor data to generate, update, and / or track a 3D model of the eye, such as an eye model that represents the shape of the eye based on the location of a landmark (e.g., the center of the eyeball, the center of the cornea, the center of the pupil, etc.), the size (e.g., the diameter of the eyeball, the diameter of the pupil, etc.), and / or the surface portion of the eye. In one example, the gaze system 5024 outputs a stream of gaze directions for each eye over time (e.g., a vector in 3D space or relative to the user's current position). In one example, the gaze system 5024 outputs a gaze direction and a confidence value corresponding to such gaze direction.

[0416] The XR system 5020 may include a computer vision (CV) system 5026 that supports or otherwise supports the hand system 5022, the gaze system 5024, and / or other XR system subsystems (not shown). For example, the CV system 5026 may execute one or more environment and / or user assessment algorithms, such as performing simultaneous localization and mapping (SLAM) to model a 3D physical environment and keep track of the user's position relative to the model. In another example, the CV system 5026 may identify objects within the physical environment, such as walls, doors, tables, appliances, etc., and / or the locations of such objects. In another example, the CV system 5026 may execute a semantic understanding algorithm to semantically label objects within the physical environment. In another example, the CV system 5026 may use associated sensor data to assess user characteristics (e.g., the type of activity currently being performed (e.g., exercise, work, meditation, etc.), gestures / movements (e.g., sitting, standing, walking, running, driving, etc.), and / or other user characteristics). Such user characteristic data may be used by the hand system 5022 and / or gaze system 5024 to improve their efficiency and / or accuracy.

[0417] exist Figure 50A In the embodiment of the present invention, the XR system 5020 provides hand / gaze data to the operating system process 5030. The hand / gaze data can be provided periodically (e.g., at a fixed frame rate corresponding to the sensor capture rate or otherwise) or irregularly. In one example, the hand data is provided at the same frame rate per second as the gaze data. In another example, the hand data is provided at a different frame rate than the gaze data.

[0418] The hand / gaze data received by the operating system process 5030 can be used by the input support process 5032 to (1) itself provide / initiate some response to the user activity on behalf of the application 5040 and / or (2) provide some information (e.g., interaction data) to the application 5040 to enable the application 5040 to respond to the user activity. The input support process 5040 / operating system process 5030 can provide information about the first type of user activity (e.g., activity determined to correspond to an intentional interaction of the user with a UI element) to the application. Thus, as Figure 50A As shown, input support process 5032 generates interaction data that is provided to application 5040. This interaction data provided to application 5040 may exclude or transform / abstract hand / gaze data (and other user-based information). Application 5040 may only receive interaction data and therefore may not receive data about other types of user activity, such as user activity that is considered to be something other than intentional interaction with UI elements, such as user activity in which the user is simply gazing at other parts of the UI or XR environment or during which the user is reading text displayed within the application's UI. Note that in Figure 50A In this example, the interaction data provided by the input support process 5032 is provided to the application 5040 via the reality system support process 5034. However, in other examples, the input support process 5032 may provide such interaction data to the application 5040 directly or via a different intermediary.

[0419] In some implementations, the input support process 5032 uses information about the user interfaces of one or more applications, such as application 5040, to provide input support to those applications. In some implementations, application 5040 sends user interface information to the input support process 5032, which uses the user interface information to interpret user interactions associated with application 5040. For example, application 5040 may provide information defining the appearance of a rectangular area that contains a set of user interface elements at a specified location and has certain desired interaction capabilities (e.g., selectable, non-selectable, hoverable, non-hoverable, expandable, non-expandable, etc.). Application 5040 may define the type, location, visual appearance, functional characteristics, or other aspects of such elements for use by the input support system process 5032. Application 5040 may provide information about its UI elements to the input support process 5032, which defines what information the application 5040 will receive. For example, based on the defined button component, when the input support system recognizes user activity (e.g., pinching, gazing, etc.) as an intentional interaction with the button element, the application 5040 can receive input events associated with the button.

[0420] In some implementations, the input support process 5032 can use information about the positioning of an application user interface and / or user interface elements within such a user interface to better understand the user's activities and / or intent and ultimately provide a more accurate, more enjoyable, or otherwise better experience for the user. For example, the input support process 5032 can use information about the positioning of an application user interface and / or user interface elements within such a user interface to (a) distinguish user activities associated with a first type of user activity (e.g., intentional interaction events) from other types of user activities, (b) determine which user activities to respond to directly and which user activities the application will respond to, and thereby selectively provide information to the application 5040 that is limited to the user activities to which the application 5040 itself will respond, (c) respond to a user activity associated with the application 5040 (e.g., provide hover feedback without notifying the application 5040), and / or (d) target the user activity to one of multiple applications that the user activity could potentially be targeted.

[0421] exist Figure 50AIn an example of , the reality system support process 5034 provides UI geometry (e.g., collision world) data to the input support process 5032 to enable the input support process 5032 to better support input to applications and / or other XR environment elements. The reality system support process 5034 can generate, update, and / or maintain information about items within the 3D XR environment, e.g., maintaining a current understanding / snapshot view of everything within the environment. This can involve determining where virtual content will be positioned within the XR environment based on the user's physical environment. For example, the reality system support process 5034 can determine that the UI of a first application will be positioned above the surface of the user's physical table in the XR environment, and the UI of a second application will be positioned in front of a window to the side of the user's table.

[0422] The reality system support process 5034 can determine where to spatially locate application UI content to be rendered within the 3D XR environment and how to configure that space (e.g., by determining container size and dimensions) based on information provided by the application. Figure 50A In an example, application 5040 provides UI information (e.g., which may include UI element declarations) that the reality system support process 5034 uses to determine where the application's user interface is spatially positioned and how to configure the space. In one example, configuring an application's UI involves determining the location of one or more containers / colliders (e.g., one or more flat rectangular or other 2D-shaped windows or one or more square or other 3D-shaped bounding areas) where the application's UI will be positioned in 3D space. The reality system support process 5034 may position a container for each of the applications based on the UI information (e.g., UI declarations) provided by one or more applications. It may take into account the surrounding environment (e.g., the size of the room or other characteristics of the XR environment), the user's location, activity, and preferences, and numerous other considerations when determining where and how to organize and manage the 3D positioning of objects within the XR environment. Applications do not need to know (or be informed of) the location of their user interfaces within the 3D XR environment.

[0423] The reality system support process 5034 can determine not only the location of user interface container / collision features such as a window that contains all application content, but also other user interface elements that may be involved in user interaction and user activity. Such user interface elements include, but are not limited to, text elements, buttons, sliders, scroll bars, pickers (e.g., color pickers), menu controls, timelines, images, vector graphics, rulers, icons, and labels.

[0424] Tracking user interface element positions within the 3D XR environment and providing such information to the input support process 5032 may enable the input support process 5032 to more efficiently, accurately, and effectively support input processes, including, but not limited to, supporting input to applications such as the application 5040. Furthermore, using such information may additionally enable the input support process 5032 to do so in a manner that protects the user's privacy by limiting information about the user's activities that is exposed to the application 5040.

[0425] For example, input support process 5032 can use hand data from hand system 5022 to determine that the user is not currently performing a gesture indicating interaction intent (e.g., their hands are not pinching), and can use gaze direction from gaze system 5024 to determine that the user is gazing at a specific button within the user interface of application 5040 within the 3D environment maintained by reality system support process 5034. Input support process 5032 can use this to initiate an out-of-process (e.g., outside of the application process) response, such as reality system support process 5034 providing hover feedback to the user that highlights the button. Application 5040 does not need to receive any information about the user's current user activity (e.g., the user's hand state and / or gaze state), and does not even need to be aware that the hover feedback is being provided by the system. In this example, input support process 5032 uses UI geometry information provided by reality system support process 5034 and based on UI information provided by application 5040 to provide a response to the user activity on behalf of application 5040, without involving the application itself. Avoiding providing user activity data about certain types of user activities (eg, unintentional activities) can help protect user private data from application 5040 .

[0426] In another example, the input support process 5032 can use hand data from the hand system 5022 to determine that the user is not currently performing a gesture indicating an intent to interact (e.g., the hands are not pinching), and use gaze direction from the gaze system 5024 to determine that the user is gazing at a menu title within the user interface of the application 5040 within the 3D environment maintained by the reality system support process 5034. The input support process 5032 can use this to initiate an out-of-process (e.g., outside of the application process) response, for example, the reality system support process 5034 can provide the user with an expansion of a menu that displays previously hidden menu options. The application 5040 does not need to receive any information about the user's current user activity (e.g., the user's hand state and / or gaze state), nor does it even need to be aware that the menu was expanded. In this example, input support process 5032 uses UI geometry information (e.g., identifying menus and their menu options and their positions in 3D space) provided by reality system support process 5034 and based on UI information provided by application 5040 to provide responses to user activities on behalf of application 5040 without involving the application itself. Avoiding providing user activity data about certain types of user activities (e.g., unintentional activities) can help protect user private data from being affected by application 5040.

[0427] The input support process 5032 may additionally (or alternatively) use the UI geometry information (provided by the reality system support process 5034 and based on the UI information provided by the application 5040) to provide information to the application 5040 to respond to the user activity itself. In some embodiments, such information is limited to user activity associated with a first type of user activity, e.g., user activity associated with intentional interaction with a user interface element. The input support process 5032 may provide information (e.g., interaction data) to enable the application 5040 to respond to the user activity itself. Such information may process or limit user activity data (e.g., hand / gaze data received from the XR system 5020) so that the application 5040 does not obtain detailed user data, such as data about a specific user gaze direction, user hand size / shape, etc.

[0428] For example, input support process 5032 may use hand data from hand system 5022 to determine that the user is currently performing a gesture indicating an intent to interact (e.g., pinching their hands), and may use gaze direction from gaze system 5024 to determine that the user is gazing at a particular button within the user interface of application 5040 within the 3D environment maintained by reality system support process 5034. Based on determining that the user activity (e.g., pinching and gazing) meets the criteria that qualifies as a first type of user activity (e.g., intentional interaction with a UI element), input support process 5032 may generate data (e.g., interaction data) distinct from the raw hand data and raw gaze data to send to application 5040 to enable application 5040 to respond to the user activity itself. Application 5040 need not receive raw hand data and / or raw gaze data associated with the user's current activity, and need not even be aware that the user activity is gaze / hand-based. Instead, the data provided to application 5040 may simply be sufficient to enable application 5040 to recognize an input event (e.g., a hit event) for a button of application 5040's user interface. Such data may have been abstracted to use an input modality agnostic format or a single input modality format that may be different from the input modalities available on the device (e.g., using a touch screen input modality format). Avoiding providing detailed user activity (e.g., detailed hand or gaze data) can help protect user private data from being compromised by application 5040.

[0429] The data provided to application 5040 to enable application 5040 to respond to input to its UI elements can take various forms. In some implementations, such data is limited to certain types of user activities, and the data format therefore reflects this. For example, if data is to be provided to application 5040 in response to an intended UI interaction event, data defining the interaction event can be provided to the application. In one example, application 5040 is provided only with information identifying the UI element with which it was interacted and the type of event (e.g., button X received a hit-type interaction).

[0430] In other specific implementations, the application 5040 has an input recognition framework for a specific input environment (e.g., a 2D input environment), and the data provided to the application 5040 mimics the format of the input environment. For example, the application 5040 may be configured with a 2D input recognition framework, wherein the application 5040 is configured to recognize 2D touch input on a touch screen device, such as receiving interaction gesture data that identifies the touch point and direction of a user's touch on the touch screen interface. Note that the term "gesture" here refers to information that identifies a 2D position on the touch screen and / or a direction associated with the touch - it may involve receiving only a 2D position or it may involve receiving both 2D position and direction data. In this example, data that mimics a touch event may be provided to such an application 5040 having a 2D input recognition framework. For example, input support process 5032 can use hand data from hand system 5022 to determine that the user is currently performing a gesture indicating interaction intent (e.g., pinching their hands together), and use gaze direction from gaze system 5024 to determine that the user is gazing at a specific button within the user interface of application 5040 within the 3D environment maintained by reality system support process 5034. Input support process 5032 can generate interaction data identifying the 2D location of the gaze direction relative to the application's user interface element and provide it to application 5040 as an interaction gesture, so that application 5040 can interpret it as a touch event (e.g., a tap) at that location (on the button) and initiate an appropriate response. Application 5040 only needs to receive this interaction gesture, without receiving raw hand or gaze data. In this example, application 5040 knows the interaction point (in its 2D space), but does not know (and, for input recognition purposes, does not need to know) the gaze direction in the 3D space of the XR environment.

[0431] In some implementations, the use of input support processes 5032 enables the execution of applications 5040 in environments that are different from their original or intended environments. For example, applications 5040 may have been originally compiled or intended for execution on a mobile device with a 2D touchscreen input environment or a laptop computer with a 2D mouse / trackpad driven environment. Applications 5040 may be executed within a device that provides a 3D input modality and receive data fr...

Claims

1. A method, comprising: At an electronic device having a processor: obtaining hand data associated with the position of the hand in three-dimensional (3D) space, The hand data is obtained based on the first sensor data; obtaining gaze data associated with a direction of gaze in the 3D space, the gaze data being obtained based on second sensor data; Select direct interaction mode; identifying, in accordance with selecting the direct interaction mode, a first interaction with the first virtual object based on identifying that the position of the hand intersects a first region associated with a position of the first virtual object in an extended reality (XR) environment corresponding to the 3D space; Select the indirect interaction mode; as well as In accordance with selecting the indirect interaction mode, identifying a second interaction with the second virtual object based on identifying that the gaze direction intersects a second region associated with a position of the second virtual object in the XR environment.

2. A method according to claim 1, wherein the direct interaction mode and the indirect interaction mode are selected based on the proximity of the hand position to the position of the virtual object in the XR environment, and wherein the indirect interaction mode is selected when the hand position in the XR environment intersects with an area associated with the virtual object in the XR environment. 3 . The method of claim 1 , wherein identifying the second interaction with the second virtual object is further based on identifying a gesture occurring with the gaze direction, the gaze direction intersecting the gaze second region. 4 . The method of claim 1 , wherein identifying the first interaction with the first virtual object comprises determining an input location on a user interface based on a movement path of the hand. 5 . The method of claim 1 , wherein identifying the first interaction with the first virtual object comprises determining whether the gesture is a tap gesture based on a movement path of the hand.

6. The method of claim 1 , wherein identifying the first interaction with the first virtual object comprises: determining whether a segment of the hand movement has characteristics that satisfy a retraction criterion, the retraction criterion configured to distinguish a retraction motion from another type of motion; as well as A subset of the segments of the movement is associated with a user interface contact based on whether the characteristic of each of the segments satisfies the retraction criteria.

7. The method of claim 1 , wherein identifying the first interaction with the first virtual object comprises: configuring an interruption volume based on the hand movement, the interruption volume defining an area of ​​the XR environment in which the movement is to be associated with a user interface contact; as well as Determining to discontinue associating the movement with the user interface contact is based on the movement crossing a boundary of the blackout volume.

8. The method of claim 1 , wherein identifying the second interaction with the second virtual object comprises: Gaze movement classification data based on the gaze data is obtained, the gaze movement classification data distinguishing gaze periods associated with gaze hold events from gaze periods associated with non-gaze hold events.

9. The method of claim 1 , wherein identifying the second interaction with the second virtual object comprises: generating a plurality of sample positions based on the gaze data; identifying a virtual object based on the plurality of sample positions; as well as The second virtual object to be associated with the user activity is selected from the virtual objects based on a selection criterion.

10. The method of claim 9, wherein identifying the second interaction with the second virtual object further comprises: A point is identified to an application providing the second virtual object so that the application can associate an action with the second virtual object.

11. The method according to claim 1, further comprising: determining an intentionality classification of the gesture based on the hand data; as well as An input action corresponding to the gesture is enabled based on the intentionality classification.

12. A system, comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform operations including: obtaining hand data associated with a position of a hand in a three-dimensional (3D) space, the hand data being obtained based on the first sensor data; obtaining gaze data associated with a direction of gaze in the 3D space, the gaze data being obtained based on second sensor data; Select direct interaction mode; identifying, in accordance with selecting the direct interaction mode, a first interaction with the first virtual object based on identifying that the position of the hand intersects a first region associated with a position of the first virtual object in an extended reality (XR) environment corresponding to the 3D space; Select the indirect interaction mode; and In accordance with selecting the indirect interaction mode, identifying a second interaction with the second virtual object based on identifying that the gaze direction intersects a second region associated with a position of the second virtual object in the XR environment.

13. A system according to claim 12, wherein the direct interaction mode and the indirect interaction mode are selected based on the proximity of the hand position to the position of the virtual object in the XR environment, and wherein the indirect interaction mode is selected when the hand position in the XR environment intersects with an area associated with the virtual object in the XR environment.

14. The system of claim 12, wherein identifying the second interaction with the second virtual object is further based on identifying a gesture occurring simultaneously with the gaze direction, the gaze direction intersecting the gaze second region.

15. The system of claim 12, wherein identifying the first interaction with the first virtual object comprises determining an input location on a user interface based on a movement path of the hand.

16. The system of claim 12, wherein identifying the first interaction with the first virtual object comprises determining whether the gesture is a tap gesture based on a movement path of the hand.

17. The system of claim 12, wherein identifying the first interaction with the first virtual object comprises: determining whether a segment of the hand movement has characteristics that satisfy a retraction criterion, the retraction criterion configured to distinguish a retraction motion from another type of motion; as well as A subset of the segments of the movement is associated with a user interface contact based on whether the characteristic of each of the segments satisfies the retraction criteria.

18. The system of claim 12, wherein identifying the first interaction with the first virtual object comprises: configuring an interruption volume based on the hand movement, the interruption volume defining an area of ​​the XR environment in which the movement is to be associated with a user interface contact; as well as Determining to discontinue associating the movement with the user interface contact is based on the movement crossing a boundary of the blackout volume.

19. The system of claim 12, wherein identifying the second interaction with the second virtual object comprises: Gaze movement classification data based on the gaze data is obtained, the gaze movement classification data distinguishing gaze periods associated with gaze hold events from gaze periods associated with non-gaze hold events.

20. A non-transitory computer-readable storage medium storing program instructions, the program instructions being executable via one or more processors to perform operations comprising: obtaining hand data associated with a position of a hand in a three-dimensional (3D) space, the hand data being obtained based on the first sensor data; obtaining gaze data associated with a direction of gaze in the 3D space, the gaze data being obtained based on second sensor data; Select direct interaction mode; identifying, in accordance with selecting the direct interaction mode, a first interaction with the first virtual object based on identifying that the position of the hand intersects a first region associated with a position of the first virtual object in an extended reality (XR) environment corresponding to the 3D space; Select the indirect interaction mode; as well as In accordance with selecting the indirect interaction mode, identifying a second interaction with the second virtual object based on identifying that the gaze direction intersects a second region associated with a position of the second virtual object in the XR environment.