Fuzzy hit testing

The use of sampling techniques and fuzzy hit testing in user interaction systems addresses inaccuracies in gaze and hand tracking, enabling accurate user activity association in 3D environments and allowing 2D applications to operate seamlessly in 3D spaces while preserving privacy.

CN119065495APending Publication Date: 2024-12-03APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410698040.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-09-29
Filing Date
2024-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing user interaction systems face inaccuracies in sensor-based detection of gaze tracking and hand/joint positions, leading to challenges in accurately associating user activities with elements within a 3D environment.

Method used

Implementing sampling techniques to generate multiple sample positions in a 3D coordinate system, identify UI targets based on collision volumes, and use selection criteria to associate user activities with appropriate elements, while accounting for inaccuracies in sensor data through fuzzy hit testing.

Benefits of technology

Enables accurate and efficient association of user activities with elements in a 3D environment, allowing 2D applications to function in 3D spaces without requiring changes to their input recognition processes, and maintaining privacy by limiting sensitive data access.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065495A_ABST
    Figure CN119065495A_ABST
Patent Text Reader

Abstract

A blur hit test is provided. Various implementations disclosed herein include devices, systems, and methods that associate user activity with user interface (UI) elements within a 3D environment. Some implementations are performed in a manner that takes into account inaccuracies (i.e., ambiguity) in sensor-based user activity detection, such as inaccuracies in gaze tracking based on sensor data or hand / joint position based on sensor data. Some implementations use sampling techniques to associate user activity in a 3D environment with appropriate portions of a UI located within the 3D environment. For example, sampling techniques may be used to identify sample locations within the 3D environment (e.g., sample locations around a gaze direction) to evaluate them and ultimately select therefrom to associate with user activity, such as to associate a particular UI button with a gaze direction.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 470,608, filed on June 2, 2023, which is incorporated herein in its entirety. Technical Field

[0003] The present disclosure generally relates to evaluating user interactions with electronic devices involving hand and body gestures, gaze, voice, handheld controller input, human interface device (HID) input, and / or other user activities. Background Art

[0004] Existing user interaction systems may be improved in facilitating interactions based on user activities. Summary of the invention

[0005] Various embodiments disclosed herein include devices, systems, and methods for associating user activities with elements within a 3D environment. Some embodiments are performed in a manner that takes into account inaccuracies (i.e., ambiguities) in sensor-based user activity detection, such as inaccuracies in gaze tracking based on sensor data or inaccuracies in hand / joint positions based on sensor data. Some embodiments use sampling techniques to associate user activities in a 3D environment with appropriate elements, such as a portion of a user interface (UI) positioned within the 3D environment. For example, sampling techniques can be used to identify sample positions within the 3D environment (e.g., sample positions around a gaze direction) to evaluate them and ultimately select from them to associate with user activities, such as associating a particular UI button with a gaze direction. In one example, the user activity is a gaze direction, and a ray pattern around the gaze direction is generated to identify candidate UI targets, from which UI elements are selected to be associated with the gaze direction.

[0006] In some embodiments, the processor performs the method by executing instructions stored on a computer-readable medium of an electronic device. The method receives data corresponding to user activity in a 3D coordinate system. The data may provide a directional representation of the user activity (e.g., a ray in a 3D coordinate system). For example, the method may identify a gaze direction associated with a user's gaze at one or more points in time (e.g., a gaze direction associated with an indirect user interaction). The method may determine the user's dominant eye when identifying the gaze direction. In another example, the method may identify the user's hand position or movement, and determine a ray from a viewpoint (e.g., an eye position) to the hand or a ray from a viewpoint to a position where the hand intersects the virtual UI (e.g., an eye-to-touch point ray synthesized on the first frame of direct user interaction). In another example, the method may use head tracking to identify the head direction, such as projecting a ray in a direction based on the direction the user's head is facing (e.g., as a substitute for a gaze direction ray).

[0007] The method generates multiple sample positions in a 3D coordinate system based on data corresponding to user activity. In some specific implementations, given a gaze direction, ray, or point in a 3D coordinate system identified based on user activity, the method can identify multiple nearby rays or points in the 3D coordinate system. In one example, the method generates a pattern of 10 to 30 rays in a pattern. The pattern may include rays or points that may be spaced relative to each other to meet predetermined criteria, such as being spaced within the pattern to ensure that UI elements of a specific size and / or shape within the pattern boundary will intersect with at least one of the rays or points. The pattern may change over time, such as frame by frame. For example, the pattern for each frame may include randomly positioned rays / points (e.g., gaze rays or gaze ray intersection points) generated based on user activity.

[0008] The method identifies a UI target within a 3D coordinate system based on multiple sample positions. In some implementations, the UI target includes a collision volume corresponding to a 2D or 3D element (e.g., corresponding to the 3D position / boundary of a 3D virtual object and / or a 2D UI element of an application defined by a layer tree, etc.). In some implementations, a collision volume corresponding to a UI hierarchical structure is identified, and then the UI hierarchical structure is checked to identify which UI element of the UI hierarchical structure is at the top at the sample position (e.g., the nearest, opaque, unobstructed element).

[0009] The method selects a UI target from the identified UI targets to be associated with the user activity based on the selection criteria. In some specific implementations, for each of the identified UI targets, the method calculates the nearest opaque point and the distance (e.g., angular distance) to the user activity location (e.g., to the gaze direction location, etc.), and then classifies and prioritizes the UI targets according to the policy. This may involve when multiple UI targets are within a threshold distance of the user activity location (e.g., within a 1° angular distance), the method prioritizes the UI targets according to the policy by ranking the UI targets according to type (e.g., effect types enabled for different element types), UI element hierarchy, nesting, and / or geometry of the UI targets, proximity to the user activity location (e.g., gaze direction location), and / or based on hysteresis logic that applies prioritization of previous targets and / or avoids flicker.

[0010] Various embodiments disclosed herein include devices, systems, and methods for associating user activities with elements within a 3D environment. Some embodiments provide output to an application in a manner that facilitates the application's identification of user activities (e.g., based on the association of user activities with elements), such as providing output based on 3D user activities so that the application can use the output to identify the user activities as being associated with appropriate / intended elements using existing touch screen / 2D inputs. In some embodiments, this can enable applications configured for a 2D environment (e.g., mobile device applications) to execute within a 3D environment without requiring the application to change its own 2D input recognition process or otherwise account for ambiguities in the underlying 3D user activities, such as inaccuracies in sensor data tracking gaze (indirectly) or hand / joint position (directly).

[0011] In some specific implementations, the processor performs the method by executing instructions stored on a computer-readable medium of the electronic device. The method receives data corresponding to user activity in a 3D coordinate system. The data may provide a directional representation of the user activity (e.g., a ray in a 3D coordinate system). For example, the method may identify a gaze direction associated with a user's gaze at one or more points in time (e.g., a gaze direction associated with an indirect user interaction). In another example, the method may identify a user's hand position or motion, and determine a ray from a viewpoint (e.g., an eye position) to the hand or a ray from a viewpoint to a position where the hand intersects the virtual UI (e.g., an eye-to-touch point ray synthesized on the first frame of a direct user interaction).

[0012] The method receives data corresponding to the positioning of a UI element of an application within a 3D coordinate system (e.g., a geometry collision world). The data corresponding to the positioning of the UI element can be based at least in part on data provided by the application (e.g., the position / shape of a 2D element intended for a 2D window area). In some implementations, the application can provide a hierarchical tree for positioning UI elements on, for example, a 2D area of ​​a 3D coordinate system. Such information can be provided to a simulation process that positions the application element in 3D space, for example, by defining a 3D position in 3D space of one or more collision bodies (e.g., each collision body having a rectangular window area) for the application element.

[0013] The method identifies one or more UI targets within a 3D coordinate system based on data corresponding to the user activity and data corresponding to the positioning of UI elements of the application within the 3D coordinate system. The method selects a UI target from the identified UI targets to be associated with the user activity based on a selection criterion. The method identifies a point within a 2D region to the application based on selecting the UI target to be associated with the user activity so that the application can associate an action (e.g., select / hit / hover / etc.) with the UI target.

[0014] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] So that the present disclosure may be understood by those of ordinary skill in the art, a more particular description may be had with reference to aspects of some illustrative embodiments, some of which are shown in the accompanying drawings.

[0016] Figure 1A to Figure 1B Example electronic devices are shown operating in a physical environment according to some implementations.

[0017] Figure 2 According to some specific implementations, the device provides Figure 1A to Figure 1B A view of virtual elements within a 3D physical environment, where the user performs direct interaction.

[0018] Figure 3According to some specific implementations, the device provides Figure 1A to Figure 1B A view of virtual elements within a 3D physical environment, where the user performs indirect interactions.

[0019] Figure 4 Exemplary direct interactions according to some implementations are shown.

[0020] Figure 5 Indirect interaction identification according to some implementations is shown.

[0021] Figure 6 Use of an exemplary input support framework to generate interaction data based on hand and gaze data and UI target data is shown according to some implementations.

[0022] FIG. 7A to FIG. 7B Exemplary hit detection corrections are shown according to some implementations.

[0023] FIG. 8A to FIG. 8B 2D elements and 3D elements are shown that may be positioned within a 3D environment according to some implementations.

[0024] Fig. 9 Exemplary control areas and fuzzy hit test areas around elements of a user interface are shown according to some implementations.

[0025] FIG. 10A to FIG. 10C Exemplary control areas and fuzzy hit test areas around elements of a user interface are shown according to some implementations.

[0026] Fig.11 Determining whether to associate a user activity with a user interface element according to some implementations is shown.

[0027] FIG. 12A to FIG. 12B The use of sampling in determining to associate user activity with an element is shown according to some implementations.

[0028] FIG. 13A to FIG. 13D Delays in determining how to associate user activities with elements are shown according to some implementations.

[0029] FIG. 14A to FIG. 14B is a flow chart illustrating a method for supporting application input recognition according to some specific implementations.

[0030] Fig.15 is a flow chart illustrating another method for supporting application input recognition according to some implementations.

[0031] Fig.16 is a flow chart illustrating a method for supporting application input recognition using sampling according to some implementations.

[0032] Fig.17 is a flow chart illustrating a method for supporting application input recognition for supporting application user activity recognition according to some implementations.

[0033] Fig.18 is a block diagram of an electronic device according to some specific implementations.

[0034] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, for clarity, the sizes of the various features may be arbitrarily expanded or reduced. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0035] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0036] Figure 1A to Figure 1B Exemplary electronic devices 105 and 110 are shown operating in physical environment 100. Figure 1A to Figure 1B In the example of , the physical environment 100 is a room including a table 120. The electronic devices 105 and 110 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 100 and objects therein, as well as information about the user 102 of the electronic devices 105 and 110 and evaluate the physical environment and objects therein. The information about the physical environment 100 and / or the user 102 can be used to provide visual and audio content and / or to identify the current location of the physical environment 100 and / or the location of the user within the physical environment 100.

[0037] In some implementations, a view of an extended reality (XR) environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via electronic devices 105 (e.g., wearable devices such as HMDs, etc.) and / or 110 (e.g., handheld devices such as mobile devices, tablet computing devices, laptop computers, etc.). Such an XR environment may include a view of a 3D environment generated based on camera images and / or depth camera images of the physical environment 100 and a representation of the user 102 based on the camera images and / or depth camera images of the user 102. Such an XR environment may include virtual content positioned at a 3D position relative to a 3D coordinate system (i.e., 3D space) associated with the XR environment, which 3D coordinate system may correspond to the 3D coordinate system of the physical environment 100.

[0038] In some implementations, video (e.g., pass-through video depicting a physical environment) is received from an image sensor of a device (e.g., device 105 or device 110). In some implementations, a 3D representation of the virtual environment is aligned with a 3D coordinate system of the physical environment. The size of the 3D representation of the virtual environment can be generated based on, among other things, the scale of the physical environment or the positioning of open spaces, floors, walls, etc., so that the 3D representation is configured to align with corresponding features of the physical environment. In some implementations, a viewpoint within the 3D coordinate system can be determined based on the position of the electronic device within the physical environment. The viewpoint can be determined based on, among other things, image data, depth sensor data, motion sensor data, etc., which can be retrieved via a virtual inertial odometer system (VIO), a simultaneous localization and mapping (SLAM) system, etc.

[0039] Figure 2 Shows the Figure 1A to Figure 1B 100 , wherein a user performs direct interaction. In this example, user 102 makes gestures relative to content presented in views 210 a - 210 b of an XR environment provided by a device (e.g., device 105 or device 110). Views 210 a - 210 b of the XR environment include an exemplary user interface 230 of an application (i.e., an example of virtual content) and a drawing 220 of a table 120 (i.e., an example of real content). Providing such a view may involve determining 3D properties of the physical environment 100 and positioning the virtual content, such as user interface 230, in a 3D coordinate system corresponding to the physical environment 100.

[0040] exist Figure 2In the example of , user interface 230 includes various content user interface elements, including background portion 235 and icons 242, 244, 246, 248. Icons 242, 244, 246, 248 can be displayed on flat user interface 230. User interface 230 can be a user interface of an application, as shown in this example. For illustrative purposes, user interface 230 is simplified, and the user interface can actually include any complexity, any number of content items and / or a combination of 2D and / or 3D content. User interface 230 can be provided by various types of operating systems and / or applications, including but not limited to messaging applications, web browser applications, content viewing applications, content creation and editing applications, or any other application that can display, present, or otherwise use visual and / or audio content.

[0041] In this example, background portion 235 of user interface 230 is flat. In this example, background portion 235 includes aspects of user interface 230 that are being displayed in addition to icons 242, 244, 246, 248. Displaying the background portion of an operating system or application's user interface as a flat surface can provide various advantages. Doing so can provide an easy to understand or otherwise easy to use portion of an XR environment to access an application's user interface. In some implementations, multiple user interfaces (e.g., corresponding to multiple different applications) are presented sequentially and / or simultaneously within an XR environment (e.g., within one or more collision volumes or other such components).

[0042] In some implementations, the position and / or orientation of such one or more user interfaces may be determined to facilitate visibility and / or use. Within a 3D environment, one or more user interfaces may be in a fixed position and orientation. In such a case, user movement will not affect the position or orientation of the user interface within the 3D environment.

[0043] The position of the user interface within the 3D environment may be based on determining the distance of the user interface from the user (e.g., from an initial or current user position). The position and / or distance from the user may be determined based on a variety of criteria, including but not limited to criteria that consider application type, application functionality, content type, content / text size, environment type, environment size, environment complexity, environment lighting, the presence of other people in the environment, use of the application or content by multiple users, user preferences, user input, and numerous other factors.

[0044] In some implementations, one or more UIs may be body-locked content, such as having a certain distance and orientation offset relative to a part of the user's body (e.g., their torso). For example, the body-locked content of the UI may be located 0.5 meters to the left of the user's torso's forward vector and at a 45-degree angle. If the user's head turns while the torso remains stationary, the body-locked UI will appear to remain stationary in the 3D environment, 2 meters to the left of the torso's forward vector and at a 45-degree angle. However, if the user does rotate their torso (e.g., by rotating in their chair), the body-locked UI will follow the torso rotation and reposition within the 3D environment so that it is still 0.5 meters to the left of the torso's new forward vector and at a 45-degree angle to it.

[0045] In other implementations, the UI content is defined at a specific distance from the user, where it remains stationary relative to the user's orientation (e.g., if initially displayed in a cardinal orientation, it will remain in that cardinal orientation regardless of any head or body movement). In this example, the orientation of the content will not reference any part of the user's body. In this different implementation, the UI will not reposition itself based on torso rotation. For example, the UI may be defined as being 2m away and may initially be displayed north of the user based on the direction the user is currently facing. If the user rotates the torso 180 degrees to face south, the UI will remain 2m north of the user, now directly behind the user.

[0046] The UI can also be configured to always remain aligned with the gravity line or horizon so that changes in the head and / or body's roll orientation will not cause the UI to move within the 3D environment. Translational movements will cause the content to be repositioned within the 3D environment so that the distance offset is maintained.

[0047] exist Figure 2 In the example of FIG. 2 , user 102 moves their hand from an initial position, as shown by the location of drawing 222 in view 210a. The hand moves along path 250 to a later position, as shown by the location of drawing 222 in view 210b. As user 102 moves their hand along this path 250, the finger intersects user interface 230. Specifically, as the finger moves along path 250, it virtually pierces icon 246 and thus a tip portion of the finger (not shown) is obscured by user interface 230 in view 210b.

[0048] Implementations disclosed herein interpret user movement, such as user 102 moving their hand / finger along path 250 relative to a user interface element, such as icon 246, to identify user input / interaction. Interpretation of user movement and other user activity may be based on identifying user intent using one or more recognition processes.

[0049] exist Figure 2 In the example of , recognizing input may involve determining that the gesture is a direct interaction and then using a direct input recognition process to recognize the gesture. For example, such a gesture may be interpreted as a tap input to icon 246. When making such a gesture, the actual movement of the user relative to icon 246 may deviate from the ideal movement (e.g., a straight path passing through the center of the user interface element in a direction completely orthogonal to the plane of the user interface element). The actual path may be curved, jagged, or otherwise nonlinear, and may be angled rather than orthogonal to the plane of the user interface element. The path may have properties that make it similar to other types of input gestures (e.g., sliding, dragging, flicking, etc.). For example, non-orthogonal motion may make the gesture similar to a sliding motion, in which the user provides input by piercing the user interface element and then moving in the direction along the plane of the user interface. In addition, in some specific implementations, such recognition involves a fuzzy hit test, such as when a finger pierces the UI interface, based on various processes and / or criteria that take into account inaccuracies in determining the position of the finger relative to the user interface in the 3D coordinate space (e.g., taking into account inaccuracies in finger / hand tracking, user perception, etc.), determine the appropriate UI element associated with the piercing.

[0050] Some embodiments disclosed herein determine that a direct interaction mode is applicable, and based on the direct interaction mode, utilize a direct interaction recognition process to distinguish or otherwise interpret user activity corresponding to direct input, such as recognizing intended user interaction based on whether and how a gesture path intercepts one or more 3D regions of space. Such a recognition process may take into account actual human tendencies associated with direct interaction (e.g., a natural arching of the hand that occurs during an intended straight motion, a tendency to move based on a shoulder or other pivot position, etc.), human perception issues (e.g., a user cannot see or know the exact position of virtual content relative to their hand), and / or other direct interaction specific issues.

[0051] Note that the user's movement in the real world (e.g., physical environment 100) corresponds to movement within a 3D space (e.g., an XR environment based on the real world and including virtual content such as a user interface positioned relative to real-world objects including the user). Thus, the user moves their hand in the physical environment 100, such as through empty space, but the hand (i.e., a drawing or representation of the hand) intersects and / or pierces the user interface 300 of the XR environment based on the physical environment. In this way, the user is directly interacting virtually with the virtual content.

[0052] Figure 3 Shows Figure 1A to Figure 1B305 of the XR environment provided by the device, wherein the user performs indirect interaction. In this example, user 102 performs gestures while viewing content presented in view 305 of the XR environment provided by the device (e.g., device 105 or device 110). View 305 of the XR environment includes Figure 2 An exemplary user interface 230 of FIG. Figure 3 In the example of , user 102 makes a pinch gesture with their hands as shown in drawing 222 while gazing at user interface icon 246 along gaze direction 310. In this example, this user activity (e.g., the pinch gesture and the gaze on the UI element) corresponds to a user intent to interact with user interface icon 246, i.e., the pinch indicates an intent to interact, and the gaze (at the time of the pinch) identifies the target of the interaction.

[0053] Implementations disclosed herein interpret user activity (such as user 102 forming a pinch gesture along with a gaze at a UI element) to identify a user / interaction. For example, such user activity may be interpreted as a tap input to icon 246, e.g., selecting icon 246. However, when performing such actions, the user's gaze direction and / or the timing between the gesture and the gaze that the user intended to be associated with the gesture may not be perfectly executed and / or timed. Furthermore, there may be errors or ambiguities in gaze tracking accuracy.

[0054] Some embodiments disclosed herein determine that an indirect interaction mode is applicable, and based on the direct interaction mode, utilize an indirect interaction recognition process to recognize intended user interactions based on user activity (e.g., based on whether and how a gesture path intercepts one or more 3D regions of space). Such recognition processes may take into account actual human tendencies associated with indirect interactions (e.g., eye glances, eye fixations, and other natural human gaze behaviors, hand arching movements, retractions that do not correspond to intended insertion directions, etc.), human perception issues (e.g., users cannot see or know the exact location of virtual content relative to their hands), and / or other indirect interaction specific issues. In some embodiments, such recognition involves fuzzy hit testing, such as determining a user interface element to be associated with a gaze direction based on various processes and / or criteria that take into account inaccuracies in determining gaze direction or other aspects of user activity.

[0055] Some implementations determine an interaction mode, such as a direct interaction mode or an indirect interaction mode, such that user behavior can be interpreted by a dedicated (or otherwise separate) recognition process for the appropriate interaction type, such as using a direct interaction recognition process for direct interactions and using an indirect interaction recognition process for indirect interactions. Such dedicated (or otherwise separate) process utilization may be more efficient, more accurate, or provide other benefits relative to using a single recognition process configured to recognize multiple types of (e.g., both direct and indirect) interactions.

[0056] Figure 2 and Figure 3 An exemplary interaction mode based on user activity in a 3D environment is shown. Other types or modes of interaction may be used in addition or alternatively, including but not limited to user activity via input devices such as keyboards, trackpads, mice, handheld controllers, etc. In one example, a user uses an input device such as a keyboard, trackpad, mouse, or handheld controller to provide an interaction intent via an activity (e.g., performing an action such as tapping a button or trackpad surface), and identifies a UI target based on the user's gaze direction when input is made on the input device (e.g., a fuzzy hit test method may be used). Similarly, user activity may involve voice commands. In one example, a user uses an input device such as a keyboard, trackpad, mouse, or handheld controller to provide an interaction intent via an activity (e.g., performing an action such as tapping a button or trackpad surface), and identifies a UI target based on the user's gaze direction when issuing a voice command (e.g., a fuzzy hit test method may be used). In another example, user activity identifies (e.g., via pinching, gestures, voice commands, input device input, etc.) an intent to interact, and UI elements are determined based on a non-gaze-based direction (e.g., based on where the user is pointing within the 3D environment) (e.g., possibly using a fuzzy hit test approach). For example, a user may pinch with one hand to provide input indicating an intent to interact while pointing at a UI button with a finger of the other hand. In another example, a user may manipulate the orientation of a handheld device in a 3D environment to control a controller direction (i.e., a virtual line extending from the controller within the 3D environment), and (e.g., possibly using a fuzzy hit test approach) a UI element with which the user is interacting may be identified based on the controller direction (i.e., based on identifying what UI element the controller direction intersects when input indicating an intent to interact is received).

[0057] Various implementations disclosed herein provide an input support process, e.g., as an OS process separate from executing applications, that processes user activity data (e.g., regarding gaze, gestures, other 3D activity, HID input, etc.) to generate data for the application that the application can interpret as user input. The application may not need to have 3D input recognition capabilities because the data provided to the application can be in a format that the application can recognize using 2D input recognition capabilities, such as those used within applications developed for use on 2D touch screens and / or 2D cursor-based platforms. In addition, the input support process can perform fuzzy hit test user activity analysis and provide information so that the application will recognize input / user activity corresponding to the expected user interface element, such as correcting 3D world errors in providing 2D information that the application will accurately recognize.

[0058] At least some aspects of interpreting user activity for an application (e.g., possibly including 3D blur hit testing / correction) can be performed by a process external to the application. Doing so can simplify or reduce the complexity, requirements, etc. of the application's own input recognition process, ensure uniform, consistent input recognition across multiple different applications, protect private usage data from application access, and provide many other benefits as described herein.

[0059] Figure 4 An exemplary direct interaction involving a user's hand 422 virtually touching a UI element of a user interface 400 is shown. In this example, the user 102 is using the device 105 to view and interact with an XR environment including the user interface 400. The direct interaction identification process may use sensor data and / or UI information to determine, for example, which UI element the user's hand is virtually touching and / or where the interaction occurs on the UI element. The direct interaction identification process may perform a fuzzy hit test / activity-to-element association. Identifying a direct interaction may additionally (or alternatively) involve evaluating user activity to determine user intent, such as whether the user intends to tap the button or the scroll bar, whether the user intends to perform a direct tap gesture through the UI element or a sliding / scrolling motion along the UI element, etc. Such identification may utilize information about UI elements, such as the location, size, type, type of interaction that can be performed on the element, type of interaction enabled on the element, which target elements of a set of possible target elements of the user's activity accept which types of interactions, etc.

[0060] The identification of such interactions may be based on functions performed both via system processes and via application processes. For example, an input support process of the OS may interpret hand data from a sensor of the device to identify an interaction event and provide limited or interpreted information about the interaction event to an application providing user interface 400. For example, instead of providing detailed hand information (e.g., identifying the 3D positions of multiple joints of a hand model representing the configuration of hand 422), the OS input support process may simply identify a 2D point on UI element 515 where an interaction (e.g., an interaction gesture) occurred within 2D user interface 400. The application process may then interpret the 2D point information (e.g., interpret the information as a selection, a mouse click, a touch screen tap, or other input received at that point) and provide a response, e.g., modifying its UI accordingly.

[0061] Figure 5 Indirect interaction recognition is shown. In this example, sensor data and / or UI information on device 105 is used to recognize a user interaction made by user 102, for example based on outward-facing image sensor data, depth sensor data, eye sensor data, motion sensor data, etc. and / or information available by an application providing a user interface. The sensor data may be monitored to detect user activity corresponding to an engagement condition corresponding to the start of a user interaction.

[0062] In this example, the process detects that the user 102 has positioned their hand 422 within the field of view of the outward-facing image sensor at block 510. The process may detect a particular hand configuration (e.g., claw, flat hand, stable hand in any configuration, etc.) as an indication of hand engagement, or may simply detect the presence of the hand within the field of view of the sensor.

[0063] At block 520, the process uses the user gaze data to identify an object. In this example, the process identifies that the gaze direction 505 of the user 102 is on the user interface element 515. This may involve the blur hit test / user activity and element association process as described herein.

[0064] At box 530, the process displays feedback based on the object identified by the gaze. In this example, the feedback graphically distinguishes the user interface element 515 to indicate that the user interface element 515 now has a different state (e.g., a "hover" state similar to the state of a traditional UI icon when the cursor is on the item without clicking / tapping). In this example, the application providing the UI information does not need to be notified of the hover state and the associated feedback. Instead, the hand engagement, object identification via gaze, and display of feedback can be processed, for example, by an operating system process outside the process (i.e., outside the application process). For example, such a process can be provided via an input support process of the operating system. Doing so can reduce or minimize potentially sensitive user information (e.g., such as a constant gaze direction vector) that might otherwise be provided to the application to enable the application to handle these functions within the application process. Whether and how to display feedback can be specified by the application, even if it is executed outside the process. For example, the application may define that an element should display hover feedback or highlight feedback, and define how the hover or highlight will appear, such that an out-of-process party (e.g., the operating system) may provide the hover or highlight according to the defined appearance. Alternatively, feedback may be defined out-of-process (e.g., defined only by the OS) or defined to use a default appearance / animation if the application does not specify an appearance.

[0065] At block 540, the process identifies a gesture to be associated with the identified object. In this example, the user gazes at user interface object 515 in gaze direction 505 while simultaneously (or at least temporally close to) performing a pinch gesture with hand 422. In this example, the pinch gesture is interpreted as initiating an action on user interface object 515, such as resulting in a cursor "click" event similar to a traditional UI icon (during which a cursor is positioned over the icon and a trigger such as a mouse click or trackpad tap is received) or a selection action similar to a touch screen "tap" event.

[0066] The identification of such interactions may be based on functions performed both via system processes and via application processes. For example, the input process of the OS may interpret hand and gaze data from the device's sensors to identify an interaction event and provide limited or interpreted / abstracted information about the interaction event to the application providing the user interface 400. For example, instead of providing gaze direction information identifying the gaze direction 505, the OS input support process may identify a 2D point within the 2D user interface 400 on a UI element 515, such as an interaction gesture, and the UI element / 2D point has been selected based on a fuzzy hit test / association process that takes into account 3D or other inaccuracies (e.g., gaze direction inaccuracies, hand configuration inaccuracies, etc.). The application process may then interpret the 2D point information (e.g., interpret it as a selection, mouse click, touch screen tap, or other input received at that point) and provide a response, such as modifying its UI accordingly.

[0067] Figure 5 An example of identifying indirect user interactions is shown. Numerous other types of indirect interactions may be identified, for example, based on one or more user actions to identify user interface elements and / or provide input (e.g., no-action / hover type input, selection type input, input with direction, path, speed, acceleration, etc.). Inputs in 3D space similar to inputs on a 2D interface may be identified, for example, inputs similar to mouse movements, mouse button clicks, touch screen touch events, trackpad events, joystick events, game controller events, etc.

[0068] Some implementations utilize an out-of-process (i.e., outside of the application process) input support framework to facilitate accurate, consistent, and efficient input recognition in a manner that preserves private user information. For example, aspects of the input recognition process can be performed out-of-process so that the application has little or no access to information about where the user is looking, such as gaze direction. In some implementations, application access to some user activity information (e.g., data based on gaze direction) is limited to specific types of user activities, such as activities that meet specific criteria. For example, an application may be restricted to receiving only information associated with intentional or intended user activities (e.g., intentional or intended actions that indicate an intent to interact with a user interface element (e.g., to select, activate, move, etc. a user interface element)).

[0069] Some implementations recognize input using functional elements executed via both an application process and a system process external to the application process. Thus, in contrast to a framework where all (or most) input recognition functionality is managed within an application process, some of the algorithms involved in input recognition may be moved out of process, i.e., outside of the application process. For example, this may involve moving algorithms that detect gaze input and intent (which may include at least some aspects of fuzzy hit testing / association) out of the application's process so that the application cannot access user activity data corresponding to where the user is looking, or only access such information in certain cases, such as only for specific instances where the user has shown intent to interact with a user interface element.

[0070] Some implementations use a model in which an application declares or otherwise provides information about its UI elements to recognize input, so that system processes outside of the application process can better facilitate input recognition. For example, an application can declare the location and / or UI behaviors / capabilities of its buttons, scroll bars, menus, objects, and other UI elements. Such declarations can identify how the user interface should behave given different types of user activity, for example, whether a button should (or should not) exhibit hover feedback when the user views it.

[0071] A system process (e.g., outside of an application process) can use such information to provide desired UI behavior (e.g., providing hover feedback in the appropriate context of user activity). For example, a system process can trigger hover feedback for a UI element based on a declaration from an application that the app's UI includes the element and that the UI should, for example, display hover feedback when gazed at. The system process can provide such hover feedback based on identifying a triggering user activity (e.g., gazing at a UI object), and can do so without revealing to the application user activity details associated with the user activity that triggered the hover, the occurrence of the user activity that triggered the hover feedback, and / or providing the hover feedback. The application may not be aware of the user's gaze direction and / or that hover feedback is provided for a UI element.

[0072] In another example, an application declares a menu as part of its user interface and declares that the menu can be extended using the identified set of extended menu options. The system process can handle the extension of the UI menu. The system process can provide menu expansion (e.g., via a system process outside the application process) based on identifying a triggering user activity (e.g., gazing at a label of a menu), and can do so without revealing to the application user activity details associated with the user activity that triggered the menu expansion, the occurrence of the user activity that triggered the menu expansion, and / or the fact that the menu was expanded. The application may not be aware of the user's gaze direction and / or that the menu was expanded.

[0073] Some aspects of input recognition may be handled by the application itself (i.e., in process). However, the system process may filter, abstract, or otherwise manage information that is made available to the application to recognize input to the application. The system process may do so (e.g., perform fuzzy hit testing / activity and element association) in a manner that facilitates efficient, accurate, and consistent (within an application and across multiple applications) input recognition, and allows applications to potentially use more easily implemented input recognition processes and / or traditional input recognition processes, such as input recognition processes developed for different systems or input environments (e.g., using touch screen input processes used in traditional mobile apps).

[0074] Some implementations use a system process to provide interactive event data to applications so that these applications can recognize input. The interactive event data can be limited so that all user activity data is unavailable to these applications. Providing only limited user activity information can help protect user privacy. The interactive event data can be configured to correspond to events that can be recognized by applications using general or traditional recognition processes. For example, the system process can interpret 3D user activity data to provide the application with interactive event data that the application can recognize in the same way as the application will recognize touch events on the touch screen. In some implementations, the application receives interactive event data corresponding only to certain types of user activities (e.g., intentions or deliberate actions on user interface objects), and may not receive information about other types of user activities (e.g., gaze activities only, users move their hands in a way that is not associated with UI interactions, users move closer to or farther away from the user interface, etc.). In one example, during a period of time (e.g., one minute, 10 minutes, etc.), the user gazes around the 3D XR environment, including gazing at certain user interface text, buttons, and other user interface elements, and eventually performs an intended UI interaction, for example, by making an intended pinch gesture while gazing at button X. The system process can handle all user interface feedback during gazes around various UI elements without providing application information about these gazes. On the other hand, the system process can provide interaction event data to the application based on the intended pinch gesture when gazing at button X. However, even this interaction event data can provide limited information to the application, for example, providing an interaction position or posture that identifies the interaction point on button X without providing information about the actual gaze direction. The application can then interpret the interaction point as an interaction with button X and respond accordingly. Therefore, user behaviors that are not associated with the user's intended interaction with the UI element (e.g., gaze hovering only, menu expansion, reading, etc.) are processed outside the process without the application being able to access user data, and information about intended user interface element interactions is limited so that the information does not include all user activity details.

[0075] Figure 6An exemplary input support framework 640 is shown for generating interaction data based on hand data 610, gaze data 620, and UI target data 630 to produce interaction data 650, which can be provided to one or more applications and / or used by system processes to provide a desired user experience. In some specific implementations, the input support process 640 is configured to understand the intent of the user interaction, generate input signals and events to create a reliable and consistent user experience across multiple applications, detect out-of-process input, and responsibly route it through the system. The input support process 640 can arbitrate which application, process, and / or UI element should receive user input, for example, based on identifying which application or UI element is the intended target of the user activity. The input support process 640 can keep sensitive user data (e.g., gaze, hand / body registration data, etc.) private; only abstracted information or high-level information is shared with the application.

[0076] The input support framework 640 can obtain hand data 610, gaze data 620, and UI target data 630 and determine the user interaction state. In some implementations, it is in a user environment where multiple input modalities are available to the user (e.g., where the user can Figure 2 Direct interaction as shown in Figure 3 For example, the input support process may determine that the user's right hand is performing an intended pinch and gaze interaction with a user interface element, the left hand is directly tapping the user interface element, or the left hand is shaking and therefore idle / not doing anything related to the user interface.

[0077] Based on determining the user interaction intent, the input support framework 640 can generate interaction data 650 (e.g., including interaction gestures, manipulator gestures, and / or interaction states). The input support framework 640 can generate input signals and events that applications can consume without requiring custom or 3D input recognition algorithms in the process. In some specific implementations, the input support framework 640 provides interaction data 650 in a format that the application can consume as a touch event on a touch screen or as a touchpad tap with a 2D cursor at a specific location. Doing so can enable the same application (with little or no additional input recognition processing) to interpret interactions across different environments (including new environments where the application was not originally created and / or using new and different input modalities). In addition, application responses to input can be more reliable and consistent across applications in a given environment and across different environments, for example, achieving consistent UI responses for 2D interactions with applications on tablet computers, mobile devices, laptops, etc. and for 3D interactions with applications on HMDs and / or other 3D / XR devices.

[0078] The input support framework may also manage user activity data so that different apps are unaware of user activity related to other apps, e.g., one app will not receive user activity information while the user types a password into another app. This may involve the input support framework 640 accurately identifying which application the user's activity corresponds to, and then routing the interaction data 650 only to the correct application. The input support framework 640 may use details about the UIs of multiple possible target apps to disambiguate input.

[0079] Fuzzy hit testing process example

[0080] Some implementations associate user activities with elements within a 3D environment. Some implementations do this in a manner that takes into account inaccuracies (i.e., ambiguities) in sensor-based detection of user activities (e.g., gaze tracking based on sensor data or inaccuracies in hand / joint positions based on sensor data). Associating user activities with elements may involve "hit testing," which generally refers to identifying user activities corresponding to possible inputs and determining where and how to route information about user activities to corresponding UI or other elements or otherwise routed to the system. Some implementations use sampling techniques to perform hit testing to associate user activities in a 3D environment with appropriate elements (e.g., a portion of a UI) positioned within the 3D environment. For example, sampling techniques can be used to identify sample locations within the 3D environment (e.g., sample locations around gaze directions / gaze intersections) to evaluate and ultimately select to associate with user activities, such as associating a particular UI button with a gaze direction. In one example, the user activity is a gaze direction, and a ray pattern around the gaze direction is generated to identify candidate UI targets, from which UI elements are selected to be associated with the gaze direction.

[0081] In a 2D user interface context (e.g., on mobile and laptop devices), hit testing can begin with a touch point (e.g., on a touch screen), and hit testing can determine to which application or process the touch / hit should be provided / associated. In such a context, the application can perform some or all of such hit testing on its own (e.g., using an in-application process specified by the application's code or functions embedded in the application's executable code (e.g., from an application development library) or otherwise used by the application, and / or using custom logic developed by the application developer). In such a process, the application itself can effectively perform in-depth hit testing, such as determining which element within the application's user interface should receive / respond to a given touch screen tap or other such user activity.

[0082] In a 2D context, hit testing may involve evaluating a hierarchical UI structure (e.g., a tree structure) that uses a hierarchical structure to define the positions and characteristics of UI elements. Hit testing may involve starting with a high-level element of the UI structure (e.g., starting with the main window) and recursively moving down the hierarchy / tree (e.g., if a child passes the hit test, the window will ask its child for a given touch position, the child will ask its child, and so on, recursively along the hierarchy / tree). The hit test pass logic may ask whether this point is within the bounds / boundaries of each hierarchical element. The hit test logic may provide a visible element area and a hittable area associated with the element, wherein the hittable area has a size or shape different from the visible element. In one example, using hit test logic, a small point-like element will have a hittable area corresponding to a large square surrounding the point-like element, such as by element hit testing involving identifying that the hit point is within the larger area.

[0083] One area of ​​problems overcome by aspects of the present disclosure is enabling hit testing logic used by 2D applications to perform 2D interactions on a touch screen via a mouse, etc. for a mobile device, laptop, etc. to work in a context based on gaze and / or other 3D user activities. Gaze and / or other 3D activity tracking may not have the same accuracy and / or characteristics as a touch screen and / or other 2D user activities. In addition, aspects of the present disclosure may provide hit testing in a manner that protects user information (e.g., associating user activities with applications and other content), such as by not providing applications with unrestricted access to user gaze direction information, hand size / shape / activity information, etc.

[0084] A touch screen can identify user activity (e.g., identifying a user touching the screen at a particular x, y location) relatively accurately (e.g., to within 100 microns). In contrast, tracking user gaze, hand position / configuration / motion, and / or other 3D user activity can be significantly less accurate. Some embodiments disclosed herein provide hit testing and / or other processes that account for the possibility of such greater "fuzziness" when tracking 3D user activity. For example, a device can track a user's gaze with an accuracy of plus or minus 1 degree. Thus, if the user is looking at a button (e.g., at the edge of the button), the detected gaze ray may appear to be completely outside the button. Some embodiments evaluate such user activity and determine to associate a gaze direction with a button despite the gaze being outside the button to account for the inaccuracies of the system.

[0085] Some implementations perform such determinations outside of the application process, such as using a system-level process to determine which applications, UI elements within an application, and / or non-application elements the user activity will be associated with. Performing some or all aspects of fuzzy hit testing or other such processes outside of the application's process (e.g., via a system process) can provide various advantages. This can extend the capabilities of applications (e.g., existing applications or applications otherwise configured for another / non-3D platform) that are not configured for 3D user activities and / or specific user activities (e.g., gaze and / or gesture-based input). This can, for example, protect user privacy by abstracting user activity data provided to the application after performing the fuzzy hit test, so that information about the user activity is limited (e.g., avoiding providing unlimited gaze / hand data). This can take into account user activities that can be associated with one of multiple different applications (due to fuzziness). This can ensure that fuzzy hit testing and related processes are executed consistently across multiple different applications. In addition, this can also make it easier to upgrade or change fuzzy hit testing and related processes, such as without reconfiguring individual applications with sensing hardware, hit test features, or other aspects of the system that change over time. This can provide a single stage for fuzzy hit testing (eg, at the operating system level) that can be more efficient, effective, and / or more accurate than performing multiple times and / or using multiple stages.

[0086] Making early, accurate decisions at the operating system level and using application-specific information can unify fuzzy hit testing for optimal system performance. For example, using a central layer with detailed sensor data (e.g., low-level gaze / pupil data and hand data) and detailed application user interface data (e.g., information about specified element positions, sizes, features, transparency, external effects, etc. from the application's UI hierarchy) can achieve a robust, accurate, and efficient system. In some specific implementations, the system-level fuzzy hit test information has enough information about the application UI elements that it can determine (e.g., without burdening the rendering / display components of the system) which element of the UI is "on top" (e.g., what the user sees) for a given sample position / sample ray direction. In the absence of such information, figuring out the top UI element can be complex and error-prone, for example, rounded transparent corners of UI elements, transparent holes within a given object, action graphics defined as cubes with transparent areas surrounding the actual shape of the graphics, etc. can affect accuracy. Similarly, UI elements can be clipped and / or prioritized in a way that hinders the accurate association of UI elements with user activities without enough information. Some implementations provide sufficiently detailed information to system processes to perform accurate unified fuzzy hit testing using detailed sensor data while accounting for transparency, clipping, prioritization, and / or while protecting user privacy.

[0087] Unifying out-of-process blur hit testing can be particularly useful in the context of determining whether to correlate user activity between elements in different apps (or different processes hosting UI within the same app, such as an app using an out-of-process photo picker widget). For example, if the gaze is between two windows, but only one window has an external effect near the gaze ray, then knowing the UI context becomes important for lower-level decisions about where to route the event so that it ends up finding the external effect. This is different from relying on heuristics such as "pick the window closest to the gaze ray" to get the event to the external effect. window mouth ” may be advantageous, this heuristic may not necessarily identify Recent external effects window.

[0088] In addition, such a system process can provide information about user activities (e.g., based on its own fuzzy hit testing) in a manner that the user activities will be associated with appropriate UI elements (e.g., based on the application's separate association / fuzzy hit testing process). For example, an application can provide a 2D user interface element positioned in a 3D world, and the system level process can correct for inaccuracies in 3D user activity sensing by determining that the user activity should be associated with a particular element of the application's UI and provide input data so that the application will recognize input to that element, such as providing a 2D interaction point within the bounds of that element.

[0089] Some implementations address issues caused by relatively low accuracy of gaze tracking or other 3D user activity tracking processes, such as gaze tracking algorithms having first-order errors such that the gaze hit location may not always be where the user is gazing and can result in gaze aiming failures where the gaze point is slightly off / outside the target. Some implementations provide a system or OS-level process for system-wide hit testing "tolerance" or "fuzziness" for aiming UI elements by gaze or other 3D user activity.

[0090] FIG. 7A to FIG. 7B An exemplary hit detection correction in the context of a user interface element 700 provided / specified by an application is shown. In this example, a user gaze direction 705a is determined based on sensor data. The detected gaze direction 705a intersects the user interface at a point 710a outside of the user interface element 700. A system-level fuzzy hit detection process is performed and a determination is made that a gaze should be associated with the user interface element 700, such as if the user had gazed in the gaze direction 705b at a point 710b within the user interface element 700. The system-level detection process can provide output to the application so that the application will recognize input to the UI element 700, for example, by identifying point 710b as an interaction point with the application.

[0091] In some specific implementations, such as FIG. 7A to FIG. 7B As shown, the fuzzy hit test process is configured to "snap" (i.e., reposition) points associated with 3D user activity to the bounds of nearby UI or other elements, so that applications can correctly interpret 3D user activity without having to be configured to address potential lack of accuracy of sensor data or processes used to detect 3D user activity. In addition, the system-level process can perform such fuzzy hit testing to select appropriate elements from elements provided by potentially multiple applications.

[0092] FIG. 8A to FIG. 8B 2D elements and 3D elements that can be positioned within a 3D environment are shown. In this example, such elements are tracked by a simulation process / engine that positions collision volumes within a 3D coordinate system. Such collision volumes can be used to describe or specify the 2D and 3D geometry of the XR environment. In some implementations, the 3D geometry can include collision volumes that contain UI features specified via multiple layers of complex data structures (e.g., using a hierarchical or tree structure). In some implementations, a UI hierarchy that encodes UI appearance and characteristics (e.g., specifying rendering pixel images, vector graphics and / or textures and / or providing animations to provide a desired user experience) can be used, for example. Core A (CA) structure) is used to specify the contents of a collision volume to an application. A hierarchical layer structure can be used to specify the appearance of, for example, a window, where a button is inside a window, text is inside a window, etc. 3D elements (e.g., a 3D model or a cube) can be specified without a hierarchical or other UI specification structure.

[0093] exist Fig. 8A and Figure 8BIn the example of , 3D cube 840 and 3D sphere 840 are positioned within corresponding collision bodies in a 3D coordinate system. A user interface window 832 having elements (e.g., element 835) specified by a hierarchical UI definition / animation structure is also provided. The user interface window (and its elements) are positioned within the collision volume 830 in the 3D coordinate system. For example, the hierarchical UI definition may specify that window 832 has a base layer, which has a layer for a disc, which has a layer for a button, and the button may have a layer for text and button properties. The hierarchical structure of the UI may be simple or complex, small or large, dense or sparse. The hierarchical structure may include a layer that specifies external effects for UI elements. External effects may be implemented as special layers with behaviors / actions associated with specific user activities (e.g., floating when gazed, etc.). Applications may use external effects to convey to the input recognition process effects intended for specific UI elements (e.g., glowing when gazed, expanding when gazed, etc.) or elements should be targeted by the input system (e.g., using a hand or gaze).

[0094] In some implementations, the hierarchical layers provide a 2D arrangement of layers displayed on a plane of a 3D coordinate system (e.g., in a 2D area). In some implementations, one or more of such 2D layers are implemented in a distinctive manner (e.g., protruding slightly from the rest of the layers). Such prominence can be considered when displaying content, determining interactions with the content, and / or prioritizing interactions with portions of the content.

[0095] In some implementations, the application framework controls element sizing. Controlling sizing may involve considering areas around UI elements that act as visual padding, such as providing an area of ​​minimum size for interacting with each element. This may involve ensuring that elements include space around visual features to achieve minimum element size, such as a fuzzy hit test may require a minimum threshold (e.g., 15 points, 18 points, 21 points, etc.). This may involve controlling visual space to keep it consistent between elements in a layout to ensure fair hit testing between elements, such as not providing some elements with significantly more space / padding than other elements. This may involve making the control size static for each element, such as not adapting the hit area to a more compact layout, gaze entry / exit, viewing angle, etc. In some implementations, fuzzy hit testing does not strictly require such parameters. In some implementations, fuzzy hit testing requires such parameters, but enables, for example, application developers who wish to have more customization to use alternatives. Some implementations utilize such parameters but allow exceptions, such as the system can enable the use of alternatives. However, because using such alternatives may be complex, e.g., for developers, some implementations provide fuzzy hit testing, e.g., default mode parameters, which enable developers to take advantage of this functionality without having to expend time or other resources to account for dynamic adjustments, etc.

[0096] Some implementations provide an input support process that provides system-wide fuzzy hit testing. This may involve providing fuzzy hit testing in a way that supplements hit box sizing by applying additional view tolerance and prioritization logic. This may involve using a process or algorithm that evaluates which visual targets are close to the user's active location (e.g., close to gaze) and snaps that location to the desired target. This can enable dynamic hit areas (i.e., based on layout, view, gaze entry / exit, etc.) and prioritization of UI elements across contexts.

[0097] Fig. 9 Exemplary control areas 910a-e and blur hit test areas 920a-e around elements of a user interface are shown. Fig. 9Reasonable visual spacing between elements 910a-e is shown. The top row buttons (i.e., UI elements 910a-d) have some spacing built into the buttons, such as the button label / icon and the space around the displayed button edges. Therefore, the control areas 910a-e correspond to the edges of the buttons. On the other hand, the slider (i.e., UI element 910a) has no built-in spacing, such as no space around the slider. Therefore, the control area 910e increases the spacing to achieve the minimum size requirement. Within a single UI or across multiple elements of different UIs, approximately equal spacing can be provided when setting the element size. The control area size of the UI element is also static, that is, the control area 910a-e is not adjusted or changed unless the corresponding element itself changes size.

[0098] The system can apply fuzzy hit testing to Fig. 9 This may involve defining a control region (e.g., by an algorithm), detecting (e.g., by an algorithm) the UI element closest to the user interaction location (e.g., the gaze point), and snapping the interaction point to the nearest edge of the UI element (e.g., by an algorithm). A disambiguation process or logic may be applied to determine which UI element (to which the user interaction location will be snapped) is between the controls. Such a process may determine the UI element closest to the user interaction location and / or consider other criteria. This Fig. 9 0a-e, i.e., a user interaction location within blur hit test area 920a is associated with control area 910a, a user interaction location within blur hit test area 920b is associated with control area 910b, a user interaction location within blur hit test area 920c is associated with control area 910c, a user interaction location within blur hit test area 920d is associated with control area 910d, and a user interaction location within blur hit test area 920e is associated with control area 910e. Blurred hit test areas 920a-e are smaller between controls 910a-e to avoid overlap, for example, by dividing the distance between adjacent controls. Note that blur hit testing in 3D can be applied based on angular size requirements (e.g., based on a 1-degree gaze angle imprecision). For example, as a user moves closer to or farther away from the user interface or looks directly at the user interface or from a significantly large angle, the size of blur hit test areas 920a-e can increase and decrease (while avoiding overlap).

[0099] FIG. 10A to FIG. 10C Exemplary control areas and blur hit test areas around elements of a user interface are shown. Fig. 10A Exemplary control areas 1010a - c and blur hit test areas 1020a - c around elements of a user interface are shown. Fig. 10BExemplary control areas 1030a-d and blur hit test areas 1040a-d around elements of a user interface are shown. Fig. 10C Exemplary control areas 1050a-c and blur hit test areas 1040a-d around elements of a user interface are shown.

[0100] Fig.11 Determining to associate a user activity with a UI element, such as an exemplary fuzzy hit test process, is shown. In this example, a gaze direction 1105 is detected and the UI position 1110 is identified using the gaze direction, such as based on determining that the gaze direction 1105 intersects the UI at a UI position 1110 in a 3D coordinate system. In this example, the UI element has a control area 1140 and a fuzzy hit test area 1150 (e.g., may be based on a minimum angular distance to the nearest edge of the control area 1140 or other criteria or strategies). Based on the UI position being within the fuzzy hit test area 1150 (e.g., the distance 1130 is less than an angular distance threshold), the process determines to associate the user activity with the user interface element. This may involve treating the gaze as if it intersects the UI at position 1120. The position on the UI associated with the user activity may be the nearest position on the UI element. The position may be the nearest opaque position on the UI element. The position may be the nearest position on the UI element that is opaque and not obscured by, for example, another UI element or a virtual object. The location 1120 may be provided to the owner (eg, the application providing the UI element) so that the application can respond appropriately to the user activity.

[0101] Determining whether to associate a user activity with a UI element and / or which UI element to associate a user activity with (e.g., fuzzy hit testing) can involve various processes. In some implementations, this involves (1) discovering UI targets, (2) finding the closest point and distance to the user activity location, and (3) classifying and / or prioritizing the targets according to a policy.

[0102] The first step, discovering UI targets, may involve finding all UI targets in an area around the user activity location. For example, the detected gaze direction may be used to determine the user interface location (e.g., the location where the gaze direction intersects the UI). Additional / sample UI locations around the user interface location (e.g., around the gaze direction or around the user activity location identified in other ways) may be determined (e.g., by generating a scattered ray pattern configured to detect all UI targets having at least a minimum size (e.g., at least 1° high) around the UI location). The additional / sample UI locations around the UI location may then be used to identify visible UI targets, for example, by considering transparency, focus level, rendering order, etc.

[0103] The second step, finding the nearest point and distance to the user activity location, may involve identifying the nearest point within each identified visible UI target. For example, for each discovered UI target, this may involve calculating the nearest opaque (e.g., non-transparent) point. This may involve identifying the distance (e.g., angular distance) between the nearest opaque point and the user interface location associated with the user activity (e.g., calculating the angular distance based on the viewpoint location).

[0104] The third step, classifying and / or prioritizing targets according to the strategy, may involve determining whether multiple UI targets are identified. When multiple targets are found (e.g., within a 1° gaze range), the process may select which of the UI targets is associated with the user activity based on the strategy. The process / strategy may rank targets based on type (e.g., prioritizing external effects, such as hovering over other layers / geometric type elements). For targets of the same type, the process / strategy may prioritize nested targets and then prioritize UI targets with the closest / smallest distance (e.g., angle) to the user activity location (e.g., gaze location). The process / strategy may apply hysteresis logic to prioritize previously identified / associated UI targets, which may avoid flickering, as described with respect to FIG. 13A to FIG. 13B More detailed explanation.

[0105] Fig. 12A The use of sampling in determining to associate a user activity with a user interface element is shown. In this example, a UI including UI elements 1210a-d is positioned within a 3D coordinate system in which the user activity is evaluated. In this example, the user activity is gaze. The system detects a gaze direction 1205, which may have an expected amount or range or error, such as the user's actual gaze direction may be expected to be within 1° of the detected gaze direction with a certain confidence level. The detected gaze direction 1205 is used to identify a user activity location 1215. In this example, the user activity location 1215 is the point at which the gaze direction 1205 is determined to intersect the UI.

[0106] The user active location 1215 is used to discover possible UI targets. This may involve finding all UI targets in the area around the user active location 1215. In this example, additional / sample UI locations 1220a-q are identified around the user interface location 1215. Additional / sample UI locations are determined by generating a pattern configured to detect all UI targets having at least a minimum size around the UI location (e.g., at least 1° in one or more directions, at least a minimum size on the plane of the UI (e.g., height, width, etc.), etc.). By using rays to generate a pattern, a pattern corresponding to the angle requirement can be generated. For example, a scattered ray pattern can be generated, and the intersection of each ray in the scattered ray pattern is used to identify a pattern of sampling locations.

[0107] Then, additional / sample UI positions 1220a-q around UI position 1215 may be used to identify visible UI targets, e.g., by taking into account transparency, focus level, rendering order, etc. A given ray (e.g., of a scattered ray pattern) may intersect multiple (e.g., 2, 3, 4, 5, 10, etc.) elements (e.g., colliders, external effects, hierarchical UI tree layers, etc.). The process may determine (e.g., via logic) which intersecting element is on top, opaque, unobstructed, etc., e.g., which element intersected by the ray is visible based on sorting, transparency, not clipped by the rendering system, etc.

[0108] exist Fig. 12A In the example of FIG. 12 , the process determines which element of the UI (if any) is on top for each of the additional / sample locations 1220a-q, e.g., for each of the 17 points of the pattern of additional sample locations. Note that in this example, one of the sample / additional locations (i.e., sample / additional location 1220i) corresponds to user active location 1215. The elements determined to be on top of each of the additional / sample locations 1220a-q provide a list of (0 or more) candidate UI targets. In the example of FIG. 12 , the candidate UI targets include UI elements 1210a, 1210b, and 1210c (based on each intersecting at least one of the additional / sample locations 1220a-q). UI element 1210d is not identified as a candidate UI target because none of the additional / sample locations 1220a-q intersects with it.

[0109] Fig. 12B An alternative pattern of additional / sample locations 1260a-s is shown. Some implementations use an optimized pattern of additional / sample locations (or rays used to generate such a pattern). For example, using a configuration of equally spaced locations (e.g., forming a Fig. 12B1215 ) can enable the use of a relatively small or minimum number of sampling positions. The configuration can utilize one or more hexagonal patterns centered on the gaze position, where the sample positions are equally spaced from each other. Patterns can be selected to achieve various purposes. For example, a pattern can be configured to ensure that objects with at least a specified minimum size (e.g., minimum height, minimum width, etc.) will be found. The pattern can be configured to ensure that an area of ​​at least a certain size will be covered by the entire pattern, such as ensuring that an area covering an area associated with an expected error threshold for user activity detection (e.g., corresponding to an expected 1° error in gaze tracking) is covered. In one example, the pattern is configured to ensure that the pattern will capture / detect rectangular UI elements with at least a minimum height and / or width size and / or that the pattern will use a minimum number of sampling positions to cover an area of ​​the UI corresponding to gaze detection accuracy. Using a relatively small or minimum number of sampling positions can reduce power consumption, reduce processing requirements, and / or increase processing speed, such as making fuzzy hit testing faster and more efficient. Using a tilted pattern can help ensure that long and thin elements (e.g., text) displayed horizontally or vertically are found. Tilt can be achieved, for example, by rotating the pattern around a center point at the gaze position 1215. Such rotation can provide rows of elements that are not exactly horizontal and thus can be well suited for some purposes. An oblique pattern can be well suited for finding (e.g., ensuring that at least one sample intersects) long, thin horizontal elements. Sample positions that are not in a horizontal line (e.g., in a slightly diagonal configuration) can provide one or more rows of sample positions, each row of sample positions spanning multiple horizontal regions (e.g., varying in both the x-direction and the y-direction).

[0110] Determining which element is on top based on rays within a 3D coordinate system may involve using a physics solver process. For example, each ray may be passed into a physics solver that uses the 3D simulation / collision world to determine which collision volume(s) intersect the ray. It may iterate along each ray to determine whether each collision volume corresponds to a 3D virtual object (e.g., a virtual cube, etc.) or hosts UI content (e.g., by hosting a UI hierarchical tree element). In some implementations, the physics solver identifies intersections with collision volumes and then uses a separate process to determine which UI element is on top within the collision volume. For example, if the UI within the collision volume is composed of Core (CA) structure is provided, CA hit testing (e.g., a 2D hit testing process) can be performed to identify which UI element is on top at a given location. Determining which element is on top may involve determining whether a given UI element is opaque or transparent (and where). Such testing may require detailed information about the application's user interface. Performing such testing via a system-level process (i.e., as opposed to within the application's own process) may require the application to disclose UI information for use by the system. In some specific implementations, one or more applications provide UI information (e.g., a hierarchical structure identifying UI element locations, layers, external effects, etc.) for use by system-level processes (e.g., input support processes) when performing system-level fuzzy hit testing.

[0111] 12 may involve extracting geometric information from the candidate UI target, such as rectangular shape, size, rounded corners, which corners are rounded, where there is transparency, etc. Such geometric information may be used to identify the nearest opaque, unobstructed point on the candidate UI target, such as the point on the candidate UI target that has the smallest distance (e.g., angle) to the gaze 1205 / user activity location 1215.

[0112] The process shown in Figure 12 may involve using information about candidate UI targets to prioritize these targets, such as using a strategy to determine which candidate UI target is associated with a gaze. Using a consistent strategy across multiple applications or otherwise across the system can provide a consistent user experience, such as ensuring that applications have similar behaviors in disambiguating user activities. This can help application developers who can design application UI layouts based on known or otherwise expected user activity interpretations. If a given UI element includes elements that are difficult to hit, these elements will always be difficult to hit / interact with, and the developer can change the UI layout to make the elements easier to hit / interact with.

[0113] Some implementations utilize a strategy that ranks candidate UI targets based on target type. In some implementations, external effect candidate targets are prioritized over other types of candidate targets. Thus, if the gaze direction is 0.9 degrees from an external effect and 0.2 degrees from a normal UI layer or 3D object, the external effect will be selected regardless of its greater distance. Such a strategy ensures that gazes within a threshold distance of an external effect (e.g., within a 1° angular distance) will always reliably hit the external effect. Such a strategy can encourage developers to use external effects to specify behaviors for important content.

[0114] For targets of the same type, the policy can prioritize nested candidate targets. For example, if a button is inside a support plane (both the button and the support plane are of the same type), the policy can prioritize the inner button. Such a policy ensures that inner / smaller UI elements have minimal ambiguity.

[0115] In some implementations, developers can specify UI elements with higher priority by specifying smaller (potentially unnoticeable) 3D positions of UI elements. For example, developers can position most UI elements on a 2D plane and position certain higher priority UI elements at elevated / pop-up positions slightly above the 2D plane. Strategies for prioritizing candidate targets can prioritize closer / pop-up UI elements, such as always selecting a UI target when it pops up.

[0116] Some implementations utilize the following strategies:

[0117] Strategy Step 1 - Categorize different types of targets:

[0118] - External effects outperform all other objectives

[0119] ○ Example: Hyperlinks with external effects “stealing” gaze from plain text

[0120] Strategy Step 2 - Same Type of Target

[0121] - Prioritize nested external effects

[0122] ○ Example: Toggle button has higher priority than list cell

[0123] - Closest angular distance wins

[0124] - Angle lag on previous target to prevent target flickering

[0125] Some UI content (such as text) can be relatively dense, e.g., the angular distance between letters / words is much denser than 1° from a typical viewing position. For example, policies can take such density into account when prioritizing certain types of dense content over other types of dense content (e.g., prioritizing hyperlinks over plain text). Some implementations enable specific / different gestures to allow a user to select low-priority content (e.g., text) such that plain text adjacent to a higher-priority hyperlink can still be selected.

[0126] FIG. 13A to FIG. 13DDetermining the lag in associating a user activity with a user interface element is shown. This may involve prioritizing candidate targets where they have been previously identified, such as prioritizing buttons when the user's gaze remains on or near a button. Such prioritization may be accomplished by expanding a fuzzy hit test area for the target. Such prioritization may reduce the appearance of flicker, displaying effects that may occur when user activity between two UI elements alternately associates with a first UI element instead of a second UI element, returns to the first UI element, etc.

[0127] exist Fig.13A , during the initial user activity association, each of the two UI elements 1302a-b has an equally sized fuzzy hit test area 1304a-b, for example, each such area is determined based on a 1° angular distance. Based on these areas and the disambiguation strategy, the user activity position above the decision boundary 1310 will be associated with the UI element 1302a, and the user activity position below the decision boundary 1310 will be associated with the UI element 1302b. The decision boundary 1310 is equidistant between the UI elements 1302a-b.

[0128] Once UI element 1302a is associated with user activity (e.g., gaze at an initial point in time), the region changes. Fig. 13B As shown, after the initial association, the two UI elements 1302a-b have fuzzy hit test areas of different sizes. The fuzzy hit test area 1306a is determined based on an angular distance of 1.2° and is therefore larger than the original fuzzy hit test area 1302a (including the fuzzy hit test area for Fig. 13B ). Blur hit test region 1302b is still determined based on 1° angular distance. Based on these regions and the disambiguation strategy, the user activity position above decision boundary 1320 will be associated with UI element 1302a, and the user activity position below decision boundary 1320 will be associated with UI element 1302b. The distance between decision boundary 1320 and UI element 1302b is closer than the distance between decision boundary 1302a, and the distance between the two is no longer equidistant.

[0129] exist Fig. 13C , during the initial user activity association, each of the two UI elements 1352a-b has an equally sized fuzzy hit test area 1354a-b, for example, each such area is determined based on a 1° angular distance. Based on these areas and the disambiguation strategy, the user activity position above the decision boundary 1360 will be associated with the UI element 1352a, and the user activity position below the decision boundary 1360 will be associated with the UI element 1352b. The decision boundary 1360 is equidistant between the UI elements 1352a-b.

[0130] Once UI element 1352a is associated with user activity (e.g., gaze at an initial point in time), the region changes. Fig.13D As shown, after the initial association, the two UI elements 1352a-b have different sized blur hit test areas. Blur hit test area 1356a is determined based on an angular distance of 1.2° and is therefore larger than the original blur hit test area 1352a (including the area used for Fig.13D ). The fuzzy hit test region 1352b is still determined based on the 1° angular distance. Based on these regions and the disambiguation strategy, the user activity position above the decision boundary 1370 will be associated with UI element 1352a, and the user activity position below the decision boundary 1370 will be associated with UI element 1352b. The distance between the decision boundary 1370 and UI element 1352b is closer than the distance to UI element 1352a, and the distance between the two is no longer equidistant.

[0131] Direct touch example for fuzzy hit testing

[0132] Fuzzy hit testing can be used to improve aiming accuracy with respect to direct touch user interactions. For example, fuzzy hit testing can be used to improve aiming accuracy for the initial touch point for a direct touch interaction. Inaccuracies in detecting the initial touch point location can be due to sensor inaccuracies (e.g., accurately detecting hand joint locations), depth perception issues (e.g., the user intended to pierce at a certain location, but actually pierced at another location), extinction and inclusion errors, parallax errors, and potentially other problems. Fuzzy hit testing can be used to adjust the detected landing location to be close to a nearby UI element (e.g., the target closest to it) or within a nearby UI element.

[0133] In some implementations, a direction-based blur hit process, such as for gaze blur hit correction, is additionally or alternatively used to adjust the direct touch interaction position. In some implementations, the direction-based blur hit process for adjusting the direct touch interaction position can use the finger interaction position. For example, the finger interaction position can be determined and the distance used to locate the sample position can be determined accordingly, such as based on an angular distance of 1 degree from the position. For example, if the fingertip is 2 feet away from the user's head position / device position, the distance of the sample around the fingertip position can be closer than if the fingertip is 3 feet away.

[0134] In some implementations, the direction-based fuzzy hit progression used to adjust the direct touch interaction location can use gaze / viewpoint-independent rays. As an example, the direct touch aiming ray can be related to the approaching trajectory of the finger. As another example, such a ray can be a ray offset from a fixed point behind the finger. As another example, such a ray can be a ray from the fingertip to the plane normal of a nearby collision body.

[0135] Adjusting a direct touch interaction location using a direction-based fuzzy hit progression may involve synthesizing a simulated gaze direction based on a detected direct touch location, such as determining a ray from a current viewpoint location (real or synthesized) through the detected direct touch location.

[0136] For example, at the beginning of a direct touch (e.g., on the first frame where a direct touch is detected), a fuzzy hit test process is performed using a synthetically generated ray based on the location of the detected direct touch. For a direct touch, the user interface or other element with which the user interacts is within arm's reach of the user. Thus, in some implementations, the synthesis of gaze rays may be based on a viewpoint that is a selected distance away from the content (e.g., 0.57 meters away). With such a distance, the tolerance of the ray-based / angular distance-based fuzzy hit test process may correspond to an expected amount of distance on the plane of the UI, e.g., a 1° angular distance may correspond to a 1 cm radial spacing on the surface of a flat UI that is 0.57 meters away.

[0137] In some implementations, the amount of tolerance (e.g., 1° angular distance / 1 cm Euclidean distance) can be static. In other implementations, the amount of tolerance (e.g., 1° angular distance / 1 cm Euclidean distance) is dynamic. For example, the amount of tolerance can be scaled in size based on the hand approach motion and / or penetration speed during direct touch. Hand approach motion and speed can be indicative of or associated with user care (e.g., precise or loose / sloppy). More precise user activity can be given less tolerance in the fuzzy hit test (e.g., a smaller radius) or different tilts based on penetration angle, etc. Using a fixed amount of tolerance can simplify debugging and ensure greater consistency.

[0138] A fuzzy hit testing process may then be applied, such as shown in FIG. 12 , to associate user activity (eg, direct touch initial location) with a UI element, such as snapping the detected location to a new location within a nearby UI element.

[0139] The system can continue to track user activity, such as as the user's hand continues to move to swipe, retract, etc., wherein subsequent positions on the UI are adjusted based on the initial adjustment / starting UI touch point position, such as correcting the hand's trajectory based on the initial correction.

[0140] Direct touch blur hit testing can ignore hysteresis. For example, it may not make sense to consider previous user activity in some or all direct touch contexts, such as if the user touches and retracts and then touches again and retracts, etc., then hysteresis may not make sense.

[0141] In some implementations, the fuzzy hit test is run only when direct touch user activity begins (e.g., only on a single frame at or near the start of the user activity). Doing so can save power, save processing resources, and make the process faster and more efficient than it would otherwise be. Doing so may be appropriate for some types of direct touches, but not for other types of direct touches. For example, it may be appropriate to run the fuzzy hit test only on the first frame of a direct touch initial touch, but it may be appropriate to run the fuzzy hit test to detect hover-type direct touch events (e.g., where a finger is just hovering over one or another UI element).

[0142] Direct touch user activities can involve different types of hover feedback. For example, the system can provide a glow that continuously tracks the user's finger (e.g., located anywhere on the UI disk directly below the user's finger) and hover feedback specific to UI elements (e.g., causing a button to pop slightly when the user's finger hovers over it). Fuzzy hit testing can be used for one or both of these different types of feedback, but can be particularly useful for the second of those that need to associate user activities with specific (and usually relatively small) UI elements rather than specific locations on a window, disk, or other relatively large UI area.

[0143] Fuzzy hit testing snaps the touch location to a point on a nearby UI element and can change the target ID associated with the user activity. For example, a user can hover over a button and the system can pop up the button, identifying the button as the new target of the user activity. If the user misses slightly and hits the supporting plane, such as missing the bounds of the button, the system can also change the target so that if the user pokes the supporting plane, they will also poke the popped-up button. This can help solve problems when a touch that pops up a button occurs, such as when the top of the button is touched or when the button is pressed.

[0144] Blur hit testing for indirect user activity (eg, gaze) can be run frequently, such as on every frame in some implementations, or less frequently, depending on the requirements for detecting interactions based on a given user activity.

[0145] In some specific implementations, a fuzzy hit test for direct touch user activity corresponding to a hand touch is guided based on the user's gaze direction. For example, a direct touch fuzzy hit test can bias the UI element association toward the UI element that the user is gazing at during the direct touch (or just before the direct touch). Although gaze is not an obvious part of the user interaction, it still helps to disambiguate the user's intention. In some specific implementations, on the first frame of the direct touch, if the fuzzy hit test point (the captured hit test point) of the gaze direction is within 1 cm of the detected touch point, the touch position is detected, for example, based on the gaze direction being approximately at the same position, only the detected touch point is accepted. This is based on the insight that users tend to look at UI elements when they touch UI elements. Similarly, if the touch fails, the user tends to look at the intended target immediately. In the context of a relatively dense UI (e.g., a virtual keyboard), simply using the gaze point as the touch point can lead to incorrect / inaccurate aiming. However, for relatively isolated targets, this can provide accuracy and efficiency. For example, when the detected touch point is located in a region between two targets, gaze can be effectively used to disambiguate between two direct touch targets.

[0146] Additionally, knowing that people tend to look before they touch, especially for small buttons, can be used to perform calibration for gaze and / or hand tracking during use of the device. For example, the system can determine that gaze direction is consistently detected to be deviating from the upper left by a certain amount (e.g., 0.5) across multiple user interface interactions, and adjust gaze tracking accordingly.

[0147] Example Input Support Process

[0148] Fig.14A An example architecture is shown for receiving user activity data and application UI information at a system process that outputs interaction event data for one or more applications for identifying input.

[0149] In this example, sensor system 1410 and AR system 1420 may be considered as a sensing layer, e.g., determining where the user's hands are in the 3D world, where the user is gazing in the 3D world, etc., and OS process 1430 and application 1440 may be considered as an interaction layer, e.g., determining what the user is interacting with, how the user is interacting, etc.

[0150] The sensor system 1410 may include various sensors, including but not limited to color / RGB image sensors, grayscale image sensors, depth sensors, dynamic vision sensors, motion sensors, etc. The sensor system may include an image signal processor (ISP) component and / or other components that process sensor data.

[0151] Sensor system 1410 can provide sensor data to XR system 420 in various forms. In some implementations, sensor data is sent over time, periodically, and / or at a fixed rate (e.g., at X frames of sensor data per second (fps)). In one example, hand data based on images captured by one or more outward-facing image sensors (e.g., on a device such as an HMD) is sent to XR system 1420 at a fixed rate (e.g., 10fps, 20fps, 30fps, 60fps, 120fps, etc.). In another example, such hand data is sent at a non-fixed rate or otherwise irregularly. In one example, gaze data based on images captured by one or more inward-facing image sensors (e.g., inside a device such as an HMD) is sent to XR system 1420 at a fixed rate (e.g., 10fps, 20fps, 30fps, 60fps, 120fps, etc.). In another example, such gaze data is sent at a non-fixed rate or otherwise irregularly. The hand data and gaze data may be sent to the XR system 1420 at the same or different rates and / or at the same or different times. The sensor data may include additional types of information and / or may provide information about other parts of the user or the physical environment in which the user is located. Such other sensor data may be provided at the same or different rates and / or at the same or different times as the hand data and / or gaze data.

[0152] The XR system 1420 utilizes the received sensor data to perform user activity monitoring and / or tracking. In one example, the XR system 1420 is configured to provide a relatively low-level tracking algorithm. For example, the hand system 1422 of the XR system 1420 may use the sensor data to execute a hand tracking algorithm to track the position, posture (e.g., position and orientation), configuration (e.g., shape), or other aspects of the hand over time. The hand system 1422 may generate, update, and / or track a 3D model of the hand, such as a model of the hand that uses defined “joints” or nodes (which may or may not correspond to the user's physiological hand joints) to represent the shape of the hand. In one example, a hand model of 20+ joints is maintained over time based on sensor data, so that the hand data generated by the hand system 1422 represents the current position / posture / configuration of the hand at different points in time, which may enable determination of 3D movements or other changes made by the hand over time. The hand system may alternatively (or additionally) track a subset of points on the surface of the user's hand, e.g., tracking the position of one or more fingertips / thumbtips of the user's hand. Such tracking may, but need not, include determining when such parts (e.g., fingertips / thumbtips) are in contact with each other and / or when they are in contact with other parts of the user or the physical environment. The hand system 1422 may output hand position, posture, and / or configuration information and confidence values ​​corresponding to such hand data.

[0153] The gaze system 1424 of the XR system 1420 can use the sensor data to execute a gaze tracking algorithm to track eye characteristics, such as but not limited to gaze direction, over time. The gaze system 1424 can use the sensor data to directly (e.g., without modeling the shape of the eyes and / or head) determine the gaze direction of one or both eyes. The gaze system 1424 can use the sensor data to generate, update, and / or track a 3D model of the eye, such as an eye model representing the shape of the eye based on the location of the identification point (e.g., the center of the eyeball, the center of the cornea, the center of the pupil, etc.), the size (e.g., the diameter of the eyeball, the diameter of the pupil, etc.), and / or the surface portion of the eye. In one example, the gaze system 1424 outputs a gaze direction stream (e.g., a vector in 3D space or relative to the current position of the user) for each eye over time. In one example, the gaze system 1424 outputs a gaze direction and a confidence value corresponding to such a gaze direction.

[0154] The XR system 1420 may include a computer vision (CV) system 1426 that supports or otherwise supports the hand system 1422, the gaze system 1424, and / or other XR system subsystems (not shown). For example, the CV system 1426 may execute one or more environment and / or user assessment algorithms, such as performing simultaneous localization and mapping (SLAM) to model a 3D physical environment and keep track of the user's position relative to the model. In another example, the CV system 1426 may identify objects within the physical environment, such as walls, doors, tables, appliances, etc. and / or the locations of such objects. In another example, the CV system 1426 may execute a semantic understanding algorithm to semantically tag objects within the physical environment. In another example, the CV system 1426 may use associated sensor data to assess user characteristics (e.g., the type of activity currently being performed (e.g., exercise, work, meditation, etc.), posture / motion (e.g., sitting, standing, walking, running, driving, etc.), and / or other user characteristics). Such user characteristic data may be used by hand system 1422 and / or gaze system 1424 to improve their efficiency and / or accuracy.

[0155] exist Fig.14A In the embodiment of the present invention, the XR system 1420 provides hand / gaze data to the operating system process 1430. The hand / gaze data can be provided periodically (e.g., at a fixed frame rate corresponding to the sensor capture rate or otherwise) or irregularly. In one example, the hand data is provided at the same frame rate per second as the gaze data. In another example, the hand data is provided at a different frame rate than the gaze data.

[0156] The hand / gaze data received by the operating system process 1430 may be used by the input support process 1432 to (1) provide / initiate some response to the user activity on behalf of the application 1440 itself and / or (2) provide some information (e.g., interaction data) to the application 1440 to enable the application 1440 to respond to the user activity. The input support process 1440 / operating system process 1430 may provide information about the first type of user activity (e.g., activity determined to correspond to an intended interaction of the user with the UI element) to the application. Thus, as Fig.14A As shown, input support process 1432 generates interaction data that is provided to application 1440. This interaction data provided to application 1440 may exclude or transform / abstract hand / gaze data (and other user-based information). Application 1440 may only receive interaction data and may therefore not receive data about other types of user activity, such as user activity that is considered to be something other than an intended interaction with a UI element, such as user activity in which the user is simply gazing at other parts of the UI or XR environment or during which the user is reading text displayed within the application's UI. Note that in Fig.14A In this example, the interaction data provided by the input support process 1432 is provided to the application 1440 via the simulation system 1434. However, in other examples, the input support process 1432 may provide such interaction data to the application 1440 directly or via a different intermediary.

[0157] In some implementations, the input support process 1432 uses information about the user interfaces of one or more applications, such as application 1440, to provide input support to those applications. In some implementations, application 1440 sends user interface information to the input support process 1432, which uses the user interface information to interpret user interactions associated with application 1440. For example, application 1440 may provide information defining the appearance of a rectangular area that contains a set of user interface elements at a specified location and has certain desired interaction capabilities (e.g., selectable, non-selectable, hoverable, non-hoverable, expandable, non-expandable, etc.). Application 1440 may define the type, location, visual appearance, functional characteristics, or other aspects of such elements for use by the input support system process 1432. Application 1440 may provide information about its UI elements to the input support process 1432, which defines what information the application 1440 will receive. For example, based on defining a button component, when the input support system recognizes user activity (eg, pinching, gazing, etc.) as an intended interaction with a button element, the application 1440 can receive an input event associated with the button.

[0158] In some implementations, input support process 1432 can use information about the positioning of an application user interface and / or user interface elements within such a user interface to better understand the user's activities and / or intent and ultimately provide a more accurate, more enjoyable, or otherwise better experience for the user. For example, input support process 1432 can use information about the positioning of an application user interface and / or user interface elements within such a user interface to (a) distinguish user activities associated with a first type of user activity (e.g., an expected interaction event) from other types of user activities, (b) determine which user activities to respond to directly and which user activities the application will respond to, and thus selectively provide information to application 1440 limited to user activities to which application 1440 itself will respond, (c) respond to a user activity associated with application 1440 (e.g., provide hover feedback without notifying application 1440), and / or (d) target the user activity to one of multiple applications to which the user activity could potentially be targeted.

[0159] exist Fig.14A In an example of , simulation system 1434 provides UI geometry (e.g., collision world) data to input support process 1432 to enable input support process 1432 to better support input to applications and / or other XR environment elements. Simulation system 1434 may generate, update, and / or maintain information about items within a 3D XR environment, such as maintaining a current understanding / snapshot view of everything within the environment. This may involve determining where virtual content will be positioned within the XR environment based on the user's physical environment. For example, simulation system 1434 may determine that a first application's UI will be positioned above the surface of a user's physical table in the XR environment, and a second application's UI will be positioned in front of a window to the side of the user's table.

[0160] The simulation system 1434 can determine where to locate the space of the application UI content to be rendered within the 3D XR environment and how to configure the space (e.g., by determining the container size and dimensions) based on the information provided by the application. Fig.14AIn an example, application 1440 provides UI information (e.g., the UI information may include UI element declarations), and simulation system 1434 uses the UI information to determine where the spatial positioning of the application's user interface is and how to configure the space. In one example, configuring the application's UI involves determining the location of one or more containers / colliders (e.g., one or more flat rectangular or other 2D shaped windows or one or more square or other 3D shaped bounding areas) where the application's UI will be positioned in 3D space. Simulation system 1434 may position containers for each of the applications based on UI information (e.g., UI declarations) provided by one or more applications. It may take into account the surrounding environment (e.g., the size of the room or other characteristics of the XR environment), the user's location, activity, and preferences, and numerous other considerations in determining where and how to organize and manage the 3D positioning of objects within the XR environment. Applications do not need to know (or be informed of) the location of their user interfaces within the 3D XR environment.

[0161] The simulation system 1434 can determine not only the location of user interface container / collision volume features such as a window containing all application content, but also other user interface elements that may be involved in user interaction and user activity. Such user interface elements include, but are not limited to, text elements, buttons, sliders, scroll bars, pickers (e.g., color pickers), menu controls, timelines, images, vector graphics, rulers, icons, and labels.

[0162] Tracking user interface element positions within the 3D XR environment and providing such information to the input support process 1432 may enable the input support process 1432 to more efficiently, accurately, and effectively support input processes, including, but not limited to, supporting input to applications such as application 1440. Moreover, using such information may additionally enable the input support process 1432 to do so in a manner that protects the user's privacy by limiting information about the user's activities that is exposed to the application 1440.

[0163] For example, the input support process 1432 can use the hand data from the hand system 1422 to determine that the user is not currently making a gesture indicating an interaction intent (e.g., the hands are not pinching), and use the gaze direction from the gaze system 1424 to determine that the user is gazing at a specific button within the user interface of the application 1440 within the 3D environment maintained by the simulation system 1434. The input support process 1432 can use this to initiate an out-of-process (e.g., outside the application process) response, such as the simulation system 1434 can provide hover feedback to the user that highlights the button. The application 1440 does not need to receive any information about the user's current user activity (e.g., the user's hand state and / or gaze state), and does not even need to know that the hover feedback is provided by the system. In this example, the input support process 1432 uses the UI geometry information provided by the simulation system 1434 and based on the UI information provided by the application 1440 to provide a response to the user activity on behalf of the application 1440 without involving the application itself. Avoiding providing user activity data regarding certain types of user activities (eg, unintentional activities) can help protect user private data from being affected by application 1440 .

[0164] In another example, the input support process 1432 may use hand data from the hand system 1422 to determine that the user is not currently making a gesture indicating an interaction intent (e.g., the hands are not pinching), and use the gaze direction from the gaze system 1424 to determine that the user is gazing at a menu title within the user interface of the application 1440 within the 3D environment maintained by the simulation system 1434. The input support process 1432 may use this to initiate an out-of-process (e.g., outside of the application process) response, such as the simulation system 1434 may provide the user with an expansion of a menu that displays previously hidden menu options. The application 1440 does not need to receive any information about the user's current user activity (e.g., the user's hand state and / or gaze state), or even know that the menu is expanded. In this example, the input support process 1432 uses the UI geometry information provided by the simulation system 1434 and based on the UI information provided by the application 1440 (e.g., identifying the menu and its menu options and their locations in 3D space) to provide a response to the user activity on behalf of the application 1440 without involving the application itself. Avoiding providing user activity data regarding certain types of user activities (eg, unintentional activities) can help protect user private data from being affected by application 1440 .

[0165] The input support process 1432 may additionally (or alternatively) use the UI geometry information (provided by the simulation system 1434 and based on the UI information provided by the application 1440) to provide information to the application 1440 to respond to the user activity itself. In some embodiments, such information is limited to user activities associated with a first type of user activity, e.g., user activities associated with intended interactions with user interface elements. The input support process 1432 may provide information (e.g., interaction data) to enable the application 1440 to respond to the user activity itself. Such information may process or limit user activity data (e.g., hand / gaze data received from the XR system 1420) so that the application 1440 does not obtain detailed user data, such as data about a specific user gaze direction, user hand size / shape, etc.

[0166] For example, the input support process 1432 may use hand data from the hand system 1422 to determine that the user is currently performing a gesture indicating an intent to interact (e.g., the hands are pinching), and use the gaze direction from the gaze system 1424 to determine that the user is gazing at a particular button within the user interface of the application 1440 within the 3D environment maintained by the simulation system 1434. Based on determining that the user activity (e.g., pinching and gazing) meets the criteria that qualifies as a first type of user activity (e.g., intended interaction with a UI element), the input support process 1432 may generate data (e.g., interaction data) that is different from the raw hand data and the raw gaze data to send to the application 1440 to enable the application 1440 to respond to the user activity itself. The application 1440 does not need to receive the raw hand data and / or raw gaze data associated with the user's current activity, and does not even need to know that the user activity is gaze / hand based. Instead, the data provided to the application 1440 may simply be sufficient to enable the application 1440 to recognize an input event (e.g., a hit event) to a button of the user interface of the application 1440. Such data may have been abstracted to use an input modality agnostic format or a single input modality format that may be different from the input modalities available on the device (e.g., using a touch screen input modality format). Avoiding providing detailed user activity (e.g., detailed hand or gaze data) can help protect user private data from application 1440.

[0167] The data provided to application 1440 to enable application 1440 to respond to input to its UI elements can have various forms. In some implementations, such data is limited to certain types of user activities, and therefore the data format reflects this, for example, where data is to be provided to application 1440 in response to an expected UI interaction event, data defining the interaction event can be provided to the application. In one example, application 1440 is provided only with information identifying the UI element with which it was interacted and the event type (e.g., button X received a hit type interaction).

[0168] In some implementations, the interaction gestures are provided to the application 1440. The interaction gestures can be 3D or 2D. In some implementations, the interaction gestures are 3D but transformed into the application's coordinate system so the application gesture recognizer (e.g., 2D gesture recognizer) can ignore the "z" value if it is not needed or used. However, in this example, the Z value is present and can be used for gesture recognition. For example, to recognize a long press in 3D space, gesture recognition can involve determining when the fingertip stays in Z to distinguish it from a tap that penetrates deep into the UI and may accidentally exceed the long press timeout (in the total time spent touching the UI).

[0169] In other specific implementations, application 1440 has an input recognition framework for a specific input environment (e.g., a 2D input environment), and the data provided to application 1440 mimics the format of the input environment. For example, application 1440 may be configured with a 2D input recognition framework, wherein application 1440 is configured to recognize 2D touch input on a touch screen device, such as receiving interaction gesture data that identifies the touch point and direction of a user's touch on the touch screen interface. Note that the term "gesture" here refers to such information that identifies a 2D position on the touch screen and / or a direction associated with the touch - it may involve receiving only a 2D position or it may involve receiving 2D position and direction data. In this example, data that mimics a touch event may be provided to such an application 1440 having a 2D input recognition framework. For example, the input support process 1432 may use hand data from the hand system 1422 to determine that the user is currently making a gesture indicating an interaction intent (e.g., the hands are pinching), and use the gaze direction from the gaze system 1424 to determine that the user is gazing at a particular button within the user interface of the application 1440 within the 3D environment maintained by the simulation system 1434. The input support process 1432 may generate interaction data identifying the gaze direction relative to the 2D location of the user interface element of the application, and provide it to the application 1440 as an interaction gesture so that the application 1440 can interpret it as a touch event (e.g., a tap) at that location (on the button) and initiate an appropriate response. The application 1440 only needs to receive the interaction gesture without receiving raw hand or gaze data. In this example, the application 1440 knows the interaction point (in its 2D space), but does not know (and does not need to know for input recognition purposes) the gaze direction in the 3D space of the XR environment.

[0170] In some implementations, use of input support processes 1432 enables execution of application 1440 in an environment different from its original or intended environment. For example, application 1440 may be originally compiled or intended for execution on a mobile device with a 2D touch screen input environment or a laptop computer with a 2D mouse / trackpad driven environment. Application 1440 may be executed within a device that provides a 3D input modality and receive data (corresponding to those 3D input modalities that application 1440 cannot directly recognize) from input support processes 1432 in a format recognizable by application 1440 (e.g., as 2D input corresponding to touch screen or mouse / trackpad driven input). Applications intended for mobile devices, laptops, or other devices may be executed within a 3D interaction enabled HMD environment without requiring significant (or any) modifications to the application's input recognition processes. In one example, the HMD is configured with binary compatibility for a mobile device and / or laptop device (e.g., enabling it to execute binary or object code executable by the mobile device and / or laptop device), and provides enhanced input capabilities to mobile applications and / or laptop applications executing on the HMD by utilizing an input support process 1432 that provides data based on user activities in a 3D environment that the application can recognize as 2D modality-based input.

[0171] In the above example, additional information may be provided to application 1440. For example, application 1440 may receive information about the position of a pinched hand, such as a manipulator pose. Such hand information may be at a higher level than the raw hand data. For example, application 1440 may receive a manipulator pose that identifies the position and / or orientation of a hand in 3D space, without receiving information about the configuration of the hand and / or information about a 3D model (e.g., of a joint) used to represent the position, pose, and / or configuration of the hand in 3D space. In another example, application 1440 may receive information about an interaction state, such as identifying an interaction type determined by input support process 1432.

[0172] In the above examples, the criteria for identifying a first type of user activity (e.g., an activity associated with an intended user element interaction) involves evaluating whether the user's hands exhibit a particular configuration (e.g., a pinch gesture), and based on identifying a hand exhibiting such a configuration, identifying other concurrent user activities, for example, identifying where the user is gazing (or approaching) at that time. Such a pinch may be based on criteria for evaluating the proximity of parts of the user's hand model to each other (e.g., the proximity of the fingertips to the thumb tip, etc.), using a classifier or other algorithm to label or classify the user's hand configuration, or otherwise determined by processing hand data. Other types of user activities and / or criteria may be used to identify a first type of user activity (e.g., an activity associated with an intended user element interaction). For example, a voice command may be identified as an indicator of an intended activity, for example, identifying a keyword or phrase such as "select" or "hit" or "tap" or "click this", and then associating gaze direction and / or other user activities occurring during or near the utterance time of the keyword or phrase with the interaction intent, for example, using other activities to identify the UI element target on which an action will be taken.

[0173] The input support process 1432 may additionally account for sensor-based or other inaccuracies in hand and / or gaze data. Tracking the position of user application interface elements within the 3D XR environment and providing such information to the input support process 1432 may enable it to account for such inaccuracies. In addition, it may be desirable to use a system (shared) process so that such inaccuracies can be consistently and effectively accounted for across multiple applications, such as providing a system-level fuzzy hit test process. In other words, it may be desirable to have a single shared process perform such corrections rather than having individual applications do so. In one example, a user gazes at a button, but the gaze system 1424 generates a user gaze direction that is slightly outside the button (e.g., 0.5 degrees outside). The input support process 1432 may correct this error. For example, it may be determined that the gaze may actually be directed at the button because it is within a threshold (e.g., 1 degree) of the button and there is nothing else nearby. The input support process 1432 may correct for this when providing data to the application 1440 that enables the application to respond to user activities. For example, instead of providing an interaction gesture slightly outside a button, it may provide an interaction gesture within the button (e.g., at the edge of the button). Application 1440 therefore does not need to consider gaze inaccuracies in its own input recognition process, e.g., it does not need to determine by itself whether an interaction gesture just outside a button should be considered an interaction gesture within the button. This may be particularly useful if application 1440 uses a framework from another input paradigm (e.g., a touch screen paradigm that utilizes a different (potentially much smaller) inaccuracy / error threshold). Such an application would not have to implement different thresholds for different inaccuracy levels expected in different input modalities. Having an input support process (e.g., an OS process) correct such inaccuracies may provide more consistent and accurate results without requiring application developers to invest significant resources to address such inaccuracies and / or differences between different input modalities.

[0174] Application 1440 can provide UI information to operating system process 1430 in various formats. In some implementations, application 1440 declares its UI elements, such as declaring a hierarchy of UI elements within its user interface. Such declarations may include information / instructions that enable operating system process 1430 to respond to a user activity on behalf of the application. In some implementations, application 1440 declares external effects for certain UI elements, such as declaring that button X should display hover feedback but button Y should not display hover feedback. Application 1440 can use external effect declarations to specify the intended behavior of application 1440 for some or all of its UI elements. An application may provide a hierarchical declaration structure (e.g., Core The application 1440 may provide an application program (CA) structure that declares UI element positions, sizes, types, hierarchical relationships, transparent portions, layered effects, special effects, and / or any other information that facilitates the functionality provided by the simulation system 1434 and / or the input support process 1432. The application 1440 may provide such information to the operating system process 1430 via inter-process communication (or otherwise).

[0175] The input support process 1432 can use such information (e.g., application declarations of UI elements and / or external effects) to better interpret user activity. For example, a given gaze may be between two UI elements, one with an external effect and the other without, and the input support process 1432 may move / snap the gaze to the UI element with the external effect because it is more likely to be the appropriate / expected UI element to which the UI response should be associated. External effects may be executed out-of-process without involving the application 1440.

[0176] In some implementations, application 1440 is enabled to request user permission for specified user data (e.g., detailed hand data and / or detailed gaze data) and, if explicitly approved by the user, is enabled to receive such information.

[0177] In various implementations, the hand system 1422 may generate different types of data, including but not limited to timestamp data, joint position data, POV correction (e.g., to joint positions), world transformation data, joint confidence data, palm center data, palm normal direction data, hand motion data, hand radius data, pinch data, object detection data (e.g., about objects being held or touching the hand), and occlusion data, such as occlusion probability data about a joint being occluded, and therefore its data may be less accurate. Similarly, in various implementations, the gaze system 1426 may generate different types of data, including but not limited to timestamp data, world transformation data, binocular gaze data, gaze confidence data, gaze tracking state data, gaze direction data, gaze origin data, pupil center data, and pupil diameter data. In some implementations, the XR system includes a frame support module that implements better frame dropping support via input frame queuing.

[0178] The input support process 1432 can access rich and detailed hand and gaze data, and use this rich and detailed information to support accurate, efficient, and consistent input responses both within and outside of the app process. However, it may be desirable to keep the details of such rich and detailed user data outside of applications (such as application 1440) to prevent such applications from knowing information about the user that the user considers private, such as what the user reads, what content they look at and for how long, how fast the user reads, how large the user's hands / fingers are, how the user interacts with other applications or aspects of the XR environment, etc.

[0179] Fig. 14B Another exemplary architecture for receiving user activity data and application UI information at a system process that outputs interaction event data for one or more applications for identifying input is shown. In this example, sensor system 1410 and AR system 1420 can be considered as a sensing layer, such as determining where the user's hands are in the 3D world, where the user is gazing in the 3D world, how the user is using an input device (e.g., a trackpad or controller, etc.), etc., and OS process 1430 and application 1440 can be considered as an interaction layer, such as determining what the user is interacting with, how the user is interacting, etc.

[0180] Sensors 1410 and XR system components (e.g., hand system 1422, gaze system 1424, and CV system 1426) are similar to those described above with respect to Fig.14A Those discussed. In addition, a human interface device (HID) 1428 (e.g., a touchpad, a 3D mouse, a handheld controller, etc.) provides device HID data to the HID system 1436. Such device data may correspond to 3D motion or position data representing the control of a point, object, ray, or other form of input affordance located within the XR environment. For example, a user may manipulate the position and orientation of a handheld controller to direct a ray / vector within the XR environment toward a UI element. HID 1428 may include multiple input mechanisms, such as one for positioning a ray and another for indicating an intention to interact with an object to which the ray is directed. In this example, a user may manipulate HID 1428 and see a visualization of a ray extending from the device in a certain direction. The user may manipulate the device to point a ray at a button on the user interface of the application, and press a physical button on HID 1428 (when the ray is directed at the button) to indicate the intention of the button interaction, such as initiating a click, tap, hit, etc. on the button.

[0181] exist Fig. 14BIn the example of , the HID system uses motion data to provide HID data to both the simulation system 1434 (which can position enabling representations (e.g., rays, points, etc.) within the 3D XR environment) and the input support process 1432 (which can use the HID data to identify intended interactions, respond to some interactions on behalf of one or more applications, and / or provide data to one or more applications to enable one or more applications to respond to HID events without providing detailed or otherwise private user data to the applications.

[0182] In one example, the input support process 1432 recognizes that a ray direction from a HID device intersects an application's UI element and initiates an out-of-process UI response (e.g., hover feedback on the element) without providing information about the user activity to the application.

[0183] In another example, the input support process 1432 identifies a first type of user activity (e.g., an intentional UI element interaction type of activity) based on HID data that meets certain criteria (e.g., including intentional physical button presses, occurring when a keyword / key phrase is spoken, occurring when another hand is pinched, etc.). The input support process 1432 sends interaction data to the application based on the user activity. It should be noted that the same interaction data can be triggered by different types of user activities, such as user activities using different interaction modalities (such as direct touch, indirect touch, HID-based, etc.). As described above, the input support process can encapsulate data provided to the application 1440 in a form that the application 1440 can recognize as input, without requiring the application to have a built-in input recognition process specific to some or all input modalities. For example, the application 1440 may not include 3D input recognition based on controller ray direction + button click, and the input support process can encapsulate interaction data associated with such input in a format that the application can understand, such as 2D touch-based input / 2D ​​mouse cursor-based input to a touch screen.

[0184] In an alternative implementation, the HID data is provided directly to the emulation system 1434 and / or the application 1440 without input support process 1432 support.

[0185] In some embodiments, HID 1428 is a trackpad and input support process 1432 fuses gaze direction and trackpad touch data, such that the trackpad touch provides an indication of an intended interaction with a UI element and the gaze direction is used to identify a target (e.g., a UI element) to be associated with the intended interaction.

[0186] In some implementations, the HID events are provided to a pointer UI process (which may be separate from OS process 1430 and / or application 1440). The pointer UI process may control the positioning of affordances (e.g., point representations, shape representations, ray representations, etc.) displayed to the user via UI framework 1442b. In some implementations, pointer UI process 1450 and UI framework 1442 are within application 1440, i.e., executed in process by application 1440.

[0187] Fig.15 is a process flow diagram illustrating an exemplary process for providing direct and indirect input using hand and eye data.

[0188] In this example, hand data is input to the hand processing block 1510. The upsampling and prediction block 1512 upsamples the hand data (e.g., adds more frames) and makes predictions about future hand positions, paths, trajectories, velocities, accelerations, etc. In some implementations, in the hand input process, the internal state of the upsampler is updated with new observation data from a computer vision-based tracking system. In the interaction detection process, data about when to render the next frame is received, which calculates an interpolated / predicted timestamp and queries the upsampler to obtain joint data corresponding to that timestamp. These steps can occur asynchronously and / or at different frame rates. The process can typically query the upsampler at different timestamps between two updates of the computer vision tracking system to obtain several predictions, for example, hand tracking can run at 30Hz and the display can be rendered at 90Hz.

[0189] The hand and POVc blending block 1514 can use the corrected and uncorrected hand joint data to mitigate POVc coupled hand jumps that may occur, for example, due to background depth and camera switching. The hand and POVc blending block can calculate new hand pose information to account for the difference between the position of the user's hand in the real world (e.g., tracked by a computer vision system) and the position of the hand from the user's perspective of viewing the display. The algorithm can smoothly blend the real version of the hand skeleton and the POV-corrected version to create hand pose information that is optimized for input recognition in a use environment where hand registration of the user interface is important (e.g., during direct input, the finger must be aligned with the button). When the user's hand is outside the user's field of view (FOV) and / or the field of view of the sensor, it may not make sense to spend resources, or corrections may not be provided in other ways, resulting in jumps in the hand position. The process can smoothly blend hand data over time, for example, based on whether the user is in direct or indirect mode, whether the hand is close to a UI element, or other factors.

[0190] Eye data is input to gaze processing block 1520. Gaze detection block 1522 removes glance, blink and / or other gaze loss events (e.g., leaving only the gaze fragment corresponding to gaze attentively). Exemplary glance detection process can detect glance and blink for removing and / or identifying gaze attentively for gaze interaction using gaze confidence, tracking state, pupil center, pupil diameter, interpupillary distance (IPD), gaze ray data and speed data. Exemplary glance detection process can be distinguished between gaze events as gaze attentively and gaze events as glances to promote more accurate input based on gaze. Gaze filtering block 1524 filters and / or smoothens gaze data signal. This can relate to smoothing gaze data during gaze. Filtering can be closely coupled to gaze motion classification. Gaze processing block 1520 can lock / remember the last smooth gaze position for those intermediate time periods, or can utilize additional algorithm / strategy to "fill" the data between gaze attentively.

[0191] The processed hand and eye data is provided to the interaction detection block 1530. It is noted that the interaction detection block 1530 may additionally use UI geometry data including data about the user interface of the application. The UI geometry data may be provided at a different rate than the hand and / or eye data. In one example, UI data is received at a first rate (e.g., 90fps), while hand and gaze data are received asynchronously at a slower average rate (e.g., 60fps). The interaction detection block 1130 may additionally use frame target times, such as the system's prediction of when to render the frame. The interaction detection block 1530 may, for example, run asynchronously when UI updates and / or gaze / hand data are received.

[0192] The gaze blur hit test and focus area 1532 block identifies the virtual elements (e.g., UI elements) and the focus area with which the user's gaze is considered to be associated, for example, based on the head or device orientation / direction. An exemplary gaze blur hit test process can use gaze ray data, confidence data, gesture data (e.g., hand movement classification), gaze clustering expansion data, etc. to relax / tighten the gaze area / cone based on the accuracy of gaze tracking and / or user behavior. Another exemplary gaze blur hit test process can use the process described with respect to Figure 12. These processes can utilize UI geometry from the simulation system 1434 based on UI information provided by applications such as application 1440, for example, to identify interactive targets (e.g., which UI elements are associated with a given user activity) based on declared external effects or UI elements. Using such UI information can facilitate more accurate hit testing. In addition to UI geometry, blur hit testing can query more detailed data, such as data about remote effects, gestures, or other contexts.

[0193] The accidental rejection block 1534 can identify user hand and gaze behaviors that are not intended to interact with the system, such as when the user moves his hand to rest, fidget, or interact with real-world objects, such as when eating. The accidental rejection block 1534 can identify hand and gaze behavior instances that are rejected (i.e., associated with subconscious user behaviors that are not intentional interactions) but are still associated with virtual elements under appropriate conditions. It attempts to understand the intention behind the user's movement to better interpret the movement as, for example, an intentional UI element interaction or other things. An exemplary accidental rejection process can use hand joint data, interaction state history, gaze focus area data, occlusion data, etc. to reject (or cancel) unintentional user interactions. The process can utilize the UI geometry from the simulation system 1434, which is based on UI information provided by an application such as application 1440, such as based on external effects near UI elements or their declarations to distinguish between user intentional activities and accidental activities. Using such UI information can promote more accurate accidental rejection. For example, based on determining (based on data declared by the application) that the UI of the application is 2D / planar, user motion in the Z direction may be identified as accidental, and therefore user motion in the Z direction is unlikely to be intentional input to the UI, i.e., it may be accidental motion that unintentionally triggers UI interaction, and therefore the input support process 1432 can be more confident in rejecting user activity as accidental.

[0194] Some implementations support two interaction modes (e.g., direct interaction mode and indirect interaction mode). Some implementations support other combinations of interaction modes (e.g., direct interaction mode, indirect interaction mode, gaze-only interaction mode, and / or peripheral device interaction). A process may be used to determine which, if any, of several interaction models a user is executing. This may involve identifying and ignoring unexpected behavior, and disambiguating the intended interaction model when the user wishes to interact intentionally. In this example, the direct vs. indirect mode detection block 1536 uses the processed hand and eye data to determine the interaction mode used to interpret the user's activity, such as selecting a direct interaction mode or an indirect interaction mode.

[0195] Then, based on the detected interaction mode, the pipeline dedicated to the mode is run to generate signals and events that drive the interaction. For example, for the direct interaction mode, the hand and eye data (processed and / or unprocessed) are provided to the direct interaction identification block 1540, which provides direct input to the user interface. The trajectory correction block 1542 can adjust the trajectory of the user's movement, such as adjusting the user's hand movement to take into account the user's behavior in the 3D environment, such as to make the motion data better correspond to the user's expected movement / expected interaction. The hover, proceed, disconnect block 1544 can use gestures, motion paths, speeds, accelerations, etc. to perform various recognition processes to identify certain types of direct interactions with user interface elements, such as "hover" interactions based on detecting that a hand / finger is close to but not touching a user interface element, "proceed" interactions based on detecting the point (in 3D space / time) at which the hand / finger has contacted the user interface element, "interrupt" interactions based on detecting the point (in 3D space / time) at which the hand / finger stops contacting the user interface element, etc.

[0196] Many types of direct interactions may be recognized and provided as user input. In another example, a direct interaction is recognized by recognizing that a user is in contact with a slider user interface element, moves their hand to the left a distance X while in contact with the slider, and then retracts their hand to break contact with the slider. The slider may move to the left, such as a distance X, based on the input. In another example, a user makes a similar motion to provide input that imparts velocity on a user interface element, such as providing a 3D swipe gesture on a user interface to cause the user interface to begin moving and continue moving after the hand breaks contact, where the continuous motion is based on the velocity already applied to the UI object, such as similar to when a user swipes upward on a multi-touch interface of a mobile phone.

[0197] If indirect interaction mode is selected, the (processed and / or unprocessed) hand and eye data is provided to an indirect interaction identification block 1550, which provides indirect input to the user interface. A user-centric motion block 1552 takes into account user-centric motion in the hand data (e.g., taking into account the fact that the user may be performing motions that pivot around a rotation point (e.g., shoulder, elbow, wrist, etc.)). The user-centric motion block 1552 can be used to map the user's hand motions to an object or affordance separate from the hand, such as an affordance displayed over the application content.

[0198] The pinch and gaze fusion block 1554 determines which gaze data will be associated with a user activity indicating an intention to interact with a UI element (such as a pinch gesture). Many types of indirect interactions can be recognized and provided as user input. In one example, the user pinches the fingers together and moves the pinched hand while gazing at the UI element to provide a movement input to the UI element. In another example, the user quickly pinches and releases while gazing at the UI element to provide a selection event (e.g., similar to a touch screen tap or mouse click input event).

[0199] like Fig.15 The input support process shown in the example of may perform one or more core functions. In some specific implementations, these functions include, but are not limited to, obtaining gaze data and associating the user's gaze with UI elements associated with one or more applications. Doing so may involve performing a fuzzy hit test process. Doing so may involve creating a focus area. A focus area may be considered a spatial area / region. Larger objects may be partially within / partially outside the focus area, so that the system needs to test where the user is interacting on the object to understand whether the interaction point is within the user's focus area. The focus area may include some or all parts of a set of zero or more UI elements that a user may focus on at a given time. Focus areas can be used to disambiguate between interaction types (e.g., direct and indirect input modalities), disambiguate which application a user is interacting with, and / or disambiguate which UI elements are the targets of the user's intent to interact with UI elements.

[0200] In some implementations, the input support process 1432 relies heavily on the simulation system 1434 to provide a geometric representation of the user interface for hit testing against hand data and gaze data. The input support process 1432 can use the collision world (e.g., 3D geometry abstraction) provided by the simulation system 1434. The input support process 1432 can use user interface priority information (e.g., hierarchical display ordering, transparency information, etc.). The input support process 1432 can utilize a separate hit testing process that produces a 3D world hit test result (RE). The input support process 1432 can utilize a hit testing process to use an application-provided hierarchical tree (e.g., declaring UI elements, relationships, and / or presenting information such as which areas are transparent, rendered on top, etc.).

[0201] In some specific implementations, executing Fig.151432 to provide an input support process 1432 that generates data for an application. The input support process 1432 can generate an interaction state of one or more UI elements, such as identifying whether the interaction is direct or indirect, whether the interaction is a hover / close proximity interaction (e.g., associated with a user activity that is not linked to a UI interaction behavior) or a gesture / pinch / touch interaction (e.g., associated with a user activity that indicates a UI intended behavior).

[0202] In some implementations, the input support process 1432 provides interaction gestures, such as tracking correction points on a UI element that are configured to drive a touch screen tap-type input.

[0203] In some implementations, the input support process 1432 provides, for example, a manipulator pose corresponding to the position and / or orientation of the hand itself. The manipulator pose can provide a 3D position of a stable hand center or pinch centroid. The manipulator pose can provide the position and / or orientation of the manipulator driving the interaction, for example, for direct touch interactions, index fingertip information (e.g., position), and for indirect pinch interactions, pinch information (e.g., pinch centroid).

[0204] In some implementations, the input support process 1432 provides an activity target (eg, a UI element).

[0205] In one example, the user initiates an interaction by pinching while gazing at a UI element. The application receives an interaction gesture on the UI element and recognizes the user's intention to interact with the UI element. The user continues to pinch and moves their hand to the left. The application receives more interaction gestures, for example, a set of positions on the user interface based on the 3D position of the hand's movement when it moves to the left, and responds by moving the associated UI element to the left. In this example, by pinching while gazing at a UI object and then moving the pinched hand to the left, the user provides an input that the application can recognize to move the object to the left. The application can respond accordingly without receiving (and in fact not receiving) information about the user's gaze direction and / or specific hand characteristics. In this example, the application only receives interaction gesture data and can therefore be an application that can only receive interaction gesture data as a proxy for touch input data. Manipulator gesture data such as the 3D position of the hand when it moves to the left can be provided to the application in addition or alternatively, and the application can determine the movement of the associated object based on the changed manipulator gesture position. Applications can be provided with information based on user changes, for example, providing user-centric acceleration deltas similar to delta updates for a trackpad that communicate the amount the user has moved from one frame to another, rather than absolute position. For example, communicating changes via a separate API can help us optimize one set of signals to accurately represent motion (e.g., for scrolling) separately from signals that also need absolute position accuracy (e.g., for drawing).

[0206] In some implementations, the input support process 1432 receives a gaze direction that may potentially be associated with a different UI element, for example, because it is between UI elements or because UI elements overlap each other. Fuzzy hit testing and other such processes can help disambiguate the user's intent to identify the appropriate UI element in such situations.

[0207] In some implementations, the input support process 1432 recognizes two-handed 3D gestures, such as a two-handed zoom gesture, and provides information to the application corresponding to the multi-touch touchscreen input gesture. For example, doing so can involve generating an interaction gesture for each hand that can be interpreted by the application's 2D touch-based gesture recognition process as two touch points moving closer or further away from each other, which can be interpreted as a pinch-to-zoom touchscreen gesture.

[0208] In some implementations, some applications may be enabled to render using a custom rendering engine, e.g., directly to the display hardware without utilizing OS rendering, and thus provide limited UI element information to the input support process 1432 for use in supporting the input recognition process. Such applications may be authorized to use an API that enables the application to use filtered hand data (e.g., with upsampling, POVc, etc.) and a single gaze ray is provided to the application when an interactive motion is intended (e.g., during a pinch). Doing so may enable some customized application input recognition while still protecting much of the user's gaze direction data. In this example, the application does not receive continuous gaze direction data. For example, such a framework may be suitable for applications migrating from another 3D XR framework that already includes 3D recognition capabilities built into the app, e.g., enabling such an app to be used with little or no modification while still protecting user privacy.

[0209] In some implementations, applications are developed for use on mobile platforms that use single-touch and / or multi-touch input gestures. Applications can be developed by application developers, include calls to gesture APIs and then specify what to do for certain types of touch events, e.g., do X for tap events, Y for swipe events, etc. It may be desirable to use such applications on new or different platforms that provide input modalities other than (or in addition to) single-touch and / or multi-touch input gestures. Rather than implementing all new 3D gestures and gesture recognition processes within applications, some implementations disclosed herein interpret 3D user activity (e.g., hand position, gaze direction, etc.) and send proxy data to applications corresponding to where the touch would be if the 3D user activity had been performed as a 2D touch input gesture. The input support process 1432 can provide such proxy data to such applications, and in doing so, enable a large number of existing 2D / touch-based applications to be easily imported and used within new 3D systems that use new and different 3D user activity-based inputs. The input support process 1432 may make the additional information available so that updated / modified applications can take advantage of / use more information about the user's 3D activities. Alternatively, this information may be utilized by older, unmodified applications that link with an XR-compatible version of the UI framework, which may utilize this additional data on behalf of the app in the background (e.g., a binary-compatible app using a standard long press recognizer may benefit from an XR-based OS implementation that uses the z-depth of a touch to more accurately recognize a long press than purely 2D native methods).

[0210] In some embodiments, to enable the use of both traditional 2D / touch-based applications and newer applications with additional 3D recognition capabilities, the input support process may provide multiple types of information, such as interaction gesture data for supporting touch-based input recognition for traditional applications and manipulator gesture data for supporting applications with additional 3D recognition capabilities.

[0211] Interpreting 3D user activities as 2D touch-based activities may require overcoming various challenges. For example, the movement of a user's hand in 3D space when performing a touch gesture may be equivalent to a touchscreen tap gesture, but may lack the precision expected of a touchscreen tap gesture. A touch on a touchscreen involves contact with a physical surface where the finger rests, while the movement of "virtually" touching a UI in a 3D XR environment may involve the user's hand piercing a UI element. A user may intend to tap the surface of a virtual button, but actually pierce the surface at one location and then retract the hand / finger at a slightly different location. This user activity involving two different UI element locations may be interpreted (by an input support process) as a tap at a single point using various algorithms and / or machine learning processes (e.g., performing trajectory correction). The application receives a single location (from the input support process) and recognizes it as a touch event. Therefore, the input support process 1432 can classify and / or interpret user activities to take into account the unique circumstances of interacting within a 3D environment, and encapsulate the data provided to the application so that the application itself does not need to distinguish between the actions intended by the user and the actions actually performed by the user.

[0212] When providing data to the application, the input support process 1432 can take into account sensor inaccuracies / limitations, such as taking into account the fact that the identified gaze direction may be different from the actual gaze direction. The application itself does not need to take such inaccuracies into account.

[0213] In formatting an application's data in a format that the application will understand, the input support process 1432 can resolve ambiguities, account for inaccuracies, and / or repackage input provided in input modalities that the application itself does not understand / needs to understand into a format that the application does recognize.

[0214] In short, the input support process 1432 can perform various processes to interpret raw 3D data for consumption by applications, so that applications (and their developers) do not need to perform those processes via manually programmed processes. Additionally, performing such processes by a shared OS process can unify the input experience across all apps, so that users receive consistent UI response behavior.

[0215] In addition, the shared input support process 1432 can enable multiple interaction modalities, for example, providing different ways for a user to select a button, and abstracting the user activity data associated with those different modalities for the same type of interaction to provide the same data to the application for different types of input. Thus, as long as the input support process 1432 can abstract new or changed input modality user activity into a format understood by the application, input modalities can be changed and / or added over time without requiring changes to the application.

[0216] In some implementations, an application utilizes a gesture recognizer running within an application process. The gesture recognizer may include a state machine that categorizes input received in an input data stream, e.g., a tap started, a tap in progress, a tap ended. The input support process 1432 may provide data to the application that is recognized by the gesture recognizer running within the application process as a particular type of input (e.g., a tap) that may correspond to a single input modality (e.g., a touch-based input modality) dedicated to the application. The input support process 1432 may configure the data provided to the application to ensure that the application's process accurately recognizes user activity, such as by providing the data in a form expected by the gesture recognizer. In one example, the input support process 1432 converts user activity involving user gaze and gestures into data that can be recognized as touch input. In another example, the input support process 1432 converts user activity involving a 6DOF controller and gaze into data that can be recognized as touch input.

[0217] Application developers do not have to write applications for different input modality platforms. Instead, application developers can write applications for touch platforms (or generic 2D input modalities), and the application can be imported with little or no effort to work within the 3D user activity platform. A single application can be used on mobile devices, laptop devices, tablet devices, desktop devices, and 3D XR devices. The technology disclosed herein can achieve binary compatibility, that is, applications can be executed in different environments that utilize different input modalities.

[0218] In some embodiments, the input support process 1432 provides gesture markers and / or gesture classifications to the application, for example, indicating to the application that the user has raised their hand or that a particular gesture has been recognized as a tap, swipe, scroll, etc., as a hint that the application can use when recognizing a gesture.

[0219] In some implementations, input support process 1432 facilitates an application to recognize gaze-only input without providing raw gaze data to the application. This may involve periodically providing gaze data (i.e., the position on the UI element that the user is gazing at). Gaze-only data may be associated with an interaction state, and data may be provided to the application based on gaze-only data only when the gaze data is associated with a particular interaction state, for example, when a gaze hover state is applicable. Such data may be provided only based on explicit user authorization. For example, when a user gazes at a UI element for more than a threshold amount of time, the application may be notified, and affordances (e.g., points) may be displayed. If the user then views the affordance, the application may be notified that the gazed UI element has been hit / selected. In another example, gazing at a point within a UI element for more than a threshold amount of time may trigger hover feedback, which may be processed outside the process or by the application. Gaze-only input modality may be another input modality that is abstracted as data provided to an application so that the application may interpret the input, for example, using a general or touch-based gesture recognition process.

[0220] In some implementations, the input support process determines the interaction state of each hand (e.g., direct versus indirect) and / or the interaction state associated with the user's gaze (e.g., gaze only or not only gaze).

[0221] Fig.16 1 is a flowchart illustrating a method 1600 for supporting application input recognition using sampling. In some implementations, a device such as electronic device 105 or electronic device 110 performs method 1600. In some implementations, method 1600 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. Method 1600 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, method 1600 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., memory). Method 1600 can be performed at an input support process, for example, via an OS or system-level process.

[0222] At block 1602, method 1600 receives data corresponding to a user activity in a 3D coordinate system. In some implementations, the data corresponding to the user activity is a gaze direction in the 3D coordinate system, the gaze direction being determined based on sensor data, e.g., a gaze direction identified at each frame during an indirect touch. In some implementations, the data corresponding to the user activity is a synthesized direction in the 3D coordinate system. This synthesized direction may be determined based on: determining a hand position in the 3D coordinate system based on the sensor data; determining an intersection position of the hand with at least one UI element based on the hand position; and determining a direction based on the intersection and a viewpoint position (e.g., a ray synthesized from an eye to a touch point on a first frame of a direct touch).

[0223] Data corresponding to user activities may include, but are not limited to, hand data, gaze data, and / or human interface device (HID) data. Various combinations of two or more different types of data may be received, such as hand data and gaze data, controller data and gaze data, hand data and controller data, voice data and gaze data, voice data and hand data, etc. Different combinations of sensor / HID data may correspond to different input modalities. In an exemplary implementation, the data includes both hand data (e.g., a hand posture skeleton identifying 20+ joint positions) and gaze data (e.g., a gaze vector stream), and both hand data and gaze data may be related to identifying input via direct touch input modalities and indirect touch input modalities.

[0224] At box 1604, method 1600 generates multiple sample positions in a 3D coordinate system based on data corresponding to user activities (e.g., based on ray patterns). The 3D coordinate system can combine 3D virtual objects and 2D application content, such as providing a 3D collision volume at a position within the 3D coordinate system. Multiple sample positions can be generated by generating a ray pattern around a gaze direction or a synthetic direction corresponding to the user activity. In various implementations, the ray pattern has 2 to 100 rays, or 5 to 35 rays. In some implementations, the ray pattern has 15 rays, 16 rays, 17 rays, 18 rays, 19 rays, 20 rays, 21 rays, 22 rays, or 23 rays. In some implementations, the ray pattern includes rays that are equally spaced. In some implementations, the pattern or rays form a square shape, a circular shape, a pentagonal shape, a hexagonal shape, an octagonal shape, etc. In some implementations, the ray pattern forms a shape rotated relative to the horizon or horizontal plane, such as a pattern of 15-25 rays in a pattern rotated 5-10 degrees. In various implementations, the pattern may include positions or rays configured to be spaced apart to ensure that UI elements of a particular size and / or shape are detected. The pattern may change over time, such as over time. For example, the pattern for each frame may include randomly positioned rays / points (e.g., gaze rays or gaze ray intersections) generated based on user activity.

[0225] At block 1606, method 1600 identifies a UI target within the 3D coordinate system based on the plurality of sample positions. Identifying the UI target may involve identifying a 3D virtual object (e.g., based on intersection with a collision volume corresponding to the 3D virtual object) and / or a UI element (e.g., based on identifying intersection with a collision volume corresponding to a UI high-level / high-level element to identify a 3D element defined by one or more applications, and then traversing the associated UI hierarchical structure to identify lower-level / lower-level elements).

[0226] Identifying a UI target may involve receiving (e.g., in a UI geometry collision world) data corresponding to the positioning of an application's UI element within a 3D coordinate system, the data corresponding to the positioning of the UI element being based at least in part on data provided by the application (e.g., the position / shape of a 2D element intended for a 2D window area). The application may provide a hierarchical tree, some of which are identified for remote input effects. Such information may be provided to a simulation process that positions the application element in 3D space, for example, by defining the 3D position of one or more collision bodies (e.g., each collision body having a rectangular window area) in 3D space for the application element. The data provided by the application may include a hierarchical tree structure that defines the position and containment relationship of UI elements relative to each other in a 2D coordinate system. The data provided by the application may identify external effects of some of the UI elements, wherein the external effect specifies that the OS process will provide a response to a specified user activity relative to a specified UI element outside the application process (e.g., performing hover feedback on this button outside the process).

[0227] Identifying the UI target may be based on identifying intersections of multiple gaze sample locations with UI elements of the application positioned within the 3D coordinate system.If a random sampling pattern is used for each frame, the intersection locations may be tracked to resample the same locations on subsequent frames.

[0228] At box 1608, method 1600 selects a UI target from the identified UI targets to be associated with the user activity based on the selection criteria. Selecting the UI target to be associated with the UI activity may involve, for each of the identified UI targets, determining a point on the corresponding UI target based on the user activity, and prioritizing the identified UI targets based on the points calculated for each corresponding UI target. Determining the point on each corresponding UI target may involve determining the nearest opaque point to a sample location associated with the user activity (e.g., the location where the gaze direction / ray intersects the UI). This may involve determining the distance (e.g., angular distance) of the nearest opaque point for each of the corresponding UI targets to the sample location associated with the user activity.

[0229] Selecting a UI target to be associated with the UI activity may be based on determining that a nearest opaque point within the UI target is within an angular distance threshold of a sample location associated with the user activity.

[0230] The UI target selected to be associated with the UI activity may be selected based on determining that the nearest opaque point within multiple UI targets is within an angular distance threshold of a sample location associated with the user activity and selecting the UI target from multiple UI targets based on a strategy that ranks UI targets according to element type, UI layer, UI geometry, or hysteresis logic. In some embodiments, when multiple targets are within a threshold of the user activity location (e.g., 1° gaze), the selected UI target is selected based on a strategy that ranks targets by type (e.g., ranking external effects above non-external effect type elements). The strategy may select among elements of the same type based on other criteria, for example, for targets of the same type, prioritizing (1) nested targets and then (2) the nearest distance (angle) to the user activity location. As described with respect to FIG. 13, some embodiments apply hysteresis logic to prioritize previous targets to avoid flickering.

[0231] In some embodiments, UI elements of an application occupy a 2D area, and method 1600 also includes identifying points within the 2D area to an associated application (e.g., an owner of the UI target) based on selecting a UI target to be associated with a user activity, so that the application can use a 2D application action recognition process to identify an action (e.g., selection / hit / hover) to be associated with the UI element.

[0232] In method 1600, the method provides a view of a 3D environment including UI targets and / or other 3D objects. Some or all of the UI targets may be 2D user interface elements provided by one or more applications. The input support process may identify user activity in a 3D coordinate system and provide data to one or more applications (e.g., to a respective owner of each UI element associated with the user activity) to identify 2D user interface input.

[0233] In some implementations, the operating system manages information about virtual and / or real content located within a 3D coordinate system. Such a 3D coordinate system may correspond to an XR environment that represents a physical environment and / or virtual content corresponding to content from one or more applications. An executing application may provide information about the positioning of its UI elements via a hierarchical tree (e.g., a declarative hierarchical tree), where some layers are identified for remote (i.e., outside of the application process) input effects. Such information may be provided to a simulation process (e.g., Fig.14A and Fig. 14B A simulation system 1434 is provided, which positions the application UI element in a 3D coordinate system, for example by defining the 3D position of one or more collision volumes (e.g., each collision volume having a rectangular, cubic, or other shaped window area) in 3D coordinates for the application UI element to be positioned therein.

[0234] Method 1600 may identify data for an application that may include interaction event data. Interaction events may be selected by identifying only certain types of activities, such as user activities determined to correspond to a first type of user activity and one or more other types of user activities. In some implementations, this involves identifying user activities that meet criteria configured to distinguish intentional actions on UI elements from other types of user activities. In some implementations, identifying the first type of user activity involves identifying user activities of the first type that are considered intentional and excluding activity types such as gaze-only activities that are considered unintentional interactions.

[0235] Interaction event data may include interaction pose (e.g., 6DOF data for a point on an application's UI), manipulator pose (e.g., 3D position of a stable hand center or pinch center of mass), interaction state (i.e., direct, indirect, hover, pinch, etc.), and / or which UI element is being interacted with.

[0236] Interaction data may exclude data associated with user activity that occurs between intentional events. Interaction event data may exclude detailed sensor / HID data, such as hand skeletal data. Interaction event data may extract detailed sensor / HID data to avoid providing applications with data that is not necessary for the application to recognize input and that may be private to the user.

[0237] An input support process can respond to some user activities on behalf of an application without necessarily involving or notifying the application of the user activity or the response provided. For example, an input support process can respond to some user activities by adjusting the appearance of displayed application content without notifying the application, such as providing hover feedback based on gaze without notifying the application of the user activity that triggered the feedback or providing hover feedback.

[0238] The application's access to at least some of the data corresponding to the user activity may be suppressed. The application process may identify input to the application based on the data it receives (e.g., based on the interaction event data). The input support process may provide data sufficient for the application to recognize the input while avoiding providing the application with access to user activity data that is not associated with the identified interaction event. Similarly, the input support process may provide abstract data to avoid providing detailed user activity data, such as not providing access to hand skeletal data. The data may be formatted to be recognized by a 2D input recognition process executed within the application, such as by an input recognition process that is configured to recognize abstract input data and / or input corresponding to an input modality from a traditional or different system, such as an input recognition process configured to receive 2D touch-based input.

[0239] Method 1600 may display a view of an XR environment corresponding to a 3D coordinate system, wherein UI elements of an application are displayed in the view of the XR environment. Such an XR environment may include UI elements from multiple application processes corresponding to multiple applications, and the input support process may identify interaction event data for multiple applications and route the interaction event data only to the appropriate application, for example, the application that the user intended to interact with. Accurately routing data only to the intended application can help ensure that one application does not misuse input data intended for another application.

[0240] The OS may provide an OS process (e.g., a shared 3D environment tracking / simulation process) configured to execute method 1600 outside of an application process. The 3D environment / simulation may be provided for tracking virtual content provided by multiple sources, such as multiple system and / or non-system applications provided by the OS itself, by an OS provider and / or a third party, etc. The OS may provide an OS process including a simulation process, which is configured to perform a simulation of a 3D environment based on a physical environment associated with a 3D coordinate system. Such a simulation process positions the UI elements of an application within a 3D coordinate system based on data provided by the application. Such a simulation process may perform the same operation on multiple applications, and may adjust the positioning of such application content within a 3D coordinate system, for example, based on which application the user focuses on, user input, and / or other criteria. In one example, the simulation process positions the UI elements by positioning one or more components within a 3D coordinate system (e.g., positioning a collision body (e.g., each collision body has a rectangular window area, a cube shape, or other shape) using a 3D coordinate system); and positioning the UI elements of the application on / within the one or more components. The positioning of the application's UI elements on the one or more components may be defined based on data provided by the application. The application may not be aware of the positioning of the one or more components within the 3D coordinate system.

[0241] In some specific implementations, the data provided by the application includes a hierarchical tree structure that defines the position and containment relationship of UI elements relative to each other in a 2D coordinate system. In some specific implementations, the hierarchical tree structure defines such positioning for traditional input modalities (e.g., touch screen modalities or 2D desktop / laptop cursor drive modalities). The data provided by the application can identify the external effects of some UI elements in the UI elements. Such external effects can specify that the OS process will provide a response to a specified user activity relative to a specified UI element outside the application process (e.g., performing hover feedback on the button outside the process). The data provided by the application can be provided to the OS process via an inter-process communication link.

[0242] The data corresponding to the user activity may have various formats and be based on or include (but not limited to being based on or including) sensor data or HID data. In some specific implementations, the data corresponding to the user activity includes gaze data, which includes a gaze vector stream corresponding to the gaze direction over time during the use of the electronic device. The data corresponding to the user activity may include hand data, which includes hand posture skeletons for multiple joints at each of multiple moments during the use of the electronic device. The data corresponding to the user activity may include both hand data and gaze data. The data corresponding to the user activity may include controller data and gaze data. The data corresponding to the user activity may include, but is not limited to, any combination of one or more types of data associated with one or more sensors or one or more sensor types, associated with one or more input methods, associated with one or more parts of the user (e.g., eyes, nose, cheeks, mouth, hands, fingers, arms, torso, etc.) or the entire user and / or associated with one or more items worn or held by the user (e.g., mobile devices, tablet computers, laptop computers, laser pointers, handheld controllers, sticks, rings, watches, bracelets, necklaces, etc.).

[0243] In some implementations, the interaction event data that may be provided to an application includes one or more of the following: interaction gestures, including position and / or orientation data of interaction points within an application's UI element; manipulator gestures, including position and / or orientation data corresponding to a hand within a 3D coordinate system (e.g., the 3D position of a stable hand center or a pinch centroid); and / or interaction states, including data identifying the type of interaction. The interaction data may include interaction event data, including interaction gestures, manipulator gestures, and interaction states, and the input recognition process of the receiving application may select which information to use. Some applications may use only some of the data (e.g., a touch-based input recognition process may use only interaction gestures), while other applications may use all of the data.

[0244] The interaction event data may identify the UI element being interacted with during the interaction event. The application may use it to identify which of its UI elements is the target of the user interaction, or may use other data (e.g., identifying which UI element the interaction gesture is on) to identify which of the UI elements is the target of the user interaction.

[0245] In some implementations, the interactivity event data provided to an application excludes data associated with interactivity events associated with applications other than the application.

[0246] In some implementations, method 1600 is performed by an electronic device that is a head-mounted device (HMD), which can provide an XR environment that is a virtual reality environment or an augmented reality environment.

[0247] Some implementations provide applications with output that the applications can recognize as existing touchscreen / 2D input, e.g., a mobile device application does not need to change its own 2D ambiguity hit testing or otherwise account for the ambiguity of the underlying 3D user activity, such as inaccuracies in sensor data tracking gaze (indirectly) or hand / joint position (directly).

[0248] Fig.17 1 is a flow chart illustrating a method 1700 for supporting application input recognition for supporting application user activity recognition. In some implementations, a device such as electronic device 105 or electronic device 110 performs method 1700. In some implementations, method 1700 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. Method 1700 is performed by processing logic (including hardware, firmware, software, or a combination thereof). In some implementations, method 1700 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., memory). Method 1700 can be performed at an input support process, for example, via an OS or system-level process.

[0249] At block 1702 , method 1700 receives data corresponding to user activity in a 3D coordinate system. This may be similar to receiving data corresponding to user activity as described with respect to block 1602 of method 1600 .

[0250] At box 1704, method 1700 receives data corresponding to the positioning of the UI element of the application within a 3D coordinate system (e.g., a UI geometry collision world). The data may correspond to the positioning of the UI element based at least in part on data provided by the application (e.g., the position / shape of a 2D element intended for a 2D window area). The data provided by the application may include a hierarchical tree structure defining the position and containment relationship of the UI elements relative to each other on the 2D coordinate system. The data provided by the application may identify external effects for some of the UI elements, where the external effect specifies that the OS process will provide a response to a specified user activity relative to a specified UI element outside the application process (e.g., performing hover feedback on this button outside the process).

[0251] At block 1706 , method 1700 identifies a UI target within the 3D coordinate system based on the data corresponding to the user activity and the data corresponding to the positioning of the UI element of the application within the 3D coordinate system.

[0252] At block 1708, method 1700 selects a UI target from among the identified UI targets to be associated with the user activity based on the selection criteria.

[0253] At block 1710, method 1700 identifies a point within the 2D region to an application based on selecting a UI target to be associated with the user activity so that the application can associate an action (e.g., select / hit / hover / etc.) with the UI target. Selecting a UI target to be associated with the user activity may involve selecting a process, such as with respect to Fig.16 Those described in box 1608 of .

[0254] The application may include a 2D input recognition framework configured to associate actions with UI targets based on 2D position data. Such a 2D input recognition framework may be configured to recognize (a) touch input corresponding to a touch point on a 2D touch screen interface and / or (b) a 2D mouse cursor position or a 2D trackpad cursor position within a 2D user interface.

[0255] Methods 1600 and 1700 may involve fuzzy hit testing as described and illustrated throughout this disclosure, such as using the exemplary process described with respect to FIG. 12 and elsewhere in this disclosure.

[0256] Fig.18 1 is a block diagram of an electronic device 1800. Device 1800 illustrates an exemplary device configuration of electronic device 110 or electronic device 105. Although some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 1800 includes one or more processing units 1802 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 1806, one or more communication interfaces 1808 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1810, one or more output devices 1812, one or more internal-facing and / or external-facing image sensor systems 1814, a memory 1820, and one or more communication buses 1804 for interconnecting these and various other components.

[0257] In some implementations, the one or more communication buses 1804 include circuitry that interconnects system components and controls communications between system components. In some implementations, the one or more I / O devices and sensors 1806 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0258] In some implementations, the one or more output devices 1812 include one or more displays configured to present a view of a 3D environment to a user. In some implementations, the one or more displays 1812 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, the one or more displays correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. In one example, the device 1800 includes a single display. As another example, the device 1800 includes a display for each eye of the user.

[0259] In some specific implementations, one or more output devices 1812 include one or more audio generating devices. In some specific implementations, the one or more output devices 1812 include one or more speakers, surround sound speakers, speaker arrays, or headphones for generating spatialized sounds such as 3D audio effects. Such devices can virtually place sound sources in a 3D environment, including behind, above, or below one or more listeners. Generating spatialized sounds can involve converting sound waves (e.g., using head-related transfer functions (HRTFs), reverberation, or elimination techniques) to simulate natural sound waves (including reflections from walls and floors), which are emitted from one or more points in a 3D environment. Spatialized sounds can induce the listener's brain to interpret the sound as if the sound occurs at one or more points in a 3D environment (e.g., from one or more specific sound sources), even if the actual sound may be generated by speakers in other locations. One or more output devices 1812 can be configured to generate tactile sensations in addition or alternatively.

[0260] In some implementations, the one or more image sensor systems 1814 are configured to obtain image data corresponding to at least a portion of a physical environment. For example, the one or more image sensor systems 1814 may include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 1814 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 1814 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.

[0261] Memory 1820 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, memory 1820 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. Memory 1820 optionally includes one or more storage devices remotely located from one or more processing units 1802. Memory 1820 includes non-transitory computer-readable storage media.

[0262] In some implementations, the memory 1820 or a non-transitory computer-readable storage medium of the memory 1820 stores an optional operating system 1830 and one or more instruction sets 1840. The operating system 1830 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 1840 includes executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set 1840 is software that can be executed by one or more processing units 1802 to implement one or more of the techniques described herein.

[0263] The instruction set 1840 includes a user interaction instruction set 1842, which is configured to identify and / or interpret user gestures and other user activities when executed, including by performing a fuzzy hit test, as described herein. The instruction set 1840 includes an application instruction set 1844 for one or more applications. In some specific implementations, each of these applications is provided as a separately executed set of code, for example, capable of being executed via an application process. The instruction set 1840 can be embodied as a single software executable file or multiple software executable files.

[0264] Although instruction set 1840 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. In addition, the drawings serve more as functional descriptions of the various features present in a particular implementation, as opposed to the structural schematics of the implementations described herein. As will be appreciated by those of ordinary skill in the art, the items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how the features are distributed among them will vary depending on the implementation, and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0265] It should be understood that the specific implementations described above are cited by way of example, and the present disclosure is not limited to what has been particularly shown and described above. On the contrary, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to a person skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0266] As described above, one aspect of the present technology is to collect and use sensor data that may include user data to improve the user experience of electronic devices. The present disclosure contemplates that, in some cases, the collected data may include personal information data that uniquely identifies a particular person or can be used to identify the interests, characteristics, or tendencies of a particular person. Such personal information data may include motion data, physiological data, demographic data, location-based data, phone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0267] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, personal information data can be used to improve the content viewing experience. Therefore, the use of such personal information data may enable planned control of electronic devices. In addition, the present disclosure also anticipates other uses of personal information data that benefit users.

[0268] The disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for the privacy and security of maintaining personal information data. For example, personal information from users should be collected for the legal and reasonable purposes of entities, and not shared or sold outside these legal purposes. In addition, such collection should only be carried out after the user's informed consent. In addition, such entities should take any required steps to safeguard and protect access to such personal information data, and ensure that other people who can access personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices.

[0269] Regardless of the foregoing, the present disclosure also contemplates specific implementations in which users selectively prevent the use or access of personal information data. That is, the present disclosure contemplates that hardware elements or software elements may be provided to prevent or block access to such personal information data. For example, with respect to a content delivery service customized for a user, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" of participating in the collection of personal information data during registration for the service. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information but allow anonymous information to be transmitted for use in improving the functionality of the device.

[0270] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences or settings based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information.

[0271] In some embodiments, data is stored using a public / private key system that only allows the owner of the data to decrypt the stored data. In some other implementations, data can be stored anonymously (e.g., without identifying and / or personal information about the user, such as legal name, username, time and location data, etc.). In this way, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some implementations, a user can access their stored data from a user device that is different from the user device used to upload the stored data. In these cases, the user may need to provide login credentials to access their stored data.

[0272] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0273] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0274] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0275] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined and / or divided into sub-blocks. Some boxes or processes can be executed in parallel.

[0276] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0277] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0278] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.

[0279] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0280] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method, comprising: At an electronic device having a processor: receiving, in the input support process, data corresponding to user activity in the 3D coordinate system; generating, in the input support process, a plurality of sample positions in the 3D coordinate system based on the data corresponding to the user activity; In the input support process, identifying a user interface (UI) target within the 3D coordinate system based on the plurality of sample positions; as well as In the input support process, a UI target among the identified UI targets is selected based on selection criteria to be associated with the user activity.

2. The method of claim 1 , wherein selecting the UI target to be associated with a UI activity comprises: For each of the identified UI targets, determining a point on the corresponding UI target based on the user activity; as well as The identified UI objects are prioritized based on the points calculated for each corresponding UI object.

3. The method of claim 2, wherein determining the point on each of the corresponding UI targets comprises: determining an opaque point closest to a sample location associated with the user activity; as well as A distance from the nearest opaque point of each of the corresponding UI objects to the sample location associated with the user activity is determined. 4 . The method of claim 2 , wherein selecting the UI target to be associated with the UI activity is based on determining that a nearest opaque point within the UI target is within an angular distance threshold of a sample location associated with the user activity.

5. The method of claim 2, wherein the UI target to be associated with the UI activity is selected based on: determining that a nearest opaque point within a plurality of UI objects is within an angular distance threshold of a sample location associated with the user activity; and The UI target is selected from the plurality of UI targets based on a policy that orders UI targets according to element type, UI layer, UI geometry, or hysteresis logic.

6. The method of claim 1, wherein the identified UI target comprises: 3D virtual objects; and A 2D element defined by one or more applications.

7. The method of claim 1 , wherein identifying the UI target comprises: receiving, at the input support process, data corresponding to a positioning of a UI element of an application within the 3D coordinate system, the data corresponding to the positioning of the UI element being based at least in part on data provided by the application; as well as The UI target is identified by identifying intersections of a plurality of gaze sample locations with the UI element of the application positioned within the 3D coordinate system.

8. The method of claim 7, wherein the UI element of the application occupies a two-dimensional (2D) area, and the method further comprises: Based on selecting the UI target to be associated with the user activity, points within the 2D area are identified to the application so that the application can use a 2D application action recognition process to identify an action to be associated with the UI element.

9. The method of claim 7, wherein the data provided by the application comprises a hierarchical tree structure that defines positions and containment relationships of the UI elements relative to each other on a two-dimensional (2D) coordinate system.

10. The method of claim 7, wherein the data provided by the application identifies external effects for some of the UI elements, wherein the external effect specifies a response that an operating system (OS) process will provide to a specified user activity relative to a specified UI element outside of an application process.

11. The method of claim 1, wherein the data corresponding to the user activity is a gaze direction within the 3D coordinate system, the gaze direction being determined based on sensor data.

12. The method of claim 1 , wherein the data corresponding to the user activity is a resultant direction within the 3D coordinate system, the direction being determined based on: determining a hand position of the hand in the 3D coordinate system based on the sensor data; Determining an intersection position of the hand with at least one UI element based on the hand position; and The direction is determined based on the intersection and a viewpoint position.

13. The method of claim 1, wherein the plurality of sample locations are generated by generating a ray pattern around a gaze direction or a synthetic direction corresponding to a user activity. The method of claim 13 , wherein the ray pattern has 2 to 100 rays.

15. The method of claim 13, wherein the ray pattern has 5 to 35 rays.

16. The method of claim 13, wherein the ray pattern comprises an outer collection of rays forming a certain shape.

17. The method of claim 16, wherein the shape is rotated relative to a horizon or a horizontal line.

18. A method according to claim 1, wherein the electronic device provides a view of a 3D environment including the UI targets, wherein at least some of the UI targets are 2D user interface elements provided by one or more applications, and wherein the input support process recognizes the user activity in the 3D coordinate system and provides data to the one or more applications to recognize 2D user interface input.

19. A system, comprising: Memory; and one or more processors coupled to the memory, wherein the memory includes program instructions that, when executed by the one or more processors, cause the system to perform operations including: receiving, in the input support process, data corresponding to user activity in the 3D coordinate system; generating, in the input support process, a plurality of sample positions in the 3D coordinate system based on the data corresponding to the user activity; In the input support process, identifying a user interface (UI) target within the 3D coordinate system based on the plurality of sample positions; and In the input support process, a UI target among the identified UI targets is selected based on selection criteria to be associated with the user activity.

20. A non-transitory computer-readable storage medium storing program instructions, the program instructions being computer-executable on a computer to perform operations comprising: receiving, in the input support process, data corresponding to user activity in the 3D coordinate system; generating, in the input support process, a plurality of sample positions in the 3D coordinate system based on the data corresponding to the user activity; In the input support process, identifying a user interface (UI) target within the 3D coordinate system based on the plurality of sample positions; as well as In the input support process, a UI target among the identified UI targets is selected based on selection criteria to be associated with the user activity.