Input recognition system capable of protecting privacy of user
By introducing input support processes in the operating system, identifying and processing intentional interaction events of users in the virtual environment, the problem of user privacy protection and input identification complexity in the virtual environment is solved, and efficient and accurate user interaction recognition and privacy is achieved. Protect.
Patent Information
- Application Number
- CN202410714665.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-31
- Filing Date
- 2024-06-04
- Publication Date
- 2024-12-06
AI Technical Summary
When existing user interaction systems identify and process users' virtual elements interactions in three-dimensional space, it is difficult to effectively protect user privacy, and the input recognition is complex, which affects the efficiency and accuracy of the system.
By introducing input support processes into the operating system, receiving three-dimensional coordinate system data and user activity data of user interface elements, identifying intentional interaction events, only providing interaction-related data to the application, eliminating unintentional activities, and using simulated processes to locate UI elements. And adjust application content through inter-process communication to protect privacy.
It realizes efficient and accurate user input recognition in a virtual environment, while protecting user privacy, reducing the complexity of application input recognition, and ensuring the security of user data.
Smart Images

Figure CN119089485A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to evaluating user interactions with electronic devices involving hand and body gestures, gaze, voice, handheld controller input, human interface device (HID) input, and / or other user activities. Background Art
[0002] Existing user interaction systems can be improved in facilitating interactions based on user activities. Summary of the invention
[0003] Various embodiments disclosed herein include devices, systems, and methods for interpreting user activities when a user interacts with virtual elements (e.g., user interface elements) positioned in a three-dimensional (3D) space, such as an extended reality (XR) environment. Some embodiments utilize an architecture that receives application user interface geometry in a system or shared simulation area and outputs data (e.g., less than all user activity data) for an application to use to identify input. An operating system (OS) process may be configured to provide an input support process to support identification of input intended for one or more separately executed applications, for example, by providing some input recognition tasks to identify user activities as input for the application, or by converting user activity data into a format that can be more easily, accurately, efficiently, or effectively interpreted by the application and / or in a manner that is conducive to protecting user privacy. The OS process may include a simulation process that utilizes application user interface information to provide 3D information (e.g., 3D world data), and the input support process uses the 3D information to support identification of input intended for one or more separate (e.g., separately executed) applications.
[0004] User privacy can be protected by providing only some user activity information to individually executed applications, for example, withholding user activity information that is not associated with intentional user actions (such as user actions where the user intends to provide input or certain types of input). In one example, raw gaze data is excluded from the data provided to the application so that when there is no intentional UI interaction, the application receives limited or no information about where the user is looking or what the user is looking at.
[0005] In some specific implementations, the processor performs the method by executing instructions stored on a computer-readable medium. The method executes the application as an application process within the OS of the electronic device. The OS may include an OS process configured to execute an input support process. The input support process may also be executed outside the application process. The OS process may additionally or alternatively include a simulation process configured to perform a shared (i.e., using content from the OS, multiple applications, and / or other sources) simulation of the 3D environment in which the UI of the application is located.
[0006] The method receives data corresponding to the positioning of a user interface (UI) element of an application within a 3D coordinate system (e.g., a UI geometry collision world) at an input support process. The data may correspond to the positioning of a UI element based at least in part on data provided by an application (e.g., application data identifying the position and / or shape of a 2D user interface element intended for a 2D window area). The 3D coordinate system may correspond to an extended reality (XR) environment representing a physical environment and / or virtual content corresponding to content from one or more applications. An executing application may provide information about the positioning of its UI elements via a hierarchical tree (e.g., a tree defining nested or hierarchical relationships between UI elements), some of which are identified for remote (i.e., outside the application process) input effects. Such application UI information may be provided from the application to a simulation process of the OS via inter-process communication, which positions the application element in the 3D coordinate system, for example, by defining a 3D position in 3D space of one or more collision volumes (e.g., each collision volume having a rectangular window area or other area) for the UI element of the application.
[0007] The method receives data corresponding to user activity in a 3D coordinate system at an input support process, the data being associated with one or more input modalities. Data of various types and combinations of types may be received. In one example, such data includes hand data (e.g., a multi-joint hand pose skeleton) and gaze data (e.g., a gaze vector stream). In other examples, such data includes controller data and gaze data.
[0008] The method identifies the data of the application based on the data corresponding to the user activity and the data corresponding to the positioning of the UI element of the application at the input support process. The data of the application may correspond to interactive events that can be selected by identifying only certain types of activities. For example, the method can select data for user activities that are considered to be associated with intentional interactions. Examples of intentional interactions include, but are not limited to, intentional pinching or other gestures when staring at a UI target, intentional hand movements that cause a virtual intersection between a part of the hand and the UI target, intentional controller input, intentional verbal commands, and gaze-only behaviors that are considered to be intentional interactions. Data can be selected in a way that excludes unintentional activities, such as resting the hand in a closed posture, or passively staring at the content without intending to interact. Interaction event data may include interaction gestures (e.g., 6DOF data of points on the UI of the application), manipulator gestures (e.g., 3D positions of stable hand centers or pinching centroids), interaction states (e.g., direct hovering, direct contact, indirect hovering, indirect pinching, etc.), and / or identifying UI elements that are being interacted. Interaction data may exclude data associated with user activities that occur during the time period between intentional events, hand bone detail data, and / or otherwise abstract data to avoid providing private user data to the application. The input support process may respond to some activities by adjusting the appearance of displayed application content without notifying the application, such as providing hover feedback / responses based on the user's gaze direction without notifying the application of those gaze directions.
[0009] The method provides data from an application of an input support process to the application. It can do so without allowing the application to access at least some data corresponding to user activity. The application process can then identify input to the application based on the interaction event data. The input support process can be configured to avoid providing access to user activity data not associated with the identified interaction event, hand skeletal detail data, continuous gaze data during interaction, input provided relative to other applications, and / or other private user data. The interaction data can be formatted to be recognized by a 2D input recognition subprocess within the application, such as by a mobile platform or laptop / desktop platform with an input modality different from that of the device executing the method.
[0010] Some implementations protect user privacy by not sending user activity data to an application unless the data is associated with the intent of an interaction, for example, not sending unintentional data such as gaze-only data to an application. In some implementations, a processor performs a method by executing instructions stored on a computer-readable medium. The method executes an application as an application process within an OS of an electronic device. The OS may include an OS process configured to execute an input support process outside of an application process.
[0011] The method receives, at an input support process, user activity data corresponding to a user activity in a 3D coordinate system.
[0012] The method generates interactive event data for interactive events based on identifying a portion of user activity data corresponding to a first type of user activity at an input support process. This may involve using criteria to identify an input portion during which a user shows an intention to interact with a UI element. The interactive event data may represent at least a subset of user activities corresponding to the first type of user activity. In some specific implementations, interactive event data is only generated to be provided to the application during the time during which it is determined that the user is showing an intention to interact with the UI element of the application. The method may not generate data to be provided to the application, while the user only stares at the content, reads, etc., or other people do not act in a manner indicating an intention to interact with the UI element. Therefore, the application does not need to receive user activity data associated with the time between interactive events. In one example, the method only provides data to the application when it is identified that there is an intention to interact with the application based on identifying that the user provides one or more specific gestures, touchpad clicks or other commands, controller button clicks or other commands, or otherwise performs an activity that is designated (e.g., one or more input modal frameworks used by the device) as an intentional interactive activity. Therefore, in one example, the application receives information based only on the user's gaze at certain points in time rather than on continuous gaze data. Additionally, the data selected for provision to the application may be limited, for example, providing only data such as the point on a UI element that the user is looking at instead of the actual gaze direction vector, or providing only the central hand position instead of all hand joint positions.
[0013] The method provides interaction event data from an input support process to an application, wherein the application process identifies input to the application based on the interaction event data.
[0014] In some implementations, input support is provided to an application in a format that the application can recognize using a non-3D gesture recognition framework (e.g., using an in-application framework that receives 2D touch screen events and uses these events to identify touch screen input). Providing information in this format can facilitate importing applications from traditional 2D systems into a 3D framework where new forms of 3D input are available, for example, enabling importing of mobile applications with little or no changes to the application's own input recognition processes.
[0015] In some specific implementations, the processor executes the method by executing instructions stored on a computer-readable medium. The method executes an application as an application process within the OS of an electronic device. The OS may include an OS process configured to execute an input support process outside the application process. The method receives user activity data corresponding to user activities in a 3D coordinate system at the input support process. The method generates interactive event data based on the user activity data at the input support process. The method provides the interactive event data from the input support process to the application, wherein the application process identifies input to the application based on the interactive event data. The interactive event data represents user activity using a certain format, wherein the format is recognized by a 2D input recognition framework within the application as input to a 2D user interface of an application including UI elements. In some specific implementations, the OS is a 3D operating system, and the 2D input recognition framework is a framework designed for a 2D operating system.
[0016] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to some exemplary embodiments, some of which are illustrated in the accompanying drawings.
[0018] Figure 1A to Figure 1B Example electronic devices are shown operating in a physical environment according to some implementations.
[0019] Figure 2 According to some specific implementations, the device provides Figure 1A to Figure 1B A view of virtual elements within a 3D physical environment, where the user performs direct interaction.
[0020] Figure 3 According to some specific implementations, the device provides Figure 1A to Figure 1B A view of virtual elements within a 3D physical environment, where the user performs indirect interactions.
[0021] Figure 4Exemplary direct interactions according to some implementations are shown.
[0022] Figure 5 Indirect interaction identification according to some implementations is shown.
[0023] Figure 6 Use of an exemplary input support framework to generate interaction data based on hand and gaze data and UI target data is shown according to some implementations.
[0024] FIG. 7A to FIG. 7B An exemplary architecture for receiving user activity data and application user interface information in a system process that outputs interaction event data for use by one or more applications in identifying input is shown according to some specific implementations.
[0025] Figure 8 is a process flow diagram illustrating an exemplary process for providing direct and indirect input using hand and eye data according to some implementations.
[0026] Fig. 9 is a flow chart illustrating a method for supporting application input recognition according to some specific implementations.
[0027] Fig.10 is a flow chart illustrating another method for supporting application input recognition according to some specific implementations.
[0028] Fig.11 is a flow chart illustrating another method for supporting application input recognition according to some specific implementations.
[0029] Fig.12 is a block diagram of an electronic device according to some specific implementations.
[0030] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION
[0031] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.
[0032] Figure 1A to Figure 1B Exemplary electronic devices 105 and 110 are shown operating in physical environment 100. Figure 1A to Figure 1B In the example of , the physical environment 100 is a room including a table 120. The electronic devices 105 and 110 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 100 and objects therein, as well as information about the user 102 of the electronic devices 105 and 110 and evaluate the physical environment and objects therein. The information about the physical environment 100 and / or the user 102 can be used to provide visual and audio content and / or to identify the current location of the physical environment 100 and / or the location of the user within the physical environment 100.
[0033] In some implementations, a view of an extended reality (XR) environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via electronic devices 105 (e.g., wearable devices such as HMDs, etc.) and / or 110 (e.g., handheld devices such as mobile devices, tablet computing devices, laptop computers, etc.). Such an XR environment may include a view of a 3D environment generated based on a camera image and / or a depth camera image of the physical environment 100 and a representation of the user 102 based on the camera image and / or the depth camera image of the user 102. Such an XR environment may include virtual content positioned at a 3D position relative to a 3D coordinate system (i.e., 3D space) associated with the XR environment, which 3D coordinate system may correspond to the 3D coordinate system of the physical environment 100.
[0034] In some implementations, video (e.g., pass-through video depicting a physical environment) is received from an image sensor of a device (e.g., device 105 or device 110). In some implementations, a 3D representation of the virtual environment is aligned with a 3D coordinate system of the physical environment. The size of the 3D representation of the virtual environment can be generated based on, among other things, the scale of the physical environment or the positioning of open spaces, floors, walls, etc., so that the 3D representation is configured to align with corresponding features of the physical environment. In some implementations, a viewpoint within the 3D coordinate system can be determined based on the position of the electronic device within the physical environment. The viewpoint can be determined based on, among other things, image data, depth sensor data, motion sensor data, etc., which can be retrieved via a virtual inertial odometer system (VIO), a simultaneous localization and mapping (SLAM) system, etc.
[0035] Figure 2 Shows the Figure 1A to Figure 1B100, wherein the user performs direct interaction. In this example, the user 102 makes gestures relative to content presented in views 210a-210b of an XR environment provided by a device (e.g., device 105 or device 110). The views 210a-210b of the XR environment include an exemplary user interface 230 of an application (i.e., an example of virtual content) and a drawing 220 of a table 120 (i.e., an example of real content). Providing such a view may involve determining 3D properties of the physical environment 100 and positioning the virtual content, such as the user interface 230, in a 3D coordinate system corresponding to the physical environment 100.
[0036] exist Figure 2 In the example of , user interface 230 includes various content user interface elements, including background portion 235 and icons 242, 244, 246, 248. Icons 242, 244, 246, 248 can be displayed on flat user interface 230. User interface 230 can be a user interface of an application, as shown in this example. For illustrative purposes, user interface 230 is simplified, and the user interface can actually include any complexity, any number of content items and / or a combination of 2D and / or 3D content. User interface 230 can be provided by various types of operating systems and / or applications, including but not limited to messaging applications, web browser applications, content viewing applications, content creation and editing applications, or any other application that can display, present, or otherwise use visual and / or audio content.
[0037] In this example, a background portion 235 of the user interface 230 is flat. In this example, the background portion 235 includes all aspects of the user interface 230 being displayed except for the icons 242, 244, 246, 248. Displaying the background portion of the user interface of the operating system or application as a flat surface can provide various advantages. Doing so can provide a user interface that is easy to understand or otherwise easy to access the application using a portion of the XR environment. In some specific implementations, multiple user interfaces (e.g., corresponding to multiple different applications) are presented sequentially and / or simultaneously within the XR environment (e.g., within one or more collision volumes or other such components).
[0038] In some implementations, the position and / or orientation of such one or more user interfaces may be determined to facilitate visibility and / or use. Within a 3D environment, one or more user interfaces may be in a fixed position and orientation. In such a case, user movement will not affect the position or orientation of the user interface within the 3D environment.
[0039] The position of the user interface within the 3D environment may be based on determining the distance of the user interface from the user (e.g., from an initial or current user position). The position and / or distance from the user may be determined based on a variety of criteria, including but not limited to criteria that consider application type, application functionality, content type, content / text size, environment type, environment size, environment complexity, environment lighting, the presence of other people in the environment, use of the application or content by multiple users, user preferences, user input, and numerous other factors.
[0040] In some implementations, one or more user interfaces may be body-locked content, e.g., at a distance and orientation offset relative to a portion of a user's body (e.g., their torso). For example, body-locked content of a user interface may be 0.5 meters away and 45 degrees to the left of a forward-facing vector of a user's torso. If the user's head turns and the torso remains stationary, the body-locked user interface will appear to remain stationary at 2 meters away and 45 degrees to the left of the torso's forward-facing vector in the 3D environment. However, if the user does rotate their torso (e.g., by rotating in their chair), the body-locked user interface will rotate with the torso and reposition within the 3D environment so that it remains 0.5 meters away and 45 degrees to the left of the new forward-facing vector of their torso.
[0041] In other implementations, user interface content is defined at a specific distance from the user, where it remains fixed relative to the user's orientation (e.g., if initially displayed in a cardinal orientation, it will remain in that cardinal orientation regardless of any head or body movement). In this example, the orientation of the body-locked content is not referenced to any part of the user's body. In this different implementation, the body-locked user interface will not reposition itself based on torso rotation. For example, a body-locked user interface may be defined as 2m away and may be initially displayed to the north of the user based on the direction the user is currently facing. If the user rotates their torso 180 degrees to face south, the body-locked user interface will remain 2m away from the north of the user, with north now directly behind the user.
[0042] The body-locked user interface may also be configured to always remain aligned with the gravity line or horizon, so that head and / or body changes in roll orientation will not cause the body-locked user interface to move within the 3D environment. Translational movements will cause the body-locked content to reposition within the 3D environment so as to maintain the distance offset.
[0043] exist Figure 2In the example of FIG. 2 , user 102 moves their hand from an initial position, as shown by the location of drawing 222 in view 210a. The hand moves along path 250 to a later position, as shown by the location of drawing 222 in view 210b. As user 102 moves their hand along this path 250, the finger intersects user interface 230. Specifically, as the finger moves along path 250, it virtually pierces icon 246 and thus a tip portion of the finger (not shown) is obscured by user interface 230 in view 210b.
[0044] Implementations disclosed herein interpret user movement, such as user 102 moving their hand / finger along path 250 relative to a user interface element, such as icon 246, to identify user input / interaction. Interpretation of user movement and other user activity may be based on identifying user intent using one or more recognition processes.
[0045] exist Figure 2 In an example of , recognizing input may involve determining that the gesture is a direct interaction and then using a direct input recognition process to recognize the gesture. For example, such a gesture may be interpreted as a tap input to icon 246. When making such a gesture, the user's actual movement relative to icon 246 may deviate from the ideal movement (e.g., a straight path passing through the center of the user interface element in a direction completely orthogonal to the plane of the user interface element). The actual path may be curved, jagged, or otherwise nonlinear, and may be at an angle to the plane of the user interface element rather than orthogonal. The path may have properties that make it similar to other types of input gestures (e.g., sliding, dragging, flicking, etc.). For example, non-orthogonal motion may make the gesture similar to a sliding motion, in which the user provides input by piercing the user interface element and then moving in a direction along the plane of the user interface.
[0046] Some embodiments disclosed herein determine that a direct interaction mode is applicable, and based on the direct interaction mode, utilize a direct interaction recognition process to distinguish or otherwise interpret user activity corresponding to direct input, such as recognizing intended user interaction based on whether and how a gesture path intercepts one or more 3D regions of space. Such a recognition process may take into account actual human tendencies associated with direct interaction (e.g., a natural arching of the hand that occurs during an intended straight motion, a tendency to move based on a shoulder or other pivot position, etc.), human perception issues (e.g., a user cannot see or know the exact position of virtual content relative to their hand), and / or other direct interaction specific issues.
[0047] Note that the user's movements in the real world (e.g., physical environment 100) correspond to movements within a 3D space (e.g., an XR environment based on the real world and including virtual content such as a user interface positioned relative to real-world objects including the user). Thus, the user moves his hand in the physical environment 100, for example, through empty space, but the hand (i.e., a drawing or representation of the hand) intersects and / or pierces the user interface 300 of the XR environment based on the physical environment. In this way, the user directly interacts virtually with the virtual content.
[0048] Figure 3 Shows the Figure 1A to Figure 1B 305 of the XR environment includes Figure 2 An exemplary user interface 230 of FIG. Figure 3 In the example of , user 102 makes a pinch gesture with their hands as shown in drawing 222 while gazing at user interface icon 246 along gaze direction 310. In this example, this user activity (e.g., the pinch gesture and the gaze on the UI element) corresponds to a user intent to interact with user interface icon 246, i.e., the pinch indicates an intent to interact, and the gaze (at the time of the pinch) identifies the target of the interaction.
[0049] Implementations disclosed herein interpret user activity (such as user 102 forming a pinch gesture in conjunction with a gaze at a UI element) to identify a user / interaction. For example, such user activity may be interpreted as a tap input to icon 246, e.g., selecting icon 246. However, when performing such actions, the user's gaze direction and / or timing between the gesture and the gaze that the user intended to be associated with the gesture may not be perfectly executed and / or timed.
[0050] Some embodiments disclosed herein determine that an indirect interaction mode is applicable and, based on the direct interaction mode, utilize an indirect interaction recognition process to recognize intended user interactions based on user activity (e.g., based on whether and how a gesture path intercepts one or more 3D regions of space). Such a recognition process may take into account actual human tendencies associated with indirect interactions (e.g., eye glances, eye fixations, and other natural human gaze behaviors, hand arching movements, retractions that do not correspond to intended insertion directions, etc.), human perception issues (e.g., users cannot see or know the exact location of virtual content relative to their hands), and / or other indirect interaction specific issues.
[0051] Some implementations determine an interaction mode, e.g., a direct interaction mode or an indirect interaction mode, such that user behavior can be interpreted by a dedicated (or otherwise separate) recognition process for the appropriate interaction type, e.g., using a direct interaction recognition process for direct interactions and an indirect interaction recognition process for indirect interactions. Such dedicated (or otherwise separate) process utilization may be more efficient, more accurate, or provide other benefits relative to using a single recognition process configured to recognize multiple types of (e.g., both direct and indirect) interactions.
[0052] Figure 2 and Figure 3 An exemplary interaction mode based on user activity in a 3D environment is shown. Other types or modes of interaction may be used additionally or alternatively, including but not limited to user activity via input devices such as keyboards, trackpads, mice, handheld controllers, etc. In one example, a user provides an interaction intent via an activity (e.g., performing an action such as tapping a button or a trackpad surface) using an input device such as a keyboard, trackpad, mouse, or handheld controller, and identifies a UI target based on the user's gaze direction when inputting on the input device. Similarly, user activity may involve voice commands. In one example, a user provides an interaction intent via an activity (e.g., performing an action such as tapping a button or a trackpad surface) using an input device such as a keyboard, trackpad, mouse, or handheld controller, and identifies a UI target based on the user's gaze direction at the time of a voice command. In another example, user activity identifies the intention of the interaction (e.g., via pinching, gestures, voice commands, input device input, etc.), and determines a UI element based on a non-gaze-based direction (e.g., based on where the user is pointing within the 3D environment). For example, a user may pinch with one hand to provide input indicating an interaction intent while pointing at a UI button with a finger of another hand. As another example, a user may manipulate the orientation of a handheld device in a 3D environment to control a controller direction (i.e., a virtual line extending from the controller within the 3D environment), and may identify a UI element with which the user is interacting based on the controller direction (i.e., based on identifying what UI element the controller direction intersects when receiving input indicating an interaction intent).
[0053] Various specific implementations disclosed herein provide an input support process, for example as an OS process separate from the executing application, that processes user activity data (e.g., regarding gaze, gestures, other 3D activity, HID input, etc.) to generate data for the application that the application can interpret as user input. The application may not need to have 3D input recognition capabilities because the data provided to the application can be in a format that the application can recognize using 2D input recognition capabilities, such as those used within applications developed for use on 2D touch screens and / or 2D cursor-based platforms. Therefore, at least some aspects of interpreting the user activity of the application can be performed by a process external to the application. Doing so can simplify or reduce the complexity, requirements, etc. of the application's own input recognition process, ensure unified, consistent input recognition across multiple different applications, protect private use data from application access, and numerous other benefits as described herein.
[0054] Figure 4 An exemplary direct interaction involving a user's hand 422 virtually touching a UI element of a user interface 400 is shown. In this example, a user 102 is using a device 105 to view and interact with an XR environment including the user interface 400. A direct interaction recognition process may use sensor data and / or UI information to determine, for example, which UI element the user's hand is virtually touching and / or where the interaction occurs on the UI element. A direct interaction may additionally (or alternatively) involve evaluating user activity to determine the user's intent, for example, whether the user intends to perform a direct tap gesture through a UI element or a sliding / scrolling motion along the UI element. Such recognition may utilize information about UI elements, for example, about position, size, element type, types of interactions that can be performed on the element, types of interactions enabled on the element, which of a set of potential target elements for user activity accepts which types of interactions, and the like.
[0055] The identification of such interactions may be based on functions performed both via system processes and via application processes. For example, an input support process of the OS may interpret hand data from a sensor of the device to identify an interaction event and provide limited or interpreted information about the interaction event to an application providing user interface 400. For example, instead of providing detailed hand information (e.g., identifying the 3D positions of multiple joints of a hand model representing the configuration of hand 422), the OS input support process may simply identify a 2D point on UI element 515 where an interaction (e.g., an interaction gesture) occurred within 2D user interface 400. The application process may then interpret the 2D point information (e.g., interpret the information as a selection, a mouse click, a touch screen tap, or other input received at that point) and provide a response, e.g., modifying its UI accordingly.
[0056] Figure 5Indirect interaction recognition is shown. In this example, sensor data and / or UI information on device 105 is used to recognize a user interaction made by user 102, for example based on outward-facing image sensor data, depth sensor data, eye sensor data, motion sensor data, etc. and / or information available by an application providing a user interface. The sensor data may be monitored to detect user activity corresponding to an engagement condition corresponding to the start of a user interaction.
[0057] In this example, the process detects that the user 102 has positioned their hand 422 within the field of view of the outward-facing image sensor at block 510. The process may detect a particular hand configuration (e.g., claw, flat hand, stable hand in any configuration, etc.) as an indication of hand engagement, or may simply detect the presence of the hand within the field of view of the sensor.
[0058] At block 520 , the process uses the user gaze data to identify an object. In this example, the process identifies that the gaze direction 505 of the user 102 is on the user interface element 515 .
[0059] At box 530, the process displays feedback based on the object identified by the gaze. In this example, the feedback graphically distinguishes the user interface element 515 to indicate that the user interface element 515 now has a different state (e.g., a "hover" state similar to the state of a traditional UI icon when the cursor is on the item without clicking / tapping). In this example, the application providing the UI information does not need to be notified of the hover state and the associated feedback. Instead, the hand engagement, object identification via gaze, and display of feedback can be processed, for example, by an operating system process outside the process (i.e., outside the application process). For example, such a process can be provided via an input support process of the operating system. Doing so can reduce or minimize potentially sensitive user information (e.g., such as a constant gaze direction vector) that might otherwise be provided to the application to enable the application to handle these functions within the application process. Whether and how to display feedback can be specified by the application, even if it is executed outside the process. For example, the application may define that an element should display hover feedback or highlight feedback, and define how the hover or highlight will appear, such that an out-of-process party (e.g., the operating system) may provide the hover or highlight according to the defined appearance. Alternatively, feedback may be defined out-of-process (e.g., defined only by the OS) or defined to use a default appearance / animation if the application does not specify an appearance.
[0060] At block 540, the process identifies a gesture to be associated with the identified object. In this example, the user gazes at user interface object 515 in gaze direction 505 while simultaneously (or at least close in time) performing a pinch gesture with hand 422. In this example, the pinch gesture is interpreted as initiating an action on user interface object 515, e.g., resulting in a cursor "click" event similar to a traditional UI icon (during which a cursor is positioned over the icon and a trigger such as a mouse click or trackpad tap is received) or a selection action similar to a touchscreen "tap" event.
[0061] The recognition of such interactions may be based on functions performed both via system processes and via application processes. For example, an input process of the OS may interpret hand and gaze data from a device's sensors to identify an interaction event and provide limited or interpreted / abstracted information about the interaction event to an application providing user interface 400. For example, instead of providing gaze direction information identifying gaze direction 505, an OS input support process may identify a 2D point within 2D user interface 400 on UI element 515, e.g., an interaction gesture. The application process may then interpret the 2D point information (e.g., interpret the information as a selection, mouse click, touch screen tap, or other input received at that point) and provide a response, e.g., modifying its UI accordingly.
[0062] Figure 5 An example of identifying indirect user interactions is shown. Numerous other types of indirect interactions may be identified, for example, based on one or more user actions to identify user interface elements and / or provide input (e.g., no-action / hover type input, selection type input, input with direction, path, speed, acceleration, etc.). Inputs in 3D space similar to inputs on a 2D interface may be identified, for example, inputs similar to mouse movements, mouse button clicks, touch screen touch events, trackpad events, joystick events, game controller events, etc.
[0063] Some implementations utilize an out-of-process (i.e., outside of the application process) input support framework to facilitate accurate, consistent, and efficient input recognition in a manner that protects private user information. For example, aspects of the input recognition process can be performed out-of-process so that the application has little or no access to information about where the user is looking, such as gaze direction. In some implementations, application access to some user activity information (e.g., data based on gaze direction) is limited to specific types of user activities, such as activities that meet specific criteria. For example, an application may be restricted to receiving only information associated with intentional or deliberate user activities (e.g., intentional or deliberate actions that indicate an intent to interact with a user interface element (e.g., select, activate, move, etc. a user interface element)).
[0064] Some implementations recognize input using functional elements executed via both an application process and a system process external to the application process. Thus, in contrast to a framework where all (or most) input recognition functionality is managed within an application process, some of the algorithms involved in input recognition may be moved out of process, i.e., outside of the application process. For example, this may involve moving algorithms that detect gaze input and intent out of the application's process so that the application cannot access user activity data corresponding to where the user is looking, or in some cases only access such information, e.g., only for specific instances where the user has expressed intent to interact with a user interface element.
[0065] Some implementations use a model in which an application declares or otherwise provides information about its UI elements to recognize input, so that system processes outside of the application process can better facilitate input recognition. For example, an application can declare the location and / or UI behaviors / capabilities of its buttons, scroll bars, menus, objects, and other UI elements. Such declarations can identify how the user interface should behave given different types of user activity, for example, whether a button should (or should not) exhibit hover feedback when the user views it.
[0066] A system process (e.g., outside of an application process) can use such information to provide desired UI behavior (e.g., providing hover feedback in the appropriate context of user activity). For example, a system process can trigger hover feedback for a UI element based on a declaration from an application that the application's UI includes the element and that the UI should, for example, display hover feedback when gazed at. The system process can provide such hover feedback based on identifying a triggering user activity (e.g., gazing at a UI object), and can do so without revealing to the application user activity details associated with the user activity that triggered the hover, the occurrence of the user activity that triggered the hover feedback, and / or providing the hover feedback. The application may not be aware of the user's gaze direction and / or that hover feedback is provided for a UI element.
[0067] In another example, an application declares a menu as part of its user interface and declares that the menu can be extended using the identified set of extended menu options. The system process can handle the extension of the UI menu. The system process can provide menu expansion (e.g., via a system process outside the application process) based on identifying a triggering user activity (e.g., gazing at a label of a menu), and can do so without revealing to the application user activity details associated with the user activity that triggered the menu expansion, the occurrence of the user activity that triggered the menu expansion, and / or the fact that the menu was expanded. The application may not be aware of the user's gaze direction and / or that the menu was expanded.
[0068] Some aspects of input recognition may be handled by the application itself (i.e., in process). However, the system process may filter, abstract, or otherwise manage information that is made available to the application to recognize input to the application. The system process may do so in a manner that facilitates efficient, accurate, consistent (within the application and across multiple applications) input recognition, and allows the application to potentially use an input recognition process that is easier to implement and / or a legacy input recognition process, such as an input recognition process developed for a different system or input environment (e.g., using a touch screen input process used in a traditional mobile application).
[0069] Some implementations use a system process to provide interactive event data to applications so that these applications can recognize input. The interactive event data can be limited so that all user activity data is unavailable to these applications. Providing only limited user activity information can help protect user privacy. The interactive event data can be configured to correspond to events that can be recognized by applications using general or traditional recognition processes. For example, the system process can interpret 3D user activity data to provide interactive event data that the application can recognize in the same way that the application will recognize touch events on a touch screen. In some implementations, the application receives interactive event data corresponding only to certain types of user activities (e.g., intentional or deliberate actions on user interface objects), and may not receive information about other types of user activities (e.g., gaze activity only, users move their hands in a way that is not associated with UI interaction, users move closer to or farther away from the user interface, etc.). In one example, during a period of time (e.g., one minute, 10 minutes, etc.), the user gazes around the 3D XR environment, including gazing at certain user interface text, buttons, and other user interface elements, and eventually performs intentional UI interactions, such as by making an intentional pinch gesture while gazing at button X. The system process can handle all user interface feedback during gazes around various UI elements without providing application information about these gazes. On the other hand, the system process can provide interaction event data to the application based on an intentional pinch gesture when gazing at button X. However, even this interaction event data can provide limited information to the application, for example, providing an interaction position or posture that identifies an interaction point on button X without providing information about the actual gaze direction. The application can then interpret the interaction point as an interaction with button X and respond accordingly. Therefore, user behavior that is not associated with the user's intentional interaction with the UI element (for example, only gaze hovering, menu expansion, reading, etc.) is processed outside the process without the application being able to access user data, and information about intentional user interface element interactions is limited so that the information does not include all user activity details.
[0070] Figure 6An exemplary input support framework 640 is shown for generating interaction data based on hand data 610, gaze data 620, and UI target data 630 to produce interaction data 650, which can be provided to one or more applications and / or used by system processes to provide a desired user experience. In some implementations, the input support process 640 is configured to understand the intent of the user interaction, generate input signals and events to create a reliable and consistent user experience across multiple applications, detect out-of-process input, and responsibly route it through the system. The input support process 640 can, for example, arbitrate which application, process, and / or UI element should receive user input based on identifying which application or UI element is the intended target of the user activity. The input support process 640 can keep sensitive user data (e.g., gaze, hand / body registration data, etc.) private; only abstracted or high-level information is shared with the application.
[0071] The input support process can obtain hand data 610, gaze data 620, and UI target data 630 and determine the user interaction state. In some implementations, it is in a user environment where multiple input modalities are available to the user (e.g., where the user can Figure 2 Direct interaction as shown in Figure 3 For example, the input support process may determine that the user's right hand is performing an intentional pinch and gaze interaction with the user interface element, the left hand is directly tapping the user interface element, or the left hand is shaking and therefore idle / not doing anything related to the user interface.
[0072] Based on determining the user interaction intent, the input support framework 640 can generate interaction data 650 (e.g., including interaction gestures, manipulator gestures, and / or interaction states). The input support framework can generate input signals and events that the application can consume without requiring customization or 3D input recognition algorithms in the process. In some specific implementations, the input support framework provides the interaction data 650 in a format that the application can consume as a touch event on a touch screen or as a touchpad tap using a 2D cursor at a specific location. Doing so can enable the same application (with little or no additional input recognition process) to interpret interactions across different environments, including new environments for which the application was not originally created and / or using new and different input modalities. In addition, the application response to input can be more reliable and consistent across applications in a given environment and across different environments, for example, achieving consistent UI responses for 2D interactions with applications on tablet computers, mobile devices, laptops, etc. and for 3D interactions with applications on HMDs and / or other 3D / XR devices.
[0073] The input support framework may also manage user activity data so that different applications are unaware of user activity relative to other applications, e.g., one application will not receive user activity information while the user is typing a password into another application. Doing so may involve the input support framework accurately identifying which application the user's activity corresponds to and then routing the interaction data 650 to only the correct application. Applications may utilize multiple processes to host different UI elements (e.g., using an out-of-process photo picker) for various reasons (e.g., privacy). The input support framework may accurately identify which process the user's activity corresponds to and route the interaction data 650 to only the correct process. The input support framework may use details about the UIs of multiple potential target applications and / or processes to disambiguate input.
[0074] Fig. 7A An exemplary architecture is shown for receiving user activity data and application UI information in a system process that outputs interaction event data for use by one or more applications to recognize input.
[0075] In this example, sensor system 710 and AR system 720 may be considered as a sensing layer, e.g., determining where the user's hands are in the 3D world, where the user is gazing in the 3D world, etc., and OS process 730 and application 740 may be considered as an interaction layer, e.g., determining what the user is interacting with, how the user is interacting, etc.
[0076] The sensor system 710 may include various sensors, including but not limited to color / RGB image sensors, grayscale image sensors, depth sensors, dynamic vision sensors, motion sensors, etc. The sensor system may include an image signal processor (ISP) component and / or other components that process sensor data.
[0077] The sensor system 710 can provide sensor data to the XR system 720 in various forms. In some implementations, the sensor data is sent over time, periodically, and / or at a fixed rate (e.g., at X frames of sensor data per second (fps)). In one example, hand data based on images captured by one or more outward-facing image sensors (e.g., on a device such as an HMD) is sent to the XR system 720 at a fixed rate (e.g., 10fps, 20fps, 30fps, 60fps, 120fps, etc.). In another example, such hand data is sent at a non-fixed rate or otherwise irregularly. In one example, gaze data based on images captured by one or more inward-facing image sensors (e.g., inside a device such as an HMD) is sent to the XR system 720 at a fixed rate (e.g., 10fps, 20fps, 30fps, 60fps, 120fps, etc.). In another example, such gaze data is sent at a non-fixed rate or otherwise irregularly. The hand data and gaze data may be sent to the XR system 720 at the same or different rates and / or at the same or different times. The sensor data may include additional types of information and / or may provide information about other parts of the user or the physical environment in which the user is located. Such other sensor data may be provided at the same or different rates and / or at the same or different times as the hand data and / or gaze data.
[0078] The XR system 720 performs user activity monitoring and / or tracking using the received sensor data. In one example, the XR system 720 is configured to provide a relatively low-level tracking algorithm. For example, the hand system 722 of the XR system 720 may use the sensor data to perform a hand tracking algorithm to track the position, posture (e.g., position and orientation), configuration (e.g., shape), or other aspects of the hand over time. The hand system 722 may generate, update, and / or track a 3D model of the hand, such as a hand model that represents the shape of the hand using defined "joints" or nodes (which may or may not correspond to the user's physiological hand joints). In one example, a hand model of 20+ joints is maintained over time based on the sensor data, so that the hand data generated by the hand system 722 represents the current position / posture / configuration of the hand at different points in time, which may enable determination of 3D movements or other changes made by the hand over time. The hand system may alternatively (or additionally) track a subset of points on the surface of the user's hand, for example, tracking the position of one or more fingertips / thumb tips of the user's hand. Such tracking may, but need not, include determining when such parts (e.g., fingertips / thumb tips) are in contact with each other and / or other parts of the user or the physical environment. The hand system 722 may output hand position, posture, and / or configuration information and confidence values corresponding to such hand data.
[0079] The gaze system 724 of the XR system 720 can use the sensor data to execute a gaze tracking algorithm to track eye characteristics over time, such as, but not limited to, gaze direction. The gaze system 724 can use the sensor data to directly (e.g., without modeling the shape of the eyes and / or head) determine the gaze direction of one or both eyes. The gaze system 724 can use the sensor data to generate, update, and / or track a 3D model of the eye, such as an eye model that represents the shape of the eye based on the location of the identification point (e.g., the center of the eyeball, the center of the cornea, the center of the pupil, etc.), the size (e.g., the diameter of the eyeball, the diameter of the pupil, etc.), and / or the surface portion of the eye. In one example, the gaze system 724 outputs a gaze direction stream (e.g., a vector in 3D space or relative to the current position of the user) for each eye over time. In one example, the gaze system 724 outputs a gaze direction and a confidence value corresponding to such a gaze direction.
[0080] The XR system 720 may include a computer vision (CV) system 726 that supports or otherwise supports the hand system 722, the gaze system 724, and / or other XR system subsystems (not shown). For example, the CV system 726 may execute one or more environment and / or user assessment algorithms, such as performing simultaneous localization and mapping (SLAM) to model a 3D physical environment and keep track of the user's position relative to the model. In another example, the CV system 726 may identify objects such as walls, doors, tables, appliances, etc. within the physical environment and / or the locations of such objects. In another example, the CV system 726 may execute a semantic understanding algorithm to semantically tag objects within the physical environment. In another example, the CV system 726 may use associated sensor data to assess user characteristics (e.g., the type of activity currently being performed (e.g., exercise, work, meditation, etc.), posture / motion (e.g., sitting, standing, walking, running, driving, etc.), and / or other user characteristics). Such user characteristic data may be used by the hand system 722 and / or the gaze system 724 to improve their efficiency and / or accuracy.
[0081] exist Fig. 7A In the embodiment of the present invention, the XR system 720 provides hand / gaze data to the operating system process 730. The hand / gaze data can be provided periodically (e.g., at a fixed frame rate corresponding to the sensor capture rate or otherwise) or irregularly. In one example, the hand data is provided at the same frame rate per second as the gaze data. In another example, the hand data is provided at a different frame rate than the gaze data.
[0082] The hand / gaze data received by the operating system process 730 may be used by the input support process 732 to (1) provide / initiate some response to the user activity on behalf of the application 740 itself and / or (2) provide some information (e.g., interaction data) to the application 740 to enable the application 740 to respond to the user activity. The input support process 740 / operating system process 730 may provide information about the first type of user activity (e.g., activity determined to correspond to an intentional user interaction with a UI element) to the application. Thus, as Fig. 7A As shown, input support process 732 generates interaction data that is provided to application 740. This interaction data provided to application 740 may exclude or transform / abstract hand / gaze data (and other user-based information). Application 740 may only receive interaction data and may therefore not receive data about other types of user activity, such as user activity that is considered to be something other than intentional interaction with UI elements, such as user activity in which the user is simply gazing at other parts of the UI or XR environment or during which the user is reading text displayed within the application's UI. Note that in Fig. 7A In this example, the interaction data provided by the input support process 732 is provided to the application 740 via the simulation system 734. However, in other examples, the input support process 732 may provide such interaction data to the application 740 directly or via a different intermediary.
[0083] In some implementations, the input support system 732 uses information about the user interfaces of one or more applications, such as application 740, to provide input support to those applications. In some implementations, application 740 sends user interface information to the input support system 732, which uses the user interface information to interpret user interactions associated with application 740. For example, application 740 may provide information defining the appearance of a rectangular area that contains a set of user interface elements at a specified location and has certain desired interaction capabilities (e.g., selectable, non-selectable, hoverable, non-hoverable, expandable, non-expandable, etc.). Application 740 may define the type, location, visual appearance, functional characteristics, or other aspects of such elements for use by the input support system 732. Application 740 may provide information about its UI elements to the input support system 732, which defines what information the application 740 will receive. For example, based on defining a button component, when the input support system recognizes user activity (e.g., pinching and gazing, etc.) as an intentional interaction with the button element, application 740 may receive input events associated with the button.
[0084] In some implementations, it may not be desirable for an input support process (e.g., a hit test process) to receive and / or use the entire UI, for example, doing so may be computationally or resource intensive and / or complex. In some implementations, less than all UI information is provided to the input support process. In some implementations, high-level information (e.g., window / collider geometry) may be shared with the input support process, and the input support process may query / request more information as needed. For example, the input support process may identify user activity / input and perform hit testing based on high-level UI information that has been provided by one or more applications or otherwise. In one example, a hit test of a collider identifies an external effect, and the input support process may then request more information from the OS's simulation system and / or the application providing the UI with the element, for example, its specific geometry, or more information about any attached gesture recognizers. In this exemplary method, the application still communicates some or all information to the system (e.g., a simulation system provided by the OS), for example, by defining external effects, for example, communicating everything to the system as a whole. However, the input support process need not receive or use comprehensive information, but may query for more details about the UI when it finds (eg, via hit testing, fuzzy hit testing, etc.) that more information is needed.
[0085] In some implementations, the input support system 732 may use information about the positioning of application user interfaces and / or user interface elements within such user interfaces to better understand the user's activities and / or intent and ultimately provide a more accurate, more enjoyable, or otherwise better experience for the user. For example, the input support system 732 may use information about the positioning of application user interfaces and / or user interface elements within such user interfaces to (a) distinguish user activities associated with a first type of user activity (e.g., intentional interaction events) from other types of user activities, (b) determine which user activities to respond to directly and which user activities the application will respond to, and thereby selectively provide information to the application 740 limited to the user activities that the application 740 itself will respond to, (c) respond to a user activity associated with the application 740 (e.g., provide hover feedback without notifying the application 740), and / or (d) target the user activity to one of multiple applications that the user activity may potentially be intended for.
[0086] In the example of Figure 730, the simulation system process 734 provides UI geometry (e.g., collision world) data to the input support process 732 to enable the input support process 732 to better support input to applications and / or other XR environment elements. The simulation system process 734 may generate, update and / or maintain information about items within the 3D XR environment, for example, maintaining a current understanding / snapshot view of everything within the environment. This may involve determining where virtual content will be positioned within the XR environment based on the user's physical environment. For example, the simulation system process 734 may determine that the UI of a first application will be positioned above the surface of a user's physical table in the XR environment, and the UI of a second application will be positioned in front of the window to the side of the user's table.
[0087] The simulation system process 734 can determine where to locate the space of the application UI content to be rendered within the 3D XR environment and how to configure the space (e.g., by determining the container size and dimensions) based on the information provided by the application. Fig. 7A In an example, application 740 provides UI information (e.g., which may include UI element declarations), which the simulation system process 734 uses to determine where the spatial positioning of the application's user interface is and how to configure the space. In one example, configuring the application's UI involves determining the location of one or more containers / colliders (e.g., one or more flat rectangular or other 2D shaped windows or one or more square or other 3D shaped bounding areas) where the application's UI will be positioned in 3D space. The simulation system process 734 may position containers for each of the applications based on the UI information (e.g., UI declarations) provided by one or more applications. It may take into account the surrounding environment (e.g., the size of the room or other characteristics of the XR environment), the user's location, activity, and preferences, and numerous other considerations in determining where and how to organize and manage the 3D positioning of objects within the XR environment. Applications do not need to know (or be informed of) the location of their user interfaces within the 3D XR environment.
[0088] The simulation system process 734 can not only determine the location of user interface container / collision volume features such as a window containing all application content, but also determine other user interface elements that may be involved in user interaction and user activity. Such user interface elements include, but are not limited to, text elements, buttons, sliders, scroll bars, pickers (e.g., color pickers), menu controls, timelines, images, vector graphics, rulers, icons, and labels.
[0089] Tracking user interface element positions within the 3D XR environment and providing such information to the input support process 732 may enable the input support process 732 to more efficiently, accurately, and effectively support input processes, including, but not limited to, supporting input to applications such as application 740. Moreover, using such information may additionally enable the input support process 732 to do so in a manner that protects the user's privacy by limiting information about the user's activities that is exposed to application 740.
[0090] For example, the input support process 732 can use hand data from the hand system 722 to determine that the user is not currently making a gesture indicating an interaction intent (e.g., the hand is not pinching), and use the gaze direction from the gaze system 724 to determine that the user is gazing at a specific button within the user interface of the application 740 within the 3D environment maintained by the simulation system process 734. The input support process 732 can use this to initiate an out-of-process (e.g., outside the application process) response, for example, the simulation system process 734 can provide hover feedback to the user that highlights the button. The application 740 does not need to receive any information about the user's current user activity (e.g., the user's hand state and / or gaze state), and does not even need to know that the hover feedback is provided by the system. In this example, the input support process 732 uses the UI geometry information provided by the simulation system 734 and based on the UI information provided by the application 740 to provide a response to the user activity on behalf of the application 740 without involving the application itself. Avoiding providing user activity data about certain types of user activities (e.g., unintentional activities) can help protect user private data from being affected by the application 740.
[0091] In another example, the input support process 732 may use hand data from the hand system 722 to determine that the user is not currently making a gesture indicating an interaction intent (e.g., the hands are not pinching), and use the gaze direction from the gaze system 724 to determine that the user is gazing at a menu title within the user interface of the application 740 within the 3D environment maintained by the simulation system process 734. The input support process 732 may use this to initiate an out-of-process (e.g., outside the application process) response, for example, the simulation system process 734 may provide the user with an expansion of a menu that displays previously hidden menu options. The application 740 does not need to receive any information about the user's current user activity (e.g., the user's hand state and / or gaze state), or even know that the menu is expanded. In this example, the input support process 732 uses the UI geometry information provided by the simulation system 734 and based on the UI information provided by the application 740 (e.g., identifying the menu and its menu options and their positions in 3D space) to provide a response to the user activity on behalf of the application 740 without involving the application itself. Avoiding providing user activity data regarding certain types of user activities (eg, unintentional activities) may help protect user private data from application 740 .
[0092] The input support process 732 may additionally (or alternatively) use the UI geometry information (provided by the simulation system 734 and based on the UI information provided by the application 740) to provide information to the application 740 to respond to the user activity itself. In some specific implementations, such information is limited to user activities associated with the first type of user activity, for example, user activities associated with intentional interactions with user interface elements. The input support process 732 may provide information (e.g., interaction data) to enable the application 740 to respond to the user activity itself. Such information may process or limit the user activity data (e.g., hand / gaze data received from the XR system 720) so that the application 740 does not obtain detailed user data, such as data about a specific user gaze direction, user hand size / shape, etc.
[0093] For example, the input support process 732 may use hand data from the hand system 722 to determine that the user is currently making a gesture indicating an interaction intent (e.g., the hands are pinching), and use the gaze direction from the gaze system 724 to determine that the user is gazing at a particular button within the user interface of the application 740 within the 3D environment maintained by the simulation system process 734. Based on determining that the user activity (e.g., pinching and gazing) meets the criteria that qualifies as a first type of user activity (e.g., intentional interaction with a UI element), the input support process 732 may generate data (e.g., interaction data) that is different from the raw hand data and the raw gaze data to send to the application 740 to enable the application 740 to respond to the user activity itself. The application 740 does not need to receive the raw hand data and / or raw gaze data associated with the user's current activity, and does not even need to know that the user activity is gaze / hand based. Instead, the data provided to the application 740 may simply be sufficient to enable the application 740 to recognize an input event (e.g., a hit event) to a button of the user interface of the application 740. Such data may have been abstracted to use an input modality agnostic format or a single input modality format that may be different from the input modalities available on the device (e.g., using a touch screen input modality format). Avoiding providing detailed user activity (e.g., detailed hand or gaze data) can help protect user private data from application 740.
[0094] The data provided to the application 740 to enable the application 740 to respond to input to its UI elements can have various forms. In some specific implementations, such data is limited to certain types of user activities, and therefore the data format reflects this, for example, in the case where data is to be provided to the application 740 in response to an intended UI interaction event, data defining the interaction event can be provided to the application. In one example, only information identifying the UI element interacted with and the event type (e.g., button X received a hit type interaction) is provided to the application 740.
[0095] In other specific implementations, the application 740 has an input recognition framework for a specific input environment (e.g., a 2D input environment), and the data provided to the application 740 mimics the format of the input environment. For example, the application 740 may be configured with a 2D input recognition framework, wherein the application 740 is configured to recognize 2D touch input on a touch screen device, for example, receiving interactive gesture data identifying the touch point and direction of the user's touch on the touch screen interface. Note that the term "gesture" here refers to such information that identifies the 2D position on the touch screen and / or the direction associated with the touch-it may involve receiving only the 2D position or it may involve receiving 2D position and direction data. In this example, data that mimics touch events may be provided to such an application 740 having a 2D input recognition framework. For example, the input support process 732 may use hand data from the hand system 722 to determine that the user is currently making a gesture indicating an intention to interact (e.g., the hand is pinching), and use the gaze direction from the gaze system 724 to determine that the user is gazing at a specific button within the user interface of the application 740 within the 3D environment maintained by the simulation system process 734. The input support process 732 can generate interaction data identifying the 2D location of the gaze direction relative to the user interface element of the application, and provide it to the application 740 as an interaction gesture so that the application 740 can interpret it as a touch event (e.g., a tap) at that location (on the button) and initiate an appropriate response. The application 740 only needs to receive the interaction gesture without receiving raw hand or gaze data. In this example, the application 740 knows the interaction point (in its 2D space), but does not know (and does not need to know for input recognition purposes) the gaze direction in the 3D space of the XR environment.
[0096] In some implementations, events may include optional "extra data," e.g., a 2D touch point may identify a 2D location and additional 3D depth information. Such extra information may be utilized in various ways. For example, if a developer rebuilds an application, they may configure the application to use this extra data. As another example, even if the developer does not rebuild the application, the underlying UI framework maintained by the OS provider / application execution environment may utilize this extra data inside its gesture recognizers, since the application will typically call higher-level APIs that are themselves built natively for the OS and linked to at runtime.
[0097] In some implementations, use of the input support process 732 enables execution of the application 740 in an environment different from its original or intended environment. For example, the application 740 may be originally compiled or intended for execution on a mobile device with a 2D touch screen input environment or a laptop computer with a 2D mouse / trackpad driven environment. The application 740 may be executed within a device that provides a 3D input modality and receive data (corresponding to those 3D input modalities that the application 740 cannot directly recognize) from the input support process 732 in a format that the application 740 can recognize (e.g., as 2D input corresponding to touch screen or mouse / trackpad driven input). An application intended for a mobile device, laptop computer, or other device may be executed within a 3D interaction enabled HMD environment without requiring significant (or any) modifications to the application's input recognition process. In one example, the HMD is configured with binary compatibility for a mobile device and / or laptop device (e.g., enabling it to execute binary or object code executable by the mobile device and / or laptop device), and provides enhanced input capabilities to mobile applications and / or laptop applications executing on the HMD by utilizing an input support process 732 that provides data based on user activities in a 3D environment that the application can recognize as 2D modality-based input.
[0098] In the above example, additional information may be provided to the application 740. For example, the application 740 may receive information about the position of the pinched hand, e.g., a manipulator pose. Such hand information may be at a higher level than the raw hand data. For example, the application 740 may receive a manipulator pose that identifies the position and / or orientation of the hand in 3D space, without receiving information about the configuration of the hand and / or information about a 3D model (e.g., of joints) used to represent the position, pose, and / or configuration of the hand in 3D space. In another example, the application 740 may receive information about the interaction state, e.g., identifying the type of interaction determined by the input support process 732.
[0099] In the above examples, the criteria for identifying a first type of user activity (e.g., an activity associated with an intentional user element interaction) involves evaluating whether the user's hands exhibit a particular configuration (e.g., a pinch gesture), and based on identifying a hand exhibiting such a configuration, identifying other concurrent user activities, for example, identifying where the user is gazing (or approaching) at that time. Such a pinch may be based on criteria for evaluating the proximity of parts of the user's hand model to each other (e.g., the proximity of the fingertips to the thumb tip, etc.), using a classifier or other algorithm to label or classify the user's hand configuration, or otherwise determined by processing hand data. Other types of user activities and / or criteria may be used to identify a first type of user activity (e.g., an activity associated with an intentional user element interaction). For example, a voice command may be identified as an indicator of an intentional activity, for example, identifying a keyword or phrase such as "select" or "hit" or "tap" or "click this", and then associating the gaze direction and / or other user activities occurring during or near the utterance time of the keyword or phrase with an interaction intent, for example, using other activities to identify a UI element target on which an action will be taken.
[0100] The input support process 732 may additionally account for sensor-based or other inaccuracies in hand and / or gaze data. Tracking the position of user application interface elements within the 3D XR environment and providing such information to the input support process 732 may enable it to account for such inaccuracies. In addition, it may be desirable to use a system (shared) process so that such inaccuracies can be accounted for consistently and effectively across multiple applications. In other words, it may be desirable to have a single shared process perform such corrections rather than having individual applications do so. In one example, a user gazes at a button, but the gaze system 724 generates a gaze direction of the user that is slightly outside the button (e.g., 0.5 degrees outside). The input support process 732 may correct this error. For example, it may be determined that the gaze may actually be directed at the button because it is within a threshold (e.g., 1 degree) of the button and there is nothing else nearby. The input support process 732 may correct for this when providing data to the application 740 that enables the application to respond to user activities. For example, instead of providing an interaction gesture that is slightly outside the button, it may provide an interaction gesture that is within the button (e.g., at the edge of the button). Application 740 therefore does not need to account for gaze inaccuracies in its own input recognition processes, e.g., it does not need to determine itself whether an interaction gesture just outside a button should be considered an interaction gesture within the button. This may be particularly useful if application 740 uses a framework from another input paradigm (e.g., a touch screen paradigm that utilizes a different (potentially much smaller) inaccuracy / error threshold). Such an application would not have to implement different thresholds for different inaccuracy levels expected in different input modalities. Having input support processes (e.g., OS processes) correct for such inaccuracies can provide more consistent and accurate results without requiring application developers to devote significant resources to addressing such inaccuracies and / or differences between different input modalities.
[0101] Application 740 can provide UI information to operating system process 730 in various formats. In some implementations, application 740 declares its UI elements, for example, declaring a hierarchy of UI elements within its user interface. Such declarations may include information / instructions that enable operating system process 730 to respond to a user activity on behalf of the application. In some implementations, application 740 declares external effects for certain UI elements, for example, declaring that button X should display hover feedback but button Y should not display hover feedback. Application 740 can use external effect declarations to specify the intended behavior of application 740 for some or all of its UI elements. An application may provide a hierarchical declaration structure (for example, Core (CA) structure), which declares UI element positions, sizes, types, hierarchical relationships, transparent portions, layered effects, special effects, and / or any other information that facilitates the functions provided by the simulation system process 734 and / or the input support process 732. The application 740 may provide such information to the operating system process 730 through inter-process communication (or otherwise).
[0102] The input support process 732 can use such information (e.g., application declarations of UI elements and / or external effects) to better interpret user activity. For example, a given gaze may be between two UI elements, one with an external effect and the other without, and the input support process 732 may move / snap the gaze to the UI element with the external effect because it is more likely to be the appropriate / expected UI element to which a UI response should be associated. External effects may be executed out-of-process without involving the application 740.
[0103] In some implementations, application 740 is enabled to request user permission for specified user data (e.g., detailed hand data and / or detailed gaze data) and, if explicitly approved by the user, is enabled to receive such information.
[0104] In various implementations, the hand system 722 may generate different types of data, including but not limited to timestamp data, joint position data, POV correction (e.g., to joint positions), world transformation data, joint confidence data, palm center data, palm normal direction data, hand motion data, hand radius data, pinch data, object detection data (e.g., about objects being held or touching the hand), and occlusion data, such as occlusion probability data about a joint being occluded, and therefore its data may be less accurate. Similarly, in various implementations, the gaze system 726 may generate different types of data, including but not limited to timestamp data, world transformation data, binocular gaze data, gaze confidence data, gaze tracking state data, gaze direction data, gaze origin data, pupil center data, and pupil diameter data. In some implementations, the XR system includes a frame support module that implements better frame dropping support via input frame queuing.
[0105] Input support process 732 can access rich and detailed hand and gaze data, and use this rich and detailed information to support accurate, efficient, and consistent input responses both within and outside of application processes. However, it may be desirable to keep the details of such rich and detailed user data outside of applications (such as application 740) to prevent such applications from knowing information about the user that the user considers private, such as what the user reads, what content they look at and for how long, how fast the user reads, how big the user's hands / fingers are, hand bone length, missing fingers, how the user interacts with other applications or aspects of the XR environment, etc.
[0106] Figure 7B Another exemplary architecture for receiving user activity data and application UI information in a system process that outputs interaction event data for one or more applications for identifying input is shown. In this example, sensor system 710 and AR system 720 can be considered as a sensing layer, such as determining where the user's hands are in the 3D world, where the user is gazing in the 3D world, how the user is using an input device such as a trackpad or controller, etc., and OS process 730 and application 740 can be considered as an interaction layer, such as determining what the user is interacting with, how the user is interacting, etc.
[0107] Sensor 710 and XR system components (e.g., hand system 722, gaze system 724, and CV system 726) are similar to those described with respect to Fig. 7A Those discussed. In addition, a human interface device (HID) 728 (e.g., a touchpad, a 3D mouse, a handheld controller, etc.) provides device HID data to the HID system 736. Such device data may correspond to 2D / 3D motion or position data representing the control of a point, object, ray, or other form of input affordance located within the XR environment. For example, a user may manipulate the position and orientation of a handheld controller to direct a ray / vector within the XR environment toward a UI element. The HID 728 may include multiple input mechanisms, such as one for positioning a ray and another for indicating an intention to interact with an object to which the ray is directed. In this example, a user may manipulate the HID 728 and see a visualization of a ray extending from the device in a certain direction. The user may manipulate the device to point a ray at a button on the user interface of the application, and press a physical button on the HID 728 (when the ray is directed at the button) to indicate the intention of the button interaction, such as initiating a click, tap, hit, etc. on the button.
[0108] exist Figure 7BIn the example of , the HID system uses motion data to provide HID data to both the simulation system 734 (which can position affordances (e.g., rays, points, etc.) within the 3D XR environment) and the input support process 732 (which can use the HID data to identify intended interactions, respond to some interactions on behalf of one or more applications, and / or provide data to one or more applications to enable one or more applications to respond to HID events without providing detailed or otherwise private user data to the applications. In other embodiments, information pointing to the input support process 732 can be provided (e.g., without sharing the HID data with the reality support system 734). In some embodiments, the interaction model involving the use of the HID device is included in the interaction state / signal derived by the input support process 732 (e.g., the simulation system 734 will receive interaction data from the input support process 732, such as touchpad plus gaze interaction, rather than raw touchpad HID event data.
[0109] In one example, the input support process 732 recognizes that a ray direction from a HID device intersects a UI element of an application and initiates an out-of-process UI response (e.g., hover feedback on the element) without providing information about the user activity to the application.
[0110] In another example, the input support process 732 identifies a first type of user activity (e.g., an intentional UI element interaction type of activity) based on HID data that meets certain criteria (e.g., including intentional physical button presses, occurring when a keyword / key phrase is spoken, occurring when another hand is pinched, etc.). The input support process 732 sends interaction data to the application based on the user activity. It should be noted that the same interaction data can be triggered by different types of user activities, such as user activities using different interaction modalities (such as direct touch, indirect touch, HID-based, etc.). As described above, the input support process can encapsulate data provided to the application 740 in a form that the application 740 can recognize as input, without requiring the application to have a built-in input recognition process specific to some or all input modalities. For example, the application 740 may not include 3D input recognition based on controller ray direction + button click, and the input support process can encapsulate interaction data associated with such input in a format that the application can understand, such as 2D touch-based input / 2D mouse cursor-based input to a touch screen.
[0111] In an alternative implementation, the HID data is provided directly to the simulation system 734 and / or the application 740 without the support of the input support process 732 .
[0112] In some embodiments, the HID 728 is a trackpad and the input support process 732 fuses gaze direction and trackpad touch data, such that the trackpad touch provides an indication of an intended interaction with a UI element and the gaze direction is used to identify a target (e.g., a UI element) to be associated with the intended interaction.
[0113] In some implementations, the HID events are provided to a pointer UI process (which may be separate from OS process 730 and / or application 740). The pointer UI process may control the positioning of affordances (e.g., point representations, shape representations, ray representations, etc.) displayed to the user via UI framework 742b. In some implementations, pointer UI process 750 and UI framework 742 are within application 740, i.e., executed in process by application 740.
[0114] Figure 8 is a process flow diagram illustrating an exemplary process for providing direct and indirect input using hand and eye data.
[0115] In this example, hand data is input to the hand processing block 810. The upsampling and prediction block 812 upsamples the hand data (e.g., adds more frames) and makes predictions about future hand positions, paths, trajectories, velocities, accelerations, etc. In some implementations, in the hand input process, the internal state of the upsampler is updated with new observation data from the computer vision-based tracking system. In the interaction detection process, data is received about when the next frame will be presented, which calculates an interpolated / predicted timestamp and queries the upsampler for joint data corresponding to that timestamp. These steps may occur asynchronously and / or at different frame rates. The process may typically query the upsample at different timestamps between updates from the computer vision tracking system for several predictions, for example, hand tracking may run at 30Hz while the display may render at 90Hz.
[0116] The hand and POVc blending block 814 can use the corrected and uncorrected hand joint data to mitigate POVc-coupled hand jumps that may occur, for example, due to background depth and camera switching. The hand and POVc blending block can calculate new hand pose information to account for the difference between where the user's hand is in the real world (e.g., tracked by a computer vision system) and where the hand appears to be from the perspective of the user viewing the display. The algorithm can smoothly blend the real and POV-corrected versions of the hand skeleton to create hand pose information that is optimized for input recognition in a usage scenario where the alignment of the hand with the UI is important (e.g., during direct input, where the finger must be aligned with the button). When the user's hand is outside the user's field of view (FOV) and / or the field of view of the sensor, it may not make sense to spend resources, or corrections may not be provided in other ways, resulting in jumps in the hand position. The process can smoothly blend hand data over time, for example, based on whether the user is in direct or indirect mode, whether the hand is close to a UI element, or other factors.
[0117] Eye data is input to gaze processing block 820. Gaze detection block 822 removes glance, blink and / or other gaze loss events (e.g., leaving only the gaze fragment corresponding to gaze attentively). Exemplary glance detection process can use gaze confidence, tracking state, pupil center, pupil diameter, interpupillary distance (IPD), gaze ray data and speed data to detect glance and blink for removal and / or to identify gaze attentively for gaze interaction. It can be distinguished between the gaze event as gaze attentively and the gaze event as glance so as to facilitate more accurate input based on gaze. Gaze filtering block 824 filters and / or smoothens gaze data signal. This can relate to smoothing gaze data during gaze attentively. Filtering can be closely coupled to gaze motion classification. Gaze processing block 820 can lock / remember the last smooth gaze position for those intermediate time periods, or can utilize additional algorithm / strategy to "fill" the data between gaze attentively.
[0118] The processed hand and eye data is provided to the interaction detection block 830. Note that the interaction detection block 830 may additionally use UI geometry data including data about the user interface of the application. The UI geometry data may be provided at a different rate than the hand and / or eye data. In one example, UI data is received at a first rate (e.g., 90fps), while hand and gaze data are received asynchronously at a slower average rate (e.g., 60fps). The interaction detection block 830 may additionally use frame target times, such as the system's prediction of when to render the frame. The interaction detection block 830 may, for example, run asynchronously when UI updates and / or gaze / hand data are received.
[0119] The gaze blur hit test and focus area 832 block identifies virtual elements (e.g., UI elements) with which the user's gaze is believed to be associated, as well as the focus area, for example, based on head or device orientation / direction. An exemplary gaze blur hit test process can use gaze ray data, confidence data, gesture data (e.g., hand movement classification), gaze clustering expansion data, etc. to relax / tighten the gaze area / cone based on the accuracy of gaze tracking and / or user behavior. The process can utilize UI geometry from the simulation system 734 based on UI information provided by applications such as application 740, for example, to identify interactive targets (e.g., which UI elements are associated with a given user activity) based on declared external effects or UI elements. The use of such UI information can facilitate more accurate hit testing. In addition to UI geometry, blur hit testing can query more detailed data, for example, data about external effects, gestures, or other contexts.
[0120] The accidental rejection block 834 identifies hand and gaze behavior instances that are rejected (i.e., associated with subconscious user behavior that is not an intentional interaction) but are still associated with virtual elements under appropriate conditions. It attempts to understand the intention behind the user's movement to better interpret the movement as, for example, an intentional UI element interaction or other thing. An exemplary accidental rejection process can use hand joint data, interaction state history, gaze focus area data, occlusion data, etc. to reject (or cancel) unintentional user interactions. The process can utilize the UI geometry from the simulation system 734, which is based on UI information provided by an application such as application 740, for example, based on UI elements or their declared nearby external effects to distinguish between intentional user activities and unexpected activities. The use of such UI information can promote more accurate accidental rejection. For example, based on (according to the data declared by the application) determining that the UI of the application is 2D / planar, user movement in the Z direction can be identified as accidental, and therefore the movement in the Z direction is unlikely to be intentionally input to the UI, that is, it may be an accidental movement that triggers UI interaction unintentionally, and therefore the input support process 732 can be more confident in rejecting user activities as accidental.
[0121] Some implementations support two interaction modes (e.g., direct interaction mode and indirect interaction mode). Some implementations support other combinations of interaction modes (e.g., direct interaction mode, indirect interaction mode, gaze-only interaction mode, and / or peripheral device interaction). A process can be used to determine which, if any, of several interaction models a user is executing. This can involve identifying and ignoring unexpected behavior, and also disambiguating which interaction model is expected when the user wants to interact intentionally. In this example, the direct vs. indirect mode detection block 836 uses the processed hand and eye data to determine the interaction mode used to interpret the user's activity, such as selecting a direct interaction mode or an indirect interaction mode.
[0122] Then, based on the detected interaction mode, the pipeline dedicated to the mode is run to generate signals and events that drive the interaction. For example, for the direct interaction mode, the hand and eye data (processed and / or unprocessed) are provided to the direct interaction recognition block 840, which provides direct input to the user interface. The trajectory correction block 842 can adjust the trajectory of the user's movement, such as adjusting the user's hand movement to take into account the user's behavior in the 3D environment, such as to make the motion data better correspond to the user's expected movement / expected interaction. The hover, make, interrupt block 844 can use gestures, motion paths, speeds, accelerations, etc. to perform various recognition processes to identify certain types of direct interactions with user interface elements, such as "hover" interactions based on detecting that a hand / finger is close to but not touching a user interface element, "make" interactions based on detecting the point (in 3D space / time) at which the hand / finger has contacted the user interface element, "interrupt" interactions based on detecting the point (in 3D space / time) at which the hand / finger stops contacting the user interface element, etc.
[0123] Many types of direct interactions can be recognized and provided as user input. In another example, a direct interaction is recognized by recognizing that a user is in contact with a slider user interface element, moves their hand to the left a distance X while in contact with the slider, and then retracts their hand to break contact with the slider. The slider can be moved to the left, for example, a distance X, based on this input. In another example, a user makes a similar motion to provide input that imparts velocity on a user interface element, such as providing a 3D swipe gesture on a user interface to cause the user interface to begin moving and continue moving after the hand breaks contact, where the continued motion is based on the velocity already imparted on the UI object, such as similar to when a user swipes upward on a multi-touch interface of a mobile phone.
[0124] If the indirect interaction mode is selected, the (processed and / or unprocessed) hand and eye data is provided to the indirect interaction identification block 850, which provides indirect input to the user interface. The user-centric motion block 852 takes into account user-centric motion in the hand data (e.g., taking into account the fact that the user may be performing a motion that pivots around a rotation point (e.g., shoulder, elbow, wrist, etc.)). The user-centric motion block 852 can be used to map the user's hand motion to an object or affordance separate from the hand, such as an affordance displayed on the application content.
[0125] The pinch and gaze fusion block 854 determines which gaze data will be associated with a user activity indicating an intention to interact with a UI element (such as a pinch gesture). Various types of indirect interactions can be recognized and provided as user input. In one example, the user pinches the fingers together and moves the pinched hand while gazing at the UI element to provide a movement input to the UI element. In another example, the user quickly pinches and releases while gazing at the UI element to provide a selection event (e.g., similar to a touch screen tap or mouse click input event).
[0126] like Figure 8 The input support process shown in the example of may perform one or more core functions. In some specific implementations, these functions include, but are not limited to, obtaining gaze data and associating the user's gaze with UI elements associated with one or more applications. Doing so may involve performing a fuzzy hit testing process. Doing so may involve creating a focus area. A focus area may be considered a spatial area / region. Larger objects may be partially inside / partially outside the focus area, so that the system needs to test where the user is interacting on the object to understand whether the interaction point is within the user's focus area. The focus area may include some or all parts of a set of zero or more UI elements that a user may focus on at a given time. Focus areas can be used to disambiguate between interaction types (e.g., direct and indirect input modalities), disambiguate which application a user is interacting with, and / or disambiguate which UI elements are the targets of the user's intent to interact with UI elements.
[0127] In some implementations, the input support process 732 relies heavily on the simulation system 734 to provide a geometric representation of the user interface for hit testing against hand data and gaze data. The input support process 732 can use the collision world (e.g., 3D geometric shape abstraction) provided by the simulation system 734. The input support process 732 can use user interface priority information (e.g., hierarchical display ordering, transparency information, etc.). The input support process 732 can utilize a separate hit test process that produces a 3D world hit test result (RE). The input support process 732 can utilize a hit test process to determine the hit / UI element target associated with the user activity using an application-provided hierarchical tree (e.g., declaring UI elements, relationships, and / or presenting information such as which areas are transparent, presented on top, etc.).
[0128] In some specific implementations, executing Figure 8The functionality shown is to provide an input support process 732 that generates data for an application. The input support process can generate an interaction state of one or more UI elements, such as identifying whether the interaction is direct or indirect, whether the interaction is a hover / close proximity interaction (e.g., associated with user activity that is not linked to UI interaction behavior) or a gesture / pinch / touch interaction (e.g., associated with user activity that indicates an intentional UI behavior).
[0129] In some implementations, the input support process 732 provides interaction gestures, such as tracking correction points on a UI element that are configured to drive a touch screen tap-type input.
[0130] In some implementations, the input support process 732 provides, for example, a manipulator pose corresponding to the position and / or orientation of the hand itself. The manipulator pose can provide a 3D position of a stable hand center or pinch centroid. The manipulator pose can provide the position and / or orientation of the manipulator driving the interaction, for example, for direct touch interaction, index fingertip information (e.g., position), and for indirect pinch interaction, pinch information (e.g., pinch centroid).
[0131] In some implementations, the input support process 732 provides an activity target (eg, a UI element).
[0132] In one example, the user initiates an interaction by pinching while gazing at a UI element. The application receives an interaction gesture on the UI element and recognizes the user's intention to interact with the UI element. The user continues to pinch and moves their hand to the left. The application receives more interaction gestures, for example, a set of positions on the user interface based on the 3D position of the hand's movement when it moves to the left, and responds by moving the associated UI element to the left. In this example, by pinching while gazing at a UI object and then moving the pinched hand to the left, the user provides an input that the application can recognize to move the object to the left. The application can respond accordingly without receiving (and in fact not receiving) information about the user's gaze direction and / or specific hand characteristics. In this example, the application only receives interaction gesture data and can therefore be an application that can only receive interaction gesture data as a proxy for touch input data. Manipulator gesture data such as the 3D position of the hand when it moves to the left can be provided to the application in addition or alternatively, and the application can determine the movement of the associated object based on the changed manipulator gesture position. Applications can be provided with information based on user changes, for example, providing user-centric acceleration deltas similar to delta updates for a trackpad, which communicates the amount the user has moved from one frame to another, rather than absolute position. For example, communicating changes via a separate API can help us optimize a set of signals to accurately represent motion (e.g., for scrolling) separately from signals that also need absolute position accuracy (e.g., for drawing).
[0133] In some implementations, the input support process 732 receives a gaze direction that may potentially be associated with different UI elements, for example, because it is between UI elements or because UI elements overlap each other. Fuzzy hit testing and other such processes can help disambiguate the user's intent to identify the appropriate UI element in such situations.
[0134] In some implementations, the input support process 732 recognizes two-handed 3D gestures, such as a two-handed zoom gesture, and provides information to the application corresponding to the multi-touch touchscreen input gesture. For example, doing so can involve generating an interaction gesture for each hand that can be interpreted by the application's 2D touch-based gesture recognition process as two touch points moving closer or further away from each other, which can be interpreted as a pinch-to-zoom touchscreen gesture.
[0135] In some implementations, some applications may be enabled to render using a custom rendering engine, e.g., directly to the display hardware without utilizing OS rendering, and thus provide limited UI element information to the input support process 732 for use in supporting the input recognition process. Such applications may be authorized to use an API that enables the application to use filtered hand data (e.g., with upsampling, POVc, etc.) and a single gaze ray is provided to the application during intentional interactive motion (e.g., during a pinch). Doing so may enable some customized application input recognition while still protecting much of the user's gaze direction data. In this example, the application does not receive continuous gaze direction data. For example, such a framework may be suitable for applications that are migrated from another 3D XR framework that already includes 3D recognition capabilities built into the application, e.g., enabling such applications to be used with little or no modification while still protecting user privacy.
[0136] In some implementations, applications are developed for use on mobile platforms that use single-touch and / or multi-touch input gestures. Applications can be developed by application developers, include calls to gesture APIs and then specify what to do for certain types of touch events, e.g., do X for tap events, Y for swipe events, etc. It may be desirable to use such applications on new or different platforms that provide input modalities other than (or in addition to) single-touch and / or multi-touch input gestures. Rather than implementing all new 3D gestures and gesture recognition processes within applications, some implementations disclosed herein interpret 3D user activity (e.g., hand position, gaze direction, etc.) and send proxy data to applications corresponding to where the touch would be if the 3D user activity had been performed as a 2D touch input gesture. The input support process 732 can provide such proxy data to such applications, and in doing so, enable a large number of existing 2D / touch-based applications to be easily imported and used within new 3D systems that use new and different 3D user activity-based inputs. The input support process 732 may make additional information available so that updated / modified applications can take advantage of / use more information about the user's 3D activities. Alternatively, this information may be utilized by older, unmodified applications that link with an XR-compatible version of the UI framework, which may utilize this additional data on behalf of the application in the background (e.g., a binary-compatible application using a standard long press recognizer may benefit from an XR-based OS implementation that uses the z-depth of a touch to more accurately recognize a long press than purely 2D native methods).
[0137] In some embodiments, to enable the use of both traditional 2D / touch-based applications and newer applications with additional 3D recognition capabilities, the input support process may provide multiple types of information, such as interaction gesture data for supporting touch-based input recognition for traditional applications and manipulator gesture data for supporting applications with additional 3D recognition capabilities.
[0138] Interpreting 3D user activities as 2D touch-based activities may require overcoming various challenges. For example, the movement of a user's hand in 3D space when performing a touch gesture may be equivalent to a touchscreen tap gesture, but may lack the precision expected of a touchscreen tap gesture. A touch on a touchscreen involves contact with a physical surface where the finger rests, while the movement of "virtually" touching a UI in a 3D XR environment may involve the user's hand piercing a UI element. A user may intend to tap the surface of a virtual button, but actually pierce the surface at one location and then retract the hand / finger at a slightly different location. This user activity involving two different UI element locations may be interpreted (by an input support process) as a tap at a single point using various algorithms and / or machine learning processes (e.g., performing trajectory correction). The application receives a single location (from the input support process) and recognizes it as a touch event. Therefore, the input support process 732 may classify and / or interpret user activities to take into account the unique circumstances of interacting within a 3D environment, and encapsulate the data provided to the application so that the application itself does not need to distinguish between the actions intended by the user and the actions actually performed by the user.
[0139] When providing data to the application, the input support process 732 can take into account sensor inaccuracies / limitations, for example, taking into account the fact that the identified gaze direction may be different from the actual gaze direction. The application itself does not need to take such inaccuracies into account.
[0140] In formatting an application's data in a format that the application will understand, the input support process 732 can resolve ambiguities, account for inaccuracies, and / or repackage input provided in input modalities that the application itself does not understand / needs to understand into a format that the application does recognize.
[0141] In short, the input support process 732 can perform various processes to interpret raw 3D data for consumption by applications, so that applications (and their developers) do not need to perform those processes via manually programmed processes. In addition, executing such processes by a shared OS process can unify the input experience across all applications, so that users receive consistent UI response behavior.
[0142] In addition, the shared input support process 732 can enable multiple interaction modalities, for example, providing different ways for a user to select a button, and abstracting the user activity data associated with those different modalities for the same type of interaction to provide the same data to the application for different types of input. Thus, as long as the input support process 732 can abstract new or changed input modality user activity into a format understood by the application, input modalities can be changed and / or added over time without requiring changes to the application.
[0143] In some implementations, an application utilizes a gesture recognizer running within an application process. The gesture recognizer may include a state machine that categorizes input received in an input data stream, e.g., a tap started, a tap in progress, a tap ended. The input support process 732 may provide data to the application that is recognized by the gesture recognizer running within the application process as a particular type of input (e.g., a tap) that may correspond to a single input modality (e.g., a touch-based input modality) dedicated to the application. The input support process 732 may configure the data provided to the application to ensure that the application's process accurately recognizes user activity, such as by providing the data in a form expected by the gesture recognizer. In one example, the input support process 732 converts user activity involving user gaze and gestures into data that can be recognized as touch input. In another example, the input support process 732 converts user activity involving a 6DOF controller and gaze into data that can be recognized as touch input.
[0144] Application developers do not have to write applications for different input modality platforms. Instead, application developers can write applications for touch platforms (or generic 2D input modalities), and the application can be imported to work within the 3D user activity platform with little or no effort. A single application can be used on mobile devices, laptop devices, tablet devices, desktop devices, and 3D XR devices. The techniques disclosed herein can achieve binary compatibility, that is, applications can be executed in different environments that utilize different input modalities.
[0145] In some embodiments, the input support process 732 provides gesture markers and / or gesture classifications to the application, for example, indicating to the application that the user has raised their hand or that a particular gesture has been recognized as a tap, swipe, scroll, etc., as a hint that the application (or UI framework) can use when recognizing a gesture.
[0146] In some implementations, the input support process 732 facilitates the application to recognize gaze-only input without providing raw gaze data to the application. This may involve periodically providing gaze data (i.e., the position on the UI element that the user is gazing at). Gaze-only data may be associated with an interaction state, and data may be provided to the application based on gaze-only data only when the gaze data is associated with a particular interaction state, for example, when a gaze hover state is applicable. Such data may be provided only based on explicit user authorization. For example, when a user gazes at a UI element for more than a threshold amount of time, the application may be notified, and affordances (e.g., points) may be displayed. If the user then views the affordance, the application may be notified that the UI element being stared at has been hit / selected. In another example, gazing at a point within a UI element for more than a threshold amount of time may trigger hover feedback, which may be processed outside the process or by the application. Gaze-only input modality may be another input modality that is abstracted as data provided to an application so that the application may interpret the input, for example, using a general or touch-based gesture recognition process.
[0147] In some implementations, the input support process determines the interaction state of each hand (e.g., direct vs. indirect) and / or the interaction state associated with the user's gaze (e.g., gaze only or not only gaze).
[0148] Fig. 9 is a flow chart illustrating a method 900 for supporting application input recognition. In some implementations, a device such as electronic device 105 or electronic device 110 performs method 900. In some implementations, method 900 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. Method 900 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 900 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).
[0149] At block 902, method 900 executes an application as an application process within an operating system (OS) of an electronic device. The OS may include and execute an OS process configured to execute input support processes other than application processes, such as Fig. 7A and Figure 7B The OS may include 3D environment management processes such as Fig. 7A and Figure 7B The simulation system 734. In other specific implementations, the input support process is executed within the application, such as in the process.
[0150] At block 904, method 900 receives data corresponding to a position of a user interface (UI) element of an application within a 3D coordinate system at an input support process. The data may correspond to the position of the UI element based at least in part on data provided by the application (e.g., a position / shape of a 2D element intended for a 2D window area), such as Fig. 7A and Figure 7B The UI information provided from application 740 to operating system process 730 is shown. This data may include FIG. 7A to FIG. 7B The user interface geometry shown (e.g., collision world data). In some embodiments, the operating system manages information about virtual and / or real content located within a 3D coordinate system. Such a 3D coordinate system may correspond to an XR environment that represents a physical environment and / or virtual content corresponding to content from one or more applications. An executing application may provide information about the positioning of its UI elements via a hierarchical tree (e.g., a declarative hierarchical tree), where some layers are identified for remote (i.e., outside the application process) input effects. Such information may be provided to a simulation process (e.g., Fig. 7A and Figure 7B A simulation system 734 is provided, which positions the application UI elements in a 3D coordinate system, for example by defining the 3D positions of one or more collision volumes (e.g., each collision volume having a rectangular, cubic, or other shaped window area) in 3D coordinates for the application UI elements to be positioned therein.
[0151] At block 906, method 900 receives data corresponding to user activity in a 3D coordinate system at an input support process, the data being associated with one or more input modalities. Fig. 7A and Figure 7B Such data may include, but are not limited to, hand data, gaze data, and / or human interface device (HID) data. Various combinations of two or more different types of data may be received, such as hand data and gaze data, controller data and gaze data, hand data and controller data, voice data and gaze data, voice data and hand data, etc. Different combinations of sensor / HID data may correspond to different input modalities. In an exemplary implementation, the data includes both hand data (e.g., a hand posture skeleton identifying 20+ joint positions) and gaze data (e.g., gaze vector stream), and both hand data and gaze data may be related to identifying input via direct touch input modality and indirect touch input modality.
[0152] At box 908, method 900 identifies data of the application based on data corresponding to the user activity and data corresponding to the location of the UI element of the application at the input support process. The data of the application may include interactive event data. Interactive events can be selected by identifying only certain types of activities, such as user activities determined to correspond to a first type of user activity and one or more other types of user activities. In some specific implementations, this involves identifying user activities that meet criteria configured to distinguish intentional actions on UI elements from other types of user activities.
[0153] In some specific implementations, the first input modality specifies a first specific user activity as an indication of an intention to interact with a UI element, for example, a direct touch input modality may specify that the intersection of a portion of a user's hand with a UI interface element (i.e., a virtual touch) is an indication of an intention to interact with a user interface object. In some specific implementations, the second input modality specifies a second specific user activity as an indication of an intention to interact with a UI element, for example, an indirect touch input modality may specify that a pinch gesture is an indication of an intention to interact with a user interface object. The input support process may be configured to monitor user activity data to indicate an intention to interact via one or more different input modalities. In the above example, this may involve simultaneously monitoring both direct virtual touches (associated with the direct input modality) and pinch gestures (associated with the indirect input modality) of a UI element. The occurrence of either type of user activity may be used to trigger the input support process to identify corresponding user event data for an interaction event to be provided to the application.
[0154] In some implementations, identifying the first type of user activity involves identifying user activities of the first type that are considered intentional and excluding activity types such as gaze-only activities that are considered unintentional interactions.
[0155] Interaction event data may include interaction pose (e.g., 6DOF data for a point on an application's UI), manipulator pose (e.g., 3D position of a stable hand center or pinch center of mass), interaction state (i.e., direct, indirect, hover, pinch, etc.), and / or which UI element is being interacted with.
[0156] Interaction data may exclude data associated with user activity that occurs between intentional events. Interaction event data may exclude detailed sensor / HID data, such as hand skeletal data. Interaction event data may extract detailed sensor / HID data to avoid providing applications with data that is not necessary for the application to recognize input and that may be private to the user.
[0157] An input support process can respond to some user activities on behalf of an application without necessarily involving or notifying the application of the user activity or the response provided. For example, an input support process can respond to some user activities by adjusting the appearance of displayed application content without notifying the application, such as providing hover feedback based on gaze without notifying the application of the user activity that triggered the feedback or providing hover feedback.
[0158] At box 910, method 900 provides data from the application of the input support process to the application. The application's access to at least some of the data corresponding to the user activity may be suppressed. The application process identifies input to the application based on the application's data (e.g., based on the interaction event data). The input support process can provide data sufficient for the application to recognize the input while avoiding providing the application with access to user activity data that is not associated with the identified interaction event. Similarly, the input support process can provide abstract data to avoid providing detailed user activity data, such as not providing access to hand bone data. The data can be formatted to be recognized by a 2D input recognition process executed within the application, such as by an input recognition process that is configured to recognize abstract input data and / or input corresponding to an input modality from a traditional or different system, such as an input recognition process configured to receive 2D touch-based input.
[0159] Method 900 may display a view of an extended reality (XR) environment corresponding to a (3D) coordinate system, wherein UI elements of an application are displayed in the view of the XR environment. Such an XR environment may include UI elements from multiple application processes corresponding to multiple applications, and the input support process may identify interaction event data for multiple applications and route the interaction event data only to the appropriate application, e.g., the application that the user intended to interact with. Accurately routing data only to the intended application can help ensure that one application does not misuse input data intended for another application (e.g., one application does not track a user entering a password into another application).
[0160] The OS may provide an OS process configured to execute an input support process outside of an application process (e.g., a shared 3D environment tracking / simulation process). The 3D environment / simulation may be provided for tracking virtual content provided by multiple sources, such as multiple system and / or non-system applications provided by the OS itself, by an OS provider and / or a third party, etc. The OS may provide an OS process including a simulation process, which is configured to perform a simulation of a 3D environment based on a physical environment associated with a 3D coordinate system. Such a simulation process locates the UI elements of an application within a 3D coordinate system based on data provided by the application. Such a simulation process may perform the same operation on multiple applications, and may adjust the positioning of such application content within a 3D coordinate system, for example, based on which application the user focuses on, user input, and / or other criteria. In one example, the simulation process locates the UI elements by: locating one or more components within a 3D coordinate system (e.g., locating collision bodies with a 3D coordinate system (e.g., each collision body has a rectangular window area, a cube shape, or other shapes)); and locating the UI elements of the application on / within the one or more components. The positioning of the application's UI elements on the one or more components may be defined based on data provided by the application. The application may not be aware of the positioning of the one or more components within the 3D coordinate system.
[0161] In some specific implementations, the data provided by the application includes a hierarchical tree structure that defines the position and containment relationship of UI elements relative to each other in a two-dimensional (2D) coordinate system. In some specific implementations, the hierarchical tree structure defines such positioning for traditional input modalities (e.g., touch screen modalities or 2D desktop / laptop cursor drive modalities). The data provided by the application can identify the external effects of some UI elements in the UI elements. Such external effects can specify that the OS process will provide a response to a specified user activity relative to a specified UI element outside the application process (e.g., performing hover feedback on the button outside the process). The data provided by the application can be provided to the OS process via an inter-process communication link.
[0162] The data corresponding to the user activity may have various formats and be based on or include (but not limited to being based on or including) sensor data or HID data. In some specific implementations, the data corresponding to the user activity includes gaze data, which includes a gaze vector stream corresponding to the gaze direction over time during the use of the electronic device. The data corresponding to the user activity may include hand data, which includes hand posture skeletons for multiple joints at each of multiple moments during the use of the electronic device. The data corresponding to the user activity may include both hand data and gaze data. The data corresponding to the user activity may include controller data and gaze data. The data corresponding to the user activity may include, but is not limited to, any combination of one or more types of data associated with one or more sensors or one or more sensor types, associated with one or more input methods, associated with one or more parts of the user (e.g., eyes, nose, cheeks, mouth, hands, fingers, arms, torso, etc.) or the entire user and / or associated with one or more items worn or held by the user (e.g., mobile devices, tablet computers, laptop computers, laser pointers, handheld controllers, sticks, rings, watches, bracelets, necklaces, etc.).
[0163] In method 900, identifying the interaction event data of the application may involve identifying only certain types of activities within the user activities to be included in the interaction event data. In some implementations, activities in the user activities that are determined to correspond to unintentional events rather than intentional UI element inputs (e.g., types of activities) are excluded from the interaction event data. In some implementations, passive gaze-only activities in the user activities are excluded from the interaction event data. This passive gaze-only behavior (non-intentional input) is different from intentional gaze-only interactions (e.g., gaze dwells, or gaze until an air gesture is performed to invoke / dismiss the gaze HUD, etc.).
[0164] In other implementations, some gaze-only activity (eg, gaze direction only at interaction events) is provided to the application.
[0165] Identifying interactive event data for an application may involve identifying only specific attributes of the data corresponding to the user's activity to include in the interactive event data, such as including the position of the hand center instead of all joints used to model the hand, including a single gaze direction or a single HID pointing direction for a given interactive event. In another example, the starting position of the gaze direction / HID pointing direction is changed or suppressed, for example to obscure data indicating how far the user is from the UI or where the user is in the 3D environment. In some specific implementations, the data corresponding to the user's activity includes hand data representing the position of multiple joints of the hand, and the interactive event data includes a single hand gesture provided in place of the hand data.
[0166] In some implementations, the interaction event data includes one or more of the following: interaction gestures, including position and / or orientation data of interaction points within the UI elements of the application; manipulator gestures, including position and / or orientation data corresponding to the hand within a 3D coordinate system (e.g., the 3D position of a stable hand center or a pinch centroid); and / or interaction state, including data identifying the type of interaction. The interaction data may include interaction event data, which includes interaction gestures, manipulator gestures, and interaction states, and the input recognition process of the receiving application may choose which information to use. Some applications may use only some of the data (e.g., a touch-based input recognition process may use only interaction gestures), while other applications may use all of the data.
[0167] The interaction event data may identify the UI element being interacted with during the interaction event. The application may use it to identify which of its UI elements is the target of the user interaction, or may use other data (e.g., identifying which UI element the interaction gesture is on) to identify which of the UI elements is the target of the user interaction.
[0168] In some implementations, the interactivity event data provided to an application excludes data associated with interactivity events associated with applications other than the application.
[0169] In some implementations, method 900 is performed by an electronic device that is a head-mounted device (HMD), and / or the XR environment is a virtual reality environment or an augmented reality environment.
[0170] Fig.10 1 is a flowchart illustrating another method 1000 for supporting application input recognition. In some implementations, a device such as electronic device 105 or electronic device 110 performs method 1000. In some implementations, method 1000 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. Method 1000 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 1000 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).
[0171] At block 1002, method 1000 executes an application as an application process within an operating system (OS) of an electronic device. The OS may include and execute an OS process configured to execute input support processes other than application processes, such as Fig. 7A and Figure 7B The OS may include 3D environment management processes such as Fig. 7A and Figure 7BThe simulation system 734. In other specific implementations, the input support process is executed within the application, such as in the process.
[0172] At block 1004, method 1000 receives data corresponding to user activity in a 3D coordinate system at an input support process, the data being associated with one or more input modalities. Fig. 7A and Figure 7B Such data may include, but are not limited to, hand data, gaze data, and / or human interface device (HID) data. Various combinations of two or more different types of data may be received, such as hand data and gaze data, controller data and gaze data, hand data and controller data, voice data and gaze data, voice data and hand data, etc. Different combinations of sensor / HID data may correspond to different input modalities. In an exemplary implementation, the data includes both hand data (e.g., a hand posture skeleton identifying 20+ joint positions) and gaze data (e.g., gaze vector stream), and both hand data and gaze data may be related to identifying input via direct touch input modality and indirect touch input modality.
[0173] At box 1006, method 1000 generates interaction event data for interaction events based on identifying a portion of user activity data corresponding to a first type of user activity (e.g., only user activities in which the user expresses an intention to interact with the UI element) at the input support process. The interaction event data represents at least a subset of user activities corresponding to the first type of user activity. In some specific implementations, method 1000 generates event data only for times / events when there is an intention to interact with the UI element. Interaction events can be selected by identifying only certain types of activities, such as user activities determined to correspond to the first type of user activity and one or more other types of user activities. In some specific implementations, this involves identifying user activities that meet criteria configured to distinguish intentional actions on UI elements from other types of user activities.
[0174] In some specific implementations, the first input modality specifies a first specific user activity as an indication of an intention to interact with a UI element, for example, a direct touch input modality may specify that the intersection of a portion of a user's hand with a UI interface element (i.e., a virtual touch) is an indication of an intention to interact with a user interface object. In some specific implementations, the second input modality specifies a second specific user activity as an indication of an intention to interact with a UI element, for example, an indirect touch input modality may specify that a pinch gesture is an indication of an intention to interact with a user interface object. The input support process may be configured to monitor user activity data to indicate an intention to interact via one or more different input modalities. In the above example, this may involve simultaneously monitoring both direct virtual touches (associated with the direct input modality) and pinch gestures (associated with the indirect input modality) of the UI element. The occurrence of either type of user activity may be used to trigger the input support process to identify corresponding user event data.
[0175] In some implementations, identifying the first type of user activity involves identifying user activities of the first type that are considered intentional and excluding activity types such as gaze-only activities that are considered unintentional interactions.
[0176] Interaction event data may include interaction pose (e.g., 6DOF data for a point on an application's UI), manipulator pose (e.g., 3D position of a stable hand center or pinch center of mass), interaction state (i.e., direct, indirect, hover, pinch, etc.), and / or which UI element is being interacted with.
[0177] Interaction data may exclude data associated with user activity that occurs between intentional events. Interaction event data may exclude detailed sensor / HID data or detailed information derived from such data, such as hand skeletal data. Interaction event data may extract detailed sensor / HID data to avoid providing applications with data that is not necessary for the application to recognize input and that may be private to the user.
[0178] An input support process can respond to some user activities on behalf of an application without necessarily involving or notifying the application of the user activity or the response provided. For example, an input support process can respond to some user activities by adjusting the appearance of displayed application content without notifying the application, such as providing hover feedback based on gaze without notifying the application of the user activity that triggered the feedback or providing hover feedback.
[0179] At box 1008, method 1000 provides the interactive event data from the input support process to the application. The application's access to at least some of the data corresponding to the user activity may be suppressed. The application process identifies input to the application based on the interactive event data. The input support process can provide data sufficient for the application to recognize the input while avoiding providing the application with access to user activity data that is not associated with the identified interactive event. Similarly, the input support process can provide abstract data to avoid providing detailed user activity data, such as not providing access to hand bone data. The data can be formatted to be recognized by a 2D input recognition process executed within the application, such as by an input recognition process that is configured to recognize abstract input data and / or input corresponding to an input modality from a traditional or different system, such as an input recognition process configured to receive 2D touch-based input.
[0180] In some implementations, the input support process identifies a second portion of the user activity data corresponding to a second type of user activity (e.g., an unconscious action, a gaze-only event, a hover-triggered action). The application may receive less detailed data (or no data) about the second type of user activity than the first type of user activity. A response to the second type of user activity may be provided to the user without notifying the application of the second type of user activity (e.g., an unconscious action, a gaze, a hover action). In some implementations, the first type of user activity corresponds to an intentional interaction with a UI element, and the second type of user activity corresponds to a user activity other than an intentional interaction with a UI element.
[0181] The user activity data used by method 900 may include one or more of the following (but is not limited to including one or more of the following): gaze data, including a stream of gaze vectors corresponding to gaze directions; hand data, including a hand posture skeleton for multiple joints; touchpad data, including touchpad input; and controller data, including controller input associated with 3D movement of a handheld controller.
[0182] In some implementations, identifying a portion of the user activity data corresponding to the first type of user activity involves identifying an interaction intent based on identifying a hand pinch based on hand data in the user activity data. In some implementations, identifying a portion of the user activity data corresponding to the first type of user activity involves identifying an interaction intent based on: identifying a user action indicating an interaction intent; determining a gaze direction associated with the interaction intent, the gaze direction being within a 3D coordinate system; and identifying a UI element of an application positioned within the 3D coordinate system based on the gaze direction, wherein the interaction event data identifies an interaction with the UI element.
[0183] The interactive event data may exclude all gaze direction data. The interactive event data may include only some gaze direction data from the gaze direction data stream, for example, only gaze direction data associated with the interactive event. The interactive event data may include interactive gesture data, which includes the position and / or orientation data of at least one interactive point within the UI element of the application. The interactive event data may include less than all attributes of the hand data in the user activity data to be included in the interactive event data, wherein the hand data represents the position of multiple joints of the hand. The interactive event data may include a single hand gesture provided in place of more / all attributes of the hand data for each interactive event. The interactive event data may include manipulator gesture data, which includes the position and orientation data of the hand within the 3D coordinate system corresponding to each interactive event (e.g., the 3D position of the stable hand center or the pinching center of mass). The interactive event data may include an interactive state, which includes data identifying the type of interaction. The interactive event data may include an interactive gesture, a manipulator gesture, and / or an interactive state including data identifying the type of interaction, and / or may identify the UI element interacted with during the interactive event. The interactivity event data provided to an application may exclude data associated with interactivity events associated with applications other than the application.
[0184] Fig.11 1 is a flowchart illustrating another method 1100 for supporting application input recognition. In some implementations, a device such as electronic device 105 or electronic device 110 performs method 1100. In some implementations, method 1100 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. Method 1100 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 1100 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).
[0185] At block 1102, method 1100 executes an application as an application process within an operating system (OS) of an electronic device. The OS may include and execute an OS process configured to execute input support processes other than application processes, such as Fig. 7A and Figure 7B The OS may include 3D environment management processes such as Fig. 7A and Figure 7B The simulation system 734. In other specific implementations, the input support process is executed within the application, such as in the process.
[0186] At block 1104, method 1100 receives data corresponding to user activity in a 3D coordinate system at an input support process, the data being associated with one or more input modalities. Fig. 7A and Figure 7B Such data may include, but are not limited to, hand data, gaze data, and / or human interface device (HID) data. Various combinations of two or more different types of data may be received, such as hand data and gaze data, controller data and gaze data, hand data and controller data, voice data and gaze data, voice data and hand data, etc. Different combinations of sensor / HID data may correspond to different input modalities. In an exemplary implementation, the data includes both hand data (e.g., a hand posture skeleton identifying 20+ joint positions) and gaze data (e.g., gaze vector stream), and both hand data and gaze data may be related to identifying input via direct touch input modality and indirect touch input modality.
[0187] At box 1106, method 1100 generates interaction event data based on the user activity data at the input support process. The interaction event data may represent at least a subset of user activities corresponding to the first type of user activities. In some specific implementations, method 1100 generates event data only for times / events when there is an intention to interact with the UI element. Interaction events can be selected by identifying only certain types of activities, such as user activities determined to correspond to the first type of user activities and one or more other types of user activities. In some specific implementations, this involves identifying user activities that meet criteria configured to distinguish intentional actions on UI elements from other types of user activities.
[0188] At block 1108, method 1100 provides the interactive event data from the input support process to the application. The application process identifies input to the application based on the interactive event data. The interactive event data may represent user activity using a format that is recognized by a 2D input recognition framework within the application as input to a 2D user interface of the application including UI elements. In some specific implementations, the OS is a 3D operating system (OS), and the 2D input recognition framework of the application is a framework designed for a 2D operating system (OS).
[0189] The 2D input recognition framework may be configured to recognize the interactive event data as corresponding to input to a 2D user interface. The 2D input recognition framework may be configured to recognize the interactive event data as corresponding to a touch input to a 2D user interface. The 2D input recognition framework may be configured to recognize the interactive event data as corresponding to a touch screen tap input to a 2D user interface. The 2D input recognition framework may be configured to recognize the interactive event data as corresponding to a touch screen touch gesture with a direction. The 2D input recognition framework may be configured to recognize the interactive event data as corresponding to a touch screen multi-touch gesture input to a 2D user interface. The 2D input recognition framework may be configured to recognize the interactive event data as corresponding to a 2D mouse cursor position within a 2D user interface. The 2D input recognition framework may be configured to recognize the interactive event data as corresponding to a 2D trackpad cursor position within a 2D user interface. The interactive event data may include additional data corresponding to the 3D aspects of the user's activities.
[0190] Fig.12 1 is a block diagram of an electronic device 1300. Device 1300 illustrates an exemplary device configuration of electronic device 110 or electronic device 105. Although some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 1300 includes one or more processing units 1302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 1306, one or more communication interfaces 1308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1310, one or more output devices 1312, one or more internal-facing and / or external-facing image sensor systems 1314, memory 1320, and one or more communication buses 1304 for interconnecting these and various other components.
[0191] In some implementations, the one or more communication buses 1304 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices and sensors 1306 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.
[0192] In some implementations, the one or more output devices 1312 include one or more displays configured to present a view of a 3D environment to a user. In some implementations, the one or more displays 1312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, the one or more displays correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. In one example, the device 1300 includes a single display. In another example, the device 1300 includes a display for each eye of the user.
[0193] In some implementations, one or more output devices 1312 include one or more audio generating devices. In some implementations, one or more output devices 1312 include one or more speakers, surround sound speakers, speaker arrays, or headphones for generating spatialized sounds such as 3D audio effects. Such devices can virtually place sound sources in a 3D environment, including behind, above, or below one or more listeners. Generating spatialized sounds can involve converting sound waves (e.g., using head-related transfer functions (HRTFs), reverberation, or cancellation techniques) to simulate natural sound waves (including reflections from walls and floors) that are emitted from one or more points in a 3D environment. Spatialized sounds can induce the listener's brain to interpret the sound as if it occurs at one or more points in a 3D environment (e.g., from one or more specific sound sources), even though the actual sound may be generated by speakers in other locations. One or more output devices 1312 may be additionally or alternatively configured to generate tactile sensations.
[0194] In some implementations, the one or more image sensor systems 1314 are configured to obtain image data corresponding to at least a portion of the physical environment. For example, the one or more image sensor systems 1314 may include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 1314 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 1314 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.
[0195] Memory 1320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some implementations, memory 1320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 1320 optionally includes one or more storage devices remotely located from one or more processing units 1302. Memory 1320 includes non-transitory computer-readable storage media.
[0196] In some implementations, the memory 1320 or a non-transitory computer-readable storage medium of the memory 1320 stores an optional operating system 1330 and one or more instruction sets 1340. The operating system 1330 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 1340 includes executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set 1340 is software that can be executed by one or more processing units 1302 to implement one or more of the techniques described herein.
[0197] The instruction set 1340 includes a user interaction instruction set 1342, which is configured to identify and / or interpret user gestures and other user activities as described herein when executed. The instruction set 1340 includes an application instruction set 1344 for one or more applications. In some specific implementations, each of these applications is provided as a separately executed set of code, for example, capable of being executed via an application process. The instruction set 1340 can be embodied as a single software executable file or multiple software executable files.
[0198] Although instruction set 1340 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. In addition, the drawings serve more as functional descriptions of the various features present in a particular implementation, as opposed to the structural schematics of the implementations described herein. As will be appreciated by those of ordinary skill in the art, items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how features are distributed among them will vary depending on the implementation, and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.
[0199] It should be understood that the specific implementations described above are cited by way of example, and the present disclosure is not limited to what has been particularly shown and described above. On the contrary, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to a person skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.
[0200] As described above, one aspect of the present technology is to collect and use sensor data that may include user data to improve the user experience of electronic devices. The present disclosure contemplates that, in some cases, the collected data may include personal information data that uniquely identifies a particular person or can be used to identify the interests, characteristics, or tendencies of a particular person. Such personal information data may include motion data, physiological data, demographic data, location-based data, phone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.
[0201] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, personal information data can be used to improve the content viewing experience. Therefore, the use of such personal information data may enable planned control of electronic devices. In addition, the present disclosure also anticipates other uses of personal information data that benefit users.
[0202] The disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for the privacy and security of maintaining personal information data. For example, personal information from users should be collected for the legal and reasonable purposes of entities, and not shared or sold outside these legal purposes. In addition, such collection should only be carried out after the user's informed consent. In addition, such entities should take any required steps to safeguard and protect access to such personal information data, and ensure that other people who can access personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices.
[0203] Regardless of the foregoing, the present disclosure also contemplates specific implementations in which users selectively prevent the use or access of personal information data. That is, the present disclosure contemplates that hardware elements or software elements may be provided to prevent or block access to such personal information data. For example, with respect to a content delivery service customized for a user, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" of participating in the collection of personal information data during registration for the service. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information but allow anonymous information to be transmitted for use in improving the functionality of the device.
[0204] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences or settings based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information.
[0205] In some embodiments, data is stored using a public / private key system that only allows the owner of the data to decrypt the stored data. In some other implementations, data can be stored anonymously (e.g., without identifying and / or personal information about the user, such as legal name, username, time and location data, etc.). In this way, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some implementations, a user can access their stored data from a user device that is different from the user device used to upload the stored data. In these cases, the user may need to provide login credentials to access their stored data.
[0206] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.
[0207] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.
[0208] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages may be used to implement the teachings contained herein in software for programming or configuring a computing device.
[0209] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined and / or divided into sub-blocks. Some boxes or processes can be executed in parallel.
[0210] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values may in practice be based on additional conditions or values beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.
[0211] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.
[0212] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.
[0213] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.
[0214] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.
Claims
1. A method, comprising: At an electronic device having a processor: executing an application as an application process within an operating system (OS) of the electronic device; receiving, at an input support process, data corresponding to a positioning of a user interface (UI) element of the application within a 3D coordinate system, the data corresponding to the positioning of the UI element being based at least in part on data provided by the application; At the input support process, receiving data corresponding to user activity in the 3D coordinate system; identifying, at the input support process, data for the application based on the data corresponding to the user activity and the data corresponding to the location of the UI element of the application; as well as The application's data from the input support process is provided to the application, wherein the application process identifies input to the application based on the application's data.
2. The method of claim 1 , further comprising displaying a view of an extended reality (XR) environment corresponding to the (3D) coordinate system, wherein the UI elements of the application are displayed in the view of the XR environment, wherein the XR environment includes UI elements from multiple application processes corresponding to multiple applications, wherein the input support process identifies data for each of the multiple applications.
3. The method according to claim 1, wherein: The OS includes an OS process configured to execute the input support process outside the application process.
4. The method according to claim 3, wherein: The OS process also includes a simulation process configured to perform a simulation of a 3D environment based on a physical environment associated with the 3D coordinate system, wherein the simulation process positions the UI element of the application within the 3D coordinate system based on data provided by the application.
5. The method according to claim 4, wherein: The simulation process locates the UI element by the following steps: positioning one or more components within the 3D coordinate system; and The UI element of the application is positioned on the one or more components, wherein the positioning of the UI element of the application on the one or more components is defined based on data provided by the application. The application is unaware of the positioning of the one or more components within the 3D coordinate system.
6. The method according to claim 3, wherein: The data provided by the application includes a hierarchical tree structure that defines the positions and containment relationships of the UI elements relative to each other on a two-dimensional (2D) coordinate system.
7. The method according to claim 3, wherein: Data provided by the application identifies external effects for ones of the UI elements, wherein the external effects specify a response to a specified user activity that the OS process is to provide relative to the specified UI element external to the application process.
8. The method according to claim 1, wherein: Data provided by the application is provided to the OS process via an inter-process communication link.
9. The method according to claim 1, wherein: The data corresponding to the user activity includes gaze data including a stream of gaze vectors corresponding to gaze directions over time during use of the electronic device.
10. The method according to claim 1, wherein: The data corresponding to the user activity includes hand data, which includes a hand posture skeleton of multiple joints at each of multiple moments during use of the electronic device.
11. The method according to claim 1, wherein: The data corresponding to the user activity includes hand data and gaze data.
12. The method according to claim 1, wherein: The data corresponding to the user activity includes controller data and gaze data.
13. The method according to claim 1, wherein: Identifying data for the application includes identifying interactivity event data by identifying only specific types of activity within the user activity to be included.
14. The method according to claim 1, wherein: Activities of the user activities that are determined to correspond to unintentional events rather than intentional input are excluded from the data for the application.
15. The method according to claim 1, wherein: Passive gaze-only activity in the user activity is excluded from data for the application.
16. The method according to claim 1, wherein: Identifying the data of the application includes identifying only specific attributes of the data corresponding to the user activity for inclusion in the data of the application.
17. The method of claim 1, wherein: The data corresponding to the user activity includes hand data representing positions of a plurality of joints of a hand; and The applied data includes a single hand gesture provided in place of the hand data.
18. The method according to claim 1, wherein: The data of the application includes: An interaction gesture comprising position and orientation data of an interaction point within the UI element of the application.
19. The method according to claim 1, wherein: The data of the application includes: A manipulator pose includes position and orientation data corresponding to a hand within the 3D coordinate system.
20. A system, comprising: a non-transitory computer-readable storage medium; as well as one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the system to perform operations including: executing an application as an application process within an operating system (OS) of the electronic device; receiving, at an input support process, data corresponding to a positioning of a user interface (UI) element of the application within a 3D coordinate system, the data corresponding to the positioning of the UI element being based at least in part on data provided by the application; At the input support process, receiving data corresponding to user activity in the 3D coordinate system; identifying, at the input support process, data for the application based on the data corresponding to the user activity and the data corresponding to the location of the UI element of the application; and The application's data from the input support process is provided to the application without allowing the application to access at least some of the data corresponding to the user activity, wherein the application process identifies input to the application based on the application's data.
