Using deep learning to determine gaze

Abstract

In one embodiment, a computing system may generate and display a virtual reality environment to a user. The computing system may determine a user's head pose based on headset tracking data associated with a headset worn by the user. The computing system may determine a user's hand pose based on hand tracking data associated with a device held by or worn by the user's hand. The computing system may access scene information associated with the displayed virtual reality environment. The computing system may determine a predicted focus of the user within the virtual reality environment by processing the head pose, hand pose, and scene information using a machine learning model.

Description

[Technical field]

[0001] The present disclosure relates generally to controls and interfaces for user interaction and experience in virtual reality environments. [Background technology]

[0002] Virtual reality is a computer-generated simulation of an environment (e.g., a 3D environment) with which a user can interact in a seemingly real or physical manner. A virtual reality system, which may be a single device or a group of devices, may generate this simulation for display to a user, for example, on a virtual reality headset or some other display device. The simulation may include images, sounds, haptic feedback, and / or other senses to mimic a real or imaginary environment. As virtual reality becomes more and more prominent, the range of useful applications of virtual reality is rapidly expanding. The most common applications of virtual reality involve games or other interactive content, but close behind are other applications, such as viewing visual media items (e.g., photos, videos) for entertainment or training purposes. The feasibility of using virtual reality to simulate real conversations and other user interactions is also being explored. Summary of the Invention

[0003] Therefore, the present invention is directed to a method, a computer-readable non-transitory storage medium, and a system according to the appended claims. Various different ways of rendering and interacting with a virtual (or augmented) reality environment are disclosed herein. A virtual reality system may render a virtual environment, which may include a virtual space that is rendered for display to one or more users. The users may observe and interact within this virtual space and the broader virtual environment through any suitable means. One objective of the disclosed method is to determine the user's gaze, or perform eye tracking. In certain embodiments, the virtual reality system may provide a method for determining the user's gaze by using a machine learning model. The virtual reality system may use input data, such as the pose of the headset worn by the user relative to the environment, the pose of the controller relative to the headset or environment, scene information, among other data, to determine the user's gaze. By training a machine learning model to determine the gaze, the virtual reality system may improve the accuracy of the user's predicted gaze. Currently, virtual reality systems require a sensor, such as an eye tracker, to determine the user's gaze. Thus, virtual reality systems that do not have the necessary sensors may not be able to take advantage of varifocal displays and gaze-contingent rendering solutions that improve the user's visual experience and the processing of the virtual reality system. ,a Input data with data from the tracker As ground truth data After the machine learning model is trained, it can then be implemented in virtual reality systems that do not have eye trackers to enable those systems to have the ability to perform eye tracking and / or determine the user's gaze.

[0004] Embodiments of the present invention may include or be implemented in conjunction with a synthetic reality system. Synthetic reality is a form of reality that is conditioned in some manner prior to presentation to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or any combination and / or derivative thereof. Synthetic reality content may include fully generated content or generated content combined with captured content (e.g., real-world photos). Synthetic reality content may include video, audio, haptic feedback, or any combination thereof, any of which may be presented in a single channel or multiple channels (such as stereo video that creates a three-dimensional effect for the viewer). Additionally, in some embodiments, synthetic reality may relate to, for example, applications, products, accessories, services, or any combination thereof that are used to create content in the synthetic reality and / or are used in the synthetic reality (e.g., perform activities in the synthetic reality). A virtual reality system that provides virtual reality content may be implemented on a variety of platforms, including a head mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing virtual reality content to one or more observers.

[0005] In one aspect, the present invention provides a method for implementing a method for a computing system comprising: generating and displaying a rendered environment to a user; determining a head pose of a user based on headset tracking data associated with a headset worn by the user; determining a user's hand pose based on the hand tracking data; accessing scene information relating to the displayed rendered environment; determining a predicted focus of a user within a rendered environment by processing head pose, hand pose, and scene information using a machine learning model; The present invention is directed to a method, comprising:

[0006] In one embodiment of the method according to the present invention, the method may further include adjusting an image presented by the computing system to the user based on a predicted focal point of the user within the rendered environment.

[0007] In one embodiment of the method according to the invention, the head pose may be determined relative to the rendered environment, and in one embodiment of the method according to the invention, the hand pose may be determined relative to one of the rendered environment or a headset worn by the user. 1. In one embodiment of the method according to the present invention, determining the user's hand pose may further include identifying the user's hand based on one or more cameras coupled to the computing system analyzing a plurality of images including the user's hand.

[0008] In one embodiment of the method according to the present invention, the hand tracking data may relate to a device held or worn by the user's hand, and the method may further include determining an action performed by the device held or worn by the user's hand, and determining the predicted focus further includes processing the action performed by the device held or worn by the user's hand using a machine learning model.

[0009] In one embodiment of the method according to the invention, the scene information may include color and depth data and / or the scene information may include semantic information of one or more elements in the rendered environment.

[0010] In one embodiment of the method according to the invention, the predicted focal point may be a three-dimensional coordinate within the rendered environment.

[0011] In one embodiment of the method according to the present invention, the rendered environment may be related to an application, the application may be related to a category, and the scene information may include a category related to the application.

[0012] In one embodiment of the method according to the invention, the method may further include generating a confidence map of one or more locations for the predicted focal point using a machine learning model, the confidence map assigning a probability that the predicted focal point is at each of the one or more locations, and determining the predicted focal point includes utilizing the confidence map.

[0013] In one embodiment of the method according to the invention, the predicted focus may be determined without an eye-tracking sensor.

[0014] In one embodiment of the method according to the invention, the rendered environment may include one or more of an augmented reality environment, a virtual reality environment, or a mixed reality environment.

[0015] In one aspect, the invention provides one or more computer-readable non-transitory storage media embodying software that, when executed, performs the methods described above or generating and displaying a virtual reality environment to a user; determining a head pose of a user based on headset tracking data associated with a headset worn by the user; determining a hand pose of a user based on hand tracking data associated with a device held or worn by the user's hand; accessing scene information associated with the displayed virtual reality environment; Determining a predicted focus of a user in a virtual reality environment by processing head pose, hand pose, and scene information using a machine learning model; The present invention is directed to one or more computer-readable non-transitory storage media operable to perform the steps of:

[0016] In one aspect, the present invention provides a system comprising one or more processors and a non-transitory memory coupled to the processors, the non-transitory memory including instructions executable by the processors, the instructions, when executed by the processor, to perform the methods described above or generating and displaying a virtual reality environment to a user; determining a head pose of a user based on headset tracking data associated with a headset worn by the user; determining a hand pose of a user based on hand tracking data associated with a device held or worn by the user's hand; accessing scene information associated with the displayed virtual reality environment; Determining a predicted focus of a user in a virtual reality environment by processing head pose, hand pose, and scene information using a machine learning model; The present invention is directed to a system operable to:

[0017] The embodiments disclosed herein are merely examples, and the scope of the disclosure is not limited thereto. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed above. Embodiments according to the invention are disclosed in the appended claims, which are particularly directed to methods, storage media, systems, and computer program products, and any feature recited in one claim category, e.g., a method, may also be claimed in another claim category, e.g., a system. Dependencies or references in the appended claims are selected for formality reasons only. However, subject matter resulting from intentional reference (especially multiple dependencies) to previous claims may also be claimed, such that any combination of a claim and its features may be claimed regardless of the dependencies disclosed and selected in the appended claims. Subject matter that may be claimed includes not only the combination of features recited in the appended claims, but also any other combination of features in the claims, and each feature recited in the claims may be combined with any other feature or other combination of features in the claims. Furthermore, any of the embodiments and features described or shown in this specification may be claimed in a separate claim and / or in any combination with any of the embodiments or features described or shown in this specification or with any of the features of the appended claims.

[0018] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 illustrates an example process of using a machine learning model to determine a user's gaze. [Diagram 2] FIG. 1 illustrates an example virtual reality environment in which a predicted line of sight is determined. [Diagram 3]FIG. 1 illustrates an example process for training a machine learning model to predict user gaze. [Figure 4] FIG. 1 illustrates an exemplary method for determining a user's gaze using a machine learning model. [Diagram 5] FIG. 1 illustrates an exemplary network environment associated with a virtual reality system. [Figure 6] FIG. 1 illustrates an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Many of the techniques for developing high-quality visual experiences on portable, personal displays may rely on accurate eye tracking. Gaze-tracking techniques that may help improve a user's visual experience in a virtual reality environment may include variable focus displays and foveated rendering. Some virtual reality systems may include eye trackers to determine the user's gaze, but eye trackers are accessible to all virtual reality systems. For example, a user rendering a virtual reality environment through their smartphone may not have eye tracking capabilities. As virtual reality becomes more accessible to users through portable and personal solutions, such as their smartphones, eye tracking / gaze determination functionality will be removed. However, by training machine learning models, data available to most virtual reality systems may be utilized to determine the user's gaze and / or perform eye tracking. As a user explores a virtual reality environment and completes a task, the location of the user's gaze may correspond to sources of information, such as head direction, hand location, scene content, and task objectives. Although the eye tracking / gaze determination process and application may be described herein in the context of virtual reality systems and environments, the process and application may also be applied to augmented reality systems, augmented reality environments, mixed reality systems, and mixed reality environments. By way of example and not by way of limitation, the rendered environment may be an augmented reality environment comprising a real-world scene with augmented reality elements, and the input data may be data used by the augmented reality system to determine the user's gaze.

[0021] In certain embodiments, input data related to at least one of the headset pose, controller information, and scene information may be used to train a machine learning model to determine the gaze. In certain embodiments, the virtual reality system may retrieve or access input data from multiple sensors. By way of example and not by way of limitation, the virtual reality system may determine the acceleration of the headset through an accelerometer coupled to the headset. The input data may be used to determine a predicted focus of where the user is looking. The predicted focus may be represented by three-dimensional coordinates in the user's environment. For example, the predicted focus may be represented by XYZ coordinates in the user's environment. For example, within the virtual reality environment, the AR / VR headset may calculate predicted three-dimensional coordinates to represent where the user is looking within the virtual reality environment. The headset pose may indicate the position and orientation of the headset. This position and orientation may be relative to the environment. The input data may also include the velocity and acceleration of the headset. The controller information may indicate pose data associated with one or more of the controllers relative to the headset and the environment, and whether there is an action being performed on the controller (e.g., the user is holding down a button to perform a pick-up function). The controller information may also include the velocity and acceleration of each of the controllers. The scene information may include RGBD data of the environment. The headset may perform object identification from the RGBD data of the scene. In a virtual reality environment, the virtual reality system may receive additional information related to the virtual reality environment. For example, object data of objects in the virtual scene, game information, and additional information of the virtual environment may be sent to the virtual reality system. The information of the environment may be used to predict where the user may be looking. By way of example and not by way of limitation, if there is a target that the user needs to aim at, it is highly likely that the user is looking at that target. This accumulation of information on the headset pose, controller information, and scene information may be fed into a machine learning model to predict the user's focus.In certain embodiments, the predicted focus may be compared to a ground truth of the user's actual gaze. In certain embodiments, the actual gaze may be first determined by an eye-tracking sensor and user verification. After the machine learning model is trained, other virtual reality systems that do not have eye-tracking sensors may use the machine learning model to predict the user's focus.

[0022] In certain embodiments, the output of the machine learning model may be a confidence map of where the focal point may be located. The confidence map may include confidence scores of various XYZ locations in the environment. This may be similar to a heat map of where the focal point may be located. By way of example and not by way of limitation, the confidence map may include a range from 0 to 1 for multiple XYZ locations in the environment. In certain embodiments, the confidence score may represent the probability that the focal point is at that location. In certain embodiments, the predicted focal point may comprise several XYZ locations. Although described as an XYZ focal point, the machine learning model may only determine an XY focal point to indicate a line of sight rather than a specific focal point in 3D space.

[0023] In certain embodiments, the virtual reality system may generate and display a virtual reality environment to a user. The virtual reality system may include a headset, one or more controllers, a processor, and additional sensors (e.g., a camera, etc.). In certain embodiments, the headset and the processor may be included in one computing device. The virtual reality headset may render the virtual reality environment, such as an application to be displayed on the virtual reality headset. The application may be related to a category and / or subcategory. By way of example and not by way of limitation, the application may be a game, and the game may be a sports game. The virtual reality system may access or retrieve information corresponding to the application associated with the rendered virtual reality environment. By way of example and not by way of limitation, if the virtual reality headset generates a virtual reality element (e.g., a book), the virtual reality system may retrieve that information to identify various objects within the virtual reality environment. In certain embodiments, the virtual reality system may perform object detection within the virtual reality environment.

[0024] In certain embodiments, the virtual reality system may determine the head pose of the user. In certain embodiments, the virtual reality system may determine the head pose based on headset tracking data associated with the headset worn by the user. In certain embodiments, the virtual reality system may access sensor data (e.g., accelerometer / gyroscope data of the headset, or a separate camera monitoring the user in the virtual reality environment) to track the pose of the headset. By way of example and not by way of limitation, the headset may have an IR / LED pattern that may be detected by a standalone sensor to be used to determine the pose of the headset. By way of another example and not by way of limitation, the headset may have an external-facing camera that may take a picture of the physical environment and use triangulation techniques to determine where the headset is relative to the physical environment. The position of the headset relative to the physical environment may be compared to the pose of the user in the virtual reality environment. In certain embodiments, the head pose may be determined relative to the virtual reality environment.

[0025] In certain embodiments, the virtual reality system may determine the hand pose of the user. In certain embodiments, the virtual reality system may determine the hand pose based on hand tracking data associated with a device held or worn by the user's hand. By way of example and not by way of limitation, the device may be a controller, a glove, etc. In certain embodiments, the virtual reality system may determine the hand pose based on hand tracking data of the user's hand. In certain embodiments, similar to determining the head pose, the virtual reality system may determine the pose of the device in the user's hand based on sensor data. By way of example and not by way of limitation, the virtual reality system may use IR / LED patterns to determine the pose of the device (e.g., a controller) in the user's hand. By way of another example and not by way of limitation, the headset may capture an image of the controller and determine the hand pose of the device relative to the user's headset. By way of another example and not by way of limitation, the headset may capture an image of the user's hand and determine the hand pose of the user's hand. In certain embodiments, the hand pose may be determined relative to the virtual reality environment or the physical real-world environment. In certain embodiments, the virtual reality system may also access controller information of the device in the user's hand. The controller information may indicate the action being performed by the device. By way of example and not by way of limitation, a user may click on a button on the device that may correspond to an action within the virtual reality environment. For example, the action may be holding an object, interacting with the object, etc. In certain embodiments, the headset may determine a gesture being performed by one or both of the user's hands through a captured image of the environment including the user's hands.

[0026] In certain embodiments, the virtual reality system may access scene information associated with the displayed virtual reality environment. In certain embodiments, the virtual reality system may retrieve scene information provided by the application. By way of example and not by way of limitation, if the application is a game and the game instructs the headset to generate virtual reality elements in the virtual reality environment, the virtual reality system may be able to retrieve information corresponding to the virtual reality elements. For example, the virtual reality system may determine XYZ coordinates corresponding to the virtual reality elements generated in the virtual reality environment. In certain embodiments, the scene information may include color and depth data. In certain embodiments, the virtual reality system may access one or more of RGB data or RGBD data associated with the virtual reality environment. By way of example and not by way of limitation, the virtual reality system may capture images within the virtual reality environment. In certain embodiments, within an augmented reality or mixed reality context, the headset may capture images of a real-world environment. The captured images of the real-world environment may be processed to identify physical objects in the real-world environment. The images may be processed by applying triangulation techniques to determine depth measurements of the physical objects. In certain embodiments, the scene information may include semantic information of the virtual reality elements in the virtual reality environment. The virtual reality system may perform object detection using various techniques to identify objects in the virtual reality environment. The virtual reality environment may determine the XYZ coordinates and / or location of the object relative to the virtual reality environment. In certain embodiments, the scene information may be classified as low-level information and high-level information. The low-level information may include one or more of intensity, color, or motion. The high-level information may include identified faces in the virtual reality environment, identified people in the virtual reality environment, and identified tasks. By way of example and not by way of limitation, the virtual reality system may determine that a user is attempting to pick up an object in the virtual reality environment, firing a weapon in the virtual reality environment, and other possible tasks that the user may be performing in the virtual reality environment.

[0027] In certain embodiments, the virtual reality system may determine a predicted focus of the user. In certain embodiments, the virtual reality system may use a machine learning model to determine a predicted focus of the user by processing head pose, hand pose, and scene information. Although the machine learning model is described as being trained to be applied to a virtual reality environment by a virtual reality system, the machine learning model may also be trained to be applied to an augmented reality environment by an augmented reality system or to be applied to a mixed reality environment by a mixed reality system. In certain embodiments, the virtual reality system may use other data to process using the machine learning model. By way of example and not by way of limitation, the virtual reality system may use a machine learning model to process an action being performed by a device held or worn by a user's hand. For example, if a user is attempting to pick up an object, the machine learning model may determine that the predicted focus is toward the object being picked up. In certain embodiments, the predicted focus may be a three-dimensional coordinate within the virtual reality environment. In certain embodiments, the virtual reality system may generate a confidence map and determine the predicted focus by using the confidence map. The predicted focus may be determined without the use of an eye-tracking sensor. While determining predicted focus is described, the virtual reality system may use machine learning models to determine user accommodation and / or convergence using input data as described herein.

[0028] In certain embodiments, the virtual reality system may access one or more of the previous frame data or the previous frame prediction results to determine the predicted focus and other outputs of the machine learning model. In certain embodiments, the machine learning model may analyze the time correlation between each frame displayed to the user. By way of example and not by way of limitation, the predicted focus in one frame may be determined to be in a precise location or relatively close to the predicted focus of the previous frame. In certain embodiments, the previous frame data may comprise the input data of the respective frame. By way of example and not by way of limitation, within a sequence of frames displayed to the user, the virtual reality system may access individual frames and their corresponding input data. In certain embodiments, the previous frame prediction results may comprise the previous predicted focus and other outputs of the machine learning model. The previous frame prediction results may indicate a current predicted focus. The machine learning model may be trained to use the previous frame data and / or the previous frame prediction results to determine a current predicted focus.

[0029] In certain embodiments, the virtual reality system may adjust the image presented to the user. In certain embodiments, the virtual reality system may adjust the image presented to the user based on the predicted focus of the user in the virtual reality environment. The virtual reality system may adjust the image presented to the user based on other data, such as the determined user accommodation and / or the determined vergence. By way of example and not by way of limitation, the virtual reality system may use the predicted focus for foveated rendering to reduce processing required to display the virtual reality environment. With foveated rendering, the virtual reality system may reduce unnecessary rendering outside of the predicted focus. In certain embodiments, the determination of the predicted focus may aid in image distortion correction, any necessary image shifting, and the like. Additionally, the predicted focus may aid in foveated rendering to reduce unnecessary rendering outside of the predicted focus. The predicted focus may also be used by a variable focus display to adjust the focus display based on the predicted focus. In certain embodiments, the virtual reality system may continuously determine the predicted focus of the user and adjust the image presented to the user based on the updated predicted focus. Although virtual reality environments and systems are described herein, in certain embodiments, the predicted focus and other outputs of the machine learning models (e.g., user accommodation and / or vergence) may be used by an augmented reality system for an augmented reality environment or by a mixed reality system for a mixed reality environment.

[0030] In certain embodiments, a user may interact within a rendered environment by one or more of an augmented reality system, a virtual reality system, or a mixed reality system. The rendered environment may be an augmented reality environment, a virtual reality environment, or a mixed reality environment. In certain embodiments, one or more of the augmented reality system, the virtual reality system, or the mixed reality system may take input data described herein to train a machine learning model to determine the user's gaze or perform eye tracking. By way of example and not by way of limitation, the augmented reality system may access input data from a sensor coupled to a headset worn by the user. In certain embodiments, the augmented reality system or the mixed reality system may use a machine learning model to determine the user's gaze or perform eye tracking. In certain embodiments, the augmented reality system or the mixed reality system may use the determined user's gaze for one or more of a variable focus lens, improved eye tracking, or user interface interaction. By way of example and not by way of limitation, the lens of the augmented reality system may be adjusted based on the determined gaze of the user. In certain embodiments, a low fidelity eye tracker may be used by an augmented reality system, a virtual reality system, or a mixed reality system to improve the results of the determined user gaze.

[0031] In certain embodiments, a user may interact with a user interface through the user's line of sight. In certain embodiments, user interface elements may be generated as augmented reality elements, virtual reality elements, or mixed reality elements. A user may perform functions related to the user interface by directing his or her gaze to a corresponding rendered element. By way of example and not by way of limitation, a messaging icon (which may be coupled to the user's view, such as a corner of the user's view, or may be coupled to the rendered environment) may be rendered as an augmented reality element in the augmented reality environment presented to the user. If the augmented reality system determines that the user is attempting to see the messaging icon, the augmented reality system may activate a messaging application for the user to interact with. In certain embodiments, the determined gaze may be used to navigate through the user interface. By way of example and not by way of limitation, a user may scroll up or down on a menu presented to the user (a menu coupled to the user's view or the rendered environment) by looking up or down to a particular point. In certain embodiments, the determined gaze may be used to perform a selection within the user interface. By way of example and not by way of limitation, if a user is presented with multiple options within an augmented reality element (which may be coupled to the user's view or augmented reality environment), the user may select one of the options by looking at the augmented reality element that corresponds to that option. In certain embodiments, the user's gaze may be used in conjunction with a gesture (e.g., the user pointing at the augmented reality element) to determine what option the user selects.

[0032] In certain embodiments, the machine learning model may develop a personalized gaze profile. By way of example and not by way of limitation, a user may sign into an account (e.g., a social networking account) on a virtual reality system, which links the account to the virtual reality system. The machine learning model may be trained to determine the gaze of a particular user based on the user's interaction with the virtual reality environment. By way of example and not by way of limitation, a user may view a virtual reality environment slightly differently than most users, and the machine learning model may accurately determine the gaze of a particular user. The machine learning model may also be trained to determine the gaze of a particular user based on eye-tracking sensors (e.g., if the device has those sensors). In certain embodiments, if the user signs into their account on another virtual reality system or device (e.g., an augmented reality system or a mixed reality system), the personalized gaze profile may be used by the machine learning model to determine the user's gaze. In certain embodiments, the personalized gaze profile may be a machine learning model trained for a particular user.

[0033] Referring to FIG. 1, an exemplary process 100 of using a machine learning model to determine a user's gaze is shown. In certain embodiments, the process 100 may include receiving input data 102, 104, 106 at a virtual reality system or other processing device. In certain embodiments, the input data may include headset tracking data 102, hand tracking data 104, and scene information 106. In certain embodiments, the headset tracking data 102 may include sensor data from a virtual reality headset of the virtual reality system, or from a standalone sensor of the virtual reality system. The headset tracking data 102 may be used to determine a head pose of the user. The head pose may indicate a position and orientation of the virtual reality headset relative to the virtual reality environment or relative to the physical environment. In certain embodiments, the hand tracking data 104 may include sensor data from a virtual reality controller of the virtual reality system, or from a standalone sensor of the virtual reality system. The hand tracking data 104 may be used to determine a hand pose of the user. The hand pose may indicate a position and orientation of the virtual reality controller relative to the virtual reality environment or the virtual reality headset. In particular embodiments, the scene information 106 may be retrieved from an application (e.g., a current application running on a virtual reality system for generating the virtual reality environment) or determined based on data (e.g., RGBD data) of the virtual reality environment. In particular embodiments, the scene information 106 may indicate semantic information of the virtual reality elements in the virtual reality environment as well as three-dimensional coordinates associated with each virtual reality element. The scene information 106 may also indicate other information, such as color and depth data associated with the virtual reality environment. The scene information 106 may include a task that a user is completing within the virtual reality environment.

[0034] In certain embodiments, the input data 102, 104, 106 may be passed to a machine learning model 108 that determines a predicted three-dimensional gaze 110. In certain embodiments, the machine learning model 108 may have been previously trained by comparing the predicted gaze to a ground truth. In certain embodiments, the machine learning model may determine that some tasks may lead to the user looking at a particular point in the virtual reality environment. For example, if the user is cooking in the virtual reality environment, the virtual reality system may send the scene information 106 along with other input data 102, 104 to the machine learning model 108. The machine learning model 108 may determine that because the user is cooking, the user should look at some objects in the virtual reality environment, such as cooking utensils. This information may be used to predict a focus 110 of where the user is looking, since users may generally follow the same gaze patterns when performing the same or similar tasks. In certain embodiments, additional information, such as the category and / or subcategory of the application, may be used to determine the predicted focus 110. By way of example and not by way of limitation, a category of application (e.g., sports games) may dictate certain actions and subsequent gaze patterns that a user may typically follow while performing actions related to the category. Categorization of these actions may train a machine learning model to identify gaze patterns that correspond to a particular category of application. Furthermore, in certain embodiments, the machine learning model may more specifically determine gaze patterns associated with certain actions that a user performs within a virtual reality environment.

[0035] In certain embodiments, the machine learning model may identify relationships between objects in the virtual reality environment. By way of example and not by way of limitation, if a user is performing a cooking action, such as making a peanut butter and jelly sandwich, the machine learning model may identify the objects that will be used for the action and the sequence of events that the user may follow to complete the task of making a sandwich. The sequence of events may dictate the gaze pattern of where the user will look while making the sandwich. For example, the user may start by grabbing bread or utensils. Thus, the user may look at the bread first and follow the process.

[0036] FIG. 2 illustrates an example virtual reality environment 200 for which a predicted line of sight is determined. The virtual reality environment 200 may comprise multiple objects 202. For example, one of the objects 202 may be a clipboard 202a. The virtual reality environment 200 may display a user's hand 204 within the virtual reality environment 200. In certain embodiments, the user's hand 204 may correspond to a virtual reality controller being held by the user. In certain embodiments, the virtual reality system may determine a head pose and a hand pose based on the sensor data described herein. The head pose may indicate a position and orientation of the user's head within the virtual reality environment 200. The hand pose may indicate a position and orientation of the user's hands within the virtual reality environment 200. The virtual reality system may access scene information of the virtual reality environment. The scene information may identify various objects 202 within the virtual reality environment 200 as well as the positions of the objects 202 within the virtual reality environment. The scene information may indicate a task the user is performing, e.g., the user is picking up the clipboard 202a on command. Thus, the user may be reading the clipboard 202a. The virtual reality system may process the determined head pose, the determined hand pose, and the scene information using a machine learning model to determine a predicted focal point 206 that represents a predicted gaze of the user. The machine learning model may determine that because the user is reading the clipboard 202a, the predicted focal point 206 may be located on or near the clipboard 202a. In certain embodiments, the virtual reality system may alter the display of the virtual reality environment 200 based on the predicted gaze.

[0037] FIG. 3 illustrates an example process 300 for training a machine learning model to predict a user's gaze. In certain embodiments, the process 300 may begin by feeding a plurality of images, such as image 302, to the machine learning model. The image 302 may be processed to identify an object of interest 304. In certain embodiments, the machine learning model may use input data described herein to determine the object of interest. In certain embodiments, the image 302 may correspond to a target practice virtual reality environment in which a user may aim at various targets 304i, 304j, 304k, 304l by using objects 304a-304h. In certain embodiments, the machine learning model may access a ground truth image 306 that includes a plurality of identified determined focal points 308a-308m of the user's gaze. In certain embodiments, the ground truth image 306 may be generated by the use of an eye tracking sensor and / or user verification. The plurality of determined focal points 308 in the ground truth image 306 may be represented by highlighted green areas. In certain embodiments, the machine learning model may generate a predicted focus image 310 including multiple predicted focal points 312a-312m. In certain embodiments, each of the predicted focal points 312 may be presented as a heat map representing the probability that the predicted focal point 312 is at that location. In certain embodiments, lighter shades of green may represent a low probability that the predicted focal point 312 may be located. Red may represent a high probability that the predicted focal point 312 may be located. The predicted focal point 312 may be compared to the determined focal point 308 of the ground truth image 306. In certain embodiments, the machine learning model may be further updated based on the comparison of the determined focal point 308 and the predicted focal point 312. Furthermore, after a large number of samples, the predicted focal point 312 generated by the machine learning model may be more accurate than the determined focal point 308. For example, eye tracking sensors are not always accurate. In certain embodiments, the machine learning model may also use input data to accurately determine the predicted focal point 312.In certain embodiments, the machine learning model may determine the predicted focal point 312 based on the task the user is accomplishing. For example, the objects 304a-304h may all have a higher probability that the user is looking at them instead of the targets 304j-304l. By way of example and not by way of limitation, the machine learning model may determine that the user is completing a particular task of hitting the targets 304j-304l and will need the objects 304a-304h to hit the targets 304j-304l. Thus, the user may be more likely to look at the objects 304a-304h first. In certain embodiments, the machine learning model may identify a gaze sequence. A gaze sequence may be a pattern that a user follows to view various rendered environments. By way of example and not by way of limitation, a user may look first at the largest object in the rendered environment and then at the next largest object, or the user may look first at the closest object in the rendered environment and then at the next closest object. For a target practice virtual reality environment, as a user is completing a task of hitting a target 304j-304l, the user may follow a particular gaze sequence. That is, the user may first look at the object 304a-304h to pick it up to throw at the target 304j-304l. After the user selects the object, the user may direct his / her attention to the target 304j-304l. Thus, the machine learning model may predict that the objects 304a-304h may initially have a high probability that the predicted focal point 312 is located at their respective locations, and then the targets 304j-304l may have a high probability that the predicted focal point 312 is located at their respective locations, and the objects 304a-304h may have a reduced probability.

[0038] FIG. 4 illustrates an exemplary method 400 for determining a user's gaze using a machine learning model. In certain embodiments, a virtual reality system may be used to determine a predicted focus of where a user is looking. Method 400 may begin at step 410, where a computing system (e.g., a virtual reality system) may generate and display a rendered environment to a user. At step 420, the computing system may determine a user's head pose based on headset tracking data associated with a headset worn by the user. At step 430, the computing system may determine a user's hand pose based on hand tracking data. At step 440, the computing system may access scene information associated with the displayed rendered environment. At step 450, the computing system may use a machine learning model to determine a user's predicted focus in the rendered environment by processing the head pose, hand pose, and scene information. Certain embodiments may repeat one or more steps of the method of FIG. 4, as appropriate. Although this disclosure describes and shows certain steps of the method of FIG. 4 as occurring in a particular order, this disclosure contemplates that any suitable steps of the method of FIG. 4 be occurring in any suitable order. Moreover, while this disclosure describes and shows an example method for determining a user's gaze using a machine learning model that includes certain steps of the method of FIG. 4, this disclosure contemplates any suitable method of determining a user's gaze using a machine learning model that includes any suitable steps, which may include all, some, or none of the steps of the method of FIG. 4, as appropriate. Furthermore, although this disclosure describes and shows certain components, devices, or systems that perform certain steps of the method of FIG. 4, this disclosure contemplates any suitable combination of any suitable components, devices, or systems that perform any suitable steps of the method of FIG. 4.

[0039] Although this disclosure describes and illustrates processes in the context of a virtual reality system performing various functions, a computing system (e.g., a server embodied as social networking system 160 or third party system 170) may handle the processing and send the results to the virtual reality system. In certain embodiments, a computing system may receive images and send the images to another computing system for processing.

[0040] FIG. 5 illustrates an exemplary network environment 500 associated with a virtual reality system. The network environment 500 includes a user 501 interacting with a client system 530, a social networking system 560, and a third party system 570, connected to each other by a network 510. Although FIG. 5 illustrates a particular arrangement of the user 501, the client system 530, the social networking system 560, the third party system 570, and the network 510, this disclosure contemplates any suitable arrangement of the user 501, the client system 530, the social networking system 560, the third party system 570, and the network 510. By way of example and not by way of limitation, two or more of the user 501, the client system 530, the social networking system 560, and the third party system 570 may be directly connected to each other, bypassing the network 510. As another example, two or more of the client system 530, the social networking system 560, and the third party system 570 may be physically or logically co-located with each other, in whole or in part. 5 illustrates a particular number of users 501, client systems 530, social networking systems 560, third party systems 570, and networks 510, this disclosure contemplates any suitable number of client systems 530, social networking systems 560, third party systems 570, and networks 510. By way of example and not by way of limitation, network environment 500 may include a plurality of users 501, client systems 530, social networking systems 560, third party systems 570, and networks 510.

[0041] This disclosure contemplates any suitable network 510. By way of example and not limitation, one or more portions of the network 510 may include an ad-hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more of these. The network 510 may include one or more networks 510.

[0042] Links 550 may connect client systems 530, social networking systems 560, and third party systems 570 to communications network 510 or to each other. This disclosure contemplates any suitable links 550. In particular embodiments, one or more links 550 include one or more wireline links (such as, for example, Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)), wireless links (such as, for example, Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)), or optical links (such as, for example, Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)). In particular embodiments, one or more links 550 each include an ad-hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a cellular technology based network, a satellite communication technology based network, another link 550, or a combination of two or more such links 550. The links 550 need not necessarily be the same throughout the network environment 500. One or more first links 550 may differ in one or more respects from one or more second links 550.

[0043] In certain embodiments, the client system 530 may be an electronic device that includes hardware, software, or embedded logic components, or a combination of two or more such components, capable of performing the appropriate functionality implemented or supported by the client system 530. By way of example and not by way of limitation, the client system 530 may include a computer system, such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular telephone, a smartphone, a virtual reality headset and controller, other suitable electronic devices, or any suitable combination thereof. This disclosure contemplates any suitable client system 530. The client system 530 may enable a network user at the client system 530 to access the network 510. The client system 530 may enable a user of the client system 530 to communicate with other users at other client systems 530. The client system 530 may generate a virtual reality environment for a user to interact with content.

[0044] In certain embodiments, the client system 530 may include a virtual reality (or augmented reality) headset 532, such as an OCULUS RIFT, and a virtual reality input device(s) 534, such as a virtual reality controller. A user at the client system 530 may wear the virtual reality headset 532 and use the virtual reality input device(s) to interact with a virtual reality environment 536 generated by the virtual reality headset 532. Although not shown, the client system 530 may also include a separate processing computer and / or any other components of a virtual reality system. The virtual reality headset 532 may generate a virtual reality environment 536 that includes system content 538 (including, but not limited to, an operating system), such as software or firmware updates, and may also include third party content 540, such as content from an application or content dynamically downloaded from the Internet (e.g., web page content). The virtual reality headset 532 may include sensor(s) 542, such as an accelerometer, gyroscope, magnetometer, etc., for generating sensor data that tracks the location of the headset device 532. The headset 532 may also include an eye tracker for tracking the position of the user's eyes or the viewing direction of the user's eyes. The client system may use data from the sensor(s) 542 to determine velocity, orientation, and gravity relative to the headset. The virtual reality input device(s) 534 may include sensor(s) 544, such as an accelerometer, gyroscope, magnetometer, and touch sensor, for generating sensor data that tracks the location of the input device 534 and the position of the user's fingers. The client system 530 may utilize outside-in tracking, where a tracking camera (not shown) is placed outside the virtual reality headset 532 and within the line of sight of the virtual reality headset 532.In outside-in tracking, the tracking camera may track the location of the virtual reality headset 532 (e.g., by tracking one or more infrared LED markers on the virtual reality headset 532). Alternatively or additionally, the client system 530 may utilize inside-out tracking, where a tracking camera (not shown) may be placed on or within the virtual reality headset 532 itself. In inside-out tracking, the tracking camera may capture images around the tracking camera in the real world and may use the changing perspective of the real world to determine its own position in space.

[0045] The third-party content 540 may include a web browser, such as MICROSOFT INTERNET EXPLORER, GOOGLE CHROME, or MOZILLA FIREFOX, and may have one or more add-ons, plug-ins, or other extensions, such as TOOLBAR or YAHOO TOOLBAR. A user at the client system 530 may enter a Uniform Resource Locator (URL) or other address that directs the web browser to a particular server (such as server 562, or a server associated with the third-party system 570), and the web browser may generate a HyperText Transfer Protocol (HTTP) request and communicate the HTTP request to the server. The server may accept the HTTP request and communicate one or more HyperText Markup Language (HTML) files to the client system 530 that respond to the HTTP request. The client system 530 may render a web interface (e.g., a web page) based on the HTML files from the server for presentation to the user. This disclosure contemplates any suitable source file. By way of example and not by way of limitation, a web interface may be rendered from an HTML file, an Extensible Hypertext Markup Language (XHTML) file, or an Extensible Markup Language (XML) file, depending on the particular needs. Such interfaces may also execute scripts, such as, for example and without limitation, those written in JAVASCRIPT, JAVA, MICROSOFT SILVERLIGHT, combinations of markup languages ​​and scripts, such as AJAX (Asynchronous JAVASCRIPT and XML), and the like. References herein to a web interface encompass, where appropriate, one or more corresponding source files (which a browser may use to render the web interface), and vice versa.

[0046] In certain embodiments, social networking system 560 may be a network addressable computing system capable of hosting an online social network. Social networking system 560 may generate, store, receive, and send social networking data, such as, for example, user profile data, concept profile data, social graph information, or other suitable data related to an online social network. Social networking system 560 may be accessed by other components of network environment 500, either directly or via network 510. By way of example and not by way of limitation, client system 530 may access social networking system 560 using a web browser of third party content 540, or a native application associated with social networking system 560 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof), either directly or via network 510. In certain embodiments, social networking system 560 may include one or more servers 562. Each server 562 may be a single server or a distributed server across multiple computers or multiple data centers. Servers 562 may be of various types, such as, for example, but not limited to, a web server, a news server, a mail server, a message server, an advertisement server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In particular embodiments, each server 562 may include hardware, software, or embedded logic components, or a combination of two or more such components, for performing the appropriate functionality implemented or supported by server 562. In particular embodiments, social networking system 560 may include one or more data stores 564.Data stores 564 may be used to store various types of information. In particular embodiments, information stored in data stores 564 may be organized according to a unique data structure. In particular embodiments, each data store 564 may be a relational database, a column-oriented database, a correlation database, or other suitable database. Although this disclosure describes or illustrates particular types of databases, this disclosure contemplates any suitable type of database. Particular embodiments may provide interfaces that allow client systems 530, social networking systems 560, or third party systems 570 to manage, retrieve, modify, add, or delete information stored in data stores 564.

[0047] In particular embodiments, social networking system 560 may store one or more social graphs in one or more data stores 564. In particular embodiments, a social graph may include multiple nodes, which may include multiple user nodes (each corresponding to a particular user) or multiple concept nodes (each corresponding to a particular concept), and multiple edges connecting the nodes. Social networking system 560 may provide users of an online social network with the ability to communicate and interact with other users. In particular embodiments, a user may join an online social network via social networking system 560 and then add connections (e.g., relationships) to some other users of social networking system 560 with whom the user wishes to be connected. As used herein, the term "friend" may refer to any other user of social networking system 560 with whom the user has connected, associated, or formed a relationship via social networking system 560.

[0048] In particular embodiments, social networking system 560 may provide users with the ability to take actions regarding various types of items or objects supported by social networking system 560. By way of example and not limitation, items and objects may include groups or social networks to which a user of social networking system 560 may belong, events or calendar entries in which a user may be interested, computer-based applications that a user may use, transactions that allow a user to buy or sell items via the service, interactions with advertisements that a user may conduct, or other suitable items or objects. Users may interact with what may be represented in social networking system 560 or by external systems of third-party systems 570 that are separate from social networking system 560 and coupled to social networking system 560 via network 510.

[0049] In particular embodiments, social networking system 560 may be capable of linking various entities. By way of example and not by way of limitation, social networking system 560 may enable users to interact with each other as well as receive content from third party systems 570 or other entities, or allow users to interact with these entities through application programming interfaces (APIs) or other communication channels.

[0050] In particular embodiments, third-party system 570 may include one or more types of servers, one or more data stores, one or more interfaces including but not limited to APIs, one or more web services, one or more content sources, one or more networks, or any other suitable components with which, for example, a server may communicate. Third-party system 570 may be operated by an entity different from the entity that operates social networking system 560. However, in particular embodiments, social networking system 560 and third-party system 570 may operate in conjunction with each other to provide social networking services to users of social networking system 560 or third-party system 570. In this sense, social networking system 560 may provide a platform or backbone that other systems, such as third-party system 570, may use to provide social networking services and functionality to users across the Internet.

[0051] In particular embodiments, third party system 570 may include a third party content object provider. A third party content object provider may include one or more sources of content objects that may be communicated to client system 530. By way of example and not by way of limitation, the content objects may include information about things or activities of interest to a user, such as, for example, movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. By way of another example and not by way of limitation, the content objects may include incentive content objects, such as coupons, discount tickets, gift certificates, or other suitable incentive objects.

[0052] In particular embodiments, social networking system 560 also includes user-generated content objects that may enhance user interaction with social networking system 560. User-generated content may include anything that a user may add, upload, send, or "post" to social networking system 560. By way of example and not by way of limitation, a user communicates posts from client system 530 to social networking system 560. Posts may include data such as status updates or other textual data, location information, photos, videos, links, music, or other similar data or media. Content may also be added to social networking system 560 by third parties through "communication channels," such as news feeds or streams.

[0053] In certain embodiments, the social networking system 560 may include various servers, subsystems, programs, modules, logs, and data stores. In certain embodiments, the social networking system 560 may include one or more of a web server, an action logger, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object exposure log, an inference module, a permission / privacy server, a search module, an ad targeting module, a user interface module, a user profile store, a connection store, a third-party content store, or a location store. The social networking system 560 may also include suitable components, such as network interfaces, security mechanisms, load balancers, failover servers, administration and network operations consoles, other suitable components, or any suitable combination thereof. In certain embodiments, the social networking system 560 may include one or more user profile stores for storing user profiles. A user profile may include, for example, biographical information, demographic information, behavioral information, social information, or other types of descriptive information, such as work history, education history, hobbies or preferences, interests, affinities, or location. Interest information may include interests related to one or more categories. The categories may be general or specific. By way of example and not by way of limitation, if a user "likes" an article about a shoe brand, the category may be the brand, or the general category of "shoes" or "clothing." A connection store may be used to store connection information about users. Connection information may indicate users who have similar or common work history, group membership, hobbies, education history, or who are related in any way or share common attributes. Connection information may also include user-defined connections between different users (both internal and external) and content.A web server may be used to link the social networking system 560 to one or more client systems 530 or one or more third party systems 570 via the network 510. The web server may include a mail server or other messaging functionality for receiving and routing messages between the social networking system 560 and one or more client systems 530. An API request server may enable the third party system 570 to access information from the social networking system 560 by calling one or more APIs. An action logger may be used to receive communication from the web server regarding user actions on or off the social networking system 560. In conjunction with the action log, a third party content object log may be maintained of user exposure to third party content objects. A notification controller may provide information about content objects to the client system 530. Information may be pushed to the client system 530 as a notification or information may be pulled from the client system 530 in response to a request received from the client system 530. The authorization server may be used to enforce one or more privacy settings of a user of the social networking system 560. A user's privacy settings determine how certain information associated with the user may be shared. The authorization server may allow a user to opt-in or opt-out of their actions being logged by the social networking system 560 or shared with other systems (e.g., third party system 570), such as by setting appropriate privacy settings. The third party content object store may be used to store content objects received from third parties, such as the third party system 570.The location store may be used to store location information received from client systems 530 associated with users. The advertising pricing module may combine social information, current time, location information, or other suitable information to provide relevant advertisements to users in the form of notifications.

[0054] FIG. 6 illustrates an exemplary computer system 600. In certain embodiments, one or more computer systems 600 perform one or more steps of one or more methods described or illustrated herein. In certain embodiments, one or more computer systems 600 provide functionality described or illustrated herein. In certain embodiments, software running on one or more computer systems 600 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Certain embodiments include one or more portions of one or more computer systems 600. As used herein, references to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, references to a computer system may encompass one or more computer systems, where appropriate.

[0055] The present disclosure contemplates any suitable number of computer systems 600. The present disclosure contemplates computer system 600 taking any suitable physical form. By way of example and not limitation, computer system 600 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (e.g., a computer on module (COM) or system on module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 600 may include one or more computer systems 600, may be single or distributed, may span multiple locations, span multiple machines, span multiple data centers, or may reside in a cloud that may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 600 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example, and not by way of limitation, the one or more computer systems 600 may perform one or more steps of one or more methods described or shown herein in real time or in batch mode. The one or more computer systems 600 may perform one or more steps of one or more methods described or shown herein at different times or in different locations, where appropriate.

[0056] In a particular embodiment, computer system 600 includes a processor 602, memory 604, storage 606, an input / output (I / O) interface 608, a communications interface 610, and a bus 612. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0057] In particular embodiments, the processor 602 includes hardware for executing instructions, such as those that make up a computer program. By way of example and not by way of limitation, to execute instructions, the processor 602 may retrieve (or fetch) instructions from an internal register, an internal cache, memory 604, or storage 606, decode and execute those instructions, and then write one or more results to an internal register, an internal cache, memory 604, or storage 606. In particular embodiments, the processor 602 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates the processor 602 including any suitable number of any suitable internal caches, where appropriate. By way of example and not by way of limitation, the processor 602 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in an instruction cache may be copies of instructions in memory 604 or storage 606, and the instruction cache may speed up retrieval of those instructions by the processor 602. The data in the data cache may be a copy of data in memory 604 or storage 606 on which instructions executing in the processor 602 operate, results of previous instructions executed in the processor 602 for access by subsequent instructions executing in the processor 602 or for writing to memory 604 or storage 606, or other suitable data. The data cache may speed up read or write operations by the processor 602. The TLB may speed up virtual-address translation for the processor 602. In particular embodiments, the processor 602 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates the processor 602 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, the processor 602 may include one or more arithmetic logic units (ALUs), be a multi-core processor, or include one or more processors 602.Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0058] In a particular embodiment, memory 604 includes a main memory for storing instructions for processor 602 to execute or data upon which processor 602 operates. By way of example and not by way of limitation, computer system 600 may load instructions into memory 604 from storage 606 or another source (such as, for example, another computer system 600). Processor 602 may then load the instructions from memory 604 into an internal register or cache. To execute instructions, processor 602 may retrieve instructions from the internal register or cache and decode those instructions. During or after execution of instructions, processor 602 may write one or more results (which may be intermediate or final results) to an internal register or cache. Processor 602 may then write one or more of those results to memory 604. In particular embodiments, the processor 602 executes only instructions in one or more internal registers or caches, or in memory 604 (as opposed to storage 606 or elsewhere), and operates only on data in one or more internal registers or caches, or in memory 604 (as opposed to storage 606 or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple the processor 602 to the memory 604. The bus 612 may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between the processor 602 and the memory 604 to facilitate accesses to the memory 604 requested by the processor 602. In particular embodiments, the memory 604 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. This RAM may be dynamic RAM (DRAM) or static RAM (SRAM), where appropriate. Moreover, this RAM may be single-ported or multi-ported RAM, where appropriate. This disclosure contemplates any suitable RAM. Memory 604 may include one or more memories 604, where appropriate.Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

[0059] In particular embodiments, storage 606 includes mass storage for data or instructions. By way of example and not limitation, storage 606 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Storage 606 may include removable or non-removable (or fixed) media, where appropriate. Storage 606 may be internal or external to computer system 600, where appropriate. In particular embodiments, storage 606 is a non-volatile solid-state memory. In particular embodiments, storage 606 includes read-only memory (ROM). Where appropriate, this ROM may be a mask program ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these. The present disclosure contemplates mass storage 606 taking any suitable physical form. Storage 606 may include one or more storage control units to facilitate communications between processor 602 and storage 606, as appropriate. Storage 606 may include one or more storages 606, as appropriate. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0060] In particular embodiments, I / O interface 608 includes hardware, software, or both that provide one or more interfaces for communication between computer system 600 and one or more I / O devices. Computer system 600 may include one or more of these I / O devices, as appropriate. One or more of these I / O devices may enable communication between a person and computer system 600. By way of example and not by way of limitation, the I / O devices may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet, a touch screen, a trackball, a video camera, another suitable I / O device, or a combination of two or more of these. The I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interfaces 608 for those I / O devices. As appropriate, I / O interface 608 may include one or more device or software drivers that enable processor 602 to drive one or more of these I / O devices. I / O interface 608 may include one or more I / O interfaces 608, where appropriate. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0061] In particular embodiments, the communication interface 610 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication, etc.) between the computer system 600 and one or more other computer systems 600 or one or more networks. By way of example and not by way of limitation, the communication interface 610 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface 610 for that network. By way of example and not by way of limitation, the computer system 600 may communicate with one or more portions of an ad-hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system 600 may communicate with a wireless PAN (WPAN) (e.g., such as a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (e.g., such as a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more of these. Computer system 600 may include any suitable communication interface 610 for any of these networks, where appropriate. Communication interface 610 may include one or more communication interfaces 610, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

[0062] In particular embodiments, bus 612 includes hardware, software, or both that couple components of computer system 600 together. By way of example, and not limitation, bus 612 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus, or a combination of two or more of these. Bus 612 may include one or more buses 612, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0063] As used herein, the one or more computer-readable non-transitory storage media may include, where appropriate, one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. The computer-readable non-transitory storage media may, where appropriate, be volatile, non-volatile, or a combination of volatile and non-volatile.

[0064] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or the context dictates otherwise. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or the context dictates otherwise. Moreover, "and" is both jointly and severally, unless expressly indicated otherwise or the context dictates otherwise. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or the context dictates otherwise.

[0065] The scope of the present disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments described or shown herein that would be understood by a person skilled in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or shown herein. Moreover, although the present disclosure describes and shows each embodiment herein as including certain components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or substitution of any of the components, elements, features, functions, operations, or steps described or shown anywhere herein that would be understood by a person skilled in the art. Furthermore, references in the appended claims to an apparatus or system, or an apparatus or system component, that is adapted, arranged, capable, configured, enabled, operable, or operative to perform a particular function encompass that apparatus, system, component, or particular function thereof, so long as the apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative, regardless of whether the apparatus, system, component, or particular function thereof is activated, turned on, or unlocked. Further, while the present disclosure describes or shows certain embodiments as providing certain advantages, a particular embodiment may provide none, some, or all of these advantages.

Claims

1. A computing system that does not include an eye tracker, generating and displaying a rendered environment to a user; determining a head pose of the user based on headset tracking data associated with a headset worn by the user; determining a hand pose of the user based on hand tracking data; accessing scene information associated with the displayed rendered environment; determining a predicted focus of the user within the rendered environment by processing the head pose, the hand pose, and the scene information using a machine learning model; Including, The method, wherein the machine learning model is trained based on ground truth data previously captured by another computing system, the ground truth data including head pose data, hand pose data, scene information data, and eye tracking data.

2. adjusting an image presented to the user by the computing system based on the predicted focus of the user within the rendered environment. The method of claim 1 further comprising:

3. The method of claim 1 , wherein the head pose is determined relative to the rendered environment.

4. The method of claim 1 , wherein the hand pose is determined relative to one of the rendered environment or the headset worn by the user.

5. Determining the hand pose of the user comprises: identifying the user's hand based on one or more cameras coupled to the computing system analyzing a plurality of images including the user's hand; The method of claim 1 , comprising:

6. The method of claim 1 , wherein the hand tracking data relates to a device held or worn by the user's hand.

7. Determining an action to be performed by the device held or worn by the hand of the user. and wherein determining the predicted focus further comprises using the machine learning model to process the action performed by the device held or worn by the hand of the user.

8. The method of claim 1 , wherein the scene information includes color and depth data, or the scene information includes semantic information of one or more elements in the rendered environment.

9. The method of claim 1 , wherein the predicted focal point is a three-dimensional coordinate within the rendered environment.

10. The method of claim 1 , wherein the rendered environment is associated with an application, the application is associated with a category, and the scene information includes the category associated with the application.

11. using the machine learning model to generate a confidence map of one or more locations for the predicted focal point.

2. The method of claim 1 , further comprising: wherein the reliability map assigns a probability that the predicted focal point is at each of the one or more locations; and wherein determining the predicted focal point comprises utilizing the reliability map.

12. The method of claim 1 , wherein the predicted focus is determined without an eye-tracking sensor.

13. The method of claim 1 , wherein the rendered environment comprises one or more of an augmented reality environment, a virtual reality environment, or a mixed reality environment.

14. One or more computer-readable, non-transitory storage media embodying software, the software being operable, when executed, to perform the method of any one of claims 1 to 13.

15. 14. A system comprising one or more processors and a non-transitory memory coupled to the processors containing instructions executable by the processors, the system being operable, when the processor executes the instructions, to perform the method of any one of claims 1 to 13.

Images