Multi-device gaze tracking

Through the gaze tracker and shared computing components, monitoring the user's gaze data and assigning tasks across devices, the inefficiency of computing devices in multi-user and multi-device scenarios is solved, and more efficient computing resource usage and user experience is achieved.

CN118922804APending Publication Date: 2024-11-08MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380029467.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-22
Filing Date
2023-02-13
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the multi-user and multi-device scenarios, existing computing devices have low efficiency in measuring input data processing, resulting in inconvenient user operation, reduced productivity, and uneven allocation of computing resources, resulting in lower video game quality and other problems.

Method used

The gaze tracker component and shared computing component are adopted to monitor user gaze data through optical sensors and acoustic sensors, allocate tasks across multiple computing devices, realize load balancing, and optimize the use of computing resources.

Benefits of technology

It improves user interface efficiency, reduces the complexity and frustration of user operations, optimizes the allocation of computing resources, and improves the performance of resource-intensive tasks such as video games.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922804A_ABST
    Figure CN118922804A_ABST
Patent Text Reader

Abstract

Aspects of the present disclosure relate to multi-user, multi-device gaze tracking. In an example, a system includes at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations. The set of operations includes: identifying a plurality of computing devices; and identifying one or more users. The set of operations may also include receiving gaze input data and load data from two or more of the plurality of computing devices. The set of operations may also include performing load balancing among the plurality of devices, wherein the load balancing includes allocating one or more tasks from a first computing device of the plurality of computing devices to a second computing device of the plurality of computing devices based on the gaze input data.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Computing devices may include optical sensors (e.g., cameras, RGB sensors, infrared sensors, LIDAR sensors), acoustic sensors (e.g., ultrasound), or other sensors (e.g., radar sensors, optical flow sensors, motion sensors) that can monitor a user's gaze. In addition, computing devices may have only limited capacity for completing certain tasks, resulting in reduced productivity, increased frustration, and other impairments.

[0002] Aspects of the present disclosure are described with respect to these and other general considerations. Likewise, although relatively specific problems have been discussed, it should be understood that the embodiments should not be limited to solving the specific problems identified in the background. Summary of the invention

[0003] Various aspects of the present disclosure relate to systems, methods, and media for processing gaze input data to adapt the behavior of one or more computing devices. Another aspect of the present disclosure relates to systems, methods, and media for processing load data to distribute tasks across multiple computing devices based on gaze input data corresponding to one or more users.

[0004] In some aspects of the present disclosure, a system is provided. The system includes: at least one processor; and a memory storing instructions, which, when executed by the at least one processor, cause the system to perform a set of operations. The set of operations includes: identifying multiple computing devices; and identifying one or more users. The set of operations may also include: receiving gaze input data and load data from two or more computing devices in the multiple computing devices. The set of operations may also include: performing load balancing between the multiple devices, wherein the load balancing includes allocating one or more tasks from a first computing device in the multiple computing devices to a second computing device in the multiple computing devices based on the gaze input data.

[0005] In some aspects disclosed herein, a system is provided. The system includes: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations. The set of operations includes: identifying two or more computing devices; and identifying two or more users. The set of operations may also include: receiving gaze input data corresponding to one or more users from the two or more computing devices. The set of operations may also include: determining an action associated with a particular computing device based on the gaze input data. The set of operations may also include: adapting the two or more computing devices based on the determined action.

[0006] In some aspects of the present disclosure, a method for processing gaze input data to control a computing device is provided. The method includes: identifying one or more computing devices; and identifying one or more users. The method may also include: displaying one or more elements on the one or more computing devices. The method may also include: receiving gaze input data corresponding to one or more users from the one or more computing devices. The method may also include: identifying metadata corresponding to the one or more elements based on the gaze input data. The method may also include: adapting the one or more computing devices to change their display based on the metadata.

[0007] This Summary is provided to introduce in a simplified form some of the concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0009] Figure 1 An overview of an example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0010] Figure 2 Shows Figure 1 Detailed diagram of the gaze tracker engine.

[0011] Figure 3 Shows Figure 1 Detailed diagram of the shared computing engine.

[0012] Figure 4 An overview of an example system for multi-user, multi-device gaze tracking in accordance with various aspects described herein is shown.

[0013] Figure 5 An overview of an example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0014] Figure 6 An overview of an example method for processing gaze input data to perform actions that affect the behavior of a computing device is presented.

[0015] Figure 7 An overview of an example method for processing gaze input data and load data to distribute tasks across computing devices is presented.

[0016] Fig. 8A An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0017] Figure 8BAn example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0018] Fig. 9 An overview of an example method for processing gaze input data to perform actions that affect the behavior of a computing device is presented.

[0019] Fig. 10A An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0020] Fig. 10B An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0021] Fig.11 An overview of an example method for processing gaze input data and load data to distribute tasks across computing devices is presented.

[0022] Fig. 12A An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0023] Fig. 12B An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0024] Fig.13 An overview of an example method for processing gaze input data to perform actions that affect the behavior of a computing device is presented.

[0025] Fig.14 An example system for gaze tracking in accordance with aspects described herein is shown.

[0026] Fig.15 An overview of an example method for processing gaze input data to perform actions that affect the behavior of a computing device is presented.

[0027] Fig.16A An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0028] Fig. 16B An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0029] Fig.17A An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0030] Fig. 17B An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0031] Fig.18An overview of an example method for processing gaze input data to perform actions that affect the behavior of a computing device is presented.

[0032] Fig.19A An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0033] Fig.19B An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0034] Fig. 20 An overview of an example method for processing gaze input data to perform actions that affect the behavior of a computing device is presented.

[0035] Fig.21A An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0036] Fig.21B An example system for multi-device gaze tracking in accordance with aspects described herein is shown.

[0037] Fig. 22 An overview of an example method for processing gaze input data to perform actions that affect the behavior of a computing device is presented.

[0038] Fig.23 An example system for device gaze tracking according to aspects described herein is shown.

[0039] Fig.24 An overview of an example method for processing gaze input data to perform actions that affect the behavior of a computing device is presented.

[0040] Fig.25 An example system 2500 for device gaze tracking according to aspects described herein is shown.

[0041] Fig.26 An overview of an example method 2600 for processing gaze input data to perform actions that affect the behavior of a computing device is shown.

[0042] Fig. 27 An example system 2700 for device gaze tracking according to aspects described herein is shown.

[0043] Fig.28 An overview of an example method 2800 for processing gaze input data to perform actions that affect the behavior of a computing device is shown.

[0044] Fig.29 An example system 2900 for device gaze tracking according to aspects described herein is shown.

[0045] Fig.30 An overview of an example method 3000 for processing gaze input data to perform actions that affect the behavior of a computing device is shown.

[0046] Fig.31 An example grid for gaze data collection in accordance with various aspects described herein is shown.

[0047] Fig.32 An example of gaze calibration in accordance with aspects described herein is shown.

[0048] Fig.33 An overview of an example method 3000 for processing gaze input data to perform actions that affect the behavior of a computing device is shown.

[0049] Fig.34 is a block diagram illustrating the physical components of a computing device that can be used to practice aspects of the present disclosure.

[0050] Fig.35A A mobile computing device is shown that can be used to practice embodiments of the present disclosure.

[0051] Fig.35B is a block diagram illustrating the architecture of one aspect of a mobile computing device.

[0052] Fig.36 One aspect of the architecture of a system for processing data received at a computing system from a remote source is shown.

[0053] Fig.37 An exemplary tablet computing device is shown that can perform one or more aspects disclosed herein. DETAILED DESCRIPTION

[0054] In the following detailed description, reference is made to the accompanying drawings which form a part thereof, and specific embodiments or examples are shown therein by way of illustration. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. The embodiments may be implemented as methods, systems, or devices. Therefore, the embodiments may take the form of hardware implementation, complete software implementation, or a combination of software and hardware implementations. Therefore, the following detailed description should not be considered restrictive, and the scope of the present disclosure is limited by the appended claims and their equivalents.

[0055] As described above, computing devices may include optical sensors, such as cameras, that are capable of monitoring a user's gaze. In addition, computing devices may have limited capacity to complete certain tasks, resulting in reduced productivity, increased frustration, and other impairments.

[0056] Typically, conventional computing devices receive input from users through multiple input devices. Conventional input devices may include a mouse, camera, joystick, keyboard, trackpad, or microphone. For example, multiple users may sit around a table where a computer is located. A first user among the multiple users may control a mouse coupled to the computer. The first user among the multiple users may use the mouse to navigate to applications on the computer. However, if a second user among the multiple users desires to navigate to a different application, they will have to control the mouse from the first user among the multiple users to navigate to the different application. When different users work on the same computing device, sharing input devices to control the computing device may be inefficient, frustrating, and interferes with the user's ability to provide a constant stream of feedback to the computing device.

[0057] In other examples, multiple users may stand in front of a computer screen during a presentation. Conventional presentation methods (e.g., marketing presentations) may require market research or focus groups to obtain feedback on how users view the presentation. Such feedback may be inaccurate regarding specific aspects of the presentation, such research may be expensive to gather, and such feedback may be imprecise. Therefore, as discussed herein, it may be useful to gather metadata about where multiple users are looking so that the presentation can be updated to increase engagement from multiple users.

[0058] In other examples, a user may be sitting at a table where a computer is located. One of the user's hands may be on a mouse coupled to the computer and the other hand on the keyboard of the computer. When performing a task on the computer, the user may be required to move both hands to the keyboard (e.g., in order to type a document). Alternatively, the user may have to remove one hand from the keyboard, swing their mouse to visually position the cursor on the display screen of the computer, and then navigate to an application or element displayed on the computer. Such conventional examples are inefficient, frustrating, and interfere with the user's ability to provide a constant stream of feedback to the computing device.

[0059] In other examples, one hand of the user may be on the keyboard, and the second hand may be holding a writing utensil, food or drink, etc. Thus, in order to navigate to another application shown on the display screen, or to select an element shown on the display screen, the user may have to put down their writing utensil, food or drink to use their second hand to operate an input device (e.g., a mouse). After successfully navigating to another application or selecting an element, the user may then remove their hand from the input device (e.g., a mouse) and retrieve their writing utensil, food or drink. Once again, such conventional examples are inefficient, frustrating, and interfere with the user's ability to provide a constant stream of feedback to the computing device.

[0060] When multiple users work across multiple devices, the inefficiency and frustration discussed above may be further exacerbated. For example, if multiple users sit at a table including both desktop computers and laptop computers placed thereon, both desktop computers and laptop computers may include their own input devices. Therefore, if one of the multiple users is working on a desktop computer and expects to navigate to an application on their laptop computer, they will have to remove one or both hands from their desktop computer and operate the input device (e.g., mouse, keyboard, or trackpad) of their laptop computer to navigate to the desired application. Then, in order to navigate to the application on their desktop computer, they will have to remove one or both hands from their laptop computer and operate the input device (e.g., mouse, keyboard, or trackpad) of their desktop computer to navigate to the desired application. At the same time, a second user among multiple users may be physically crossing arms and hands with a first user among multiple users, trying to navigate to an application different from the application that the first user among multiple users expects to navigate. Therefore, when working across multiple computing devices, the inefficiency and frustration discussed above about a single computing device may be exacerbated.

[0061] Still referring to the conventional example of a user working across multiple devices, computing resources may be allocated inefficiently. For example, if the user is running a video game with high-definition graphics on a first computing device (e.g., their laptop) and has multiple background applications running (e.g., email, an internet browser, video streaming, or other applications), the user may experience a degradation in the quality of the video game due to insufficient capacity in the processor or memory of the first computing device that is running the video game with high-definition graphics. This experience may be frustrating, particularly if there are multiple other computing devices (e.g., a second computing device and / or a third computing device) near or in communication with the first computing device that have sufficient capacity in their processors or memory to share some of the load handled by the first computing device, particularly the load of background processes (e.g., the aforementioned email application, internet browser, or video streaming).

[0062] Aspects of the present disclosure may be beneficial in addressing some of the deficiencies, frustrations, and inefficiencies discussed above. Further aspects of the present disclosure may provide additional advantages and benefits, as will become apparent in view of the following discussion.

[0063] For example, aspects of the present disclosure may generally include one or more computing devices configured to interface with one or more users. The computing devices may interface with one or more users via an optical device (e.g., a camera) that receives visual data or gaze data. The visual data may be used to track the gaze of one or more users. One or more computing devices may include a gaze tracker component or engine that monitors the gaze of one or more users relative to a display screen of one or more computing devices.

[0064] The gaze tracker component can also determine one or more actions to be performed across one or more computing devices based on gaze data corresponding to one or more users. In this regard, users can navigate to applications, open applications, select elements displayed on a computing device, or perform any other task based on where they are looking relative to the position of a display screen of the computing device. Such capabilities provide a simplified user interface that improves efficiency and productivity using computing devices.

[0065] The computing device may also include a shared computing component, such as a shared memory space, that monitors load capacity (e.g., processor availability or memory availability) across multiple computing devices to distribute tasks across computing devices. The shared computing component may prioritize tasks determined to be of interest based on gaze data corresponding to one or more users received or generated by the gaze tracker component. The shared computing component may also deprioritize tasks determined to be of no interest (e.g., currently not of interest) based on gaze data corresponding to one or more users received or generated by the gaze tracker component.

[0066] Typically, a shared computing component seeks to optimize load capacity across computing devices by reducing non-essential tasks on a primary computing device (e.g., a computing device on which one or more users are found to be viewing) and offloading non-essential tasks to other computing devices. For example, in the scenario discussed above, where a user is frustrated by the poor quality of their high-definition graphics video game, according to examples disclosed herein, a shared computing component may offload background processes being completed for an email application, an Internet browser, and / or a video stream to other computing devices in communication with the first computing device, thereby freeing up resources for executing the video game on the device that is currently the focus of the user. In this regard, a shared computing component may distribute tasks across multiple computing devices based on the available load capacity (e.g., processor availability or memory availability) of the computing devices.

[0067] Figure 1An overview of an example system 100 for multi-device gaze tracking according to various aspects described herein is shown. As shown, the system 100 includes a synchronization platform 102, a computing device 103, a computing device 104, a peripheral device 106, a peripheral device 108, and a network 110. As shown, the synchronization platform 102, the computing device 103, the computing device 104, the peripheral device 106, and the peripheral device 108 communicate via the network 110, which can include a local area network, a wireless network, or the Internet, or any combination thereof, among other examples.

[0068] Although system 100 shows in an example that computing devices 103 and 104 can communicate with peripheral devices 106 and 108 via network 110, it should be understood that in other examples, peripheral devices 106 and / or 108 can be directly connected to the computing device, such as using a wired (e.g., Universal Serial Bus (USB) or other electrical connection) and / or wireless (e.g., Bluetooth Low Energy (BTLE) or Wi-Fi Direct) connection, etc. Devices 103 to 108 can each be any of a variety of devices, including but not limited to mobile computing devices, tablet computing devices, laptop computing devices, desktop computing devices, Internet of Things (IoT) devices, smart computing devices, or server computing devices, as well as other examples. For example, computing device 103 can be a mobile computing device, peripheral device 106 can be a VR computing device, computing device 104 can be a desktop computing device, and peripheral device 108 can be a tablet device, as well as other examples. One or more of devices 103, 104, and 108 may include or be coupled to gaze tracking hardware, such as an optical sensor (e.g., a camera, an RGB sensor, an infrared sensor, a LIDAR sensor), an acoustic sensor (e.g., ultrasound), or other sensors (e.g., a radar sensor, an optical flow sensor, a motion sensor) that can be used to perform gaze tracking of one or more users.

[0069] The synchronization platform 102 includes a request processor 112, a signal data store 114, and a gaze tracking data store 116. In an example, the synchronization platform 102 synchronizes a set of gaze tracking data between a group of devices (e.g., computing device 103, computing device 104, peripheral device 106, and / or peripheral device 108). Thus, the request processor 112 can receive requests for synchronization data from devices 103 to 108, including, but not limited to, the set of gaze tracking data, training data associated with the identified user signal and associated actions performed by one or more computing devices, and / or environmental context information (e.g., software applications currently being run or currently attached hardware), as well as other examples. These data can be stored by the signal data store 114 and the gaze tracking data store 116. In an example, synchronization can be performed by one or more of the devices 103 to 108 as an alternative or in addition to using centralized synchronization (e.g., centralized synchronization can be provided by the synchronization platform 102).

[0070] The computing device 103 is shown to include a signal identification component or engine 118, a gaze tracker component or engine 120, and a shared computing component or engine 122. In an example, according to various aspects described herein, the signal identification component 118 processes gaze tracking data (e.g., visual data) to generate a set of signals. For example, the signal identification component 118 can process gaze tracking data obtained from any of a variety of other sources of sensors (e.g., cameras), software, or computing devices 103, computing devices 104, peripheral devices 106, and / or peripheral devices 108. In some examples, at least a portion of the obtained data may have been processed on the device receiving the data, such as by the signal identification component 130 of the peripheral device 106. The signal identification component 118 can process the gaze tracking data according to any of a variety of techniques, including but not limited to using a set of rules, machine learning models, according to computer vision techniques and / or mechanisms disclosed herein. The generated set of user signals can be processed by the gaze tracker component 120 and / or the shared computing component 122, as discussed below.

[0071] The gaze tracker component 120 can obtain, receive, update, or otherwise determine gaze tracking data for one or more users from the environment in which the computing device 103 is located. For example, the environment can be a room, a building, or a geographic area with a given radius or other area, among other examples. For example, one or more devices 103 to 108 can be located within the environment in which the gaze tracking data is determined. Similar to the signal identification component 118, the gaze tracking component 120 can generate gaze tracking data based at least in part on data received from one or more devices 104 to 108. For example, the gaze tracking component 120 can receive visual data corresponding to one or more users from a camera on one or more devices 104 to 108. In some cases, at least a portion of the data processed by the signal identification component 118 can be processed by the gaze tracking component 120 (or vice versa in other examples).

[0072] According to some examples, the gaze tracker component 120 can obtain, receive, update, or otherwise determine gaze tracking data for multiple users from an environment in which multiple computing devices 103 to 108 are located. Multiple users can be identified by multiple computing devices (e.g., devices 103 to 108). The gaze tracking component 120 can receive visual data corresponding to each of the multiple users from sensors (e.g., cameras) located on the multiple computing devices. In this regard, the system disclosed herein can identify multiple computing devices, identify multiple users, receive gaze data associated with each of the multiple users from the multiple computing devices, determine actions based on the gaze data corresponding to the multiple users, and / or adapt the behavior of the multiple computing devices based on the determined actions.

[0073] Shared computing component 122 processes a set of signals (e.g., signals generated by signal identification component 118, or signals that may be received from devices 104 to 108) to facilitate pooling of computing resources (e.g., computing resources from computing device 103, computing device 104, peripheral device 106, and / or peripheral device 108). Shared computing component 122 further balances computing loads and may communicate information (e.g., transient information, state information, and key-value pair information) across devices (e.g., device 103, device 104, device 106, and device 108) to facilitate processing and / or execution of various tasks.

[0074] Shared computing component 122 can transmit transient information across devices, such as information that is synchronized but not stored. For example, transient information can be information indicating mouse movement, mouse clicks, keyboard clicks, touch positions on a touch screen, etc. Typically, transient information is used relatively instantaneously by subroutines of software applications that perform certain processes. Transient information may become obsolete immediately after being used by these subroutines, and therefore does not need to be stored. For example, if a user moves their mouse to a file to highlight the file, and then moves their mouse away from the file to cancel highlighting the file, the mouse movement is used to perform a subroutine that highlights or cancels highlighting the file, but the mouse movement does not need to be stored for further processing. Other examples of transient information can include copy and paste commands, cut and paste commands, selection commands, scroll commands, and other similar input commands that do not need to be stored for further processing after the corresponding subroutine is completed.

[0075] The shared computing component 122 can transmit state information across devices. The state information can be information that is synchronized immediately and stored for a short period of time. The state information indicates information that describes the state of the entire system (e.g., system 100) and maintains consistency. As the system is used over time, the state information can be updated and / or overwritten. For example, the state information can include the ID of the computing device being followed, the selected color, the brush size, the active / disabled components, etc. For example, if the computing device is not the main computing device (i.e., not the device currently being watched or followed), and its display brightness is adjusted to the highest, the shared computing component 122 can dim the display brightness of the computing device and assign another task to the computing device. As another example, the shared computing component 122 can store the state of a specific document or application in a memory. The shared computing component 122 monitors which application is currently selected (e.g., followed or watched). The user can select the first application on the first computing device, thereby changing the state of the first application to the first ("focused") state. If the user were to gaze at the second application on the second computing device, the shared computing component 122 would have stored in memory that the first application on the first computing device is currently in the first (e.g., “focused”) state. Thus, the shared computing component 122, through interaction with the gaze tracker component 120, can change the first application to the second (e.g., “not focused”) state and the second application to the first (e.g., “focused”) state.

[0076] Shared computing component 122 can transmit key-value pair information across devices. Key-value information can be synchronized and stored for a long time. In addition, key-value pair information can indicate information that can be searched or found by a search key or a search string or key. Examples of information that can be synchronized using key-value pairs can be text, numerical values, images, messages, files, etc. The relevant key-value pair information can be retrieved from the first computing device and used by the second computing device and / or peripheral devices. In some examples, the size of the key-value pair information (e.g., the size of the data in the memory) may be relatively large, and may not need to be synchronized immediately, but is stored until requested for use. For example, a user can provide an image input to a first computing device. The image input can be stored in a memory and accessed via shared computing component 122. The user can select or retrieve an image via his gaze and transfer the image to a second computing device.

[0077] Although examples of assigning computing tasks to specific devices based on computing load are provided, it should be understood that various additional or alternative techniques can be used. For example, a set of rules, heuristics and / or machine learning can be used. In some instances, a task can have an associated default action that can be modified or removed based on interaction data. Similarly, a new task can be generated, for example as a result of identifying a set of gaze tracking data associated with a user request to perform one or more actions by a computing device.

[0078] As described above, the software application may utilize a framework (e.g., the framework may be provided by an operating system of the computing device 103) to associate software application functionality with determined gaze tracking data from a user. For example, the software application may register functions or other functionality of the application with the framework such that when an action associated with the gaze tracking data is determined to be performed, the functionality of the registered software application is invoked (e.g., as a result of determining that the software application is the active application of the computing device or the software application is the intended target of the gaze tracking data).

[0079] Peripheral device 106 is shown to include audio / video output 124, sensor array 126, input controller 128 and signal identification component 130. In an example, audio / video output 124 includes one or more speakers and / or displays. Therefore, data can be received from computing device 103 and used to provide auditory and / or visual output to the user via audio / video output 124. Sensor array 126 includes one or more sensors as described herein, and can generate gaze tracking data (e.g., can be processed by a signal identification component (such as signal identification component 118 and / or signal identification component 130)). In an example, signal identification component 130 processes at least a portion of the gaze tracking data from sensor array 126 (and / or software executed on peripheral device 106) to generate a group of user signals so that the gaze tracking data itself does not need to leave peripheral device 106. In various aspects, the group of user signals may include gaze tracking data for multiple users. Alternatively, each individual user in a multi-user environment can be associated with a group of user signals. Such aspects may improve user privacy and reduce bandwidth utilization by reducing or eliminating the amount of gaze tracking data transmitted by the peripheral device 106. As another example, such processing may be performed by a computing device (e.g., computing device 103) to reduce resource utilization of the peripheral device 106. The input controller 128 may provide the interaction data and / or an indication of the generated set of user signals to a computing device (such as computing device 103).

[0080] In some instances, peripheral device 106 may include a shared computing component, aspects of which are discussed above with respect to shared computing component 122 of computing device 103. Thus, it should be understood that user signals may be used to determine actions associated with user gaze at any of a variety of devices. In addition, such processing need not be limited to data generated or otherwise obtained by a computing device performing processing. For example, a shared computing component of peripheral device 106 may use gaze tracking data from one or more of sensor array 126 and / or device 103, device 104, and / or device 106, as well as computational load data generated by peripheral device 106 or another device, to assign tasks to devices 104 to 108. Aspects of computing device 104 and peripheral device 108 are similar to computing device 103 and peripheral device 106, respectively, and therefore do not need to be re-described in detail below.

[0081] Typically, the shared computing component 122 can distribute tasks across computing devices to balance loads across computing devices (e.g., processor usage, memory usage, storage device usage, network usage, and / or general resource requirements, etc.). Additionally or alternatively, the shared computing component 122 can distribute tasks across computing devices to reduce the load on the main computing device (e.g., increase processing resources, memory resources, etc.). In this regard, the shared computing component 122 can reduce the allocation of non-essential tasks to the main computing device and offload non-essential tasks to other computing devices. Specifically, the shared computing component 122 can receive gaze tracking data from, for example, a gaze tracker component 120 and / or a gaze tracking data store 116. The shared computing component 122 can determine which computing device among multiple computing devices is the main computing device based on the gaze tracking data (e.g., by identifying which computing device among multiple computing devices is being viewed by one or more users). In addition, the gaze tracking data can be used to determine which tasks are necessary (e.g., applications that are the focus of one or more users' gaze).

[0082] Similarly, the shared computing component 122 can distribute tasks across computing devices to increase the load (e.g., reduce processing resources, memory resources, etc.) on a secondary computing device (e.g., a computing device that is found not to be the focus of the user's gaze). In this regard, the shared computing component 122 can increase the task allocation to the secondary computing device, thereby offloading tasks from the primary computing device (e.g., a computing device that is found to be the focus of the user's gaze). Specifically, the shared computing component 122 can receive gaze tracking data from, for example, the gaze tracker component 120 and / or the gaze tracking data store 116. The shared computing component 122 can determine which computing device among the multiple computing devices is the secondary computing device based on the gaze tracking data (e.g., by identifying which computing device among the multiple computing devices is not being looked at by one or more users). In addition, the gaze tracking data can be used to determine which tasks are not necessary (e.g., applications that are not the focus of one or more users' gaze).

[0083] Additionally or alternatively, the shared computing component 122 can distribute tasks across computing devices based on the type of computing device. For example, the shared computing component 122 can reallocate tasks from low computing devices (e.g., smart phones or smart watches) to high computing devices (e.g., GPU-enabled laptops or desktop computers). In this regard, the shared computing component 122 can reference the processing power of each computing device in the plurality of computing devices as a factor in determining which computing device the task will be assigned to.

[0084] Figure 2 Shows Figure 1 A detailed schematic diagram of the gaze tracker component 120 of FIG. Figure 2As shown, the gaze tracker component 120 includes a distributed artificial intelligence (AI) execution component or engine 120a, an application distribution component or engine 120b, and a multi-device interaction component or engine 120c.

[0085] In general, distributed AI refers to a way to solve complex learning, planning, and decision-making problems by distributing problems to autonomous processing nodes (e.g., functional agents) and / or by using machine learning models. Distributed AI execution component 120a can communicate with shared computing component 122 to coordinate the execution of tasks across heterogeneous devices (e.g., device 103 to device 108). As used herein, the term "heterogeneous device" refers to devices that may have different brands, models, operating systems, or locations. Heterogeneous devices can interact to share each other's computing resources. The examples disclosed herein can use machine learning algorithms to determine how applications are executed across heterogeneous devices. Such a determination can be made within AI execution component 120a.

[0086] The application distribution component 120b can handle the migration of one or more applications between multiple computing devices. For example, a first computing device may be running a word processing application. A user may wish to migrate the word processing application from the first computing device to a second computing device. Using the gaze input and corresponding commands disclosed herein, the system can migrate the word processing application from the primary computing device to the secondary computing device based on current or recent gaze data. Such operations can be performed by the application distribution component 120b or performed within the application distribution component 120b.

[0087] Additionally or alternatively, the application distribution component 120b can handle the distribution of one or more applications across multiple computing devices. For example, a user may desire to display a demonstration across multiple devices. Such operations can be performed by the application distribution component 120b. Specifically, the application distribution component can receive gaze input data indicating that multiple users are looking at multiple computing devices. Therefore, the application distribution component 120b can display the application desired to be viewed by the user across each of the multiple computing devices that the multiple users are looking at.

[0088] Typically, the multi-device interaction component facilitates communication between multiple computing devices. The multi-device interaction component 120c enables interaction between computing devices (e.g., devices 103 to 108). For example, if multiple users desire that their gazes adapt the behavior of multiple computing devices based on the determined actions corresponding to the gazes, the multiple computing devices can communicate gaze data corresponding to the user gazes to each other through the multi-device interaction component 120c. Additionally or alternatively, the multiple computing devices can communicate actions based on the gaze data to each other through the multi-device interaction component 120c, so that the actions can be performed by one or more computing devices in the multiple computing devices.

[0089] Figure 3 Shows Figure 1 A detailed schematic diagram of the shared computing component 122 of FIG. Figure 3 As shown, the shared computing components 122 include a context server component or engine 122a, a load balancer component or engine 122b, and a shared memory component or engine 122c.

[0090] Typically, the context server component manages resource pooling. The context server component 122a can receive information corresponding to resources available across multiple computing devices. For example, the context server component 122a can receive information corresponding to how much processor capacity is available across multiple computing devices. Additionally or alternatively, the context server component 122a can receive information corresponding to how much memory is available across multiple computing devices. The context server component 122a can also access or declare resources across multiple computing devices (e.g., processor capacity, memory capacity, and / or dedicated hardware, such as a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), a holographic processing unit (HPU), an infrared camera, a LIDAR sensor, other types of dedicated hardware (xPU), etc.). For example, the context server component 122a can include a set of access management policies that allow read and / or write access to processors or memories across multiple computing devices.

[0091] Typically, a load balancer component handles workload distribution. Load balancer component 122b can handle workload distribution across multiple computing devices (e.g., device 103 to device 108). Load balancer component 122b can facilitate balancing computing resources (e.g., processors or memory) based on factors such as time, space, user, and / or device. For example, if it is known that a user uses a smartphone during the morning and a laptop during the afternoon, load balancer component 122b can offload tasks from the smartphone to the laptop in the morning. Similarly, load balancer component 122b can offload tasks from the laptop to the smartphone in the afternoon.

[0092] As another example, if a laptop computer contains more resource capacity (e.g., processor capacity and / or memory capacity) than another form of computing device, the load balancer component 122b can offload tasks from a smartphone to the laptop computer. As another example, the mechanisms described herein can identify a user who is known to use a first computing device (e.g., a gaming console) more frequently than a second computing device (e.g., a desktop computer). Thus, the mechanisms described herein can identify the user, and the load balancer component can receive instructions to offload tasks from the first computing device to the second computing device, enabling the first computing device to operate at a relatively higher performance (e.g., complete operations faster, display high-quality graphics, process applications relatively quickly, etc.).

[0093] The load balancer component 122b can facilitate computing resources based on the gaze data received by the system 100 (e.g., from one or more users). Specifically, the load balancer component 122b can move non-essential tasks from the device that is the focus of the one or more users' gaze so as to make resources available for the application that the user is looking at. Alternatively, the load balancer component 122b can move the desired tasks to the device that is the focus of the one or more users' gaze so as to meet the one or more users' desire to view.

[0094] In general, the shared memory component handles the distribution of various memory products (e.g., information data). As previously discussed herein, the shared computing component 122 can balance the computing load, and can also transfer information (e.g., transient, state, and key-value information) across devices (e.g., devices 103, 104, 106, and 108) to facilitate the processing and / or execution of various tasks. The shared memory component 122c is a component that allows the shared computing component 122 to transfer information across devices. Therefore, the earlier description of transient, state, and key-value information can be applied in a similar manner to the operation performed by the shared memory component 122c. That is, the shared memory component 122c can store shared information across several linked devices. The shared memory component 122c allows the cross-linked devices to execute applications and tasks, thereby allowing the application executed on the first device to be migrated and executed on another device with the same state as when executed on the first device. In an example, the linked device can copy the shared memory 122c so that each linked device contains a similar copy of the shared memory 122c.

[0095] Figure 4 4 shows an overview of an example system 400 for multi-user, multi-device gaze tracking according to various aspects described herein. The system 400 includes a plurality of users 402 and a plurality of computing devices 404. Figure 4As shown, multiple computing devices 404 can identify one or more users. In some cases, user 402 is identified by multiple computing devices (e.g., several computing devices 404). Computing device 404 can be the above-mentioned Figure 1 Any type of computing device described (e.g., laptop, tablet, smartphone, etc.). Furthermore, computing devices may be heterogeneous devices that vary in model, brand, operating system, capacity, resources, and / or location.

[0096] The computing device 404 may include the Figure 2 The AI ​​execution 406 may include a cross-architecture AI platform 408 and an execution manager 410. The AI ​​execution 406 may be an accelerator for machine learning models with multi-platform support. The AI ​​execution 406 may be integrated with hardware-specific libraries across various computing devices.

[0097] Execution manager 410 loads and / or exports machine learning models (e.g., models generated or used by AI execution 406). Execution manager can load and / or export machine learning models. Execution manager 410 can apply optimizations, select hardware acceleration frameworks, manage training of models, and / or manage interfaces between computing devices.

[0098] The computing device 404 may also include a shared computing 412 component or engine. The shared computing 412 component may be similar to the one described above with respect to Figure 1 and Figure 3 The shared computing component 122 discussed above is similar. The shared computing 412 may include a context server 414 (eg, similar to the context server component 122a), a load balancer 416 (eg, similar to the load balancer component 122b), and a shared memory 418 (eg, similar to the shared memory component 122c).

[0099] Figure 5 An overview of an example system 500 for multi-device gaze tracking according to various aspects described herein is shown. The system 500 includes a user 502 and multiple computing devices 504. The multiple computing devices 504 each include a sensor 506 (e.g., a camera) that is configured to receive gaze input data from the user (e.g., by monitoring the eyes of the user 502). Typically, the camera 506 on the computing device 504 tracks where the user is looking (e.g., gaze data). The gaze data can be received by the computing device 504 to determine and / or perform specific actions corresponding to the gaze data based on the user 502.

[0100] Figure 6An overview of an example method 600 for processing gaze input data to perform an action that affects the behavior of a computing device is shown. According to some examples, aspects of the method 600 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0101] Method 600 begins at operation 602, where one or more computing devices are identified. For example, a user may use the Figure 1 The method 600 may be used to link one or more devices (e.g., device 103 to device 108) using any communication method discussed. The device may be identified by a previous link association (e.g., indicated in a device profile or a shared profile). Alternatively, one or more devices may be identified based on user login information for different devices (e.g., each device with the same user login may be linked). In another aspect, one or more devices may be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity may be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical properties around the device (e.g., based on device camera feed if the user has given the device permission to use the camera for this purpose). In another example, the user may then manually select one or more devices linked together to be identified by method 600 to identify the device at operation 602. Additionally or alternatively, the network may be configured to automatically identify one or more devices connected to the network. In another example, the network may be configured to detect computing devices within a specified geographic proximity.

[0102] At operation 604, one or more users are identified. The one or more users can be identified by one or more computing devices (e.g., devices 103 to 106). Specifically, the one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify the one or more users. The visual data can be processed using the mechanisms described herein to perform facial identification on the one or more users if the one or more users have provided permission to do so. For example, the one or more computing devices can create a grid on the face of each of the one or more users to identify facial features, such as nose location, mouth location, cheek location, hair location, eye location, and / or eyelid location.

[0103] Additionally or alternatively, at operation 604, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged in, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged in, the user is thereby identified. Additionally or alternatively, at operation 604, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0104] Additionally or alternatively, at operation 604, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements (e.g., eyes, face, body, hands, etc.) of one or more users may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0105] At operation 606, gaze input data is received from one or more computing devices (e.g., computing devices 103 to 108) corresponding to one or more users. Once one or more users are identified at 604, method 600 can monitor the orientation of the users' eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can be provided to a multi-device gaze tracking system (e.g., as described above with respect to Figures 1 to 5 The systems 100 , 400 , and 500 ) discussed provide an indication of where a user may be looking relative to a display screen (eg, the display screens of devices 103 - 108 ).

[0106] Still referring to operation 606, one or more computing devices (e.g., computing devices 103 to 108) can receive gaze data from multiple users (e.g., the computing devices can track the orientation of the eyes of multiple users and receive gaze data therefrom). Specifically, one or more computing devices can track the device (e.g., computing devices 103 to 108) that each user is looking at, and even further can determine what each user is looking at on the device (e.g., an application being displayed on one or more computing devices, or some other element). The gaze data can be received in real time (e.g., providing a continuous feedback stream about what multiple users are looking at). Alternatively, the gaze data can be received periodically (e.g., at regular or irregular time intervals that can be specified by the user).

[0107] Further, referring to operation 606, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data first began to be received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0108] At determination 608, it is determined whether there is a gaze command associated with the gaze input data. For example, determination 608 may include evaluating the received gaze input data to generate a set of user signals that may be processed in light of an environmental context (e.g., an application currently running on the device or a task currently being performed). Thus, the evaluation may identify a gaze command as a result of an association between the gaze input data and the environmental context.

[0109] In some examples, at determination 608, it is determined for each user whether there is a gaze command associated with the gaze input data corresponding to the user. For example, determination 608 may include evaluating the received gaze input data to generate a set of user signals, wherein each user signal corresponds to one of the multiple users. The user signal may be processed in view of the environmental context (e.g., an application currently running on the device or a task currently being executed). Therefore, the evaluation may identify one or more gaze commands as a result of the association between the gaze input data for each user and the environmental context. It should be appreciated that different gaze commands may be identified for each user based on different gaze input data (e.g., different users viewing different computing devices). Alternatively, the same gaze command may be identified for each user based on the same gaze input data (e.g., different users viewing the same computing device).

[0110] If it is determined that there is no gaze command associated with the gaze input data, the flow branches "no" to operation 610, where a default action is performed. For example, the gaze input data may have an associated preconfigured action. In some other examples, the method 600 may include determining whether the gaze input data has an associated default action, so that in some instances, as a result of the received gaze input data, the action may not be performed. The method 600 may terminate at operation 610. Alternatively, the method 600 may return from operation 610 to operation 602 to create a continuous feedback loop of gaze input data and execution commands for the user.

[0111] However, if it is determined that there is a gaze command associated with the received gaze input data, then the flow branches "yes" to operation 612, where an action is determined based on the gaze input data. For example, various actions that can be performed as a result of gaze input data are discussed below through some aspects disclosed herein.

[0112] The process proceeds to operation 614, where the behavior of the computing device is adapted according to the action determined at operation 612. For example, the action can be performed by the computing device executing method 600. In another example, an indication of the action can be provided to another computing device. For example, various aspects of method 600 can be performed by a peripheral device, so that operation 614 includes providing input to an associated computing device. As another example, operation 614 can include using an application programming interface (API) call to influence the behavior of the computing device accordingly based on the determined action. Method 600 can terminate at operation 614. Alternatively, method 600 can return from operation 614 to operation 602 to create a continuous feedback loop for the user to look at input data and execute commands.

[0113] Although method 600 is described as an example in which association is used to identify and perform actions based on gaze tracking data, it will be understood that any of a variety of additional or alternative techniques (e.g., reinforcement learning, rule sets, user interface commands) may be used to determine actions based on received gaze input data.

[0114] Figure 7 An overview of an example method 700 for processing gaze input data and load data to distribute tasks across computing devices is shown. According to some examples, aspects of the method 700 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0115] Method 700 begins at operation 702, where a plurality of computing devices are identified. For example, a user may use the above description of Figure 1 The method 700 may be used to link multiple devices (e.g., device 103 to device 108) by any communication method discussed. The device may be identified by a previous link association (e.g., indicated in a device profile or a shared profile). Alternatively, one or more devices may be identified based on user login information for different devices (e.g., each device with the same user login may be linked). In another aspect, one or more devices may be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity may be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical properties around the device (e.g., based on the device camera feed if the user has given the device permission to use the camera for this purpose). In another example, the user may then manually select multiple computing devices linked together to be identified by method 700 to identify the device at operation 702. Additionally or alternatively, the network may be configured to automatically identify multiple computing devices connected to the network. In another example, the network may be configured to detect computing devices within a specified geographic proximity.

[0116] At operation 704, one or more users are identified. The one or more users can be identified by one or more computing devices (e.g., devices 103 to 106). Specifically, the one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify the one or more users. The visual data can be processed using the mechanisms described herein to perform facial identification on the one or more users if the one or more users have provided permission to do so. For example, the one or more computing devices can create a grid on the face of each of the one or more users to identify facial features, such as nose location, mouth location, cheek location, hair location, eye location, and / or eyelid location.

[0117] Additionally or alternatively, at operation 704, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged in, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged in, the user is thereby identified. Additionally or alternatively, at operation 604, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0118] Additionally or alternatively, at operation 704, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0119] At operation 706, load data and gaze input data are received from each of the plurality of computing devices corresponding to the one or more users (e.g., computing devices 103 to 108). Once the one or more users are identified at 704, method 700 can monitor the orientation of the user's eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can be provided to a multi-device gaze tracking system (e.g., as described above with respect to Figures 1 to 5The systems 100, 400, and 500 discussed provide an indication of where a user may be looking relative to a display screen (e.g., the display screens of devices 103 to 108). For example, gaze data for a user may be tracked by multiple nearby devices. The gaze data tracked by each device may be stored in a shared memory (e.g., shared memory 122c) so that user gaze data associated with different devices may be shared.

[0120] The load data received from each computing device of the plurality of computing devices may indicate computing resources (e.g., processor availability and / or memory availability) available on each computing device. The load data may be generated by a shared computing component (e.g., Figure 1 The load data may be received by the shared computing component 122 discussed above. The load data may be received in real time (e.g., providing a continuous stream of feedback about the computing resources available across the computing devices). Alternatively, the load data may be received periodically (e.g., at regular or irregular time intervals that may be specified by a user).

[0121] The load data can be stored (e.g., in a memory). In some examples, only the most recent load data is stored, such that as load data is received, older load data is overwritten (e.g., in a memory) by new gaze data. Alternatively, in some examples, load data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data first began to be received). Typically, such implementations allow for a history of load data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future). For example, the history of load data may provide an indication of which computing devices are often overloaded, times when certain computing devices tend to be overloaded, or other indications that can be discerned from analysis of the stored load data.

[0122] Still referring to operation 706, one or more computing devices (e.g., computing devices 103 to 108) may receive gaze data from multiple users (e.g., the computing devices may track the orientation of the eyes of multiple users and receive gaze data therefrom). Specifically, one or more computing devices may track the device (e.g., computing devices 103 to 108) that each user is looking at, and may even further determine what each user is looking at on the device (e.g., an application being displayed on one or more computing devices, or some other element). The gaze data may be received in real time (e.g., providing a continuous stream of feedback about what multiple users are looking at). Alternatively, the gaze data may be received periodically (e.g., at regular or irregular time intervals that may be specified by the user).

[0123] Further, referring to operation 706, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten (e.g., in memory) by the new gaze data. Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data first began to be received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0124] At operation 708, the load data is processed to determine resource availability for one or more linked computing devices. For example, the load data may be generated by a shared computing component (e.g., Figure 1 The shared computing component 122 discussed above) is received to determine the efficiency of each computing device. Computing devices typically have processors and memories that can be relied upon to perform actions on a computer. The resource availability of one or more computing devices can be determined by receiving processor usage data and / or memory usage data from one or more computing devices, and receiving processor capability data and / or memory capability data from each computing device. The ratio between the processor usage data and the processor capability data can be calculated (e.g., by dividing the former by the latter) to determine how much processor space is available on the computing device when no actions are performed relative to the received load data. Similarly, the ratio between the memory availability data and the memory capability data can be calculated (e.g., by dividing the former by the latter) to determine how much memory space is available on the computing device when no actions are performed relative to the received load data. The ratio can be compared with a predetermined threshold, as will be discussed further below, to determine whether one or more tasks need to be reallocated across computing devices based on the determined efficiency of each computing device.

[0125] At operation 710, the gaze data is processed to determine which computing device of the one or more computing devices is a focus device. Among the plurality of computing devices, the focus device may be the device that most users among the plurality of users are found to be looking at (or the device currently being viewed in a single user environment). Alternatively, in some examples, among the plurality of users, when special importance is assigned to a particular user, the focus device is the device that the particular user is looking at. For example, if a presentation is being given, the focus device may be the device that the presenter is looking at, as opposed to the devices that any presentation observers may be looking at.

[0126] At determination 710, it is determined whether the focus device is greater than an efficiency threshold. For example, determination 710 may include evaluating the determined efficiency of operation 708 relative to a predetermined threshold that is automatically calculated (e.g., based on device specifications) or set by a user. The efficiency threshold may be a threshold at which computing performance decreases based on computing resource overload on a particular device. In an example, the efficiency threshold may be dynamic. That is, the efficiency threshold may be higher for resource-intensive tasks (e.g., video games, high-resolution video, etc.), or may be lower for tasks that are not resource-intensive (e.g., email applications, word processing applications, etc.). That is, the efficiency threshold may be dynamically determined based on resource requirements for a particular application or task in focus.

[0127] If it is determined that the focus device is not greater than the efficiency threshold, the flow branches "no" to operation 714, where the task is assigned to the focus device. For example, if it is determined that the focus device is not overloaded with tasks, the focus device can be assigned tasks from another computing device that is overloaded. Method 700 can terminate at operation 714. Alternatively, method 700 can return to operation 706, where additional load data and gaze data are received from each of the multiple computing devices. In some examples, when the flow branches "no", operation 714 can be skipped and method 700 can flow directly from operation 712 to operation 706.

[0128] However, if it is determined that the focus device is greater than the efficiency threshold, then the flow branches "yes" to operation 716. At operation 716, one or more tasks from the focus device are assigned to different devices in the plurality of devices. In general, the shared computing component can dispatch tasks across multiple computing devices based on available load capacity. If the focus device is found to be greater than the efficiency threshold (e.g., based on the above-mentioned processor ratio or memory ratio, focus application or task resource requirements, or general resource availability), the shared computing component can reallocate tasks from the focus device to different devices (e.g., devices less than the efficiency threshold).

[0129] For example, returning a reference Figure 4, one of the devices 404 is a tablet computer. Relative to the other devices 404, the tablet computer 404 "Device-2" is tracking gazes from the majority of users (four users). Therefore, the tablet computer 404 may be the focus device. If the tablet computer 404 is rendering a high-definition video for viewing by the user 402, and also has multiple background processes running (e.g., refreshing email, monitoring changes to a website, etc.), the background processes may be reallocated to another device by the shared computing 412. As an example, the background processes may be assigned to "Device-1", which may be a laptop computer, because "Device-1" is only tracking two users, and the laptop computer may have greater processor and memory capabilities than the tablet computer.

[0130] Fig. 8A and Figure 8B An example system 800 for multi-device gaze tracking according to various aspects described herein is shown. The system 800 includes a user 802 and multiple computing devices, such as a first computing device 804 and a second computing device 806. The multiple computing devices 804, 806 communicate via a network 808. The multiple computing devices 804, 806 can be connected to the network 808 described previously in this document. Figure 1 Devices 103 through 108 are similar to those discussed above. In addition, network 808 may be similar to that discussed previously herein. Figure 1 The network 110 discussed is similar.

[0131] The first computing device 804 may include multiple applications running thereon. For example, the first computing device 804 may include a word processing application 810 and an email application 812 running thereon. The second computing device 806 may include multiple applications running thereon. For example, the second computing device 806 may include a mobile email application 814 running thereon. It should be appreciated that application 810, application 812, and application 814 may be any one of multiple applications or software programs that provide some visual displays that a user can watch. Examples of such applications, elements, or computer programs include, but are not limited to, word processors, graphics software, database software, spreadsheet software, web browsers, enterprise software, information worker software, multimedia software, presentation software, educational software, content access software, communication software, etc.

[0132] The mechanisms described herein provide a user with an opportunity to select, focus on, or navigate between applications on a computing device based on where the user is looking. Fig. 8A , user 802 is shown viewing word processing application 810. Thus, according to the mechanisms described herein, word processing application 810 is selected or focused. In contrast, and with specific reference to Figure 8B, user 802 is shown viewing mobile email application 814. Thus, according to the mechanisms described herein, mobile email application 814 is selected or focused. When a user (e.g., user 802) switches their gaze from a first application to a second application, the first application may be deselected or unfocused so that the second application is selected or focused. Similarly, when a user (e.g., user 802) switches their gaze from a second application to a first application, the second application may be deselected or unfocused so that the first application is selected or focused.

[0133] Fig. 9 An overview of an example method 900 for processing gaze input data to perform an action that affects the behavior of a computing device is shown. According to some examples, aspects of the method 900 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0134] Method 900 begins at operation 902, where one or more computing devices are identified. For example, a user may use the above description of Figure 1 Any communication method discussed is used to link one or more devices (e.g., device 103 to device 108). The device can be identified by a previous link association (e.g., indicated in a device profile or a sharing profile). Alternatively, one or more devices can be identified based on user login information for different devices (e.g., each device with the same user login can be linked). In another aspect, one or more devices can be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity can be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical characteristics around the device (e.g., if the user has given the device permission to use the camera for this purpose, based on the device camera feed). In another example, the user can then manually select one or more devices linked together to be identified by method 900. Additionally or alternatively, the network (e.g., network 808) can be configured to automatically identify one or more devices connected to the network. In another example, the network (e.g., network 808) can be configured to detect computing devices within a specified geographic proximity.

[0135] At operation 904, one or more users are identified. The one or more users can be identified by one or more computing devices (e.g., devices 103 to 106). Specifically, the one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify one or more users (e.g., user 802). The visual data can be processed using the mechanisms described herein to perform facial identification on the one or more users if the one or more users have provided permission to do so. For example, the one or more computing devices can create a grid on the face of each of the one or more users to identify facial reminders, such as nose locations, mouth locations, cheek locations, hair locations, eye locations, and / or eyelid locations.

[0136] Additionally or alternatively, at operation 904, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged in, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged in, the user is thereby identified. Additionally or alternatively, at operation 904, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0137] Additionally or alternatively, at operation 904, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0138] At operation 906, gaze input data is received from one or more computing devices (e.g., computing devices 103 to 108, and / or computing devices 804, 806) corresponding to one or more users (e.g., user 802). Once one or more users are identified at 904, method 900 can monitor the orientation of the user's eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can be provided to a multi-device gaze tracking system (e.g., as described above with respect to Fig. 8A and Figure 8B The system 800 discussed provides an indication of where a user may be looking relative to a display screen (eg, a display screen of device 804 and / or device 608).

[0139] Still referring to operation 906, one or more computing devices (e.g., computing devices 103 to 108 and / or computing devices 804, computing devices 806) can receive gaze data from multiple users (e.g., the computing devices can track the orientation of the eyes of multiple users and receive gaze data therefrom). Specifically, one or more computing devices can track the device each user is looking at (e.g., computing devices 103 to 108 and / or computing devices 804, computing devices 806), and even further can determine what each user is looking at on the device (e.g., an application being displayed on one or more computing devices, or some other element). The gaze data can be received in real time (e.g., providing a continuous feedback stream about what multiple users are looking at). Alternatively, the gaze data can be received periodically (e.g., at regular or irregular time intervals that can be specified by the user).

[0140] Further, referring to operation 906, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data was first received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0141] At determination 908, it is determined whether there is an application associated with the gaze input data. For example, determination 908 may include evaluating the received gaze input data to generate a set of user signals that may be processed in light of an environmental context (e.g., an application currently running on the device or a task currently being performed). Thus, the evaluation may identify an application as a result of an association between the gaze input data and the environmental context.

[0142] In some examples, at determination 908, it is determined for each user whether there is an application associated with the gaze input data corresponding to the user. For example, determination 908 may include evaluating the received gaze input data to generate a set of user signals, wherein each user signal corresponds to one of the multiple users. The user signal can be processed in view of the environmental context (e.g., an application currently running on the device or a task currently being executed). Therefore, the evaluation can identify one or more applications as a result of the association between the gaze input data for each user and the environmental context. It should be appreciated that different applications can be identified for each user based on different gaze input data (e.g., different users viewing different computing devices). Alternatively, the same application can be identified for each user based on the same gaze input data (e.g., different users viewing the same computing device).

[0143] If it is determined that there is no application associated with the gaze input data, the flow branches "NO" to operation 910, where a default action is performed. For example, the gaze input data may have an associated predetermined application. In some other examples, the method 900 may include determining whether the gaze input data has an associated default application, so that in some instances, as a result of the received gaze input data, no action may be performed. The method 900 may terminate at operation 910. Alternatively, the method 900 may return from operation 910 to operation 902 to create a continuous feedback loop for the user to gaze input data and select or focus on an application.

[0144] However, if it is determined that there is a gaze command associated with the received gaze input data, then the flow branches "yes" to operation 912, where an application is determined based on the gaze input data. Fig. 8A and Figure 8B When the user 802 looks at the word processing application 810, it is determined that the user is looking at the word processing application 810. When the user 802 looks at the mobile email application 814, it is determined that the user is looking at the mobile email application 814.

[0145] The process proceeds to operation 914, where one or more computing devices are adapted to select the determined application. Alternatively, in some examples, one or more computing devices may be adapted to focus on the determined application. For example, the determined application may be selected or focused on by the computing device executing method 900. In another example, an indication of the determined application may be provided to another computing device. For example, various aspects of method 900 may be performed by a peripheral device so that operation 914 includes providing input to an associated computing device. As another example, operation 914 may include using an application programming interface (API) call to adapt one or more computing devices to select or focus on the determined application, and / or deselect or not focus on an application that is not determined based on gaze input data. Method 900 may terminate at operation 914. Alternatively, method 900 may return to operation 902 from operation 914 to create a continuous feedback loop for the user to gaze input data and select or focus on the application.

[0146] Fig. 10A and Fig. 10B An example system 1000 for multi-device gaze tracking according to various aspects described herein is shown. The system 1000 includes a user 1002 and multiple computing devices, such as a first computing device 1004 and a second computing device 1006. The multiple computing devices 1004, 1006 communicate via a network 1008. The multiple computing devices 1004, 1006 can be connected to the network 1008 described previously in this document. Figure 1 Devices 103 through 108 are similar to those discussed above. Additionally, network 1008 may be similar to that discussed previously herein. Figure 1 The network 110 discussed is similar.

[0147] The first computing device 1004 may include one or more applications running thereon. For example, the first computing device 1004 may include a word processing application 1010. The second computing device 1006 may include multiple applications running thereon. For example, the second computing device 1006 may include an email application 1012 running thereon. It should be appreciated that the applications 1010 and 1012 may be any of a plurality of applications or software programs that provide some visual displays that a user may gaze upon. Examples of such applications, elements, or computer programs include, but are not limited to, word processors, graphics software, database software, spreadsheet software, web browsers, enterprise software, information worker software, multimedia software, presentation software, educational software, content access software, communication software, and the like.

[0148] Various aspects described herein provide an opportunity to distribute tasks across computing devices and / or reduce tasks on computing devices based on where a user is looking. Fig. 10A, user 1002 is shown viewing word processing application 1010. Therefore, according to the mechanisms described herein, word processing application 1010 is selected or focused on on first computing device 1004. At the same time, non-essential tasks on second computing device 1006 may be reduced. For example, the brightness on second computing device 1006 is Fig. 10A By reducing the brightness on the second computing device 1006, the computing device 1006 can increase the load capacity to perform one or more tasks (e.g., tasks reallocated from the first computing device 1004). Additionally or alternatively, with the brightness being reduced, background processes on the second computing device 1006 can be deprioritized (e.g., slowed down, interrupted, or stopped) based on the gaze data to increase the capacity of resources to the prioritized tasks.

[0149] In comparison, and with reference to Figure 8B , user 802 is shown viewing mobile email application 814 on second computing device 1006. Thus, according to the mechanisms described herein, mobile email application 814 is selected or focused. At the same time, non-essential tasks on first computing device 1004 may be reduced. For example, the brightness on first computing device 1004 may be reduced. Fig. 10B By reducing the brightness on the first computing device 1004, the computing device 1004 can increase the load capacity to perform one or more tasks (e.g., tasks reallocated from the second computing device 1006). Additionally or alternatively, in order for the brightness to be reduced, background processes on the first computing device 1006 can be deprioritized (e.g., slowed down, interrupted, or stopped) based on the gaze data to increase the capacity of resources to the prioritized tasks.

[0150] When a user (e.g., user 1002) switches their gaze from the first computing device 1004 to the second computing device 1006, tasks on the first computing device 1004 may be de-prioritized and tasks on the second computing device 1006 may be prioritized. Thus, a shared computing component (e.g., shared computing component 122) may allocate or execute tasks based on priorities and available resources (e.g., available processor capacity and / or memory capacity across computing devices). Similarly, when a user (e.g., user 1002) switches their gaze from the second computing device 1006 to the first computing device 1004, tasks on the second computing device 1006 may be de-prioritized and tasks on the first computing device 1004 may be prioritized. Thus, a shared computing component (e.g., shared computing component 122) may allocate or execute tasks based on priorities and available resources (e.g., available processor capacity and / or memory capacity across computing devices).

[0151] Fig.11An overview of an example method 1100 for processing gaze input data and load data to distribute tasks across computing devices is shown. According to some examples, aspects of the method 1100 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0152] Method 1100 begins at operation 1102, where one or more computing devices are identified. For example, a user may use the above description of Figure 1 Any communication method discussed is used to link one or more devices (e.g., device 103 to device 108 and / or device 1004, device 1006). The device can be identified by a previous link association (e.g., indicated in a device profile or a shared profile). Alternatively, one or more devices can be identified based on user login information for different devices (e.g., each device with the same user login can be linked). In another aspect, one or more devices can be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity can be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical characteristics around the device (e.g., if the user has given the device permission to use the camera for this purpose, based on the device camera feed). In another example, the user can then manually select one or more devices linked together to be identified by method 1100 to identify the device at operation 1102. Additionally or alternatively, the network (e.g., network 1008) can be configured to automatically identify one or more devices connected to the network. In yet another example, a network (eg, network 1008 ) may be configured to detect computing devices within a specified geographic proximity.

[0153] At operation 1104, one or more users are identified. One or more users can be identified by one or more computing devices (e.g., devices 103 to 106 and / or devices 1004, 1006). Specifically, one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify one or more users (e.g., user 1002). The visual data can be processed using the mechanisms described herein to perform facial identification on one or more users if one or more users have provided permission to do so. For example, one or more computing devices can create a grid on the face of each of the one or more users to identify facial features, such as nose location, mouth location, cheek location, hair location, eye location, and / or eyelid location.

[0154] Additionally or alternatively, at operation 1104, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged in, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged in, the user is thereby identified. Additionally or alternatively, at operation 1104, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0155] Additionally or alternatively, at operation 1104, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0156] At operation 1106, load data and gaze data are received from each computing device (e.g., computing devices 103 to 108 and / or devices 1004, 1006) of a plurality of computing devices corresponding to one or more users. Once the one or more users are identified at 1104, method 1100 can monitor the orientation of the user's eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can be provided to a multi-device gaze tracking system (e.g., as described above with respect to Fig. 10A and 10BThe system 1000 discussed provides an indication of where a user may be looking relative to a display screen (eg, the display screens of devices 103 to 108 and / or devices 1004 , 1006 ).

[0157] The load data received from each of the plurality of computing devices may indicate computing resources available on each computing device (e.g., processor usage, memory usage, storage device usage, network usage, and / or general resource requirements, etc.). The load data may be generated by a shared computing component (e.g., Figure 1 The load data may be received by the shared computing component 122 discussed above. The load data may be received in real time (e.g., providing a continuous stream of feedback about the computing resources available across the computing devices). Alternatively, the load data may be received periodically (e.g., at regular or irregular time intervals that may be specified by a user).

[0158] The load data can be stored (e.g., in a memory). In some examples, only the most recent load data is stored, such that as load data is received, older load data is overwritten (e.g., in a memory) by new gaze data. Alternatively, in some examples, load data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data first began to be received). Typically, such implementations allow for a history of load data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future). For example, the history of load data may provide an indication of which computing devices are often overloaded, times when certain computing devices tend to be overloaded, or other indications that can be discerned from analysis of the stored load data.

[0159] Still referring to operation 1106, one or more computing devices (e.g., computing devices 103 to 108 and / or devices 1004, 1006) can receive gaze data from multiple users (e.g., the computing device can track the orientation of the eyes of multiple users and receive gaze data from them). Specifically, one or more computing devices can track the device that each user is looking at (e.g., computing devices 103 to 108 and / or devices 1004, 1006), and even further can determine what each user is looking at on the device (e.g., an application being displayed on one or more computing devices, or some other element). The gaze data can be received in real time (e.g., providing a continuous feedback stream about what multiple users are looking at). Alternatively, the gaze data can be received periodically (e.g., at regular or irregular time intervals that can be specified by the user).

[0160] Further, referring to operation 1106, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data first began to be received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0161] At operation 1108, the load data is processed to determine resource availability for one or more linked computing devices. For example, the load data may be generated by a shared computing component (e.g., Figure 1 The shared computing component 122 discussed above) is received to determine the efficiency of each computing device. Computing devices typically have processors and memories that can be relied upon to perform actions on a computer. The resource availability of each computing device can be determined by receiving processor usage data and / or memory usage data from each computing device, and receiving processor capability data and / or memory capability data from each computing device. A ratio between the processor usage data and the processor capability data can be calculated (e.g., by dividing the former by the latter) to determine how much processor space is available on the computing device when no actions are performed relative to the received load data. Similarly, a ratio between the memory usage data and the memory capability data can be calculated (e.g., by dividing the former by the latter) to determine how much memory space is available on the computing device when no actions are performed relative to the received load data. The ratio can be compared to a predetermined threshold, as will be discussed further below, to determine whether one or more tasks need to be reallocated across computing devices based on the determined efficiency of each computing device.

[0162] At operation 1110, the gaze data is processed to determine which computing device of the one or more computing devices is a focus device. Among the plurality of computing devices, the focus device may be the device that most users among the plurality of users are found to be looking at. Alternatively, in some examples, among the plurality of users, when special importance is assigned to a particular user, the focus device is the device that the particular user is looking at. For example, if a presentation is being given, the focus device may be the device that the presenter is looking at, as opposed to the devices that any presentation observers may be looking at.

[0163] At determination 1110, it is determined whether the focus device is above an efficiency threshold. For example, determination 1110 may include evaluating the determined efficiency of operation 1108 relative to an efficiency threshold that is automatically calculated (e.g., based on device specifications) or set by a user. The efficiency threshold may be a threshold at which computing performance decreases based on computing resource overload on a particular device. In an example, the efficiency threshold may be dynamic. That is, the efficiency threshold may be higher for resource-intensive tasks (e.g., video games, high-resolution video, etc.), or may be lower for tasks that are not resource-intensive (e.g., email applications, word processing applications, etc.). That is, the efficiency threshold may be dynamically determined based on resource requirements for a particular application or task in focus.

[0164] If it is determined that the focus device is not greater than the efficiency threshold, the process branches "no" to operation 1114, where the task is assigned to the focus device. For example, if it is determined that the focus device is not overloaded with tasks, the focus device can be assigned tasks from another computing device that is overloaded. Method 1100 can terminate at operation 1114. Alternatively, method 1100 can return to operation 1106, where additional load data and gaze data are received from each of the multiple computing devices. In some examples, when the process branches "no", operation 1114 can be skipped and method 1100 can flow directly from operation 1112 to operation 1106.

[0165] However, if it is determined that the focus device is greater than the efficiency threshold, then the flow branches "yes" to operation 1116. At operation 1116, one or more tasks from the focus device are assigned to different devices in the plurality of devices. In general, the shared computing component can allocate tasks across the plurality of computing devices based on available load capacity. If the focus device is found to be greater than the efficiency threshold (e.g., based on the processor ratio or memory ratio described above, the focus application or task resource requirements, or general resource availability), then the shared computing component can reallocate tasks from the focus device to different devices (e.g., devices less than the efficiency threshold).

[0166] Additionally, non-essential tasks may be reduced on different devices that were assigned tasks from the focus device at operation 1116. In general, it may be beneficial to prioritize tasks that are considered important based on user data (e.g., tasks running on an application that is currently being gazed at by the user, or tasks running on an application that is predicted to be gazed at by the user based on historical gaze data).

[0167] Operation 1116 can be illustrated with respect to system 1000. For example, brightness can be a non-essential task on a computing device that is not currently being viewed by a user (eg, user 1002). Fig. 10A, when the user 1002 is looking at the first computing device 1004, the gaze data can be received by the gaze tracker component (e.g., the gaze tracker component 120) to identify that the user is viewing an application (e.g., the application 1010) on the first computing device 1004. Therefore, the shared computing component 122 can receive the identification made by the gaze tracker component 120 to determine how to allocate tasks or execute programs based on the load capacity data and the gaze data. The second computing device 1006 can be determined to have one or more non-essential tasks (e.g., high brightness) running, which are reduced to make computing resources available for higher priority tasks (e.g., tasks running on the first computing device 1004).

[0168] Fig. 12A and Fig. 12B An example system 1200 for multi-device gaze tracking according to various aspects described herein is shown. The system 1200 includes a user 1202 and multiple computing devices, such as a first computing device 1204 and a second computing device 1206. The multiple computing devices 1204, 1206 communicate via a network 1208. The multiple computing devices 1204, 1206 can be connected to the network 1208 described previously in this document. Figure 1 Devices 103 to 108 are similar to those discussed above. In addition, network 1208 may be similar to that discussed previously in this document. Figure 1 The network 110 discussed is similar.

[0169] The system 1200 may also include a first posture 1210 (see Fig. 12A ) and the second posture 1212 (see Fig. 12B ). Gestures 1210, 1212 may be gestures made by user 1202 with their hands. For example, gestures 1210, 1212 may be waving gestures, pinching gestures, clenching fist gestures, opening gestures, blinking gestures, blinking gestures, gaze lingering, snapping gestures, clicking gestures, clapping gestures, or similar gestures that are preconfigured to serve functions in system 1200. In some examples, user input (such as voice input or switch input) may be combined with gestures (e.g., gestures 1210, 1212) or used independently to adapt the computing device to perform the desired action. Gestures 1210, 1212 may be detected by computing device 1204, 1206, for example, via a sensor (e.g., a camera, an RGB sensor, an infrared sensor, a LiDAR sensor, a motion sensor, or any other type of sensor capable of recognizing gestures made by a user).

[0170] The first computing device 1204 may include a plurality of applications running thereon. For example, the first computing device 1204 may include a word processing application 1214. Additionally or alternatively, the second computing device 1206 may include a plurality of applications (not shown) running thereon. It should be appreciated that the application 1212 may be any of a plurality of applications or software programs that provide some visual display that a user may look at. Examples of such applications, elements, or computer programs include, but are not limited to, word processors, graphics software, database software, spreadsheet software, web browsers, enterprise software, information worker software, multimedia software, presentation software, educational software, content access software, communication software, and the like.

[0171] The mechanisms described herein provide a user (e.g., user 1202) with the ability to transfer applications between computing devices by using a combination of their gaze and gestures. Computing device 1204, computing device 1206 may receive gaze data corresponding to user 1202, and gesture data corresponding to gesture 1210 and / or gesture 1212. Fig. 12A , user 1202 is shown viewing a word processing application 1212 on a first computing device 1204 while also performing a first gesture 1210 (e.g., a pinching gesture using a hand). According to the mechanisms described herein, word processing application 1210 is selected on first computing device 1204. Then, referring to Fig. 12B , user 1202 is shown looking at second computing device 1206 and making second gesture 1212 (e.g., using a lowering or releasing gesture of the hand). As a result, word processing application 1212 is transferred to second computing device 1206 and deselected by user 1202.

[0172] In some examples, an application (e.g., a word processing application 1212) can be completely transferred or completely migrated from a first computing device (e.g., first computing device 1204) to a second computing device (e.g., second computing device 1206). In other examples, some components of an application can be migrated from a first computing device to a second computing device. For example, using a word processing application running on a first computing device, typesetting preferences (e.g., bold, italics, fonts, colors, etc.) can be displayed on a second computing device. As another example, using a drawing application running on a first computing device (e.g., a laptop computer), a color palette can be migrated to a second computing device (e.g., a smart phone, smart watch, or tablet computer). In such an example, a user can use a second computing device to select a color (e.g., with their finger, stylus, or input device) and then paint on a canvas displayed on the first computing device using the selected color. This enables users to select paint and paint in a similar way to the real world (e.g., picking paint from a palette and painting on a canvas). Other examples mimic how users participate in activities.

[0173] Although Fig. 12A and Fig. 12B The example of is shown as including a single application being transferred across computing devices, but it is also contemplated that multiple applications can be transferred across computing devices (e.g., device 1204 and / or device 1206) using gaze data and gesture data. In addition, although Fig. 12A and Fig. 12B The example is shown as including only a single user (e.g., user 1202), but it is also contemplated that computing device 1204 and / or computing device 1206 can receive gaze data and gesture data from multiple users, as will be further discussed herein with respect to method 1300.

[0174] Fig.13 An overview of an example method 1300 for processing gaze input data to perform an action that affects the behavior of a computing device is shown. According to some examples, aspects of the method 1300 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0175] Method 1300 begins at operation 1302, where one or more computing devices are identified. For example, a user may use the above description of Figure 1Any communication method discussed is used to link one or more devices (e.g., device 103 to device 108 and / or device 1204, device 1206). The device can be identified by a previous link association (e.g., indicated in a device profile or a shared profile). Alternatively, one or more devices can be identified based on user login information for different devices (e.g., each device with the same user login can be linked). In another aspect, one or more devices can be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity can be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical characteristics around the device (e.g., if the user has given the device permission to use the camera for this purpose, based on the device camera feed). In another example, the user can then manually select one or more devices linked together to be identified by method 1300 to identify the device at operation 1302. Additionally or alternatively, the network (e.g., network 1208) can be configured to automatically identify one or more devices connected to the network. In yet another example, a network (eg, network 1208 ) may be configured to detect computing devices within a specified geographic proximity.

[0176] At operation 1304, one or more users are identified. One or more users can be identified by one or more computing devices (e.g., devices 103 to 106 and / or devices 1204, 1206). Specifically, one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify one or more users (e.g., user 1202). The visual data can be processed using the mechanisms described herein to perform facial identification on one or more users if one or more users have provided permission to do so. For example, one or more computing devices can create a grid on the face of each of the one or more users to identify facial features, such as nose location, mouth location, cheek location, hair location, eye location, and / or eyelid location.

[0177] Additionally or alternatively, at operation 1304, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged in, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged in, the user is thereby identified. Additionally or alternatively, at operation 1304, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0178] Additionally or alternatively, at operation 1304, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0179] At operation 1306, gaze input data and gesture input data are received from one or more computing devices (e.g., computing devices 103 to 108, and / or computing devices 1204, 1206) corresponding to one or more users (e.g., user 1202). Once one or more users are identified at 1304, method 1300 can monitor the orientation of the user's eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can be provided to a multi-device gaze tracking system (e.g., as described above with respect to Fig. 12A and Fig. 12BThe system 1200 discussed provides an indication of where a user may be looking relative to a display screen (e.g., a display screen of device 1204 and / or device 1206). In addition, once one or more users are identified at 1304, the method 1300 can monitor the user's hands, wrists, or other body parts to determine gestures and thereby receive gesture input data.

[0180] Still referring to operation 1306, one or more computing devices (e.g., computing devices 103 to 108 and / or computing devices 1204, computing device 1206) may receive gaze input data from multiple users (e.g., the computing devices may track the orientation of the eyes of multiple users and receive gaze data therefrom). Additionally, one or more computing devices (e.g., computing devices 103 to 108 and / or computing devices 1204, computing device 1206) may receive gesture input data from multiple users (e.g., the computing devices may track the orientation of the bodies of multiple users and receive gaze data therefrom). Specifically, one or more computing devices may track the device (e.g., computing devices 103 to 108 and / or computing devices 1204, computing device 1206) that each of the users is looking at, and may even further determine what each of the users is looking at on the device (e.g., an application being displayed on one or more computing devices, or some other element). In addition, one or more computing devices may monitor parts of the user's body (e.g., hands, wrists, arms) to determine when a particular gesture is performed by a user.

[0181] Gaze input data may be received in real time (e.g., providing a continuous stream of feedback about what multiple users are looking at). Alternatively, gaze input data may be received periodically (e.g., at regular or irregular time intervals that may be specified by the user). Additionally, gesture input data may be received in real time (e.g., providing a continuous stream of feedback about the configuration of a user's body part). Alternatively, gesture input data may be received periodically (e.g., at regular or irregular time intervals that may be specified by the user).

[0182] Further, referring to operation 1306, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data first began to be received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0183] Similarly, with reference to operation 1306, gesture data may be stored (e.g., in a memory). In some examples, only recent gesture data is stored, such that as gesture data is received, older gesture data is overwritten (e.g., in a memory) by new gesture data. Alternatively, in some examples, gesture data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data was first received). Typically, such implementations allow for a history of gesture data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0184] In some examples, a user may perform a series of gestures (e.g. Fig. 12A and Fig. 12B ). Thus, gesture data may be stored across a specified duration so that a series of gestures (eg, gestures 1210 to 1212) may be recognized by a system disclosed herein (eg, system 1200).

[0185] At determination 1308, it is determined whether there is an action associated with the gaze input data and the gesture input data. For example, determination 1308 may include evaluating the received gaze input data and gesture input data to generate a user signal group that may be processed in view of an environmental context (e.g., an application currently running on the device or a task currently being performed). Thus, the evaluation may identify an application or task as a result of an association between the gaze input data, the gesture input data, and the environmental context.

[0186] In some examples, at determination 1308, it is determined for each user whether there is an action associated with the gaze input data and gesture input data corresponding to the user. For example, determination 1308 may include evaluating the received gaze input data to generate one or more groups of user signals, wherein each user signal corresponds to one of the multiple users. The user signal can be processed in view of the environmental context (e.g., the application currently running on the device or the task currently being executed). Therefore, the evaluation can identify one or more actions as the result of the association between the gaze input data and gesture input data for each user and the environmental context. It should be recognized that different actions can be identified for each user based on different gaze input data (e.g., different users viewing different computing devices) and different gesture input data (e.g., different users using their hands to perform different configurations). Alternatively, the same action can be identified for each user based on the same gaze input data (e.g., different users looking at the same computing device) and the same gesture input data (e.g., different users using their hands to perform the same configuration).

[0187] If it is determined that there is no application associated with the gaze input data and the gesture input data, the process branches "no" to operation 1310, where a default action is performed. For example, the gaze input data and the gesture input data may have an associated predetermined action. In some other examples, the method 1300 may include determining whether the gaze input data and the gesture input data have an associated default action, so that in some instances, as a result of the received gaze input data and gesture input data, no action may be performed. The method 1300 may terminate at operation 1310. Alternatively, the method 1300 may return from operation 1310 to operation 1302 to create a continuous feedback loop of receiving gaze and gesture input data and executing commands based on the gaze and gesture input data.

[0188] However, if it is determined that there is a gaze command associated with the received gaze input data, then flow branches "yes" to operation 1312 where an action is determined based on the gaze and gesture input data. Fig. 12A and Fig. 12B , when user 1202 gazes at word processing application 1214 on first computing device 1204 and performs a first gesture (e.g., gesture 1210), application 1214 is selected. Then, when user 1202 moves their gaze to second computing device 1206 and performs a second gesture (e.g., gesture 1212), the application is transferred from first computing device 1204 to second computing device 1206. Fig. 12A and Fig. 12BThe sequence of gestures shown in can be referred to as a “pinch and drop” sequence, where the pinch gesture selects an application on a first computing device and the drop gesture releases the application on a second computing device.

[0189] The process proceeds to operation 1314, where one or more computing devices are adapted to perform the determined action. In some examples, one or more computing devices may be adapted to perform the action determined by the computing device executing method 1300. In another example, an indication of the determined action may be provided to another computing device. For example, various aspects of method 1300 may be performed by a peripheral device so that operation 1314 includes providing input to an associated computing device. As another example, operation 1314 may include using an application programming interface (API) call to perform the determined action (e.g., transferring an application from a first computing device to a second computing device). Method 1300 may terminate at operation 1314. Alternatively, method 1300 may return from operation 1314 to operation 1302 to create a continuous feedback loop that receives gaze and gesture input data and adapts one or more computing devices to perform associated actions.

[0190] It should be appreciated that although the methods described herein refer to a multi-device configuration, similar operations may be performed on a single device configuration. For example, a user may move an application from one portion of a display screen to a second portion of the display screen, rather than transferring the application across computing devices (as described with respect to FIG. Fig. 12A and Fig. 12B described).

[0191] Fig.14 An example system 1400 for gaze tracking according to various aspects described herein is shown. The system 1400 includes a user 1402 and multiple computing devices, such as a first computing device 1404 and a second computing device 1406. The multiple computing devices 1404, 1406 communicate via a network 1408. The multiple computing devices 1404, 1406 can be connected to the network 1408 described previously in this document. Figure 1 Devices 103 through 108 are similar to those discussed above. Additionally, network 1408 may be similar to that discussed previously herein. Figure 1 The network 110 discussed is similar.

[0192] The first computing device 1404 may include a plurality of applications running thereon. For example, the first computing device 1404 may include a spreadsheet application 1410 running thereon. The second computing device 1406 may include a plurality of applications (not shown) running thereon. It should be appreciated that the application 1410 may be any of a plurality of applications or software programs that provide some visual displays that a user may gaze upon. Examples of such applications or computer programs include, but are not limited to, word processors, graphics software, database software, spreadsheet software, web browsers, enterprise software, information worker software, multimedia software, presentation software, educational software, content access software, communication software, and the like.

[0193] The mechanisms described herein provide users with the opportunity to select or focus on a specific element (e.g., a cell, file, folder, button, text box, string variable, etc.) on a computing device based on where one or more users (e.g., user 1402) are looking. Fig.14 , user 1402 is shown looking at element 1412 on computing device 1404. In the specific example of system 1400, element 1412 is a cell of spreadsheet application 1410. By viewing cell 1412, user 1402 is able to select cell 1412 on spreadsheet application 1410 so that further actions (e.g., typing) can be performed. Additionally or alternatively, in some examples, the user can view a specific file on the computing device to open the file, and / or view a specific title of the file to receive an option to change the name of the file, and / or view a specific button of the application to change the state of the button (e.g., from not pressed to pressed), and / or view a specific location to move a mouse cursor to the location, and / or view a string variable to highlight the string variable, etc.

[0194] Fig.15 An overview of an example method 1500 for processing gaze input data to perform an action that affects the behavior of a computing device is shown. According to some examples, aspects of the method 1500 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0195] Method 1500 begins at operation 1502, where one or more computing devices are identified. For example, a user may use the above description of Figure 1The method 1500 may be used to link one or more devices (e.g., device 103 to device 108) using any of the communication methods discussed above. The user may then manually select one or more devices to be linked together to be identified by the method 1500. Additionally or alternatively, the network (e.g., network 1408) may be configured to automatically identify one or more devices connected to the network. In yet another example, the network (e.g., network 1408) may be configured to detect computing devices within a specified geographic proximity.

[0196] At operation 1504, one or more users are identified. The one or more users can be identified by one or more computing devices (e.g., devices 103 to 106 and / or devices 1404, 1406). Specifically, the one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify one or more users (e.g., user 1402). The visual data can be processed using the mechanisms described herein to perform facial identification on the one or more users if the one or more users have provided permission to do so. For example, the one or more computing devices can create a grid on the face of each of the one or more users to identify facial features, such as nose location, mouth location, cheek location, hair location, eye location, and / or eyelid location.

[0197] Additionally or alternatively, at operation 1504, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged in, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged in, the user is thereby identified. Additionally or alternatively, at operation 1504, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0198] Additionally or alternatively, at operation 1504, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0199] At operation 1506, gaze input data is received from one or more computing devices (e.g., computing devices 103 to 108, and / or computing devices 1404, 1406) corresponding to one or more users (e.g., user 1402). Once one or more users are identified at 1504, method 1500 can monitor the orientation of the user's eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can be provided to a multi-device gaze tracking system (e.g., as described above with respect to Fig.14 The system 1400 discussed provides an indication of where a user may be looking relative to a display screen (eg, a display screen of device 1404 and / or device 1406).

[0200] Still referring to operation 1506, one or more computing devices (e.g., computing devices 103 to 108 and / or computing devices 1404, computing device 1406) can receive gaze data from multiple users (e.g., the computing device can track the orientation of the eyes of multiple users and receive gaze data from them). Specifically, one or more computing devices can track the device that each user is looking at (e.g., computing devices 103 to 108 and / or computing devices 1404, computing device 1406), and even further can determine what each user is looking at on the device (e.g., elements being displayed on one or more computing devices, such as cells, files, buttons, text boxes, string variables, etc.). The gaze data can be received in real time (e.g., providing a continuous feedback stream about what multiple users are looking at). Alternatively, the gaze data can be received periodically (e.g., at regular or irregular time intervals that can be specified by the user).

[0201] Further, referring to operation 1506, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data was first received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0202] At determination 1508, it is determined whether there is an element associated with the gaze input data. For example, determination 1508 may include evaluating the received gaze input data to generate a set of user signals that may be processed in light of an environmental context (e.g., an application currently running on the device or a task currently being performed and a specific element being displayed therewith). Thus, the evaluation may identify the element as a result of an association between the gaze input data and the environmental context.

[0203] In some examples, at determination 1508, it is determined for each user whether there is an element associated with the gaze input data corresponding to the user. For example, determination 1508 may include evaluating the received gaze input data to generate a set of user signals, wherein each user signal corresponds to a user in a plurality of users. The user signal can be processed in view of the environmental context (e.g., the application currently running on the device or the task currently being performed and the specific element being displayed therewith). Therefore, the evaluation can identify one or more elements as the result of the association between the gaze input data for each user and the environmental context. It should be appreciated that different actions can be identified for each user based on different gaze input data (e.g., different users viewing different computing devices). Alternatively, the same element can be identified for each user based on the same gaze input data (e.g., different users viewing the same computing device).

[0204] If it is determined that there is no element associated with the gaze input data, the process branches "no" to operation 1510, where a default action is performed. For example, the gaze input data may have an associated predetermined element. In some other examples, operation 1500 may include determining whether the gaze input data has an associated default element, so that in some instances, as a result of the received gaze input data, no action may be performed. Method 1500 may terminate at operation 1510. Alternatively, method 1510 may return from operation 1510 to operation 1502 to create a continuous feedback loop for the user to gaze input data and select an element.

[0205] However, if it is determined that there is a gaze command associated with the received gaze input data, then flow branches "yes" to operation 1512 where an element is determined based on the gaze input data. Fig.14 , when the user 1402 gazes at a cell 1412 (e.g., a type of element) of the spreadsheet application 1410, it is determined that the user is gazing at the cell 1412 of the spreadsheet application 1410. In other examples, if the user gazes at a text box in a search engine, it can be determined that the user is gazing at the text box of the search engine, and therefore may desire to search for something. In other examples, if the user gazes at the title of a document, it can be determined that the user is gazing at the title of the document, and therefore may desire to edit the title of the document.

[0206] The process proceeds to operation 1514, where one or more computing devices are adapted to select the determined element. Alternatively, in some examples, one or more computing devices may be adapted to focus on the determined element, or change the state of the determined element. For example, the determined element may be selected, focused on and / or changed state by the computing device that performs method 1500. In another example, the indication of the determined element may be provided to another computing device. For example, various aspects of method 1500 may be performed by a peripheral device so that operation 1514 includes providing input to the associated computing device. As another example, operation 1514 may include using an application programming interface (API) call to adapt one or more computing devices to select, focus on or change the state of the determined element. Method 1500 may terminate at operation 1514. Alternatively, method 1500 may return to operation 1502 from operation 1514 to create a continuous feedback loop of looking at input data and selecting, focusing on or changing the state of an element on a computing device.

[0207] Fig.16A and Fig. 16B An example system 1600 for multi-device gaze tracking according to aspects described herein is shown. The system 1600 includes a user 1602 and one or more computing devices 1604. The system 1600 also includes a first user interface input 1606 and a second user interface input 1608 (e.g., input that may be received from a keyboard or touchpad that receives keystroke input from a user).

[0208] The first computing device 1604 may include a plurality of applications running thereon. For example, the computing device 1604 may include a spreadsheet application 1610 running thereon. It should be appreciated that the application 1610 may be any one of a plurality of applications or software programs at which some visual display is provided that a user may look at. Examples of such applications or computer programs include, but are not limited to, word processors, graphics software, database software, spreadsheet software, web browsers, enterprise software, information worker software, multimedia software, presentation software, educational software, content access software, communication software, and the like.

[0209] In addition to user interface data received from one or more computing devices, the mechanisms described herein also provide users with the ability to perform actions based on where one or more users (e.g., user 1602) are looking. Fig.16A , the user 1602 looks at the first cell or element 1612 of the spreadsheet application 1610 while also entering the first user interface input 1606 (e.g., copy or "Ctrl+C"). Then, referring to Fig. 16B, user 1602 looks at a second cell or element 1614 of spreadsheet application 1610, and also simultaneously and subsequently or previously inputs a second user interface input 1608 (e.g., paste or "Ctrl+V"). Thus, the user can use gaze data and user interface data to perform copy and paste commands.

[0210] It should be appreciated that the user can perform other keyboard commands based on the location the user is looking at on the display screen of the computing device in combination with the gaze data based on numerous keyboard shortcuts known to those of ordinary skill in the art. In addition, although the first user interface input 1606 and the second user interface input 1608 are discussed above as keyboard inputs, the first user interface input 1606 and the second user interface input 1608 may be any of a variety of user interface inputs that are not keyboard-specific. For example, on a computing device with a touch screen, the first user interface input 1606 and the second user interface input 1608 may be variations of touch commands (e.g., long press on the display screen to perform a copy operation, short press on the display screen to perform a paste operation, etc.). As another example, the first user interface input 1606 and the second user interface input 1608 may be voice inputs (e.g., verbally instructing the voice command module to perform a copy operation, verbally instructing the voice command module to perform a paste operation, etc.). As another example, the first user interface input 1606 and the second user interface input 1608 may be gaze commands (e.g., long gaze to perform a copy operation, short gaze to perform a paste operation, etc.). As another example, the first user interface input 1606 and the second user interface input 1608 may be input from a peripheral device (eg, stepping on or releasing a foot pedal, or pressing a display of a peripheral computing device, etc.).

[0211] Fig.17A and Fig. 17B An example system 1700 for multi-device gaze tracking according to aspects described herein is shown. The system 1700 includes a user 1702 and one or more computing devices 1704. The system 1700 also includes a first user interface input 1706 and a second user interface input 1708 (e.g., input that may be received from a keyboard or touchpad that receives keystroke input from a user).

[0212] The first computing device 1704 may include a plurality of applications running thereon. For example, the computing device 1704 may include a spreadsheet application 1710 running thereon. It should be appreciated that the application 1710 may be any one of a plurality of applications or software programs at which some visual display is provided that a user may look at. Examples of such applications or computer programs include, but are not limited to, word processors, graphics software, database software, spreadsheet software, web browsers, enterprise software, information worker software, multimedia software, presentation software, educational software, content access software, communication software, and the like.

[0213] In addition to user interface data received from one or more computing devices, the mechanisms described herein also provide users with the ability to perform actions based on where one or more users (e.g., user 1702) are looking. Fig.17A , the user 1702 looks at the first cell or element 1712 of the spreadsheet application 1710, and simultaneously or previously enters the first user interface input 1706 (e.g., cut or "Ctrl+X"). Then, referring to Fig. 17B , the user 1702 looks at a second cell or element 1714 of the spreadsheet application 1710 and simultaneously enters a second user interface input 1708 (e.g., paste or "Ctrl+V"). Thus, the user can use gaze data and user interface data to perform cut and paste commands.

[0214] It should be appreciated that the user can perform other keyboard commands based on the location the user is looking at on the display screen of the computing device, based on numerous keyboard shortcuts known to those of ordinary skill in the art, in combination with gaze data. In addition, although the first user interface input 1706 and the second user interface input 1708 are discussed above as keyboard inputs, the first user interface input 1706 and the second user interface input 1708 may be any of a variety of user interface inputs that are not keyboard-specific. For example, on a computing device with a touch screen, the first user interface input 1706 and the second user interface input 1708 may be variations of touch commands (e.g., long press on the display screen to perform a cut operation, short press on the display screen to perform a paste operation, etc.). As another example, the first user interface input 1706 and the second user interface input 1708 may be voice inputs (e.g., verbally instructing the voice command module to perform a cut operation, verbally instructing the voice command module to perform a paste operation, etc.). As another example, the first user interface input 1706 and the second user interface input 1708 may be gaze commands (e.g., long gaze to perform a cut operation, short gaze to perform a paste operation, etc.). As another example, the first user interface input 1706 and the second user interface input 1708 may be input from a peripheral device (eg, stepping on or releasing a foot pedal, or pressing a display of a peripheral computing device, etc.).

[0215] Fig.18 An overview of an example method 1800 for processing gaze input data to perform an action that affects the behavior of a computing device is shown. According to some examples, aspects of the method 1800 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0216] Method 1800 begins at operation 1802, where one or more computing devices are identified. For example, a user may use the above description of Figure 1The method 1500 may be used to link one or more devices (e.g., device 103 to device 108) by any communication method discussed. The device may be identified by a previous link association (e.g., indicated in a device profile or a sharing profile). Alternatively, one or more devices may be identified based on user login information for different devices (e.g., each device with the same user login may be linked). In another aspect, one or more devices may be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity may be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical properties around the device (e.g., based on device camera feed if the user has given the device permission to use the camera for this purpose). In another example, the user may then manually select one or more devices linked together to be identified by method 1500 to identify the device at operation 1802. Additionally or alternatively, the network may be configured to automatically identify one or more devices connected to the network. In another example, the network may be configured to detect computing devices within a specified geographic proximity.

[0217] At operation 1804, one or more users are identified. One or more users can be identified by one or more computing devices (e.g., devices 103 to 106 and / or devices 1604, 1704). Specifically, one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify one or more users (e.g., user 1602, user 1702). The visual data can be processed using the mechanisms described herein to perform facial identification on one or more users if one or more users have provided permission to do so. For example, one or more computing devices can create a grid on the face of each of the one or more users to identify facial features, such as nose location, mouth location, cheek location, hair location, eye location, and / or eyelid location.

[0218] Additionally or alternatively, at operation 1804, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged in, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged in, the user is thereby identified. Additionally or alternatively, at operation 1804, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0219] Additionally or alternatively, at operation 1804, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0220] At operation 1804, gaze input data is received from one or more computing devices (e.g., computing devices 103 to 108, and / or computing devices 1604, 1706) corresponding to one or more users (e.g., user 1602, user 1702). Once one or more users are identified at 1804, method 1800 can monitor the orientation of the user's eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can provide a multi-device gaze tracking system (e.g., system 1600 discussed above with respect to FIG. 16, and / or system 1700 discussed above with respect to FIG. 17) with an indication of where the user may be looking relative to a display screen (e.g., the display screens of device 1604 and device 1704).

[0221] Still referring to operation 1806, one or more computing devices (e.g., computing devices 103 to 108 and / or computing devices 1604, computing devices 1704) can receive gaze data from multiple users (e.g., the computing device can track the orientation of the eyes of multiple users and receive gaze data from them). Specifically, one or more computing devices can track the device that each user is looking at (e.g., computing devices 103 to 108 and / or computing devices 1604, computing devices 1704), and even further can determine what each user is looking at on the device (e.g., elements being displayed on one or more computing devices, such as cells, files, buttons, text boxes, string variables, etc.). The gaze data can be received in real time (e.g., providing a continuous feedback stream about what multiple users are looking at). Alternatively, the gaze data can be received periodically (e.g., at regular or irregular time intervals that can be specified by the user).

[0222] Further, referring to operation 1806, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data was first received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0223] At operation 1808, user interface input data is received from one or more computing devices (e.g., computing device 103 to computing device 108 and / or computing device 1604, computing device 1706). Such user interface input data can be received from, for example, a keyboard, touchpad, touch screen, or other user interface. For example, the user interface input data can correspond to key strokes received by a keyboard, touchpad, or other computer input device or interface. The user interface input data can be received in real time (e.g., to provide feedback about user interface commands being executed by a user). Alternatively, the user interface input data can be received periodically (e.g., at regular or irregular time intervals that can be specified by the user) to save computing resources.

[0224] At determination 1810, it is determined whether there is an action associated with the gaze input data and the user interface input data. For example, determination 1810 may include evaluating the received gaze and user interface input data to generate a set of user signals that may be processed in light of an environmental context (e.g., an application currently running on the device or a task currently being performed and a specific element being displayed therewith). Thus, the evaluation may identify an action as a result of an association between the gaze input data, the user interface input data, and the environmental context.

[0225] In some examples, at determination 1810, it is determined for each user whether there is an action associated with the gaze and user interface input data corresponding to the user. For example, determination 1810 may include evaluating the received gaze and user interface input data to generate a set of user signals, wherein each user signal corresponds to one of the multiple users. The user signal can be processed in view of the environmental context (e.g., the application currently running on the device or the task currently being executed, and the specific element being displayed with it). Therefore, the evaluation can identify one or more actions as the result of the association between the gaze input data for each user and the user interface input data of each user and the environmental context. It should be recognized that different actions can be determined for each user based on different gaze and user interface input data (e.g., different users viewing different computing devices, and / or different users entering different user interface inputs). Alternatively, the same action can be determined for each user based on the same gaze and user interface input data (e.g., different users viewing the same computing device and entering the same user interface input).

[0226] If it is determined that there is no action associated with the gaze and user interface input data, the process branches "no" to operation 1812, where a default action is performed. For example, the gaze input data and the user interface input data may have an associated predetermined action. In some other examples, method 1812 may include determining whether the gaze input data and the user interface input data have an associated default action, so that in some instances, as a result of the received gaze input data and the user interface input data, no action may be performed. Method 1800 may terminate at operation 1810. Alternatively, method 1812 may return from operation 1812 to operation 1802 to create a continuous feedback loop for the user to receive gaze input data and user interface input data and execute commands.

[0227] However, if it is determined that there is a gaze command associated with the received gaze input data and the received user interface input data, then the flow branches "yes" to operation 1814, where an action is determined based on the gaze input data and the user interface input data. Fig.16A and Fig. 16B , when user 1602 gazes at a first cell 1612 of spreadsheet application 1610 and enters a first user interface input 1606 (e.g., Ctrl+C), it is determined that user 1602 is copying the contents of first cell 1612. Then, when user 1602 gazes at a second cell 1614 of spreadsheet application 1610 and enters a second user interface input 1608 (e.g., Ctrl+V), it is determined that user 1602 is pasting the contents of first cell 1612 into second cell 1614.

[0228] As another example, refer to Fig.17A and Fig. 17B , when user 1702 gazes at a first cell 1712 of spreadsheet application 1710 and enters a first user interface input 1706 (e.g., Ctrl+X), it is determined that user 1702 is cutting the contents of first cell 1712. Then, when user 1702 gazes at a second cell 1714 of spreadsheet application 1710 and enters a second user interface input 1708 (e.g., Ctrl+V), it is determined that user 1702 is removing the contents from first cell 1712 and pasting the contents into second cell 1714.

[0229] The process proceeds to operation 1816, where one or more computing devices are adapted to perform the determined action. For example, the determined action can be performed by the computing device that performs method 1800. In another example, the indication of the determined action can be provided to another computing device. For example, various aspects of method 1800 can be performed by peripheral devices, so that operation 1816 includes providing input to the associated computing device. As another example, operation 1816 can include using an application programming interface (API) call to adapt one or more computing devices to perform the determined action. Method 1800 can terminate at operation 1816. Alternatively, method 1800 can return to operation 1802 from operation 1816 to create a continuous feedback loop that receives gaze and user interface input data and performs actions based on gaze and user interface input data.

[0230] Fig.19A and Fig.19B An example system 1900 for multi-device gaze tracking in accordance with aspects described herein is shown. System 1900 includes a plurality of users 1902, a first computing device 1904, and a second computing device 1906. The plurality of users 1902 may include a presenter 1902a and an audience member 1902b.

[0231] The first computing device 1904 may include a plurality of applications running thereon. For example, the first computing device 1904 may include a first application (e.g., a word processor application) 1908 and a second application (e.g., a spreadsheet application) 1910 running thereon. It should be appreciated that the applications 1908, 1910 may be any of a plurality of applications or software programs that provide some visual display that a user may look at. Examples of such applications or computer programs include, but are not limited to, word processors, graphics software, database software, spreadsheet software, web browsers, enterprise software, information worker software, multimedia software, presentation software, educational software, content access software, communication software, and the like.

[0232] The mechanisms described herein provide users with the ability to present applications based on the location of one or more users (e.g., presenter 1902a and / or audience member 1902b). Fig.19A , user 1902 looks at first application 1908 of first computing device 1904. As a result, first application 1908 is demonstrated on second computing device 1906. Alternatively, now referring to Fig.19B , user 1902 looks at second application 1910 on first computing device 1904. As a result, second application 1910 is demonstrated on second computing device 1906. Such a capability may be beneficial to a user when giving a presentation and / or when screens are shared (e.g., in a video call or telephone conference).

[0233] In other examples, a decision is made as to which application (e.g., of application 1908 and application 1910) to display based on gaze data corresponding to audience member 1920b. For example, if a majority of audience members 1920b are determined to be looking at an application (e.g., application 1908, application 1910) on a first computing device (e.g., device 1904), the application may be presented or screen shared on a second computing device (e.g., device 1906).

[0234] Although Fig.19A and Fig.19B Example system 1900 shows a multi-device system. It should be appreciated that the aspects disclosed above may also be applied to a single-device system. For example, if a user has multiple tabs open on a device, each tab corresponding to an application, and is giving a presentation or participating in a video call, by viewing one of the multiple tabs, it can be determined which tab will be magnified, presented, screen shared, or otherwise displayed. The multiple tabs can be tabs of applications that are minimized (e.g., in a taskbar), applications that overlap each other (e.g., on a desktop), or applications that are arranged in a display (e.g., tabs of an internet browser shown on a display of a computing device).

[0235] Fig. 20 An overview of an example method 2000 for processing gaze input data to perform an action that affects the behavior of a computing device is shown. According to some examples, aspects of the method 2000 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0236] Method 2000 begins at operation 2002, where one or more computing devices are identified. For example, a user may use the above description of Figure 1 Any communication method discussed is used to link one or more devices (e.g., device 103 to device 108 and / or device 1904, device 1906). The device can be identified by a previous link association (e.g., indicated in a device profile or a shared profile). Alternatively, one or more devices can be identified based on user login information for different devices (e.g., each device with the same user login can be linked). In another aspect, one or more devices can be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity can be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical characteristics around the device (e.g., if the user has given the device permission to use the camera for this purpose, based on the device camera feed). In another example, the user can then manually select one or more devices linked together to be identified by method 2000 to identify the device at operation 2002. Additionally or alternatively, the network can be configured to automatically identify one or more devices connected to the network. In yet another example, a network may be configured to detect computing devices within a specified geographic proximity.

[0237] At operation 2004, one or more users are identified. One or more users can be identified by one or more computing devices (e.g., devices 103 to 106 and / or devices 1904, device 1906). Specifically, one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify one or more users (e.g., presenter 1902a and / or audience member 1902b). The visual data can be processed using the mechanisms described herein to perform facial identification on one or more users if one or more users have provided permission to do so. For example, one or more computing devices can create a grid on the face of each of the one or more users to identify facial features, such as nose location, mouth location, cheek location, hair location, eye location, and / or eyelid location.

[0238] Additionally or alternatively, at operation 2004, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged in, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged in, the user is thereby identified. Additionally or alternatively, at operation 1804, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0239] Additionally or alternatively, at operation 2004, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0240] At operation 2006, gaze input data is received from one or more computing devices (e.g., computing devices 103 to 108, and / or computing devices 1904, 1906) corresponding to one or more users (e.g., presenter 1902a and / or audience member 1902b). Once one or more users are identified at 2004, method 2000 can monitor the orientation of the users' eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can be provided to a multi-device gaze tracking system (e.g., as described above with respect to Fig.19A and Fig.19BThe system 1900 discussed provides an indication of where a user may be looking relative to a display screen (eg, the display screens of device 1904 and device 1906).

[0241] Still referring to operation 2006, one or more computing devices (e.g., computing devices 103 to 108 and / or computing devices 1904, computing device 1906) can receive gaze data from multiple users (e.g., the computing device can track the orientation of the eyes of multiple users and receive gaze data therefrom). Specifically, one or more computing devices can track the device each user is looking at (e.g., computing devices 103 to 108 and / or computing devices 1904, computing device 1906), and even further can determine what each user is looking at on the device (e.g., an application, a tab, or a portion of an application). The gaze data can be received in real time (e.g., providing a continuous feedback stream about what multiple users are looking at). Alternatively, the gaze data can be received periodically (e.g., at regular or irregular time intervals that can be specified by the user).

[0242] Further, referring to operation 2006, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data was first received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0243] At operation 2008, context data is received from one or more computing devices (e.g., computing device 103 to computing device 108 and / or computing device 1904, computing device 1906). Such context data can be received from, for example, a shared computing component (e.g., shared computing component 122). The context data can correspond to an application currently running or a command currently being executed on a computing device. As a specific example, if a user is making a video call and the user is looking at an application (e.g., a word processing application, an Internet browser, a spreadsheet application, etc.), the context data can be useful for determining the application they are looking at that the user wants to screen share or demonstrate. In addition, if the computing device is coupled to a specific hardware device (e.g., a projector, an HDMI cable, a DisplayPort cable, a VGA cable, a DVI cable, a USB cable, or a USB Type-C cable), it can be determined that the user expects a specific action to be performed when looking at the application (e.g., a screen sharing application, a demonstration application, etc.). The context data can be received in real time (e.g., to provide feedback on an application or command currently being run by a computing device). Alternatively, user interface input data may be received periodically (eg, at regular or irregular time intervals that may be specified by the user) to conserve computing resources.

[0244] At determination 2010, it is determined whether there is an application associated with the gaze input data. For example, determination 2010 may include evaluating the gaze input data to generate a set of user signals that may be processed in light of an environmental context (e.g., an application currently running on the device or a task currently being performed and a specific element being displayed therewith). Thus, the evaluation may identify an application as a result of an association between the gaze input data and the environmental context.

[0245] In some examples, at determination 2010, it is determined for each user whether there is an application associated with the gaze input data corresponding to the user. For example, determination 2010 may include evaluating the received gaze input data to generate a set of user signals, wherein each user signal corresponds to one of the multiple users. The user signal can be processed in view of the environmental context (e.g., the application currently running on the device or the task currently being performed, and the specific element being displayed therewith). Therefore, the evaluation can identify one or more applications as the result of the association between the gaze input data for each user and the environmental context. It should be appreciated that different applications can be determined for each user based on different gaze input data (e.g., different users viewing different computing devices). Alternatively, the same application can be determined for each user based on the same gaze input data (e.g., different users viewing the same application on the same computing device).

[0246] If it is determined that there is no application associated with the gaze input data, the process branches "NO" to operation 2012, where a default action is performed. For example, the gaze input data may have an associated predetermined application. In some other examples, the method 2012 may include determining whether the gaze input data has an associated default action, so that in some instances, as a result of the received gaze input data, an application may not be identified. The method 2000 may terminate at operation 2012. Alternatively, the method 2000 may return from operation 2012 to operation 2002 to create a continuous feedback loop of receiving gaze input data and determining an application associated with the gaze data.

[0247] However, if it is determined that there is an application associated with the received gaze input data, the flow branches "yes" to operation 2014, where the application is determined based on the gaze input data. Fig.19A and Fig.19B , when the user 1902 gazes at the first application 1908, it is determined that the first application 1908 is associated with the user's gaze. Alternatively, when the user 1902 gazes at the second application 1910, it is determined that the second application 1910 is associated with the user's gaze.

[0248] At determination 2016, it is determined whether there is an action associated with the application and contextual data determined from operation 2014. For example, if presenter 1902a is giving a presentation (e.g., an HDMI cable is plugged into computing device 1904, and / or a video calling application is running on computing device 1904), then the application determined from operation 2014 may be presented (e.g., magnified or screen shared).

[0249] If it is determined that there is no action associated with the context data and the determined application (i.e., from operation 2014), the process branches "no" to operation 2012, where a default action is performed. For example, the context data and the determined application may have an associated predetermined action. In some other examples, operation 2012 may include determining whether the context data and the determined application have an associated default action, so that in some instances, as a result of the received context data and gaze input data, the action may not be performed. Method 2000 may terminate at operation 2012. Alternatively, method 2000 may return from operation 2012 to operation 2002 to create a continuous feedback loop for the user to receive gaze input data and context data and execute commands.

[0250] However, if it is determined that there is a gaze command associated with the determined application and context data, the flow branches "yes" to operation 2018, where the action is determined based on the determined application and context data. For example, when the user (e.g., presenter 1902a) is in a video call, or the user's computing device is coupled to an HDMI line, and the determined application is a word processor document (e.g., application 1908), then the determined action with respect to method 2000 can be screen sharing or presenting the word processor document on a second computing device (e.g., computing device 1906). Alternatively, the determined action can be screen sharing or presenting the word processor document via the video call within the video call interface.

[0251] The process proceeds to operation 2020, where one or more computing devices are adapted to perform the determined action. For example, the determined action can be performed by the computing device that executes method 2000. In another example, an indication of the determined action can be provided to another computing device. For example, various aspects of method 2000 can be performed by a peripheral device, so that operation 2020 includes providing input to an associated computing device. As another example, operation 2020 can include using an application programming interface (API) call to adapt one or more computing devices to perform the determined action. Method 2000 can terminate at operation 2020. Alternatively, method 2000 can return from operation 2020 to operation 2002 to create a continuous feedback loop that receives gaze input data and context data and performs actions based on gaze input data and context data.

[0252] Fig.21A and Fig.21B An example system 2100 for multi-device gaze tracking according to various aspects described herein is shown. The system 2100 includes multiple users (e.g., a primary user 2102a and a participant 2102b) and multiple computing devices, such as a first computing device 2104 and a second computing device 2106. The multiple computing devices 2104, 2106 communicate via a network 2108. The multiple computing devices 2104, 2106 can be connected to the network 2108 described previously in this document. Figure 1 Devices 103 to 108 are similar to those discussed above. In addition, network 2108 may be similar to that discussed previously herein. Figure 1 The network 110 discussed is similar.

[0253] The first computing device 2104 may include one or more applications running thereon. For example, the first computing device 2104 may include a video conferencing application 2110. Additionally or alternatively, the second computing device 1206 may include one or more applications running thereon. For example, the second computing device 1206 may include an information prompt application 2112 (e.g., an application that provides information about an object indicated by a user (such as the primary user 2102a)).

[0254] The mechanisms described herein provide a user (e.g., primary user 2102) with the ability to display information about a participant (e.g., participant 2102b) on a video call by looking at the participant. Computing device 2104 and / or computing device 2106 may receive gaze data corresponding to primary user 2102a, as well as contextual data (e.g., applications currently running on computing device 2104 and / or computing device 2106).

[0255] Specific reference Fig.21A , a primary user 2102a is participating in a video call with multiple participants 2102b via a computing device 2104 (e.g., via a video conferencing application 2110). When the primary user 2102a looks at one of the multiple participants 2102b (e.g., person 1), information corresponding to the participant 2102b (e.g., person 1) that the primary user 2102a is looking at is displayed on the computing device 2106 (e.g., via an information prompt application 2112). The displayed information corresponding to the participant 2102b that is being looked at by the primary user 2102a may include details about the participant 2102b, previous conversations, email threads, previously shared files, and / or side conversations.

[0256] refer to Fig.21B , when the primary user 2102a switches their gaze between multiple participants 2102b (e.g., from person 1 to person 3), the information displayed on the second computing device 2106 (e.g., via the information prompt application 2112) may also switch (e.g., from information corresponding to person 1 to information corresponding to person 3). Similarly, the information displayed corresponding to the participant 2102b being gazed at by the primary user 2102a may include details about the participant 2102b, previous conversations, email threads, previously shared files, and / or one-on-one conversations.

[0257] In some examples, the information corresponding to the participant 2102b being displayed on the computing device 2106 may be related to the amount of time that the primary user 2102a is looking at one or more participants 2102b. For example, if the primary user 2102a looks at "person 1" for a first duration (e.g., 3 seconds), then information corresponding to "person 1" may be displayed on the computing device 2106. In this regard, the information displayed on the computing device 2106 may be based on the duration that the primary user 2102a spends looking at one or more participants 2102b (e.g., "person 1", "person 2", or "person 3"). In some examples, the primary user 2102a may look at "person 1" for 3 seconds and then briefly look at "person 2" (e.g., 1 second or less). In such an example, the information corresponding to "person 1" will continue to be displayed on the computing device 2106 (i.e., the information corresponding to "person 2" will not be displayed in the brief moment that the primary user 2102a is looking at "person 2"). Additionally or alternatively, if user 2102a stops looking up "Person 1" for a second duration (e.g., 3 seconds), the information may fade from being displayed on computing device 2106 (e.g., information corresponding to another person may be displayed, or such information may not be displayed on the computing device).

[0258] It should be recognized that although Fig.21A and Fig.21B The example system 2100 of is shown and described with respect to a multi-device configuration, but aspects of the system 2100 may also be applied to a single device configuration, as will be appreciated by one of ordinary skill in the art. For example, rather than displaying information on the second computing device 2106 via the information prompt application 2112, the information may instead be displayed on the primary computing device 2104 (e.g., within a portion of the video conferencing application 2110, or via a separate graphical interface displayed on the computing device 2104).

[0259] Fig. 22 An overview of an example method 2200 for processing gaze input data to perform an action that affects the behavior of a computing device is shown. According to some examples, aspects of the method 2200 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0260] Method 2200 begins at operation 2202, where one or more computing devices are identified. For example, a user may use the above description of Figure 1Any communication method discussed is used to link one or more devices (e.g., device 103 to device 108 and / or device 2104, device 2106). The device can be identified by a previous link association (e.g., indicated in a device profile or a shared profile). Alternatively, one or more devices can be identified based on user login information for different devices (e.g., each device with the same user login can be linked). In another aspect, one or more devices can be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity can be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical characteristics around the device (e.g., if the user has given the device permission to use the camera for this purpose, based on the device camera feed). In another example, the user can then manually select one or more devices linked together to be identified by method 2200 to identify the device at operation 2202. Additionally or alternatively, the network (e.g., network 2108) can be configured to automatically identify one or more devices connected to the network. In yet another example, a network (eg, network 2108 ) may be configured to detect computing devices within a specified geographic proximity.

[0261] At operation 2204, one or more users are identified. One or more users can be identified by one or more computing devices (e.g., devices 103 to 106 and / or devices 2104, device 2106). Specifically, one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify one or more users (e.g., primary user 2102a). The visual data can be processed using the mechanisms described herein to perform facial identification on one or more users if one or more users have provided permission to do so. For example, one or more computing devices can create a grid on the face of each of the one or more users to identify facial features, such as nose location, mouth location, cheek location, hair location, eye location, and / or eyelid location.

[0262] Additionally or alternatively, at operation 2204, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged in, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged in, the user is thereby identified. Additionally or alternatively, at operation 2204, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0263] Additionally or alternatively, at operation 2204, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0264] At operation 2206, gaze input data is received from one or more computing devices (e.g., computing devices 103 to 108, and / or computing devices 2104, computing devices 2106) corresponding to one or more users (e.g., primary user 2102a). Once one or more users are identified at 2204, method 2200 can monitor the orientation of the user's eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can be provided to a multi-device gaze tracking system (e.g., as described above with respect to Fig.21A and Fig.21B The system 2100 discussed provides an indication of where a user may be looking relative to a display screen (eg, the display screens of device 2104 and device 2106).

[0265] Still referring to operation 2206, one or more computing devices (e.g., computing devices 103 to 108 and / or computing devices 2104, computing device 2106) may receive gaze input data from multiple users (e.g., the computing devices may track the orientation of the eyes of multiple users and receive gaze data therefrom). Specifically, the one or more computing devices may track the device (e.g., computing devices 103 to 108 and / or computing devices 2104, computing device 2106) that each user is looking at, and may even further determine what each user is looking at on the device (e.g., an application, or some other element being displayed on one or more computing devices, such as participant 2102b or an indication of participant 2102b).

[0266] The gaze input data may be received in real time (eg, providing a continuous stream of feedback about what multiple users are looking at). Alternatively, the gaze input data may be received periodically (eg, at regular or irregular time intervals that may be specified by the user).

[0267] Further, referring to operation 2206, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data was first received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0268] At operation 2208, context data is received from one or more computing devices (e.g., computing device 103 to computing device 108 and / or computing device 2104, computing device 2106). The context data may indicate a specific application (e.g., video conferencing software, presentation software, etc.) being run on the one or more computing devices. The context data may be received in real time (e.g., providing a continuous stream of feedback about the applications being run on the computing devices). Alternatively, the context data may be received periodically (e.g., at regular or irregular time intervals that may be specified by a user).

[0269] Additionally, with reference to operation 2208, context data may be stored (e.g., in a memory). In some examples, only recent context data is stored such that as context data is received, older context data is overwritten (e.g., in a memory) by new context data. Alternatively, in some examples, context data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data first began to be received). Typically, such implementations allow for a history of context data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0270] At determination 2210, it is determined whether there is an action associated with the gaze input data and the context data. For example, determination 2210 may include evaluating the received gaze input data to generate a user signal group that can be processed in view of the context data. Thus, the evaluation may identify an application or task as a result of the association between the gaze input data and the context data.

[0271] In some examples, at determination 2210, it is determined for each user whether there is an action associated with the gaze input data and the context data. For example, determination 2210 may include evaluating the received gaze input data to generate one or more user signal groups, wherein each user signal corresponds to one of the multiple users. The user signals can be processed in view of the context data. Thus, the evaluation may identify one or more actions as a result of the association between the gaze input data and the context data. It should be appreciated that different actions may be identified for each user based on different gaze input data (e.g., different users viewing different aspects of a computing device). Alternatively, the same action may be identified for each user based on the same gaze input data (e.g., different users viewing the same aspects of a computing device).

[0272] If it is determined that there is no application associated with the gaze input data and the context data, the process branches "NO" to operation 2212, where a default action is performed. For example, the gaze input data and the context data may have an associated predetermined action. In some other examples, method 2200 may include determining whether the gaze input data and the context data have an associated default action, so that in some instances, as a result of the received gaze input data and context data, no action may be performed. Method 2200 may terminate at operation 2212. Alternatively, method 2200 may return from operation 2212 to operation 2202 to create a continuous feedback loop of receiving gaze input data and context data and executing commands based on the gaze input data and context data.

[0273] However, if it is determined that there is a gaze command associated with the received gaze input data, then flow branches "yes" to operation 2214 where an action is determined based on the gaze input data and the contextual data. Fig.21A and Fig.21B , when the primary user 2102a looks at a participant (e.g., one of the participants 2102b) during a video call, it can be determined that the primary user 2102a is requesting information about the participant. Therefore, the system 2100 can determine that the requested information should be displayed on the second computing device 2106. As another example, the user can look at an element of another application (such as a presentation application or a word processing application). By looking at the element, the mechanisms described herein can determine that the user is requesting further information about the element.

[0274] The process proceeds to operation 2216, where one or more computing devices are adapted to perform the determined action. In some examples, one or more computing devices may be adapted to perform the determined action by the computing device executing method 2200. In another example, an indication of the determined action may be provided to another computing device. For example, various aspects of method 2200 may be performed by a peripheral device so that operation 2216 includes providing input to an associated computing device. As another example, operation 2216 may include using an application programming interface (API) call to adapt one or more computing devices to perform the determined action. Method 2200 may terminate at operation 2216. Alternatively, method 2200 may return to operation 2202 from operation 2216 to create a continuous feedback loop that receives gaze input data and context data and performs associated actions.

[0275] Fig.23 An example system 2300 for device gaze tracking according to aspects described herein is shown. System 2300 includes a user 2302 and a computing device 2304. Computing device 2304 includes an application 2306 running thereon. Application 2306 requires a login. Computing device 2304 can be used with the device previously described herein. Figure 1 The devices 103 to 108 discussed are similar.

[0276] The mechanisms described herein provide a user (e.g., user 2302) with the ability to log into a computing device or application based on gaze data. For example, computing device 2304 may receive gaze data corresponding to the position at which user 2302 is looking relative to a display of computing device 2304. User 2302 may formulate a pattern using their gaze (e.g., by looking at different points on a display screen of computing device 2304). Computing device 2304 may authenticate user 2302's pattern against a stored login pattern. If user 2302's pattern is the same as a stored login pattern, computing device 2304 or an application running thereon may be unlocked or logged in.

[0277] Although example system 2300 illustrates a single-device system, it should be appreciated that aspects of system 2300 may also be applied to multi-device systems. For example, a user may provide gaze data (e.g., a login pattern) to a first computing device in order to log into or unlock a second computing device or an application (e.g., application 2306) running on the second computing device.

[0278] Fig.24 An overview of an example method 2400 for processing gaze input data to perform an action that affects the behavior of a computing device is shown. According to some examples, aspects of the method 2400 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0279] Method 2400 begins at operation 2402, where one or more computing devices are identified. For example, a user may use the above description of Figure 1 Any communication method discussed is used to link one or more devices (e.g., device 103 to device 108 and / or device 2304). The device can be identified by a previous link association (e.g., indicated in a device profile or a shared profile). Alternatively, one or more devices can be identified based on user login information for different devices (e.g., each device with the same user login can be linked). In another aspect, one or more devices can be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity can be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical properties around the device (e.g., if the user has given the device permission to use the camera for this purpose, based on the device camera feed). In another example, the user can then manually select one or more devices identified by method 2400 to identify the device at operation 2402. Additionally or alternatively, the network can be configured to automatically identify one or more devices connected to the network. In another example, the network can be configured to detect computing devices within a specified geographic proximity.

[0280] At operation 2404, one or more users are identified. The one or more users can be identified by one or more computing devices (e.g., devices 103 to 106 and / or device 2304). Specifically, the one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify one or more users (e.g., user 2302). The visual data can be processed using the mechanisms described herein to perform facial identification on the one or more users if the one or more users have provided permission to do so. For example, the one or more computing devices can create a grid on the face of each of the one or more users to identify facial features, such as nose location, mouth location, cheek location, hair location, eye location, and / or eyelid location.

[0281] Additionally or alternatively, at operation 2404, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, opening an application, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged into, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged into, the user is thereby identified. Additionally or alternatively, at operation 2404, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0282] Additionally or alternatively, at operation 2404, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0283] At operation 2406, gaze input data is received from one or more computing devices (e.g., computing devices 103 to 108, and / or computing device 2304) corresponding to one or more users (e.g., user 2302). Once the one or more users are identified at 2404, method 2400 can monitor the orientation of the users' eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can be provided to a multi-device gaze tracking system (e.g., as described above with respect to Fig.23 The system 2300 discussed provides an indication of where a user may be looking relative to a display screen (e.g., the display screen of device 2304).

[0284] Still referring to operation 2406, one or more computing devices (e.g., computing devices 103 to 108 and / or computing device 2304) can receive gaze input data from multiple users (e.g., the computing device can track the orientation of the eyes of multiple users and receive gaze data therefrom). Specifically, one or more computing devices can track the device (e.g., computing devices 103 to 108 and / or computing device 2304) that each user is looking at, and even further can determine what each user is looking at on the device (e.g., an application or some other element being displayed on one or more computing devices, such as a password or login mode). The gaze input data can be received in real time (e.g., providing a continuous feedback stream about what multiple users are looking at). Alternatively, the gaze input data can be received periodically (e.g., at regular or irregular time intervals that can be specified by the user).

[0285] Further, referring to operation 2406, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data was first received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0286] At operation 2408, the gaze input data is compared to the lock data. The lock data may be stored (e.g., in a memory). The lock data may be a default setting generated by a device manufacturer or application programmer. Alternatively, the lock data may be generated by a user, such as by setting a specific pattern of eye movement as a password or login for an application or device.

[0287] At determination 2410, it is determined whether the gaze input data matches the lock data. For example, determination 2410 may include evaluating the received gaze input data to generate a user signal set that can be compared to the lock data stored in the memory. Thus, the evaluation may determine whether the application or device should be logged in or unlocked.

[0288] If it is determined that the gaze input data does not match the lock data, the process branches "no" to operation 2412, where a default action is performed. For example, the gaze input data may have an associated predetermined action. In some other examples, operation 2412 may include determining whether the gaze input data has an associated default action, so that in some instances, as a result of the received gaze input data, no action may be performed. Method 2400 may terminate at operation 2412. Alternatively, method 2400 may return from operation 2412 to operation 2402 to create a continuous feedback loop of receiving gaze input that determines whether the device or application should be unlocked or logged in.

[0289] However, if it is determined that there is a gaze command associated with the received gaze input data, then flow branches "yes" to operation 2414, where the computing device or application is adapted to be unlocked. Fig.23 When the user 2302 uses their gaze to execute a pattern that matches the lock data stored by the computing device 2304, the computing device 2304 is unlocked.

[0290] Fig.25An example system 2500 for device gaze tracking according to aspects described herein is shown. System 2500 includes a user 2502 and a computing device 2504. Computing device 2504 includes an application 2506 (e.g., a word processing application) running thereon. Computing device 2504 can be similar to the one previously described herein with respect to Figure 1 The devices 103 to 108 discussed are similar.

[0291] The mechanisms described herein provide a user (e.g., user 2502) with the ability to zoom in or out an application based on gaze data. For example, computing device 2504 may receive gaze data corresponding to a position that user 2502 is looking at relative to a display of computing device 2504. An application (e.g., application 2506) may be identified as corresponding to a position that user 2502 is looking at (e.g., based on the gaze data received). It may be determined based on the gaze data that an application should be zoomed in. Alternatively, it may be determined based on the gaze data that an application should be zoomed out. In some examples, a user may enter a key command or other user interface input to specify whether an application should be zoomed in or out.

[0292] Although example system 2500 illustrates a single-device system, it should be appreciated that aspects of system 2500 may also be applied to multi-device systems. For example, a user may provide gaze data to a first computing device to zoom in or out on an application on a second computing device.

[0293] Fig.26 An overview of an example method 2600 for processing gaze input data to perform an action that affects the behavior of a computing device is shown. According to some examples, aspects of the method 2600 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0294] Method 2600 begins at operation 2602, where one or more computing devices are identified. For example, a user may use the above description of Figure 1Any communication method discussed is used to link one or more devices (e.g., device 103 to device 108 and / or device 2504). The device can be identified by a previous link association (e.g., indicated in a device profile or a shared profile). Alternatively, one or more devices can be identified based on user login information for different devices (e.g., each device with the same user login can be linked). In another aspect, one or more devices can be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity can be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical characteristics around the device (e.g., if the user has given the device permission to use the camera for this purpose, based on the device camera feed). In another example, the user can then manually select one or more devices to be identified by method 2600 to identify the device at operation 2602. Additionally or alternatively, the network can be configured to automatically identify one or more devices connected to the network. In another example, the network can be configured to detect computing devices within a specified geographic proximity.

[0295] At operation 2604, one or more users are identified. The one or more users can be identified by one or more computing devices (e.g., devices 103 to 106 and / or device 2504). Specifically, the one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify one or more users (e.g., user 2502). The visual data can be processed using the mechanisms described herein to perform facial identification on the one or more users if the one or more users have provided permission to do so. For example, the one or more computing devices can create a grid on the face of each of the one or more users to identify facial features, such as nose location, mouth location, cheek location, hair location, eye location, and / or eyelid location.

[0296] Additionally or alternatively, at operation 2604, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, opening an application, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged in, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged in, the user is thereby identified. Additionally or alternatively, at operation 2604, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0297] Additionally or alternatively, at operation 2604, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0298] At operation 2606, gaze input data is received from one or more computing devices (e.g., computing devices 103 to 108, and / or computing device 2504) corresponding to one or more users (e.g., user 2502). Once the one or more users are identified at 2604, method 2600 can monitor the orientation of the users' eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can be provided to a multi-device gaze tracking system (e.g., as described above with respect to Fig.25 The system 2500 discussed provides an indication of where a user may be looking relative to a display screen (e.g., the display screen of device 2504).

[0299] Still referring to operation 2606, one or more computing devices (e.g., computing devices 103 to 108 and / or computing device 2504) can receive gaze input data from multiple users (e.g., the computing device can track the orientation of the eyes of multiple users and receive gaze data therefrom). Specifically, one or more computing devices can track the device (e.g., computing devices 103 to 108 and / or computing device 2504) that each user is looking at, and even further can determine what each user is looking at on the device (e.g., an application or some other element being displayed on one or more computing devices). The gaze input data can be received in real time (e.g., providing a continuous feedback stream about what multiple users are looking at). Alternatively, the gaze input data can be received periodically (e.g., at regular or irregular time intervals that can be specified by the user).

[0300] Further, referring to operation 2606, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data was first received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0301] At determination 2608, it is determined whether there is an application associated with the gaze input data. For example, determination 2608 may include evaluating the received gaze input data to generate a set of user signals that may be processed in view of an environmental context (e.g., an application currently running on the device or a task currently being performed). Thus, the evaluation may identify an application as a result of an association between the gaze input data and the environmental context.

[0302] In some examples, at determination 2608, it is determined for each user whether there is an application associated with the gaze input data corresponding to the user. For example, determination 2608 may include evaluating the received gaze input data to generate a set of user signals, wherein each user signal corresponds to one of the multiple users. The user signal can be processed in view of the environmental context (e.g., an application currently running on the device or a task currently being executed). Therefore, the evaluation can identify one or more applications as a result of the association between the gaze input data for each user and the environmental context. It should be appreciated that different applications can be identified for each user based on different gaze input data (e.g., different users viewing different computing devices). Alternatively, the same application can be identified for each user based on the same gaze input data (e.g., different users viewing the same computing device).

[0303] If it is determined that there is no application associated with the gaze input data, the flow branches "NO" to operation 2610, where a default action is performed. For example, the gaze input data may have an associated predetermined application. In some other examples, the method 2600 may include determining whether the gaze input data has an associated default application, so that in some instances, as a result of the received gaze input data, no action may be performed. The method 2600 may terminate at operation 2610. Alternatively, the method 2600 may return from operation 2610 to operation 2602 to create a continuous feedback loop for the user of gaze input data and identifying associated applications.

[0304] However, if it is determined that there is an application associated with the received gaze input data, then the flow branches "yes" to operation 2612, where the application is determined based on the gaze input data. Fig.25 , when the user 2502 looks at the word processing application 2506, it is determined that the user is looking at the word processing application 2506.

[0305] At determination 2614, it is determined whether there is a zoom action associated with the gaze input data and the determined application. For example, determination 2614 can include evaluating the type of application determined from operation 2612. If the determined application requires reading, it may be desirable for the application to be magnified when the user is looking at the application. Additionally or alternatively, in some examples, the user can provide a user interface input (e.g., keyboard input, mouse input, trackpad input, etc.) to indicate whether the user desires to magnify the determined application.

[0306] If it is determined that there is no zoom action associated with the gaze input data and the determined application, the process branches "no" to operation 2610, where a default action is performed. For example, the gaze input data may have an associated predetermined zoom action. In some other examples, method 2600 may include determining whether the gaze input data and the determined application have an associated default zoom action, so that in some instances, as a result of the received gaze input data and the determined application, no action may be performed. Method 2600 may terminate at operation 2610. Alternatively, method 2600 may return from operation 2610 to operation 2602 to create a continuous feedback loop for the user of gaze input data and identifying the associated application.

[0307] However, if it is determined that there is a zoom action associated with the received gaze input data, then the flow branches "yes" to operation 2612, where the one or more computing devices are adapted to perform the zoom action on the determined application. Fig.25 , when the user 2502 is looking at the word processing application 2506, it is determined that the user wants to zoom in on the point at which they are looking at the word processing application 2506. Therefore, a zoom operation is performed on the word processing application 2506, which zooms in on the word processing application 2506 on the display of the computing device 2504.

[0308] Fig. 27 An example system 2700 for device gaze tracking according to aspects described herein is shown. System 2700 includes a user 2702 and a computing device 2704. Computing device 2704 includes an application 2706 (e.g., a word processing application) running thereon. Computing device 2704 can be similar to the one previously described herein with respect to Figure 1 The devices 103 to 108 discussed are similar.

[0309] The mechanisms described herein provide a user (e.g., user 2702) with the ability to scroll (e.g., downward, upward, left, right, or diagonally) on an application based on gaze data. For example, computing device 2704 may receive gaze data corresponding to the position at which user 2702 is looking relative to a display of computing device 2704. An application (e.g., application 2706) may be identified as corresponding to the position at which user 2702 is looking (e.g., based on the gaze data received). It may be determined that the application should scroll downward based on the position on the application being viewed by the user. Alternatively, it may be determined that the application should scroll upward, left, right, and / or diagonally based on the position of the application being viewed by the user. In some examples, a user may enter a key command or other user interface input to specify whether the application should scroll upward, downward, left, right, and / or diagonally.

[0310] Although example system 2700 illustrates a single-device system, it should be appreciated that aspects of system 2700 may also be applied to multi-device systems. For example, a user may provide gaze data to a first computing device in order to scroll an application on a second computing device.

[0311] Fig.28 An overview of an example method 2800 for processing gaze input data to perform an action that affects the behavior of a computing device is shown. According to some examples, aspects of the method 2800 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0312] Method 2800 begins at operation 2802, where one or more computing devices are identified. For example, a user may use the above description of Figure 1 The method 2800 may be used to link one or more devices (e.g., device 103 to device 108 and / or device 2504) by any communication method discussed. The device may be identified by a previous link association (e.g., indicated in a device profile or a sharing profile). Alternatively, one or more devices may be identified based on user login information for different devices (e.g., each device with the same user login may be linked). In another aspect, one or more devices may be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity may be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical properties around the device (e.g., based on device camera feed if the user has given the device permission to use the camera for this purpose). In another example, the user may then manually select one or more devices to be identified by method 2800 to identify the device in operation 2802. Additionally or alternatively, the network may be configured to automatically identify one or more devices connected to the network. In another example, the network may be configured to detect computing devices within a specified geographic proximity.

[0313] At operation 2804, one or more users are identified. The one or more users can be identified by one or more computing devices (e.g., devices 103 to 106 and / or device 2704). Specifically, the one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify one or more users (e.g., user 2702). The visual data can be processed using the mechanisms described herein to perform facial identification on one or more users if the one or more users have provided permission to do so. For example, the one or more computing devices can create a grid on the face of each of the one or more users to identify facial features, such as nose location, mouth location, cheek location, hair location, eye location, and / or eyelid location.

[0314] Additionally or alternatively, at operation 2804, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, opening an application, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged in, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged in, the user is thereby identified. Additionally or alternatively, at operation 2804, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0315] Additionally or alternatively, at operation 2804, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0316] At operation 2806, gaze input data is received from one or more computing devices (e.g., computing devices 103 to 108, and / or computing device 2704) corresponding to one or more users (e.g., user 2702). Once the one or more users are identified at 2804, method 2800 can monitor the orientation of the user's eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can be provided to a multi-device gaze tracking system (e.g., as described above with respect to Fig. 27 The system 2700 discussed provides an indication of where a user may be looking relative to a display screen (e.g., the display screen of device 2704).

[0317] Still referring to operation 2806, one or more computing devices (e.g., computing devices 103 to 108 and / or computing device 2704) can receive gaze input data from multiple users (e.g., the computing device can track the orientation of the eyes of multiple users and receive gaze data therefrom). Specifically, one or more computing devices can track the device (e.g., computing devices 103 to 108 and / or computing device 2704) that each user is looking at, and even further can determine what each user is looking at on the device (e.g., an application or some other element being displayed on one or more computing devices). The gaze input data can be received in real time (e.g., providing a continuous feedback stream about what multiple users are looking at). Alternatively, the gaze input data can be received periodically (e.g., at regular or irregular time intervals that can be specified by the user).

[0318] Further, referring to operation 2806, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data was first received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0319] At determination 2808, it is determined whether there is an application associated with the gaze input data. For example, determination 2808 may include evaluating the received gaze input data to generate a set of user signals that may be processed in view of an environmental context (e.g., an application currently running on the device or a task currently being performed). Thus, the evaluation may identify an application as a result of an association between the gaze input data and the environmental context.

[0320] In some examples, at determination 2808, it is determined for each user whether there is an application associated with the gaze input data corresponding to the user. For example, determination 2808 may include evaluating the received gaze input data to generate a set of user signals, wherein each user signal corresponds to one of the multiple users. The user signal can be processed in view of the environmental context (e.g., an application currently running on the device or a task currently being executed). Therefore, the evaluation can identify one or more applications as a result of the association between the gaze input data for each user and the environmental context. It should be appreciated that different applications can be identified for each user based on different gaze input data (e.g., different users viewing different computing devices). Alternatively, the same application can be identified for each user based on the same gaze input data (e.g., different users viewing the same computing device).

[0321] If it is determined that there is no application associated with the gaze input data, the flow branches "NO" to operation 2810, where a default action is performed. For example, the gaze input data may have an associated predetermined application. In some other examples, the method 2800 may include determining whether the gaze input data has an associated default application, so that in some instances, as a result of the received gaze input data, no action may be performed. The method 2800 may terminate at operation 2810. Alternatively, the method 2800 may return from operation 2810 to operation 2802 to create a continuous feedback loop for the user of gaze input data and identifying the associated application.

[0322] However, if it is determined that there is an application associated with the received gaze input data, then the flow branches "yes" to operation 2812, where the application is determined based on the gaze input data. Fig. 27 , when the user 2702 looks at the word processing application 2706, it is determined that the user is looking at the word processing application 2706.

[0323] At determination 2814, it is determined whether there is a scrolling action associated with the gaze input data and the determined application. For example, determination 2614 may include evaluating the type of application determined from operation 2612. If the determined application requires reading, then when the user is looking at the bottom of the application, it may be desirable for the application to be scrolled down. Similarly, when the user is looking at the top, or the right side, or the left side of the application, it may be desirable for the application to be scrolled up, or to the right, or to the left. Additionally or alternatively, in some examples, the user may provide a user interface input (e.g., keyboard input, mouse input, trackpad input, etc.) to indicate whether the user desires to perform a scrolling action on the determined application, and / or in which direction the scrolling action is desired to be performed on the determined application.

[0324] If it is determined that there is no scrolling action associated with the gaze input data and the determined application, the process branches "no" to operation 2810, where a default action is performed. For example, the gaze input data may have an associated predetermined scrolling action. In some other examples, method 2800 may include determining whether the gaze input data and the determined application have an associated default scrolling action, so that in some instances, as a result of the received gaze input data and the determined application, no action may be performed. Method 2800 may terminate at operation 2810. Alternatively, method 2800 may return from operation 2810 to operation 2802 to create a continuous feedback loop for the user of gaze input data and identifying the associated application.

[0325] However, if it is determined that there is a scrolling action associated with the received gaze input data, then the flow branches "yes" to operation 2812, where the one or more computing devices are adapted to perform a scrolling action on the determined application. Fig. 27 , when the user 2702 looks at the bottom of the word processing application 2706, it is determined that the user wants to scroll down the word processing application 2706. Therefore, a scrolling operation is performed on the word processing application 2706 displayed on the computing device 2704.

[0326] Fig.29 An example system 2900 for device gaze tracking according to aspects described herein is shown. System 2900 includes multiple users 2902 and multiple computing devices, such as a first computing device 2904 and a second computing device 2906. The first computing device 2904 includes one or more elements 2908 displayed thereon, such as a shiny element 2908a and an attractive element 2908b. The second computing device 2904 includes one or more elements 2910 displayed thereon, such as an ignored element 2910a. System 2900 also includes a network 2912 that can be configured to communicate information between the first computing device 2904 and the second computing device 2906. The computing device 2304 can communicate with the computing device 2304 as previously described herein. Figure 1 Devices 103 to 108 are similar to those discussed above. In addition, network 2912 may be similar to that discussed previously herein. Figure 1 The network 110 discussed is similar.

[0327] The mechanism described herein provides the ability to aggregate useful metadata about the location that one or more users (e.g., user 2902) are looking at on one or more computing devices (e.g., device 2904, device 2906). For example, computing device 2904, computing device 2906 may receive gaze data corresponding to the location that user 2302 is looking at relative to the display of computing device 2904, computing device 2906. The display may include multiple elements (e.g., element 2908, element 2910) located thereon. Based on the received gaze data, metadata may be determined based on multiple elements. For example, an element that attracts the first attention of multiple users may be classified as a shining element (e.g., element 2908a). Elements that are ignored by most users may be classified as ignored elements (e.g., element 2910a). Elements that are most noticed by users may be classified as attractive elements (e.g., 2908b). In addition, the order in which elements are noticed by users may be identified as a pattern.

[0328] The determined metadata may be useful in organizing a display or presentation. For example, a user may change which elements are displayed in a presentation based on the amount of engagement the elements have from users (e.g., audience members of the presentation). If an element is ignored, the element may be moved to another location in the display or presentation; alternatively, the ignored element may be removed from the presentation. Additionally or alternatively, the arrangement of elements in the presentation may be modified based on the received mode metadata. For example, if an element is "shiny" and not meant to attract first attention, the "shiny" element may be moved to the back in the presentation.

[0329] In some examples, the determined metadata from system 2900 may be useful in marketing presentations. For example, it may be useful to track which elements are being noticed by users in order to customize the elements advertised to those users. For example, if the shiny element for one or more users is an item of clothing, then the additional elements that may be presented to the users are also items of clothing. Conversely, if the ignored element for one or more users is an item of clothing, then the elements presented to the users may no longer be items of clothing, or may be presented with less frequency.

[0330] One of ordinary skill in the art will appreciate that the elements discussed above with respect to system 2900 may be images, videos, animations, or any other form of graphics that may be displayed on a computing device.

[0331] Fig.30 An overview of an example method 3000 for processing gaze input data to perform an action that affects the behavior of a computing device is shown. According to some examples, aspects of the method 3000 are performed by a device, such as described above with respect to Figure 1Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0332] Method 3000 begins at operation 3002, where one or more computing devices are identified. For example, a user may use the above description of Figure 1 Any communication method discussed herein is used to link one or more devices (e.g., device 103 to device 108 and / or device 2904, device 2906). The device may be identified by a previous link association (e.g., indicated in a device profile or a shared profile). Alternatively, one or more devices may be identified based on user login information for different devices (e.g., each device with the same user login may be linked). In another aspect, one or more devices may be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity may be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical properties around the device (e.g., based on the device camera feed if the user has given the device permission to use the camera for this purpose). In yet another example, the user may then manually select one or more devices identified by method 3000 to identify the device at operation 3002. Additionally or alternatively, the network (e.g., network 2912) may be configured to automatically identify one or more devices connected to the network. In yet another example, a network (eg, network 2912 ) may be configured to detect computing devices within a specified geographic proximity.

[0333] At operation 3002, one or more users are identified. One or more users can be identified by one or more computing devices (e.g., devices 103 to 106 and / or devices 2904, 2906). Specifically, one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify one or more users (e.g., user 2902). The visual data can be processed using the mechanisms described herein to perform facial identification on one or more users if one or more users have provided permission to do so. For example, one or more computing devices can create a grid on the face of each user in one or more users to identify facial features, such as nose position, mouth position, cheek position, hair position, eye position, and / or eyelid position.

[0334] Additionally or alternatively, at operation 3004, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, opening an application, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged into, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged into, the user is thereby identified. Additionally or alternatively, at operation 3004, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0335] Additionally or alternatively, at operation 3004, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0336] At operation 3006, the one or more elements may be displayed on one or more computing devices. The one or more elements may be images, videos, animations, or any other form of graphics that may be displayed on a computing device. Examples of the one or more elements may be displayed on the Fig.29 is found (e.g., element 2908a, element 2908b, element 2910a).

[0337] At operation 3008, gaze input data is received from one or more computing devices (e.g., computing devices 103 to 108, and / or computing devices 2904, 2906) corresponding to one or more users (e.g., user 2902). Once one or more users are identified at 3004, method 3000 can monitor the orientation of the user's eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can be provided to a multi-device gaze tracking system (e.g., as described above with respect to Fig.29 The system 2900 discussed provides an indication of where a user may be looking relative to a display screen (e.g., a display screen of device 2904 or device 2906) and / or what element the user may be looking at on the display screen (e.g., element 2908a, element 2908b, element 2910a).

[0338] Still referring to operation 3008, one or more computing devices (e.g., computing devices 103 to 108 and / or computing devices 2904, computing devices 2906) can receive gaze input data from multiple users (e.g., the computing device can track the orientation of the eyes of multiple users and receive gaze data from them). Specifically, one or more computing devices can track the device that each user is looking at (e.g., computing devices 103 to 108 and / or computing devices 2904, computing devices 2906), and even further can determine what each user is looking at on the device (e.g., an application or some other element being displayed on one or more computing devices). The gaze input data can be received in real time (e.g., providing a continuous feedback stream about what multiple users are looking at). Alternatively, the gaze input data can be received periodically (e.g., at regular or irregular time intervals that can be specified by the user).

[0339] Further, referring to operation 3008, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data was first received). Typically, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future, and / or to determine useful metadata corresponding to the gaze data).

[0340] At operation 3010, metadata corresponding to one or more elements is identified based on the gaze input data. For example, the gaze input data can be evaluated to generate a set of user signals, which can be processed in view of metadata categories. Examples of metadata categories include: shiny, ignored, attractive, and pattern. Shiny metadata refers to an element that attracts the first attention of one or more users. It should be recognized that among multiple users, not all users may view the same element first. Therefore, for different users, there may be different elements that are classified as shiny. Ultimately, the element classified as shiny may be the element that most users focus on first.

[0341] Ignored metadata refers to elements that are ignored (e.g., not looked at by the user, last looked at by the user, or looked at for a relatively short period of time). It should be appreciated that among multiple users, not all users may ignore the same elements. Therefore, for different users, there may be different elements that are classified as ignored. Ultimately, the elements classified as ignored may be the elements that are ignored by most users.

[0342] Attractive metadata refers to elements that attract the most attention (e.g., are looked at by most users, or looked at for a relatively long period of time). It should be recognized that among multiple users, not all users find the same elements attractive. Therefore, for different users, there may be different elements that are classified as attractive. Ultimately, the elements classified as attractive may be the elements that most users find attractive.

[0343] Pattern metadata refers to the order in which elements are focused on by a user (e.g., if ignored elements are included in a pattern, the shiny element may be first in the pattern, the attractive element may be second in the pattern, and the ignored element may be third in the pattern). It should be recognized that among multiple users, not all users view the same elements. Therefore, there may be different patterns determined for different users based on the gaze data corresponding to each user. Patterns from multiple users may be aggregated to determine a dominant or common pattern (e.g., a pattern followed by an average user or a majority of users when viewing multiple elements).

[0344] At operation 3012, one or more computing devices may be adapted to change their display based on the metadata identified in operation 3010. For example, the determined metadata may be useful in organizing a display or presentation. A user may change which elements are displayed in a presentation based on the amount of engagement that the element has from users (e.g., audience members of the presentation). If an element is ignored, the element may be moved to another location in the display or presentation; alternatively, the ignored element may be removed from the presentation, or modified to a new element. Additionally or alternatively, the arrangement of elements in the presentation may be modified based on the received mode metadata. For example, if an element is "shiny" and is not meant to attract first attention, the "shiny" element may be moved to the back in the presentation. Additionally or alternatively, "attractive" elements may be presented at moments when high user engagement is expected.

[0345] Still referring to operation 3012, and in other examples, the metadata identified in operation 3010 may be useful in marketing. For example, it may be useful to track which elements are being noticed by users in order to customize the elements advertised to those users. For example, if the shiny element for one or more users is an item of clothing, then an additional element may be presented to the user via a display on one or more computing devices that is also an item of clothing. Conversely, if the ignored element for one or more users is an item of clothing, then the element presented to the user via a display on one or more computing devices may no longer be an item of clothing, or may be presented with less frequency.

[0346] Fig.31 An example grid for gaze data collection according to aspects described herein is shown. 3100. Typically, an eye tracker or sensor (e.g., camera) is located at the top edge of a device. When the user's gaze moves away from the sensor to the bottom edge of the device, occlusion may occur due to the user closing their eyelids. Such occlusion can be prevented by using gaze correction as a pre-processing step on the gaze data received via the sensor.

[0347] Additionally, as the angular distance from the sensor increases, the ability of the sensor to track the user's gaze may diminish due to, for example, poor angular resolution. Using a polar coordinate system with sensor 3102 (e.g., camera) at its center allows for the creation of a polar grid (e.g., grid 3100) that controls data density. As the user moves to the outer ring (e.g., moves away from the sensor collecting gaze tracking data), data can be collected more densely.

[0348] Fig.32An example of gaze calibration 3200 according to various aspects described herein is shown. Eye characteristics may vary across multiple users. For example, different users may have eyes that include different pupil axis angles (kapp angles), diopters, or general eye abnormalities. Therefore, in order to use the methods and systems outlined herein throughout the present disclosure, user-specific calibration may be performed to personalize the device (e.g., with respect to Figure 1 Gaze tracking on the discussed devices 103 to 108).

[0349] Fig.32 The gaze calibration example 3200 relies on Delaunay triangulation. Specifically, a set of predicted gaze points can be mapped against reference fact gaze points. A Delaunay triangulation grid can be constructed to map each triangle formed by the predicted gaze points to a corresponding triangle formed by the reference fact gaze points. Any differences between the predicted gaze points and the reference fact gaze points can be stored as errors and used to calibrate gaze data received by a computing device according to any example disclosed herein. Alternatively, conventional gridding methods can be recognized by a person of ordinary skill in the art and substituted for Delaunay triangulation for gaze calibration according to the examples disclosed herein.

[0350] Fig.33 An overview of an example method 3300 for processing gaze input data to perform an action that affects the behavior of a computing device is shown. According to some examples, aspects of the method 3300 are performed by a device, such as described above with respect to Figure 1 Computing device 103, computing device 104, peripheral device 106, or peripheral device 108 are discussed.

[0351] Method 3300 begins at operation 3302, where one or more computing devices are identified. For example, a user may use the above description of Figure 1The method 3300 may be used to link one or more devices (e.g., device 103 to device 108) by any communication method discussed. The device may be identified by a previous link association (e.g., indicated in a device profile or a shared profile). Alternatively, one or more devices may be identified based on user login information for different devices (e.g., each device with the same user login may be linked). In another aspect, one or more devices may be identified based on network connection (e.g., devices linked on the same network) or based on device proximity. Device proximity may be determined based on direct device communication (e.g., via RF or Bluetooth) or via determining similar physical properties around the device (e.g., based on the device camera feed if the user has given the device permission to use the camera for this purpose). In another example, the user may then manually select one or more devices to be identified by method 3300 to identify the device at operation 3302. Additionally or alternatively, the network may be configured to automatically identify one or more devices connected to the network. In another example, the network may be configured to detect computing devices within a specified geographic proximity.

[0352] At operation 3304, one or more users are identified. The one or more users can be identified by one or more computing devices (e.g., devices 103 to 108). Specifically, the one or more computing devices can receive visual data from a sensor (e.g., a camera) to identify the one or more users. The visual data can be processed using the mechanisms described herein to perform facial identification on the one or more users if the one or more users have provided permission to do so. For example, the one or more computing devices can create a grid on the face of each of the one or more users to identify facial features, such as nose location, mouth location, cheek location, hair location, eye location, and / or eyelid location.

[0353] Additionally or alternatively, at operation 3304, one or more users may be identified by participating in specific software (e.g., joining a call, joining a video call, joining a chat, opening an application, etc.). In addition, some users may be identified by logging into one or more computing devices. For example, a user may be the owner of a computing device, and the computing device may be linked to the user (e.g., via a password, a biometric entry, etc.). Therefore, when a computing device is logged in, the user is thereby identified. Similarly, a user may be identified by logging into a specific application (e.g., via a password, a biometric entry, etc.). Therefore, when a specific application is logged in, the user is thereby identified. Additionally or alternatively, at operation 3304, one or more users may be identified using a radio frequency identification tag (RFID), an ID badge, a barcode, a QR code, or some other identification method that can identify a user via some technical interface.

[0354] Additionally or alternatively, at operation 3304, one or more users may be identified as being present within the proximity of the computing device. In some examples, only specific elements of one or more users (e.g., eyes, face, body, hands, etc.) may be identified or recognized. In other examples, at least a portion of one or more users may be identified or recognized. For example, the system disclosed herein may not necessarily identify one or more users as specific individuals (e.g., individuals with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more users as being within the proximity of the computing device, so that one or more users may be tracked and / or monitored by the computing device. Similarly, the system disclosed herein may not necessarily identify one or more features of interest on a user as specific features of interest (e.g., features of interest with paired unique IDs for authentication or other purposes); instead, the system disclosed herein may only identify one or more features of interest (e.g., eyes, face, body, hands, etc.) as being within the proximity of the computing device, so that the features of interest may be tracked and / or monitored by the computing device.

[0355] At operation 3306, a sensor (e.g., a camera) is calibrated on each of the one or more devices (e.g., devices 103 to 108) to receive gaze input data corresponding to one or more users. The sensor may be based on the information provided herein. Fig.32 The present invention can be calibrated according to the various aspects disclosed herein. For example, a set of predicted fixation points can be mapped against the reference fact fixation points. A Delaunay triangulation grid can be constructed to map each triangle formed by the predicted fixation points to the corresponding triangle formed by the reference fact fixation points. Any difference between the predicted fixation points and the reference fact fixation points can be stored as errors and used to calibrate the fixation data received by the computing device. Alternatively, conventional gridding methods can be recognized by a person of ordinary skill in the art and substituted for Delaunay triangulation for fixation calibration according to the examples disclosed herein.

[0356] At operation 3308, gaze input data is received from one or more computing devices (e.g., computing devices 103 to 108) corresponding to one or more users. Once one or more users are identified at 3304, method 3300 can monitor the orientation of the user's eyes to determine their gaze, thereby receiving gaze input data. Such gaze input data can provide a multi-device gaze tracking system and / or a single-device gaze tracking system (such as any of those previously disclosed herein) with an indication of where the user may be looking relative to a display screen (e.g., a display screen of any of devices 103 to 108).

[0357] Still referring to operation 3308, one or more computing devices (e.g., computing devices 103 to 108) may receive gaze input data from multiple users (e.g., the computing devices may track the orientation of the eyes of multiple users and receive gaze data therefrom). Specifically, one or more computing devices may track the device (e.g., computing devices 103 to 108) that each user is looking at, and may even further determine what each user is looking at on the device (e.g., an application or some other element being displayed on one or more computing devices, such as a password or login mode). The gaze input data may be received in real time (e.g., providing a continuous stream of feedback about what multiple users are looking at). Alternatively, the gaze input data may be received periodically (e.g., at regular or irregular time intervals that may be specified by the user).

[0358] Further, referring to operation 3308, the gaze data may be stored (e.g., in the gaze tracking data store 116 or another form of memory). In some examples, only the most recent gaze data is stored, such that as gaze data is received, older gaze data is overwritten by the new gaze data (e.g., in memory). Alternatively, in some examples, gaze data is stored from a specified duration (e.g., the last hour, last day, last week, last month, last year, or since gaze data was first received). In general, such implementations allow for a history of gaze data from one or more users to be reviewed for further analysis (e.g., to infer or predict data that may be collected in the future).

[0359] At operation 3310, the gaze input data may be preprocessed using gaze correction. Fig.31 As discussed, when the user's gaze moves away from the location of the sensor (e.g., camera), occlusion may occur due to, for example, closed eyelids. Therefore, operation 3310 can determine when the user's gaze is directed to a location at a predetermined distance from the sensor so that occlusion of the gaze can be corrected.

[0360] Still referring to operation 3310, a radial grid may be used to control the data density for data collection. Fig.31 , using a polar coordinate system, the mechanism disclosed herein can receive gaze input data based on the position on the polar coordinate system. Such an implementation allows for more dense data collection at locations relatively far from the sensor. This is beneficial for preventing poor angular resolution that may occur.

[0361] At determination 3312, it is determined whether there is an action associated with the gaze input data. For example, determination 3312 may include evaluating the received gaze input data to generate a user signal group that may be processed in view of an environmental context (e.g., an application currently running on the device or a task currently being executed). Thus, the evaluation may identify an application or task as a result of an association between the gaze input data and the contextual data.

[0362] In some examples, at determination 3312, it is determined for each user whether there is an action associated with the gaze input data corresponding to the user. For example, determination 3312 may include evaluating the received gaze input data to generate one or more groups of user signals, wherein each user signal corresponds to one of the multiple users. The user signal can be processed in view of the environmental context (e.g., an application currently running on the device or a task currently being executed). Therefore, the evaluation can identify one or more actions as a result of the association between the gaze input data and the environmental context for each user. It should be appreciated that different actions can be identified for each user based on different gaze input data (e.g., different users viewing different computing devices). Alternatively, the same action can be identified for each user based on the same gaze input data (e.g., different users viewing the same computing device).

[0363] If it is determined that there is no application associated with the gaze input data, the flow branches "NO" to operation 3314, where a default action is performed. For example, the gaze input data may have an associated predetermined action. In some other examples, the method 3300 may include determining whether the gaze input data has an associated default action, so that in some instances, as a result of the received gaze input data, no action may be performed. The method 3300 may terminate at operation 3314. Alternatively, the method 3300 may return from operation 3314 to operation 3302 to create a continuous feedback loop of receiving gaze input data and executing commands based on the gaze input data.

[0364] However, if it is determined that there is a gaze command associated with the received gaze input data, then flow branches "yes" to operation 3316 where an action is determined based on the gaze input data. Examples of such actions may be found in this disclosure.

[0365] The process proceeds to operation 3318, where one or more computing devices are adapted to perform the determined action. In some examples, one or more computing devices may be adapted to perform the determined action by the computing device executing method 3300. In another example, an indication of the determined action may be provided to another computing device. For example, various aspects of method 3300 may be performed by a peripheral device so that operation 3318 includes providing input to an associated computing device. As another example, operation 3318 may include using an application programming interface (API) call to perform the determined action (e.g., transferring an application from a first computing device to a second computing device). Method 3300 may terminate at operation 3318. Alternatively, method 3300 may return to operation 3302 from operation 3318 to create a continuous feedback loop that receives gaze input data and adapts one or more computing devices to perform associated actions.

[0366] Figure 34 to Figure 37 and the associated descriptions provide a discussion of various operating environments in which aspects of the present disclosure may be implemented. Figures 6 to 9 The devices and systems illustrated and discussed are for purposes of example and explanation and are not limiting of the wide variety of computing device configurations that may be used to implement the various aspects of the present disclosure described herein.

[0367] Fig.34 3400 is a block diagram showing physical components (eg, hardware) of a computing device 3400 that can be used to implement various aspects of the present disclosure. The computing device components described below may be suitable for the computing devices described above, including Figure 1 102, device 103, device 104, device 106, and / or device 108. In a basic configuration, computing device 3400 may include at least one processing unit 3402 and system memory 3404. Depending on the configuration and type of computing device, system memory 3404 may include, but is not limited to, volatile storage devices (e.g., random access memory), non-volatile storage devices (e.g., read-only memory), flash memory, or any combination of these memories.

[0368] The system memory 3404 may include an operating system 3405 and one or more program modules 3406 suitable for running software applications 3420, such as one or more components supported by the system described herein. As an example, the system memory 3404 may store a gaze tracking component 3424 and a load balancer component 3426. The operating system 3405 may, for example, be suitable for controlling the operation of the computing device 3400.

[0369] Furthermore, embodiments of the present disclosure may be implemented in conjunction with a graphics library, other operating systems, or any other application, and are not limited to any particular application or system. Fig.343408. Computing device 3400 may have additional features or functionality. For example, computing device 3400 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data or other data. Fig.34 , illustrated by a removable storage device 3409 and a non-removable storage device 3410.

[0370] As described above, a number of program modules and data files can be stored in system memory 3404. When executed on processing unit 3402, program modules 3406 (e.g., applications 3420) can perform processes including, but not limited to, the various aspects described herein. Other program modules that can be used in accordance with various aspects of the present disclosure can include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-assisted applications, and the like.

[0371] In addition, embodiments of the present disclosure may be implemented on electronic circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or single chips containing electronic components or microprocessors. For example, embodiments of the present disclosure may be implemented via a system on a chip (SOC), wherein Fig.34 Each or many of the components shown can be integrated onto a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operated via the SOC, the functions described herein regarding the ability to switch protocols for the client can be operated via application-specific logic integrated with other components of the computing device 3400 on a single integrated circuit (chip). Embodiments of the present disclosure may also be implemented using other technologies capable of performing logical operations (e.g., AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present disclosure may be implemented in a general-purpose computer or in any other circuit or system.

[0372] The computing device 3400 may also have one or more input devices 3412, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or sliding input device, etc. Output device(s) 614 may also be included, such as a display, a speaker, a printer, etc. The above devices are examples, and other devices may be used. The computing device 3400 may include one or more communication connections 3416 that allow communication with other computing devices 650. Examples of suitable communication connections 3416 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuits; universal serial bus (USB), parallel and / or serial ports.

[0373] The term "computer-readable medium" as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures or program modules). System memory 3404, removable storage device 3409 and non-removable storage device 3410 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other product that can be used to store information and accessed by computing device 3400. Any such computer storage media may be part of computing device 3400. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0374] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in a manner that encodes information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0375] Fig.35A and Fig.35B A mobile computing device 3500 is shown, such as a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc., on which embodiments of the present disclosure may be implemented. In some aspects, the client may be a mobile computing device. Fig.35A , one aspect of a mobile computing device 3500 for implementing these aspects is shown. In a basic configuration, the mobile computing device 3500 is a handheld computer having both input elements and output elements. The mobile computing device 3500 typically includes a display 3505 and one or more input buttons 3510 that allow a user to enter information into the mobile computing device 3500. The display 3505 of the mobile computing device 3500 can also be used as an input device (e.g., a touch screen display).

[0376] When included, optional side input element 3515 allows further user input. Side input element 3515 can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, mobile computing device 3500 can incorporate more or fewer input elements. For example, in some embodiments, display 3505 may not be a touch screen.

[0377] In yet another alternative embodiment, the mobile computing device 3500 is a portable telephone system, such as a cellular phone. The mobile computing device 3500 may also include an optional keypad 3535. The optional keypad 3535 may be a physical keypad or a "soft" keypad generated on the touch screen display.

[0378] In various embodiments, output elements include a display 3505 for showing a graphical user interface (GUI), a visual indicator 3520 (e.g., a light emitting diode), and / or an audio transducer 3525 (e.g., a speaker). In some aspects, the mobile computing device 3500 incorporates a vibration transducer for providing tactile feedback to the user. In yet another aspect, the mobile computing device 3500 incorporates input and / or output ports for sending or receiving signals to or from external devices, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port).

[0379] Fig.35B 3500 can be incorporated into a system (e.g., architecture) 3502 to implement some aspects. In one embodiment, the system 3502 is implemented as a "smart phone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, the system 3502 is integrated into a computing device, such as an integrated personal digital assistant (PDA) and wireless phone.

[0380] One or more application programs 3566 may be loaded into the memory 3562 and run on or in association with the operating system 3564. Examples of application programs include a phone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 3502 also includes a non-volatile storage area 3568 within the memory 3562. The non-volatile storage area 3568 may be used to store persistent information that should not be lost if the system 3502 loses power. The application programs 3566 may use and store information in the non-volatile storage area 3568, such as emails or other messages used by the email application, and the like. A synchronization application (not shown) also resides on the system 3502 and is programmed to interact with a corresponding synchronization application residing on the host computer to keep the information stored in the non-volatile storage area 3568 synchronized with the corresponding information stored at the host computer. It should be appreciated that other applications may be loaded into the memory 3562 and executed on the mobile computing device 3500 described herein (eg, a signal identification component, a gaze tracker component, a shared computing component, etc.).

[0381] The system 3502 has a power source 3570, which can be implemented as one or more batteries. The power source 3570 can also include an external power source, such as an AC adapter or a powered docking cradle that replenishes or recharges the batteries.

[0382] The system 3502 may also include a radio interface layer 3572 that performs the function of sending and receiving radio frequency communications. The radio interface layer 3572 enables wireless connectivity between the system 3502 and the "outside world" via a communications carrier or service provider. Transmissions to and from the radio interface layer 3572 are performed under the control of the operating system 3564. In other words, communications received by the radio interface layer 3572 may be propagated to the application 3566 via the operating system 3564, and vice versa.

[0383] The visual indicator 3520 can be used to provide a visual notification, and / or the audio interface 3574 can be used to generate an audible notification via the audio transducer 3525. In the illustrated embodiment, the visual indicator 3520 is a light emitting diode (LED), and the audio transducer 3525 is a speaker. These devices can be directly coupled to the power supply 3570 so that when activated, even if the processor 3560 and other components may be turned off to save battery power, the device remains turned on until the duration indicated by the notification mechanism. The LED can be programmed to remain turned on indefinitely until the user takes action to indicate the power-on state of the device. The audio interface 3574 is used to provide an audible signal to the user and receive an audible signal from the user. For example, in addition to being coupled to the audio transducer 3525, the audio interface 3574 can also be coupled to a microphone to receive an audible input, such as to enable a telephone conversation. According to an embodiment of the present disclosure, the microphone can also be used as an audio sensor to enable control of the notification, as will be described below. The system 3502 may also include a video interface 3576 that enables operation of the onboard camera 3530 to record still images, video streams, and the like.

[0384] The mobile computing device 3500 implementing the system 3502 may have additional features or functionality. For example, the mobile computing device 3500 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data on the mobile computing device 3500. Fig.35B In the figure, non-volatile storage area 3568 is shown.

[0385] As described above, data / information generated or captured by the mobile computing device 3500 and stored via the system 3502 may be stored locally on the mobile computing device 3500, or the data may be stored on any number of storage media that may be accessed by the device via the radio interface layer 3572 or via a wired connection between the mobile computing device 3500 and a separate computing device associated with the mobile computing device 3500 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information may be accessed via the mobile computing device 3500 via the radio interface layer 3572 or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use in accordance with well-known data / information transmission and storage means, including email and collaborative data / information sharing systems.

[0386] Fig.36One aspect of the architecture of a system for processing data received at a computing system from a remote source, such as a personal computer 3604, a tablet computing device 3606, or a mobile computing device 3608, is shown, as described above. The content displayed at the server device 3602 can be stored in different communication channels or other storage types. For example, a directory service 3622, a web portal 3624, a mailbox service 3626, an instant messaging storage 3628, or a social networking site 3630 can be used to store various documents.

[0387] A gaze tracking component or engine 3620 may be employed by a client communicating with the server device 3602, and / or a load balancer component or engine 3621 may be employed by the server device 3602. The server device 3602 may provide data to and from client computing devices such as a personal computer 3604, a tablet computing device 3606, and / or a mobile computing device 3608 (e.g., a smart phone) over a network 3615. As an example, the above-described computer system may be embodied in a personal computer 3604, a tablet computing device 3606, and / or a mobile computing device 3608 (e.g., a smart phone). In addition to receiving graphics data that may be used for pre-processing at a graphics originating system or post-processing at a receiving computing system, any of these embodiments of the computing device may also obtain content from storage 3616.

[0388] Fig.37 An exemplary tablet computing device 3700 that can perform one or more aspects disclosed herein is shown. In addition, the various aspects and functions described herein can be operated on a distributed system (e.g., a cloud-based computing system), where application functions, memory, data storage and retrieval, and various processing functions can be remotely operated from each other through a distributed computing network (such as the Internet or an intranet). Various types of user interfaces and information can be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, user interfaces and various types of information can be displayed on a wall surface on which various types of user interfaces and information are projected and interacted with. Interaction with numerous computing systems that can be used to implement various embodiments of the present invention includes key input, touch screen input, voice or other audio input, where the associated computing device is equipped with a detection (e.g., camera) functional gesture input for capturing and interpreting user gestures to control the functionality of the computing device, etc.

[0389] For example, various aspects of the present disclosure are described above with reference to the block diagrams and / or operational diagrams of the methods, systems, and computer program products according to various aspects of the present disclosure. The functions / actions indicated in the blocks may not occur in the order shown in any flow chart. For example, depending on the functions / actions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order.

[0390] The description and explanation of one or more aspects provided in this application are not intended to limit or restrict the scope of the present disclosure in any way. The various aspects, examples and details provided in this application are considered to be sufficient to convey the claimed aspects of the present disclosure that possess and enable others to make and use. The disclosure claimed for protection should not be interpreted as being limited to any aspect, example or detail provided in this application. Whether it is shown and described in combination or individually, various features (structures and methods) are intended to be selectively included or omitted to produce an embodiment with a specific feature set. The description and explanation of the present application have been provided, and those skilled in the art can envision variations, modifications and alternative aspects that fall within the spirit of the broader aspects of the overall inventive concept embodied in this application, and these variations, modifications and alternative aspects do not depart from the broader scope of the disclosure claimed for protection.

Claims

1. A system comprising: at least one processor; as well as a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations comprising: identifying a plurality of computing devices; Identify one or more users; receiving gaze input data and load data from two or more computing devices of the plurality of computing devices; and Load balancing is performed between the plurality of devices, wherein the load balancing includes allocating one or more tasks from a first computing device among the plurality of computing devices to a second computing device among the plurality of computing devices based on the gaze input data.

2. The system of claim 1, wherein the load data is processed to determine the efficiency of each computing device. 3 . The system of claim 2 , wherein the efficiency of each computing device is calculated as a ratio between processor usage data and processor capability data. 4 . The system of claim 2 , wherein gaze data is processed to determine which of the one or more computing devices is a focus device, and wherein the assigned operation occurs if the focus device is greater than an efficiency threshold. 5 . The system of claim 1 , wherein the assigned operation is based on the gaze input data corresponding to the one or more users.

6. The system of claim 1, wherein the gaze input data is received by a sensor on each of the plurality of computing devices, and wherein the sensor is calibrated using a grid that includes predicted gaze points and ground truth gaze points.

7. A system comprising: at least one processor; as well as a memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operations, the set of operations comprising: identifying two or more computing devices; Identify one or more users; receiving gaze input data corresponding to the one or more users from the two or more computing devices; determining an action associated with a particular computing device based on the gaze input data; and The two or more computing devices are adapted based on the determined action.

8. The system of claim 7, wherein the one or more computing devices is a plurality of computing devices.

9. The system of claim 7, wherein: The receiving operation also includes receiving gesture input data corresponding to the one or more users from the one or more computing devices, and The determining operation also includes determining the action based on the gaze input data and the gesture input data.

10. The system of claim 7, wherein: The receiving operation also includes receiving user interface input data from the one or more computing devices, and The operation of determining further includes determining the action based on the gaze input data and the user interface input data.

11. The system of claim 7, wherein: The receiving operation also includes receiving context data from the one or more computing devices, and The operation of determining further includes determining the action based on the gaze input data and the context input data.

12. A method for processing gaze input data to control a computing device, the method comprising: identifying one or more computing devices; Identify one or more users; displaying one or more elements on the one or more computing devices; receiving gaze input data corresponding to the one or more users from the one or more computing devices; identifying metadata corresponding to the one or more elements based on the gaze input data; as well as The one or more computing devices are adapted based on the metadata to alter a display of the one or more computing devices. 13 . The method of claim 12 , wherein the identified metadata corresponds to which of the one or more elements was first viewed by the one or more users.

14. The method of claim 12, wherein the identified metadata corresponds to which of the one or more elements is of most interest to the one or more users.

15. The method of claim 12, wherein the metadata identified corresponds to a sequence in which the one or more elements are viewed by the one or more users.