Positioning graphical keyboard within graphical user interface

Abstract

A computing system is described that includes a memory that stores instructions that, when executed by one or more processors, cause the one or more processors to output a graphical user interface for display, and determining that the first portion of the graphical user interface includes content having at least a threshold probability that a user of the computing device 1) will view the content or 2) is viewing the content, where the graphical user interface further includes a second portion, and where the first portion and the second portion are non-overlapping portions of the graphical user interface. The instructions may also cause the one or more processors to output, for display by the display device, an updated graphical user interface including a graphical keyboard such that a majority of the graphical keyboard is visually superimposed on a second portion of the graphical user interface.

Description

Background Art

[0001] Some computing devices (e.g., mobile phones, tablet computers) may output a graphical keyboard for display as part of a graphical user interface for receiving text input (e.g., using a presence-sensitive input device and / or a display, such as a touch screen). The graphical keyboard may enable a user of the computing device to enter text (e.g., emails, text messages, documents, etc.). For example, a display device of the computing device may output a graphical keyboard at a presence-sensitive input device that enables a user to enter data by tapping or swiping on one or more keys of the graphical keyboard to indicate the keys displayed at the display device. Summary of the invention

[0002] The technology of the present disclosure can enable a computing device to display a graphical keyboard in an area of ​​a graphical user interface so as to reduce the amount of content being viewed or likely to be viewed that is blocked by the graphical keyboard. For example, the computing device may determine that at least a portion of the content included in the graphical user interface is being viewed by a user of the computing device or is likely to be viewed by the user of the computing device and / or at least a portion of the graphical user interface does not include content being viewed by the user or is likely to be viewed by the user. The computing device may determine to output the graphical keyboard within a portion of the graphical user interface that does not include content being viewed by the user or is likely to be viewed by the user. In various examples, if the portion of the graphical user interface that does not include content being viewed by the user or is likely to be viewed by the user is not sufficient to display the graphical keyboard, the computing device may resize the content (e.g., by switching the active application from operating in full-screen mode to operating in split-screen mode, by reducing the amount of the screen that is outputting content, etc.) to create a portion of the graphical user interface that does not include content and output the graphical keyboard within the newly created portion of the graphical user interface.

[0003] In one example, the present disclosure describes a method, the method comprising: outputting, by one or more processors of a computing device, a graphical user interface for display by a display device; and determining, by the one or more processors, that a first portion of the graphical user interface includes content having at least a threshold probability of being viewed by a user of the computing device, wherein the graphical user interface also includes a second portion, and wherein the first portion and the second portion are non-overlapping portions of the graphical user interface. The method may also include outputting, by the one or more processors, an updated graphical user interface including a graphical keyboard for display by the display device, such that a majority of the graphical keyboard is visually superimposed on the second portion of the graphical user interface.

[0004] In another example, a computing system includes a display device, one or more processors, and a memory storing instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to output a graphical user interface for display by the display device, and determine that a first portion of the graphical user interface includes content having at least a threshold probability of being viewed by a user of the computing device, wherein the graphical user interface also includes a second portion, and wherein the first portion and the second portion are non-overlapping portions of the graphical user interface. The instructions may also cause the one or more processors to output an updated graphical user interface including a graphical keyboard for display by the display device, such that a majority of the graphical keyboard is visually superimposed on the second portion of the graphical user interface.

[0005] In an additional example, a computer-readable storage medium is encoded with instructions that, when executed, cause one or more processors of a computing device to: output a graphical user interface for display by a display device of the computing device; and determine that a first portion of the graphical user interface includes content having at least a threshold probability of being viewed by a user of the computing device, wherein the graphical user interface also includes a second portion, wherein the first portion and the second portion are non-overlapping portions of the graphical user interface. The instructions may also cause the one or more processors to output an updated graphical user interface including a graphical keyboard for display by the display device, such that a majority of the graphical keyboard is visually superimposed on the second portion of the graphical user interface.

[0006] In another example, a computing system includes: means for outputting a graphical user interface for display; and means for determining that a first portion of the graphical user interface includes content having at least a threshold probability of being viewed by a user of the computing system, wherein the graphical user interface also includes a second portion, and wherein the first portion and the second portion are non-overlapping portions of the graphical user interface. The computing system may also include means for outputting an updated graphical user interface including a graphical keyboard for display, such that a majority of the graphical keyboard is visually superimposed on the second portion of the graphical user interface.

[0007] The details of one or more examples are set forth in the accompanying drawings and the description below.Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a conceptual diagram illustrating an example computing device configured to display a graphical keyboard, in accordance with one or more techniques of this disclosure.

[0009] Figure 2 is a block diagram illustrating an example computing device configured to determine user engagement with a graphical user interface of the device, in accordance with one or more techniques of this disclosure.

[0010] Figure 3A and Figure 3B is a conceptual diagram illustrating an example graphical user interface before and after executing a split-screen mode in accordance with one or more techniques of this disclosure.

[0011] Figure 4A and Figure 4B is a conceptual diagram illustrating an example graphical user interface with empty space in accordance with one or more techniques of this disclosure.

[0012] Figure 5 is a conceptual diagram illustrating an example computing device configured to track eye gaze of a user, in accordance with one or more techniques of this disclosure.

[0013] Fig. 6A and Figure 6B is a conceptual diagram illustrating an example graphical user interface including user interfaces for multiple applications, in accordance with one or more techniques of this disclosure.

[0014] Figure 7 is a flow diagram illustrating example operations for outputting a graphical keyboard within a graphical user interface, in accordance with one or more techniques of this disclosure. DETAILED DESCRIPTION

[0015] Figure 1 is a conceptual diagram illustrating an example computing device configured to display a graphical keyboard in accordance with one or more techniques of this disclosure. Figure 1 As shown, computing device 100 is a mobile computing device (e.g., a tablet computer). However, in other examples, computing device 100 may be a mobile phone, a laptop computer, a desktop computer, a gaming system, a media player, an e-book reader, a television platform, a car navigation system, a wearable computing device (e.g., a computerized watch, a computerized headset, computerized glasses, a computerized glove), or any other type of mobile or non-mobile computing device.

[0016] The computing device 100 includes a user interface device (UID) that can act as an input device for the computing device 100 and an output device for the computing device 100. The UID can be implemented using various technologies. For example, the UID can act as an input device using a presence-sensitive input screen (such as a resistive touch screen, a surface acoustic wave touch screen, a capacitive touch screen, a projected capacitive touch screen, a pressure-sensitive screen, an acoustic pulse recognition touch screen, or another presence-sensitive display technology). The UID can act as an output (e.g., display) device using any one or more display devices (such as a liquid crystal display (LCD), a dot matrix display, a light emitting diode (LED) display, a micro-LED, an organic light emitting diode (OLED) display, electronic ink, or a similar monochrome or color display capable of outputting visible information to a user of the computing device 100).

[0017] The UID of computing device 100 may include a presence-sensitive display that may receive tactile input from a user of computing device 100. The UID may receive one or more indications of tactile input by detecting one or more gestures from a user of computing device 100 (e.g., a user touching or pointing to one or more locations of the UID with a finger or stylus). The UID may present output to the user, for example, at a presence-sensitive display. The UID may present the output as a graphical user interface (e.g., graphical user interface 110), which may be associated with functionality provided by computing device 100. For example, the UID may present various user interfaces of components of a computing platform, operating system, application, or service (e.g., an electronic messaging application, an Internet browser application, a mobile operating system, etc.) executed at computing device 100 or accessible by the computing device. The user may interact with the respective user interfaces to cause computing device 100 to perform operations associated with the functionality.

[0018] like Figure 1 As shown, a graphical user interface (GUI) 110 includes content 101. Generally speaking, content 101 may include text, one or more pictures, videos, graphics, or any other content of GUI 110 displayed by a display device. More specifically, in various examples, content 101 may include content such as words of a text document, images of a website, streaming movies, graphics interchange format (GIF) images in an instant messaging application, control elements of a game, etc. Content 101 may be included in a graphical user interface (e.g., GUI 110) of an application executed in the foreground. Content 101 may be of interest to a user of computing device 100. For example, the user may be viewing the content, interacting with the content, and / or about to view the content. In some examples, content 101 may not be of interest to the user, or may be of less interest to the user than other content of GUI 110.

[0019] An application or operating system executed at the computing device 100 can selectively cause the UID to display a graphical keyboard. For example, if the user is watching a movie, the application and / or operating system of the computing device 100 can determine that text input is not required, and therefore does not cause the UID to output a graphical keyboard. In another example, since the cursor may be included in a text input field of a graphical user interface, the application and / or operating system can determine that text input is required and cause the UID to output a graphical keyboard. When the graphical keyboard is output, a portion of the content 101 or other content portions included in the GUI 110 may be obscured or visually moved so that the portion of the content is no longer displayed or otherwise visible to the user of the computing device 100. In instances where the content that is no longer visible is content that the user wishes to view, the user may be required to provide additional input to scroll or otherwise adjust the content to make the desired content visible. The additional input causes the computing device 100 to consume additional power due to detecting additional input, updating the content displayed by the UID, and potentially energizing various components of the computing device 100 (e.g., the UID) for a longer time than the computing device implementing the technology of the present disclosure would otherwise require.

[0020] According to one or more techniques of the present disclosure, computing device 100 may, via one or more processors, identify different portions of GUI 110 as including content of different relative importance to a user of computing device 100 and output graphical keyboard 112 for display within GUI 110, such that graphical keyboard 112 obscures as little of the content of interest as possible. For example, computing device 100 may determine that first portion 102 of GUI 110 includes content that is of interest to the user and is therefore relatively more important to the user, such that the content included within first portion 102 should not be obscured. Similarly, computing device 100 may determine that second portion 104 of GUI 110 includes content that is of no or less interest to the user and is therefore relatively less important to the user, such that the content included in second portion 104 may be obscured, moved, or otherwise adjusted to accommodate the inclusion of graphical keyboard 112 within GUI 110.

[0021] In determining what content of GUI 110 includes relatively more important or less important content, computing device 100 may determine the probability that the user of computing device 100 will view or is viewing various portions of the content of GUI 110. If the determined probability does not meet the threshold probability, computing device 100 may determine that the associated content portion is relatively less important. Conversely, if the determined probability meets the threshold probability, computing device 100 may determine that the associated content portion is relatively more important. Computing device 100 may determine the probability that the user is viewing or will view content by tracking the user's eye movement, based on active cursor focus (e.g., in an active text field), based on media content, by determining the amount of time each application from one or more currently executed applications has been visible on the display of the computing device, based on whether the user has viewed a portion of the content of GUI 110 within a threshold amount of time, based on whether promotional content is included in GUI 110, based on whether a portion of GUI 110 is blank (e.g., does not include text or graphical elements other than a background image, background color, etc.), or any combination thereof.

[0022] Computing device 100 may output graphical keyboard 112 such that content included in second portion 104 is obscured and a minimal amount of content included in first portion 102 is obscured. However, in some examples, second portion 104 may be too small (or non-existent) to output graphical keyboard 112 in a size or position that provides sufficient usability for a user of computing device 100. In such examples, computing device 100 may resize first portion 102 and second portion 104, or may also split GUI 110 into a first partition and a second partition, wherein the first partition includes content 101 and graphical keyboard 112 is included in the second partition.

[0023] By positioning the graphical keyboard 112 within the GUI 110 so as to reduce the amount of content of interest to the user that is obscured by the graphical keyboard 112, the techniques of this disclosure may improve the operational efficiency of the computing device 100. For example, the techniques described herein may reduce the amount of user input required, may reduce the amount of time that the UID of the computing device 100 is powered on, may reduce the amount of content rendering performed by the computing device 100, or any combination thereof. Improving such operational efficiency may reduce the amount of power consumed by the computing device 100 during operation.

[0024] Figure 2 is a block diagram illustrating an example computing device configured to determine user engagement with a graphical user interface of the device, in accordance with one or more techniques of this disclosure. Figure 2 The computing device 200 is Figure 1 The computing device 200 is only an example of a computing device 100. Figure 1A particular example of computing device 100, and many other examples of computing device 200 may be used in other instances and may include a subset of the components included in the example computing device 200 or may include Figure 2 Additional components not shown.

[0025] like Figure 2 As shown in the example of , the computing device 200 includes one or more processors 250, one or more input devices 252, one or more communication units 256, one or more output devices 258, one or more storage devices 270, and a user interface device 254. The one or more storage devices 270 of the computing device 200 can store an operating system 272, a UI manager module 276, an I / O module 278, application modules 274A to 274N, an interest content module 280, an eye gaze module 282, a content identification (ID) module 284, an application manager module 286, a machine learning (ML) module 288, a screen portion module 290, a priority module 292, and a graphical keyboard module 294. The communication channel 260 can interconnect each of the components 250, 252, 254, 256, 258, and 270 for inter-component communication (physically, communicatively, and / or operationally). In some examples, communication channel 260 may include a system bus, a network connection, one or more inter-process communication data structures, or any other method for communicating data between hardware and / or software.

[0026] One or more input devices 252 of computing device 200 can receive input. Examples of input are tactile input, audio input, and video input. In one example, one or more input devices 252 of computing device 200 include a mouse, a keyboard, a voice response system, a camera, a microphone, or any other type of device for detecting input from a human or a machine. In some examples, input device 252 can be a presence-sensitive input device, which can include a presence-sensitive screen, a touch-sensitive screen, etc.

[0027] One or more output devices 258 of computing device 200 may generate output. Examples of output are tactile output, audio output, and video output. In one example, output device 258 of computing device 200 includes a presence-sensitive display, a sound card, a video graphics adapter card, a speaker, a liquid crystal display (LCD), a mini LCD, a micro LCD, an organic light emitting diode (OLED), or any other type of device for generating output to a human or a machine.

[0028] The one or more communication units 256 of the computing device 200 can communicate with external devices via one or more networks by sending and / or receiving network signals on one or more networks. For example, the computing device 200 can use the communication unit 44 to send and / or receive radio signals on a radio network (such as a cellular radio network). Similarly, the communication unit 256 can send and / or receive satellite signals on a satellite network (such as a GPS network). Examples of the communication unit 256 include a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and / or receive information. Other examples of the communication unit 256 can include Bluetooth®, GPS, 3G, 4G, and Wi-Fi® radios found in mobile devices, and universal serial bus (USB) controllers.

[0029] In some examples, computing device 200 includes an I / O (input / output) module 278. I / O module 278 may receive information from UI manager module 276 and output such information to a device having output functionality (e.g., user interface device 254, communication unit, etc.). I / O module 278 may receive information from a device having input functionality (e.g., user interface device 254, communication unit, etc.). For example, I / O module 278 may receive information defining a GUI of computing device 200 from UI manager module 276 and one or more of application modules 274A to 274N, content of interest module 280, eye gaze module 282, content ID module 284, application manager module 286, ML module 288, screen partition module 290, priority module 292, and graphical keyboard module 294. Based on the received information, I / O module 278 may cause user interface device 254 to output a GUI for display. In some examples, I / O module 278 can determine an indication of user input performed at user interface device 254 and send such information to one or more of UI manager module 276, application modules 274A to 274N, content of interest module 280, eye gaze module 282, content ID module 284, application manager module 286, ML module 288, screen partitioning module 290, priority module 292, and / or graphical keyboard module 294.

[0030] In some examples, user interface device 254 of computing device 200 may include the functionality of input device 252 and / or output device 258. In some examples, a GUI may refer to a presence-sensitive display. Figure 2In an example, the user interface device 254 may include a presence-sensitive input device, such as a presence-sensitive screen, a touch-sensitive screen, and the like. In some examples, the presence-sensitive input device may detect objects at and / or near the presence-sensitive input device. As an example range, the presence-sensitive input device may detect an object, such as a finger or stylus, that is within 2 inches or less of the presence-sensitive screen. The presence-sensitive input device may determine the location (e.g., (x, y) coordinates) at which the presence-sensitive input device detects the object. In another example range, the presence-sensitive input device may detect an object that is 6 inches or less from the presence-sensitive input device, and other ranges are possible. The presence-sensitive input device may use capacitive, inductive, and / or optical recognition technology to determine the location of the screen selected by the user's finger. In some examples, the user interface device 254 uses tactile, audio, or video stimulation to provide output to the user, as described with respect to the output device 258. For example, the user interface device 254 may include a display device that presents a user interface, such as Figure 1 GUI 110. In some examples, a user can interact with a presence-sensitive input device displaying GUI 110 to select characters from a graphical keyboard.

[0031] The presence sensitive input device may include one or more touch sensors capable of detecting touch input from a user. Each touch sensor may include one or more electrical devices, such as a capacitive sensor or a resistive sensor, or may include one or more mechanical devices, such as a switch or a pressure plate. In one example, each touch sensor may include one or more simple capacitance sensors that detect changes in the capacitance of a material. Electrical hardware attached to each simple capacitance sensor may monitor changes in the capacitance of the corresponding simple capacitance sensor. The change in capacitance may be determined by the attached electrical hardware, and an indication may be output by the touch sensor. Other known sensor technologies that provide the ability to sense touch input may also be employed to receive touch input from a user.

[0032] Although shown as an integrated component of computing device 200, in some examples, user interface device 254 may be an external component that shares a data path for sending and / or receiving input and output with other components of computing device 200. For example, user interface device 254 may be a built-in component of computing device 200 that is located inside and physically connected to an external package of computing device 200 (e.g., a screen on a mobile phone). In another example, user interface device 254 may be an external component of computing device 200 that is located outside and physically separated from the package of computing device 200 (e.g., a monitor, projector, etc. that shares a wired and / or wireless data path with a tablet computer). In some examples, user interface device 254, when located outside and physically separated from the package of computing device 200, may collectively refer to two components: a presence-sensitive input device for receiving input and a display device for providing output.

[0033] One or more storage devices 270 within computing device 200 may store information for processing during operation of computing device 200. In some examples, storage device 270 is temporary storage, meaning that the primary purpose of storage device 270 is not long-term storage. Storage device 270 of computing device 200 may be configured as volatile memory for short-term storage of information, and thus the stored contents are not retained when the computer is shut down. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art.

[0034] In some examples, storage device 270 also includes one or more computer-readable storage media. Storage device 270 can be configured to store a larger amount of information than volatile memory. Storage device 270 can also be configured as a non-volatile memory space for long-term storage of information and retain information after power on / off cycles. Examples of non-volatile memory include magnetic hard disks, optical disks, floppy disks, flash memory, or the form of electrically programmable memory (EPROM) or electrically erasable and programmable (EEPROM) memory. Storage device 270 can store program instructions and / or data associated with operating system 272, UI manager module 276, I / O module 278, application modules 274A to 274N, content of interest module 280, eye gaze module 282, content ID module 284, application manager module 286, ML module 288, screen partitioning module 290, priority module 292, and graphical keyboard module 294.

[0035] The one or more processors 250 may implement functionality and / or execute instructions within the computing device 200. For example, the one or more processors 250 on the computing device 200 may receive and execute instructions stored by the storage device 270 that provide functionality for the operating system 272, the UI manager module 276, the I / O module 278, the application modules 274A through 274N, the content of interest module 280, the eye gaze module 282, the content ID module 284, the application manager module 286, the ML module 288, the screen partitioning module 290, the priority module 292, and the graphical keyboard module 294. These instructions executed by the one or more processors 250 may cause the computing device 200 to store and / or modify information within the storage device 270 during program execution. The one or more processors 250 may execute instructions of the operating system 272, the UI manager module 276, the I / O module 278, the application modules 274A to 274N, the content of interest module 280, the eye gaze module 282, the content ID module 284, the application manager module 286, the ML module 288, the screen partition module 290, the priority module 292, and the graphical keyboard module 294 to cause the user interface device 254 to display the GUI. That is, the operating system 272, the UI manager module 276, the I / O module 278, the application modules 274A to 274N, the content of interest module 280, the eye gaze module 282, the content ID module 284, the application manager module 286, the ML module 288, the screen partition module 290, the priority module 292, and the graphical keyboard module 294 may be operated by the one or more processors 250 to perform various functions described herein, including determining a portion of the GUI that includes content having at least a threshold probability of being viewed by a user of the computing device, and causing the user interface device 254 to present the GUI to the user at the user interface device 254. It will be appreciated that descriptions of computing device 200 or one or more components thereof (e.g., one or more modules in storage device 270) performing one or more operations of the present disclosure may include one or more processors 250 implementing functionality and / or executing instructions thereof.

[0036] The content of interest module 280 may include instructions executable by the one or more processors 250 to determine whether one or more portions of the GUI of the computing device 200 include content of interest to the user. That is, the content of interest module 280 may determine whether one or more portions of the GUI include content that satisfies at least a threshold probability that the user of the computing device 200 will view the content or is viewing the content. The content of interest module 280 may determine a first portion of the GUI, the first portion comprising one or more portions of the GUI that include content that has at least a threshold probability that the user of the computing device 200 will view the content or is viewing the content. The content of interest module 280 may also determine a second portion of the GUI, the second portion comprising one or more portions of the GUI that do not include content of interest to the user (e.g., do not satisfy the threshold probability that the user of the computing device 200 will view or is viewing the content). After determining the first portion and the second portion of the GUI, the content of interest module 280 may send information defining the first portion and the second portion of the GUI to the screen partition module 290 to determine where in the GUI to display the graphical keyboard. The content of interest module 280 may determine whether one or more portions of the GUI include content of interest based on output from one or more sub-modules including the eye gaze module 282 , the application manager module 286 , the content ID module 284 , and the ML module 288 .

[0037] In determining which portions of the GUI include content of interest to the user of computing device 200, interested content module 280 may use information determined by one or more of eye gaze module 282, content ID module 284, or application manager module 286. For example, eye gaze module 282 may determine what portion of the GUI the user of computing device 200 is looking at, content ID module 284 may determine various types of content included in the GUI, and application manager module 286 may determine various information about one or more applications (e.g., one or more application modules 274) executing at computing device 200.

[0038] The eye gaze module 282 may determine which parts of the GUI displayed by the user interface device 254 the user is looking at by, for example, tracking eye movements of the user. When tracking such eye movements, the eye gaze module 282 may determine one or more gaze points of the user on the GUI displayed by the user interface device 254. The eye gaze module 282 may determine the distance of the user's head or eyes from one or more cameras located on the computing device 200 (e.g., one or more cameras of the input device 252), and may determine the position of the user's head or eyes in 3D space relative to the one or more cameras.

[0039] In some examples, eye gaze module 282 may analyze one or more images captured by one or more cameras of computing device 200 to determine the distance between the one or more cameras and the user's eyes. As an example, assuming that the position of each of two or more different cameras is known, eye gaze module 282 may analyze the disparity angle between the images captured by the two or more different cameras. Using the disparity angle and the distance between the cameras, eye gaze module 282 determines the distance between one or more of the two or more cameras and the user's eyes. As another example, eye gaze module 282 may analyze images captured by a single infrared camera to determine the distortion of the images captured by the infrared camera to determine the distance between the infrared camera and the user's eyes.

[0040] Using this determined distance between one or more cameras and the user's eyes and the position of the user's eyes in the captured images, the eye gaze module 282 can place the user's eyes in three-dimensional space relative to the one or more cameras. That is, the eye gaze module 282 can determine the position of the user's eyes with reference to the computing device 200 (i.e., relative to the position of at least one camera of the computing device 200). The position of the user's eyes is a position defined in 3D space relative to at least one camera. For example, the 3D space may be spherical and have a center of mass corresponding to the position of the camera. In such examples, the position of the user's eyes can be defined in terms of (x, y, z) coordinates, where (0, 0, 0) is the position of the camera used as the center of mass of the sphere. Such coordinates can be referred to as being in a "camera-based coordinate system."

[0041] Using the position of the user's eyes and the movement of the pupil or other features of the eyes, the eye gaze module 282 can determine one or more eye gaze vectors. In some examples, the eye gaze module 282 can determine one or more eye gaze vectors using the pitch, roll, and / or yaw of the user's head about a particular axis. The eye gaze module 282 uses the one or more eye gaze vectors and the position of the user's eyes relative to one or more cameras to determine one or more gaze points of the user on the GUI displayed on the display of the computing device 200. The eye gaze module 282 may include instructions executable by one or more processors 250 for determining the number, time, and / or location of gaze points on the GUI. The eye gaze module 282 may output the number, time, and / or location of gaze points on the GUI to the content of interest module 280 to determine whether the portion of the GUI including the gaze points includes content of interest to the user.

[0042] In various examples, the eye gaze module 282 can apply a trained machine learning model (e.g., ML module 288) to the camera image to determine one or more gaze points of a user on a GUI displayed by a display device of the computing device 200. The machine learning model can be trained using images of a user with known gaze points on an equivalent GUI displayed on an equivalent computing device. The machine learning model can continuously learn based on user feedback and feedback provided by the eye gaze module 282. For example, the eye gaze module 282 can rank or adjust parameters of the machine learning model based on updated training data received since the last time the machine learning model was trained. After training, the ML module 288 can be trained to receive image data as input and determine one or more gaze points of a user on the GUI as output.

[0043] The content ID module 284 may include instructions executable by the one or more processors 250 for identifying the content of a GUI running on the computing device 200. For example, the content ID module 284 may determine whether the GUI includes promotional content, whether the GUI includes a video or an image, whether the GUI includes an active text field, and / or whether the GUI includes blank space. After determining the type of content in the GUI, the content ID module 284 may output data indicating the type of content and the location of the content in the GUI to the content of interest module 280.

[0044] The content ID module 284 may also determine a priority level for the content of the GUI and output the priority level to the content of interest module 280. The content ID module 284 may determine the priority level based on the determined type of content of interest included in the GUI. In various instances, the priority level may indicate the likelihood that the user is viewing or will view the content of interest. Content of interest with a higher priority level may be of high interest to the user, while content of interest with a lower priority level may be of least interest to the user and, therefore, may be identified as being obscured by the graphical keyboard before content of interest with a higher priority level. The content ID module 284 may output the priority level to the content of interest module 280.

[0045] In response to receiving data indicating the type of content from content ID module 284, interested content module 280 may determine whether the content is of interest to the user, that is, whether the content satisfies at least a threshold probability that the user will view or is viewing the content. For example, interested content module 280 may receive data indicating promoted content from content ID module 284, may determine that the promoted content is not of interest to the user, and may determine that the portion of the GUI occupied by the promoted content is part of the second portion.

[0046] The application manager module 286 may include instructions executable by the one or more processors 250 for determining usage information for each of one or more currently executed applications on the computing device 200. For example, the application manager module 286 may determine the amount of time that each application from the one or more currently executed applications has been executed. The application manager module 286 may also determine a corresponding amount of time that each of the one or more currently executed applications has been visible on a display device (e.g., user interface device 254) of the computing device 200. The application manager module 286 may also include instructions for determining a foreground application. The application manager module 286 may output data indicating the amount of time each application has been executed, the amount of time each application has been visible on the display device, and / or whether the application is a foreground application to the content of interest module 280.

[0047] The content of interest module 280 may determine whether one or more of the application GUIs included in the GUI of the computing device 200 include content of interest. After receiving data from the application manager module 286, the content of interest module 280 may determine whether the content of the one or more application GUIs included in the GUI of the computing device 200 meets a threshold probability that the user is viewing or will view the content based on the data from the application manager module 286. The content of interest module 280 may also determine that one or more portions of the GUI occupied by the one or more application GUIs are part of the first portion or the second portion. For example, the content of interest module 280 may determine that the foreground application includes content of interest. In some examples, the content of interest module 280 may determine that the application GUI that has been visible on the user interface device 254 for the longest amount of time includes content of interest.

[0048] The ML module 288 may include instructions for determining whether one or more portions of the GUI include content of interest to a user of the computing device 200. The ML module 288 may determine whether one or more portions of the GUI include content with at least a threshold probability that the user will view or is viewing the content. The ML module 288 may include one or more machine learning algorithms configured to accept one or more representations of the content of the GUI as input and output whether the content satisfies a threshold probability that the user is viewing or will view the content. Example machine learning techniques that may be employed include various learning styles, such as supervised learning, unsupervised learning, and semi-supervised learning. Example types of machine learning algorithms include Bayesian algorithms, clustering algorithms, decision tree algorithms, regularization algorithms, regression algorithms, instance-based algorithms, artificial neural network algorithms, deep learning algorithms, dimensionality reduction algorithms, and the like. Various examples of specific algorithms include Bayesian linear regression, boosted decision tree regression and neural network regression, back-propagation neural network, convolutional neural network (CNN), long short-term network (LSTM), Apriori algorithm, K-means clustering, k-nearest neighbor (kNN), learning vector quantization (LVQ), self-organizing map (SOM), locally weighted learning (LWL), ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net and least angle regression (LARS), principal component analysis (PCA) and principal component regression (PCR).

[0049] In some examples, one or more machine learning models of ML module 288 may be configured to accept as input an instruction for outputting a GUI, and determine as output from one or more machine learning models one or more portions of the GUI that include content having at least a threshold probability that a user is viewing or will view the content. In some examples, one or more machine learning models of ML module 288 may be configured to accept as input an instruction for outputting a GUI, and determine as output from one or more machine learning models one or more portions of the GUI that include content that does not satisfy at least a threshold probability that a user is viewing or will view the content. In some examples, one or more machine learning models may determine as outputs both one or more portions of the GUI that include content having at least a threshold probability and one or more portions of the GUI that include content that does not satisfy at least a threshold probability. One or more machine learning models of ML module 288 may be trained using training data stored in storage device 270. The training data may include multiple inputs to one or more machine learning models corresponding to known outputs.

[0050] In some examples, one or more machine learning models of the ML module 288 may be trained to accept as input the output of one or more of the eye gaze module 282, the content ID module 284, and / or the application manager module 286. The one or more machine learning models of the ML module 288 may be trained to determine as output the content of interest in the GUI of the computing device 200 based on the input received from the one or more modules 282, 284, and 286. For example, the ML module 288 may accept as input the number and location of gaze points from the eye gaze module 282, the data indicating the type of content from the content ID module 284, and / or the amount of time each of the one or more currently executed applications has been visible on the user interface device 254 from the application manager module 286. Based on the number and location of gaze points, the data indicating the type of content, and / or the amount of time each of the one or more currently executed applications has been visible on the user interface device 254, the ML module 288 may output a portion of the GUI containing the content of interest.

[0051] Using information from one or more of modules 282, 284, 286, and 288, content of interest module 280 may determine one or more sizes of one or more portions of the GUI that include content that has at least a threshold probability that the user of computing device 200 will view the content or is viewing the content (e.g., a first portion) and content that is not of interest to the user (e.g., a second portion), and whether the sizes meet one or more threshold sizes. In some examples, the second portion of the GUI may be too small to display a graphical keyboard without the graphical keyboard obscuring at least some of the content of interest in the first portion. That is, content of interest module 280 may determine that the size of the second portion does not meet a threshold size, where the threshold size is associated with a minimum-sized graphical keyboard.

[0052] In some instances where the threshold size is not met, the content of interest module 280 may determine whether the priority of the content of interest in the first portion that would be obscured by the graphical keyboard meets a lower priority threshold. In some examples, the content of interest module 280 may determine that enough content with a low enough relative importance is included in the first portion so that the graphical keyboard can be output without partitioning the screen. The content of interest module 280 may additionally or alternatively adjust the threshold probability to include less content in the first portion, and resize the first portion and the second portion based on the priority level in view of the updated threshold probability.

[0053] In response to determining that the second portion is too small to display the graphical keyboard and / or determining that the content that would be obscured by the keyboard is not of sufficiently low relative importance, the content of interest module 280 may determine whether the GUI should enter split-screen mode. To determine whether the GUI should enter split-screen mode, the content of interest module 280 may determine whether the content of interest in the first portion can be reduced without making the content of interest too difficult for the user to see. For example, the content of interest 280 may compare the adjusted first portion size to a minimum first portion size associated with the type of content included in the first portion. Content types (such as images, videos, advertisements, etc.) may have a smaller minimum size than other types of content (such as text, links, input fields, etc.).

[0054] In response to determining that the content of interest in the first portion can be reduced without making the content of interest too difficult to see, the content of interest module 280 can instruct the screen division module 290 to perform the split screen mode. In response to determining that the content of interest in the first portion cannot be reduced without making the content of interest too difficult to see, the content of interest module 280 can redetermine the first portion and the second portion based on the priority level output from the content ID module 284 so that the second portion is large enough to display the graphical keyboard.

[0055] The screen splitting module 290 may include instructions executable by one or more processors 250 for executing a split-screen mode on the computing device 200 and for determining where to output a graphical keyboard within the GUI for display. The screen splitting module 290 may receive information indicating a first portion of the GUI containing content that is believed to be of interest to the user and a second portion of the GUI containing content that is believed to be of no interest to the user and instructions on whether to execute the split-screen mode from the content of interest module 280. For example, as described above, the content of interest module 280 may determine that the second portion is too small to display the graphical keyboard without obscuring at least some of the content of interest in the first portion, and send instructions to the screen splitting module 290 to execute the split-screen mode on the computing device 200 to create more space to display the graphical keyboard. To execute the split-screen mode, the screen splitting module 290 may split the GUI into at least a first partition and a second partition, wherein the first partition is smaller than the first portion of the GUI before the split, and the second partition is larger than the second portion of the GUI before the split. The screen splitting module 290 may resize the content for display in the first partition and the second partition. For example, the screen partition module 290 may shrink the content of the first portion so that the content fits into the first partition.The screen partition module may determine where in the second portion and / or second partition of the GUI to output a graphical keyboard for display.

[0056] The graphical keyboard module 294 may include instructions executable by one or more processors 250 to output a graphical keyboard in a GUI of the computing device 200 for display by a display device. The graphical keyboard module 294 may be a standalone application, service, or module executed at the computing device 200, and in other examples, the graphical keyboard module 294 may be a subcomponent thereof. For example, the graphical keyboard module 294 may be integrated into a chat or messaging application executed at the computing device 200, while in other examples, the graphical keyboard module 294 may be a standalone application or subroutine called by an application or operating platform of the computing device 200 at any time when the application or operating platform requires graphical keyboard input functionality. In some examples, the computing device 200 may download and install the graphical keyboard module 294 from an application distribution platform (e.g., via the Internet) such as an application repository of a service provider. In other examples, the graphical keyboard module 294 may be preloaded during the production of the computing device 200. In various examples, the graphical keyboard module 294 may include instructions for determining the size of the user input device and the corresponding minimum threshold size of the graphical keyboard to be displayed.

[0057] Application modules 274A-274N may include instructions executable by one or more processors 250 to cause one or more applications to run on computing device 200. For example, application modules 274A-274N may include instructions for one or more of a messaging application, a browser application, a camera application, a music streaming application, or any other application for execution on computing device 200.

[0058] Figure 3A and Figure 3B is a conceptual diagram illustrating an example graphical user interface before and after executing a split-screen mode according to one or more techniques of this disclosure. Figure 2 The computing device 200 is described Figure 3A and Figure 3B .exist Figure 3A In the example of FIG. 3 , GUI 310 includes content 301 that applies GUI 316. Content 301 may include media 320, such as an image or video.

[0059] In many computing devices (e.g., computing device 200), a graphical keyboard may not be displayed as part of GUI 310 until computing device 200 determines that a user may wish to enter text at computing device 200. For example, a user may reply to a comment on a website in a browser application, or a user may tap an incoming notification for a messaging application. However, once computing device 200 outputs the graphical keyboard for display, the graphical keyboard may obscure content of interest to the user of computing device 200 (e.g., content of interest in first portion 302 of GUI 310). Computing device 200 may determine whether one or more portions of GUI 310 include content of interest to the user, and output the graphical keyboard for display so that it does not obscure (or obscures as little as possible) the content of interest to the user.

[0060] Interested content module 280 of computing device 200 may determine whether one or more portions of GUI 310 include content of interest to the user, and determine first portion 302 and second portion 304 of GUI 310 from the one or more portions. Interested content module 280 may determine that first portion 302 includes each of the one or more portions of GUI 310 that include content having at least a threshold probability of being viewed by a user of the computing device. That is, first portion 302 includes each of the one or more portions of GUI 310 that include content having at least a threshold probability that the user of computing device 200 1) will view the content or 2) is viewing the content. Interested content module 280 may determine that second portion 304 includes each of the one or more portions of GUI 310 that do not include content that meets at least the threshold probability. First portion 302 and second portion 304 may be non-overlapping portions of the GUI.

[0061] Interested content module 280 may determine whether one or more portions of GUI 310 include content of interest to the user based on data output by content ID module 284. Content ID module 284 may determine the type of content 301 and output data indicating the type of content 301 to interested content module 280. For example, content ID module 284 may determine that a video or an image is included in the graphical user interface. Content ID module 284 may examine tags of content 301 within application GUI 316 of GUI 310. One or more of the tags may indicate that content 301 within application GUI 316 includes an image or video, and may also indicate a location in application GUI 316 where such content may be found. Using the location data, content ID module 284 may determine the location of the image or video in GUI 310 and the portion of GUI 310 that contains the image or video. In some examples, content ID module 284 may determine that content 301 includes an active text field (e.g., active text field 324), where the active text field has cursor focus. The content ID module 284 may output data indicating the type of content 301 and the location of the content 301 in the GUI to the content of interest module 280 .

[0062] In response to receiving the type and location of content 301 from content ID module 284, content of interest module 280 may determine whether the content is or is not content of interest. That is, content of interest module 284 may determine whether the content meets a threshold probability that a user will view or is viewing the content. For example, content of interest module 280 may determine that an image or video meets a threshold probability that a user will view or is viewing the content. When visual media is visible in GUI 310, the user may be interested in it. Therefore, data in the memory of computing device 200 may indicate that the image or video meets the threshold probability. Content of interest module 280 may determine that one or more portions of GUI 310 containing images and / or videos are included in first portion 302. Content of interest module 280 may send data indicating first portion 302 and second portion 304 to screen partition module 290 to determine where to output graphical keyboard 312 for display.

[0063] exist Figure 3AIn the example of , content ID module 284 may determine that content 301 includes images, videos, text, and graphic buttons in application GUI 316 of GUI 310. Content ID module 284 may output data indicating content 301 and its location in GUI 310 to content of interest module 280. Content of interest module 280 may determine that the images, videos, text, and number buttons are content of interest, and determine first portion 302 from the portion of GUI 310 that includes the images, videos, text, and number buttons. Content of interest module 280 may determine that the remaining space in GUI 310 is second portion 304.

[0064] The content of interest module 280 may also determine that the active text field meets the threshold probability that the user will view or is viewing the content. The active text field 324 may display the typed text being submitted to the active text field 324, which may allow the user to actively confirm the correct selection of characters when the user types on the graphical keyboard (e.g., graphical keyboard 312). Therefore, the data in the memory of the computing device 200 may indicate that the active text field meets the threshold probability. The content of interest module 280 may determine a third portion 306 of the GUI 310 that includes the active text field 324. In some examples, the first portion 302 may include the third portion 306. The content of interest module 280 may determine a fourth portion 308 of the GUI 310 that is near or adjacent to the active text field 324, wherein the second portion 302 includes the fourth portion 308. In this way, the content of interest module 280 may identify a portion of the GUI 310 that is near or adjacent to the active text field, which portion may be obscured by the graphical keyboard 312 without obscuring the content of interest to the user.

[0065] In some examples, active text field 324 may not be output for display in GUI 310 until computing device 200 determines that a user may wish to enter text at computing device 200. In these instances, active text field 324 and graphical keyboard 312 may be output for display at substantially the same time. Interested content module 280 may determine the size and location of third portion 306 that will contain active text field 324 based on application data in one or more applications 274 that contain instructions for the behavior of the active text field.

[0066] exist Figure 3AIn the example of FIG. 3 , the content of interest module 280 can determine that the second portion 304 of the GUI 310 is too small to fit a majority of the graphical keyboard 312. For example, the content of interest module 280 can determine the size of the second portion 304 and whether the size of the second portion 304 satisfies a threshold size. The threshold size can be any size preset in the memory of the computing device 200 (e.g., a rectangle of a specific pixel size). The threshold size can be a minimum threshold size for the second portion 304 to satisfy the condition that a majority of the graphical keyboard 312 of the minimum size can be visually superimposed on the second portion 304.

[0067] Although computing device 200 can output graphical keyboard 312 for display on GUI 310 in any size or configuration, certain sizes and / or configurations may be disadvantageous to a user of computing device 200. For example, graphical keyboard 312 may be too small for a user's finger to accurately select one or more characters of graphical keyboard 312. Therefore, graphical keyboard 312 may occupy a certain minimum space on GUI 310.

[0068] Graphic keyboard module 294 may determine the size of the minimum-sized graphical keyboard and output data indicating the size to content of interest module 280. In some examples, graphical keyboard module 294 may determine the size of a user input device used to select one or more characters of graphical keyboard 312, and graphical keyboard module 294 may adjust the minimum size of graphical keyboard 312 based on the determined size of the user input device. One or more sensors of the capacitive touch screen of computing device 200 may determine the surface area of ​​interaction of the user input device with the capacitive touch screen and determine the size of the user input device based on the surface area of ​​interaction. In this manner, graphical keyboard module 294 may set a graphical keyboard 312 of a larger minimum threshold size for a larger user input device (e.g., a finger), and may set a graphical keyboard 312 of a reduced minimum threshold size for a smaller user input device (e.g., a stylus).

[0069] In response to determining that the second portion 304 is too small to fit a majority of the graphical keyboard 312 (e.g., the size of the second portion 304 does not meet the threshold size), the content of interest module 280 may determine whether the screen partitioning module 290 should perform the split-screen mode. To determine whether the screen partitioning module 290 should perform the split-screen mode, the content of interest module 280 may determine whether the content of interest in the first portion can be reduced without making the content of interest too difficult for the user to see in the GUI 310. One or more minimum size thresholds for the content of interest may be stored in a memory of the computing device 200. The content of interest module 280 may determine whether the size of the reduced content of interest in the potential split-screen mode meets the minimum size threshold, and based on the determination, determine whether the screen partitioning module 290 should perform the split-screen mode. The content of interest module 280 may send data to the screen partitioning module 290 indicating whether the screen partitioning module 290 should perform the split-screen mode.

[0070] In response to determining that the content of interest in the first portion cannot be reduced without making the content of interest too difficult to see, the content of interest module 280 can redetermine the first portion and the second portion based on the priority level output from the content ID module 284 so that the second portion is large enough to display the graphical keyboard. For example, the content ID module 284 can determine that the GUI 310 includes both a still image and a playing video. The content ID module 284 can determine that the still image has a lower priority level than the playing video, that is, the still image should be covered before the playing video. Therefore, the content ID module 284 can assign a higher priority level to the playing video than to the still image. The content of interest module 280 can receive one or more priority levels corresponding to one or more parts of the identified content from the content ID module 284. The content of interest module 280 can determine that the content of interest with the lowest priority level should be part of the second portion 304. For example, the content of interest module 280 can receive the priority level of the still image in the GUI 310, the priority level of the playing video in the GUI 310, and the priority level of the active text field in the GUI 310. The priority level of the still image may be the lowest priority level. The content of interest module 280 may determine that the portion of the GUI 310 that includes the still image is part of the second portion 302. The content of interest module 280 may repeat this process until the second portion 302 is large enough to display the graphical keyboard 312. In this way, the content of interest module 280 may prioritize displaying content that is deemed to be of greater interest to the user. The content of interest module 280 may send data indicating the updated first portion 302 and second portion 304 to the screen partitioning module 290.

[0071] The screen partitioning module 290 may receive data from one or more other modules to help the screen partitioning module 290 determine where to output the graphical keyboard 312. The screen partitioning module 290 may receive data indicating the first portion 302 and the second portion 304 from the content of interest module 280. In some examples, the data indicating the first portion 302 and the second portion 304 include data indicating the third portion 306 and / or the fourth portion 308. The screen partitioning module 290 may also receive data indicating whether the screen partitioning module 290 should execute a split-screen mode from the content of interest module 280. The screen partitioning module 290 may also receive data indicating the size of the graphical keyboard 312 from the graphical keyboard module 294. Based on the received data, the screen partitioning module 290 may determine where to output the graphical keyboard 312.

[0072] In some examples, screen split module 290 determines where to output graphical keyboard 312 based on receiving data indicating that screen split module 290 should execute split screen mode. For example, in response to determining that the content of interest in first portion 302 can be reduced without making the content of interest too difficult to see, content of interest module 280 can instruct screen split module 290 to execute split screen mode (i.e., send data indicating that screen split module 290 should execute split screen mode). Screen split module 290 can execute split screen mode, such as Figure 3B . To execute the split screen mode, the screen splitting module 290 may split the GUI 310 into at least a first partition 323A and a second partition 323B, wherein the second partition 232B is large enough to fit the graphical keyboard 312. The screen splitting module 290 may include instructions for maintaining and controlling the number and arrangement of partitions within the GUI 310. In some examples, each partition of the split screen mode may be a separate GUI that is output for display. The screen splitting module 290 may resize at least some of the content of interest 301 to fit within the first partition 323A. In some examples, the entire first portion 302 prior to the split is resized to fit into the first partition 323A. Figure 3B In the example of FIG. 3 , the screen division module 290 has split the GUI 310 into the first partition 323A and the second partition 323B, and has reduced the size of some of the content of interest to fit into the first partition 323A.

[0073] Screen partition module 290 can resize second portion 304 of GUI 310 to be within second partition 323B so that a majority of graphical keyboard 312 can be visually superimposed on second portion 304 when output for display in GUI 310, and determine that graphical keyboard 312 should be output in second portion 304. In some examples, resized second portion 304 can occupy the entire second partition 323B. In some examples, second partition 323B can include some content of interest. Figure 3B In the example of , the screen partition module 204 has increased the size of the second portion 304 and includes both the second portion 304 and some interesting content in the second partition 323B. Figure 3B In the example of , the content of interest includes active text field 324 .

[0074] The graphical keyboard module 294 may output the updated GUI 310 including the graphical keyboard 312 for display by the display device of the computing device 200 through the one or more processors 250, so that in the second partition 323B, most of the graphical keyboard 312 is visually superimposed on the second portion 304 of the GUI 310. In this way, the graphical keyboard 312 can be output for display without obstructing (or obstructing as little as possible) the content of interest to the user. In order to output the graphical keyboard 312, the UI manager module 276 may send information defining the updated GUI 310 (including the graphical keyboard 312 according to the instructions from the graphical keyboard module 294) to the I / O module 278. Based on the information, the I / O module 278 may cause the user interface device 254 to output the GUI 310 for display on the display device, the GUI including the graphical keyboard 312.

[0075] In some examples, screen partition module 290 may determine where to output graphical keyboard 312 based on receiving data indicating fourth portion 308 in second portion 304. For example, computing device 200 may provide a better user experience to a user by outputting graphical keyboard 312 for display in an area of ​​GUI 310 that is directly adjacent to an active text field (e.g., active text field 324). In doing so, computing device 200 may allow a user to see one or more characters of graphical keyboard 312 (when the user selects them) as well as typed text being submitted to active text field 324. As described above, fourth portion 308 is a portion of second portion 304 that is near or adjacent to active text field 324. Screen partition module 290 may determine that graphical keyboard 312 should be output for display in fourth portion 308 and send instructions to graphical keyboard module 294 about where to output graphical keyboard 312. I / O module 278 may cause user interface device 254 to output a GUI 310 for display on a display device that includes a graphical keyboard 312 based on information received from graphical keyboard module 294 such that a majority of graphical keyboard 312 is visually superimposed on fourth portion 308 .

[0076] Figure 4A and Figure 4B is a conceptual diagram illustrating an example graphical user interface having blank space before and after a graphical keyboard is output for display in accordance with one or more techniques of this disclosure. Figure 2 The computing device 200 is described Figure 4A and Figure 4B .exist Figure 4A In the example of , GUI 410 includes content 401 of application GUI 416, including blank space 426. GUI 410 includes first portion 402 and second portion 404.

[0077] Graphical keyboard 412 may not be displayed as part of GUI 410 until computing device 200 determines that a user wishes to enter text at computing device 200. For example, a user may open a text editing application. However, once computing device 200 outputs graphical keyboard 412 for display, graphical keyboard 412 may obscure content of interest to the user (e.g., content of interest in first portion 402 of GUI 410). Computing device 200 may determine whether one or more portions of GUI 410 include content of interest to the user, and output graphical keyboard 412 for display so that it does not obscure (or obscures as little as possible) the content of interest to the user.

[0078] Interested content module 280 of computing device 200 may determine whether one or more portions of GUI 410 include content of interest to the user, and determine first portion 402 and second portion 404 of GUI 410 from the one or more portions. Interested content module 280 may determine first portion 402 to include each of the one or more portions of GUI 410 that include content having at least a threshold probability of being viewed by the user of computing device 200. Interested content module 280 may determine second portion 404 to include each of the one or more portions of GUI 410 that do not include content that satisfies at least the threshold probability. First portion 402 and second portion 404 may be non-overlapping portions of GUI 410.

[0079] Interested content module 280 may determine whether one or more portions of GUI 410 include content of interest to the user based on data output by content ID module 284. Content ID module 284 may identify the type of content 401 and output data indicating the type of content 401 to interested content module 280. For example, content ID module 284 may determine that text or blank space 426 is included in the graphical user interface.

[0080] For example, application GUI 416 may be a text editing program. The portion of the text editing program displayed on the display of computing device 200 may include text in the body, and blank space 426 where no text exists. Instructions for execution of the text editing program may be included in one or more of application modules 274A to 274N. Content ID module 284 may examine the tags of content 401 within application GUI 416 of GUI 410. One or more of the tags may indicate that content 401 within application GUI 416 includes text, and may also indicate the location in application GUI 416 where such content may be found. Using the location data, content ID module 284 may determine the location of the text in GUI 410 and the portion of GUI 410 that contains the text.

[0081] In some examples, content ID module 284 can determine that content 401 includes an active text field, where the active text field has cursor focus. Figure 4A and Figure 4B In the example of , content ID module 284 can determine that the active text field includes the text of application GUI 416, as well as additional space around the text of application GUI 416 where typed text may appear.

[0082] exist Figure 4AIn some examples, content ID module 284 may also identify blank space 426 in application GUI 416. In some examples, content ID module 284 may determine that a portion of GUI 410 that includes less than a threshold number of features includes blank space 426. For example, content ID module 284 may determine that blank space 426 includes a portion of GUI 410 that is displayed with a monochrome color scheme (e.g., the background of a web page). In some examples, content ID module 284 may determine that blank space 426 includes a portion of GUI 410 where the color contrast ratio is below a threshold ratio. In some examples, content ID module 284 may determine that blank space 426 includes a portion of GUI 410 that includes less than a threshold number of straight lines (e.g., the edge of an open document in a text editing application). In some examples, content ID module 284 may determine that blank space 426 includes a portion of GUI 410 that includes a simple repeating pattern (e.g., a series of cells in a spreadsheet application). Content ID module 284 may analyze information received from one or more application modules (e.g., application modules 274A-27N) in memory of computing device 200 that contain instructions for output of application GUI 416 to determine where blank space 426 may be displayed in application GUI 416. Content ID module 284 may output data indicating the type of content 401 and the location of content 401 in the GUI to content of interest module 280.

[0083] In response to receiving the type and location of content 401 from content ID module 284, content of interest module 280 may determine whether the content is or is not content of interest. That is, content of interest module 284 may determine whether the content meets the threshold probability that the user will view or is viewing the content. For example, content of interest module 280 may determine that the text of a text editing program meets the threshold probability that the user will view or is viewing the content, and may determine that the portion of GUI 410 displaying the text is included in first portion 402 of GUI 410. The text of the text editing program is the main focus of the program. Therefore, data in the memory of computing device 200 may indicate that a block of text in the text editing program meets the threshold probability. Content of interest module may determine that one or more portions of GUI 410 containing text are included in first portion 402. Content of interest module 280 may determine that a portion of GUI 410 including blank space 426 is included in second portion 404 and does not meet the threshold probability that the user of computing device 200 will view or is viewing the content.

[0084] The content of interest module 280 can determine a third portion 406 of the GUI 410 that includes an active text field. In some examples, the first portion 402 can include the third portion 406. The content of interest module 280 can determine a fourth portion 408 of the GUI 310 that is near or adjacent to the active text field 324, wherein the second portion 402 includes the fourth portion 408. In this manner, the content of interest module 280 can identify a portion of the GUI 410 that is near or adjacent to the active text field that can be obscured by the graphical keyboard 412 without obscuring the content of interest to the user. The content of interest module 280 can send data indicating the first portion 402 and the second portion 404 to the screen partition module 290 to determine where to output the graphical keyboard 412 for display.

[0085] The screen partition module 290 can receive data indicating the first portion 402 and the second portion 404 from the content of interest module 280 and determine where to output the graphical keyboard 412 for display based on the data. The screen partition module 290 can determine that the graphical keyboard 412 should be displayed in the second portion 402. In this way, the graphical keyboard 412 will not obscure (or obscure as little as possible) the content of interest in the GUI 410.

[0086] In some examples, the data indicating the first portion 402 and the second portion 404 include data indicating the third portion 406 (a portion including an active text field) and / or the fourth portion 408 (a segment of the second portion 404 that is near or adjacent to the active text field). The computing device 200 can provide a better user experience to the user by outputting the graphical keyboard 412 for display in an area of ​​the GUI 410 that is near or adjacent to the active text field. In doing so, the computing device 200 can allow the user to see one or more characters of the graphical keyboard 412 (when the user selects them) and the typed text that is being submitted to the active text field. As described above, the fourth portion 408 can be included in the second portion 404 near or adjacent to the active text field. The screen division module 290 can determine that the graphical keyboard 412 should be output for display in the fourth portion 408.

[0087] The screen partitioning module 290 may send instructions to the graphical keyboard module 294 regarding where to output the graphical keyboard 312. Based on the data received from the screen partitioning module 290, the graphical keyboard module 294 may output an updated GUI 410 including the graphical keyboard 412 such that a majority of the graphical keyboard 412 is visually superimposed on the second portion 404 of the GUI 410 and / or the fourth portion 408 of the GUI 410. In this manner, the graphical keyboard 412 may be output for display without obstructing (or obstructing as little as possible) content of interest to the user.

[0088] Figure 5 is a conceptual diagram illustrating an example computing device 200 configured to track a user's eye gaze in accordance with one or more techniques of this disclosure. Figure 2 The computing device 200 is described Figure 5 .exist Figure 5 In the example of , GUI 510 includes multiple gaze points 530A to 530N (collectively referred to as gaze points 530) and promotional content 528A to 528B (collectively referred to as promotional content 528).

[0089] Graphical keyboard 512 may not be displayed as part of GUI 510 until computing device 200 determines that the user wishes to enter text at computing device 200. However, once computing device 200 outputs graphical keyboard 512 for display, graphical keyboard 512 may obscure content of interest to the user (e.g., content of interest in first portion 502 of GUI 510). Computing device 200 may determine whether one or more portions of GUI 510 include content of interest to the user, and output graphical keyboard 512 for display so that it does not obscure (or obscures as little as possible) the content of interest to the user.

[0090] Interested content module 280 of computing device 200 may determine whether one or more portions of GUI 410 include content of interest to the user, and determine first portion 502 and second portion 504 of GUI 510 from the one or more portions. Interested content module 280 may determine that first portion 502 includes each of the one or more portions of GUI 510 that include content having at least a threshold probability of being viewed by a user of computing device 200. Interested content module 280 may determine that second portion 504 includes each of the one or more portions of GUI 510 that do not include content that satisfies at least the threshold probability. First portion 502 and second portion 504 may be non-overlapping portions of GUI 510.

[0091] Interested content module 280 may determine whether one or more portions of GUI 510 include content of interest to the user based on data output by content ID module 284. Content ID module 284 may identify the type of content 501 and output data indicating the type of content 501 to interested content module 280. For example, content ID module 284 may determine that promoted content 528 is included in GUI 510. Promoted content may include advertisements, sponsored content, third-party content for an application, or any other type of promoted content. Figure 5 In the example of FIG. 5 , GUI 510 includes promotional content 528A and 528B.

[0092] For example, the application GUI in GUI 510 may be a web browser. A portion of the web browser may include advertisements. Instructions for the execution of the web browser may be included in one or more of the application modules 274A to 274N. Content ID module 284 may examine the tags of the content within the web browser. One or more of the tags may indicate that the content within the web browser includes promotional content 528, and may also indicate the location in the web browser where such content can be found. Using the location data, content ID module 284 may determine the location of promotional content 528 in GUI 510 and the portion of GUI 510 that contains the promotional content.

[0093] In response to receiving the type and location of content 501 from content ID module 284, interested content module 280 may determine that the content is or is not interested content. That is, interested content module 284 may determine whether the content meets a threshold probability that a user will view or is viewing the content. For example, interested content module 280 may determine that the promoted content does not meet the threshold probability that a user will view or is viewing the content, and may determine that the portion of GUI 510 displaying the promoted content is included in second portion 504 of GUI 510.

[0094] The interesting content module 280 can determine whether one or more portions of the GUI 510 include content of interest to the user based on data output by the eye gaze module 282. The input device 252 of the computing device 200 can include one or more cameras. The eye gaze module 282 can determine which portions of the GUI 510 the user is looking at by, for example, tracking the user's eye movements using the one or more cameras. When tracking such eye movements, the eye gaze module 282 can determine one or more gaze points 530 of the user on the GUI 510. The eye gaze module 282 can determine the one or more gaze points 530, as described above with reference to FIG. Figure 2 described.

[0095] Interested content module 280 may determine a probability that a user will view or is viewing at least a portion of the content of GUI 510 based on data indicating eye movement of the user received from eye gaze module 282. Based on the number and / or time of gaze points 530 on GUI 510, interested content module 280 may determine whether the content of GUI 510 satisfies a threshold probability that the user is viewing or will view the content.

[0096] The eye gaze module 282 may determine the number of times the user looks at the content included in the third portion 506 of the GUI 510 based on the eye movement of the user. For example, the eye gaze module 282 may determine a plurality of gaze points 530. For each of the plurality of portions of the content included in the GUI 510, the eye gaze module 282 may determine the number of gaze points 530 located in the corresponding portion. For example, the eye gaze module 282 may determine the number of gaze points 530 located in the third portion 506. The eye gaze module 282 may also determine a plurality of timestamps corresponding to each of the gaze points 530, wherein each of the gaze points 530 corresponds to one of the plurality of timestamps. The eye gaze module 282 may output the number of gaze points 530, their locations in the GUI 510, and the plurality of timestamps corresponding to the gaze points 530 to the content of interest module 280.

[0097] In response to receiving the number, location, and timestamp associated with gaze points 530, interesting content module 280 can determine whether the content at those locations of GUI 510 is or is not interesting content. That is, interesting content module 284 can determine whether the content in those portions of GUI 510 meets a threshold probability that the user will view or is viewing the content.

[0098] For example, if the number of fixation points 530 located in the third portion 506 of the GUI 510 satisfies the threshold number, the content of interest module 280 may determine that the probability that the user will view or is viewing the content in the third portion 506 satisfies the threshold probability. The content of interest module 280 may receive data from the eye gaze module 282 indicating that eleven fixation points appear in the third portion 506. A threshold in the memory of the computing device 200 may indicate that any number of fixation points equal to or greater than ten satisfies the threshold probability. In response to determining that the probability satisfies the threshold probability, the content of interest module 280 may determine that the third portion 506 is included in the first portion 502.

[0099] In some examples, the content of interest module 280 may determine a proportion of the gaze points 530 that are located in the third portion 506. The content of interest module 280 may divide the number of gaze points 530 in the third portion 506 by the total number of gaze points in a time period. For example, the eye gaze module 282 may determine 100 gaze points in the past five minutes. Sixty gaze points may be located in the third portion 506, or sixty percent of the determined gaze points may be located in the third portion. The time period may be a preset time period in the memory, or a time period since the eye gaze module 282 began determining the gaze points. The content of interest module 280 may determine whether the proportion of the gaze points 530 in the third portion 506 meets a threshold proportion. For example, sixty percent of the determined gaze points may be located in the third portion 506, and according to the data in the memory of the computing device 200, any percentage above fifty percent may meet the threshold proportion. In response to the proportion of gaze points 530 in the third portion 506 satisfying the threshold proportion, the interested content module 280 can determine that the content of the third portion 506 satisfies the threshold likelihood that the user will view or is viewing the content, and determine that the third portion 506 is included in the first portion 502.

[0100] The content of interest module 280 may determine whether the content of the GUI 510 meets a threshold probability that the user is viewing or will view the content based on the timestamp of the gaze point 530. For each of the plurality of gaze points 530, the content of interest module 280 may determine whether the difference between the corresponding timestamp of each gaze point and the current time (e.g., received from the system clock application in the applications 274A to 274N) meets a threshold amount of time. In response to determining that the time difference meets the threshold amount of time, the content of interest module 280 may determine that the portion of the GUI 510 including the gaze point with the corresponding timestamp meets a threshold probability that the user will view or is viewing the content of the portion. For example, the difference between the corresponding timestamp of the gaze point and the current time may be five seconds. The data in the memory may indicate that a time difference of less than twenty seconds meets the threshold probability. The content of interest module 280 may determine that the portion of the GUI 510 including the gaze point with the corresponding timestamp that meets the threshold probability is included in the first portion 302. In this manner, the interested content module 280 may determine that recently viewed portions of the GUI 510 meet a threshold likelihood that the user will view or is viewing those portions, and determine those portions to be included in the first portion 302. The interested content module 280 may determine that portions of the GUI 510 that have not been viewed for a threshold amount of time do not meet a threshold likelihood that the user will view or is viewing those portions, and determine those portions to be included in the second portion 304. For example, the data in the memory may indicate that a time difference of greater than five minutes does not meet a threshold probability that the user is viewing or will view content.

[0101] The content of interest module 280 may send data indicating the first portion 502 and the second portion 504 to the screen partition module 290 to determine where to output the graphical keyboard 512 for display. The screen partition module 290 may receive data indicating the first portion 502 and the second portion 504 from the content of interest module 280 and determine where to output the graphical keyboard 512 for display based on the data. The screen partition module 290 may determine that the graphical keyboard 512 should be displayed in the second portion 504. In this way, the graphical keyboard 512 will not obscure (or obscure as little as possible) the content of interest in the GUI 510. The screen partition module 290 may send instructions to the graphical keyboard module 294 regarding where to output the graphical keyboard 512. Based on the data received from the screen partition module 290, the graphical keyboard module 294 may output an updated GUI 510 including the graphical keyboard 512, such that a majority of the graphical keyboard 512 is visually superimposed on the second portion 504 of the GUI 510.

[0102] Fig. 6A and Figure 6B is a conceptual diagram illustrating an example graphical user interface including user interfaces for multiple applications in accordance with one or more techniques of this disclosure. Figure 2 The computing device 200 is described Fig. 6A and Figure 6B .exist Fig. 6A In the example of FIG. 6 , GUI 610 includes application GUIs 616A to 616C. Fig. 6A and Figure 6B 610, but in other examples, GUI 610 may output any number of application GUIs. In some examples, the portion of GUI 610 displaying application GUI 616A may include playing a video, the portion of GUI 610 displaying application GUI 616B may include a messaging interface, and the portion of GUI 610 displaying application GUI 616C may include one or more animated graphics. Instructions for each of the application GUIs 616 may be included in one or more application modules 274A to 274N.

[0103] Interested content module 280 of computing device 200 may determine whether one or more portions of GUI 610 include content of interest to the user, and determine first portion 602 and second portion 604 of GUI 610 from the one or more portions. Interested content module 280 may determine first portion 602 to include each of the one or more portions of GUI 610 that include content having at least a threshold probability of being viewed by the user of computing device 200. Interested content module 280 may determine second portion 604 to include each of the one or more portions of GUI 610 that do not include content that satisfies at least the threshold probability. First portion 602 and second portion 604 may be non-overlapping portions of GUI 610.

[0104] The content of interest module 280 may determine whether one or more portions of the GUI 610 include content of interest to the user based on data output by the application manager module 286. The application manager module 286 may determine usage information for each of one or more currently executing applications (e.g., the application that outputs the application GUI 616) on the computing device 200. The application manager module 286 may output data indicating the usage information to the content of interest module 280. For example, the application manager module 286 may output data to the content of interest module 280 indicating the amount of time each application has been executing, the amount of time each application GUI has been visible on the display device, and / or whether the application is the foreground application.

[0105] The application manager module 286 may determine the amount of time that one or more currently executing applications of the output application GUI 616 have been executing. The application manager module 286 may examine data from one or more application modules 274A to 274N corresponding to each of the one or more currently executing applications to obtain the application runtime. The application manager module 286 may examine data from one or more application modules 274A to 274N corresponding to each of the one or more currently executing applications to obtain a timestamp corresponding to the start time of each application and compare the start time to a current time (e.g., a current time obtained from a system clock). The application manager module 286 may output the amount of time that the application has been executing to the content of interest module 280.

[0106] Application manager module 286 may determine an amount of time that one or more currently executing application GUIs 616 have been visible on the display device displaying GUI 610. Application manager module 286 may determine that application GUI 616A has been visible on the display device for one hour and fourteen minutes, application GUI 616B has been visible on the display device for twenty minutes, and application GUI 616C has been visible on the display device for ten minutes. Application manager module 286 may output the amount of time that each application GUI has been visible on the display device to content of interest module 280.

[0107] In some examples, application manager module 286 can determine an executing application as a foreground application. That is, application manager module 286 can determine which task, process, application, window, etc., the user is currently using (e.g., the user is actively interacting with it, has cursor focus, etc.). Application manager module 286 can output data indicating the foreground application to content of interest module 280.

[0108] In response to receiving the application usage data from the application manager module 286, the content of interest module 280 can determine whether the content associated with each currently executed application is content of interest (e.g., satisfies a threshold probability that the user will view or is viewing the content). For example, the content of interest module 280 can determine whether the content associated with each currently executed application is content of interest based on whether the application is a foreground application. The application GUI with which the user is actively interacting is likely the application GUI with which the user wants to continue to interact. Data in the memory of the computing device 200 can indicate that the foreground application satisfies a threshold probability that the user will view or is viewing the content. The content of interest module 280 can determine that the portion of the GUI 610 that includes the application GUI as the foreground application satisfies the threshold probability and is included in the first portion 602.

[0109] In some examples, the interesting content module 280 may determine whether content associated with each currently executing application is interesting content based on a respective amount of time that each application GUI 616 from one or more currently executing applications has been visible on the display device.

[0110] The content of interest module 280 can determine from one or more application GUIs 616 that one application GUI has been visible on the display device for a maximum amount of time based on the corresponding amount of time that the one or more application GUIs 616 have been visible on the display device. The application manager module 286 can determine that the content associated with one application GUI meets the threshold probability that the user will view or is viewing the content. For example, the application manager module 286 can determine that the content of application GUI 616A that has been visible on the display device for one hour and fourteen minutes meets the threshold probability. Application GUIs 616 that have been visible on the display device for an extended period of time can indicate that the user is actively engaging with the content of those applications. For example, application GUI 616A can include a movie or documentary that is being played, which may contain content that the user is interested in throughout the length of the movie / documentary. The content of interest module can determine that the portion of GUI 610 that includes application GUI 616A is included in the first portion 602.

[0111] The content of interest module can determine from one or more application GUIs 616 that one application GUI has been visible on the display device for a minimum amount of time based on the corresponding amount of time that the one or more application GUIs 616 have been visible on the display device. The application manager module 286 can determine that the content associated with an application GUI does not meet the threshold probability that the user will view or is viewing the content. For example, the application manager module 286 can determine that the application GUI 616C that has been visible on the display device for ten minutes does not meet the threshold probability. An application that has not been visible for a long time may not contain content that the user is interested in for a longer time than a brief moment. For example, the application GUI 616C may include a weather application that the user is interested in only for as long as the time it takes for the user to determine whether it will rain that day. The content of interest module 280 can determine that the portion of the GUI 610 that includes the application GUI 616C is included in the second portion 604.

[0112] In some examples, application manager module 286 can determine that an application GUI that has been visible on the display device for a minimum amount of time includes content that satisfies a threshold probability that the user will view or is viewing the content. Application manager module 286 can determine whether an application GUI 616B that is visible on the display device for sixty seconds satisfies the threshold probability. An application GUI 616B that the user has recently opened may contain content that is of interest to the user, e.g., the user may not have time to perform any intended interaction with application GUI 616B. For example, application GUI 616B may include a recently opened messaging application with unread messages. An application that the user has recently opened may also be a foreground application. Interested content module 280 can determine that the portion of GUI 610 that includes application GUI 616B is included in first portion 602.

[0113] The content of interest module 280 may send data indicating the first portion 602 and the second portion 604 to the screen partition module 290 to determine where to output the graphical keyboard 612 for display. The screen partition module 290 may receive data indicating the first portion 602 and the second portion 604 from the content of interest module 280 and determine where to output the graphical keyboard 612 for display based on the data. The screen partition module 290 may determine that the graphical keyboard 612 should be displayed in the second portion 604. In this way, the graphical keyboard 612 will not obscure (or obscure as little as possible) the content of interest in the GUI 610. The screen partition module 290 may send instructions to the graphical keyboard module 294 regarding where to output the graphical keyboard 612. Based on the data received from the screen partition module 290, the graphical keyboard module 294 may output an updated GUI 610 including the graphical keyboard 612, such that a majority of the graphical keyboard 612 is visually superimposed on the second portion 604 of the GUI 610.

[0114] Figure 7 is a flowchart illustrating an example operation for outputting a graphical keyboard within a graphical user interface according to one or more techniques of the present disclosure. Figure 2 components to describe the operation.

[0115] The one or more processors 250 of the computing device 200 may output a GUI (such as, by way of non-limiting example, Figure 1 GUI 110, Figure 3A GUI 310, Figure 4A GUI 410, Figure 5 GUI 510 or Fig. 6AThe GUI 610 is provided for display by the user interface device 254 (702). The GUI may include a plurality of contents, such as text, pictures, videos, graphics, etc. For example, the contents may include the words of a text document, images of a news website, a streaming movie, a graphics interchange format file in an instant messaging application, and / or buttons of a digital game. The contents may be part of one or more currently executed applications of the GUI.

[0116] Content interest module 280 may determine interesting content in the GUI (704). Certain content of the GUI may be of interest to a user of computing device 200. For example, the user may be viewing the content, interacting with the content, and / or about to view the content. Certain content may not be of interest to the user, or may be of less interest to the user than other content of the GUI. In some examples, content interest module 280 determines the interesting content based on gaze tracking data received from eye gaze module 282, content ID module 284, application manager module 268, and / or ML module 288. Content interest module 280 may determine that a first portion of the GUI includes interesting content, and a second portion of the GUI includes content that is not of interest or of less interest to the user.

[0117] The content of interest module 280 may determine whether to split the graphical user interface (706). To determine whether to split the GUI, the content of interest module 280 may determine the size of the first portion and the size of the second portion, and whether the sizes meet one or more threshold sizes. In some examples, the second portion of the GUI may be too small to display the graphical keyboard without obscuring at least some of the content of interest in the first portion. That is, the content of interest module 280 may determine that the size of the second portion does not meet a threshold size, where the threshold size is associated with a minimum-sized graphical keyboard.

[0118] In response to the content of interest module 280 determining that the size of the second part does not meet the threshold size, the screen partitioning module 290 can execute a split-screen mode in the GUI (the "yes" branch of 706). The screen partitioning module 290 can split the GUI into at least a first partition and a second partition (708). The screen partitioning module 290 can include the content of the first part of the GUI in the first partition and include the content of the second part of the GUI in the second partition. The size of the first partition on the GUI may be smaller than the first part before the split. The screen partitioning module 290 can resize the content in the first part of the GUI so that it fits in the smaller first partition. The size of the second partition on the GUI may be larger than the second part before the split. The second partition after the split may be large enough so that a large part of the graphical keyboard can be visually superimposed on the second partition.

[0119] UI manager module 276 may cause user interface device 254 to output an updated GUI for display on user interface device 254, the updated GUI including a graphical keyboard according to the information received from graphical keyboard module 294, such that a majority of the graphical keyboard is visually superimposed on the second partition (710). For example, UI manager module 276 of computing device 200 may update the GUI so that it includes the first GUI in the first partition and includes at least the second GUI in the second partition. The first GUI may include the interesting content of the GUI before the split, and the second GUI may include the graphical keyboard.

[0120] In response to determining that the size of the second portion satisfies the threshold size, the method may not include splitting the GUI (the "No" branch of 706). That is, the content of interest module 280 may determine the second portion of the GUI that does not contain the content of interest and determine that the second portion is large enough to display the graphical keyboard. The UI manager module 276 may cause the user interface device 254 to output an updated GUI for display on the user interface device 254, the updated GUI including the graphical keyboard according to the information received from the graphical keyboard module 294, such that a majority of the graphical keyboard is visually superimposed on the second portion of the GUI (712).

[0121] Various aspects of the techniques described in this disclosure may relate to the examples listed below.

[0122] Example 1: A method, comprising: outputting, by one or more processors of a computing device, a graphical user interface for display by a display device; determining, by the one or more processors, that a first portion of the graphical user interface includes content having at least a threshold probability of being viewed by a user of the computing device 200, wherein the graphical user interface also includes a second portion, wherein the first portion and the second portion are non-overlapping portions of the graphical user interface; and outputting, by the one or more processors, an updated graphical user interface including a graphical keyboard for display by the display device, such that a majority of the graphical keyboard is visually superimposed on the second portion of the graphical user interface.

[0123] Example 2: The method of Example 1, wherein the second portion of the graphical user interface does not include content having at least a threshold probability of being viewed by the user of the computing device.

[0124] Example 3: The method as described in Example 1 or 2 also includes: determining, by the one or more processors, whether the size of the second part of the graphical user interface meets a threshold size; and in response to determining that the size of the second part does not meet the threshold size, splitting, by the one or more processors, the graphical user interface into at least a first partition and a second partition, wherein: the first part of the graphical user interface is included in the first partition of the graphical user interface; and the graphical keyboard is included in the second partition of the graphical user interface.

[0125] Example 4: A method as described in any of Examples 1 to 3, wherein the content included in the first part of the graphical user interface is a first content part from multiple content parts included in the graphical user interface, and the method further includes: determining, by the one or more processors, whether the user has viewed a corresponding content part from the multiple content parts within a threshold amount of time; and in response to determining that the user has not viewed the corresponding part within the threshold amount of time, determining, by the one or more processors, that the corresponding part of the content of the graphical user interface is part of the second part of the graphical user interface.

[0126] Example 5: A method as described in any of Examples 1 to 4, wherein determining that the first portion of the graphical user interface includes content having at least the threshold probability of being viewed by the user of the computing device includes: tracking, by the one or more processors, eye movements of the user; determining, by the one or more processors and based on the eye movements, a probability that the user will view or is viewing at least a portion of the content of the graphical user interface; determining, by the one or more processors, whether the probability satisfies the threshold probability; and in response to determining that the probability satisfies the threshold probability, determining, by the one or more processors, that the portion of the content of the graphical user interface includes content having at least the threshold probability of being viewed by the user of the computing device.

[0127] Example 6: The method as described in any of Examples 1 to 5, further comprising: tracking, by the one or more processors, the eye movements of the user; determining, by the one or more processors and based on the eye movements of the user, the number of times the user looks at content included in the third part of the graphical user interface; determining, by the one or more processors, whether the number meets a threshold number; and in response to determining that the number does not meet the threshold number, determining, by the one or more processors, that the third part does not include content that meets the threshold probability of being viewed by the user of the computing device.

[0128] Example 7: The method as described in any of Examples 1 to 6, further includes: determining, by the one or more processors, whether the promotional content is included in the graphical user interface; and in response to determining that the promotional content is included in the graphical user interface, determining, by the one or more processors, that the space occupied by the promotional content on the graphical user interface does not include content that meets the threshold probability of being viewed by the user of the computing device.

[0129] Example 8: The method as described in any of Examples 1 to 7 further includes: determining, by the one or more processors, whether a video or image is included in the graphical user interface; and in response to determining that the video or image is included in the graphical user interface, determining, by the one or more processors, that the video or the image includes content that meets the threshold probability of being viewed by the user of the computing device.

[0130] Example 9: The method as described in any one of Examples 1 to 8, further includes: determining, by the one or more processors, whether the third portion of the graphical user interface includes an active text field, wherein the active text field is a text field with current cursor focus; and in response to determining that the third portion includes the active text field, determining, by the one or more processors, that the third portion includes content that meets the threshold probability of being viewed by the user of the computing device.

[0131] Example 10: The method as described in Example 9 also includes: determining, by the one or more processors, whether a fourth portion of the graphical user interface includes content that meets the threshold probability of being viewed by the user of the computing device, wherein the fourth portion is positioned adjacent to the active text field; and in response to determining that the fourth portion does not include content that meets the threshold probability of being viewed by the user of the computing device, outputting, by the one or more processors, an updated graphical user interface including a graphical keyboard for display by the display device, such that a majority of the graphical keyboard is visually superimposed on the fourth portion of the graphical user interface.

[0132] Example 11: The method as described in any of Examples 1 to 10 also includes: determining, by the one or more processors, a corresponding amount of time that each application from one or more currently executing applications has been visible on the display device of the computing device; identifying, by the one or more processors and based on the corresponding amount of time that the one or more currently executing applications have been visible on the display device, one application of the one or more currently executing applications that has been visible on the display device for a maximum amount of time; and determining, by the one or more processors, that content associated with the one application includes content that meets the threshold probability of being viewed by the user of the computing device.

[0133] Example 12: The method as described in Examples 1 to 11 also includes: determining, by the one or more processors, whether the third portion of the graphical user interface includes blank space; and in response to determining that the third portion of the graphical user interface includes blank space, determining, by the one or more processors, that the third portion does not include content that meets the threshold probability of being viewed by the user of the computing device.

[0134] Example 13: A computing system includes a display device, one or more processors, and a memory storing instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to output a graphical user interface for display by the display device, and determine that a first portion of the graphical user interface includes content having at least a threshold probability of being viewed by a user of the computing device, wherein the graphical user interface also includes a second portion, and wherein the first portion and the second portion are non-overlapping portions of the graphical user interface. The instructions may also cause the one or more processors to output an updated graphical user interface including a graphical keyboard for display by the display device, such that a majority of the graphical keyboard is visually superimposed on the second portion of the graphical user interface.

[0135] Example 14: The computing system of Example 13, wherein the second portion of the graphical user interface does not include content having at least a threshold probability of being viewed by the user of the computing device.

[0136] Example 15: A computing system as described in Example 13 or 14, wherein the instructions further cause the one or more processors to: determine whether the size of the second portion of the graphical user interface satisfies a threshold size; and in response to determining that the size of the second portion does not satisfy the threshold size, split the graphical user interface into at least a first partition and a second partition, wherein the first portion of the graphical user interface is included in the first partition of the graphical user interface, and wherein the graphical keyboard is included in the second partition of the graphical user interface.

[0137] Example 16: A computing system as described in any of Examples 13 to 15, wherein the content included in the first portion of the graphical user interface is a first content portion from a plurality of content portions included in the graphical user interface, and wherein the instructions further cause the one or more processors to: determine whether the user has viewed a corresponding content portion from the plurality of content portions within a threshold amount of time; and in response to determining that the user has not viewed the corresponding portion within the threshold amount of time, determine that the corresponding portion of the content of the graphical user interface is part of the second portion of the graphical user interface.

[0138] Example 17: A computing system as described in any of Examples 13 to 16, wherein the instructions that cause the one or more processors to determine that the first portion of the graphical user interface includes content having at least the threshold probability of being viewed by the user of the computing device also include instructions that cause the one or more processors to: track eye movements of the user; determine a probability that the user will view or is viewing at least a portion of the content of the graphical user interface based on the eye movements; determine whether the probability satisfies the threshold probability; and in response to determining that the probability satisfies the threshold probability, determine that the portion of the content of the graphical user interface includes content having at least the threshold probability of being viewed by the user of the computing device.

[0139] Example 18: A computing system as described in any of Examples 13 to 17, wherein the instructions further cause the one or more processors to: track eye movements of the user; determine a number of times the user looks at content included in a third portion of the graphical user interface based on the eye movements of the user; determine whether the number of times meets a threshold number of times; and in response to determining that the number of times does not meet the threshold number of times, determine that the third portion does not include content that meets the threshold probability of being viewed by the user of the computing device.

[0140] Example 19: A computing system as described in any of Examples 13 to 18, wherein the instructions further cause the one or more processors to: determine whether promotional content is included in the graphical user interface; and in response to determining that promotional content is included in the graphical user interface, determine that the space occupied by the promotional content on the graphical user interface does not include content that meets the threshold probability of being viewed by the user of the computing device.

[0141] Example 20: A computing system as described in any of Examples 13 to 19, wherein the instructions further cause the one or more processors to: determine whether a video or image is included in the graphical user interface; and in response to determining that a video or image is included in the graphical user interface, determine that the video or the image includes content that meets the threshold probability of being viewed by the user of the computing device.

[0142] Example 21: A computing system as described in any of Examples 13 to 20, wherein the instructions further cause the one or more processors to: determine that a third portion of the graphical user interface includes an active text field, wherein the active text field is a text field having current cursor focus; and in response to determining that the third portion includes the active text field, determine that the third portion includes content that meets the threshold probability of being viewed by the user of the computing device.

[0143] Example 22: A computing system as described in Example 21, wherein the instructions further cause the one or more processors to: determine whether a fourth portion of the graphical user interface includes content that meets the threshold probability of being viewed by the user of the computing device, wherein the fourth portion is positioned adjacent to the active text field; and in response to determining that the fourth portion does not include content that meets the threshold probability of being viewed by the user of the computing device, output an updated graphical user interface including a graphical keyboard for display by the display device, such that a majority of the graphical keyboard is visually superimposed on the fourth portion of the graphical user interface.

[0144] Example 23: A computing system as described in any of Examples 13 to 22, wherein the one or more processors are further configured to: determine a corresponding amount of time that each application from one or more currently executing applications has been executing in the foreground; identify one of the one or more currently executing applications that has been executing in the foreground for a maximum amount of time based on the corresponding amount of time that the one or more currently executing applications have been executing in the foreground; and determine that content associated with the one application includes content that meets the threshold probability of being viewed by the user of the computing device.

[0145] Example 24: A computing system as described in any of Examples 13 to 23, wherein the instructions further cause the one or more processors to: determine whether a third portion of the graphical user interface includes blank space; and in response to determining that the third portion of the graphical user interface includes blank space, determine that the third portion does not include content that meets the threshold probability of being viewed by the user of the computing device.

[0146] Example 25: A computer-readable storage medium is encoded with instructions that, when executed, cause one or more processors of a computing device to: output a graphical user interface for display by a display device of the computing device; and determine that a first portion of the graphical user interface includes content having at least a threshold probability of being viewed by a user of the computing device, wherein the graphical user interface also includes a second portion, wherein the first portion and the second portion are non-overlapping portions of the graphical user interface. The instructions may also cause the one or more processors to output an updated graphical user interface including a graphical keyboard for display by the display device, such that a majority of the graphical keyboard is visually superimposed on the second portion of the graphical user interface.

[0147] Example 26: The computer-readable storage medium of Example 25, wherein the second portion of the graphical user interface does not include content having at least a threshold probability of being viewed by the user of the computing device.

[0148] Example 27: A computer-readable storage medium as described in Example 25 or 26, wherein the instructions further cause the one or more processors to: determine whether the size of the second portion of the graphical user interface satisfies a threshold size; and in response to determining that the size of the second portion does not satisfy the threshold size, split the graphical user interface into at least a first partition and a second partition, wherein: the first portion of the graphical user interface is included in the first partition of the graphical user interface; and the graphical keyboard is included in the second partition of the graphical user interface.

[0149] Example 28: A computer-readable storage medium as described in any of Examples 25 to 27, wherein the content included in the first portion of the graphical user interface is a first content portion from a plurality of content portions included in the graphical user interface, and wherein the instructions further cause the one or more processors to: determine whether the user has viewed a corresponding content portion from the plurality of content portions within a threshold amount of time; and in response to determining that the user has not viewed the corresponding portion within the threshold amount of time, determine that the corresponding portion of the content of the graphical user interface is part of the second portion of the graphical user interface.

[0150] Example 29: A computer-readable storage medium as described in any of Examples 25 to 28, wherein the instructions that cause the one or more processors to determine the first portion of the graphical user interface also include instructions that cause the one or more processors to determine content having at least the threshold probability of being viewed by the user of the computing device, and wherein the instructions also cause the one or more processors to: track eye movements of the user; determine a probability that the user will view or is viewing at least a portion of the content of the graphical user interface based on the eye movements; determine whether the probability satisfies the threshold probability; and in response to determining that the probability satisfies the threshold probability, determine that the portion of the content of the graphical user interface includes content having at least the threshold probability of being viewed by the user of the computing device.

[0151] Example 30: A computer-readable storage medium as described in any of Examples 25 to 29, wherein the instructions further cause the one or more processors to: track eye movements of the user; determine a number of times the user looks at content included in a third portion of the graphical user interface based on the eye movements of the user; determine whether the number of times meets a threshold number of times; and in response to determining that the number of times does not meet the threshold number of times, determine that the third portion does not include content that meets the threshold probability of being viewed by the user of the computing device.

[0152] Example 31: A computer-readable storage medium as described in any of Examples 25 to 30, wherein the instructions further cause the one or more processors to: determine whether promotional content is included in the graphical user interface; and in response to determining that promotional content is included in the graphical user interface, determine that the space occupied by the promotional content on the graphical user interface does not include content that meets the threshold probability of being viewed by the user of the computing device.

[0153] Example 32: A computer-readable storage medium as described in any of Examples 25 to 31, wherein the instructions further cause the one or more processors to: determine whether a video or image is included in the graphical user interface; and in response to determining that a video or image is included in the graphical user interface, determine that the video or the image includes content that meets the threshold probability of being viewed by the user of the computing device.

[0154] Example 33: A computer-readable storage medium as described in any of Examples 25 to 32, wherein the instructions further cause the one or more processors to: determine that a third portion of the graphical user interface includes an active text field, wherein the active text field is a text field having current cursor focus; and in response to determining that the third portion includes the active text field, determine that the third portion includes content that meets the threshold probability of being viewed by the user of the computing device.

[0155] Example 34: A computer-readable storage medium as described in Example 33, wherein the instructions further cause the one or more processors to: determine whether a fourth portion of the graphical user interface includes content that meets the threshold probability of being viewed by the user of the computing device, wherein the fourth portion is positioned adjacent to the active text field; and in response to determining that the fourth portion does not include content that meets the threshold probability of being viewed by the user of the computing device, output an updated graphical user interface including a graphical keyboard for display by the display device, such that a majority of the graphical keyboard is visually superimposed on the fourth portion of the graphical user interface.

[0156] Example 35: A computer-readable storage medium as described in any of Examples 25 to 34, wherein the instructions further cause the one or more processors to: determine a corresponding amount of time that each application from one or more currently executing applications has been executing in the foreground; identify one of the one or more currently executing applications that has been executing in the foreground for a maximum amount of time based on the corresponding amount of time that the one or more currently executing applications have been executing in the foreground; and determine that content associated with the one application includes content that meets the threshold probability of being viewed by the user of the computing device.

[0157] Example 36: A computer-readable storage medium as described in any of Examples 25 to 35, wherein the instructions further cause the one or more processors to: determine whether a third portion of the graphical user interface includes blank space; and in response to determining that the third portion of the graphical user interface includes blank space, determine that the third portion does not include content that meets the threshold probability of being viewed by the user of the computing device.

[0158] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium or sent via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include one or more computer-readable storage media corresponding to a tangible medium (such as a data storage medium) or a communication medium (including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol). In this manner, a computer-readable medium may generally correspond to (1) a non-temporary tangible computer-readable storage medium or (2) a communication medium (such as a signal or carrier wave). A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0159] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, disk storage devices or other magnetic storage devices, flash memory or any other storage medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to send instructions from a website, server or other remote source, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave) is included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but are directed to non-temporary tangible storage media. Disks and optical disks as used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks and blue-ray disks, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The combination of the above items should also be included in the scope of computer-readable media.

[0160] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules. Moreover, the techniques may be fully implemented in one or more circuits or logic elements.

[0161] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but the various components, modules, or units are not necessarily required to be implemented by different hardware units. Rather, as described above, the various units may be combined in a hardware unit or provided by a collection of interoperable hardware units including one or more processors as described above in combination with appropriate software and / or firmware.

[0162] Various embodiments have been described. These and other embodiments are within the scope of the following claims.

Claims

1. A method, include: Outputting a graphical user interface by one or more processors of the computing device for display by a display device; determining, by the one or more processors, that a first portion of the graphical user interface includes content having at least a threshold probability of being viewed by a user of the computing device, wherein the graphical user interface further includes a second portion, wherein the first portion and the second portion are non-overlapping portions of the graphical user interface; as well as An updated graphical user interface including a graphical keyboard is output by the one or more processors for display by the display device such that a majority less than the graphical keyboard is visually superimposed on the first portion of the graphical user interface.

2. The method of claim 1, wherein the second portion of the graphical user interface does not include content having at least the threshold probability of being viewed by the user of the computing device.

3. The method according to claim 1 or 2, further comprising: include: determining, by the one or more processors, whether a size of the second portion of the graphical user interface satisfies a threshold size; as well as In response to determining that the size of the second portion does not satisfy the threshold size, splitting, by the one or more processors, the graphical user interface into at least a first partition and a second partition, wherein: The first portion of the graphical user interface is included within the first section of the graphical user interface; and The graphical keyboard is included within the second partition of the graphical user interface.

4. The method of any one of claims 1 to 3, wherein the content included in the first portion of the graphical user interface is a first content portion from a plurality of content portions included in the graphical user interface, the method further comprising: include: determining, by the one or more processors, whether the user has viewed a corresponding content portion from the plurality of content portions within a threshold amount of time; as well as In response to determining that the user has not viewed the corresponding portion within the threshold amount of time, determining, by the one or more processors, that the corresponding portion of the content of the graphical user interface is part of the second portion of the graphical user interface.

5. The method of any one of claims 1 to 4, wherein determining that the first portion of the graphical user interface includes content having at least the threshold probability of being viewed by the user of the computing device include: tracking, by the one or more processors, eye movements of the user; determining, by the one or more processors and based on the eye movement, a probability that at least a portion of the content of the graphical user interface is viewed by the user of the computing device; determining, by the one or more processors, whether the probability satisfies the threshold probability; as well as In response to determining that the probability satisfies the threshold probability, determining, by the one or more processors, that the portion of the content of the graphical user interface includes content having at least a threshold probability of being viewed by the user of the computing device.

6. The method according to any one of claims 1 to 5, further comprising: include: tracking, by the one or more processors, eye movements of the user; determining, by the one or more processors and based on the eye movement of the user, a number of times the user looked at content included in a third portion of the graphical user interface; determining, by the one or more processors, whether the number of times satisfies a threshold number of times; as well as In response to determining that the number does not satisfy the threshold number, determining, by the one or more processors, that the third portion does not include content that satisfies the threshold probability of being viewed by the user of the computing device.

7. The method according to any one of claims 1 to 6, further comprising: include: determining, by the one or more processors, whether promotional content is included within the graphical user interface; as well as In response to determining that promoted content is included within the graphical user interface, determining, by the one or more processors, that the space on the graphical user interface occupied by the promoted content does not include content that meets the threshold probability of being viewed by the user of the computing device.

8. The method according to any one of claims 1 to 7, further comprising: include: determining, by the one or more processors, whether a video or image is included in the graphical user interface; as well as In response to determining that a video or image is included in the graphical user interface, determining, by the one or more processors, that the video or the image includes content that satisfies the threshold probability of being viewed by the user of the computing device.

9. The method according to any one of claims 1 to 8, further comprising: include: determining, by the one or more processors, whether a third portion of the graphical user interface includes an active text field, wherein the active text field is a text field having current cursor focus; as well as In response to determining that the third portion includes the active text field, determining, by the one or more processors, that the third portion includes content that meets the threshold probability of being viewed by the user of the computing device.

10. The method according to claim 9, further comprising: include: determining, by the one or more processors, whether a fourth portion of the graphical user interface includes content that meets the threshold probability of being viewed by the user of the computing device, wherein the fourth portion is positioned adjacent to the active text field; as well as In response to determining that the fourth portion does not include content that meets the threshold probability of being viewed by the user of the computing device, the one or more processors output an updated graphical user interface including a graphical keyboard for display by the display device, such that a majority of the graphical keyboard is visually superimposed on the fourth portion of the graphical user interface.

11. The method according to any one of claims 1 to 10, further comprising: include: determining, by the one or more processors, a respective amount of time that each application from one or more currently executing applications has been visible on the display device of the computing device; identifying, by the one or more processors and based on the respective amounts of time that the one or more currently executing applications have been visible on the display device, an application of the one or more currently executing applications that has been visible on the display device for a maximum amount of time; as well as Determining, by the one or more processors, that content associated with the one application includes content that satisfies the threshold for being viewed by the user of the computing device.

12. The method according to claims 1 to 11, further comprising: include: determining, by the one or more processors, whether a third portion of the graphical user interface includes blank space; as well as In response to determining that the third portion of the graphical user interface includes blank space, determining, by the one or more processors, that the third portion does not include content that meets the threshold probability of being viewed by the user of the computing device.

13. A computing system comprising at least one processor configured to perform any one of the methods of claims 1 to 12.

14. A system comprising means for performing any one of the methods of claims 1 to 12.

15. A computer-readable storage medium comprising instructions which, when executed, cause at least one processor to perform any one of the methods of claims 1 to 12.

Images