POSITIONING A GRAPHIC KEYBOARD WITHIN A GRAPHIC USER INTERFACE

Abstract

Described is a computing system that includes a memory which stores instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to output a graphical user interface for display and to determine that a first section of the graphical user interface contains content which has at least a threshold probability that a user of the computing device 1) will view the content or 2) is currently viewing the content, wherein the graphical user interface further includes a second section and wherein the first section and the second section are non-overlapping sections of the graphical user interface.The instructions can further instruct one or more processors to output an updated graphical user interface, including a graphical keyboard, for display by the display device, such that a large portion of the graphical keyboard visually overlays the second section of the graphical user interface.

Description

GENERAL STATE OF THE ART

[0001] Some computing devices (e.g., mobile phones, tablet computers) can output a graphical keyboard for display as part of a graphical user interface for receiving text input (e.g., using a presence-sensitive input device and / or display, such as a touchscreen). The graphical keyboard can allow a user of the computing device to enter text (e.g., an email, a text message, a document, etc.). For example, a display device of a computing device can output a graphical keyboard to a presence-sensitive input device, allowing the user to enter data by tapping or swiping one or more keys on the graphical keyboard that are displayed on the display device. SUMMARY

[0002] Techniques described in this disclosure enable a computing device to display a graphical keyboard within a portion of a graphical user interface in order to reduce the amount of viewed or likely viewed content that is obscured by the graphical keyboard. For example, the computing device may determine that at least one portion of the content contained in the graphical user interface is being viewed or likely to be viewed by a user of the computing device, and / or that at least one portion of the graphical user interface that is being viewed or likely to be viewed by the user does not contain any content. The computing device may determine to display the graphical keyboard within the portion of the graphical user interface that does not contain any content that is being viewed or likely to be viewed by the user.In various examples, if the section of the graphical user interface that does not contain content viewed or likely to be viewed by the user is insufficient to display the graphical keyboard, the computing device can resize the content (e.g., by switching an active application from full-screen mode to split-screen mode, by reducing the amount of screen displaying the content, etc.) to create a section of the graphical user interface that does not contain any content and display the graphical keyboard within the newly created section of the graphical user interface.

[0003] In an example, this disclosure describes a method for outputting, by one or more processors of a computing device and for display by a display device, a graphical user interface, and determining, by the one or more processors, that a first section of the graphical user interface contains content that has at least a threshold probability of being viewed by a user of the computing device, wherein the graphical user interface further includes a second section, and wherein the first section and the second section are non-overlapping sections of the graphical user interface.The method may further include output, by one or more processors and for display by the display device, of an updated graphical user interface, including a graphical keyboard, such that a large part of the graphical keyboard visually overlays the second section of the graphical user interface.

[0004] In another example, a computer system includes a display device, one or more processors, and a memory for storing instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to output a graphical user interface for display by the display device and to determine that a first section of the graphical user interface contains content that has at least a threshold probability of being viewed by a user of the computing device, wherein the graphical user interface further includes a second section, and wherein the first section and the second section are non-overlapping sections of the graphical user interface.The instructions can further instruct one or more processors to output an updated graphical user interface, including a graphical keyboard, for display by the display device, such that a large portion of the graphical keyboard visually overlays the second section of the graphical user interface.

[0005] In an additional example, a computer-readable storage medium is encoded with instructions which, when executed, cause one or more processors of a computing device to do the following: output, for display by a display device of the computing device, a graphical user interface, and determine that a first section of the graphical user interface contains content which has at least a threshold probability of being viewed by a user of the computing device, wherein the graphical user interface further includes a second section, the first section and the second section being non-overlapping sections of the graphical user interface.The instructions can further instruct one or more processors to output an updated graphical user interface, including a graphical keyboard, for display by the display device, such that a large portion of the graphical keyboard visually overlays the second section of the graphical user interface.

[0006] In another example, a computer system includes means for outputting a graphical user interface for display and means for determining that a first section of the graphical user interface contains content that has at least a threshold probability of being viewed by a user of the computer system, wherein the graphical user interface further includes a second section, and wherein the first and second sections are non-overlapping sections of the graphical user interface. The computing device may further include means for outputting an updated graphical user interface for display, including a graphical keyboard, such that much of the graphical keyboard visually overlays the second section of the graphical user interface.

[0007] The details of one or more examples are set forth in the accompanying drawings and the description below. Further features, functions, and advantages of the disclosure will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a conceptual diagram illustrating an exemplary computing device configured to display a graphic keyboard according to one or more techniques of the present disclosure. Fig. Figure 2 is a block diagram illustrating an exemplary computing device configured to determine a user interaction with a graphical user interface of the device according to one or more techniques of the present disclosure. Fig. 3A and Fig. 3B are conceptual diagrams illustrating exemplary graphical user interfaces before and after the execution of a split-screen mode according to one or more techniques of disclosure. Fig. 4A and Fig. 4B are conceptual diagrams illustrating exemplary graphical user interfaces with white space according to one or more techniques of disclosure. Fig. Figure 5 is a conceptual diagram illustrating an exemplary computing device configured to track a user's gaze direction according to one or more techniques of revelation. Fig. 6A and Fig. 6B are conceptual diagrams illustrating exemplary graphical user interfaces, which include user interfaces for a variety of applications according to one or more techniques of disclosure. Fig. Figure 7 is a flowchart illustrating exemplary operations for outputting a graphical keyboard within a graphical user interface according to one or more techniques of the disclosure. DETAILED DESCRIPTION

[0008] Fig. Figure 1 is a conceptual diagram illustrating an exemplary computing device configured to display a graphic keyboard according to one or more techniques of the present disclosure. As in Fig. As shown in Figure 1, a computing device 100 is a mobile computing device (e.g., a tablet computer). In other examples, however, the computing device 100 may be a mobile phone, a laptop computer, a desktop computer, a gaming system, a media player, an e-book reader, a television platform, a motor vehicle navigation system, a wearable computing device (e.g., a computerized watch, a computerized headset, computerized glasses, a computerized glove), or some other type of mobile or non-mobile computing device.

[0009] The computing device 100 includes a user interface device (UID) that can function as both an input device and an output device for the computing device 100. The UID can be implemented using various technologies. For example, as an input device, the UID can utilize a presence-sensitive input screen, such as a resistive touchscreen, an acoustic surface wave touchscreen, a capacitive touchscreen, a projective capacitive touchscreen, a pressure-sensitive screen, an acoustic pulse detection touchscreen, or another presence-sensitive display technology. The UID can also function as an output device (e.g., a display).Display device) using one or more display devices, such as a liquid crystal display (LCD), a dot matrix display, a light emitting diode display (LED display), a microLED display, an organic light-emitting diode (OLED) display, an e-ink display or a similar monochrome or color display capable of outputting 100 visible pieces of information to the user of the computing device.

[0010] The user interface (UID) of the computing device 100 can include a presence-sensitive display that can receive tactile input from a user of the computing device 100. The UID can receive one or more pieces of tactile input by detecting one or more gestures from a user of the computing device 100 (e.g., when the user touches or points to one or more parts of the UID with a finger or stylus). The UID can present an output to a user, for example, on a presence-sensitive display. The UID can present the output as a graphical user interface (e.g., the graphical user interface 110), which may be associated with functionality provided by the computing device 100.For example, the UID can present various user interfaces of components of a computing platform, an operating system, applications, or services that run on or can be accessed by the computing device 100 (e.g., an electronic messaging application, a web browser application, a mobile operating system, etc.). A user can interact with a particular user interface to cause the computing device 100 to perform operations related to a specific function.

[0011] As in Fig. As shown in Figure 1, a graphical user interface (GUI) 110 contains content 101. In general, the content 101 can include text, one or more images, videos, graphics, or any other content of the GUI 110 that is displayed by the display device. More specifically, in various examples, the content 101 can include content such as words from a text document, an image from a website, a streaming movie, an image in Graphics Interchange Format (GIF) in an instant messaging application, a game control, and so on. The content 101 can be included in a graphical user interface of a foreground application (e.g., the GUI 110). The content 101 can be of interest to the user of the computing device 100. For example, the user can be currently viewing the content, interacting with the content, and / or viewing the content soon.In some examples, content 101 may not be of interest to a user, or may be of less interest to the user than other content in GUI 110.

[0012] An application or operating system running on computer 100 can selectively cause the UID to display a graphical keyboard. For example, if a user is watching a movie, the application and / or operating system of computer 100 may determine that no text input is required and therefore not cause the UID to display the graphical keyboard. In another example, because a cursor may be contained within a text input field of the graphical user interface, the application and / or operating system may determine that text input is required and cause the UID to display the graphical keyboard.When the graphical keyboard is displayed, a section of content 101, or other sections of content included in the GUI 110, may be obscured or visually shifted in such a way that the sections of content are no longer displayed or otherwise visible to a user of the computing device 100. In cases where the obscured content is content the user wishes to view, the user may need to provide additional input to scroll or otherwise adjust the content to make the desired content visible. This additional input causes the computing device 100 to consume additional processing power by detecting the additional input, updating the content displayed by the UID, and potentially activating various components of the computing device 100 (e.g.,the UID) are switched on for longer than is otherwise required by a computing device implementing the techniques of this disclosure.

[0013] According to one or more techniques of this disclosure, the computing device 100 can, by means of one or more processors, identify different sections of the GUI 110 such that they contain content of varying relative importance to the user of the computing device 100, and output a graphical keyboard 112 for display within the GUI 110 in such a way that the graphical keyboard 112 obscures as little as possible of the content of interest. For example, the computing device 100 can determine that the first section 102 of the GUI 110 contains content that is of interest to the user and thus relatively more important to the user, so that the content contained in the first section 102 should not be obscured.Similarly, the computing device 100 can determine that the second section 104 of the GUI 110 contains content that is of no interest or less interest to the user and is therefore relatively less important to the user, so that the content contained in the second section 104 can be hidden, moved or otherwise modified to accommodate the graphic keyboard 112 within the GUI 110.

[0014] In determining which content of GUI 110 is relatively more or less important, the computing device 100 can determine a probability that the user of the computing device 100 will either view or is currently viewing different sections of the content of GUI 110. If the determined probability does not meet a threshold probability, the computing device 100 can determine that the associated content sections are relatively less important. Conversely, if the determined probability meets the threshold probability, the computing device 100 can determine that the associated content sections are relatively more important. The computing device 100 can determine the probability that the user is currently viewing or will view the content by tracking the user's eye movements based on an active cursor focus (e.g.,in an active text field), based on media content, by determining how long each application of one or more currently running applications was visible on the display of the computing device, based on whether the user viewed a section of the content of the GUI 110 within a threshold period, based on whether advertising content is included in the GUI 110, based on whether the section of the GUI 110 is empty (e.g., contains no text or graphic elements other than a background image, background color, etc.), or any combination thereof.

[0015] The computing device 100 can output a graphical keyboard 112 such that content contained in the second section 104 is obscured, and a minimal amount of content contained in the first section 102 is obscured. However, in some examples, the second section 104 may be too small (or non-existent) to output the graphical keyboard 112 at a size or in a location that provides sufficient usability for the user of the computing device 100. In such examples, the computing device 100 can change the size of the first section 102 and the second section 104, or it can divide the GUI 110 into a first partition and a second partition, with the first partition containing the content 101 and the graphical keyboard 112 contained in the second partition.

[0016] By positioning the graphical keyboard 112 within the GUI 110 to reduce the amount of content of interest to the user that is obscured by the graphical keyboard 112, the techniques of this disclosure can improve the operational efficiency of the computing device 100. For example, the techniques described in this document can reduce the number of user inputs required, reduce the time the UID of the computing device 100 is powered on, reduce the amount of content rendered by the computing device 100, or any combination thereof. Improving such operational efficiencies can reduce the amount of power consumed by the computing device 100 during operation.

[0017] Fig. Figure 2 is a block diagram illustrating an exemplary computing device configured to determine user interaction with a graphical user interface of the device according to one or more techniques of the present disclosure. The computing device 200 from Fig. 2 is an example of the calculating device 100 from Fig. 1. The calculating device 200 is only one concrete example of the calculating device 100 from Fig. 1 and many other examples of the computing device 200 can be used in other cases and may include a subset of the components included in the exemplary computing device 200, or may include additional components that are in Fig. 2 are not shown.

[0018] As in the example from Fig. As shown in Figure 2, the computing device 200 includes one or more processors 250, one or more input devices 252, one or more communication units 256, one or more output devices 258, one or more storage devices 270, and a user interface device 254. One or more storage devices 270 of the computing device 200 can store an operating system 272, a UI management module 276, an I / O module 278, application modules 274A-274N, a content of interest module 280, a gaze direction module 282, a content identification (ID) module 284, an application management module 286, a machine learning (ML) module 288, a screen section module 290, a priority module 292, and a graphic keyboard module 294. Communication channels 260 can connect each of the components 250, 252, 254, 256, 258 and 270 for communication between the components (physical, communicative and / or effective).In some examples, the communication channels 260 may include a system bus, a network connection, one or more data structures for communication between processes, or any other method for data communication between hardware and / or software.

[0019] One or more input devices 252 of the computing device 200 can receive inputs. Examples of inputs are tactile, acoustic, and visual. One or more input devices 252 of the computing device 200 include, in one example, a mouse, a keyboard, a voice-controlled system, a video camera, a microphone, or any other type of device for detecting inputs from a person or a machine. In some examples, the input device 252 can be a presence-sensitive input device, which may include a presence-sensitive screen, a touch-sensitive screen, etc.

[0020] One or more output devices 258 of the computing device 200 can generate outputs. Examples of outputs are tactile, acoustic, and visual outputs. The output devices 258 of the computing device 200 include, in one example, a presence-sensitive screen, a sound card, a video graphics adapter card, a loudspeaker, a liquid crystal display (LCD), a miniLCD, a microLCD, an organic light-emitting diode (OLED), or any other type of device for generating an output to a person or a machine.

[0021] One or more communication units 256 of the computing device 200 can communicate with external devices over one or more networks by transmitting and / or receiving network signals in the one or more networks. For example, the computing device 200 can use the communication unit 44 to transmit and / or receive radio signals in a radio network, such as a cellular network. Likewise, the communication units 256 can transmit and / or receive satellite signals in a satellite network, such as a GPS network. Examples of the communication units 256 include a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device capable of transmitting and / or receiving information.Other examples of the communication units 256 may include Bluetooth®, GPS, 3G, 4G and Wi-Fi® radios found in mobile devices, as well as Universal Serial Bus (USB) controllers.

[0022] In some examples, the computing device 200 includes an I / O module (input / output module) 278. The I / O module 278 can receive information from the UI management module 276 and output such information to devices with output functionality (e.g., the user interface device 254, a communication unit, etc.). The I / O module 278 can also receive information from devices with input functionality (e.g., the user interface device 254, a communication unit, etc.). For example, the I / O module 278 can receive information that defines the GUI of the computing device 200 from the UI management module 276 and one or more of the application modules 274A-274N, the content of interest module 280, the gaze direction module 282, the content ID module 284, the application management module 286, the ML module 288, the screen section module 290, the priority module 292 and the graphic keyboard module 294.Based on the information it receives, the I / O module 278 can cause the user interface device 254 to output the GUI for display. In some examples, the I / O module 278 can determine a user input input made at the user interface device 254 and send such information to one or more of the UI management module 276, the application modules 274A-274N, the content of interest module 280, the gaze direction module 282, the content ID module 284, the application management module 286, the ML module 288, the screen section module 290, the priority module 292, and / or the graphic keyboard module 294.

[0023] In some examples, the user interface device 254 of the computing device 200 can include functionality of the input devices 252 and / or the output devices 258. In some examples, the GUI can refer to a presence-sensitive display. In the example from Fig. 2. The user interface device 254 can include a presence-sensitive input device, such as a presence-sensitive screen, a touch-sensitive screen, etc. In some examples, a presence-sensitive input device can detect an object at and / or near the presence-sensitive input device. As one example, a presence-sensitive input device can detect an object, such as a finger or a stylus, that is within 2 inches or less of the presence-sensitive screen. The presence-sensitive input device can determine a location (e.g., an (x,y) coordinate) of the presence-sensitive input device at which the object was detected. In another example, a presence-sensitive input device can detect an object 6 inches or less from the presence-sensitive input device, and other ranges are also possible.The presence-sensitive input device can determine the position of the screen selected by a user's finger using capacitive, inductive, and / or optical detection techniques. In some examples, the user interface device 254 provides an output to a user using tactile, acoustic, or visual stimuli, as described in relation to the output device 258. For example, the user interface device 254 may include a display device that presents a user interface, such as the GUI 110 from [reference missing]. Fig. 1. Present. In some examples, a user can interact with the presence-sensitive input device displayed by GUI 110 to select characters from a graphical keyboard.

[0024] The presence-sensitive input device can include one or more touch sensors capable of detecting touch input from a user. Each touch sensor can include one or more electrical devices, such as capacitive or resistive sensors, or one or more mechanical devices, such as switches or pressure plates. For example, each touch sensor can include one or more simple capacitance sensors that detect a change in the capacitance of a material. Electrical hardware attached to each simple capacitance sensor can monitor it for a change in capacitance. A change in capacitance can be determined by the attached electrical hardware, and a reading can be output by the touch sensor.Other well-known sensor technologies that provide the ability to detect touch input can also be used to receive touch input from a user.

[0025] While the user interface device 254 is illustrated as an integrated component of the computing device 200, in some examples it can be an external component that shares a data path with other components of the computing device 200 for transmitting and / or receiving inputs and outputs. For example, the user interface device 254 can be an integrated component of the computing device 200 that is located inside the external packaging of the computing device 200 and physically connected to it (e.g., a screen on a mobile phone). In another example, the user interface device 254 can be an external component of the computing device 200 that is located outside of and physically separate from the packaging of the computing device 200 (e.g., a monitor, a projector, etc., that shares a wired and / or wireless data path with a tablet computer).In some examples, the user interface device 254, when located outside the packaging of the computing device 200 and physically separate from it, can collectively refer to two components: a presence-sensitive input device for receiving inputs and a display device for providing outputs.

[0026] One or more storage devices 270 within the computing device 200 can store information for processing during the operation of the computing device 200. In some examples, the storage device 270 is temporary storage; that is, long-term storage is not its primary purpose. The storage device 270 of the computing device 200 can be configured as volatile memory for short-term storage of information, and therefore, stored contents are not retained when the computing device is powered off. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in engineering.

[0027] In some examples, the storage device 270 also includes one or more computer-readable storage media. The storage device 270 can be configured to store larger amounts of information than volatile memory. The storage device 270 can also be configured for long-term storage of information as non-volatile memory, where information is retained after power cycles. Examples of non-volatile memory include magnetic hard disks, optical media, floppy disks, flash memory, or forms of electrically programmable memory (EPROM) or electrically erasable and programmable memory (EEPROM).The storage device 270 can store program instructions and / or data associated with the operating system 272, the UI management module 276, the I / O module 278, the application modules 274A-274N, the content of interest module 280, the gaze direction module 282, the content ID module 284, the application management module 286, the ML module 288, the screen section module 290, the priority module 292, and the graphic keyboard module 294.

[0028] One or more processors 250 can implement functionality and / or execute instructions within the computing device 200. For example, one or more processors 250 can receive and execute instructions stored on the computing device 200 by the storage device 270, which provide the functionality of the operating system 272, the UI management module 276, the I / O module 278, the application modules 274A-274N, the content of interest module 280, the gaze direction module 282, the content ID module 284, the application management module 286, the ML module 288, the screen section module 290, the priority module 292, and the graphic keyboard module 294. These instructions, executed by one or more processors 250, can cause the computing device 200 to store and / or modify information in the storage device 270 during program execution.One or more processors 250 can execute instructions from the operating system 272, the UI management module 276, the I / O module 278, the application modules 274A-274N, the content of interest module 280, the gaze direction module 282, the content ID module 284, the application management module 286, the ML module 288, the screen section module 290, the priority module 292, and the graphic keyboard module 294 to cause the user interface device 254 to display the GUI.That is to say, the operating system 272, the UI management module 276, the I / O module 278, the application modules 274A-274N, the content of interest module 280, the gaze direction module 282, the content ID module 284, the application management module 286, the ML module 288, the screen section module 290, the priority module 292, and the graphic keyboard module 294 can be operated by one or more processors 250 to perform various functions described in this document, including determining sections of the GUI that contain content which has at least a threshold probability of being viewed by the user of the computing device, and causing the user interface device 254 to present the GUI to the user at the user interface device 254. It is understood that the description of the computing device 200 or one or more components thereof (e.g.one or more modules in the storage device 270), which perform one or more operations of the disclosure, may include one or more processors 250 which implement a functionality and / or execute instructions thereof.

[0029] The Content of Interest module 280 can contain instructions executable by one or more processors 250 to determine whether one or more sections of the GUI of the computing device 200 contain content of interest to the user. That is, the Content of Interest module 280 can determine whether one or more sections of the GUI contain content that meets at least a threshold probability that the user of the computing device 200 will view or is currently viewing the content. The Content of Interest module 280 can determine a first section of the GUI that includes the one or more sections of the GUI containing content that meets at least the threshold probability that the user of the computing device 200 will view or is currently viewing the content.The Content of Interest module 280 can also determine a second section of the GUI, which includes one or more sections of the GUI that do not contain content of interest to the user (e.g., do not meet the threshold probability that the user of the computing device 200 will view or is currently viewing the content). After determining the first and second sections of the GUI, the Content of Interest module 280 can send information defining the first and second sections of the GUI to the Screen Section module 290 to determine where in the GUI a graphical keyboard should be displayed.The Content of Interest module 280 can determine whether one or more sections of the GUI contain content of interest, based on output from one or more submodules, including the View Direction module 282, the Application Management module 286, the Content ID module 284, and the ML module 288.

[0030] In determining which sections of the GUI contain content of interest to the user of the computing device 200, the Content of Interest module 280 can use information determined by one or more of the View Direction module 282, the Content ID module 284, or the Application Management module 286. For example, the View Direction module 282 can determine which section(s) of the GUI the user of the computing device 200 is viewing, the Content ID module 284 can determine different types of content included in the GUI, and the Application Management module 286 can determine various information about one or more applications (e.g., one or more Application Modules 274) running on the computing device 200.

[0031] The gaze direction module 282 can determine which sections of the GUI displayed by the user interface device 254 the user is viewing, for example, by tracking the user's eye movements. By tracking such eye movements, the gaze direction module 282 can determine one or more gaze direction points of the user on the GUI displayed by the user interface device 254. The gaze direction module 282 can determine the distance of the user's head or eyes from one or more cameras located on the computing device 200 (e.g., one or more cameras of the input devices 252) and can determine the position of the user's head or eyes in 3D space relative to the one or more cameras.

[0032] In some examples, the gaze direction module 282 can analyze one or more images captured by one or more cameras of the computing device 200 to determine the distance between the one or more cameras and the user's eyes. For example, the gaze direction module 282 can analyze parallax angles between images captured by two or more different cameras, since the position of each of the two or more cameras is known. Using the parallax angles and the distance between the cameras, the gaze direction module 282 determines the distance between one or more of the two or more cameras and the user's eyes.As another example, the gaze direction module 282 can analyze an image captured by a single infrared camera to determine the distortion of the image captured by the infrared camera in order to determine the distance between the infrared camera and the user's eyes.

[0033] Using this specific distance between the one or more cameras and the user's eyes, as well as the position of the user's eyes in the captured image(s), the gaze direction module 282 can position the user's eyes in a three-dimensional space relative to the one or more cameras. That is, the gaze direction module 282 can determine a position of the user's eyes with respect to the computing device 200 (i.e., relative to a position of at least one camera of the computing device 200). The position of the user's eyes is a position within a 3D space defined relative to at least one camera. For example, the 3D space can be spherical and have a center point corresponding to a camera position. In such examples, the position of the user's eyes can be defined by (x,y,z) coordinates, where (0,0,0) is the position of the camera used as the center point of the sphere.Such a coordinate can be described as being located within a "camera-based coordinate system".

[0034] Using the position of the user's eyes and the movement of a pupil or other features of the eye, the gaze direction module 282 can determine one or more gaze direction vectors. In some examples, the gaze direction module 282 can use a tilt, roll, and / or yaw movement of the user's head about a specific axis to determine the one or more gaze direction vectors. Using the one or more gaze direction vectors and the position of the user's eyes relative to the one or more cameras, the gaze direction module 282 determines one or more user gaze direction points on the GUI displayed on a screen of the computing device 200. The gaze direction module 282 can include instructions that can be executed by one or more processors 250 to determine the number, timing, and / or location of gaze direction points on the GUI.The gaze direction module 282 can output the number, time and / or location of gaze direction points on the GUI to the content of interest module 280 to determine whether the sections of the GUI containing the gaze direction points contain content of interest to the user.

[0035] In various cases, the gaze direction module 282 can apply a trained machine learning model (e.g., the ML module 288) to the camera image to determine one or more user gaze direction points on the GUI displayed by a display device of the computing device 200. The machine learning model can be trained using images of users with known gaze direction points on an equivalent GUI displayed on an equivalent computing device. The machine learning model can continuously learn based on user feedback and feedback provided by the gaze direction module 282. For example, the gaze direction module 282 can classify or adjust the parameters of the machine learning model based on updated training data received since the previous time the machine learning model was trained.After training, the ML module 288 can be trained to receive image data as input and to determine one or more viewing direction points of the user on the GUI as output.

[0036] The Content ID module 284 can contain instructions that can be executed by one or more processors 250 to identify content in the GUI running on the computing device 200. For example, the Content ID module 284 can determine whether the GUI contains advertising content, videos or images, an active text field, and / or white space. After determining the content type in the GUI, the Content ID module 284 can output data specifying the content type and its location in the GUI to the Content of Interest module 280.

[0037] The Content ID module (284) can also determine priority levels for GUI content and output these levels to the Content of Interest module (280). The Content ID module can determine priority levels based on specific content types included in the GUI. In some cases, the priority levels can indicate the probability that the user is currently viewing or will view the content of interest. Content of interest with a higher priority level may be of greater interest to the user, while content of interest with a lower priority level is likely to be of less interest and thus be identified before content of interest with a higher priority level for potential obscuration by the graphical keyboard. The Content ID module (284) can output these priority levels to the Content of Interest module (280).

[0038] In response to receiving data specifying the content type from module 284 for Content ID, module 280 for Content of Interest can determine whether the content is of interest to the user, i.e., whether the content meets at least the threshold probability that the user will view or is currently viewing it. For example, module 280 for Content of Interest can receive data specifying advertising content from module 284 for Content ID, determine that the advertising content is not of interest to the user, and determine that the section of the GUI occupied by the advertising content is part of the second section.

[0039] The application management module 286 can contain instructions, executable by one or more processors 250, to determine usage information for each application of one or more currently running applications on the computing device 200. For example, the application management module 286 can determine the period of time that each application of the one or more currently running applications was executed. The application management module 286 can also determine the respective period of time that each of the one or more currently running applications was visible on the display device (e.g., the user interface device 254) of the computing device 200. The application management module 286 can also include instructions for determining a foreground application.The application management module 286 can output data indicating the period of time each application was running, the period of time each application was visible on the display device, and / or whether an application is the foreground application to the content of interest module 280.

[0040] The Content of Interest module 280 can determine whether one or more of the application GUIs included in the Computing Device GUI 200 contain content of interest. After receiving data from the Application Management module 286, the Content of Interest module 280 can determine, based on the data from the Application Management module 286, whether content from one or more application GUIs included in the Computing Device GUI 200 meets the probability threshold that the user is currently viewing or will view the content. The Content of Interest module 280 can also determine whether one or more sections of the GUI occupied by the one or more application GUIs are part of the first section or the second section. For example, the Content of Interest module 280 can determine that the foreground application contains content of interest.In some examples, the Content of Interest module 280 can determine that the application GUI that was visible on the user interface device 254 for the longest period of time contains content of interest.

[0041] The ML module 288 can contain instructions to determine whether one or more sections of the GUI contain content of interest to the user of the computing device 200. The ML module 288 can determine whether one or more sections of the GUI contain content that has at least a threshold probability that the user will view or is currently viewing. The ML module 288 can include one or more machine learning algorithms configured to accept one or more representations of GUI content as input and output whether the content meets the threshold probability that the user is currently viewing or will view it. Examples of machine learning techniques that can be used include various learning styles, such as supervised learning, unsupervised learning, and semi-supervised learning.Examples of machine learning algorithm types include Bayesian algorithms, clustering algorithms, decision tree algorithms, regularization algorithms, regression algorithms, instance-based algorithms, artificial neural network algorithms, deep learning algorithms, dimensionality reduction algorithms, and the like.Various examples of specific algorithms include Bayesian linear regression, boosted decision tree regression and neural network regression, backpropagation neural networks, convolutional neural networks (CNNs), long-short-term memory (LSTM) networks, the Apriori algorithm, K-means clustering, k-nearest neighbor (kNN), learning vector quantization (LVQ), self-organizing map (SOM), locally weighted learning (LWL), ridge regression, the least absolute shrinkage and selection operator (LASSO), plastic network and least-angle regression (LARS), principal component analysis (PCA), and principal component regression. PCR).

[0042] In some examples, one or more machine learning models of the ML module 288 can be configured to accept GUI output instructions as input and, as output, determine one or more sections of the GUI containing content that meets at least the threshold probability that the user is currently viewing or will view. In other examples, one or more machine learning models of the ML module 288 can be configured to accept GUI output instructions as input and, as output, determine one or more sections of the GUI containing content that does not meet at least the threshold probability that the user is currently viewing or will view.In some examples, the one or more machine learning models can determine as output both the one or more sections of the GUI containing content that meets at least the threshold probability, and the one or more sections of the GUI containing content that does not meet at least the threshold probability. The one or more machine learning models of the ML module 288 can be trained using training data stored in the memory device 270. The training data can include multiple inputs to the one or more machine learning models, corresponding to known outputs.

[0043] In some examples, one or more machine learning models of the ML module 288 can be trained to accept as input the output of one or more from the gaze direction module 282, the content ID module 284, and / or the application management module 286. The one or more machine learning models of the ML module 288 can be trained to determine, based on the input received from one or more modules 282, 284, and 286, content of interest in the GUI of the computing device 200 as output. For example, the ML module 288 can accept as input the number and location of gaze direction points from the gaze direction module 282, data specifying content types from the content ID module 284, and / or the period that each application of one or more currently running applications on the user interface device 254 was visible from the application management module 286.Based on the number and location of the viewing direction points, the data specifying content types, and / or the period of time that each application of one or more currently running applications was visible on the user interface device 254, the ML module 288 can output sections of the GUI containing content of interest.

[0044] Using information from one or more of Modules 282, 284, 286, and 288, Module 280 can determine, for content of interest, one or more sizes of the one or more sections of the GUI containing the content that has at least the threshold probability that the user of the computing device 200 will view the content or is currently viewing the content (e.g., the first section) and will view the content of no interest to the user (e.g., the second section), as well as whether the sizes satisfy one or more threshold sizes. In some examples, the second section of the GUI may be too small to display a graphical keyboard without the graphical keyboard obscuring at least some of the content of interest in the first section.That is, the module 280 for content of interest can determine that a size of the second section does not meet a threshold size, where the threshold size is correlated with a minimum-size graphical keyboard.

[0045] In some cases where the threshold size is not met, the Content of Interest (CFI) module (280) can determine whether the priority of the CFI in the first section, which would be obscured by the graphical keyboard, meets a lower priority threshold. In some examples, the CFI module (280) can determine that the first section contains enough content of such low relative importance that the graphical keyboard can be displayed without partitioning the screen. The CFI module (280) can additionally or alternatively adjust the threshold probability to include less content in the first section and recalculate the sizes of the first and second sections based on their priority levels with respect to the updated threshold probability.

[0046] In response to determining that the second section is too small to display the graphical keyboard, and / or that the content that would be obscured by the keyboard is not of sufficiently minor relative importance, Module 280 can determine whether the GUI should enter split-screen mode. To determine whether the GUI should enter split-screen mode, Module 280 can assess whether the content of interest in the first section can be reduced in size without making it too difficult for the user to see. For example, Module 280 can compare an adjusted first section size with a minimum first section size associated with a type of content included in the first section, such as images, videos, advertisements, etc., can have a smaller minimum size than content of other types, such as text, links, input fields, etc.

[0047] In response to a determination that the content of interest in the first section can be reduced in size without making it too difficult to see, the Content of Interest module 280 can instruct the Screen Section module 290 to execute split-screen mode. In response to a determination that the content of interest in the first section cannot be reduced in size without making it too difficult to see, the Content of Interest module 280 can redefine the first and second sections based on priority levels issued by the Content ID module 284, such that the second section is large enough to display the graphical keyboard.

[0048] The screen section module 290 can contain instructions that can be executed by one or more processors 250 to run a split-screen mode on the computing device 200 and to determine where within the GUI a graphical keyboard should be displayed. The screen section module 290 can receive information from the content of interest module 280, specifying a first section of the GUI containing content deemed to be of interest to the user and a second section of the GUI containing content deemed not to be of interest to the user, and instructions on whether or not to run the split-screen mode.For example, as described above, the Content of Interest module 280 can determine that the second section is too small to display a graphical keyboard without the graphical keyboard obscuring at least part of the Content of Interest in the first section, and send instructions to the Screen Section module 290 to execute split-screen mode on the Computing Device 200 to provide more space for displaying the graphical keyboard. To execute split-screen mode, the Screen Section module 290 can divide the GUI into at least one first partition and one second partition, the first partition being smaller than the first section of the GUI before the division and the second partition being larger than the second section of the GUI before the division. The Screen Section module 290 can resize the content for display in the first and second partitions.For example, screen section module 290 can shrink the content of the first section so that it fits into the first partition. The screen section module can determine where within the second section and / or the second partition of the GUI a graphical keyboard should be displayed.

[0049] The graphic keyboard module 294 can output instructions, executable by one or more processors 250, to display a graphic keyboard in the GUI of the computing device 200 for display by the display device. The graphic keyboard module 294 can be a standalone application, a service, or a module running on the computing device 200, and in other examples, the graphic keyboard module 294 can be a subcomponent thereof.For example, the Graphic Keyboard Module 294 may be integrated into a chat or messaging application running on the Computing Device 200, while in other examples, the Graphic Keyboard Module 294 may be a standalone application or subroutine called by an application or operating platform of the Computing Device 200 whenever that application or operating platform requires input functionality via a graphic keyboard. In some examples, the Computing Device 200 may download and install the Graphic Keyboard Module 294 from an application distribution platform (e.g., via the Internet), such as a service provider's application repository. In other examples, the Graphic Keyboard Module 294 may be preloaded during the manufacturing of the Computing Device 200.In various examples, module 294 for graphic keyboards can include instructions for determining a size of the user input device and a corresponding minimum threshold size of the graphic keyboard for display.

[0050] The application modules 274A-274N can contain executable instructions for one or more applications to be run on the computing device 200 by one or more processors 250. For example, the application modules 274A-274N can contain instructions for one or more applications from a messaging application, a browser application, a camera application, a music streaming application, or another application to be run on the computing device 200.

[0051] Fig. 3A and Fig. 3B are conceptual diagrams illustrating exemplary graphical user interfaces before and after the execution of a split-screen mode according to one or more techniques of disclosure. Fig. 3A and Fig. 3B are from the computing device 200. Fig. 2 described. In the example from Fig. 3A includes the GUI 310 content 301 of the application GUI 316. The content 301 can contain media 320, such as an image or a video.

[0052] In many computing devices (e.g., the Computing Device 200), a graphical keyboard as part of the GUI 310 may not appear until the Computing Device 200 determines that the user wants to enter text on the Computing Device 200. For example, the user might reply to a comment on a website in a browser application, or they might tap on an incoming notification for a messaging application. However, once the Computing Device 200 outputs the graphical keyboard for display, the graphical keyboard may obscure content of interest to a user of the Computing Device 200 (e.g., content of interest in the first section 302 of the GUI 310).The computing device 200 can determine whether one or more sections of the GUI 310 contain content of interest to the user, and output the graphical keyboard for display in such a way that it does not (or as little as possible) obscure content of interest to the user.

[0053] The Content of Interest module 280 of the computing device 200 can determine whether one or more sections of the GUI 310 contain content of interest to the user, and based on the one or more sections, determine a first section 302 and a second section 304 of the GUI 310. The Content of Interest module 280 can determine that the first section 302 includes each of the one or more sections of the GUI 310 that contain content with at least a threshold probability of being viewed by the user of the computing device. That is, the first section 302 includes each of the one or more sections of the GUI 310 that contain content with at least a threshold probability that the user of the computing device 200 1) will view the content or 2) is currently viewing the content.The content of interest module 280 can determine that the second section 304 includes any of the one or more sections of the GUI 310 that do not contain content that meets at least the threshold probability. The first section 302 and the second section 304 can be non-overlapping sections of the GUI.

[0054] Module 280 for Content of Interest can determine whether one or more sections of GUI 310 contain content of interest to the user, based on data output by Module 284 for Content ID. Module 284 for Content ID can determine a type of content 301 and output data specifying the type of content 301 to Module 280 for Content of Interest. For example, Module 284 for Content ID can determine that a video or image is included in the graphical user interface. Module 284 for Content ID can examine labels of content 301 within the application GUI 316 of GUI 310. One or more of the labels can indicate that the content 301 within the application GUI 316 contains images or videos, and can also specify the location within the application GUI 316 where such content may be found.Using location data, the Content ID module (284) can determine the position of images or videos in the GUI (310) and the section of the GUI that contains them. In some examples, the Content ID module (284) can determine that content (301) contains an active text field (e.g., an active text field (324)) where the active text field has cursor focus. The Content ID module (284) can output data specifying the type of content (301) and its location in the GUI to the Content of Interest module (280).

[0055] In response to receiving the type and location of content 301 from module 284 for content ID, module 280 for content of interest can determine whether the content is of interest or not. That is, module 280 for content of interest can determine whether the content meets the probability threshold for the user to view or is currently viewing the content. For example, module 280 for content of interest can determine that images or videos meet the probability threshold for the user to view or is currently viewing the content. Users may focus on visual media if it is visible in the GUI 310. Therefore, data in the memory of computing device 200 can indicate that images or videos meet the probability threshold.The Content of Interest module 280 can determine that one or more sections of the GUI 310 containing images and / or video are included in the first section 302. The Content of Interest module 280 can send data specifying the first section 302 and the second section 304 to the screen section module 290 to determine where to output the graphical keyboard 312 for display.

[0056] In the example from Fig. Module 3A can determine, using Content ID module 284, that Content 301 contains an image, a video, text, and a graphical button in the application GUI 316 of GUI 310. Content ID module 284 can output data specifying Content 301 and its locations in GUI 310 to Content of Interest module 280. Content of Interest module 280 can determine that the image, video, text, and digital button constitute Content of Interest and identify the first section 302 based on the sections of GUI 310 that contain the image, video, text, and digital button. Content of Interest module 280 can determine that the remaining space in GUI 310 is the second section 304.

[0057] The Content of Interest module 280 can also determine that active text fields meet the probability threshold for the user to view or currently view the content. The active text field 324 can display the typed text being sent to the active text field 324, which can allow users to actively confirm the correct selection of characters while typing on a graphical keyboard (e.g., the graphical keyboard 312). Therefore, data in the memory of the computing device 200 can indicate that active text fields meet the probability threshold. The Content of Interest module 280 can determine a third section 306 of the GUI 310 that contains the active text field 324. In some examples, the first section 302 can contain the third section 306.The module 280 for content of interest can determine a fourth section 308 of the GUI 310 near or adjacent to the active text field 324, where the second section 302 contains the fourth section 308. In this way, the module 280 for content of interest can identify a section of the GUI 310 near or adjacent to the active text field that can be obscured by the graphical keyboard 312 without obscuring content of interest to the user.

[0058] In some examples, the active text field 324 may not be output for display in the GUI 310 until the computing device 200 determines that the user wants to enter text on the computing device 200. In these cases, the active text field 324 and the graphical keyboard 312 can be output for display approximately simultaneously. The Content of Interest module 280 can determine the size and location of the third section 306 containing the active text field 324 based on application data in one or more applications 274 that contain instructions for the behavior of the active text field.

[0059] In the example from Fig. Module 280 can determine for content of interest that the second section 304 of the GUI 310 is too small to fit into most of the graphical keyboard 312. For example, module 280 can determine the size of the second section 304 and whether the size of the second section 304 satisfies a threshold size. The threshold size can be any size previously defined in the memory of the computing device 200 (e.g., a rectangle with specific pixel dimensions). The threshold size can be a minimum threshold size for the second section 304 to satisfy the condition that most of a graphical keyboard 312 of minimum size can visually overlap the second section 304.

[0060] While the Computing Device 200 can output the Graphic Keyboard 312 for display on the GUI 310 in any size or configuration, certain sizes and / or configurations may be disadvantageous for a user of the Computing Device 200. For example, the Graphic Keyboard 312 may be too small to allow the user to precisely select one or more characters with their fingers. Therefore, the Graphic Keyboard 312 may occupy a minimal amount of space on the GUI 310.

[0061] The graphic keyboard module 294 can determine a minimum size for the graphic keyboard and output data specifying the size to the content of interest module 280. In some examples, the graphic keyboard module 294 can determine the size of a user input device used to select one or more characters of the graphic keyboard 312 and adjust the minimum size of the graphic keyboard 312 based on the determined size of the user input device. One or more sensors of a capacitive touchscreen of the computing device 200 can determine a surface area of ​​interaction between the user input device and the capacitive touchscreen and determine the size of the user input device based on this surface area of ​​interaction.In this way, the graphic keyboard module 294 can specify a larger minimum threshold size for graphic keyboard 312 for larger user input devices (e.g., fingers) and a smaller minimum threshold size for graphic keyboard 312 for smaller user input devices (e.g., pens).

[0062] In response to determining that the second section 304 is too small to fit on most of the graphical keyboard 312 (e.g., the size of the second section 304 does not meet the threshold size), the Content of Interest module 280 can determine whether the screen section module 290 should operate in split-screen mode. To determine whether the screen section module 290 should operate in split-screen mode, the Content of Interest module 280 can determine whether the Content of Interest in the first section can be reduced in size without making the Content of Interest too difficult for the user to see on the GUI 310. One or more minimum size thresholds for the Content of Interest can be stored in the memory of the computing device 200.The Content of Interest (CFI) module 280 can determine whether the size of the resized CFI in a potential split-screen mode meets the minimum size thresholds and, based on this determination, decide whether the screen section module 290 should execute the split-screen mode. The Content of Interest (CFI) module 280 can send data to the screen section module 290 indicating whether the screen section module 290 should execute the split-screen mode.

[0063] In response to the determination that the content of interest in the first section cannot be reduced in size without making it too difficult to see, the Content of Interest module 280 can redefine the first and second sections based on priority levels issued by the Content ID module 284, so that the second section is large enough to display the graphical keyboard. For example, the Content ID module 284 might determine that the GUI 310 contains both still images and playing videos. The Content ID module 284 might determine that still images have a lower priority level than playing videos; that is, still images should be obscured by the playing video. The Content ID module 284 can therefore assign a higher priority level to the playing video than to still images.The Content of Interest (CFI) module (280) can receive one or more priority levels corresponding to one or more sections of the identified content from the Content ID module (284). The Content of Interest module (280) can specify that the Content of Interest with the lowest priority level should be part of the second section (304). For example, the Content of Interest module (280) can receive a priority level for still images in the GUI (310), a priority level for a playing video in the GUI (310), and a priority level for an active text field in the GUI (310). The priority level for still images can be the lowest priority level. The Content of Interest module (280) can specify that the section of the GUI (310) containing the still images is part of the second section (302).The Content of Interest module 280 can repeat this procedure until the second section 302 is large enough to display the graphical keyboard 312. In this way, the Content of Interest module 280 can prioritize the display of content deemed more interesting to the user. The Content of Interest module 280 can send data displaying the updated first section 302 and second section 304 to the screen section module 290.

[0064] The screen section module 290 can receive data from one or more other modules to help it determine where to output the graphical keyboard 312. The screen section module 290 can receive data specifying the first section 302 and the second section 304 from the Content of Interest module 280. In some examples, the data specifying the first section 302 and the second section 304 includes data specifying the third section 306 and / or the fourth section 308. The screen section module 290 can also receive data from the Content of Interest module 280 specifying whether to operate in split-screen mode. The screen section module 290 can also receive data specifying the size of the graphical keyboard 312 from the Graphical Keyboards module 294.Based on the received data, the screen section module 290 can determine where to output the graphic keyboard 312.

[0065] In some examples, the screen section module 290 determines where to output the graphical keyboard 312 based on receiving data indicating that the screen section module 290 should operate in split-screen mode. For example, in response to determining that content of interest in the first section 302 can be shrunk without making the content of interest too difficult to see, the content of interest module 280 can instruct the screen section module 290 to operate in split-screen mode (i.e., send data indicating that the screen section module 290 should operate in split-screen mode). The screen section module 290 can then operate in split-screen mode, as in the example from Fig. 3B shown. To execute split-screen mode, the screen section module 290 can divide the GUI 310 into at least one first partition 323A and a second partition 323B, the second partition 232B being large enough to accommodate a graphic keyboard 312. The screen section module 290 can contain instructions for maintaining and controlling the number of partitions and their arrangement within the GUI 310. In some examples, each partition of the split-screen mode can be a separate GUI output for display. The screen section module 290 can resize at least a portion of the content 301 of interest to fit into the first partition 323A. In some examples, the entire first section 302 is resized before splitting to fit into the first partition 323A. In the example from Fig. 3B has the screen section module 290, the GUI 310, divided into a first partition 323A and a second partition 323B, and reduced the size of a portion of the content of interest to fit into the first partition 323A.

[0066] The screen section module 290 can resize the second section 304 of the GUI 310 within the second partition 323B so that a large portion of the graphical keyboard 312 can visually overlay the second section 304 when output for display in the GUI 310, and can specify that the graphical keyboard 312 should be output in the second section 304. In some examples, the resized second section 304 can occupy the entire second partition 323B. In some examples, the second partition 323B can contain part of the content of interest. In the example from Fig. In partition 3B, screen section module 290 increased the size of the second section 304 and included both the second section 304 and part of the content of interest in the second partition 323B. In the example from Fig. 3B contains the content of interest in the active text field 324.

[0067] The graphic keyboard module 294 can output an updated GUI 310, including the graphic keyboard 312, to one or more processors 250 and the display device of the computing device 200, such that a large portion of the graphic keyboard 312 visually overlays the second section 304 of the GUI 310 in the second partition 323B. In this way, the graphic keyboard 312 can be output for display without obscuring (or obscuring as little as possible) the content of interest to the user. To output the graphic keyboard 312, the UI management module 276 can send information defining an updated GUI 310—including the graphic keyboard 312, according to instructions from the graphic keyboard module 294—to the I / O module 278.Based on the information, the I / O module 278 can cause the user interface device 254 to output the GUI 310 for display on the display device, including the graphic keyboard 312.

[0068] In some examples, the screen section module 290 can determine where to output a graphical keyboard 312 based on receiving data specifying the fourth section 308 in the second section 304. For example, the computing device 200 can provide a better user experience by outputting the graphical keyboard 312 for display in an area of ​​the GUI 310 that is directly adjacent to an active text field (e.g., the active text field 324). In doing so, the computing device 200 can allow users to see both the one or more characters of the graphical keyboard 312 when the user selects it and the typed text being transmitted to the active text field 324. As described above, the fourth section 308 is part of the second section 304 near or adjacent to the active text field 324.The screen section module 290 can determine that the graphic keyboard 312 should be output for display in the fourth section 308 and send instructions to the graphic keyboard module 294 specifying where the graphic keyboard 312 should be output. The I / O module 278 can cause the user interface device 254 to output a GUI 310 for display on the display device, including the graphic keyboard 312, according to information received from the graphic keyboard module 294, such that a large portion of the graphic keyboard 312 visually overlays the fourth section 308.

[0069] Fig. 4A and Fig. 4B are conceptual diagrams illustrating exemplary graphical user interfaces with whitespace before and after the output of a graphical keyboard for display according to one or more techniques of disclosure. Fig. 4A and Fig. 4B are in relation to the computing device 200 from Fig. 2 described. In the example from Fig. 4A includes a GUI 410, content 401 of an application GUI 416, including a blank space 426. The GUI 410 includes a first section 402 and a second section 404.

[0070] A graphical keyboard 412 might not appear as part of the GUI 410 until the computing device 200 determines that the user wants to enter text on the computing device 200. For example, the user might open a word processing application. However, once the computing device 200 outputs the graphical keyboard 412 for display, the graphical keyboard 412 might obscure content of interest to the user (for example, content of interest in the first section 402 of the GUI 410). The computing device 200 can determine whether one or more sections of the GUI 410 contain content of interest to the user and output the graphical keyboard 412 for display in such a way that it does not obscure (or obscures as little as possible) content of interest to the user.

[0071] The Content of Interest module 280 of the computing device 200 can determine whether one or more sections of the GUI 410 contain content of interest to the user and, based on the one or more sections, determine a first section 402 and a second section 404 of the GUI 410. The Content of Interest module 280 can determine that the first section 402 includes any of the one or more sections of the GUI 410 that contain content with at least a threshold probability of being viewed by the user of the computing device 200. The Content of Interest module 280 can determine that the second section 404 includes any of the one or more sections of the GUI 410 that do not contain content with at least the threshold probability. The first section 402 and the second section 404 can be non-overlapping sections of the GUI 410.

[0072] Module 280 for Content of Interest can determine whether one or more sections of the GUI contain content of interest to the user, based on data output by Module 284 for Content ID. Module 284 for Content ID can identify a type of content (401) and output data specifying the type of content (401) to Module 280 for Content of Interest. For example, Module 284 for Content ID can determine that text or whitespace (426) is included in the graphical user interface.

[0073] For example, the application GUI 416 can be a text editing program. A section of the text editing program, displayed on the screen of the computing device 200, can contain text in the body as well as blank spaces 426 where no text is present. Instructions for executing the text editing program can be contained in one or more of the application modules 274A-274N. The Content ID module 284 can examine labels of content 401 within the application GUI 416 of the GUI 410. One or more of the labels can indicate that the content 401 within the application GUI 416 contains text and can also indicate the location within the application GUI 416 where such content can be found. Using the location data, the Content ID module 284 can determine the position of the text within the GUI 410 and the section of the GUI 410 that contains the text.

[0074] In some examples, module 284 can determine for content ID that content 401 contains an active text field, where the active text field has cursor focus. In the example from Fig. 4A and Fig. 4B can determine the module 284 for content ID that the active text field contains the text of the application GUI 416 as well as additional space around the text of the application GUI 416 in which typed text can appear.

[0075] In the example from Fig. 4A can also use the Content ID module 284 to identify the whitespace 426 in the application GUI 416. In some examples, the Content ID module 284 can determine that a section of the GUI 410 containing fewer than a threshold number of features includes whitespace 426. For example, the Content ID module 284 can determine that whitespace 426 contains a section of the GUI 410 display with a monochromatic color scheme (e.g., the background of a web page). In some examples, the Content ID module 284 can determine that whitespace 426 contains a section of the GUI 410 where the color contrast ratio is below a threshold. In some examples, the module 284 for content ID can determine that the empty space 426 contains a section of the GUI 410 that contains less than a threshold number of straight lines (e.g., the edges of an open document in a word processing application).In some examples, the Content ID module 284 can determine that the blank space 426 contains a section of the GUI 410 that includes simple, repeating patterns (e.g., a series of cells in a spreadsheet application). The Content ID module 284 can analyze information received from one or more application modules (e.g., application modules 274A-274N) in the memory of the computing device 200, which includes instructions for outputting the application GUI 416, to determine where the blank space 426 can be displayed in the application GUI 416. The Content ID module 284 can output data specifying the type of content 401 and its location in the GUI to the Content of Interest module 280.

[0076] In response to receiving the type and location of content 401 from module 284 for content ID, module 280 for content of interest can determine whether the content is of interest or not. That is, module 280 for content of interest can determine whether the content meets the threshold probability that the user will view or is currently viewing the content. For example, module 280 for content of interest can determine that the text of the word processor meets the threshold probability that the user will view or is currently viewing the content, and can determine that the section of GUI 410 that displays the text is contained in the first section 402 of GUI 410. The text of a word processor is the main focus of the program. Therefore, data in the memory of computing device 200 can indicate that blocks of text in a word processor meet the threshold probability.A Content of Interest module can determine that one or more sections of GUI 410 containing text are included in the first section 402. The Content of Interest module 280 can determine that the section of GUI 410 containing the blank space 426 is included in the second section 404 and does not meet the threshold probability that the user of the computing device 200 will view or is currently viewing the content.

[0077] The Content of Interest module 280 can determine a third section 406 of the GUI 410 that contains the active text field. In some examples, the first section 402 can contain the third section 406. The Content of Interest module 280 can determine a fourth section 408 of the GUI 410 near or adjacent to the active text field 324, where the second section 404 contains the fourth section 408. In this way, the Content of Interest module 280 can identify a section of the GUI 410 near or adjacent to the active text field that can be obscured by the graphical keyboard 412 without obscuring content of interest for the user. The Content of Interest module 280 can send data specifying the first section 402 and the second section 404 to the screen section module 290 to determine where to output the graphical keyboard 412 for display.

[0078] The screen section module 290 can receive data from the content of interest module 280 specifying the first section 402 and the second section 404, and, based on this data, determine where to output the graphical keyboard 412 for display. The screen section module 290 can specify that the graphical keyboard 412 should be displayed in the second section 404. In this way, the graphical keyboard 412 does not obscure (or obscures as little as possible) the content of interest in the GUI 410.

[0079] In some examples, the data specifying the first section 402 and the second section 404 include data specifying the third section 406 (the section containing the active text field) and / or the fourth section 408 (the segment of the second section 404 near or adjacent to the active text field). The computing device 200 can provide users with a better user experience by outputting a graphical keyboard 412 for display in an area of ​​the GUI 410 near or adjacent to an active text field. The computing device 200 can allow users to see both the one or more characters of the graphical keyboard 412, when the user selects it, and the typed text being transmitted to the active text field. As described above, the fourth section 408 can be included in the second section 404 near or adjacent to the active text field.The screen section module 290 can determine that the graphic keyboard 412 should be output for display in the fourth section 408.

[0080] The screen section module 290 can send instructions to the graphic keyboard module 294 specifying where to output the graphic keyboard 412. The graphic keyboard module 294 can output an updated GUI 410, including the graphic keyboard 412, such that a large portion of the graphic keyboard 412 visually overlays the second section 404 and / or the fourth section 408 of the GUI 410, according to data received from the screen section module 290. In this way, the graphic keyboard 412 can be output for display without obscuring (or obscuring as little as possible) the content of interest to the user.

[0081] Fig. Figure 5 is a conceptual diagram illustrating the exemplary computing device 200, which is configured to track a user's gaze direction according to one or more techniques of revelation. Fig. 5 is in relation to the computing device 200 from Fig. 2 described. In the example from Fig. 5 includes a GUI 510, a variety of viewing direction points 530A-530N (together viewing direction points 530) and advertising content 528A-528B (together advertising content 528).

[0082] A graphical keyboard 512 may not be displayed as part of the GUI 510 until the computing device 200 determines that the user wants to enter text on the computing device 200. However, once the computing device 200 outputs the graphical keyboard 512 for display, the graphical keyboard 512 may obscure content of interest to the user (e.g., content of interest in an initial section 502 of the GUI 510). The computing device 200 can determine whether one or more sections of the GUI 510 contain content of interest to the user and output the graphical keyboard 512 for display in such a way that it does not obscure (or obscures as little as possible) content of interest to the user.

[0083] The Content of Interest module 280 of the computing device 200 can determine whether one or more sections of the GUI 510 contain content of interest to the user and, based on the one or more sections, determine a first section 502 and a second section 504 of the GUI 510. The Content of Interest module 280 can determine that the first section 502 includes any of the one or more sections of the GUI 510 that contain content with at least a threshold probability of being viewed by the user of the computing device 200. The Content of Interest module 280 can determine that the second section 504 includes any of the one or more sections of the GUI 510 that do not contain content with at least the threshold probability. The first section 502 and the second section 504 can be non-overlapping sections of the GUI 510.

[0084] Module 280 for Content of Interest can determine whether one or more sections of GUI 510 contain content of interest to the user, based on data output by Module 284 for Content ID. Module 284 for Content ID can identify a type of content 501 and output data specifying the type of content 501 to Module 280 for Content of Interest. For example, Module 284 for Content ID can determine that advertising content 528 is included in GUI 510. Advertising content can include advertisements, sponsored content, third-party content for the application, or any other type of promotional content. In the example from Fig. 5 includes the GUI 510 advertising content 528A and 528B.

[0085] For example, an application GUI in GUI 510 can be a web browser. A section of the web browser can contain advertisements. Instructions for running the web browser can be included in one or more of the application modules 274A-274N. Module 284 for Content ID can examine labels of the content within the web browser. One or more of the labels can indicate that content within the web browser contains the advertising content 528, and can also indicate the location within the web browser where such content may be found. Using the location data, Module 284 for Content ID can determine the position of the advertising content 528 in GUI 510 and the section of GUI 510 that contains the advertising content.

[0086] In response to receiving the type and location of content 501 from module 284 for Content ID, module 280 for Content of Interest can determine whether the content is of interest or not. That is, module 280 for Content of Interest can determine whether the content meets the probability threshold for the user to view or is currently viewing it. For example, module 280 for Content of Interest can determine that advertising content does not meet the probability threshold for the user to view or is currently viewing it, and can determine that the section of GUI 510 displaying the advertising content is contained within the second section 504 of GUI 510.

[0087] The Content of Interest module 280 can determine whether one or more sections of the GUI 510 contain content of interest to the user, based on data output by the Gaze Direction module 282. The input devices 252 of the computing device 200 can include one or more cameras. The Gaze Direction module 282 can determine which sections of the GUI 510 the user is viewing, for example, by tracking the user's eye movements using the one or more cameras. By tracking such eye movements, the Gaze Direction module 282 can determine the user's one or more gaze direction points 530 on the GUI 510. The Gaze Direction module 282 can determine the one or more gaze direction points 530 as described above with respect to Fig. 2 described.

[0088] The Content of Interest module 280 can determine—based on data indicating the user's eye movements, received by the Gaze Direction module 282—a probability that the user will view or is currently viewing at least one section of the GUI 510's content. Based on the number and / or timing of the Gaze Direction points 530 on the GUI 510, the Content of Interest module 280 can determine whether the GUI 510's content meets the threshold probability that the user is currently viewing or will view the content.

[0089] The gaze direction module 282 can determine, based on the user's eye movements, the number of times the user views content contained in a third section 506 of the GUI 510. For example, the gaze direction module 282 can determine a plurality of gaze direction points 530. For each section of a plurality of sections of content contained in the GUI 510, the gaze direction module 282 can determine a number of gaze direction points 530 located in that section. For example, the gaze direction module 282 can determine a number of gaze direction points 530 located in the third section 506. The gaze direction module 282 can also determine a plurality of timestamps corresponding to each gaze direction point of the gaze direction points 530, where each gaze direction point of the gaze direction points 530 corresponds to one timestamp of the plurality of timestamps.The viewing direction module 282 can output the number of viewing direction points 530, their location in the GUI 510 and the multitude of timestamps corresponding to the viewing direction points 530 to the module 280 for content of interest.

[0090] In response to receiving the number, location, and timestamps associated with the view direction points 530, the Content of Interest module 280 can determine whether the content at these locations in the GUI 510 is of interest or not. That is, the Content of Interest module 280 can determine whether the content in these sections of the GUI 510 meets the probability threshold for the user to view or currently view the content.

[0091] For example, if the number of gaze direction points 530 located in the third section 506 of the GUI 510 meets a threshold, the Content of Interest module 280 can determine that the probability that the user will view, or is currently viewing, content in the third section 506 meets the threshold probability. The Content of Interest module 280 can receive data from the gaze direction module 282 indicating that eleven gaze direction points occurred in the third section 506. A threshold stored in the memory of the computing device 200 can indicate that any number of gaze direction points equal to or greater than ten meets the threshold probability. In response to determining that the probability meets the threshold probability, the Content of Interest module 280 can determine that the third section 506 is contained within the first section 502.

[0092] In some examples, the Content of Interest module 280 can determine a proportion of the gaze direction points 530 that are located in the third section 506. The Content of Interest module 280 can separate the number of gaze direction points 530 in the third section 506 from the total number of gaze direction points within a given time period. For example, the gaze direction module 282 can determine 100 gaze direction points within the last five minutes. Sixty gaze direction points may be located in the third section 506, or sixty percent of the determined gaze direction points. The time period may be a previously defined time interval in memory or a time interval since the gaze direction module 282 began determining gaze direction points. The Content of Interest module 280 can determine whether the proportion of gaze direction points 530 in the third section 506 meets a threshold.For example, sixty percent of the determined gaze direction points may be located in the third section 506, and any percentage above fifty may meet the threshold proportion according to the data in the memory of the computing device 200. In response to the fact that the proportion of gaze direction points 530 in the third section 506 meets the threshold proportion, the module 280 for content of interest may determine that the content of the third section 506 meets the threshold possibility that the user will view or is currently viewing the content, and that the third section 506 is included in the first section 502.

[0093] The Content of Interest module 280 can determine, based on the timestamps of the gaze direction points 530, whether content in the GUI 510 meets the threshold probability that the user is currently viewing or will view the content. For each gaze direction point of the plurality of gaze direction points 530, the Content of Interest module 280 can determine whether a difference between the corresponding timestamp for each gaze direction point and a current time (e.g., received from a system clock application of applications 274A-274N) meets a threshold period. In response to determining that the time difference meets a threshold period, the Content of Interest module 280 can determine that the section of the GUI 510 containing the gaze direction point with the corresponding timestamp meets the threshold probability that the user is currently viewing or will view the content of that section.For example, the difference between a corresponding timestamp for a gaze direction point and the current time could be five seconds. Data in memory can indicate that time differences of less than twenty seconds meet the threshold probability. The Content of Interest module 280 can determine that the section of GUI 510 containing the gaze direction point with the corresponding timestamp that meets the threshold probability is included in the first section 502. In this way, the Content of Interest module 280 can determine that recently viewed sections of GUI 510 meet the threshold probability that the user will view or is currently viewing these sections, and that these sections are included in the first section 502.Module 280 for Content of Interest can determine that sections of the GUI 510 that have not been viewed within a threshold period do not meet the probability threshold for the user to view or currently view those sections, and that these sections are included in the second section 504. For example, data in memory can indicate that time differences greater than five minutes do not meet the probability threshold for the user to view or will view the content.

[0094] The Content of Interest module 280 can send data specifying the first section 502 and the second section 504 to the Screen Section module 290 to determine where to output the graphical keyboard 512 for display. The Screen Section module 290 can receive data specifying the first section 502 and the second section 504 from the Content of Interest module 280 and, based on this data, determine where to output the graphical keyboard 512 for display. The Screen Section module 290 can specify that the graphical keyboard 512 should be displayed in the second section 504. In this way, the graphical keyboard 512 does not obscure (or obscures as little as possible) the Content of Interest in the GUI 510. The Screen Section module 290 can send instructions to the Graphical Keyboard module 294 regarding where to output the graphical keyboard 512.The graphic keyboard module 294 can output an updated GUI 510, including the graphic keyboard 512, such that a large part of the graphic keyboard 512 visually overlays the second section 504 of the GUI 510 according to data received from the screen section module 290.

[0095] Fig. 6A and Fig. 6B are conceptual diagrams illustrating exemplary graphical user interfaces, which include user interfaces for a variety of applications according to one or more techniques of disclosure. Fig. 6A and Fig. 6B are in relation to the computing device 200 from Fig. 2 described. In the example from Fig. 6A includes a GUI, 610 application GUIs 616A-616C. Although in Fig. 6A and Fig. While three application GUIs 616A-616C are shown in example 6B, GUI 610 can output any number of application GUIs in other examples. In some examples, the section of GUI 610 displaying application GUI 616A may include a playing video, the section of GUI 610 displaying application GUI 616B may include a message transmission interface, and the section of GUI 610 displaying application GUI 616C may include one or more animated graphics. Instructions for each of the application GUIs 616 may be contained in one or more application modules 274A-274N.

[0096] The Content of Interest module 280 of the computing device 200 can determine whether one or more sections of the GUI 610 contain content of interest to the user and, based on the one or more sections, determine a first section 602 and a second section 604 of the GUI 610. The Content of Interest module 280 can determine that the first section 602 includes any of the one or more sections of the GUI 610 that contain content with at least a threshold probability of being viewed by the user of the computing device 200. The Content of Interest module 280 can determine that the second section 604 includes any of the one or more sections of the GUI 610 that do not contain content with at least the threshold probability. The first section 602 and the second section 604 can be non-overlapping sections of the GUI 610.

[0097] The Content of Interest module 280 can determine whether one or more sections of the GUI 610 contain content of interest to the user, based on data output by the Application Management module 286. The Application Management module 286 can determine usage information for each application of one or more currently running applications on the Computing Device 200 (e.g., the applications that output the Application GUIs 616). The Application Management module 286 can output data specifying the usage information to the Content of Interest module 280. For example, the Application Management module 286 can output data specifying the duration that each application was running, the duration that each application GUI was visible on the display device, and / or whether an application is the foreground application to the Content of Interest module 280.

[0098] Application Management Module 286 can determine the time period during which one or more currently running applications, which output Application GUIs 616, have been running. Application Management Module 286 can examine data from the one or more Application Modules 274A-274N corresponding to each of the one or more currently running applications to obtain an application runtime. Application Management Module 286 can examine data from the one or more Application Modules 274A-274N corresponding to each of the one or more currently running applications to obtain a timestamp corresponding to the start time of each application and compare the start time to a current point in time (e.g., a current time obtained from a system clock). Application Management Module 286 can output the time period during which the applications have been running to Module 280 for content of interest.

[0099] The application management module 286 can determine the period during which one or more currently running application GUIs 616 were visible on the display device that shows GUI 610. The application management module 286 can determine that application GUI 616A was visible on the display device for one hour and fourteen minutes, application GUI 616B was visible on the display device for twenty minutes, and application GUI 616C was visible on the display device for ten minutes. The application management module 286 can output the period during which each application GUI was visible on the display device to module 280 for content of interest.

[0100] In some examples, the Application Management Module 286 can determine which running application is a foreground application. That is, the Application Management Module 286 can determine which task, process, application, window, etc., the user is currently using (e.g., actively interacting with the user, holding cursor focus, etc.). The Application Management Module 286 can output data specifying the foreground application to the Content of Interest Module 280.

[0101] In response to receiving application usage data from the application management module 286, the Content of Interest module 280 can determine whether content associated with each currently running application is Content of Interest (e.g., whether it meets the probability threshold that the user will view or is currently viewing the content). For example, the Content of Interest module 280 can determine whether content associated with each currently running application is Content of Interest based on whether an application is the foreground application. The application GUI with which a user is actively interacting is likely to be an application GUI with which the user will want to continue interacting. Data in the memory of the compute device 200 can indicate that the foreground application meets the probability threshold that the user will view or is currently viewing the content.Module 280 for Content of Interest can determine that the section of GUI 610, which includes an application GUI that is the foreground application, meets the threshold probability and is included in the first section 602.

[0102] In some examples, the Content of Interest module 280 can determine whether content associated with each currently running application is content of interest, based on the respective period of time that each application GUI 616 was visible on the display device by the one or more currently running applications.

[0103] The Content of Interest module 280 can determine, based on the duration that one or more application GUIs 616 were visible on the display device, which application GUI(s) 616 were visible for the longest period. The application management module 286 can determine that content associated with an application GUI meets the probability threshold for the user to view or currently view that content. For example, the application management module 286 can determine that content from application GUI 616A, which was visible on the display device for one hour and fourteen minutes, meets the probability threshold. Application GUIs 616 that were visible on the display device for a longer period can indicate that the user is actively interacting with the content of those applications.For example, the application GUI 616A can contain a playing movie or documentary that may contain content of interest to the user for the entire duration of the movie / documentation. The content of interest module can determine that the section of GUI 610 containing the application GUI 616A is included in the first section 602.

[0104] The Content of Interest module can determine, based on the respective time periods that one or more application GUIs 616 were visible on the display device, which application GUI was visible for the shortest period. The Application Management module 286 can determine that content associated with an application GUI does not meet the probability threshold for the user to view or currently view the content. For example, the Application Management module 286 can determine that the application GUI 616C, which was visible on the display device for ten minutes, does not meet the probability threshold. An application that was not visible for a very long time may not contain content that is of interest to a user for more than a brief moment.For example, the application GUI 616C might contain a weather application that is only of interest to a user for as long as the user needs to determine whether it will rain that day. The Content of Interest module 280 can specify that the section of GUI 610 containing the application GUI 616C is included in the second section 604.

[0105] In some examples, the Application Management Module 286 can determine that the application GUI that was visible on the display device for the shortest period of time contains content that meets the probability threshold for the user to view or currently view that content. The Application Management Module 286 can determine that the application GUI 616B, which was visible on the display device for sixty seconds, meets the probability threshold. An application GUI 616B that the user recently opened may contain content of interest to the user; for example, the user may not have had time to perform intended interactions with the application GUI 616B. For instance, the application GUI 616B may contain a recently opened messaging application with unread messages. An application that the user recently opened may also be the foreground application.Module 280 for content of interest can determine that the section of GUI 610, which includes the application GUI 616B, is included in the first section 602.

[0106] The Content of Interest module 280 can send data specifying the first section 602 and the second section 604 to the Screen Section module 290 to determine where to output a graphical keyboard 612 for display. The Screen Section module 290 can receive data specifying the first section 602 and the second section 604 from the Content of Interest module 280 and, based on this data, determine where to output the graphical keyboard 612 for display. The Screen Section module 290 can specify that the graphical keyboard 612 should be displayed in the second section 604. In this way, the graphical keyboard 612 does not obscure (or obscures as little as possible) the Content of Interest in the GUI 610. The Screen Section module 290 can send instructions to the Graphical Keyboard module 294 regarding where to output the graphical keyboard 612.The graphic keyboard module 294 can output an updated GUI 610, including the graphic keyboard 612, such that a large part of the graphic keyboard 612 visually overlays the second section 604 of the GUI 610 according to data received from the screen section module 290.

[0107] Fig. Figure 7 is a flowchart illustrating exemplary operations for displaying a graphical keyboard within a graphical user interface according to one or more techniques of the disclosure. For better description, the operations can be related to components from Fig. 2 will be described.

[0108] One or more processors 250 of the computing device 200 can be used for display by the user interface device 254, a GUI, such as the GUI 110 from Fig. 1, the GUI 310 from Fig. 3A, the GUI 410 from Fig. 4A, the GUI 510 from Fig. 5 or the GUI 610 from Fig. 6A, output as non-restrictive examples (702). The GUI can contain a variety of content, such as text, images, video, graphics, etc. For example, content can include the words of a text document, the images of a news website, a streaming movie, a file in Graphics Interchange format in an instant messaging application, and / or buttons of a digital game. The content can be part of one or more currently running applications of the GUI.

[0109] The Content of Interest module 280 can determine the content of interest in the GUI (704). A portion of the GUI content may be of interest to the user of the computing device 200. For example, the user may be currently viewing the content, interacting with the content, and / or viewing the content soon. A portion of the content may not be of interest to a user, or may be of less interest than other content in the GUI. In some examples, the Content of Interest module 280 determines content of interest based on eye-tracking data received from the Eye-Direction module 282, the Content ID module 284, the Application Management module 268, and / or the Machine Learning module 288. The Content of Interest module 280 can determine that a first section of the GUI contains content of interest, and a second section of the GUI contains content that is not of interest to the user, or is of less interest.

[0110] Module 280 for Content of Interest can determine whether the graphical user interface (GUI) should be split (706). To determine whether the GUI should be split, Module 280 for Content of Interest can determine a size for the first section and a size for the second section, and whether the sizes satisfy one or more threshold sizes. In some examples, the second section of the GUI might be too small to display a graphical keyboard without the graphical keyboard obscuring at least some of the content of interest in the first section. That is, Module 280 for Content of Interest can determine that a size for the second section does not satisfy a threshold size, where the threshold size is correlated with a minimum-sized graphical keyboard.

[0111] In response to the determination by module 280 that the size of the second section does not meet the threshold size for content of interest, the screen section module 290 can implement a split-screen mode in the GUI (JA branch of 706). The screen section module 290 can split the GUI into at least one first partition and one second partition (708). The screen section module 290 can contain the content of the first section of the GUI within the first partition and the content of the second section of the GUI within the second partition. The first partition can be smaller on the GUI than the first section was before the split. The screen section module 290 can resize the content within the first section of the GUI to fit the smaller first partition. The second partition can be larger on the GUI than the second section was before the split.The second partition after the split can be large enough that a large portion of the graphical keyboard can visually overlay the second partition.

[0112] The UI Management Module 276 can cause the User Interface Device 254 to output an updated GUI for display on the User Interface Device 254, including the graphical keyboard, according to information received from the Graphical Keyboard Module 294, such that a large portion of the graphical keyboard visually overlays the second partition (710). For example, the UI Management Module 276 of the Computing Device 200 can update the GUI to include a first GUI in the first partition and at least one second GUI in the second partition. The first GUI can contain the content of interest from the GUI prior to the partitioning, and the second GUI can contain the graphical keyboard.

[0113] In response to determining that the size of the second section meets the threshold size, the procedure cannot involve splitting the GUI (NO branch of 706). That is, the Content of Interest module 280 can determine a second section of the GUI that contains no content of interest and that the second section is large enough to display a graphical keyboard. The UI Management module 276 can cause the User Interface Device 254 to output an updated GUI for display on the User Interface Device 254, including the graphical keyboard, according to information received from the Graphical Keyboards module 294, such that a large portion of the graphical keyboard visually overlays the second section of the GUI (712).

[0114] Various aspects of the techniques described in this revelation may be directed at the examples listed below.

[0115] Example 1: Method comprising: outputting, by one or more processors of a computing device and for display by a display device, a graphical user interface; determining, by the one or more processors, that a first section of the graphical user interface contains content that has at least a threshold probability of being viewed by the user of the computing device, wherein the graphical user interface further comprises a second section, wherein the first section and the second section are non-overlapping sections of the graphical user interface;and output, by the one or more processors and for display by the display device, an updated graphical user interface, including a graphical keyboard, such that a large part of the graphical keyboard visually overlays the second section of the graphical user interface.;

[0116] Example 2: Method according to Example 1, wherein the second section of the graphical user interface does not contain any content which has at least a threshold probability of being viewed by the user of the computing device.

[0117] Example 3: A method according to Example 1 or 2, further comprising: determining, by one or more processors, whether a size of the second section of the graphical user interface satisfies a threshold size; and in response to determining that the size of the second section does not satisfy the threshold size, splitting, by one or more processors, the graphical user interface into at least a first partition and a second partition, wherein: the first section of the graphical user interface is included in the first partition of the graphical user interface; and the graphical keyboard is included in the second partition of the graphical user interface.

[0118] Example 4: A method according to any one of Examples 1 to 3, wherein the content included in the first section of the graphical user interface is a first section of content from a plurality of sections of content included in the graphical user interface, the method further comprising: determining, by the one or more processors, whether the user has viewed a respective section of content from the plurality of sections of content within a threshold period; and in response to determining that the user has not viewed the respective section within the threshold period, determining, by the one or more processors, that the respective section of content of the graphical user interface is part of the second section of the graphical user interface.

[0119] Example 5: A method according to any one of Examples 1 to 4, wherein determining that the first section of the graphical user interface contains content that has at least the threshold probability of being viewed by the user of the computing device comprises: tracking, by the one or more processors, of the user's eye movements; determining, by the one or more processors and based on the eye movements, a probability that the user will view or is currently viewing at least one section of the content of the graphical user interface; determining, by the one or more processors, whether the probability satisfies the threshold probability;and in response to determining that the probability satisfies the threshold probability, determining, by the one or more processors, that the section of the graphical user interface content contains content that has at least a threshold probability of being viewed by the user of the computing device.

[0120] Example 6: A method according to any one of Examples 1 to 5, further comprising: tracking, by the one or more processors, of the user's eye movements; determining, by the one or more processors and based on the user's eye movements, the number of times the user views content contained in a third section of the graphical user interface; determining, by the one or more processors, whether the count meets a threshold count; and in response to determining that the count does not meet the threshold count, determining, by the one or more processors, that the third section does not contain any content that meets the threshold probability of being viewed by the user of the computing device.

[0121] Example 7: A method according to any one of Examples 1 to 6, further comprising: determining, by one or more processors, whether advertising content is included in the graphical user interface; and in response to determining that advertising content is included in the graphical user interface, determining, by one or more processors, that the space on the graphical user interface occupied by the advertising content does not include any content that meets the threshold probability of being viewed by the user of the computing device.

[0122] Example 8: A method according to any one of Examples 1 to 7, further comprising: determining, by one or more processors, whether a video or image is included in the graphical user interface; and in response to determining that a video or image is included in the graphical user interface, determining, by one or more processors, that the video or image contains content that satisfies the threshold probability of being viewed by the user of the computing device.

[0123] Example 9: A method according to any one of Examples 1 to 8, further comprising: determining, by the one or more processors, whether a third section of the graphical user interface contains an active text field, wherein the active text field is a text field that has a current cursor focus; and in response to determining that the third section contains the active text field, determining, by the one or more processors, that the third section contains content that satisfies the threshold probability of being viewed by the user of the computing device.

[0124] Example 10: The method of Example 9, further comprising: determining, by the one or more processors, whether a fourth section of the graphical user interface contains content that meets the threshold probability of being viewed by the user of the computing device, wherein the fourth section is adjacent to the active text box; and in response to determining that the fourth section does not contain content that meets the threshold probability of being viewed by the user of the computing device, outputting, by the one or more processors and for display by the display device, an updated graphical user interface, including a graphical keyboard, such that a large portion of the graphical keyboard visually overlays the fourth section of the graphical user interface.

[0125] Example 11: A method according to any one of Examples 1 to 10, further comprising: determining, by the one or more processors, the respective period of time that each application from one or more currently running applications was visible on the display device of the computing device; identifying, by the one or more processors and based on the respective period of time that the one or more currently running applications were visible on the display device, one application from the one or more currently running applications that was visible on the display device for the longest period of time; and determining, by the one or more processors, that content associated with the one application includes content that meets the threshold probability of being viewed by the user of the computing device.

[0126] Example 12: Method according to Examples 1 to 11, further comprising: Determining, by the one or more processors, whether a third section of the graphical user interface contains whitespace; and in response to determining that the third section of the graphical user interface contains whitespace, determining, by the one or more processors, that the third section does not contain any content which satisfies the threshold probability of being viewed by the user of the computing device.

[0127] Example 13: A computing system includes a display device, one or more processors, and a memory for storing instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to output a graphical user interface for display by the display device and to determine that a first section of the graphical user interface contains content that has at least a threshold probability of being viewed by a user of the computing device, wherein the graphical user interface further includes a second section, and wherein the first section and the second section are non-overlapping sections of the graphical user interface.The instructions can further instruct one or more processors to output an updated graphical user interface, including a graphical keyboard, for display by the display device, such that a large portion of the graphical keyboard visually overlays the second section of the graphical user interface.

[0128] Example 14: Computing system according to Example 13, wherein the second section of the graphical user interface does not contain any content which has at least a threshold probability of being viewed by the user of the computing device.

[0129] Example 15: Computing system according to Example 13 or 14, wherein the instructions further cause the one or more processors to: determine whether a size of the second section of the graphical user interface satisfies a threshold size; and in response to determining that the size of the second section does not satisfy the threshold size, divide the graphical user interface into at least a first partition and a second partition, wherein the first section of the graphical user interface is included in the first partition of the graphical user interface and wherein the graphical keyboard is included in the second partition of the graphical user interface.

[0130] Example 16: Computing system according to any of Examples 13 to 15, wherein the content included in the first section of the graphical user interface is a first section of content from a plurality of sections of content included in the graphical user interface, and wherein the instructions further cause the one or more processors to: determine whether the user has viewed a particular section of content from the plurality of sections of content within a threshold period; and in response to determining that the user has not viewed the particular section within the threshold period, determine that the particular section of content of the graphical user interface is part of the second section of the graphical user interface.

[0131] Example 17: Computing system according to any one of Examples 13 to 16, wherein the instructions causing the one or more processors to determine that the first section of the graphical user interface contains content that has at least the threshold probability of being viewed by the user of the computing device, further include instructions causing the one or more processors to: track the user's eye movements; determine, based on the eye movements, a probability that the user will view or is currently viewing at least one section of the content of the graphical user interface;Determine whether the probability meets the threshold probability, and in response to determining that the probability meets the threshold probability, determine that the section of the graphical user interface content contains content that has at least a threshold probability of being viewed by the user of the computing device.

[0132] Example 18: Computing system according to any of Examples 13 to 17, wherein the instructions further cause the one or more processors to: track user eye movements; determine, based on the user eye movements, the number of times the user views content contained in a third section of the graphical user interface; determine whether the count meets a threshold count; and, in response to determining that the count does not meet the threshold count, determine that the third section does not contain any content that meets the threshold probability of being viewed by the user of the computing device.

[0133] Example 19: Computing device according to any of Examples 13 to 18, wherein the instructions further cause the one or more processors to: Determine whether advertising content is included in the graphical user interface; and in response to determining that advertising content is included in the graphical user interface, determine that the space on the graphical user interface occupied by the advertising content does not contain any content that meets the threshold probability of being viewed by the user of the computing device.

[0134] Example 20: Computing device according to any of Examples 13 to 19, wherein the instructions further cause the one or more processors to: determine whether a video or image is included in the graphical user interface; and in response to determining that a video or image is included in the graphical user interface, determine that the video or image contains content that meets the threshold probability of being viewed by the user of the computing device.

[0135] Example 21: Computing device according to any of Examples 13 to 20, wherein the instructions further cause the one or more processors to: Determine whether a third section of the graphical user interface contains an active text field, wherein the active text field is a text field that has a current cursor focus; and in response to determining that the third section contains the active text field, determine that the third section contains content that meets the threshold probability of being viewed by the user of the computing device.

[0136] Example 22: Computing system according to Example 21, wherein the instructions further cause the one or more processors to: determine whether a fourth section of the graphical user interface contains content that meets the threshold probability of being viewed by the user of the computing device, wherein the fourth section is adjacent to the active text box; and in response to determining that the fourth section does not contain content that meets the threshold probability of being viewed by the user of the computing device, output, for display by the display device, an updated graphical user interface, including a graphical keyboard, such that a large portion of the graphical keyboard visually overlays the fourth section of the graphical user interface.

[0137] Example 23: Computing system according to any of Examples 13 to 22, wherein the one or more processors are further configured to: determine the respective period of time that each application from among the currently running applications has been in the foreground; identify, based on the respective period of time that the one or more currently running applications have been in the foreground, one application from among the one or more currently running applications that has been in the foreground for the longest period of time; and determine that content associated with the one application includes content that meets the threshold probability of being viewed by the user of the computing device.

[0138] Example 24: Computing system according to any of Examples 13 to 23, wherein the instructions further cause the one or more processors to: Determine whether a third section of the graphical user interface contains whitespace; and in response to determining that the third section of the graphical user interface contains whitespace, determine that the third section does not contain any content that satisfies the threshold probability of being viewed by the user of the computing device.

[0139] Example 25: A computer-readable storage medium is encoded with instructions which, when executed, cause one or more processors of a computing device to: output, for display by a display device of the computing device, a graphical user interface and determine that a first section of the graphical user interface contains content which has at least a threshold probability of being viewed by a user of the computing device, wherein the graphical user interface further includes a second section, wherein the first section and the second section are non-overlapping sections of the graphical user interface.The instructions can further instruct one or more processors to output an updated graphical user interface, including a graphical keyboard, for display by the display device, such that a large portion of the graphical keyboard visually overlays the second section of the graphical user interface.

[0140] Example 26: Computer-readable storage medium according to Example 25, wherein the second section of the graphical user interface does not contain any content which has at least a threshold probability of being viewed by the user of the computing device.

[0141] Example 27: A computer-readable storage medium according to Example 25 or 26, wherein the instructions further cause the one or more processors to determine whether a size of the second section of the graphical user interface satisfies a threshold size; and in response to determining that the size of the second section does not satisfy the threshold size, to partition the graphical user interface into at least a first partition and a second partition, wherein: the first section of the graphical user interface is included in the first partition of the graphical user interface; and the graphical keyboard is included in the second partition of the graphical user interface.

[0142] Example 28: A computer-readable storage medium according to any of Examples 25 to 27, wherein the content included in the first section of the graphical user interface is a first section of content from a plurality of sections of content included in the graphical user interface, and wherein the instructions further cause the one or more processors to: determine whether the user has viewed a particular section of content from the plurality of sections of content within a threshold period; and in response to determining that the user has not viewed the particular section within the threshold period, determine that the particular section of content of the graphical user interface is part of the second section of the graphical user interface.

[0143] Example 29: Computer-readable storage medium according to any one of Examples 25 to 28, wherein the instructions cause the one or more processors to determine that the first section of the graphical user interface further includes instructions that cause the one or more processors to determine content that has at least the threshold probability of being viewed by the user of the computing device, and wherein the instructions further cause the one or more processors to: track the user's eye movements; determine, based on the eye movements, a probability that the user will view or is currently viewing at least one section of the content of the graphical user interface;Determine whether the probability meets the threshold probability, and in response to determining that the probability meets the threshold probability, determine that the section of the graphical user interface content contains content that has at least a threshold probability of being viewed by the user of the computing device.

[0144] Example 30: A computer-readable storage medium according to any of Examples 25 to 29, wherein the instructions further cause the one or more processors to: track the user's eye movements; determine, based on the user's eye movements, the number of times the user views content contained in a third section of the graphical user interface; determine whether the count meets a threshold count; and, in response to determining that the count does not meet the threshold count, determine that the third section does not contain any content that meets the threshold probability of being viewed by the user of the computing device.

[0145] Example 31: Computer-readable storage medium according to any of Examples 25 to 30, wherein the instructions further cause the one or more processors to: determine whether advertising content is included in the graphical user interface; and in response to determining that advertising content is included in the graphical user interface, determine that the space on the graphical user interface occupied by the advertising content does not contain any content that meets the threshold probability of being viewed by the user of the computing device.

[0146] Example 32: Computer-readable storage medium according to any of Examples 25 to 31, wherein the instructions further cause the one or more processors to: determine whether a video or image is included in the graphical user interface; and in response to determining that a video or image is included in the graphical user interface, determine that the video or image contains content that meets the threshold probability of being viewed by the user of the computing device.

[0147] Example 33: Computer-readable storage medium according to any of Examples 25 to 32, wherein the instructions further cause the one or more processors to: determine whether a third section of the graphical user interface contains an active text field, wherein the active text field is a text field that has a current cursor focus; and in response to determining that the third section contains the active text field, determine that the third section contains content that meets the threshold probability of being viewed by the user of the computing device.

[0148] Example 34: Computer-readable storage medium according to Example 33, wherein the instructions further cause the one or more processors to: determine whether a fourth section of the graphical user interface contains content that meets the threshold probability of being viewed by the user of the computing device, wherein the fourth section is adjacent to the active text box; and in response to determining that the fourth section does not contain content that meets the threshold probability of being viewed by the user of the computing device, output, for display by the display device, an updated graphical user interface, including a graphical keyboard, such that a large portion of the graphical keyboard visually overlays the fourth section of the graphical user interface.

[0149] Example 35: A computer-readable storage medium according to any of Examples 25 to 34, wherein the instructions further cause the one or more processors to: determine a respective period of time that each application from among the currently running applications has been in the foreground; identify, based on the respective period of time that the one or more currently running applications have been in the foreground, one application from among the one or more currently running applications that has been in the foreground for the longest period of time; and determine that content associated with the one application includes content that meets the threshold probability of being viewed by the user of the computing device.

[0150] Example 36: Computer-readable storage medium according to any of Examples 25 to 35, wherein the instructions further cause the one or more processors to: determine whether a third section of the graphical user interface contains whitespace; and in response to determining that the third section of the graphical user interface contains whitespace, determine that the third section does not contain any content that meets the threshold probability of being viewed by the user of the computing device.

[0151] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted on a computer-readable medium as one or more instructions or as code and executed by a hardware-based processing unit. Computer-readable medium may include computer-readable storage media, media equivalent to a tangible medium such as data storage media, or communication media, which includes any medium that facilitates the transmission of a computer program from one location to another, for example, according to a communication protocol. In this way, computer-readable medium can generally correspond to (1) tangible, computer-readable storage media that are non-transient, or (2) a communication medium such as a signal or carrier wave.Data storage media can be any available media that one or more computers or one or more processors can access to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product can include a computer-readable medium.

[0152] As an example, but not limited to, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other storage media that can be used to store the desired program code in the form of instructions or data structures and that a computer can access. Furthermore, any such connection is correctly referred to as a computer-readable medium.For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted-pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted-pair cable, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it is understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other volatile media, but instead refer to non-volatile, tangible storage media.In this context, "disk" and "disc" include Compact Disc (CD), Laser Disc, Optical Disc, Digital Versatile Disc (DVD), floppy disk, and Blu-ray Disc, whereby "discs" typically reproduce data magnetically, while "discs" reproduce data optically using lasers. Combinations of the above are also included in the scope of computer-readable media.

[0153] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor" in this context can refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Furthermore, some aspects of the functionality described herein can be provided in dedicated hardware and / or software modules. Additionally, the techniques could be implemented entirely in one or more circuits or logic elements.

[0154] The techniques of this disclosure can be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). This disclosure describes various components, modules, or units to highlight functional aspects of devices configured to perform the disclosed techniques, but does not necessarily require implementation by different hardware units. Rather, as described above, different units can be combined in a single hardware unit or provided by a collection of interoperable hardware units, including one or more processors, as described above in conjunction with suitable software and / or firmware.

[0155] Several embodiments have been described. These and other embodiments are within the scope of the following claims.

Claims

[1] Method comprising: Outputting, by one or more processors of a computing device and displaying by a display device, a graphical user interface; Determining, by the one or more processors, that a first portion of the graphical user interface includes content that has at least a threshold probability of being viewed by a user of the computing device, the graphical user interface further including a second portion, the first portion and the second portion being non-overlapping portions of the graphical user interface; and Outputting, by the one or more processors and for display by the display device, an updated graphical user interface, including a graphical keyboard, such that less than a majority of the graphical keyboard visually overlays the first portion of the graphical user interface. [2] The method of claim 1, wherein the second portion of the graphical user interface does not include content that has at least a threshold probability of being viewed by the user of the computing device. [3] The method of claim 1 or 2, further comprising: Determining, by the one or more processors, whether a size of the second portion of the graphical user interface meets a threshold size; and in response to determining that the size of the second section does not meet the threshold size, partitioning, by the one or more processors, the graphical user interface into at least a first partition and a second partition, wherein: the first section of the graphical user interface is included in the first partition of the graphical user interface; and the graphical keyboard is included in the second partition of the graphical user interface. [4] The method of any one of claims 1 to 3, wherein the content included in the first portion of the graphical user interface is a first portion of content from a plurality of portions of content included in the graphical user interface, the method further comprising: Determining, by the one or more processors, whether the user has viewed a respective portion of content from the plurality of portions of content within a threshold period of time; and in response to determining that the user has not viewed the respective portion within the threshold time, determining, by the one or more processors, that the respective portion of the graphical user interface content is part of the second portion of the graphical user interface. [5] The method of any one of claims 1 to 4, wherein determining that the first portion of the graphical user interface includes content having at least the threshold probability of being viewed by the user of the computing device comprises: Tracking, by the one or more processors, the eye movements of the user; determining, by the one or more processors and based on the eye movements, a probability of being viewed by the user of the computing device for at least a portion of the content of the graphical user interface; Determining, by the one or more processors, whether the probability satisfies the threshold probability; and in response to determining that the probability meets the threshold probability, determining, by the one or more processors, that the portion of the graphical user interface content includes content that has at least a threshold probability of being viewed by the user of the computing device. [6] A method according to any one of claims 1 to 5, further comprising: Tracking, by the one or more processors, the eye movements of the user; determining, by the one or more processors and based on the user's eye movements, a number of times the user views content included in a third portion of the graphical user interface; Determining, by the one or more processors, whether the number meets a threshold number; and in response to determining that the number does not meet the threshold number, determining, by the one or more processors, that the third portion does not include content that meets the threshold probability of being viewed by the user of the computing device. [7] A method according to any one of claims 1 to 6, further comprising: Determining, by the one or more processors, whether advertising content is included in the graphical user interface; and in response to determining that advertising content is included in the graphical user interface, determining, by the one or more processors, that a space on the graphical user interface occupied by the advertising content does not include content that meets the threshold probability of being viewed by the user of the computing device. [8] A method according to any one of claims 1 to 7, further comprising: Determining, by the one or more processors, whether a video or image is included in the graphical user interface; and in response to determining that a video or image is included in the graphical user interface, determining, by the one or more processors, that the video or image includes content that meets the threshold probability of being viewed by the user of the computing device. [9] A method according to any one of claims 1 to 8, further comprising: determining, by the one or more processors, whether a third portion of the graphical user interface includes an active text field, wherein the active text field is a text field that has a current cursor focus; and in response to determining that the third portion includes the active text field, determining, by the one or more processors, that the third portion includes content that meets the threshold probability of being viewed by the user of the computing device. [10] The method of claim 9, further comprising: determining, by the one or more processors, whether a fourth portion of the graphical user interface includes content that meets the threshold probability of being viewed by the user of the computing device, the fourth portion being adjacent to the active text field; and in response to determining that the fourth portion does not include content that meets the threshold probability of being viewed by the user of the computing device, outputting, by the one or more processors and for display by the display device, an updated graphical user interface, including a graphical keyboard, such that a majority of the graphical keyboard visually overlays the fourth portion of the graphical user interface. [11] Method according to one of claims 1 to 10, further comprising: Determining, by the one or more processors, a respective period of time that each application from one or more currently executing applications was visible on the display device of the computing device; Identifying, by the one or more processors and based on the respective period of time that the one or more currently executing applications were visible on the display device, an application from the one or more currently executing applications that was visible on the display device for the longest period of time; and determining, by the one or more processors, that content associated with the one application includes content that meets the threshold for being viewed by the user of the computing device. [12] The method of claims 1 to 11, further comprising: determining, by the one or more processors, whether a third portion of the graphical user interface includes white space; and in response to determining that the third portion of the graphical user interface includes white space, determining, by the one or more processors, that the third portion does not include content that meets the threshold probability of being viewed by the user of the computing device. [13] A computing system comprising at least one processor configured to perform any of the methods of claims 1-12. [14] A system comprising means for performing any of the methods of claims 1-12. [15] A computer-readable storage medium comprising instructions that, when executed, cause at least one processor to perform any of the methods of claims 1-12.

Images