Dynamically display reused data in the UI

By dynamically converting the keyboard layer and symbol layer in the user interface, the virtual keyboard experience on small monitors is optimized based on user input frequency and interactive behavior, solving the problem of virtual keyboard rendering and improving interface usability and efficiency.

CN112400155BActive Publication Date: 2024-11-05GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980011144.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-06-12
Filing Date
2019-07-16
Publication Date
2024-11-05
Estimated Expiration
2039-07-16

AI Technical Summary

Technical Problem

On small displays, effective rendering of virtual keyboards and other data input methods is difficult to implement, resulting in limited user interface experience.

Method used

By dynamically converting between the keyboard layer and the symbol layer in the user interface, different parts of the symbol layer are exposed or hidden according to the user input frequency and interaction behavior to optimize the utilization of the display area.

Benefits of technology

The user interface features are realized in a limited display area, reducing the time and number of interactions between users when accessing different interface elements, and improving the usability and efficiency of the user interface.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112400155B_ABST
    Figure CN112400155B_ABST
Patent Text Reader

Abstract

Methods, systems, and apparatus for enhancing user interaction with an interface are described, including a computer program encoded on a computer storage medium. In one aspect, the method includes displaying a keyboard layer of a user interface, the keyboard layer including a keyboard having at least one user interface element configured to receive user input. Detecting an initial input at a given user interface element. In response to detecting the initial interaction, while maintaining display of the keyboard layer of the user interface, revealing a first portion of a symbol layer of the user interface. Detecting a subsequent input at the given user interface element. In response to detecting the subsequent input, while maintaining a position of the first portion of the symbol layer previously revealed in response to detecting the initial input, revealing a larger portion of the symbol layer of the user interface.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This specification relates to data processing and dynamically updating user interfaces. Background Art

[0002] User interfaces enable users to input and consume data. The user interfaces of some user devices include virtual keyboards (also known as on-screen keyboards) in which a user can interact with a visual representation of keys to enter characters displayed within the visual representation of the keys. As the use of mobile devices continues to grow, users spend a disproportionately more time on small displays than on traditional desktop displays, and are limited in their experience by the amount of space available to present information to and receive information from users on these handheld devices. Given the variety of available display sizes and the growing use of small displays, it can be difficult to effectively render virtual keyboards and other methods of supporting data entry. Summary of the invention

[0003] Generally, one aspect of the subject matter described in this specification can be embodied as a method, the method comprising the following operations: displaying a keyboard layer of a user interface in a user interface, the keyboard layer including a keyboard having at least one user interface element configured to receive user input; detecting an initial input at a given user interface element; in response to detecting the initial interaction, while maintaining the display of the keyboard layer of the user interface, revealing a first portion of a symbol layer of the user interface, the symbol layer having at least one user interface element configured to receive user input indicating a symbol; detecting a subsequent input at a given user interface element; in response to detecting the subsequent input, while maintaining the position of the first portion of the symbol layer previously revealed in response to detecting the initial input, revealing a larger portion of the symbol layer of the user interface, wherein the larger portion of the revealed symbol layer obscures at least a portion of the keyboard layer. Other embodiments of this aspect include corresponding systems, devices, apparatuses, and computer programs configured to perform the actions of the method. The computer program (e.g., instructions) can be encoded on a computer storage device. These and other embodiments can each optionally include one or more of the following features.

[0004] Revealing the first portion of the symbol layer may include revealing at least one row of user interface elements, each user interface element in the row configured to receive user input indicating a frequently used symbol, and wherein the row is displayed at a location above the keyboard.

[0005] Revealing the first portion of the symbol layer may include revealing at least one row of user interface elements, each user interface element in the row configured to receive user input indicating an emoticon, a Chinese character, a shape, or a web design widget.

[0006] The method may include the following operations: based on the frequency of use of symbols included in the larger portion of the symbol layer, selecting a set of symbols included in the first portion of the symbol layer. Selecting the set of symbols may include: identifying a specific symbol with the highest frequency of use; including a user interface element configured to receive user input indicating the specific symbol at a first position in the first portion of the symbol layer; identifying an additional set of symbols, each additional symbol having a frequency of use lower than the specific symbol but higher than the remaining symbols in the symbol layer; and including a plurality of user interface elements, each of the plurality of user interface elements configured to receive user input indicating a respective one of the additional sets of symbols in other positions in the first portion of the symbol layer.

[0007] Identifying the particular symbol having the highest frequency of usage may be based on frequency of usage data collected from applications other than the particular application that caused the first portion of the symbol layer to be displayed.

[0008] The method may include the following operations: before receiving user input at the symbol layer, initially filling a first portion of the symbol layer with a default set of symbols; in response to the initial set of user input at the symbol layer, while maintaining display of the first portion of the symbol layer, updating the first portion of the symbol layer to include a user interface element configured to receive user input indicating a specific symbol with a highest frequency of use at a first position in the first portion of the symbol layer.

[0009] The method may include the following operations: determining a default symbol set based on one or more of a machine learning model that provides a recommended symbol set, contextual information about a current user session, or based on a set of user-specified preferences.

[0010] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. For example, some embodiments discussed in this document enable intuitive access to user interface features in a limited display area by cycling between different user interface states in response to sequential user interactions with a particular user interface element. More specifically, in some embodiments discussed in this document, different portions of the symbol input interface are revealed as the user continues to interact with the user interface element (e.g., button or icon) that triggers the presentation of the symbol input interface. In some cases, the first interaction with the user interface element reveals only a preview view of the symbol input interface, which may include only the most frequently used and / or recently used single symbol row, thereby providing the user with access to a single symbol row while also providing the user with access to different input interfaces presented before the interaction. Subsequent interactions may reveal more of the symbol input interface and / or replace (hide) different interfaces. As discussed in more detail below, a series of user interface transitions not only provide the user with the different levels of access to multiple different input interfaces that the user desires, but also convey information about the transitions that are occurring, without having to explicitly convey that information to the user. This allows for more efficient use of the limited display area. Using layers to define various aspects of a user interface enables efficient transitions between different user interface states by enabling each aspect to be defined in a specified layer and then revealed or hidden based on interaction with the user interface. This simplifies the ability to transition between user interface states using various visual effects during the transition.

[0011] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of a series of user interfaces illustrating example transitions between user interface states.

[0013] Figure 2 is a block diagram of a series of user interfaces showing how different layers of a user interface may be revealed / hidden.

[0014] Figure 3 is a block diagram of an example process by which symbols may be selected to populate a preview view (or another portion) of a user input tool.

[0015] Figure 4 is a block diagram of an example system that can perform ranking and / or updating of the user interfaces discussed in this document.

[0016] Figure 5 is a flow chart of an example process for implementing a user interface and transformations.

[0017] Figure 6 is a block diagram of an example computer system.

[0018] Like reference numerals and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0019] This document discloses methods, systems, devices, and computer-readable media for improving user interfaces by improving user access to user interface features. More specifically, the user interfaces discussed herein can enable users to access a subset of symbols that are frequently used and / or otherwise specified for presentation when a default keyboard (or user interface layer) is presented. The user interfaces discussed herein can also enable users to quickly switch from a default keyboard to a full symbol keyboard in a seamless manner, and use animations that intuitively convey the relationship between the various views throughout the transition process. The technology for implementing these transitions between different user interface features (e.g., different keyboards) reduces the amount of time and number of interactions required for users to access features provided in different user interfaces. In addition, the user interface can be dynamically adjusted so that functions that are more frequently used by a particular user are repositioned within the user interface, making it easier for the user to access the function. These user interfaces are particularly useful in the context of mobile devices or other devices with limited display space, thereby providing users with access to functions that the user is more likely to use.

[0020] In some embodiments, the user interface initially presents a keyboard layer of the user interface presenting a keyboard in the display. For example, the user can focus (e.g., click) a text input box, which can initiate the presentation of a standard keyboard. The standard keyboard can be a virtual keyboard with multiple virtual keys, each of which is configured to receive user input. The user input can indicate characters of, for example, ASCII or ISO / IEC 8859-1 character sets. The virtual keyboard can have a QWERTY layout (or a portion thereof), or can have some or all of any other suitable layout. When the user interacts with a given user interface element, the presentation of the user interface can be converted to a "preview" view, in which only a portion (e.g., less than all) of the symbol layer of the symbol user interface is presented together with the keyboard layer. The symbol user interface can also be a virtual keyboard with multiple virtual keys, but each virtual key of the symbol user interface can be configured to receive user input indicating a symbol that cannot be accessed from a standard keyboard. Subsequent user interactions with a given user interface element will result in a conversion from a preview view to a full view of the symbol layer, which replaces the presentation of the keyboard layer and enables the user to interact with the full version of the symbol user interface. Further interactions with a given user interface element may return to the preview view and then to the presentation of the keyboard layer, such that the user may easily access each view through additional interactions with the same given user interface element, as described in more detail below.

[0021] Figure 1 1 is a block diagram of a series of user interfaces 100 illustrating example transitions between user interface states. The series of user interfaces 100 includes a keyboard interface 102, a preview view interface 104, and a symbol interface 106. The keyboard interface 102 can be the default interface presented when a user interacts with the text input portion of an application. For example, when a user focuses on a text input area 108, the keyboard interface 102 can be activated (e.g., revealed). The user can focus on the text input area 108, for example, by clicking at a location on a touch screen where the text input area 108 is presented. Other ways of focusing on the text input area 108 can include interacting with the text input area 108 using an indicator (e.g., a mouse or stylus), or using a voice command to activate the text input area 108 for data entry.

[0022] As discussed in more detail below, the keyboard area 110 within the dashed rectangle of the keyboard interface 102 can be implemented as a layer in the overall user interface definition. When the keyboard interface 102 is presented, the layer including the keyboard area 110 is fully revealed and is therefore visually perceptible in the keyboard interface 102. The keyboard area 110 enables a user to enter alphanumeric characters into the text input area 108 by interacting with the keyboard area 110.

[0023] The text input area 108 includes a given user interface element that triggers a transition between user interface views, which in this example is a smiley face emoticon 112. In some cases, a first user interaction with the smiley face emoticon 112 results in a first transition from the keyboard interface 102 to the preview view interface 104. The preview view interface 104 still includes the keyboard area 110 presented in the keyboard interface 102, but also includes a single symbol row 114 presented above the keyboard area 110. In this particular example, the symbol row 114 is presented between the text input area 108 and the keyboard area 110, but other arrangements are possible. In some embodiments, the symbol row 114 can be the most commonly used symbols, the most recently used symbols, or a favorite set of symbols. As shown in FIG. Figure 3 As discussed in greater detail, the symbols presented in the symbol row 114 may be dynamically selected and / or updated based on user interactions and / or contextual data of a user session.

[0024] As discussed in more detail below, the symbol row 114 can be implemented in a different layer of the user interface than the keyboard area 110, and can be only a portion of the different layer (e.g., less than all). For example, when a user interaction with the smiley face emoticon 112 of the keyboard interface 102 is detected, the detected user interaction can trigger the execution of a user interface transition routine that causes a specified portion of a different layer to be presented while the keyboard area 110 of the keyboard interface 102 remains presented. The user interface transition routine can also include an animation sequence that causes the text input area 108 to move upward along the display to make room for the symbol row 114 in the preview view interface 104. In this example, a portion of the layer containing the symbol row 114 remains obscured, such as transparent or behind the user interface layer in which the keyboard area 110 is defined, so that the keyboard area 110 remains visually perceptible after the symbol row 114 of other layers is revealed.

[0025] When the symbol row 114 is presented, each of the symbol row 114 and the keyboard area 110 are active for user input. For example, user interaction with any symbol presented in the symbol row 114 triggers entry of that symbol in the text input area 108. Similarly, interaction with a character in the keyboard area 110 triggers entry of that character in the text input area 108. In this way, the preview view interface 104 provides an efficient and effective way for the user to continue to access the keyboard area 110 while also being able to access symbols from the symbol row 114.

[0026] Subsequent user interaction with the smiley face emoticon 112 while the preview view interface 104 is being presented triggers a second transition from the preview view interface 104 to the symbol interface 106. The transition to the symbol interface 106 includes presentation of a full symbol keyboard 116, in this example a full emoticon keyboard. In some embodiments, the transition from the preview view interface 104 to the symbol interface 106 is achieved by revealing a portion of a layer that includes the full symbol keyboard 116, which is not presented in the preview view interface 104. In other words, the layer that includes the full symbol keyboard 116 can be the same layer as the layer that includes the symbol row 114, so that when the full layer (or at least a portion of the layer that includes the full symbol keyboard 116) is revealed, the full symbol keyboard 116 that is visually perceptible in the symbol interface 106 includes the symbol row 114. The transition from the preview view interface 104 to the symbol interface 106 also includes hiding or obscuring the layer that includes the keyboard area 110, so that the full symbol keyboard 116 is presented in the portion of the display that previously presented the keyboard area 110, as described below with reference to Figure 2 In some cases, for example, the keyboard area can be reduced by removing the number row and / or adjusting the spacing of other items in the keyboard area, which will also reveal the symbol row.

[0027] When the full symbol keyboard 116 is presented, a user interaction with any symbol presented in the full symbol keyboard 116 will trigger entry of that symbol into the text input area 108. Additionally, a subsequent user interaction with the smiley face emoticon 112 will again trigger a transition to the user interface. In some embodiments, when a subsequent user interaction occurs while the symbol interface 106 is presented, the interaction triggers a transition back to the preview view interface 104. The transition back to the preview view interface 104 includes re-revealing the keyboard area 110, and obscuring the portion of the full symbol keyboard 116 that is presented in the portion of the display where the keyboard area 110 is presented in the preview view interface 104. The transition back to the preview view interface 104 also includes obscuring or hiding the full symbol keyboard 116, except for the portion of the full symbol keyboard 116 that includes the symbol row 114.

[0028] After transitioning back to the preview view interface 104 from the symbol interface 106, subsequent user interactions with the preview view interface 104 trigger a transition back to the keyboard interface 102. The transition back to the keyboard interface 102 includes obscuring or hiding the symbol row 114. In some embodiments, the transition from the preview view interface 104 back to the keyboard interface 102 also includes repositioning the text input area 108 closer to the keyboard area 110 (e.g., repositioning the text input area 108 to a location in the keyboard interface 102 that corresponds to the location where the symbol row 114 is presented in the preview view interface 104). The repositioning of the keyboard interface 102 may include an animation in which the movement of the text input area 108 down the display toward the keyboard area 110 is visually perceptible. Once the transition to the keyboard interface 102 is completed, further interactions with the smiley face emoticon 112 will cycle between the keyboard interface 102, the preview view interface 104, and the symbol interface 106 in the manner described above.

[0029] Figure 2 is a reference showing how to show / hide different layers of the UI to achieve Figure 1 2 is a block diagram of a series of user interfaces 200 for the operation discussed herein. The series of user interfaces 200 includes an initial interface 202, an intermediate interface 204, and a final interface 206. The initial interface 202 can be a default interface presented when a user is interacting with a text input portion of an application. For example, when a user focuses on a text input area, the initial interface 202 can be launched (e.g., revealed), as shown in FIG. Figure 1 In this example, the initial interface 202 presents a specific user interface layer 208, which presents a specific set of user interface elements. For example, the specific user interface layer 208 may present Figure 1 208. In some implementations, the specific user interface layer 208 may be the only content defined by the specific layer, or the specific user interface layer 208 may include additional content (e.g., user interface elements) that may be located in the region 210. The region 210 may be defined in the specific user interface layer 208, or may be defined in another layer of the initial interface 202. For the purposes of this example, it is assumed that the specific user interface layer 208 has a size corresponding to the shaded area of ​​the initial interface 202, and it is assumed that the region 210 is implemented in a separate layer of the initial interface 202.

[0030] When a particular user interface layer 208 is revealed, unhidden, activated, and / or brought to the front of a layer of the initial interface 202, the particular user interface layer 208 is visually perceptible in the initial interface 202. For the sake of brevity, this document refers to a user interface layer (or portion thereof) that is visible when revealed. Thus, the particular user interface layer 208 and the area 210 of the initial interface 202 are in a revealed state. In some cases, a layer of a user interface may be fully revealed or partially revealed. For example, in order to provide varying degrees of access to user interface elements defined in a given layer, a user interface may be designed so that specified portions of a layer (e.g., defined by coordinates and / or reference position and size information) are revealed in certain circumstances, while different portions of the layer may be revealed in other circumstances. In this example, the particular user interface layer 208 is fully revealed, thereby revealing the user interface elements defined in the particular user interface layer 208. Examples of such user interface elements that may be depicted in a particular user interface layer 208 are shown in the accompanying drawings. Figure 1 , is shown in the keyboard area 110 of the user interface. In general, when one layer of the user interface is revealed in a particular portion of the user interface, other layers of the user interface are not visually perceptible in the particular portion of the user interface (assuming that the transparency of the presented layer is set to 0%). Revealing the layer may include reducing the transparency of the layer while increasing the transparency of other layers so that the revealed layer becomes visible. Alternatively or additionally, revealing the layer may include configuring the user interface to receive user input indicating a character or symbol associated with (e.g., displayed in or near) a user interface element of the revealed layer.

[0031] The user interface can transition from one state to another state in response to user interaction with the user interface, or based on the satisfaction of certain timing conditions or certain other conditions. For example, when the user device detects an interaction with a user interface element presented in the initial interface 202, the user device can transition from the initial interface 202 to the intermediate interface 204. Examples of this type of transition are shown below. Figure 1 104. In this example, the specific user interface layer 208 remains exposed so that the user interface elements defined in the specific user interface layer 208 remain visually perceptible in the intermediate interface 204. This is similar to Figure 1 The keyboard area 110 remains visually perceptible after the transition from the keyboard interface 102 to the preview view interface 104 .

[0032] In the intermediate interface 204, portions of different layers 212 (e.g., different from the specific user interface layer 208 and the region 210) are exposed in the designated portion 214 of the intermediate interface 204. This is similar to Figure 1The symbol row 114 is revealed in the preview view interface 104. Portions of different layers 212 can be revealed in various ways to create a variety of different visual effects. For example, portions of different layers 212 can be defined as fading in (e.g., by changing the opacity of different layers 212 and region 210 in opposite directions), or portions of different layers 212 can be moved from a lower position on the display (e.g., hidden behind a specific user interface layer 208) to a higher position on the screen (e.g., above a specific user interface layer 208). In some cases, a specific user interface layer 208 can be reduced so that it no longer includes an area replaced by a specified portion. Assuming that different layers 212 are defined at a higher level of the user interface than region 210, but at a lower level of the user interface than specific user interface layer 208, the upward movement of different layers 212 along the display will cause different layers 212 to be revealed as it moves out from behind a specific user interface layer 208, thereby causing portions of different layers 212 to be revealed.

[0033] When revealing a different layer 212 in the designated portion 214, the region 210 can be animated to visually indicate that the different layer 212 is displacing the region 210. For example, when revealing the different layer 212, the region 210 can move up along the display to "make room" for the revealed portion of the different layer 212. This can be accomplished by moving the position of the region 210 up along the display based on the size of the designated portion 214 to be occupied by the different layer 212, or by revealing the different layer in the portion of the intermediate interface 204 that remains occupied by the region 210 after the transition from the initial interface 202 to the intermediate interface 204. For example, instead of shifting the position of the region 210 in the display, the user device can hide the region 210 while revealing another layer that shows the user interface elements of the region 210 in a different display position. In either case, the resulting user interface of intermediate interface 204 will include at least some user interface elements defined in region 210 , some user interface features defined in different layers 212 , and the entirety of the particular user interface layer 208 presented in initial interface 202 . Figure 1 The preview view interface 104 is an example of an end state of such a transition from the initial interface 202 to the intermediate interface 204 .

[0034] When the portion of different layer 212 is revealed, another portion of different layer 212 may remain hidden (or obscured) by particular user interface layer 208 located in the same portion of the display as different layer 212. In this case, the remainder of different layer 212 (or some other larger portion) may be revealed through user interaction with intermediate interface 204. For example, as described in reference to Figure 1As discussed above with respect to the transition from the preview view interface 104 to the symbolic interface 106, a user interaction with the intermediate interface 204 can trigger a transition to the final interface 206 in which the different layer 212 is fully revealed, thereby replacing the particular user interface layer 208 presented in the initial interface 202 and the intermediate interface 204. The portion of the different layer 212 presented in the designated portion 214 of the intermediate interface 204 can remain in the same display position while the rest of the different layer 212 is revealed, so that the transition from the intermediate interface 204 to the final interface 206 visually communicates to the user that the transition from the intermediate interface 204 to the final interface 206 is revealing the rest of the different layer 212 that was only partially presented previously.

[0035] In some embodiments, Figure 1 As shown in the preview view interface 104 and the symbol interface 106 of , the user interface elements presented in the designated portion 214 of the intermediate interface 204 also remain in the same position to emphasize to the user that they have not yet switched to a different set of user interface features, but rather fully reveal more user interface features previewed in the designated portion 214 of the intermediate interface 204. For example, in Figure 1 In the symbolic interface 106 of FIG. 1 , the symbols presented in the symbol row 114 remain in the same position and order during the transition from the preview view interface 104 to the symbolic interface 106. This helps to visually convey to the user that the remainder of the full symbol keyboard 116 revealed in the symbolic interface 106 is a larger portion of the symbol row 114 previously presented in the preview view interface 104. In this way, a particular combination of user interface transitions can be used to convey implicit contextual information about the content that the user is viewing. This is particularly useful in situations where the display size is limited (e.g., small), because in these situations, the space used to explain the content that the user is viewing detracts from the features that can be presented in the limited display space.

[0036] Once the user interface transitions from the initial interface 202 to the intermediate interface 204, further interaction with the user interface element (or other conditions) may trigger a transition from the intermediate interface 204 to the final interface 206. Examples of this type of transition are shown below. Figure 1 206. In this example, a larger portion of the different layers 212 is revealed, and the specific user interface layer 208 is hidden, so that after the transition from the intermediate interface 204 to the final interface 206, the specific user interface layer 208 is no longer visible. This is similar to the transition from Figure 1 After the conversion from the preview view interface 104 to the symbol interface 106 , the full symbol keyboard 116 replaces the keyboard area 110 .

[0037] In the final interface 206, the rest (or at least a larger portion) of the different layers 212 is revealed in the area where the specific user interface layer 208 was previously revealed. As described above, the larger portion of the different layers 212 can be revealed in various ways to create a variety of different visual effects. For example, it can be defined (e.g., programmed) that the larger portion of the different layers 212 fades in (e.g., by changing the opacity of the different layers 212 and the area 210 in opposite directions). Alternatively, the different layers 212 can be moved to a layer higher than the specific user interface layer 208 defined in the user interface, thereby obscuring the specific user interface layer 208. In some embodiments, the portion of the different layers 212 presented in the designated portion 214 of the intermediate interface 204 will remain fixed during the transition from the intermediate interface 204 to the final interface 206, thereby visually notifying the user that the rest of the different layers 212 presented in the final interface 206 is a larger view of the different layers 212, e.g., not a different layer or information set. As described above, this helps convey contextual information about what is happening during the transition from the intermediate interface 204 to the final interface 206 without having to utilize limited display space to convey that information to the user.

[0038] After transitioning to the final interface 206, further user interaction (e.g., through one or more user interface elements) may trigger further transitions between user interface states. Figure 1 As discussed above with respect to the transition from the symbolic interface 106 to the preview view interface 104, interaction with the final interface 206 may trigger a transition back to the intermediate interface 204, and then as described with reference to Figure 1 As discussed above with respect to the transition from preview view interface 104 to keyboard interface 102 in , a subsequent interaction with intermediate interface 204 can trigger a transition back to initial interface 202. In another example, an interaction with final interface 206 can trigger a transition back to initial interface 202, as shown by curved arrow 216. In yet another example, an interaction (or some other condition, such as a timing condition) can trigger a transition to another user interface that is not depicted. In any case, a transition to another user interface can be achieved by manipulating / adjusting layers defined in the user interface and / or exposing different layers and / or portions of layers defined in the user interface. For example, each user interaction (or condition) in each user interface state can be mapped to a set of layer adjustments that result in a visual transition.

[0039] As mentioned above, in Figure 1 The symbols presented in the symbol row 114 may be a set of most commonly used symbols, recently used, or otherwise recommended symbols. Figure 3 is a visual representation of the symbol that can be selected to fill a Figure 114. Block diagram 300 of an example process for displaying a preview view (or another portion) of a user input tool presented in symbol row 114 of an image processing apparatus. Block diagram 300 includes three display states, referred to as initial display state 302, intermediate display state 304, and subsequent display state 306. As used throughout the discussion, the phrase "initial display state" is used simply as a reference to the first display state discussed, and is not intended to be limited to the first presented state or even the unchanged display state, but the initial display state can be the first presented state or the unchanged display state.

[0040] The initial display state 302 includes a recommended symbol area 308 and a common symbol area 310, the recommended symbol area 308 including a set of recommended symbols. In this example, the symbols are shapes, but the symbols can be emoticons, Chinese characters, Arabic characters, web design widgets, or any other symbols. In some embodiments, the recommended symbol area 308 can be part of a user interface that presents the most commonly used symbols, the most recently used, or otherwise recommended symbols, with the highest ranking symbols in the set. As discussed in more detail below, the ranking can be performed based on one or more of the frequency of use of the symbols, the context of the user input (e.g., the application being used or the input previously entered), and / or the output of an artificial intelligence agent that selects the recommended symbols based on various signals input into the artificial intelligence agent (e.g., user characteristics, time of day, context of the input, input from other applications, or other signals).

[0041] For purposes of this discussion, the universal symbol area 310 is depicted as a static symbol area, such as a static symbol keyboard including a fixed arrangement of symbols that are entered (e.g., into a text input field) in response to user interaction with the symbols. In other words, for purposes of the following examples, the set of symbols presented in the universal symbol area 310 will not change. However, the universal symbol area 310 may also be an extension of the recommended symbol area 308, such that the universal symbol area 310 may also change dynamically in response to user interaction with the symbols, as will be discussed in more detail below.

[0042] Prior to user interaction with the universal symbol region 310, the recommended symbol region 308 may be free of any symbols (e.g., empty), may include a default set of symbols, or may include a user-specified set of symbols. In this example, the recommended symbol region 308 includes a set of symbols that may be considered to have a ranking of 1 (left side) to 6 (right side), wherein ranking 1 is considered to be the most prominent or most recommended symbol. As the user interacts with the symbols in the universal symbol region 310, the device presenting the initial display state 302 may monitor the user interaction (and / or other information) and dynamically update the recommended symbol region 308 by rearranging and / or replacing the symbols presented in the recommended symbol region 308 based on the monitored user interaction (and / or other information). For example, prior to user interaction with the symbols in the universal symbol region 310, the pentagon 312 is the highest ranked symbol in the recommended symbol region 308, and the diamond 314 is the lowest ranked symbol in the recommended symbol region 308. Again, this initial ranking may be a default ranking, a user-specified ranking, or some other ranking.

[0043] For purposes of example, assume that a device presenting the initial display state 302 detects a user interaction with a four-pointed star symbol 316 in the common symbols region 310 of the initial display state 302. Assume further that the initial ranking of the symbols in the recommended symbols region 308 is not based on frequency of use, but that the recommended symbols region 308 is programmed to dynamically update based on symbol frequency of use or recent symbol use following user interaction with the symbols in the common symbols region 310. In this example, the user interaction with the four-pointed star symbol 316 will cause the four-pointed star symbol 316 to become the most commonly used symbol (as well as the most recently used symbol), which results in updating the user interface by inserting the four-pointed star symbol 316 into the highest ranked position (e.g., the leftmost position) in the recommended symbols region 308, as shown in the intermediate display state 304. In some implementations, the insertion of the four-pointed star symbol 316 into the highest ranked position will cause the symbol (in this case, the pentagon 312) that was previously presented in the highest ranked position (e.g., second from the left) to shift rightward to a lower ranked position, as shown in the transition from the initial display state 302 to the intermediate display state 304. When this shift occurs, the remaining symbols included in the recommended symbol area 308 may also be shifted to the next lower ranked position before the four-pointed star symbol 316 is inserted, and the lowest ranked symbol in the initial display state 302 (in this example, the diamond 314) is removed from the recommended symbol area 308.

[0044] In some embodiments, inserting a symbol into the recommended symbol region 308 will cause the symbol to be removed from the universal symbol region 310, which reduces redundancy in the symbols presented. Referring to the above example, inserting the four-pointed star symbol 316 into the recommended symbol region 308 may cause the four-pointed star symbol 316 to be removed from the universal symbol region 310. In these cases, in response to removing the four-pointed star symbol 316 (or another symbol) from the universal symbol region 310, the remaining symbols in the universal symbol region 310 may be rearranged. For example, the symbol may be shifted and / or shifted upward to occupy the position of the universal symbol region 310 occupied by the four-pointed star symbol 316. In addition, the new symbol (or the symbol replaced from the recommended symbol region 308) may be moved from the recommended symbol region 308 to the universal symbol region 310. The position of the universal symbol region 310 occupied by the symbol removed from the recommended symbol region 308 may be the position previously occupied by the symbol inserted into the recommended symbol region 308, or may be another position.

[0045] The transition from the initial display state 302 to the intermediate display state 304 may occur while the recommended symbols area 308 and the universal symbols area 310 continue to be presented, or the transition from the initial display state 302 to the intermediate display state 304 may occur after the recommended symbols area 308 and / or the universal symbols area 310 have been removed (e.g., hidden) in the user interface. In other words, the transition between the initial display state 302 and the intermediate display state 304 may occur while the user continues to use the recommended symbols area 308 and the universal symbols area 310, or the transition may occur after the user closes the recommended symbols area 308 and the universal symbols area 310. The transition from the initial display state 302 to the intermediate display state 304 while the recommended symbols area 308 and the universal symbols area 310 continue to be presented provides the user with additional visual communication about how the recommended symbols area 308 is arranged and / or the importance of those symbols without having to otherwise communicate that information using limited screen space.

[0046] After the transition from the initial display state 302 to the intermediate display state 304, the device presenting the intermediate display state 304 can continue to monitor user interactions with the user interface (and / or other information), re-rank symbols based on the monitored user interactions (and / or other information), and update the order of symbols presented in the recommended symbol area 308 (and / or the common symbol area 310). For example, after the transition from the initial display state 302 to the intermediate display state 304, assume that the device detects one or more user interactions with the five-pointed symbol pentagon 312. Such interactions with the pentagon 312 can trigger the device to re-rank the symbols (discussed in more detail below) and update the symbols presented in the recommended symbol area 308. For example, assume that the detected interactions result in the pentagon 312 being ranked higher than the four-pointed star symbol 316. In this example, as visually indicated in the transition from the intermediate display state 304 to the subsequent display state 306, the recommended symbol area 308 can be updated to swap the positions of the pentagon 312 and the four-pointed star symbol 316.

[0047] As described above, in the recommended symbol area 308 (and / or Figure 1 The updating of the arrangement of symbols presented in the symbol row 114 of the recommended symbol area 308 can be based on the symbol ranking. For example, the symbol ranking can be performed by a device (e.g., a mobile device, a tablet device, a wearable computing device, or a desktop device) presenting the recommended symbol area 308 or a server-side device communicating with the device presenting the recommended symbol area 308. Figure 4 4 is a block diagram of an example system 400 that can perform ranking and / or updating of user interfaces discussed in this document. System 400 can be implemented, for example, in a mobile device, a tablet device, a wearable device (e.g., a smart watch), or another computing device.

[0048] System 400 includes display management means 402, interaction means 404, symbol use means 406, UI layer definition 408, and native applications 410a-410n. System 400 may also include a network interface to facilitate communication with networked devices such as user devices and / or server devices.

[0049] Display management device 402 includes one or more data processing devices and is configured to monitor interactions (e.g., interactions by user 412) with a user interface presented by system 400. For example, display management device 402 may accept data specifying user interactions with a user interface presented by display management device 402, evaluate actions taken in response to the user interactions, and update the presentation of the user interface presented by system 400, as discussed in more detail below.

[0050] The interaction device 404 detects user interaction with the display of the system 400. For example, the interaction device 404 can be connected to a touch display or a portion of a touch display that detects user interaction with the touch display. The interaction device 404 can also receive signals indicating other user interactions (e.g., from the device periphery), such as a user's single click on an element of the user interface, a wireless pencil click or move, or other user interaction with an element of the user interface. The interaction device 404 can accept these signals indicating user interaction and provide interaction data to the display management device 402. The interaction data can specify, for example, the type of user interaction that occurred (e.g., a single click, click, slide, pinch, or other type of interaction), and the user interface element (e.g., button, symbol, drop-down menu, icon, etc.) with which the interaction occurred. In some embodiments, the interaction data can also specify which native application is in focus or active when the user interaction occurs and / or the state of the native application when the interaction occurs.

[0051] Display management device 402 receives interaction data from interaction device 404 and processes the information based on one or more rules to take action. In some embodiments, display management device 402 can classify the interaction based on the type of interaction and the user interface element with which the interaction occurs to determine whether the interaction is with a symbol as an input to an application. For example, suppose the interaction data received from interaction device 404 indicates that the interaction is with a Figure 3 410a is active. In this example, the display management device 402 can classify this as a symbol interaction and pass the symbol usage data to the symbol usage device 406. The symbol usage data can include, for example, the symbol interacted with, any information provided to the display management device 402 in the interaction data, and additional information. The additional information can include, for example, contextual data, such as time of day information, day of the week information, and / or characteristics corresponding to the user 412 (e.g., profile information, application usage data, etc.).

[0052] The symbol usage device 406 can store symbol usage data on a per-application basis. For example, the symbol usage device 406 can store, with reference to application 410a, each symbol interaction detected while application 410a is active. Similarly, the symbol usage device 406 can store, with reference to application 410b, symbol interactions detected while application 410b is active. The symbol usage device 406 can also store symbol interactions with a timestamp indicating the time of the interaction with the symbol.

[0053] The display management device 402 can use the symbol usage data to rank the symbols. For example, the display management device 402 can rank the symbols based on the frequency of use. For example, the display management device 402 can determine how many times each symbol is interacted with based on the symbol usage data. In one example, the display management device 402 can count the number of interactions with each given symbol stored by the symbol usage device 406. In another example, the symbol usage device 406 can update the usage counter of each symbol as the symbol usage data specifying the use of each symbol is received, and the display management device 402 can read the value of the counter for each symbol. Using the number of interactions with the symbols, the display management device 402 can assign a ranking (e.g., 1-x) to the symbols in descending order of frequency of use, with ranking 1 being the highest. In turn, the display management device 402 can update the presentation of the symbols based on the ranking, as described above with reference to Figure 1-3 discussed.

[0054] In some embodiments, the frequency of use of those symbols may be used to determine a ranking on a per-application basis as each specified application is used. In some embodiments, the ranking may be determined based on aggregate symbol usage across multiple applications. Aggregate symbol usage may, for example, consider symbol usage across all applications installed in the system 400, or less than symbol usage across all those applications. For example, the display management device 402 may determine aggregate usage on a per-application category basis (e.g., for gaming applications, text applications, social media applications, or other categories of applications), and may also create a ranking on a per-application category basis.

[0055] The display management device 402 may use the ranking created based on the aggregate symbol usage to update (or initialize) a "preview view" displayed for any application corresponding to the aggregate symbol usage. For example, if the ranking is based on an application category, the ranking may be used to arrange symbols for newly installed applications included in the category to be displayed in a preview view interface (e.g., Figure 1 The symbols are initially arranged in the preview view interface 104).

[0056] The display management device 402 may rank the symbols based on other or additional information. Figure 3 As discussed, the display management device 402 may rank the symbols based on how recently they were used, eg, the most recently used symbols are ranked highest and are placed in the highest ranked display position.

[0057] The display management device 402 can also rank or adjust the ranking based on a combination of features or information. For example, the display management device 402 can implement a weighting function that determines a ranking score based on the sum of the value of each corresponding feature (e.g., frequency of use, recent use, or other feature) and the weight assigned to the corresponding feature. The ranking score can then be used to rank and present the symbols in a manner similar to that discussed above.

[0058] In some embodiments, the ranking of symbols can be performed and / or adjusted based on the output of an artificial intelligence agent that generates the ranking based on predictions of the symbols most likely to be used by users 412 of the system 400. For example, any of the data discussed above (or other data) can be input to an artificial intelligence agent that has been trained to predict those symbols most likely to be used by users in various contexts. The output of the artificial intelligence agent can be used to rank and present symbols based on the predictions. For example, supervised learning techniques can be used to train an artificial intelligence agent to predict the most likely symbols to be selected, where each symbol is a class.

[0059] The system 400 also includes a UI layer definition 408, which the display management device 402 can use to implement the above reference Figure 1-3 The UI layer definition 408 may define the structure and UI components of each UI layer of the user interface. For example, Figure 2 , UI layer definitions 408 may define the size and / or orientation of each layer relative to other layers, the UI elements presented in each UI layer, and the default order of the layers. UI layer definitions 408 may also include instructions that specify how to rearrange and / or reveal the layers of the user interface in response to user interactions with the user interface. For example, the instructions may specify moving and / or revealing layers and / or portions of layers to achieve a reference Figure 1-2 Animation for discussion.

[0060] When display management device 402 receives interaction data indicating a specified user interaction with a user interface, display management device 402 can utilize UI layer definition 408 to execute instructions mapped to the specified user interaction and update the user interface based on those instructions. Figure 1, when the display management device 402 receives interaction data indicating a user interaction with the smiley face emoticon 112 of the keyboard interface 102, the display management device 402 can access the UI layer definition 408 to determine how to modify the arrangement of the layers of the user interface to implement the transition from the keyboard interface 102 to the preview view interface 104. More specifically, the display management device 402 can identify instructions that map to the specified user interaction that occurred and the user interface state of the application that existed at the time of the user interaction. Once the display management device 402 has identified the instructions, the display management device 402 modifies the user interface according to the instructions to perform the above reference. Figure 1 The discussed approach converts the keyboard interface 102 to the preview view interface 104 .

[0061] Figure 5 is a flow chart of an example process 500 for implementing the user interface and conversion discussed above. The process 500 can be performed by one or more data processing devices (such as the device 400 discussed above). The process can also be implemented as instructions stored on a non-transitory computer readable medium. The instructions can be executed by one or more data processing devices. Execution of the instructions can cause one or more data processing devices to perform the operations of the process 500.

[0062] A keyboard layer (502) presenting a user interface. The keyboard layer is a layer presenting a user interface of a keyboard. For example, the keyboard can be a QWERTY keyboard or other keyboards that are convenient for user input. Figure 1 The keyboard area 110 is an example of a keyboard layer presented in a user interface. In some implementations, the keyboard layer is revealed in response to a user interaction with a keyboard icon or a text input field of the user interface.

[0063] An initial interaction with a given user interface element is detected (504). The initial interaction can be the first user interaction with a given user interface (e.g., by a particular user after installing an application), the first user action with the user interface during the current user session, or simply a reference interaction used to determine the relative position and timing of other user interactions.

[0064] While maintaining the presentation of the keyboard layer of the user interface, in response to the initial interaction, revealing a first portion of the symbol layer of the user interface (506). In some embodiments, as described above, the first portion of the symbol layer is revealed by shifting the position of the symbol layer so that the first portion is no longer obscured by another layer of the user interface. In some embodiments, as described above, the first portion of the symbol layer is revealed by adjusting the transparency settings of various portions of the layer of the user interface. For example, the first portion of the symbol layer revealed may be at least one row of frequently used symbols. For example, Figure 1As shown, the first part of the symbol layer can be revealed in the user interface above the keyboard presentation. The revealed first part of the symbol layer can reveal at least one row of emoticons, Chinese characters, shapes or web design widgets.

[0065] A symbol set is selected to be presented in the first portion of the symbol layer (508). In some embodiments, the selection of the symbol set is selected based on the frequency of use of symbols included in the larger portion of the symbol layer. For example, as described in reference Figure 3 and Figure 4 As discussed, symbols may be ranked based on their frequency of use, and a symbol set may be selected to include a specified number of the highest ranked symbols. More specifically, the symbols may be ranked similarly to those in the above reference. Figure 4 The discussed method identifies a specific symbol with the highest frequency of use. In some embodiments, as shown above with reference to Figure 4 As discussed, a specific symbol with the highest frequency of use is identified based on usage frequency data (e.g., symbol usage data and / or interaction data) collected from an application different from the specific application presenting the first portion of the symbol layer. The specific symbol with the highest frequency of use can be presented in a first position (e.g., a highest ranked position) of the symbol layer.

[0066] In some embodiments, a set of additional symbols is identified, wherein each additional symbol is used less frequently than a particular symbol. Figure 3 As discussed, additional symbol sets are presented in other locations in the first portion of the symbol layer.Selection of a symbol set may occur at any time prior to the initial interaction, or may even occur in response to the initial interaction.

[0067] In some embodiments, as described above, the first portion of the symbol layer is initially populated with a default symbol set (e.g., prior to user interaction with the symbol layer). The default symbol set can be determined or selected based on one or more of a machine learning model that provides a recommended symbol set (e.g., an artificial intelligence agent discussed above), contextual information about the current user session, or based on a set of user-specified preferences.

[0068] The first portion of the symbol layer may be updated in response to an initial set of user interactions with the symbol layer. For example, the update may cause a particular symbol having the highest frequency of use to be presented at a first position in the first portion of the symbol layer, while the first portion of the symbol layer remains presented, as described above with reference to Figure 3 discussed.

[0069] Detecting subsequent interactions with a given user interface element (510). Subsequent interactions can be detected in a manner similar to that discussed above. For example, a given user interaction can be a click or a single click. Figure 1 Smiley emoticon 112.

[0070] In response to detecting the subsequent interaction, a larger portion of the symbol layer of the user interface is revealed (512). In some embodiments, the larger portion of the symbol layer is displayed while maintaining the position of the first portion of the symbol layer that was previously revealed in response to detecting the initial interaction. Figure 1 and 2 As discussed in detail, the revealing of a larger portion of the symbol layer can obscure at least a portion of the keyboard layer. In some embodiments, as the user continues to interact with the same given user interface element (e.g., Figure 1 The user interface can continue to cycle between multiple different user interface states, and the multiple different user interface states can include three or more different user interface states. In a specific example, the portion of the specific keyboard that is exposed can change as the user interface cycles between various user interface states.

[0071] Figure 6 6 is a block diagram of an example computer system 600 that can be used to perform the above operations. System 600 includes a processor 610, a memory 620, a storage device 630, and an input / output device 640. Each of components 610, 620, 630, and 640 can be interconnected, for example, using a system bus 650. Processor 610 is capable of processing instructions that are executed within system 600. In one embodiment, processor 610 is a single-threaded processor. In another embodiment, processor 610 is a multi-threaded processor. Processor 610 is capable of processing instructions stored in memory 620 or storage device 630.

[0072] The memory 620 stores information within the system 600. In one embodiment, the memory 620 is a computer readable medium. In one embodiment, the memory 620 is a volatile memory unit. In another embodiment, the memory 620 is a non-volatile memory unit.

[0073] The storage device 630 can provide mass storage for the system 600. In one embodiment, the storage device 630 is a computer-readable medium. In various embodiments, the storage device 630 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices (e.g., a cloud storage device) on a network, or some other mass storage device.

[0074] The input / output device 640 provides input / output operations for the system 600. In one embodiment, the input / output device 640 may include one or more of a network interface device (e.g., an Ethernet card), a serial communication device (e.g., an RS-232 port), and / or a wireless interface device (e.g., an 802.11 card). In another embodiment, the input / output device may include a driver device configured to receive input data and send output data to other input / output devices, such as a keyboard, a printer, and a display device 660. However, other embodiments may also be used, such as a mobile computing device, a mobile communication device, a set-top TV client device, and the like.

[0075] Despite Figure 6 An example processing system is described in the specification, but the implementation and functional operations of the subject matter described in this specification may be implemented in other types of digital electronic circuits or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them.

[0076] An electronic document (referred to simply as a document for simplicity) does not necessarily correspond to a file. A document may be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files.

[0077] The subject matter and the embodiments of the operation described in this specification can be implemented in digital electronic circuits or in computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions, which are encoded on a computer storage medium (or multiple media) to be run by a data processing device or to control the operation of a data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal (e.g., an electrical, optical or electromagnetic signal generated by a machine), and the propagation signal is generated to encode information for transmission to a suitable receiver device to be run by a data processing device. The computer storage medium can be or be included in a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. In addition, although the computer storage medium is not a propagation signal, the computer storage medium can be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer storage medium can also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks or other storage devices).

[0078] The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0079] The term "data processing apparatus" covers all kinds of apparatus, devices and machines for processing data, including, for example, programmable processors, computers, systems on chips, or a combination of one or more of them. The apparatus may include dedicated logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). In addition to hardware, the apparatus may also include code that creates an operating environment for the computer program mentioned, such as code that constitutes processor firmware, protocol stacks, database management systems, operating systems, cross-platform operating environments, virtual machines, or a combination of one or more of them. The apparatus and operating environment can implement a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0080] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language (including compiled or interpreted languages, declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to run on one computer located at one site, or on multiple computers distributed across multiple sites and interconnected by a communications network.

[0081] The processes and logic flows described in this specification can be performed by one or more programmable processors running one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, and the apparatus can also be implemented as special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits).

[0082] For example, processors suitable for running computer programs include general and special microprocessors. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions according to instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data (e.g., magnetic disks, magneto-optical disks, or optical disks), or be operably coupled to receive data from one or more mass storage devices for storing data or transfer data to one or more mass storage devices for storing data, or both. However, a computer does not have to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0083] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or an LCD (liquid crystal display)) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user. For example, a web page is sent to a web browser on a user's client device in response to a request received by the web browser.

[0084] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., a data server), or includes an intermediate component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser) through which a user can interact with an implementation of the subject matter described in this specification, or includes a combination of one or more of such back-end, intermediate, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., adhoc peer-to-peer networks).

[0085] A computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., the result of a user interaction) can be received from the client device at the server.

[0086] Although this specification contains many specific implementation details, these should not be interpreted as limitations on any invention or the scope of protection that can be claimed, but should be interpreted as a specific description of the features of a specific embodiment of a specific invention. Certain features described in this specification in the context of a single embodiment may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments or in any suitable sub-combination. In addition, although the features may be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features in the claimed combination may be cut out from the combination, and the claimed combination may lead to a sub-combination or a variation of the sub-combination.

[0087] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0088] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired results. Additionally, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method (500) for enhancing user interaction with an interface, comprising: displaying (502) a keyboard layer (110) of a user interface in the user interface (100), the keyboard layer comprising a keyboard having at least one user interface element configured to receive user input; detecting (504) an initial input at a given user interface element (112); In response to the detected initial interaction, while maintaining display of a keyboard layer (110) of the user interface, revealing (506) a first portion (114) of a symbol layer of the user interface, the symbol layer having at least one user interface element configured to receive user input indicating a symbol; detecting (510) a subsequent input at the given user interface element (112); In response to detecting the subsequent input, a larger portion (116) of the symbol layer of the user interface relative to the first portion is revealed (512) while maintaining a position of the first portion (114) of the symbol layer that was previously revealed in response to detecting the initial input, wherein revealing the larger portion (116) of the symbol layer relative to the first portion obscures at least a portion of the keyboard layer (110).

2. The method according to claim 1, wherein: Revealing the first portion (114) of the symbol layer includes revealing at least one row of user interface elements, each user interface element in the row being configured to receive user input indicating a frequently used symbol, and wherein the row is displayed at a location above the keyboard.

3. The method according to claim 1 or claim 2, wherein: Revealing the first portion (114) of the symbol layer includes revealing at least one row of user interface elements, each user interface element in the row being configured to receive user input indicating an emoticon, a Chinese character, a shape, or a web design widget.

4. The method according to claim 1 or claim 2, further comprising: A set of symbols included in a first portion (114) of the symbol layer is selected (508) based on a frequency of use of symbols included in a larger portion (116) of the symbol layer relative to the first portion.

5. The method according to claim 4, wherein: Selecting the symbol set includes: Identify specific symbols with the highest frequency of use; comprising a user interface element configured to receive a user input indicating the particular symbol at a first position in a first portion (114) of the symbol layer; identifying a set of additional symbols, each additional symbol being used less frequently than the particular symbol but more frequently than the remaining symbols in the symbol layer; and A plurality of user interface elements are included, each of the plurality of user interface elements being configured to receive user input indicating a corresponding one of the additional set of symbols at other locations of the first portion (114) of the symbol layer.

6. The method according to claim 5, further comprising: Initially populating a first portion of the symbol layer with a default symbol set (114) prior to receiving user input at the symbol layer; In response to an initial set of user inputs at the symbol layer, while maintaining display of the first portion of the symbol layer, updating the first portion (114) of the symbol layer to include a user interface element configured to receive user input indicating the particular symbol with the highest frequency of use at a first position in the first portion (114) of the symbol layer.

7. The method according to claim 6 also includes determining the default symbol set based on one or more of a machine learning model that provides a recommended symbol set, contextual information about a current user session, or based on a set of user-specified preferences.

8. The method according to claim 4, wherein: The specific symbol having the highest frequency of use is identified based on frequency of use data collected from applications other than the specific application that caused the first portion (114) of the symbol layer to be displayed.

9. A computer program product having instructions which, when executed by one or more data processing devices (400), cause the data processing devices to perform the method of any one of claims 1 to 8.

10. A computer readable medium storing instructions which, when executed by one or more data processing devices (400), cause the data processing devices to perform the method of any one of claims 1 to 8.

11. A system (600) for enhancing user interaction with an interface, comprising: Data storage device (630); and One or more processors (610) configured to interact with the data storage device and execute the method of any one of claims 1 to 8.