Intelligent virtual keyboard

Abstract

Systems, methods, and computer media for dynamically generating and displaying task icons within a virtual keyboard based on intent are provided herein. A system can include a processor, an intent classifier, and a user interface generator. The intent classifier can be configured to determine a user intent candidate based on context information. The user interface generator can be configured to generate a virtual keyboard for display and generate a task icon within the virtual keyboard upon receiving an indication of a user intent determined based on the user intent candidate. The task icon represents a function associated with the determined user intent. Interaction with the task icon in the virtual keyboard can initiate the function associated with the determined intent. The user interface generator can also be configured to present a search interface upon receiving an indication of a user interaction with the virtual keyboard.

Description

Background Technology

[0001] With the advent of touchscreens and mobile devices, virtual keyboards have become commonplace. For example, a virtual keyboard typically appears when a user taps the screen to type text while using an app on a touchscreen device. Despite their advantages, virtual keyboards are often difficult to use and cumbersome for certain activities. Summary of the Invention

[0002] The examples described herein relate to the dynamic generation and display of task icons and search functionality within a virtual keyboard, based on intent. The example system may include a processor, an intent classifier, a ranking unit, and a user interface generator. The intent classifier can be configured to be determined by the processor based on contextual information. Contextual information may be, for example, text entered via the virtual keyboard, information related to the application active when the virtual keyboard is displayed, text received in a session within the active application, etc.

[0003] User intent candidates can be selected in different ways. For example, a ranking unit can be configured to rank one or more user intent candidates by the processor and select the candidate as the determined user intent based on the ranking. A user interface generator can be configured to generate a virtual keyboard for display by the processor. Upon receiving an indication of the determined user intent, the user interface generator can also be configured to generate a task icon within the virtual keyboard based on the determined user intent. For example, a task icon can be displayed in the input method editor (IME) of the virtual keyboard. Selecting a task icon in the virtual keyboard or other interactions with the task icon can activate a function associated with the determined intent. User intents can be updated based on additional contextual information, and task icons can be removed from the virtual keyboard if they no longer reflect the updated user intent.

[0004] The user interface generator can also be configured to, in response to receiving an instruction to interact with a portion of the virtual keyboard's IME, render a search interface (e.g., one or more search icons) by the processor. Upon receiving an instruction to interact with the IME, a portion of the virtual keyboard can also be replaced with a search results interface.

[0005] This summary is provided to introduce some concepts in a simplified form, which will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0006] The foregoing and other objects, features and advantages of the claimed subject matter will become more apparent from the following detailed description with reference to the accompanying drawings. Attached Figure Description

[0007] Figure 1 This is a block diagram of an example system capable of dynamically generating intent-based task icons.

[0008] Figure 2 This is a block diagram of an example system capable of dynamically generating intent-based task icons, which has multiple intent classifiers and consortiums.

[0009] Figure 3 This is a diagram illustrating an example method for reconfiguring the user interface, where intent-based task icons are dynamically generated and presented on a virtual keyboard.

[0010] Figures 4A-4D The diagram illustrates the determination of intent and the presentation of calendar task icons on the virtual keyboard.

[0011] Figure 5 The diagram illustrates the determination of intent and the presentation of web service task icons on the virtual keyboard.

[0012] Figures 6A-6D The diagram illustrates the determination of intent, the presentation of map task icons, and the selection of shareable content (current location and estimated time of arrival) from the task icon user interface.

[0013] Figures 7A-7D The diagram illustrates the determination of intent, the presentation of map task icons, and the selection of shareable content (a link to a restaurant) from the task icon user interface.

[0014] Figures 8A-8B The diagram illustrates how intent is defined and instant responses are presented on the virtual keyboard.

[0015] Figures 9A-9B The diagram illustrates the definition of intent and the presentation of media task icons on the virtual keyboard.

[0016] Figures 10A-10B The diagram illustrates the definition of intent and the representation of movie mission icons on the virtual keyboard.

[0017] Figure 11 The diagram illustrates the determination of intent, the presentation of the payment task icon on the virtual keyboard, and the display of the payment task user interface.

[0018] Figures 12A-12H The diagram illustrates various features related to the determination of intent and the presentation of task icons in the virtual keyboard, where the task user interface is presented above the virtual keyboard.

[0019] Figures 13A-13BThe diagram illustrates various features related to the determination of intent and the presentation of task icons in the virtual keyboard, where the task user interface is presented above the virtual keyboard when the messaging app is active and essentially replaces the messaging app user interface.

[0020] Figures 14A-14D The diagram illustrates various features associated with the search tools within the virtual keyboard.

[0021] Figures 15A-15D The diagram illustrates various features associated with the search tools within the virtual keyboard, including search category selection.

[0022] Figure 16 The image shows the search tools within the virtual keyboard and the search user interface above the virtual keyboard.

[0023] Figure 17 The flowchart illustrates an example method for reconfiguring a user interface, in which intent-based task icons are dynamically generated and rendered on a virtual keyboard, and the task icon user interface is rendered to replace a portion of the user interface.

[0024] Figure 18 The flowchart illustrates an example method for reconfiguring a user interface that includes a virtual keyboard, where a task icon user interface is rendered to replace a portion of the virtual keyboard.

[0025] Figure 19 This is a diagram of an example computing system in which some of the described embodiments can be implemented.

[0026] Figure 20 This is an example mobile device that can be used in conjunction with the technologies described in this article.

[0027] Figure 21 This is an example cloud-supported environment that can be used in conjunction with the technologies described in this article.

[0028] Figures 22A-22J The figure shows the search interface presented in response to user interaction with the IME section of the virtual keyboard.

[0029] Figure 23 The figure illustrates an example method for reconfiguring the graphical user interface, where a search interface is presented within a virtual keyboard.

[0030] Figure 24 The diagram illustrates an example method for reconfiguring a graphical user interface, in which an indicator is presented in association with a search icon to reflect that a task icon corresponding to the search category of the search icon is presented but has not yet been interacted with.

[0031] Figures 25A-25GThe diagram illustrates a presentation indicator associated with the search icon, reflecting that a task icon corresponding to the search category of the search icon is presented but has not yet been interacted with.

[0032] Figures 26A-26B The image shows the presentation of the search interface menu in response to user interaction with the virtual keyboard.

[0033] Figures 27A-27B The diagram illustrates the curved layout of the search interface menu in response to user interaction with the virtual keyboard. Detailed Implementation

[0034] Using the systems, methods, and computer-readable media described herein, the intent of a user interacting with a user interface of a computing device can be dynamically determined based on contextual information (e.g., text entered by the user or received from another user), and task icons reflecting the user's intent can be generated and displayed within a virtual keyboard in the user interface. Interaction with the task icons can cause the display of a task icon user interface to replace a portion of the entire user interface (e.g., displayed above or replacing a portion of the virtual keyboard). The task icon user interface provides convenient access to functions corresponding to the user's intent (e.g., via application user interfaces, links, deep links, etc.).

[0035] Unlike traditional methods, the dynamic, intent-based approach described in this paper allows users to access desired functionality directly from the virtual keyboard and / or task icon user interface, without forcing users to exit one application, open another to perform an action, and then switch back to the original application.

[0036] As an example, the described method can be used to determine a user's intent while the user is in a conversation within a messaging application. In an example conversation, a user receives a message saying "Want to grab something to eat?" and then types "Okay, where?" via a virtual keyboard. One or both of these questions can be analyzed to determine that the user's intent is to find a restaurant to meet for dinner. After determining this user intent, a map task icon can be generated and displayed on the virtual keyboard. Selecting, swiping, or otherwise interacting with the map task icon causes the map task icon user interface to appear. The map task icon user interface can show the locations of nearby restaurants, provide links to the restaurant's webpage, and provide shareable restaurant information (e.g., names and addresses that can be inserted into the messaging conversation), etc.

[0037] Therefore, the map task icon user interface provides users with access to map functions (searching for restaurants near the user's current location) without requiring them to exit the messaging app, launch the map app, locate the restaurant, copy the message, switch back to the messaging app, and so on. Users can simply select a restaurant from the map task icon user interface, have the restaurant's information added to the conversation, and continue the uninterrupted conversation flow.

[0038] Search functionality can also be provided to users via a virtual keyboard. As an example, a virtual keyboard could be presented without a search icon, search query text input box, or other search features, and the user could interact with the IME portion of the virtual keyboard to display one or more search icons and / or other aspects of the search interface within the virtual keyboard. By providing search functionality when a user requests it, rather than including the search interface as a permanent part of the virtual keyboard, it avoids cluttering the virtual keyboard with numerous icons or features that the user is not currently interested in.

[0039] In the described example, a dynamic, intent-based approach reduces the computational complexity of performing the desired action, eliminating the computational costs of exiting / starting / restarting the application and navigating through the user interface to locate the desired application. See below for reference. Figures 1 to 27B To describe the example.

[0040] Figure 1 The figure illustrates a system 100 implemented on one or more computing devices 102. The computing device 102 includes at least one processor 104. The computing device 102 may be, for example, a mobile device (such as a smartphone or tablet), a personal computer (such as a desktop, laptop, or notebook computer), or other device.

[0041] User interface generator 106 is configured to generate a virtual keyboard 108 by at least one processor 104 for display in a user interface. The user interface is presented on the display of computing device(s) 102. As used herein, a “virtual keyboard” refers to a user interface having numbers and letters corresponding to those of a physical keyboard (e.g., a laptop computer or PC keyboard). Typically, the characters of the virtual keyboard are arranged similarly to those of a physical keyboard. The virtual keyboard is typically displayed on a touchscreen, and individual characters are selected by touch selection, hover selection, or other interactions with the touchscreen. Projected keyboards, AirType keyboards, and other non-physical keyboards are also considered virtual keyboards.

[0042] User interface generator 106 is also configured to, upon receiving an indication of a determined user intent 110, at least one processor 104 generate a task icon 112 within the virtual keyboard 108 based on the determined user intent 110, as used herein. A “task icon” refers to a graphic, image, text, symbol, or other user interface element that represents a function. The appearance of task icon 112 may reflect the determined user intent 110. For example, if the determined user intent 110 is to watch a movie, task icon 112 could be a graphic or image of a movie studio, theater, ticket, or projector, the text “movie,” etc. In some examples, multiple user intents (e.g., multiple user intents exceeding a confidence or probability threshold) are determined, and multiple task icons are presented. Figure 4A-Figure 1 Example of a task icon is shown in Figure 5.

[0043] In some examples, task icons 112 are displayed in the Input Method Editor (IME) section 114 of the virtual keyboard 108. Figure 1 In the diagram, IME section 114 is shown at the top of virtual keyboard 108. IME section 114 may be part of virtual keyboard 108, displaying autocorrect or word suggestions. IME section 114 may contain various important locations. Using examples of autocorrect suggestions appearing in the IME, the most likely suggestion may be presented in a first position on the left side of IME section 114, the second most likely suggestion may be presented in a second position to the right of the first position, and so on. Task icon 112 may be presented in any location within IME section 114. Task icon 112 may also be presented in different parts of virtual keyboard 108, such as the left or right side, lower left or lower right, etc. Task icon 112 may also be partially obscured as a "peek" of additional information that can be obtained using swiping. In some examples, user interface generator 106 is also configured to remove task icon 112 by at least one processor 104 upon receiving an indication that the determined user intent 110 has been updated (e.g., based on additional contextual information or after a time threshold has elapsed).

[0044] Interaction with the task icon 112 in the virtual keyboard 108 initiates a function associated with the determined user intent 110. Interaction with the task icon 112 can be, for example, touch or hover to select, swipe right (or left, up, or down), pinch, select, press and hold for a threshold amount of time, or other interactions.

[0045] User interface generator 106 can be configured to launch a function associated with the determined user intent 110 in the task icon user interface (e.g., in...). Figure 4B , Figure 6B , Figure 7BFigure 8D Figure 9B , Figure 12G , Figure 12H , Figure 13B (As illustrated in other figures). For example, the user interface generator 106 can be configured to replace a portion of the user interface with a task icon user interface upon receiving an instruction to interact with task icon 112. Functionality associated with the determined user intent 110 can be accessed via the task icon user interface. In some examples, the task icon user interface is displayed to replace a portion of the virtual keyboard 108. As a specific example, the task icon user interface can be displayed to replace a portion of the virtual keyboard 108 below IME section 114 (e.g., in...). Figure 4B and Figure 6B (As illustrated in the diagram). In some examples, a portion of the user interface, except for the virtual keyboard 108, is replaced with a task icon user interface. This is, for example, in... Figure 12B and Figure 13B The image is shown in the diagram. In various examples, the virtual keyboard 108 can be displayed simultaneously with the task icon user interface.

[0046] The task icon user interface may include an application user interface for an application launched by interacting with task icon 112, or may include links or deep links to functions of the application or web services. The application launched by interacting with task icon 112 may be, for example, a map application, an organization application, a money transfer application, a media application (audio, video, and / or images), or a user review application. In some examples, the application user interface for an application launched by interacting with task icon 112 includes shareable content generated by the application and related to the determined user intent 110. Shareable content may include, for example, estimated arrival or departure times, event start or end times, restaurant suggestions, movie suggestions, calendar items, suggested meeting times or locations, public transport information, traffic information, weather or temperature information, or instant response results. In some examples, certain portions of the information may be selected individually for sharing. For example, if restaurant suggestions are shareable content, any combination of address, hours, menu links, directions, reviews, etc., may be shareable. Figure 4B-Figure 1 5. A detailed discussion of the task icon user interface.

[0047] The user interface generator 106 can be implemented as, for example, computer-executable instructions (e.g., software) stored in memory (not shown) and executable by at least one processor 104. The user interface generator 106 can also be implemented, at least in part, using programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)), application-specific integrated circuits (ASICs), or other computing hardware.

[0048] The determined user intent 110 provided to the user interface generator 106 is determined by the intent classifier 116, and in some examples by the ranking classifier 118. The intent classifier 116 is configured to determine one or more user intent candidates based on context information 120 by at least one processor 104. In some examples, the system 100 includes multiple intent classifiers. The context information 120 may include, for example, text entered via the virtual keyboard 108, text received from a remote computing device, commands received via speech recognition, information related to an application that was active when the virtual keyboard 108 was displayed, task icons previously interacted with, the location of user interaction with the user interface, the current time, day or date, or history or preference information.

[0049] As an example, if the messaging app is active when the virtual keyboard 108 is displayed, the context information 120 may include: a message entered via the virtual keyboard 108; a message received from another user; the fact that the messaging app is active; preferences or history associated with other users with whom the user is communicating; or preferences or history associated with a conversation between two users (e.g., the history of the two users meeting for coffee). The context information 120 may also include: actions recently performed by the user (e.g., a recent search for “coffee” in a map app), reminders / alerts, calendar events, or organizing items (e.g., storing “coffee with Kyle” as an upcoming appointment in a calendar app), and so on. As used herein, a messaging app includes a dedicated “chat” app, chat or messaging features in other apps, email apps, or other apps for sending messages between users.

[0050] The intent classifier 116 can determine user intent candidates through various methods, including artificial intelligence and machine learning methods such as Natural Language Understanding (NLU). NLU involves parsing, organizing, and classifying human language (whether received via speech recognition, touch / typing input, or via electronic messages or other electronic communications). For example, NLU can be performed using a template matching method, where text is analyzed to identify specific co-location strings corresponding to known intents. For instance, the template “(airport_code_1) to (airport_code_2)” could correspond to the intent to purchase a flight ticket. In template matching methods, the received text can be compared with many different templates.

[0051] Intent classification can also be performed using statistical models such as logistic regression, augmented decision trees, neural networks, conditional Markov language models, or conditional random fields. In such methods, a training set of text portions labeled with known intents is used to build a statistical model, which is then used to predict the intents of other text encountered at runtime. Collecting a wide variety of large amounts of training data can improve the performance of such methods.

[0052] In some examples, system 100 includes different intent classifiers for different types of functions that can be associated with and accessed via task icon 112. For example, system 100 may include an intent classifier for restaurant recommendations, an intent classifier for directions, an intent classifier for media items, etc. In examples using template matching techniques, different intent classifiers may have different associated templates. Intent classifier 116 can be trained using training data based on previously input text and subsequent actions taken (for the user and / or other users). For example, if a user receives the text “Hungry?” and replies “Hungry,” and then opens a restaurant review or map application, this data can be used as training data to match future received “Hungry” and “I’m hungry” events with the intent to find a restaurant to eat at. Training can also be done based on user interaction with a previously presented task icon or the lack thereof. For example, if a task icon is presented based on a determined intent, and the user selects, swipes, or otherwise interacts with the task icon, it is interpreted that the determined user intent is accurate for the contextual information. Conversely, if a task icon is presented but the user does not interact with it, the interpretation is that the determined user intent is inaccurate. In some examples, training data may be aggregated across users and stored, for example, in a cloud environment accessible to different users. In other examples, the training data is user-specific.

[0053] Search tools can also be used to train intent classifiers. The user's search and the corresponding subsequent action or selected result are used to inform future intent classification. Search tools (e.g., in...) can be used to train intent classifiers. Figure 13A-Figure 1 3D Figures 14A-14D , Figures 15A-15D and Figure 16The search tool (shown in the diagram) is included in the virtual keyboard 108. The search tool may be presented in the IME section 114, for example, and can be accessed by interacting with a search icon (e.g., a magnifying glass, a question mark, etc.) or by performing a specific gesture or combination of gestures. For example, swiping the IME section 114 (e.g., swiping left or right) can cause the search tool, which has a text input area, to be displayed in the IME section 114. Swiping the IME 114 in the opposite direction or selecting the exit button can cause the search tool to disappear. The search tool can also be a voice recognition search tool, which begins to "listen" when interacting with the search icon. In some examples, the search tool is displayed when the virtual keyboard 108 is displayed. As a specific example, when the virtual keyboard 108 is activated, the search tool appears in or below the IME section 114. In some examples where the search tool is presented in the IME section 114, the search tool disappears when a task icon 112 is generated and presented due to the use of an application using the virtual keyboard 108 (or when an autocorrect suggestion is generated). System 100 may include a search engine (not shown) configured to perform searches received via a search tool, and user interface generator 106 may be configured to render the search tool.

[0054] The intent classifier 116 can be implemented as, for example, computer-executable instructions (e.g., software) stored in memory (not shown) and executable by at least one processor 104. The intent classifier 116 can also be implemented, at least in part, using programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)), application-specific integrated circuits (ASICs), or other computing hardware. The intent classifier 116 can also be implemented using neural networks (e.g., deep neural networks, convolutional neural networks, etc.). In some examples, the intent classifier 116 can be implemented in a cloud computing environment (e.g., such as...). Figure 21 The system 100 is implemented in the cloud (2110), and the system 100 communicates with the intent classifier 116 via a network (such as the Internet). In some examples, some functions of the intent classifier 116 can be implemented in the system 100, while other functions of the intent classifier 116 are implemented in the cloud.

[0055] In some examples, the intent classifier 116 performs ranking of one or more user intent candidates or otherwise selects one or more user intent candidates (e.g., selecting the candidate with the highest probability, etc.) as the determined user intent 110. In other examples, the ranker 118 is configured to rank one or more user intent candidates determined by the intent classifier 116 by at least one processor 104, and select at least one user intent candidate as the determined user intent 110 based on the ranking. The ranker 118 can be configured to apply various ranking methods, such as selecting the user intent candidate with the highest confidence level (e.g., the probability of being correct). In some examples, the functionality of the ranker 118 is combined with that of the intent classifier 116. Ranking can be accomplished, for example, by calibrating the intent classifier 116 using isotonic or logistic regression, and then using an augmented decision tree or neural network trained under a ranking loss function, or other methods of sorting by classifier output. For example, the selection of the determined user intent 110 can be accomplished by thresholding the scores used for ranking.

[0056] For example, the ranking unit 118 can be implemented as computer-executable instructions (e.g., software) stored in memory (not shown) and executable by at least one processor 104. The ranking unit 118 can also be implemented at least in part using programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)), application-specific integrated circuits (ASICs), or other computing hardware.

[0057] Figure 2 The figure illustrates a system 200 implemented on one or more computing devices 202. The computing device 202 includes at least one processor 204. The computing device 202 may be similar to... Figure 1 The computing device 102. The user interface generator 206 and ranking device 208 can also be similar. Figure 1 The corresponding components in the system 200 include multiple intent classifiers 210. A concatenator 212 is configured to distribute contextual information to the intent classifiers 210 by at least one processor 204. The concatenator 212 is also configured to determine aggregate groups of user intent candidates based on the user intent candidates determined by the intent classifiers 210. A ranking unit 208 is further configured to rank the user intent candidates in the aggregate groups by at least one processor 204.

[0058] The user interface generator 206, the ranking unit 208, and the combiner 212 can be implemented as, for example, computer-executable instructions (e.g., software) stored in memory (not shown) and executable by at least one processor 204. The user interface generator 206, the ranking unit 208, and the combiner 212 can also be implemented, at least in part, using programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)), application-specific integrated circuits (ASICs), or other computing hardware.

[0059] Decoder 214 is configured to interpret and recognize touch and hover input to a virtual keyboard (not shown) by at least one processor 204. Decoder 214 may be configured to recognize touch / tapping as well as swiping. Decoder 214 may be configured to interpret the input based on one or more touch models 216 and language models 218 (e.g., a user's language model). Touch model 216 is configured to evaluate how well various hypotheses about which word the user wants when inputting text match the touch and hover input. Language model 218 is configured to evaluate how well these hypotheses fit the input words. Autocorrector 220 is configured to provide, for example, autocorrect suggestions, spelling suggestions, etc., that can be presented in the IME of the virtual keyboard by at least one processor 204.

[0060] Similar to Figure 1 User interface generator 106 and user interface generator 206 are configured to generate task icons within a virtual keyboard that reflect determined user intents, and to generate task icon user interfaces upon receiving an instruction to interact with the task icons, providing access to functionality corresponding to the determined intents. The task icon user interface may include links or deep links to one or more local services or applications 222 or web services or applications 224. The task icon user interface may also include application interfaces for one or more local services or applications 222 or web services or applications 224.

[0061] Figure 3 The figure illustrates an example method 300 for reconfiguring a user interface. In processing box 302, a virtual keyboard is generated. In processing box 304, context information is received. Context information may include, for example, text entered via the virtual keyboard, text received from a remote computing device, commands received via voice recognition, or information related to the application that is active when the virtual keyboard is displayed. In processing box 306, the user intent is determined. For example, an intent classifier and ranking unit can be used to determine the user intent. In processing box 308, a task icon is generated within the virtual keyboard. For example, the task icon can be presented in the keyboard's IME. For example, it can be used... Figure 1 System 100 or Figure 2 System 200 executes method 300.

[0062] Figures 4A-4D The diagram illustrates the determination of user intent and the presentation of calendar task icons. Figure 4A In the user interface 400, the messaging application is active and a session 402 is displayed. The user interface 400 also includes a virtual keyboard 404 with an IME portion 406. Based on contextual information obtained from the session 402 (e.g., the question “Want to go out for dinner this week?” and the reply “I’m free”), the user’s intent is determined to identify a time when they can meet for dinner. For example, the contextual information may also include: whether “Ellen” is a frequent contact (and therefore someone the user is more likely to meet for dinner with), the user’s statement “Happy Birthday” with the accompanying birthday cake emoji, whether the email indicates that Ellen will be in the same location as the user, etc. After determining this intent, a task icon 408 is generated and presented in the IME portion 406 of the virtual keyboard 404. A text input box 409 is also shown in the user interface 400. The text input box 409 is part of the messaging application and not part of the virtual keyboard 404.

[0063] like Figure 4B As shown, interactions with task icon 408 (such as selection, swiping left or right, etc.) cause the task icon user interface 410 to replace a portion of user interface 400. Figure 4B In this context, the task icon user interface 410 is presented to replace a portion of the virtual keyboard 404. Specifically, the portion of the virtual keyboard 404 below the IME portion 406 has been replaced by the task icon user interface 410, but the IME portion 406 is still displayed. The portion of the user interface 400 displaying session 402 remains unchanged. Functions associated with the determined intent can be accessed via the task icon user interface 410. The task icon user interface 410 includes an application user interface for a calendar application, which displays time blocks for different dates. The appearance of the task icon 408 (month view of the calendar) reflects the determined user intent and the functions that can be initiated in the task icon user interface 410 by interacting with the task icon 408. The task icon 408 can be highlighted (e.g., as shown in the image). Figure 4A and Figure 4B The underlines and bolding, distortion, or other emphasis shown indicate that interaction with task icon 408 initiates a function. Keyboard icon 412 is presented in IME section 406 to allow the user to replace the task icon user interface 410 with the character keys of the virtual keyboard 404. Figure 4C and Figure 4D As shown, in the task icon user interface 410, the time frame is selectable and can be added to the session 402.

[0064] exist Figure 4C In the text input box 409, two possible times have been selected: timeframe 414 and timeframe 416. The text input box 409 indicates "Selected Event: 2". Timeframes 414 and 416 are shareable content that can be selected and added to session 402. Figure 4D As shown, by taking another action (such as pressing "Send", dragging and dropping, etc.), a "Available Time:" session entry 418, including timeframes 414 and 416, is added to session 402. After session entry 418 is added, task icon 408 is no longer displayed in IME section 406. Figures 4A-4D As illustrated, dynamically determining user intent and generating and presenting task icons corresponding to that intent allows users to perform actions and access other applications without interrupting the session.

[0065] Figure 5 The diagram illustrates a user interface 500 including a session 502 being implemented in a messaging application. A virtual keyboard 504 includes an IME section 506. The user's intent to open a file from a web service is determined based on the statement "Hi, I just updated salespitch.pptx" in session 502. A web service task icon 508, along with auto-suggestions 510 ("In a meeting") and 512 ("I'm busy"), are presented in the IME section 506. Figure 5 In the IME, Auto Suggestion 510 is located in IME location 1, Auto Suggestion 512 is located in IME location 2, and Task Icon 508 is located in IME location 3. Interaction with Task Icon 508 initiates functions associated with a web service (not shown), such as links or deep links to the "salespitch.pptx" file, access to shared workspaces or folders, and the web service user interface.

[0066] Figure 6A The diagram illustrates a user interface 600 including a session 602 being implemented in a messaging application. A virtual keyboard 604 includes an IME section 606. Based on the statements in session 602, such as “Hi, where are you? Just got a table…” and / or “I’m at…”, the user’s intent is determined to be sharing their current location (and / or providing an estimated time of arrival, etc.). A map task icon 608 is then presented in the IME section 606 to reflect the determined intent. Figure 6B As shown, interaction with the map task icon 608 causes the map task icon user interface 610 to be displayed in place of a portion of the virtual keyboard 604.

[0067] The map task icon user interface 610 includes a map of the user's current location and destination, as well as shareable content items 612, 614, and 616, which respectively indicate the user's estimated arrival time by car, public transport, or on foot. Figure 6C In the map task icon user interface 610, shareable content item 614 has been selected (as indicated by the bold text of shareable content item 614), and the acquired public transport route between the user's current location and destination is displayed. Figure 6D In this context, session entry 618 has been added to session 602. Session entry 618 reflects shareable content item 614.

[0068] Figure 7A The diagram illustrates a user interface 700 including a session 702 being implemented in a messaging application. A virtual keyboard 704 includes an IME section 706. The user's intent is determined to be meeting somewhere (e.g., having dinner at a restaurant) based on the statements "Want to get together tonight?" and / or "Sure, how about it?". Map task icons 708 are presented in the IME section 706. Figure 7B As shown, interaction with the map task icon 708 causes the map task icon user interface 710 to appear, replacing a portion of the virtual keyboard 704. The map task icon user interface 710 displays the user's current location and lists nearby restaurants.

[0069] like Figure 7A and Figure 7B As shown, IME section 706 also includes additional task icons (such as movie task icon 712) generated based on context information, which... Figure 7A Partially obscured, but Figure 7B The virtual keyboard is visible. In some cases, multiple user intentions are possible. Based on the statements in session 702, a user might be interested in meeting for dinner, meeting for coffee, meeting for a movie, meeting to return an item, etc. Therefore, multiple task icons can be generated and presented within the virtual keyboard. These task icons can be presented in an order of probability determined (e.g., based on confidence level, user history, etc.). Figure 7B As shown, map task icon 708, movie task icon 712, restaurant task icon 714 and other task icons are presented in IME section 706.

[0070] In some examples, the first task icon can be associated with other task icons that represent a subset of the functions of the first task icon. For example, map task icon 708 can activate various different map functions (e.g., estimated arrival time, restaurant location, shop location, etc.). Therefore, as... Figure 7B As illustrated, map task icon 708 has been selected, and map task icon user interface 710 is associated with the restaurant aspect of the map represented by restaurant task icon 714. In user interface 700, the user can slide or select other task icons to change the task icon user interface displayed below IME section 706.

[0071] exist Figure 7C In the list, Shareable Content Item 716 (Restaurant "Mamnoon") has been selected (as indicated by the bold text of Shareable Content Item 716). Figure 7D In this context, session entry 718 has been added to session 702. Session entry 702 reflects the shareable content item 718 and lists the restaurant's name and address. After adding session entry 702, the task icons, including task icons 708 and 712, have been removed from IME section 706.

[0072] Figure 8A and Figure 8B This involves immediate, intent-based responses. Figure 8A In the user interface 800 (e.g., in an email or messaging application), a new message is being composed containing the text "I'm inviting 25 kids x $9 meals x $4 drinks". Some or all of this text can be used to determine that the user's intent is to calculate the total cost. An instant response result 802 indicating the total cost is displayed within the IME section 804 of the virtual keyboard 806. Similarly, in Figure 8B In the user interface 808, a new message containing the text "∈45 Gift" is being composed. Some or all of this text can be used to determine that the user intends to calculate the dollar equivalent price. An instant response result 810 indicating the dollar price is displayed within the IME section 804 of the virtual keyboard 806.

[0073] Figure 9A and Figure 9B The diagram illustrates a user interface 900 displaying session 902. Based on contextual information (e.g., the session entry “Did you watch the Seahawks game???”), the user's intent is determined to be to view or share a video of a Seattle Seahawks football game. An instant response result 904, showing the score of the most recent game, is provided in the IME section 906 of the virtual keyboard 908. Figure 9B As illustrated, one or more task icons can also be generated and displayed on the virtual keyboard 908. For example, additional task icons can be revealed by swiping the instant response result 904. Figure 9B Multiple task icons, including a rugby media task icon 910, are shown. Interacting with the rugby media task icon 910 causes a portion of the virtual keyboard 908 to be replaced by a rugby media task user interface 912, which displays shareable and / or viewable rugby game video clips and / or images. For example, a user can select or drag and drop thumbnails of video clips into session 902.

[0074] Figure 10A and Figure 10BThe diagram illustrates a user interface 1000 in which session 1002 is displayed. Based on contextual information (e.g., session entries “We can also watch a movie after dinner” and / or “Yes! Let’s go buy Star Wars tickets”), the user’s intent is determined to be to watch the movie “Star Wars”. Movie task icons 1004 are presented in the IME section 1006 of the virtual keyboard 1008. Figure 10B As shown, interaction with the movie mission icon 1004 causes a movie mission icon user interface 1010 to appear, replacing a portion of the virtual keyboard 1008. The movie mission icon user interface 1010 displays the showing times of "Star Wars" at theaters near the user's current location. The movie mission icon user interface 1010 may be a movie ticket purchase / reservation service application user interface and / or may contain links to movie services or deep links for purchasing tickets for specific performances. Figure 10B In the middle, the theater “iPic Redmond(2D)” is selected and displayed as a text entry in text input box 1012. This text entry can then be added to session 1002.

[0075] Figure 11 The diagram illustrates the user interface 1100 in which session 1102 is displayed. Based on contextual information (e.g., session entries “Hi, got tickets for us, $25 each” and / or “Thanks! I’ll send you the money”), the user’s intent is determined to be a fund transfer. The fund transfer is considered a transaction service; the transaction service application could be a fund transfer application or other transaction-based application. The fund transfer task icon 1104 is presented in the IME section 1106 of the virtual keyboard. Figure 11 Only IME section 1106 is shown in the image. Figure 11 Also shown is a funds transfer task icon user interface 1108, which replaces a portion of the virtual keyboard after interaction with the funds transfer task icon 1104. The funds transfer task icon user interface contains deep links 1110 and 1112 that can be selected to transfer funds using different funds transfer services. Instructions to send funds can then be added to session 1102.

[0076] Figures 12A-12H The image shows an example where the task icon user interface replaces a portion of the entire user interface (above the virtual keyboard) instead of replacing a portion of the virtual keyboard. Figure 12A The diagram illustrates user interface 1200, where "I'm meeting Kyle" has been entered into a text input box 1202 of a messaging application. Based on contextual information (e.g., the text entry "I'm meeting Kyle"), the user's intent is determined to be accessing Kyle's contact information. A contact task icon 1204 is displayed in the IME section 1206 of the virtual keyboard 1208. Figure 12B As shown, interaction with task icon 1204 causes a portion of user interface 1200 (the portion immediately above IME portion 1206) to be replaced with contact task icon user interface 1210. Contact task icon user interface 1210 includes functionality for contact or organizer applications and displays Kyle's contact information.

[0077] Figure 12C The diagram illustrates the relationship with Figure 12C An extended contact task icon user interface 1212 is presented when interacting with the contact task icon user interface 1210. The extended contact task icon user interface 1212 is presented to replace a portion of the virtual keyboard 1208 (i.e., replace the character keys). The IME portion 1206 is moved to the bottom of the user interface 1200. Figure 12C In the process, before the extended contact task icon user interface 1212 is presented, the portion of the user interface 1200 that can be used to display messages remains consistent with... Figure 12B Same as above.

[0078] exist Figure 12D In the current context, the user has exited the extended contact task icon user interface 1212 and continues typing in the text input box 1204, which now reads "I'm meeting Kyle for lunch at Din Tai Fung". Based on the updated context information (the additional text "having lunch at Din Tai Fung"), the updated user intent is determined to be confirming / sharing restaurant locations. The restaurant task icon 1214 is displayed in the IME section 1206 to reflect the updated user intent, and the contact task icon 1204 is removed from the IME section 1206. In some examples, task icons that are presented due to updated user intent but subsequently removed can be represented by indicators (e.g., numbers or other indicators in the IME or elsewhere on the virtual keyboard), and these task icons can be redisplayed when the user interacts with the indicators.

[0079] exist Figure 12E In this context, user interaction with the restaurant task icon 1214 causes a portion of the user interface 1200 to be replaced by the restaurant task icon user interface 1216, which provides contact information for the restaurant "Din Tai Fung". Figure 12F In the middle, the user interface 1200 reflects the user's selection. Figure 12E The exit button 1218 shown in the figure, as well as the restaurant task icon user interface 1216 and restaurant task icon 1214, have disappeared.

[0080] In some examples, after receiving additional contextual information but determining the additional intent based on that contextual information, if it is determined that the task icon still reflects the determined user intent, then both the additional task icon and the original task icon can be displayed. Figure 12G and Figure 12H In the IME section 1206, based on updated contextual information (with the additional text "Having lunch at Din Tai Fung, want to join?"), the contact task icon 1204 and restaurant task icon 1214 are displayed. User interaction with the contact task icon 1204 causes the contact task icon user interface 1220 to replace a portion of the user interface 1200. Figure 12G In this context, multiple task icon user interfaces are presented to correspond to multiple task icons. A portion of the restaurant task icon user interface 1222 is visible next to the contact task icon user interface 1220. The different task icon user interfaces can be scrollable. For example, a user can swipe or scroll the contact task icon user interface 1220 left or right, or swipe / select the restaurant task icon 1214 to display the restaurant task icon user interface 1222. Figure 12H The following example illustrates a scenario where the restaurant task icon user interface 1222 is selected, and then a portion of the virtual keyboard 1208 is replaced by an extended restaurant task icon user interface 1224 during user interaction with the restaurant task icon user interface 1222.

[0081] Figure 13A The figure illustrates a user interface 1300 in which the phrase "I met Kyle for lunch at Din Tai Fung" has been entered into a text input box 1302 in a messaging application. Based on contextual information (e.g., the text entry "I met Kyle for lunch at Din Tai Fung"), the user's intent is determined to be to access / share restaurant information. A restaurant task icon 1304 is presented in the IME portion 1306 of the virtual keyboard 1308. As shown in Figure 13B, interaction with the task icon 1304 causes a portion of the user interface 1300 (the portion above the IME portion 1206) to be replaced by a restaurant task icon user interface 1310. While the restaurant task icon user interface 1310 is presented, the virtual keyboard 1308 continues to be displayed.

[0082] Figures 14A-16 The diagram illustrates various example user interfaces where search tools are presented within a virtual keyboard. Figure 14A In this configuration, the user interface 1400 includes a virtual keyboard 1402 with an IME section 1404. A search tool, represented by a magnifying glass icon 1406, is presented in the IME section 1404. In some examples, the search tool is accessed (and the magnifying glass icon 1406 is displayed) by swiping the IME or selecting another icon or character displayed within the virtual keyboard 1402. Figure 14AIn the search results, the user entered "pizza". The search results user interface 1408 then displayed results for various categories, such as emojis, restaurants, etc.

[0083] exist Figure 14B In the IME section 1404, different task icons corresponding to various categories are displayed. Figure 14B In the search results, selecting emoji icon 1410 results only for "pizza" in the search results user interface 1408. Emoji icon 1410 is not considered a task icon because emojis are not associated with a function. Figure 14C In the middle, select restaurant task icon 1412, and the search results user interface 1408 will only display restaurant results. Similarly, in Figure 14D In the search results user interface 1408, image task icon 1414 is selected, and only image results are displayed. In some examples, similar to emoji icon 1410, image task icon 1414 is not considered a task icon unless a function is associated with it. Similar to the task icon user interface, the search results user interface 1408 can provide access to functions (e.g., via links, deep links, or applied user interfaces).

[0084] Figure 15A The figure shows a sample user interface 1500 including a virtual keyboard 1502 with an IME section 1504. Figure 15A The diagram illustrates an example of the initial display of the search tool (as shown by the magnifying glass icon 1506). For example, the search tool may appear after the user swipes or otherwise interacts with the IME section 1504. The search tool can be hidden by selecting the exit button 1508. In some examples, the search tool offers the option to select search results corresponding to various categories. Figure 15B The diagram illustrates various task icons corresponding to the categories presented in IME section 1504. Category-based search can... Figure 15A The alternative layouts shown in the image, or Figure 15A and Figure 15B The layout can be switched or selected between the two.

[0085] Figure 15C The diagram illustrates user choices for specific search categories. Figure 15C In this context, as indicated by the image task icon 1510 shown in IME section 1504, image search is selected. Figure 15D In the search bar, the user interface 1512 displays the search results, including image results, for the search term "speaker".

[0086] Figure 16The figure illustrates user interface 1600, in which search results user interface 1602 is presented above virtual keyboard 1604 to replace a portion of user interface 1600.

[0087] exist Figures 4A to 16 In this example, the task icon user interface, the extended task icon user interface, and the search results user interface are presented in different locations, replacing different parts of the overall user interface. It can be expected that any location of the task icon user interface described herein can be used with any example. The specific configuration and examples chosen are for illustrative purposes and are not intended to be limiting.

[0088] In some examples, task icons are generated and presented based on previous manual searches. For instance, a task icon corresponding to a previous search entered via a search tool (e.g., in the IME section of the virtual keyboard) can be presented in the virtual keyboard. In some examples, a task icon corresponding to a previous search can be presented before the current user intent is determined.

[0089] Figure 17 The figure illustrates a method 1700 for reconfiguring a user interface on a computing device. In processing block 1702, a virtual keyboard is presented in the graphical user interface. In processing block 1704, one or more text entries are received. In processing block 1706, a user intent is determined using one or more intent classifiers, at least in part based on the one or more text entries. Upon determining the user intent, in processing block 1708, a task icon representing a function corresponding to the user intent is presented within the virtual keyboard. In processing block 1710, after the user selects a task icon, a task icon user interface providing access to the function corresponding to the user intent is presented, replacing a portion of the graphical user interface.

[0090] Figure 18The figure illustrates a method 1800 for reconfiguring a graphical user interface. In processing box 1802, a virtual keyboard is presented in the graphical user interface when a first application is active. The virtual keyboard has an Input Method Editor (IME) portion. In processing box 1804, a user intent is determined based at least in part on contextual information for the first application. The contextual information includes at least one of the following: text entered via the virtual keyboard, text received via the first application, or information related to the first application. In processing box 1806, a task icon is presented within the IME portion of the virtual keyboard. The task icon is linked to a function reflecting the user intent. In processing box 1808, upon receiving an instruction to select a task icon, a task icon user interface is presented to replace a portion of the virtual keyboard. The task icon user interface includes at least one of the following: an application user interface for a second application, shareable content generated by the second application, or deep links to functions of the second application or web services.

[0091] Example computing system

[0092] Figure 19 A generalized example of a suitable computing system 1900 in which the described innovations can be realized is depicted. The computing system 1900 is not intended to impose any limitations on the scope of its use or functionality, as innovations can be realized in various general-purpose or special-purpose computing systems.

[0093] refer to Figure 19 The computing system 1900 includes one or more processing units 1910, 1915 and memories 1920, 1925. Figure 19 In the diagram, the basic configuration 1930 is included within the dashed lines. Processing units 1910 and 1915 execute computer-executable instructions. Processing units can be general-purpose central processing units (CPUs), processors in application-specific integrated circuits (ASICs), or any other type of processor. In a multiprocessor system, multiple processing units execute computer-executable instructions to increase processing power. For example, Figure 19 A central processing unit 1910 and a graphics processing unit or coprocessor 1915 are shown. Physical memories 1920 and 1925 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination thereof, accessible by the processing unit(s). Memories 1920 and 1925 store software 1980 implementing one or more innovations described herein in the form of computer-executable instructions suitable for execution by the processing unit(s). For example, memories 1920 and 1925 may store... Figure 1 Intent classifier 116, ranking unit 118 and / or user interface generator 106 and / or Figure 2 The user interface generator 206, the ranking unit 208, the concatenator 212, the decoder 214, the autocorrector 220, and / or the intent classifier 210.

[0094] The computing system may have additional features. For example, computing system 1900 includes storage 1940, one or more input devices 1950, one or more output devices 1960, and one or more communication connections 1970. Interconnection mechanisms (not shown), such as buses, controllers, or networks, interconnect the components of computing system 1900. Typically, operating system software (not shown) provides an operating environment for other software executing in computing system 1900 and coordinates the activities of the components of computing system 1900.

[0095] Physical storage 1940 may be removable or non-removable and includes magnetic disks, magnetic tapes or cartridges, CD-ROMs, DVDs, or any other medium that can be used to store information and is accessible within computing system 1900. Storage 1940 stores instructions for implementing one or more of the innovative software 1980 described herein. For example, storage 1940 may store… Figure 1 Intent classifier 116, ranking unit 118 and / or user interface generator 106 and / or Figure 2 The user interface generator 206, the ranking unit 208, the concatenator 212, the decoder 214, the autocorrector 220, and / or the intent classifier 210.

[0096] Multiple input devices 1950 may be touch input devices (such as keyboards, mice, pens, or trackballs), voice input devices, scanning devices, or other devices that provide input to the computing system 1900. For video encoding, multiple input devices 1950 may be cameras, video cards, television tuner cards, or similar devices that accept video input in analog or digital form, or CD-ROMs or CD-RWs that read video samples into the computing system 1900. Multiple output devices 1960 may be monitors, printers, speakers, CD burners, or other devices that provide output from the computing system 1900.

[0097] Multiple communication connections established in 1970 enable communication with another computing entity via a communication medium. The communication medium transmits information or other data, such as computer-executable instructions, audio or video input or output, in the form of modulated data signals. Modulated data signals are signals whose information is encoded in a manner that sets or alters one or more of their characteristics. By way of example and not limitation, the communication medium may be electrical, optical, RF, or other carriers.

[0098] Innovations can be described within the general context of computer-executable instructions, such as those included in a program module, which are executed on a computing system targeting a real or virtual processor. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among them as desired. The computer-executable instructions for program modules can be executed within a local or distributed computing system.

[0099] The terms “system” and “device” are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computing system or computing device. Generally, a computing system or computing device can be local or distributed and can include any combination of dedicated hardware and / or general-purpose hardware with software that implements the functions described herein.

[0100] For presentational purposes, the specific implementation uses terms such as "determine" and "use" to describe computer operations in a computing system. These terms are high-level abstractions of the operations performed by the computer and should not be confused with actions performed by humans. The actual computer operations corresponding to these terms vary depending on the implementation.

[0101] Example mobile devices

[0102] Figure 20 This is a system diagram depicting an example mobile device 2000 including various optional hardware and software components, generally shown as 2002. Any component 2002 in the mobile device can communicate with any other component, but not all connections are shown for illustration purposes. The mobile device can be any of a variety of computing devices (e.g., cellular phones, smartphones, handheld computers, personal digital assistants (PDAs), etc.) and can allow for two-way wireless communication with one or more mobile communication networks 2004 (such as cellular, satellite, or other networks).

[0103] The illustrated mobile device 2000 may include a controller or processor 2010 (e.g., a signal processor, microprocessor, ASIC, or other control and processing logic circuitry) for performing tasks such as signal encoding, data processing, input / output processing, power control, and / or other functions. The operating system 2012 may control the allocation and use of components 2002 and support for one or more applications 2014. Applications may include common mobile computing applications (e.g., email applications, calendars, contact managers, web browsers, messaging applications) or any other computing application. Applications 2014 may also include virtual keyboards, task icons, and user interface reconfiguration technologies. A function 2013 for accessing an app store may also be used to obtain and update applications 2014.

[0104] The illustrated mobile device 2000 may include memory 2020. Memory 2020 may include non-removable memory 2022 and / or removable memory 2024. Non-removable memory 2022 may include RAM, ROM, flash memory, hard disk, or other known memory storage technologies. Removable memory 2024 may include flash memory or a Subscriber Identity Module (SIM) card, which is known in GSM communication systems, or other known memory storage technologies such as a "smart card." Memory 2020 may be used to store data and / or code for running operating system 2012 and applications 2014. Example data may include web pages, text, images, sound files, video data, or other datasets to be sent to and / or received from one or more network servers or other devices via one or more wired or wireless networks. Memory 2020 may be used to store device identifiers, such as subscriber identifiers like International Mobile Subscriber Identity (IMSI), and such as International Mobile Equipment Identity (IMEI). Such identifiers may be transmitted to a network server to identify users and devices.

[0105] Mobile device 2000 may support one or more input devices 2030 (such as touchscreen 2032, microphone 2034, camera 2036, physical keyboard 2038, and / or trackball 2040) and one or more output devices 2050 (such as speaker 2052 and display 2054). Other possible output devices (not shown) may include piezoelectric or other haptic output devices. Some devices may provide more than one input / output function. For example, touchscreen 2032 and display 2054 may be combined into a single input / output device.

[0106] Input device 2030 may include a Natural User Interface (NUI). An NUI is any interface technology that enables users to interact with a device in a “natural” manner, free from the artificial constraints imposed by input devices such as a mouse, keyboard, or remote control. Examples of NUI methods include those relying on speech recognition, touch and stylus recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, speech and voice, vision, touch, gestures, and machine intelligence. Other examples of NUIs include motion gesture detection using accelerometers / gyroscopes, facial recognition, 3D displays, head, eye, and gaze tracking, immersive augmented reality and virtual reality systems—all of which provide a more natural interface—and technologies for sensing brain activity using electric field-sensing electrodes (EEG and related methods). Thus, in a specific example, operating system 2012 or application 2014 may include speech recognition software as part of a voice user interface that allows users to operate device 2000 via voice commands. Furthermore, device 2000 may include input devices and software that allows user interaction via spatial gestures (such as detecting and interpreting gestures to provide input to a game application).

[0107] As is well known in the art, the wireless modem 2060 may be coupled to an antenna (not shown) and may support bidirectional communication between the processor 2010 and external devices. The modem 2060 is generally shown and may include a cellular modem for communicating with the mobile communication network 2004 and / or other radio-based modems (e.g., Bluetooth 2064 or Wi-Fi 2062). The wireless modem 2060 is typically configured to communicate with one or more cellular networks (such as GSM networks) for data and voice communication within a single cellular network, between cellular networks, or between mobile devices and the Public Switched Telephone Network (PSTN).

[0108] Mobile devices may also include at least one input / output port 2080, a power supply 2082, a satellite navigation system receiver 2084 such as a Global Positioning System (GPS) receiver, an accelerometer 2086, and / or a physical connector 2090, which may be a USB port, an IEEE 1394 (FireWire) port, and / or an RS-232 port. Component 2002 illustrated is not required or includes all components, as any component can be removed and other components can be added.

[0109] Example cloud supported environments

[0110] Figure 21The figure illustrates a generalized example of a suitable cloud-supported environment 2100 in which the described embodiments, technologies, and techniques can be implemented. In the example environment 2100, cloud 2110 provides various types of services (e.g., computing services). For example, cloud 2110 may include a collection of computing devices that may be centralized or distributed, providing cloud-based services to various types of users and devices connected via a network (such as the Internet). Computing tasks can be performed using implementation environment 2100 in different ways. For example, some tasks (e.g., processing user input and presenting a user interface) may be performed on local computing devices (e.g., connected devices 2130, 2140, 2150), while other tasks (e.g., storing data that will be used in subsequent processing) may be performed in cloud 2110.

[0111] In example environment 2100, cloud 2110 provides services with various screen capabilities to connected devices 2130, 2140, and 2150. Connected device 2130 represents a device with a computer screen 2135 (e.g., a medium-sized screen). For example, connected device 2130 can be a personal computer, such as a desktop computer, laptop computer, notebook computer, netbook, etc. Connected device 2140 represents a device with a mobile device screen 2145 (e.g., a small-sized screen). For example, connected device 2140 can be a mobile phone, smartphone, personal digital assistant, tablet computer, etc. Connected device 2150 represents a device with a large-sized screen 2155. For example, connected device 2150 can be a television screen (e.g., a smart TV) or another device connected to a television (e.g., a set-top box or game console), etc. One or more of connected devices 2130, 2140, and 2150 may include touchscreen capabilities. Touchscreens can accept input in various ways. For example, a capacitive touchscreen detects touch input when an object (e.g., a fingertip or pen tip) twists or interrupts the current flowing through its surface. As another example, a touchscreen can use an optical sensor to detect touch input when a beam of light from the sensor is interrupted. For some touchscreen inputs, physical contact with the screen surface is not required. Devices without screen capabilities can also be used in example environment 2100. For example, cloud 2110 can provide services to one or more computers (e.g., server computers) without a display.

[0112] The service may be provided by the cloud 2110 through service provider 2120 or through other online service providers (not shown). For example, the cloud service may be customized to the screen size, display capabilities, and / or touchscreen capabilities of specific connected devices (e.g., connected devices 2130, 2140, 2150).

[0113] In example environment 2100, cloud 2110 utilizes service provider 2120 at least in part to provide the technologies and solutions described herein to various connected devices 2130, 2140, 2150. For example, service provider 2120 can provide centralized solutions for various cloud-based services. Service provider 2120 can manage service subscriptions for users and / or devices (e.g., for connected devices 2130, 2140, 2150 and / or their respective users). Cloud 2110 can store training data 2160 used in the user intent determination described herein. Intent classifier 2162 can, for example, be similar to... Figure 1 The intent classifier 116 can also be implemented in the cloud 2110.

[0114] Example Implementation

[0115] Although some operations in the disclosed methods are described in a specific order for ease of presentation, it should be understood that this description includes rearrangement—unless the specific language described below requires a particular order. For example, operations described sequentially may be rearranged or performed concurrently in certain situations. Furthermore, for simplicity, the accompanying figures may not show the various ways in which the disclosed methods can be used in conjunction with other methods.

[0116] Any of the disclosed methods can be implemented as computer-executable instructions or a computer program product stored on one or more computer-readable storage media and executed on a computing device (e.g., any available computing device, including smartphones or other mobile devices that include computing hardware). A computer-readable storage medium is any available tangible medium that can be accessed within a computing environment (e.g., one or more optical media discs, such as DVDs or CDs, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as flash memory or hard disks)). As an example and reference. Figure 19 Computer-readable storage media include memory 1920, memory 1925, and storage 1940. As an example and reference. Figure 20 Computer-readable storage media include memory 2020, memory 2022, and memory 2024. The term "computer-readable storage medium" does not include signals and carrier waves. Furthermore, the term "computer-readable storage medium" does not include communication connections (e.g., 1970, 2060, 2062, and 2064).

[0117] Any computer-executable instructions used to implement the disclosed technology, and any data created and used during the implementation of the disclosed embodiments, may be stored on one or more computer-readable storage media. The computer-executable instructions may be, for example, part of a dedicated software application or a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software may be executed, for example, on a single local computer (e.g., any suitable commercial computer) using one or more networked computers, or in a networked environment (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such networks.

[0118] For clarity, only selected aspects of the software-based implementation are described. Other details well-known in the art are omitted. For example, it should be understood that the disclosed techniques are not limited to any particular computer language or program. For instance, the disclosed techniques can be implemented by software written in C++, Java, Perl, JavaScript, Adobe Flash, or any other suitable programming language. Similarly, the disclosed techniques are not limited to any particular computer or hardware type. Certain details of suitable computers and hardware are well-known and do not need to be elaborated in this disclosure.

[0119] Furthermore, any software-based implementation (including, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded, or remotely accessed via suitable communication devices. Such suitable communication devices include, for example, the Internet, the World Wide Web, intranets, software applications, cable (including fiber optic cables), magnetic communication, electromagnetic communication (including RF, microwave, and infrared communication), electronic communication, or other such communication methods.

[0120] The disclosed methods, apparatuses, and systems should not be construed as limiting in any way. Rather, this disclosure is directed toward all novel and non-obvious features and aspects of the various disclosed embodiments—individually and in various combinations and sub-combinations of each other. The disclosed methods, apparatuses, and systems are not limited to any particular aspect or feature or combination thereof, nor are the disclosed embodiments required to have any one or more particular advantages or problems solved.

[0121] The techniques from any example can be combined with the techniques described in any one or more other examples. Given the many possible embodiments to which the principles of the disclosed techniques can be applied, it should be recognized that the illustrated embodiments are examples of the disclosed techniques and should not be considered as limiting the scope of the disclosed techniques.

[0122] Additional examples

[0123] In some example systems (similar to) Figure 1 System 100 and / or Figure 2 In system 200), a user interface generator (e.g., user interface generator 106 or user interface generator 206) is configured to generate a virtual keyboard for display in the user interface by at least one processor. The user interface generator can also be configured to present a search icon upon receiving an instruction to interact with the IME portion of the virtual keyboard. Also regarding... Figures 14A-16 The search user interface and search icon are discussed. For example, a search icon can be displayed in the IME section of a virtual keyboard. User interaction can be pressing and holding the IME section, swiping the IME section, or other user interactions. In some examples, interaction with any part of the IME section (e.g., swiping) can cause the search icon to be displayed.

[0124] In some examples, the virtual keyboard does not include a search icon or search function until it receives a user interaction instruction. This approach avoids cluttering the virtual keyboard with a large number of icons or features that the user is not currently interested in, while still allowing the user to access the search function when they expect it.

[0125] The search icon can be one of the following: a web search icon, an emoji search icon, a contact search icon, a graphics exchange format (GIF) search icon, a general search icon, or another suitable search icon. In some examples, the search icons correspond to at least some of the same categories as the task icons. In some examples, the user interface generator is configured to present multiple search icons in the IME upon receiving a user interaction instruction. Each search icon represents a different type of search functionality accessible via the virtual keyboard. As a specific example, a web search icon (e.g., a magnifying glass) and at least one additional search icon (e.g., an emoji search icon, an image search icon, a video search icon, a GIF search icon, a contact search icon, etc.) are presented together to allow the user to select a specific category of search. In some examples, the general search icon is presented alone or in combination with other search icons.

[0126] In some examples, the search query text input area is presented in conjunction with multiple search icons. In examples where multiple search icons are presented, an indicator or emphasis of one search icon, or the de-emphasis of other search icons (e.g., underline, bold, grayed out from other search icons, etc.), indicates which search icon is active. The active search icon determines the search category of the presented search results. For example, if the image search icon is underlined and bold, image search results are returned. In some examples, the search query text input area is presented upon further interaction with the search icon (e.g., press and hold, select, swipe, etc.), rather than simultaneously with (multiple) search icons.

[0127] In some examples, the user interface generator is also configured to have at least one processor replace a portion of the virtual keyboard with a search results interface. This search results interface can be presented to replace a portion of the virtual keyboard before a search query is received. That is, upon receiving an instruction for user interaction with the IME portion of the virtual keyboard, the virtual keyboard can be modified from an initial state that does not include a search icon or search functionality to a state that includes search icons(s) and a search results interface. In examples where the search results interface is presented before a query is received, search results for previous or frequently asked queries can be presented (for a specific user or a group of users).

[0128] Search results in the search results interface can be presented in a "card" format, where each result is enclosed in a rectangular (or other shaped) area resembling a business card. You can slide a result card left or right to reveal another result in the card format. In other examples, search results may be presented as a list, thumbnails, or other formats.

[0129] The search results interface can be displayed anywhere within the virtual keyboard. In some examples, the search results interface is displayed above the IME section of the virtual keyboard, with the virtual keyboard expanding upwards to accommodate it. In another example, a portion of the virtual keyboard below the IME section can be replaced by the search results interface. The user interface generator can also be configured to animate transitions in which the letter keys of the virtual keyboard are replaced by the search results of the previously queried queries. For example, the letter keys can transition horizontally to the left or right, or vertically upwards or downwards. The search results user interface can transition along with the letter keys, and appear as if the search results user interface is pushing the letter keys out of the visible area.

[0130] Figures 22A-22J The image shows various search interfaces presented in the virtual keyboard. Figure 22A A graphical user interface 2200 is shown. The user interface 2200 includes a virtual keyboard with an IME section 2202. Figure 22AThe image shows an introductory screen that introduces the search function on the keyboard to the user. In the virtual keyboard section 2204, instructions for displaying the search function on the virtual keyboard are shown (i.e., swiping the IME section 2202). In some examples, under normal operating conditions (i.e., when not using the virtual keyboard for the first time), the virtual keyboard section 2204 will display the keys of the virtual keyboard (e.g., ...). Figure 4A (as shown in the image).

[0131] exist Figure 22B In the middle, the user swipes on the IME section 2202, and the search interface begins to appear (swipe from right to left). The search interface includes a first search icon 2206 (displayed as "HA!", which is in the GIF search category) and a second search icon 2208, which is in the emoji search category. As the introductory content 2210 is also moved into the view, the virtual keyboard section 2204 is transforming into the search results interface.

[0132] Figure 22C The diagram illustrates the user interface 2200 after the search interface has completely moved across the display from right to left. The IME section 2202 has been expanded and now includes a search query text input area 2212, as well as a first search icon 2206, a second search icon 2208, a third search icon 2214 (contact search), and a fourth search icon 2216 (web search). The first search icon 2206 is the "active" search category, as indicated by its bold / colored and underlined appearance relative to the second, third, and fourth search icons 2208, 2214, and 2216. A keyboard icon 2218 is also presented on the right side of the IME section 2202. Interaction with the keyboard icon 2218 causes the keys of a virtual keyboard to replace the search results interface. Figure 22B The virtual keyboard section 2204 has been transformed into the search results interface 2219. The search results interface 2219 may also include search expanders (e.g., search expander 2220), alternative search suggestions, related queries, etc. In some examples, the search query text input area 2212 is not presented until the user interacts with the search icon.

[0133] like Figure 22A As shown, the search results interface 2219 displays introductory content 2210 that can be shown when a user uses the virtual keyboard for the first time. For example... Figure 22D As shown in the image, during normal operation, previous or common search results can be displayed. Figure 22D In the example, after a user swipes to move the description content 2210 to the left and outside of the display, the previous search result 2221 is displayed. Figures 22A-22DAs shown, the search icon, search query text input area, or other search functions are not displayed in the virtual keyboard until user interaction with IME section 2202. Also... Figures 22A-22D As shown, after the user causes the search interface to be displayed but before the search query has actually been executed, previous or common search results 2221 can be provided in the search results interface 2219.

[0134] Figure 22E The diagram illustrates user interface 2200, where the user has selected the second search icon 2208 (emoji search) as the active search category. The search results user interface 2219 changes to display emoji search results. Similarly, in... Figure 22F In this context, the user has selected the third search icon 2214 (Contact Search) as the active search category, and the search results user interface 2219 has changed to display contact search results. Figure 22G In this example, the user has selected the fourth search icon 2216 (web search) as the active search category, and the search results user interface 2219 changes to display introductory content about web search. In some examples, after the user has performed at least one web search, the previous web search results are presented in the search results interface 2219.

[0135] exist Figures 22A-22G In the search results, the results are displayed as cards, but other formats (such as lists, thumbnail icons) are also possible. Figure 22H The figure illustrates the user interface 2200 after a user taps in the search query text input area 2212. The search results interface 2219 has been replaced by keys 2222 of a virtual keyboard. In some examples, transitions between the virtual keyboard section 2204, the search results interface 2219 (and between search results within the search results interface 2219), and / or the keys 2222 are animated in a horizontal or vertical direction (e.g., from left to right, from right to left, from top to bottom, from bottom to top).

[0136] exist Figure 22I In the middle, users via Figure 22H The key 2222 shown indicates that "Colonie" was entered. As a result, the search results interface 2219 is re-displayed instead of key 2222, and the search results for the restaurant Colonie are displayed in the search results interface 2219. Search results can include shareable content. In some examples, as indicated by option 2224 "Share All," the entire search results can be shared, and in some examples, specific information bars (e.g., only phone number, address, hours, reviews, menu, etc.) can be shared. Figure 22JIn the middle, user selection for sharing Coloney's address is shown via touch marker 2226. Then, address information 2228 for Coloney is presented in a text input box for a messaging application currently using a virtual keyboard. Figure 22J As shown, each of "Phone," "Website," and "Bing" has selectable sharing options, and addresses sent to the text input box for sharing have delete options. Using such an interface, users can select only one, two, three, etc., pieces of information to share, rather than sharing the entire search results card, which may contain information irrelevant to the intended recipient.

[0137] exist Figures 22A-22J In this version, common decorative themes are maintained for the search interface, virtual keyboard, and search results interface. In some examples, different themes are used to indicate where to enter the search interface.

[0138] Figure 23 The figure illustrates a method 2300 for reconfiguring a graphical user interface on a computing device. In processing block 2302, a virtual keyboard is presented in the graphical user interface. The virtual keyboard does not include search icons or a search text input area. In processing block 2304, a swipe instruction on the IME portion of the virtual keyboard is received. In processing block 2306, in response to the swipe instruction, a search interface is presented within the virtual keyboard. The search interface includes multiple search icons representing different search categories. In some examples, the search interface also includes a search query text input area.

[0139] The search interface also includes a search results interface. The search results interface is presented to replace a portion of the virtual keyboard below the IME section, and the search results interface may include search results for previously received or frequently used search queries. In some examples, the search results interface is presented above the IME section of the virtual keyboard, and the overall size of the virtual keyboard is expanded. Previously received or frequently used search queries included in the search results interface may correspond to search categories represented by a search icon (“active” search icon) among a plurality of currently selected search icons. The plurality of search icons may include, for example, a web search icon and at least one of the following: an image search icon, a video search icon, an emoji search icon, a contact search icon, a graphics exchange format (GIF) search icon, or a general search icon.

[0140] Figure 24The figure illustrates a method 2400 for reconfiguring a graphical user interface on a computing device. In processing block 2402, a virtual keyboard is presented in the graphical user interface. In processing block 2404, one or more text entries are received. In processing block 2406, a user intent is determined using one or more intent classifiers, at least in part based on the one or more text entries. In processing block 2408, upon determining the user intent, task icons representing functions corresponding to the user intent are presented in the virtual keyboard. Interaction with the task icons causes a portion of the graphical user interface to be replaced by a task icon user interface, which provides access to the functions corresponding to the user intent.

[0141] In processing box 2410, it is determined that there has been no interaction with the task icon. A threshold time can be used for this purpose. For example, if the task icon has not been selected, swiped, etc., within 3 seconds, 5 seconds, 10 seconds, 15 seconds, etc., it can be determined that there has been no interaction with the task icon. Other time thresholds can also be used. In some examples, when it is determined that there has been no interaction with the task icon, the task icon is removed from the virtual keyboard or moved to the edge of the graphical user interface (e.g., the edge of the IME section), so that only a portion of the task icon is visible. This is also referred to as providing a "peek" of the task icon. In some examples, if the lack of interaction with the task icon's peeking exceeds a second time threshold (e.g., 20 seconds, 30 seconds, one minute, etc.), the peeking of the task icon is removed from the virtual keyboard.

[0142] In processing box 2412, an instruction for user interaction with the IME portion of the virtual keyboard is received. This could be, for example, a swipe, selection, press and hold, or other user interaction. In processing box 2414, in response to receiving the instruction, a search interface is presented on the virtual keyboard. The search interface includes multiple search icons representing different search categories. The search interface also includes indicators presented in association with search icons corresponding to task icons. These indicators indicate that the task icon has been previously presented but not interacted with. Figure 25F and Figure 25G The image shows an example indicator. In some examples, the task icon is removed from the graphical user interface when the search interface is displayed.

[0143] Figures 25A-25G The image illustrates the features of the additional task icon. For example... Figures 4A-16 A detailed discussion of task icons. Figure 25A In the middle, the user interface 2500 includes a virtual keyboard, which includes an IME part 2502 and keyboard keys 2504. Figure 25A The diagram illustrates the generation and presentation of task icon 2506 based on the determined user intent to meet someone named Kyle. Figure 25BIn the process, after determining that the user has not interacted with task icon 2506 (e.g., after a time threshold has passed), task icon 2506 has moved to the right edge of IME section 2502. After a second time threshold has passed, this "peeping" of task icon 2506 may continue or may disappear from IME section 2502.

[0144] exist Figure 25C In the middle, the user continued typing "lunch" on the original input text "I'm meeting Kyle," and task icon 2506 was removed from the view. Figure 25D In the process, the user additionally types "in Colonee," and based on the determined intention to go to the Colonee restaurant, a second task icon 2508 is generated and displayed in IME section 2502. Figure 25E As indicated by sliding mark 2510, the user slides the IME section 2502 to reveal the search interface (e.g., Figure 25F (As shown). In Figure 25F In the IME section 2502, the number of search icons has been expanded and now includes four search icons and a search query text input area, similar to... Figure 22C As shown in the image, keyboard key 2504 has been replaced by search results interface 2512.

[0145] The user did not interact with either task icon 2506 or the second task icon 2508. Figure 25F In this example, indicators 2514 and 2516 indicate that a task icon has been presented but not interacted with; this is shown as a dot below the search icon, corresponding to the category of task icon 2506 and the second task icon 2508. That is, because a contact task icon is presented, a dot indicator for contacts is presented below the search icon, and because a restaurant task icon is presented, a dot indicator is presented below the web search icon (the restaurant task icon provides access to restaurant information accessed via the web). In some examples, a separate indicator is provided next to the search icon to indicate that the previously presented but uninterrupted task icon is available. In some examples, the task icon itself is presented next to the search icon (e.g., in...). Figure 25F The web search (magnifying glass) icon shown is accompanied by task icon 2506 and a second task icon 2508 to the right. In some examples, only the latest or most recent few (e.g., two) task icons are shown along with the search icon.

[0146] exist Figure 25G In the middle, the user selected the web search icon (as indicated by bold / colored and underlined), and information linking to the second task icon 2508 was displayed. The indicator associated with the web search icon was subsequently removed.

[0147] Figures 26A-26B The diagram illustrates alternative or additional search interface entry points. The user interface 2600 includes an IME section 2602 and keyboard keys 2604. Figure 26A In the middle, the user presses and holds the emoji icon indicated by touch mark 2606 to launch search menu 2608 (e.g., Figure 26B (As shown). Search menu 2608 includes about Figures 22A-22J The discussion includes four search icons and corresponding text descriptions for the search categories. In some examples, the user selects, swipes, or interacts with one of the search icons to initiate, for example... Figures 22A-22J The search interface is shown in the figure. In other examples, the search query text input area is initiated and the search menu is still displayed (2608).

[0148] Figures 27A-27B The diagram illustrates yet another alternative or additional search interface entry point. User interface 2700 includes an "arc-shaped" virtual keyboard. The virtual keyboard includes an IME section 2702 and keyboard keys 2704. As indicated by touch marker 2706, user interaction with the emoji icon initiates a search menu 2708, which extends in an arc shape to the right of the emoji icon. The search menu 2708 includes... Figures 26A-26B The same four search categories are illustrated, but other search categories are also possible. In some examples, curved or other shaped menus are displayed in a separate part of the virtual keyboard instead of extending from the emoji icon. The emoji icon is used as an example starting point because the emoji search category is one of the four search categories shown. Other categories (e.g., image search, video search, etc.) are possible, and other starting points are possible (e.g., Figure 27A and Figure 27B (HA! in the bottom right corner). In some examples, when the search menu 2708 is presented, a search query text input box is displayed.

[0149] In some examples, the search icon is included within the virtual keyboard (e.g., in the IME section of the virtual keyboard), and after a task icon is generated based on the determined user intent, the task icon is presented to replace the search icon. For example, if a user is typing the message "We should book our flight to New York" in a messaging application, it can be determined that the user's intent is to buy a ticket to New York, and a flight or travel task icon can be presented to replace the search icon in the virtual keyboard (e.g., in the IME section).

Claims

1. A system comprising: at least one processor; and a user interface generator configured to be executed by the at least one processor to: generate a virtual keyboard for display in a user interface; receive one or more text entries; determine a user intent based at least in part on the one or more text entries; upon determining the user intent, present within the virtual keyboard a task icon representing a function corresponding to the user intent, wherein an interaction with the task icon causes a portion of the graphical user interface to be replaced with a task icon user interface that provides access to the function corresponding to the user intent; upon receiving an indication of an interaction with an input method editor (IME) portion of the virtual keyboard, present a plurality of search icons in the IME portion of the virtual keyboard, respective ones of the search icons representing different types of search functions accessible via the virtual keyboard; and for at least one search icon of the plurality of search icons that is presented: determine that the at least one search icon corresponds to a task icon, determine that the corresponding task icon was previously presented but not yet interacted with, and present an indicator associated with the at least one search icon, the indicator indicating that the at least one search icon corresponds to a task icon that was previously presented but not yet interacted with.

2. The system of claim 1, wherein the interaction is a swipe of the IME portion of the virtual keyboard.

3. The system of claim 2, wherein the swipe of the IME portion of the virtual keyboard is a swipe of any portion of the IME portion of the virtual keyboard.

4. The system of claim 1, wherein the plurality of search icons includes at least one of a web search icon, an emoji search icon, a contact search icon, a graphic interchange format (GIF) search icon, or a general search icon.

5. The system of claim 1, wherein the user interface generator is further configured to be executed by the at least one processor to present a search query text entry area in conjunction with the plurality of search icons.

6. The system of claim 1, wherein the user interface generator is further configured to be executed by the at least one processor to replace a portion of the virtual keyboard with a search results interface.

7. The system of claim 6, wherein the search results interface replaces a portion of the virtual keyboard below the IME portion.

8. The system of claim 6, wherein the search results interface replaces the portion of the virtual keyboard prior to receiving a search query.

9. The system of claim 6, wherein the search results interface includes search results of a previous query.

10. The system of claim 9, wherein the user interface generator is further configured to animate a transition in which one or more letter keys of the virtual keyboard are replaced with the search results of the previous query. ​ ​ ​ 11. The system of claim 1, wherein the virtual keyboard does not include a search icon prior to receiving the indication of the interaction.

12. A method for reconfiguring a graphical user interface on a computing device, the method comprising: presenting a virtual keyboard in the graphical user interface, wherein the virtual keyboard does not include a search icon or a search text entry area; receiving one or more text entries; determining a user intent based at least in part on the one or more text entries; upon determining the user intent, presenting a task icon within the virtual keyboard representing a function corresponding to the user intent, wherein an interaction with the task icon causes a portion of the graphical user interface to be replaced with a task icon user interface that provides access to the function corresponding to the user intent; receiving an indication of a swipe of an input method editor (IME) portion of the virtual keyboard; in response to the indication of the swipe, presenting a search interface within the virtual keyboard, the search interface comprising: a plurality of search icons representing different search categories; and a search results interface, wherein the search results interface is presented to replace a portion of the virtual keyboard below the IME portion, and wherein the search results interface includes search results for previously received or common search queries; and for at least one search icon of the plurality of search icons that is presented: determining that the at least one search icon corresponds to a task icon that was previously presented but has not yet been interacted with, and presenting an indicator associated with the at least one search icon, the indicator indicating that the at least one search icon corresponds to a task icon that was previously presented but has not yet been interacted with.

13. The method of claim 12, wherein the search interface further includes a search query text entry area.

14. The method of claim 12, wherein the previously received or common search queries included in the search results interface correspond to the search category represented by a search icon of the plurality of search icons that is currently selected.

15. The method of claim 12, wherein the plurality of search icons includes a web search icon and at least one of: an image search icon, a video search icon, an emoji search icon, a contacts search icon, a graphic interchange format (GIF) search icon, or a general search icon.

16. One or more computer-readable storage media storing computer-executable instructions for reconfiguring a graphical user interface, the reconfiguring comprising: presenting a virtual keyboard in the graphical user interface; receiving one or more text entries; determining a user intent based at least in part on the one or more text entries using one or more intent classifiers; while determining the user intent, presenting, within the virtual keyboard, a task icon representing a function corresponding to the user intent, wherein interaction with the task icon causes a portion of the graphical user interface to be replaced with a task icon user interface that provides access to the function corresponding to the user intent; determining that the task icon has not been interacted with; receiving an indication of user interaction with an input method editor (IME) portion of the virtual keyboard; and in response to receiving the indication, presenting, in the virtual keyboard, a search interface, wherein the search interface includes a plurality of search icons representing different search categories, and wherein the search interface includes a presented indicator associated with a search icon corresponding to the task icon, the indicator representing that the task icon was previously presented but not interacted with.

17. The one or more computer-readable storage media of claim 16, wherein the reconfiguring further comprises: while determining that the task icon has not been interacted with, moving the task icon to an edge of the graphical user interface such that only a portion of the task icon is visible.

18. The one or more computer-readable storage media of claim 17, wherein the reconfiguring further comprises: while the search interface is presented, removing the task icon from the graphical user interface.

Images