Real-time morphing interface for computer screen display

Abstract

The morphing interface system updates, or morphs, the display of the client device as the user provides a portion of the input. The system receives a first portion of the user input and selects an intent related to the user's predicted intent that can match or be associated with the portion of the user input. The system displays an interface related to the intent on the client device. The system additionally extracts entity values ​​related to the execution of the intent from the first portion of the user input. When the user provides an additional portion of the input, the morphing interface system re-evaluates the user input to determine whether the user input still indicates the same user intent, i.e., the same execution intent. If it determines that the additional input indicates a different user intent, the system updates the display to display an interface related to the new intent.

Description

Technical Field

[0001] The present invention relates to the field of interface technologies for computer screen display, and more particularly to real-time morphing interfaces.

[0002] (Cross-reference to related applications) This application claims the benefit of U.S. Provisional Application No. 62 / 895,944, filed Sep. 4, 2019, and U.S. Provisional Application No. 63 / 038,064, filed Jun. 12, 2020, which are hereby incorporated by reference in their entirety.

Background Art

[0003] Computer assistants such as smart speakers and artificial intelligence programs are becoming increasingly popular and are used in a variety of user-oriented systems. Computerized systems can often be implemented such that the entire process is automated without the human system user having insight into the process. For example, a computer can complete a series of tasks without displaying content on the user's screen. However, many users prefer to receive feedback regarding computerized processes, and it can be useful or necessary for the user to understand the status of a series of tasks when the user needs to provide feedback at a particular step.

[0004] Furthermore, users desire that assistant systems respond as quickly as possible. However, if the system responds to the user before receiving all of a series of instructions from the user, the system may perform inappropriate tasks for the user or may not receive sufficient information to display on the screen the content that the user should view. Thus, for example, a system that includes a function of displaying in real time the progress of an automated task and adjusting the display in response to additional input would be beneficial.

Brief Description of the Drawings

[0005] [Figure 1] Figure 1 is a high-level block diagram of the system architecture of a morphing interface system according to an exemplary embodiment. [Figure 2] Figure 2 is a high-level diagram of the interactions between the components of the morphing interface system 130 in an exemplary embodiment. [Figure 3] Figure 3 is a block diagram showing an exemplary NLP signal in an exemplary embodiment. [Figure 4A] Figure 4A is a flowchart showing the process of generating an interface from user input in an exemplary embodiment. [Figure 4B] Figure 4B is a flowchart illustrating the process of morphing the interface when additional user input is received in an exemplary embodiment. [Figure 5A] Figure 5A shows a first layout displayed for an interface related to a flight reservation intent in an exemplary embodiment. [Figure 5B] Figure 5B shows a second layout displayed for the interface related to the flight reservation intent in an exemplary embodiment. [Figure 5C] Figure 5C shows a third layout displayed for the interface related to the flight reservation intent in an exemplary embodiment. [Figure 5D] Figure 5D shows a fourth layout displayed for the interface related to the flight reservation intent in an exemplary embodiment. [Figure 5E] Figure 5E shows a fifth layout displayed for the interface related to the flight reservation intent in an exemplary embodiment. [Figure 5F] Figure 5F shows a sixth layout displayed for the interface related to the flight reservation intent in an exemplary embodiment. [Figure 5G] Figure 5G shows an email confirming the execution of a flight reservation intent in an exemplary embodiment. [Figure 6A] Figure 6A shows the receipt of a first portion of user input related to a pizza order intent, according to an exemplary embodiment. [Figure 6B] Figure 6B shows a layout displayed for an interface related to pizza ordering, according to an exemplary embodiment. [Figure 6C] Figure 6C shows a layout displayed for an interface related to purchasing a pizza T-shirt, according to an exemplary embodiment. [Figure 6D] Figure 6D shows an exemplary embodiment of an additional exemplary interface that may be related to a T-shirt purchase intent. [Figure 7] Figure 7 is a block diagram showing components of an exemplary machine, according to an exemplary embodiment, that can read instructions from a machine-readable medium and execute them on one or more processors.

[0006] The letters following a reference number, such as "105A," indicate that the text specifically refers to the component with that particular reference number. A reference number in text without any following letters, such as "105," indicates that it refers to any or all components in the figure to which that reference number belongs.

[0007] The drawings illustrate various embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein can be employed without departing from the principles described herein. [Modes for carrying out the invention]

[0008] The figures and the following description relate to preferred embodiments for illustrative purposes only. It should be noted from the following description that alternative embodiments of the structures and methods disclosed herein will be readily recognizable as viable alternatives that may be adopted without departing from the principles of the claims.

[0009] Herein, several embodiments are referenced in detail, and examples thereof are shown in the accompanying drawings. Note that, wherever possible, similar or identical reference numerals are used in the drawings to indicate similar or identical functionality. The drawings illustrate embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

[0010] (Overview of the structure) Disclosed is a system (and method, and computer program code stored in a non-temporary computer-readable medium) configured to generate and enable displays on a user interface of a computer screen. In one embodiment, the system may include one or more computer processors for executing computer program instructions and a non-temporary computer-readable storage medium containing stored instructions executable by at least one processor. In an exemplary embodiment, the instructions may, when executed, cause a processor to receive a first input from a first user device, including an input string, and to generate a set of natural language processing signals based on the first user input. The instructions may further include instructions for selecting a user intent that matches the first user input, the selection being based on natural language processing signals, instructions for identifying an interface associated with the intent, and instructions for extracting a set of values ​​from the user input that are associated with entities of the interface. Entities may be variables of the interface to which values ​​can be assigned. In one embodiment, the instructions may further include instructions for making a display interface containing values ​​from a set of values ​​available as an interface on a user device, and the display may occur almost simultaneously with the reception of a first user input. In one embodiment, instructions executable by at least one processor of the system may further include instructions that, when executed by the processor, cause the processor to receive a second user input containing a text string from the user device, and generate an updated set of natural language processing signals based on the combination of the first and second user inputs.The instructions may further include instructions for selecting an objective that matches a combination of a first user input and a second user input based on an updated set of natural language processing signals; instructions for identifying a second interface associated with the newly selected intent; instructions for extracting a second set of values ​​associated with entities of the second interface from the combination of the first user input and the second user input; and instructions for enabling the second interface for display on a user device, including values ​​from the second set of values.

[0011] In various exemplary embodiments, the first user input and / or the second user input may be voice input. Furthermore, the first interface and the second interface may include a set of related entities and may be the same or different interfaces in various exemplary embodiments. The input string may be a text string, voice input, and / or another form of user input.

[0012] Example of system configuration Figure 1 is a high-level block diagram of the system architecture of a morphing interface system according to an exemplary embodiment. Figure 1 includes a morphing interface system 130, a network 120, and a client device 110. For clarity, only one client device and one morphing interface system 130 are shown in Figure 1. Alternative embodiments of the system environment may have multiple morphing interface systems 130 in addition to any number of client devices 110. The functions performed by the various entities in Figure 1 may vary in different embodiments. The client device 110 and the morphing interface system 130 may include some or all of the components of a computing device, such as those described in Figure 7, and an appropriate operating system.

[0013] In exemplary embodiments, the morphing interface system 130 generates (or renders, or enables rendering) a user interface for display to the user in response to user input (e.g., typed or spoken text strings). In some embodiments, the system may also receive visual input, for example, from the camera or camera roll of a client device 110, and perform a search process on an online marketplace. The morphing interface system 130 determines a user intent. A user intent corresponds to what the machine (e.g., a computer or computing system) predicts based on the received user input. Thus, a user intent can be a computer-executable function or request that corresponds to and / or is described by the received user input. An executable function is something that can be performed and can be instantiated by generating and / or adding (e.g., in rendering) one or more user interfaces for the function that corresponds to what could be the predicted intent.

[0014] When the morphing interface system 130 receives additional user input (for example, words added to a typed or spoken text string), it re-evaluates whether the determined user intent is still the most relevant match to the user input. If another user intent better fits the updated user input, the morphing interface system 130 generates and adds a new user interface that applies to the new intent. In other words, as the morphing interface system 130 receives more input information about which user intent best applies to the input, i.e., which function or request best addresses the user input, the user interface "morphs" from one interface to another. If the morphing interface system 130 determines that there are several equivalent intents, the morphing interface system 130 may prompt the user with an interface preview (for example, by providing information for rendering the interface preview on the client device 110) so that the user can select from the equivalent intents, or the morphing interface system 130 may automatically select an intent based on stored user settings (for example, learned based on past user interactions with the system).

[0015] The user can input user input such as typed text or spoken voice input via the client device 110. The client device 110 can be any personal or mobile computing device such as a smartphone, tablet, notebook computer, laptop, desktop computer, and smartwatch, as well as any home entertainment device such as a television, video game console, TV box, and receiver. The client device 110 can present the information received from the morphing interface system 130 to the user, for example, in the form of a user interface. In some embodiments, the morphing interface system 130 can be stored and executed from the same machine as the client device 110.

[0016] The client device 110 can communicate with the morphing interface system 130 via the network 120. The network 120 can include any combination of local area networks and wide area networks using wired or wireless communication links. In some embodiments, all or part of the communication of the network 120 can be encrypted.

[0017] The morphing interface system 130 includes various modules and data stores for determining an intent and / or generating an interface. The morphing interface system 130 includes a natural language processing (NLP) pipeline 135, an intent recognition module 140, an entity recognition module 145, an intent model store 150, an interface store 155, and an entity recognition model store 160. To avoid obscuring the details of the system architecture, computer components such as web servers, network interfaces, security functions, load balancers, failover servers, management and network operation consoles are not shown. Further, the morphing interface system 130 may include more, fewer, or different components than those shown in FIG. 1, and the functions of the components described herein may be distributed differently than described in this specification. Note that the pipeline and modules may be embodied as program code (e.g., software or firmware), hardware (e.g., application specific integrated circuit (ASIC), field programmable gate array (FPGA), controller, processor), or combinations thereof.

[0018] The NLP pipeline 135 receives user input, for example, in the form of text or speech, and generates NLP signals that the morphing interface system 130 can use for intent recognition and entity extraction. In some embodiments, the NLP pipeline 135 performs tokenization, part-of-speech tagging, stemming, lemmatization, stop word identification, dependency parsing, entity extraction, chunking, semantic role assignment, and coreference analysis. In one embodiment, the input to the NLP pipeline is, for example, a set of one or more words in the form of a complete or partially complete sentence or phrase. In one embodiment, the NLP pipeline 135 generates an annotated version of the input set of words. In another embodiment, the NLP pipeline 135 constructs or retrieves numerical representations or feature embeddings for immediate consumption by downstream modules that may use neural networks, such as the intent recognition module 140 or the entity recognition module 145. For example, the input to the NLP pipeline 135 may be a partial sentence, and the output may be a partial sentence accompanied by metadata about the partial sentence.

[0019] The intent recognition module 140 identifies what the user's intention is based on the input received from the user (via the client device 110). In particular, the intent recognition module 140 predicts the available intents (i.e., functions) that the morphing interface system 130 can perform. The available intents correspond to a set of words that make up the user input. The user input may match one or more predefined intents. For simplicity of explanation, the system is described in terms of words. However, it should be noted that the principles described in this specification may also apply to any set of signals that may include information inputs including sound actions (such as voice commands or audio tones), video streams (such as ambient computing scenarios), and other potential forms. In different embodiments, the intent recognition module 140 may use various machine learning models to determine intents that can be associated with the user input. For simplicity of explanation, the system will be described in terms of supervised machine learning. However, it should be noted that the principles described in this specification may also apply to semi-supervised and unsupervised systems.

[0020] In one exemplary embodiment, the intent recognition module 140 may use text classification to predict the intent most likely to correspond to user input. In this exemplary embodiment, the text classification model may be trained using labeled examples of input strings. For example, the morphing interface system 130 may store labeled examples of input strings. The labels associate each example of an input string with one of the available intents. The training data may include examples of input strings in the form of words, partial sentences, partial phrases, complete sentences, and complete phrases. The classification model may also be trained to use various natural language processing signals generated by the NLP pipeline 135, and the training data may further include natural language processing signals. The classification model may also utilize signals from the entity recognition module 145, for example, using the “airline” entity recognized by the entity recognition module 145 to confirm that an intent, or function, is “book a flight”. Therefore, classification models are trained, for example, using semantic similarity, to predict which of the available intents is most likely to correspond to a particular user input string (i.e., to determine the closest query from a set of example queries for each action).

[0021] In another exemplary embodiment, the intent recognition module 140 may use a model that calculates semantic similarity scores between user inputs and sets of input examples across the entire set of available intents. That is, rather than training the model to directly predict applicable intents based solely on labeled training data, the intent recognition module 140 may also compare a given user input with some or all of previously received user input strings when determining the intent that best matches the user input. For example, the morphing interface system 130 may store a record of previously matched intents and input strings, and the intent recognition module 140 may predict the same intent for the current user input string if it determines that the current user input string has the same or the same sentence structure or associated NLP signals as a previously stored input string. In addition to comparison with correctly matched past user input strings, the user input may also be compared with computer-generated strings created by both semantic rule-based approaches and generative deep learning algorithms.

[0022] In another exemplary embodiment, the intent recognition module 140 may use a simpler rule-based method to infer the most likely intent for a user's input string. This may include regular expression matching, i.e., identifying specific predefined syntactic and grammatical patterns in the input string to determine the user's intent. This may also include using signals from the NLP pipeline 130, such as dependency parsing, component parsing, chunking, and / or semantic role assignment, to retrieve verbs, subjects, predicates, etc., of the query and match them against data from a stored knowledge base. For example, if the user's input is "Buy me bananas," the intent recognition module 140 may determine that the word "bananas" is the direct object of the query and retrieve a match for its lemma "banana" from the knowledge base to learn that "banana" is a type of food or ingredient—which may indicate a match with the intent to buy groceries, for example.

[0023] In some exemplary embodiments, the morphing interface system 130 includes an intent model store 150. The intent model store 150 may store program code for a computer model trained and applied by an intent recognition module 140 to predict the intent most likely to be associated with a given user input string. In some embodiments, labeled training data and records of previously matched intents and user inputs may be stored in the intent model store 150. The intent model store 150 may also store a list of available intents, i.e., a list of tasks that the morphing interface system 130 can perform for the user in response to user input. The intent model store 150 may also store a list of unavailable intents, i.e., a list of tasks that the morphing interface system 130 recognizes as tasks that it cannot currently perform and are independent of the available intents. Furthermore, the intent model store 150 may store user-created custom intents that are only available to the user. For example, the user input string "Turn on device" ("Device power on") may not be included in the list of globally available intents, but if the user generates this intent for their own use, the intent logic will be stored in the intent model store 150.

[0024] In one embodiment, the interface store 155 stores program code for a user interface for each of the available intents that can be executed by the morphing interface system 130. Interfaces stored by the interface store 155 may include a layout for displaying the interface on the client device 110, instructions for executing intents, and a list of entities related to adding layouts and executing intents. In various embodiments, the user interface may be a custom-made interface for each potential intent. In other embodiments, the interface store 155 may include custom interfaces for custom intents designed by users and used only by those users.

[0025] The entity recognition module 145 predicts a set of entity values ​​associated with a given user input. These entity values ​​can be used to execute intents that match the user input. In various embodiments, the entity recognition module 145 accepts a user input string, relevant NLP signals from the NLP pipeline 135, and matched intents from the intent matching module 140 as input. The entity recognition module 145 may also access the interface store 155 to use interfaces associated with the matched intent as input, thereby obtaining a list of entity values ​​required for the morphing interface system 130 to execute the intent. The entity recognition module 145 may apply a trained computer model to extract a set of values ​​from the user input string and associate the extracted values ​​with entities in the matched intent. In one embodiment, the entity recognition module 145 first extracts high-level entity values ​​from the input string, and then extracts detailed entity values. For example, the entity recognition module 145 may apply a model that determines that the user input string contains a title, and then apply different models to predict whether the title is a movie title, a book title, etc. In one embodiment, one or more computer models applied by the entity recognition module 145 are classifiers or sequence taggers trained on examples of training data, including examples of user input strings having labeled entity values. These classifiers or sequence taggers may be further trained using multiple subjects (such as language modeling and autoencoding) on ​​large amounts of unstructured, unlabeled text information from the internet to incorporate real-world knowledge and an understanding of syntax, grammar, and semantics.

[0026] In other exemplary embodiments, the entity recognition module 145 may extract a set of values ​​from a user input string by applying rule-based techniques, such as matching against a regular expression pattern. This can then help in the rapid customization of the entity recognition module 145 to extract values ​​for new custom intents designed by the user.

[0027] The model and training data applied by the entity recognition module 145 may be stored in the entity recognition model store 160. The entity recognition model store 160 may also contain labeled training data for training the computer model used by the entity recognition module 145.

[0028] In some embodiments, the entity recognition module 145 and the intent recognition module 140 may be the same system. That is, the entity recognition module 145 and the intent recognition module 140 may be configured as a joint intent and entity recognition system so that both systems can work together to make more accurate decisions.

[0029] (Example of a morphing interface system) Figure 2 is a high-level diagram of the interactions between the components of the morphing interface system 130 in an exemplary embodiment. The morphing interface system 130 receives a user query 210. The user query may be a complete sentence or concept, or a partial sentence or phrase, expressed by the user, for example, in the form of typed text or spoken audio. In embodiments where the input is received as an audio file or audio stream, automatic speech recognition or other types of speech models may be used to generate an input string representing the input, for example, as text. The morphing interface system 130 may begin to respond to the user by displaying the interface while the user is providing input. Thus, in some cases, the user query 210 received by the morphing interface system 130 may be only the first part of the user input, for example, a single word or a set of words.

[0030] The user query 210 is provided as input to an NLP pipeline 135 that analyzes the user query 210 and outputs a corresponding NLP signal 220. The NLP signal 220 and the user query 210 are provided to the intent recognition module 140. The intent recognition module 140 outputs a predicted user intent 230. That is, the intent recognition module 140 predicts what intent the user is requesting or intending to perform. The predicted intent 230 or function, the NLP signal 220, and the user query 210 are provided to the entity recognition module 145, which generates extracted entity values ​​240 related to the predicted intent. The predicted intent information 250 regarding the user's predicted intent, the extracted entities 240, and additional generated metadata can be used by the morphing interface system 130 to enable the display of a user interface related to the system prediction corresponding to the user intent (on a computing device, e.g., on the screen of a client device), and to add the extracted entity values ​​to the user interface area. Therefore, the interface generated and enabled (or provided) for display on the client device can, advantageously, begin to change in virtually real time.

[0031] In some embodiments, the components of the morphing interface system 130 may be configured to interact in ways other than those shown in the example in Figure 2. In one embodiment, the morphing interface system may be configured to include a feedback loop between the intent recognition module 140 and the entity recognition module 145. For example, the intent recognition module 140 may provide the entity recognition module 145 with information about predicted intents, and the entity recognition module 145 may use the information about predicted intents as input to identify entities and potential entity types in the user query 210. The information about the identified entities may then be returned to the intent recognition module 140 for use in regenerating predictions of intents to be associated with the user query 210. In some embodiments, the entity recognition module 145 may analyze an NLP signal or user input to predict entities related to the user input, and then provide the predicted entities and entity types to the intent recognition module 140, in addition to the input and NLP signal. In such a case, the intent recognition module 140 may then use the predicted entity information to predict intent types related to the user input. A feedback loop between the intent recognition module 140 and the entity recognition module 145 may also exist in this embodiment (i.e., the intent recognition module 140 may send predicted intent information back to the entity recognition module 145 to improve or add to predictions about existing entities). In some embodiments where the entity recognition module 145 receives input data before the intent recognition module 140, the intent recognition module 140 may filter the extracted entities provided by the entity recognition module 145 to entities corresponding to predicted intents.

[0032] In other exemplary embodiments, a single module may be configured to perform the functions of both the intent recognition module 140 and the entity recognition module 145. For example, a model may be trained to perform both intent recognition and entity recognition. In another exemplary embodiment, the morphing interface system 130 may include submodels related to entity recognition for each intent type (i.e., each domain). That is, the morphing interface system 130 may store different entity recognition models for determining the values ​​of entities associated with each potential intent type, and transfer learning may be used to automate the creation of models for new potential intent types based on a past set of entity recognition models. For example, if the intent recognition module 140 predicts an intent to order a pizza, the entity recognition module 145 may access and use an entity recognition model trained to identify entities associated with an intent for ordering food. In another exemplary embodiment, the intent recognition module 140 may consist of a hierarchical model in which a first model infers a higher-level domain (e.g., "food"), and submodels of that domain then predict specific intents of the user within the predicted domain (e.g., whether the user wants to reserve a table, order takeout, search for a recipe, etc.).

[0033] In another exemplary embodiment, the morphing interface system may not include the NLP pipeline 135. In such an embodiment, the intent recognition module 140 and the entity recognition module 145 are trained to predict intents and determine entities directly based on user queries 210.

[0034] Figure 3 is a detailed block diagram showing exemplary NLP signals in an exemplary embodiment. The NLP pipeline 135 may include tokenization, part-of-speech (POS) tagging, text chunking, semantic role assignment (SRL), and coreference analysis functions for generating various NLP signals from a user input string. Some other example NLP signals include lemmatization, stemming, dependency parsing, entity extraction, and stop word identification. In various embodiments, various combinations of NLP signals may be used in the NLP pipeline 135, and the signals may be determined in various orders. For example, in another exemplary embodiment, the NLP pipeline 135 may determine the NLP signals in the order of tokenization, stemming, lemmatization, stop word identification, dependency parsing, entity extraction, chunking, SRL, and coreference analysis.

[0035] Figure 4A is a flowchart illustrating the process of generating an interface from user input in an exemplary embodiment. The morphing interface system 130 receives a first portion of user input (405). The first portion of user input may be, for example, the first word or multiple words of a document and may be received by the morphing interface system 130 in various input forms, including text or voice input. The morphing interface system 130 generates a natural language processing signal based on the first portion of user input received (410). The NLP pipeline 135 may generate the natural language processing signal using various parsing techniques, including tokenization, part-of-speech (POS) tagging, dependency parsing, entity extraction, stemming, lemmatization, stop word identification, text chunking, semantic role assignment, and coreference analysis.

[0036] The morphing interface system 130 selects an intent that matches a first portion of the user input (420). In some embodiments, the intent recognition module 140 applies a trained computer model to predict which intent is most applicable to responding to the first portion of the user input. That is, the intent recognition module 140 selects an intent implied by the received user input.

[0037] The morphing interface system 130 extracts entity values ​​associated with a predicted intent from a first portion of user input (425). In one embodiment, an entity recognition module 145 applies a trained computer model to extract relevant values ​​from the received user input. In some embodiments, the morphing interface system 130 is configured to include a feedback loop such that information about the extracted entities is sent to the intent recognition module 140 as additional input. In some embodiments, an automatic retraining and self-improvement cycle may also be included. The morphing interface system 130 obtains an interface associated with a selected intent (430). In some embodiments, the selected interface is used in the process of extracting entities associated with the intent. For example, the intent-associated interface may include an input area for values ​​associated with that particular intent, and the entity recognition module 145 may use information about the input area to identify values ​​from user input. The extracted entity values ​​associated with the selected intent are used to add to the interface (435). In some embodiments, the interface may be partially populated, for example, if some of the user input received so far contains only some of the information needed to complete the input areas included in the interface layout. For example, if the user pauses after providing input including "book me flight," the morphing interface system 130 may highlight the calendar and prompt the user to provide more user input to further add to the user interface using a query such as "when do you want to fly?". The morphing interface system 130 then displays the added interface to the user, for example, via the client device 110 (440).

[0038] In some embodiments, the morphing interface system 130 may select one or more applicable intents if several intents have similar or equivalent predictability to be applicable to a first part of user input (420). For example, if the first part of user input is "Order me coffee," the morphing interface system 130 may determine that intents for purchasing a bag of coffee beans and ordering a cup of coffee from a cafe are equally applicable. That is, the morphing interface system 130 may determine that the user's intent is equally likely to be a drink delivery and an order for groceries. In such a case, the morphing interface system 130 may extract entity values ​​related to two or more intents that have equal probability (425), obtain preview interfaces related to the multiple possible intents (430), add the extracted entity values ​​related to the intents to the preview interfaces (435), and provide the added preview interfaces for display to the user on the client device 110 (440). In such embodiments, the morphing interface system may await input from the user to select one of several possible intents before proceeding to analyze further user input. In some embodiments, the morphing interface system 130 may store information about the user selection of the preview interface with respect to the provided user input so that in the future, the system can increase the likelihood that the user will select a particular intent based on the pattern of the user input. Thus, after a history of several interactions with the morphing interface system 130, as the user's preferences and input patterns are stored, the user may be presented with previews of fewer intents.

[0039] Figure 4B is a flowchart illustrating the process of morphing an interface when additional user input is received in an exemplary embodiment. As soon as the morphing interface system 130 receives a portion of the user input for parsing, it may enable (or provide) a display interface on the client device 110. Enabling (or providing) a display interface on the client device 110 may include providing code, commands, and / or data for the operating system of the client device 110 to display a user interface corresponding to intents and / or morphing structures defined by the morphing interface system 130. Upon receiving additional user input, the morphing interface system 130 repeatedly re-evaluates whether the selected intent still best matches the user input string being expanded. If the morphing interface system 130 determines that the same intent still applies, additional entity values ​​may be extracted from the expanded user input. However, if the morphing interface system 130 is given additional user input and determines that a different intent is more applicable, the system may enable (or provide) a change in the user interface to a different, more applicable intent (e.g., "morphing") for display. The visual change on the screen (e.g., on the client device 110) could be, for example, a gradual visual transition to a complete update of the user interface. Furthermore, the morphing interface system may allow the user interface to change any number of times as user input is added over time.

[0040] In particular, as shown in Figure 4A, after the first part of the user input is received and analyzed, the morphing interface system 130 continues to parse the expanding string of user input as it receives additional parts of the user input, as shown in Figure 4B. The morphing interface system 130 receives additional parts of the user input (445). For example, the first part of the user input may be the first word of a sentence, and the process in Figure 4B may be initiated in response to receiving the second word of the user input sentence.

[0041] The morphing interface system 130 generates a natural language processing signal based on a combination of previously received portions of the user input and additional portions of the user input (450). An intent that matches the combination of user inputs is selected, for example, by the intent matching module 140 (455).

[0042] The morphing interface system 130 determines whether the matched intent is the same as the most recently matched intent, or whether the matched intent is a new intent (460). The new intent may be a different intent from the intent associated with the interface currently displayed on the client device 110.

[0043] If the intents are the same intent, the morphing interface system 130 then extracts additional entity values ​​associated with the intent from the combination of the previous user input and the current (i.e., most recent) user input (465). The interface then adds the additional entity values ​​associated with the intent (470).

[0044] If the intent is not the same as a previous intent that matched the previously collected user input, the morphing interface system 130 extracts entity values ​​associated with the new intent from the combination of user input (475). The interface associated with the new intent is obtained, for example, from the interface store 155 (480). The obtained interface is then supplemented with the extracted entity values ​​associated with the new intent (485).

[0045] The morphing interface system 130 enables (or provides) the display of the added interface to the user, for example via the client device 110, regardless of whether it is the same interface as before or a newly acquired interface (490). The process shown in Figure 4B may be repeated each time the morphing interface system 130 receives an additional portion of user input (445).

[0046] Figures 5A to 5G below illustrate an example of one embodiment of an interface that executes an intent when additional user input is received. In one embodiment, the interface is a user interface presented for display on the screen of a client device 110, such as a computing device, e.g., a smartphone, tablet, laptop, or desktop computer. Figures 5A to 5G show an example where user input received via the client device 110 matches a general intent (i.e., a feature or user request) for booking a flight. As additional entity values ​​associated with the selected interface are determined in response to the receipt of additional user input, the layout of the displayed interface morphs and changes. However, in the example of Figures 5A to 5G, the morphing interface system 130 has not determined that the new intent better matches the user input, and therefore the displayed layout remains associated with the flight booking interface.

[0047] Figure 5A shows a first layout displayed for an interface associated with a flight booking intent in an exemplary embodiment. The interface 500 associated with flight booking is selected by the morphing interface system 130 based on the received user input 530A. In the drawings of Figures 5A to 5E, the selected interface 500 and associated display layouts are shown on the left side of the drawing, and two possible types of displays for user input 530 are shown on the right side of the drawing. In various embodiments, the actual display on the user device may appear as a layout of interface 500, an alternation between interface 500 and user input layouts, a split screen showing part of interface 500 and part of the input display layout for user input 530, or a different display scheme. In one exemplary embodiment, when interface 500 is updated, for example, in response to a new user input 530, the display, as shown on the left side of the drawing, begins to morph to show the next layout or interface. For example, the user interface 500 on the left in Figure 5A morphs to the layout of the user interface 500 on the left in Figure 5B. This interface morphing can occur between consecutive user interfaces, for example, as a transition between interfaces 500 shown in consecutive Figures 5A-5G and 6A-6D. Exemplary types of user input 530 shown in Figure 5A include voice input 510 and text input 520.

[0048] In the example in Figure 5A, the morphing interface system 130 receives initial user input 530A containing the text string "Book me a flight". The morphing interface system 130 determines that user input 530A is most likely associated with a flight booking intent, and accordingly, the flight booking interface 500 is displayed almost instantly. Interface 500 may include widgets 540. Widgets may be part of an interface layout that displays or collects information related to entities associated with the interface. In some cases, widgets 540 may be input fields (e.g., text fields, checkboxes, or other data input formats). Interface 500 may display widgets 540 that can populate entity values ​​determined by the entity recognition module 145. In various embodiments, interface 500 may display some or all of the widgets 540 for entities to be included in interface 500 even before the entity recognition module 145 determines the values ​​to populate in the input fields. For example, Figure 5A includes a widget 540A with a space for entering the destination value of a flight reservation. Similarly, Figure 540B includes a departure / arrival date widget 540B with a space for entering the date value of a flight reservation, and a widget 540C showing the number of passengers, with the entity recognition module 145 predicting that the value "1 passenger Economy" is the most likely input value, and the value added to widget 540C accordingly is displayed.

[0049] Figure 5B shows a second layout displayed for an interface related to a flight booking intent in an exemplary embodiment. In Figure 5B, user input 530B includes additional information. In particular, because the user has added further input, user input 530B now displays "Book me a flight from SFO". The morphing interface system 130 determines that the selected intent should still remain a flight booking and identifies additional entity value information for adding a widget 540 within interface 500. Thus, user interface 500 morphs from the layout in Figure 5A to the layout shown in Figure 5B. For example, interface 500 shown in Figure 5B includes the origin "SFO" ("San Francisco") entered for the flight origin information in widget 540A. In some embodiments, morphing an interface from one layout to another may involve displaying animations such as a moving expansion of sections of the layout that can be further added using newly received information from user input, for example, so that new entity information is extracted from user input.

[0050] Figure 5C shows a third layout displayed for an interface related to a flight booking intent in an exemplary embodiment. In Figure 5C, user input 530C includes additional information. In particular, the user has added input such that user input 530C includes "Book me a flight from SFO to LAX". The morphing interface system 130 determines that the selected intent should still be a flight booking. The morphing interface system 130 identifies additional entity value information to further add widget 540 into interface 500, for example, including the destination "LAX" ("Los Angeles") in the destination field of widget 540A. Thus, user interface 500 morphs from the layout shown in Figure 5B to the layout shown in Figure 5C.

[0051] Figure 5D shows a fourth layout displayed for an interface related to a flight booking intent in an exemplary embodiment. In Figure 5D, user input 530D includes additional information. In particular, because the user has added further input, user input 530B now includes "Book me a flight from SFO to LAX on April 22". The morphing interface system 130 determines that the selected intent should still be a flight booking and extracts additional entity value information to add further to interface 500. Thus, user interface 500 morphs from the layout shown in Figure 5C to the layout shown in Figure 5D. For example, Figure 5D shows an extended widget 540D displaying departure date information for the flight requested by the user, with April 22 selected.

[0052] Figure 5E shows a fifth layout displayed for an interface related to a flight booking intent in an exemplary embodiment. When the morphing interface system 130 determines that all entity values ​​related to the selected intent have been extracted and applied to the intent, the intent may be executed to generate a response for the user. The layout of interface 500 may then include the display of the response. For example, when the entity recognition module 145 identifies the values ​​related to all entities necessary to book a flight, the intent is executed and a layout of interface 500 is presented that identifies flights that may meet the criteria specified in user input 530. For example, the user interface 500 morphs from the layout shown in Figure 5D to the layout shown in Figure 5E. The user may then select a flight from the presented options, for example, presented choices that match flights that are possible according to the specified criteria, e.g., widgets 540E, 540F, 540G, 540H, and 540I.

[0053] Figure 5F shows a sixth layout displayed for an interface related to a flight booking intent in an exemplary embodiment. In the example of Figure 5F, interface 500 displays selected flight information, allowing the user to review the data before submitting an order to purchase a flight. In some embodiments, the morphing interface system 130 may use various personalization techniques to determine the values ​​of user-related entities. In one embodiment, the morphing interface system 130 may store user profile information for use in executing the intent. For example, the morphing interface system 130 may store usernames, credit card numbers, home and work locations, etc. For example, in Figure 5F, the booking user and credit card information in widgets 540J and 540K is populated by the morphing interface system 130 without the information being included in user input 530. In some embodiments, the user morphing interface system 130 may use, for example, a voiceprint to verify the user's identity and determine whether to automatically include user information or whether additional user input is required. In some embodiments, the morphing interface system 130 may add or remove parts of the interface layout and entities pre-added as inputs within the layout, using a trained model that can predict layout parts and entity values ​​that the user is more likely to use. For example, a user's specific credit card input may be modified after several weeks of use to reflect the most recent of the user's regular actions.

[0054] Figure 5G shows a portion of the user interface displayed on the screen. This portion of the user interface is an email confirmation for the execution of a flight booking intent, according to an exemplary embodiment. An example in Figure 5G shows an email 550 received by the user confirming that the user has purchased a flight. In some intents, such an email confirmation may not be included as part of the intent execution.

[0055] Figures 6A to 6D below show an example of an interface that executes an intent when additional user input is received in one embodiment. Figures 6A to 6D show an example where user input 530E matches the pizza order intent. When additional entity values ​​related to the selected interface are determined in response to the receipt of additional user input, the displayed interface layout morphs and changes. In the example in Figures 6A to 6D, the morphing interface system 130, upon receiving and analyzing the additional user input, determines that the new intent matches the user input better than the initially selected intent, and the displayed interface layout morphs accordingly from the pizza order interface to the shirt purchase interface.

[0056] Figure 6A shows the receipt of a first portion of user input associated with a pizza order intent, according to an exemplary embodiment. The interface 500 associated with the pizza order is selected by the morphing interface system 130 based on the received user input 530E. In the particular example of Figure 6A, the morphing interface system 130 receives initial user input 530E, which includes a text string (on 520) "I want to order a pizza" or a voice input (on 510) corresponding to "I want to order a pizza". The morphing interface system 130 determines that the user input 530E is most likely associated with a pizza order intent, and the interface 500 initiates a transition (e.g., morphing) to display the pizza order layout accordingly, as shown by the graphical example 501 on the interface 500.

[0057] Figure 6B shows the layout displayed when transitioning from Figure 6A to Figure 6B for an interface related to ordering pizza, according to an exemplary embodiment. As in Figure 5A, the interface 500 may include a widget 540 that can populate entity values ​​determined by the entity recognition module 145. For example, Figure 6B includes a widget 540L that includes an area for entering a pizza restaurant. Furthermore, additional information, such as the price of pizzas and delivery times from a noted restaurant, may begin to appear on the morphed screen. In the example of Figure 6B, the entity recognition module 145 predicts which pizza restaurant the user might want to order from and populates the widget 540 of the user interface 500 with information about the restaurant.

[0058] Figure 6C shows a layout displayed for the interface related to purchasing a pizza T-shirt, as the interface continues to morph from Figure 6A, receiving more user input 530F according to an exemplary embodiment. In Figure 6C, user input 530F includes additional information. In particular, because the user has added input, user input 530F now includes "I want to order a pizza T-Shirt". The intent matching module 140 analyzes the additional user input and determines that the previously selected pizza order intent is no longer the most applicable intent, and the new most relevant intent is the shirt order intent. The morphing interface system 130 selects an interface 500 applicable to the shirt purchase intent and morphs the display on the client device 110 to display a layout related to the selected intent. For example, instead of displaying suggestions for pizza restaurants, the interface 500 in Figure 6C now displays a selection of pizza-themed T-shirts available for purchase. In other words, as the user interface 500 continues to provide additional input information, it morphs from the interface shown in Figure 6D, which displays a pizza order, to the interface shown in Figure 6C, which clearly and instantly displays the pizza T-shirt purchase option. The widget 540M associated with the exemplary shirt purchase intent may then contain pictures of pizza T-shirts that the user can select for purchase.

[0059] Figure 6D shows an example of an exemplary embodiment in which additional exemplary interfaces may be associated with a T-shirt purchase intent. For example, once the morphing interface system 130 determines that all the entity values ​​necessary to execute the intent are available, the process may be taken over by the user. In the example in Figure 6D, the user may select one of the pizza-themed T-shirt options, as shown in interface 500A; the user may view additional information about the selected item, as shown in interface 500B; and the user may review the order details and order the pizza T-shirt.

[0060] Examples in Figures 5A–5G and 6A–6D favorably illustrate a rapidly changing (e.g., morphing) user interface that changes via substantially (or nearly) simultaneous refreshes as the received user input gradually increases with additional information. Unlike conventional systems, here the user does not need to analyze the recommendations that conventional systems might present to them as the user provides user input. Furthermore, unlike conventional systems, the user does not need to wait for a complete selection of user inputs to begin seeing which user interface will be displayed for the instantaneous display (e.g., currently provided) corresponding to the increasing input string. Moreover, unlike conventional systems, the user interface that can be displayed on the computing device screen begins to reflect partial input almost instantaneously (or immediately), rapidly evolving to reflect the current input as additional items are contextually added to the user input, and ending with an appropriate final user interface corresponding to the complete user input. That is, the user interface enabled for display with text inputs TX0+TX1 is updated substantially immediately from the user interface enabled for displaying the original user input TX0.

[0061] (Example of a computing system) Figure 7 is a block diagram showing exemplary machine components that, according to an exemplary embodiment, can read instructions from a machine-readable medium and execute them on one or more processors (or controllers). Specifically, Figure 7 shows a schematic representation of a morphing interface system 130 in an exemplary form of computer system 700. Computer system 700 can be used to execute instructions 724 (e.g., program code, or software) causing the machine to perform one or more of the methodologies (or processes) disclosed herein. In an alternative embodiment, the machine operates as a standalone device or as a connected (e.g., networked) device connected to another machine. In a networked deployment, the machine may operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0062] A machine can be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a smartphone, an Internet of Things (IoT) device, a network router, a switch or bridge, or any machine capable of executing instruction 724 (sequential or otherwise) which specifies the actions to be performed by that machine. Furthermore, although only one machine is shown, the term “machine” is also construed to include any set of machines that execute instruction 724 individually or in combination to perform one or more of the methodologies described herein.

[0063] An exemplary computer system 700 includes one or more processing units (typically, a processor 702). The processor 702 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a controller, a state machine, one or more application-specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof. The processor runs an operating system for the computing system 700. The computer system 700 also includes main memory 704. The computer system may include a storage unit 716. The processor 702, memory 704, and storage unit 716 communicate via a bus 708.

[0064] Furthermore, the computer system 706 may include static memory 706, a graphics display (for example, for driving a plasma display panel (PDP), a liquid crystal display (LCD), or a projector). The computer system 700 may also include a character / number input device 712 (for example, a keyboard), a cursor control device 714 (for example, a mouse, trackball, joystick, motion sensor, or other pointing device), a signal generation device 718 (for example, a speaker), and a network interface device 720, which are also configured to communicate via bus 708.

[0065] The storage unit 716 includes a machine-readable medium 722 in which instructions 724 (e.g., software) embodying any one or more of the methodologies or functions described herein are stored. For example, instructions 724 may include instructions for implementing the functions of the NLP pipeline 135, the function matching module 140, and / or the entity recognition module 145. Instructions 724 (e.g., software) may also reside entirely or at least partially in the main memory 704 or in the processor 702 (e.g., in the processor's cache memory) during their execution by the computer system 700, and the main memory 704 and the processor 702 also constitute the machine-readable medium. Instructions 724 may be transmitted or received via the network interface device 720, via a network 726 such as network 120. Furthermore, for a client device (or user device), received instructions may be instructions from a server system that enable functions on the client device. For example, how a user interface is displayed may involve receiving code about how the user interface should be enabled for display (e.g., how it should be rendered), based on how the code properly interfaces with the client device's operating system.

[0066] In one embodiment, the machine-readable medium 722 is shown as a single medium, but the term “machine-readable medium” is to be interpreted as including one or more mediums capable of storing instructions 724 (e.g., a centralized or distributed database, or associated caches and servers). The term “machine-readable medium” is also to be interpreted as including any medium capable of storing instructions 724 for machine execution, causing a machine to execute any one or more methodologies disclosed herein. The term “machine-readable medium” includes, but is not limited to, data repositories in the form of solid memory, optical media, and magnetic media.

[0067] (Further considerations) The foregoing description of embodiments is presented for illustrative purposes only and is not intended to be exhaustive or to limit patent rights to the exact forms disclosed. Those skilled in the art will understand that, in consideration of the above disclosure, many modifications and variations are possible.

[0068] Certain parts of this description describe embodiments in terms of algorithms and symbolic representations of operations on information. These descriptions and representations of algorithms are commonly used by those skilled in the art in data processing techniques and effectively convey the essence of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, they are understood to be implemented by computer programs or equivalent electrical circuits, or microcode, etc. Furthermore, it is sometimes convenient to refer to sequences of these operations as modules, without losing universality. The operations described and the modules associated with them may be embodied in software, firmware, hardware, or any combination thereof.

[0069] Any step, operation, or process described herein may be executed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one example of an embodiment, the software module is implemented using a computer program product comprising a computer-readable medium containing computer program code executable by one or more computer processors to perform any or all of the steps, operations, or processes described herein.

[0070] Embodiments may also relate to apparatus for performing the calculations described herein. Such apparatus may consist of a computing device that is specifically constructed for a required purpose and / or selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-temporary, tangible, computer-readable storage medium, or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. For example, a computing device coupled to a data storage device for storing a computer program may correspond to a special-purpose computing device. Furthermore, any computing system referenced herein may include a single processor or may be an architecture employing multiple processor designs to enhance computing power.

[0071] Embodiments may also relate to products generated by computing processes described herein. Such products may consist of information arising from computing processes, which is stored in non-temporary, tangible, computer-readable storage media and may include any embodiment of combinations with computer program products or other data described herein.

[0072] Finally, the language used in this specification has been selected primarily for readability and educational purposes, and may not have been selected to describe or limit the subject matter of the invention. Therefore, the scope of the patent rights is intended to be limited not by the detailed description of this invention, but by any claims issued in an application based on this specification. Accordingly, the disclosure of embodiments is intended to illustrate, not to limit, the scope of the patent rights set forth in the following claims.

Claims

1. receiving a first user input from a user device, the first user input including an input string; generating a set of natural language processing signals based on the first user input; selecting an intent that matches the first user input, the selection being based on the natural language processing signal, the intent corresponding to a computer-executable function; identifying an interface associated with the intent; extracting a set of values ​​associated with entities of the interface from the first user input; substantially simultaneously with the receipt of the first user input, enabling a display of the interface on the user device, the displayed interface including a value from the set of values; A computer-implemented method comprising:

2. receiving a second user input from the user device, the second user input comprising a text string; generating an updated set of natural language processing signals based on a combination of the first user input and the second user input; selecting an intent that matches the combination of the first user input and the second user input, the selection being based on the updated set of natural language processing signals; Identifying a second interface associated with the newly selected intent; extracting a second set of values ​​associated with entities of the second interface from the combination of the first user input and the second user input; enabling the second interface for display on the user device, the second interface for display including values ​​from the second set of values; The computer-implemented method of claim 1 further comprising:

3. The computer-implemented method of claim 1 , wherein the first user input is a voice input.

4. 2. The computer-implemented method of claim 1, wherein selecting an intent matching the user input comprises comparing the first user input to one or more previously received user input strings; and in response to the first user input matching the one or more previously received user input strings, selecting an intent selected in response to at least one of the previously received user input strings.

5. 10. The computer-implemented method of claim 1, wherein selecting an intent that matches the user input comprises applying a trained computer model to predict a most applicable intent.

6. The computer-implemented method of claim 1 , wherein the interface comprises a set of related entities.

7. The computer-implemented method of claim 1 , wherein the first interface and the second interface are the same interface.

8. The computer-implemented method of claim 1 , wherein the first interface and the second interface are different interfaces.

9. The computer-implemented method of claim 1 , wherein the input string is a text string.

10. The computer-implemented method of claim 1 , wherein the input string is a voice input.

11. one or more computer processors for executing computer program instructions; A non-transitory computer-readable storage medium containing stored instructions executable by at least one processor, the instructions, when executed, causing the processor to: receiving a first user input from a first user device, the first user input including an input string; generating a set of natural language processing signals based on the first user input; selecting an intent that matches the first user input, the selection being based on the natural language processing signal, the intent corresponding to a computer-executable function; Identifying an interface associated with the intent; Extracting a set of values ​​associated with entities of the interface from the first user input; and substantially simultaneously with the receipt of the first user input, enabling the interface on the user device for display, the interface for display including a value from the set of values. a computer-readable storage medium; A computer system comprising:

12. and further including stored instructions that, when executed, cause the processor to further: receiving a second user input from the user device, the second user input comprising a text string; generating an updated set of natural language processing signals based on a combination of the first user input and the second user input; selecting an intent that matches the combination of the first user input and the second user input, the selection being based on the updated set of natural language processing signals; Identifying a second interface associated with the newly selected intent; extracting a second set of values ​​associated with entities of the second interface from the combination of the first user input and the second user input; and enabling the second interface for display on the user device, the second interface for display including values ​​of the second set of values; 12. The computer system of claim 11, wherein the computer system is configured to:

13. 12. The computer system of claim 11, wherein the first user input is a voice input.

14. 12. The computer system of claim 11, wherein the instructions to cause the processor to select an intent that matches the user input include instructions for comparing the first user input to one or more previously received user input strings, and in response to the first user input matching the one or more previously received user input strings, selecting an intent selected in response to at least one of the previously received user input strings.

15. 12. The computer system of claim 11, wherein the instructions to cause the processor to select an intent that matches the user input comprise instructions to cause the processor to apply a trained computer model to predict a most applicable intent.

16. The computer system of claim 11 , wherein an interface comprises a set of related entities.

17. 12. The computer system of claim 11, wherein the first interface and the second interface are the same interface.

18. 12. The computer system of claim 11, wherein the first interface and the second interface are different interfaces.

19. The computer system of claim 11 , wherein the input string is a text string.

20. The computer system of claim 11 , wherein the input string is a voice input.

Images