Intention recognition technology with reduced training

Abstract

The present disclosure relates generally to determining intent based on speech input using a dialogue system. More particularly, techniques are described for identifying intents corresponding to speech input in a dialogue system using matching-based machine learning techniques. These procedures do not require training when intents are added or removed from a set of possible intents.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority under 35 U.S.C. §119(e) to U.S. Application No. 62 / 899,654, filed September 12, 2019, entitled "REDUCED TRAINING INTENT RECOGNITION TECHNIQUES," and U.S. Application No. 16 / 942,535, filed July 29, 2020, entitled "REDUCED TRAINING INTENT RECOGNITION TECHNIQUES," the contents of which are incorporated herein by reference in their entirety for all purposes.

[0002] Technical Field The present disclosure relates generally to dialogue systems. More specifically, but not exclusively, the present disclosure describes techniques for performing natural language dialogue tasks using match functions that can significantly reduce the amount of training required to set up and maintain a dialogue system. [Background technology]

[0003] background

[0003] Currently, an increasing number of devices allow users to directly interact with the device using voice or oral speech. For example, a user can speak to such a device in natural language, where the user may ask a question or make an utterance requesting some action to be performed. In response, the device performs the requested action or responds to the user's question using a voice output. Because direct interaction using voice is a more natural and intuitive way for humans to communicate with their surroundings, the popularity of such voice-based systems is growing at an astronomical rate.

[0004] Voice-enabled systems capable of interacting with users via voice input and voice output come in a variety of forms. For example, such systems may be provided as standalone devices, digital or virtual assistants, voice-enabled services, etc. In each of these forms, the system can receive voice or speech inputs, understand these inputs, generate responses or take actions in response to these inputs, and output the responses using voice output. In certain embodiments, the interactivity in such voice-enabled systems is provided by a dialogue system or infrastructure (“dialogue system”).

[0005] Machine learning-based techniques are commonly used in dialogue systems for a variety of functions. For example, for voice input, a dialogue system may use supervised machine learning techniques to predict the intent of the input, where the intent is selected from a set of preconfigured intents. As another example, in question-and-answer applications, a dialogue system may use machine learning techniques to predict an answer to an input question. This is typically achieved using a classifier-based approach, which must be retrained when the target value or the underlying data associated with the value changes. As a result, traditional approaches require frequent retraining, which is cumbersome and requires the investment of significant time and computational resources in these retraining procedures. Summary of the Invention [Means for solving the problem]

[0006] Quick Overview

[0001] The present disclosure relates generally to automatically answering questions using a dialogue system. More particularly, techniques are described that use a matching-based machine learning system to identify values ​​(e.g., intents or answers) corresponding to speech inputs in a dialogue system. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.

[0007] In certain embodiments, a dialogue system receives an input verbal utterance. The dialogue system generates a vector representation of the input verbal utterance. The dialogue system uses a match function to generate a match score for each key / value pair among a plurality of key / value pairs, the match score generated for each key / value pair indicating a degree of match between a key in the key / value pair and the vector representation of the input verbal utterance, each key in the plurality of key / value pairs being a multidimensional vector representation of a sentence, and the match function is configured to generate a higher match score for a match of two different keys having the same associated value than for a match of two different keys having different associated values. The dialogue system identifies a particular key / value pair from the plurality of key / value pairs having the highest match score based on the match scores generated for the plurality of key / value pairs. The dialogue system outputs the value associated with the particular key / value pair as a value for the input verbal utterance.

[0008] In some aspects, the dialogue system further comprises training the match function using training data, the training comprising adjusting a set of parameters associated with the match function by minimizing a loss function. In some aspects, the dialogue system further comprises training the match function using training data, the training comprising adjusting a set of parameters associated with the match function by maximizing a likelihood function.

[0009] In some aspects, the values ​​of the key / value pairs are intents. In some aspects, the keys of the key / value pairs are expressions of questions, the values ​​of the key / value pairs are answers corresponding to each question, the output value associated with the particular key / value pair corresponds to a selected answer, and the value is output as a response to the input verbal utterance.

[0010] In some aspects, the input verbal utterance is a first input verbal utterance, the value is a first value, and the match score is a first match score, and the method further includes: updating the plurality of key / value pairs to include an additional key / value pair after the dialogue system outputs the first value for the first input verbal utterance; receiving a second input verbal utterance by the dialogue system; and generating a second match score for each key / value pair in the updated plurality of key / value pairs using the match function, wherein the match function does not change between generating the first match score and generating the second match score, and the method further includes: identifying a second particular key / value pair from the updated plurality of key / value pairs having a highest match score; and outputting a second value associated with the second particular key / value pair as a value for the input verbal utterance by the dialogue system. In some aspects, the match function includes one of a cosine similarity function, a bilinear function, a single layer network, a multi-layer network, or a nearest neighbor function.

[0011] In some embodiments, a training method for training a match function for use in a dialogue system is disclosed. The method includes training the match function using training data. The match function is trained to generate a match score for a key of a key / value pair in a plurality of key / value pairs against another key of another key / value pair. The generated match score indicates a degree to which the two keys match. Each key in the plurality of key / value pairs is a multidimensional vector representation of a sentence. The match function is configured to generate a higher match score for a match of two different keys with the same associated value than for a match of two different keys with different associated values.

[0012] In some embodiments, a training apparatus for training a match function used in a dialogue system is disclosed. The apparatus comprises means for training the match function using training data. The match function is trained to generate a match score for a key of a key / value pair in a plurality of key / value pairs against another key of another key / value pair. The generated match score indicates a degree of matching between the two keys. Each key in the plurality of key / value pairs is a multidimensional vector representation of a sentence. The match function is configured to generate a higher match score for a match of two different keys with the same associated value than for a match of two different keys with different associated values.

[0013] In some embodiments, a non-transitory computer-readable medium is disclosed. The non-transitory computer-readable medium stores a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform a process, the process comprising training a match function using training data. The match function is trained to generate a match score for a key of a key / value pair in a plurality of key / value pairs against another key of another key / value pair. The generated match score indicates a degree to which the two keys match. Each key in the plurality of key / value pairs is a multidimensional vector representation of a sentence. The match function is configured to generate a higher match score for a match of two different keys with the same associated value than for a match of two different keys with different associated values.

[0014] Embodiments further include systems, computer readable memories and programs configured to perform the methods described herein.

[0015] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims and accompanying drawings. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a simplified block diagram illustrating a dialogue system according to certain embodiments. [Figure 2] 1 is a simplified flowchart of a method for matching an input spoken utterance with a value, according to some embodiments. [Figure 3] 1 is a simplified flowchart illustrating exemplary processing performed during a setup phase, according to certain embodiments. [Figure 4]10 is a simplified flowchart illustrating another example of processing performed during a setup phase according to a further embodiment. [Figure 5] 1 is a simplified diagram of a distributed system for implementing one embodiment. [Figure 6] FIG. 1 is a simplified block diagram of a cloud-based system environment in which various services may be provided as cloud services, according to certain embodiments. [Figure 7] FIG. 1 illustrates an exemplary computer system that may be used to implement certain embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0017] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting. As used herein, the word "exemplary" is used to mean "serving as an example, illustration, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0018] The present disclosure relates to techniques for identifying values ​​for speech inputs in dialogue systems using matching-based machine learning techniques. The values ​​may correspond to intents or answers to questions. The identified values ​​may be tied to functions supported by the system (such as making a purchase and getting directions). These techniques for determining values ​​using match functions do not require training when values ​​are added or removed from a set of possible values. This allows developers to add intents or answers to a chatbot or digital assistant and immediately test the system with the new intents or answers without waiting for a training step.

[0019] Most machine learning-based techniques require a training step, which can be computationally intensive and take a day or even longer. Even very fast training takes a few minutes (e.g., 5-15 minutes is considered fast training). As a result, for existing chatbot systems that use traditional machine learning-based classifiers, it is very difficult to update the system (e.g., add features to the chatbot system) and deploy the system in a timely manner because the update requires retraining the system.

[0020] For example, when a user query or utterance is provided, traditionally, a classifier based on a traditional classification algorithm is used to determine the intent of the query. Models built using machine learning techniques are used as classifiers to perform classification, and building these models requires a training step. These classifier models require retraining when the underlying data (e.g., the set of candidate classes) changes. As a specific example, with traditional classifier approaches (e.g., supervised classification), when a new intent is added to a chatbot or digital assistant, the machine learning training algorithm must be re-run to be able to process the new intent. Thus, there is a significant time delay between adding a new intent to the training data and having an updated procedure for mapping user requests to intents. This time delay makes it difficult for developers to interactively develop chatbots or digital assistants with new intents or to demonstrate chatbot or digital assistant development toolkits that include developers adding new intents.

[0021] The techniques described herein do not require such retraining. For example, developers can add new intents to the system and try them out without retraining. New techniques are disclosed for performing tasks (such as intent recognition or question answering) in dialogue systems using different types of algorithms that do not require retraining when intents are added or removed. As described herein, instead of traditional classification algorithms, match functions are used to identify values ​​for utterances.

[0022] Match Function Example In some embodiments, the intent determination problem is solved using information retrieval techniques (e.g., match functions) that do not need to be retrained each time the set of candidate intents changes. Thus, the intent determination problem is solved using a different technique than traditional classification algorithm-based techniques. That is, the information retrieval technique (match function) is used to solve the problem. Alternatively or additionally, the match function may be used to perform other tasks in the dialogue system (such as question answering).

[0023] In a chatbot system environment, a match function is obtained through training. This match function is then used to match a user request utterance with the candidate utterances provided for each intent in the set of candidate intents. The user request utterance that is the subject of intent determination is matched against the set of candidate utterances provided for the set of candidate intents.

[0024] In some embodiments, a key / value pair lookup technique is used. For example, in a question and answer (QnA) setting for a chatbot system, a question or query asked by a user is matched with a key from a set of multiple keys to look up a value associated with the matched key. This value can be output as a response to the query or used to determine the response to the query.

[0025] For example, the matching-based value calculation may be performed as follows: (1) A query is received from a user.

[0026] (2) This query is compared with keys in a set of key / value pairs. For example, these keys are sample utterances (e.g., sentences) and these values ​​are the corresponding intents. The key / value pairs can be configured by the customer for whom the chatbot system is built. The best match key for the query is identified.

[0027] (3) The value associated with this best match key is then used for downstream processing. For example, this value may be output as a response to the query. As another example, this value may identify an intent corresponding to the query, which may then be used to generate a response.

[0028] Generally, there are multiple keys that are associated with the same value. In such a scenario, a match function (or model) is generated that preferentially matches two keys that map to the same value over keys that point to different values. As an example, a model is trained to preferentially match the key "Create a new account for me" to the key "How can I open an account?" with a higher match score than "Can I close my account?" because both keys match the same value (e.g., OPEN_ACCOUNT intent), while "Can I close my account?" maps to a different value (e.g., CLOSE_ACCOUNT intent). Similarly, "How can I open an account?" matches "Create a new account for me" with a higher match score than "Can I close my account?". The training process is to learn the match function so that keys with the same value match each other preferentially over keys with different values. The match function is trained so that the match score, which indicates the degree of match, is higher for keys that map to the same value compared to keys that map to different values.

[0029] In certain embodiments, the match function returns a ranked list of matches based on the match score. In the context of a chatbot, the match with the highest score is selected. Various types of match functions may be used. These match functions may have parameters, and as part of the learning process, these parameters are adjusted so that keys with identical values ​​match better than keys with different values. Some match functions do not have their own parameters (e.g., dot product / cosine similarity algorithms).

[0030] In particular embodiments, a neural network may be used to generate embeddings based on input training key / value pairs that are used to generate the match function. During the training process, the parameters of the neural network and the parameters of the match function are adjusted so that keys with the same associated value match better than keys with different values.

[0031] In some embodiments, to ensure that the match function matches all keys that map to the same value with a higher match score than keys that map to different values, keys associated with the same value are treated as a positive pair and keys with different values ​​are treated as a negative pair. The match function is trained to ensure that a positive pair (e.g., two keys that map to the same value) has a higher match score than a corresponding negative pair (e.g., two keys that map to different values). The match score of a positive pair should be higher than the match score of a negative pair. There may be multiple positive pairs, and two different positive pairs may have different match scores, as long as these scores are higher than the match score of the negative pair. The training is done to ensure that the match function is such that positive pairs are at the top of the list of matches, and that positive pairs are at the top of the list (i.e., have the highest match score).

[0032] The match function may be generated using a variety of techniques. In some embodiments, the match function may be determined based on a ranking algorithm. In some embodiments, the match function may be determined based on probability instead of ranking. Thus, there are a variety of ways to generate a match function.

[0033] Using the Match Function The match function learned as described above is then used to find a matching key (from a dataset comprising multiple key / value pairs) for a query from a user. The query is matched against all keys using the match function, and the best match key (having the highest match score) is identified. The matching key is identified using the match function. The value associated with the best match key is then retrieved and used for downstream processing. For example, this value may represent the intent of the query. As another example, this value may be output as a response to the query.

[0034] This model can be used to find a matching key for a query when there is an exact match between the query and the key, or even when the query differs from the key in certain parts.

[0035] Differences from standard classifiers In standard classification techniques, a classifier is trained using a dataset D. Standard classifier-based approaches have the property that they will not return values ​​that were not seen in the training dataset D. In contrast, the techniques disclosed in this disclosure use a match function rather than a classification algorithm. This match function is trained on a different dataset (or the match function is not trained at all), and the dataset used at runtime may not be the same as the dataset used to train the match function. Because the dataset used at runtime may not be the same as the dataset used to train the match function, the match function has the potential to return values ​​that were not present in the training dataset used to learn the match function. This allows, for example, new intents to be added without having to go through a retraining process.

[0036] Exemplary Dialogue System FIG. 1 illustrates an example of a dialogue system 100 according to some embodiments. The dialogue system 100 is configured to receive voice or speech inputs 104 (also referred to as speech utterances) from a user 102. The dialogue system 100 may then interpret these voice inputs. The dialogue system 100 may maintain a dialogue with the user 102 and, in some cases, perform or cause to be performed one or more actions based on the interpretation of these voice inputs. The dialogue system 100 may prepare appropriate responses and output these responses to the user using voice or speech output. The dialogue system 100 is a dedicated computing system that may be used to process large amounts of data using many computer processing cycles. The multiple devices shown in FIG. 1 are provided for illustrative purposes. A different number of devices may be used. For example, although each device, server, and system in FIG. 1 is shown as a single device, multiple devices may instead be used.

[0037] In particular embodiments, the processing performed by the dialogue system is implemented by a pipeline of components or subsystems, including a speech input component 105, a wake word detection (WD) subsystem 106, an automatic speech recognition (ASR) subsystem 108, and a natural language understanding (NLU) subsystem 110, which includes a named entity recognizer (NER) subsystem 112, an intent matching subsystem 113, and a semantic parser subsystem 114, which further include a dialogue manager (DM) subsystem 116, which may include a matching database 115, a QnA matching subsystem 117, a natural language generator (NLG) subsystem 118, a text-to-speech (TTS) subsystem 120, and a speech output component 124. The above subsystems may be implemented solely in software (e.g., using code, programs, or instructions executable by one or more processors or cores), solely in hardware, or a combination of hardware and software. In particular implementations, one or more of the subsystems may be combined into a single subsystem. Additionally or alternatively, in some implementations, functionality described herein as being performed by a particular subsystem may be performed by multiple subsystems.

[0038] The audio input component 105 includes hardware and software configured to receive audio input 104. In some examples, the audio input component 105 may be part of the dialogue system 100. In some other examples, the audio input component 105 may be separate from and communicatively coupled to the dialogue system 100. The audio input component 105 may include, for example, a microphone coupled to software configured to digitize and transmit the audio input to the wake word detection subsystem 106.

[0039] The wake word detection (WD) subsystem 106 is configured to listen and monitor the audio input stream for input corresponding to a special sound or word or set of words (referred to as a wake word). Upon detecting a wake word configured for the dialogue system 100, the WD subsystem 106 is configured to activate the ASR subsystem 108. In particular implementations, a user may be provided with the ability to activate and deactivate the WD subsystem 106 (e.g., by speaking the wake word and pressing a button). Once activated (or operating in an active mode), the WD subsystem 106 is configured to continuously receive an audio input stream and process the audio input stream to identify audio or speech input corresponding to the wake word. When audio input corresponding to the wake word is detected, the WD subsystem 106 activates the ASR subsystem 108.

[0040] As described above, the WD subsystem 106 activates the ASR subsystem 108. In some implementations of a voice-enabled system, mechanisms other than a wake word may be used to trigger or activate the ASR subsystem 108. For example, in some implementations, a push button on the device may be used to trigger processing of the ASR subsystem 108 without requiring a wake word. In such implementations, the WD subsystem 106 may not be provided. When the push button is pressed or activated, voice input received after the button activation is provided to the ASR subsystem 108 for processing. In some implementations, the ASR subsystem 108 may be activated when it receives input to be processed.

[0041] The ASR subsystem 108 is configured to receive and monitor verbal speech input after a trigger or wake-up signal (e.g., a wake-up signal may be sent by the WD subsystem 106 upon detecting a wake word in the speech input, a wake-up signal may be received upon activation of a button, etc.) and convert this verbal speech input to text. As part of its processing, the ASR subsystem 108 performs speech-to-text conversion. The verbal speech or speech input may be in a natural language format, and the ASR subsystem 108 is configured to generate corresponding natural language text in the language of the speech input. The text generated by the ASR subsystem is then fed to the NLU subsystem 110 for further processing. The speech input received by the ASR subsystem 108 may include one or more words, phrases, clauses, sentences, questions, etc. The ASR subsystem 108 is configured to generate a text utterance for each verbal clause and feed these text utterances to the NLU subsystem 110 for further processing.

[0042] The NLU subsystem 110 receives the text generated by the ASR subsystem 108. The text received by the NLU subsystem 110 from the ASR subsystem 108 may include text utterances that correspond to spoken words, phrases, clauses, etc. The NLU subsystem 110 converts each text utterance (or a sequence of text utterances) into its corresponding logical form.

[0043] In a particular implementation, NLU subsystem 110 includes a named entity recognizer (NER) subsystem 112, an intent matching subsystem 113, and a semantic parser (SP) subsystem 114. NER subsystem 112 receives a text utterance as input, identifies named entities within the text utterance, and tags the text utterance with information related to the identified named entities. The tagged text utterance is then fed to SP subsystem 114 (configured to generate a logical form for each tagged text utterance (or for a series of tagged text utterances)) and to intent matching subsystem 113 (configured to identify one or more intents corresponding to the text utterance that can be used by SP subsystem 114 in generating the logical form).

[0044] The intent of an utterance identifies the purpose of the utterance. Example intents include "Order_Pizza" and "Find_Directions." Each intent is given an intent identifier or name. For example, intents that the intent matching subsystem 113 is trained to recognize may include "CheckBalance," "TransferMoney," "DepositCheck," etc. An intent may, for example, identify an action requested to be performed. An intent may represent a category or class that the dialogue system is trained to infer for an input utterance. When the intent matching subsystem 113 receives an utterance, it may determine the intent of the utterance, and the inferred intent is selected from a set of predefined intents used to train the intent matching subsystem 113. The dialogue system then takes an appropriate action in response to the utterance based on the inferred intent for the utterance.

[0045] For each specified intent, the designer may provide one or more example utterances that correspond to and demonstrate the intent. These example utterances are intended to represent utterances that a user may input into the dialogue system for that intent. For example, for the CheckBalance intent, example utterances may include, "What's the balance in my savings account?", "How much is in my checking account?", "How much money do I have in my account?", etc. Thus, various permutations of typical user utterances may be specified as example utterances for the intent.

[0046] These intents and associated example utterances are used as training data to train the intent matching subsystem 113. This training results in a predictive model (e.g., a match function) configured to take an utterance as input and output an inferred intent for the utterance.

[0047] In some examples, an input utterance is provided to the intent matching subsystem 113, which is configured to predict or infer the intent of the input utterance using a trained model. The dialogue system may then take one or more actions based on the inferred intent. As described herein, the intent matching subsystem 113 may apply a match function to the text utterance to predict the intent corresponding to the text utterance. The intent matching subsystem 113 may interact with the matching database 115 to apply the match function. The matching database 115 may store a plurality of key / value pairs that map representations of sample text utterances to values. In some embodiments, the text utterance is the key in the key / value pair, and the intent is the value in the key / value pair. Alternatively, or in addition, the value in the key / value pair may correspond to an answer to a question. The intent matching subsystem 113 may apply the match function to identify a key stored in the matching database 115 that best matches the text utterance. The intent matching subsystem 113 can then identify the corresponding value and thereby predict the intent corresponding to the text utterance.

[0048] In addition to intent, the logical form generated for a text utterance may also identify slots (also referred to as parameters or arguments) for the identified intent. For example, for a speech input of "I would like to order a large pepperoni pizza with mushrooms and olives," the NLU subsystem 110 can identify the intent to order pizza. The NLU subsystem can also identify and fill slots (e.g., pizza_size (filled with "large") and pizza_toppings (filled with "mushrooms and olives")). The NLU subsystem may generate the logical form using machine learning-based techniques, rules (which may be domain-specific), or a combination of the two. The logical form generated by the NLU subsystem 110 is then fed into the DM subsystem 116 for further processing.

[0049] The DM subsystem 116 is configured to manage interactions with the user based on the logical forms received from the NLU subsystem 110. As part of interaction management, the DM subsystem 116 is configured to track the interaction state and initiate or itself perform one or more actions or tasks to determine how to interact with the user. These actions may include, for example, querying one or more databases, generating execution results, and other actions. For example, the DM subsystem 116 is configured to interpret the intent identified in the logical forms received from the NLU subsystem 110. Based on this interpretation, the DM subsystem 116 may initiate one or more actions that are interpreted as being requested by the voice input provided by the user.

[0050] In particular embodiments, the DM subsystem 116 performs dialogue state tracking based on current and past voice inputs and based on a set of rules (e.g., dialogue policies) configured for the DM subsystem 116. These rules may specify various dialogue states, conditions for transitioning between states, actions to be performed when in a particular state, etc. These rules may be domain-specific.

[0051] In some embodiments, the DM subsystem 116 uses a question and answer (QnA) matching subsystem 117 to identify answers to queries. As described herein, the QnA matching subsystem 117 may apply a match function to the text utterance to predict an answer corresponding to the query. The QnA matching subsystem 117 may interact with the matching database 115 to apply the match function. As described above, the matching database 115 may store multiple key / value pairs that map the text utterance to values ​​that may correspond to answers to a question. The QnA matching subsystem 117 may apply the match function to identify a key stored in the matching database 115 that best matches the text utterance (e.g., a query (e.g., a query posed by a user via voice input)). The intent matching subsystem 113 may then identify the corresponding value, thereby predicting an answer corresponding to the text utterance.

[0052] The DM subsystem 116 also generates responses that are returned to users involved in the interaction. These responses may be based on actions initiated by the DM subsystem 116 and their outcomes. The responses generated by the DM subsystem 116 are fed into the NLG subsystem 118 for further processing.

[0053] The NLG subsystem 118 is configured to generate natural language texts corresponding to the responses generated by the DM subsystem 116. These texts may be generated in a format that allows them to be converted into speech by the TTS subsystem 120. The TTS subsystem 120 receives these texts from the NLG subsystem 118 and converts each of them into speech or voice audio, which may then be output to a user via an audio or voice output component 124 of the dialogue system (e.g., a speaker or a communication channel coupled to an external speaker). In some examples, the voice output component 124 may be part of the dialogue system 100. In some other examples, the voice output component 124 may be separate from the dialogue system 100 and communicatively coupled to the dialogue system 100.

[0054] As described above, the various subsystems of the dialogue system 100, working together, provide functionality that enables the dialogue system 100 to receive speech input 104, respond using speech output 122, and maintain a dialogue with a user using natural language speech. The various subsystems described above may be implemented using a single computer system or multiple computer systems working together. For example, in a device implementing a voice-enabled system, the subsystems of the dialogue system 100 described above may be implemented solely on the device with which the user interacts. In some other implementations, some components or subsystems of the dialogue system 100 may be implemented on the device with which the user interacts, while other components may be implemented remotely from the device, possibly on other computing devices, platforms, or servers.

[0055] As noted above, in certain embodiments, the dialogue system 100 may be implemented using a pipeline of subsystems. In some embodiments, one or more of the subsystems may be combined into a single subsystem. In certain embodiments, the functionality provided by a particular subsystem may be provided by multiple subsystems. Also, a particular subsystem may be implemented using multiple subsystems.

[0056] In particular embodiments, machine learning techniques may be used to perform one or more functions of the dialogue system 100. For example, supervised machine learning techniques, such as those implemented using neural networks (e.g., deep neural networks), may be used to perform one or more functions of the dialogue system 100. As an example, a neural network trained to perform the ASR function to be performed may be provided, and such trained model may be used by the ASR subsystem 108 for its processing. Such a neural network implementation may take speech input as input and output text utterances to the NLU subsystem. Machine learning-based models may also be used by other subsystems of the dialogue system 100.

[0057] Use the match function to find a value FIG. 2 is a simplified flowchart illustrating a method 200 for determining a value using a match function, according to some embodiments. The process illustrated in FIG. 2 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method illustrated in FIG. 2 and described below is intended to be exemplary and non-limiting. While FIG. 2 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, these steps may be performed in some different order, or some steps may be performed in parallel. In certain embodiments, the process illustrated in FIG. 2 may be performed by the dialogue system 100 described above with respect to FIG. 1.

[0058] As shown in FIG. 2 , at 202, an input verbal utterance is received. The input verbal utterance may result from words spoken by a user. The dialogue system may receive the input verbal utterance from a user, for example, via the voice input component 105. The input verbal utterance may be received as a waveform. The input verbal utterance may be in a particular language and in the form of a natural language query. The input verbal utterance may specify a question the user wants the dialogue system to answer, specify one or more actions the user wants the dialogue system to initiate (e.g., "I'll have a medium pepperoni pizza"), etc. The input verbal utterance may be processed by the ASR subsystem 108 to generate a text utterance, which is passed to the NLU subsystem 110 for further processing.

[0059] In certain embodiments, instead of receiving voice input (as at 202), the input may be received in text form, for example, when a user types text while chatting with the dialogue system. The techniques described in this disclosure can be used with systems that receive input in the form of oral speech or text input, or a combination thereof.

[0060] At 204, the dialogue system generates a vector representation of the input verbal utterance. The dialogue system may generate a sentence encoding of the input verbal utterance using the text utterance generated at 202. The dialogue system may generate an encoded version of the input verbal utterance for processing (e.g., a set of word embeddings including a separate embedding for each word in the input verbal utterance, where each embedding is a multidimensional feature vector including values ​​for the features of the corresponding word).

[0061] At 206, the dialogue system uses the match function to generate a match score for each key / value pair in the plurality of key / value pairs. The dialogue system may calculate the plurality of match scores by applying the match function to the plurality of keys in each key / value pair and the vector representation of the input verbal utterance generated at 204. The match score generated for the key / value pair indicates the degree of matching between the key in the key / value pair and the vector representation of the input verbal utterance.

[0062] The dialogue system may store multiple key / value pairs 125 in the matching database 115. Each key / value pair includes a key, which may be a string or a representation thereof. These keys may correspond to user queries (such as "I need directions to the store" or "How do I change my cable subscription?").

[0063] In some embodiments, each key in the multiple key / value pairs is a multidimensional vector representation of a sentence, phrase, word, or set of words. These keys may be encoded as dense vectors (e.g., as sentence embeddings), so that each i ∈R d where d is the dimension of the embedding. Embeddings encode words and sentences into fixed-length vectors. For example, embeddings may be sentence embeddings (such as Bidirectional Encoder Representations with Transformers (BERT) or Universal Sentence Encoder Embeddings). Embeddings may be trained using machine learning techniques (e.g., using neural networks). Embeddings may be used to generalize speech input types. Thus, the dialogue system may generate a trained set of embeddings representing multiple sample utterances, and these embeddings may be used to calculate match scores. In some embodiments, universal embeddings may be performed. The use of universal embeddings can facilitate identifying values ​​based on input regardless of the language in which the input is received.

[0064] In some embodiments, a key can incorporate more information than the sample utterance (e.g., a question or utterance) alone. For example, a key can include information extracted from an associated answer or value (e.g., a key can be a linear combination of a question embedding and an answer embedding). As a specific example, the value corresponding to a key is a link to a website. The key can include information derived from the link to the website (e.g., words on the website), which can be used to facilitate the matching process. For example, in the key "close an account," another word for "close" is "shut." The website might include the phrase "procedures for closing an account." Including such information derived from the value can make it easier to distinguish intent across more types of speech input.

[0065] For each key / value pair, the key may be mapped to a respective value, which may be an intent, an answer to a question, or other suitable value. i ∈V, where V is a set of values. These values ​​may or may not be embedded. Each key of multiple key / value pairs may be stored in association with a corresponding value in the matching database 115. For example, a vector representation of the utterance "I need directions to the store" may be mapped to the intent "FIND-DIRECTIONS." Alternatively or additionally, the keys and values ​​may correspond to questions and answers, where the key corresponds to the question and the value provides the answer to the question. As an example, Table 1 shows a set of key / value pairs.

[0066] [Table 1]

[0067] Three keys are stored in a table in association with values. In the example shown in Table 1, the keys are queries (e.g., "How can I open an account?"). The values ​​correspond to the answers paired with each query. In the example shown in Table 1, the values ​​are URLs that can be used to instruct a user how to open or close an account. These keys and values ​​are stored in a matching database 115 as D=((k1,v1),...,(k n ,v n )) where k i is the i-th speech input type or key, and v i is the corresponding value.

[0068] In some embodiments, the values ​​are intents. In this case, each key may represent a sample utterance, and each corresponding value is the intent of the sample utterance. For example, a key / value pair may consist of the key "1 large pepperoni" and the corresponding value "Order_Pizza." Alternatively or additionally, the keys of the multiple key / value pairs are representations of questions, and the values ​​of the multiple key / value pairs are answers corresponding to each question. For example, a key / value pair may include the key "How much is a large pepperoni pizza?" and the corresponding value "$19.99."

[0069] The dialogue system calculates a match score by applying a match function to the plurality of sample utterances and the speech input received at 202. The match function may be applied to each stored key along with the input verbal utterance received at 202. The match function may return how well the received input verbal utterance matches the stored sample utterance. The match function may be associated with a parameter θ, which may be learned or fine-tuned using information retrieval machine learning methods (such as ranking algorithms or probabilistic algorithms). The match function may be configured to generate a higher match score for a match of two different keys with the same associated value than for a match of two different keys with different associated values.

[0070] In some embodiments, a match function Match() maps a query q and a key k to a real-valued match score Match(q,k). The query corresponds to the input verbal utterance at 202. As a specific example, the corpus of stored key / value pairs is dataset D. Dataset D=((k1,v1),...,(k n ,v n )) is a collection of n key / value pairs (k i ,v i ) and k i is the key and v iis the associated value. Given a speech input q, the system returns the key / value pair in D with the best matching key. The function BestMatch() returns the best matching key / value pair.

[0071]

number

[0072] where q is the sentence encoding of the speech input and θ is a vector of parameters for the match function. As further described below with respect to Figure 3, the match function is further based on training triples T of positive and negative key pairs. Exemplary match function implementations include:

[0073] [Table 2]

[0074] Some match functions return the inverse distance (e.g., dot product score), in which case the matching problem is considered to be finding the pair in D where key k is closest to the speech input / query q. The dot product / cosine implementation uses the cosine similarity function, which provides the degree of similarity between two vectors based on the cosine of the angle between them. (See, for example, Jiawei Han et al., "Getting to Know Your Data," Data Mining (3rd ed.) (2012).) The dot product / cosine match function has no parameters that require training data, although training data can still be useful for fine-tuning a sentence embedding model that generates sentence encodings for the speech input type.

[0075] The remaining match functions shown in Table 2 have a parameter θ (third column), which can be estimated from training data as described below with respect to Figures 3 and 4. A bilinear implementation uses a bilinear function, such as that described by Wei Wu and Zhendong-Lu and Han Li, "Learning Bilinear Model for Matching Queries and Documents," Journal of Machine Learning Research, 14, 2519-2548 (2013). A single-layer network is a single-layer neural network. A multilayer network is a multilayer neural network. An exemplary match function for the multilayer network implementation shown in Table 2 includes the rectified linear unit activation function (ReLU). However, other nonlinearities (such as tanh()) may be used instead of ReLU().

[0076] Alternatively or additionally, a probabilistic model (e.g., a nearest neighbor function) may be implemented. The probabilistic model involves computing a partition function. Backpropagation through the partition function may be difficult when the training data set is large. Error-driven approximation should require relatively low memory usage. An exemplary probabilistic model is as follows:

[0077]

number

[0078] D v is the subset of pairs in D that have value v. The probabilistic model shown by equations [2] and [3] is an exponential model over D that marginalizes over key k and uses match scores as potentials.

[0079] The match function may be initially trained to generate parameters θ. This training may include optimizing θ for a dataset D using a neural network, as described below with respect to Figures 3 and 4. The dialogue system may initially train the match function using training data, where the training comprises adjusting a set of parameters associated with the match function by minimizing a loss function, as described with respect to Figure 3. Alternatively or additionally, the training may comprise adjusting a set of parameters associated with the match function by maximizing a likelihood function, as described with respect to Figure 4.

[0080] Thus, the dialogue system may determine a set of parameters for the match function based on a set of sample key-value pairs before receiving the input verbal utterance at 202. The sample key-value pairs used for training may be the same as those stored and used to calculate the match score. Alternatively, the sample key-value pairs used for training may be different from those stored and used to calculate the match score. Thus, keys and values ​​may be added or removed after initial training.

[0081] Using the match function, the dialogue system may generate various match scores corresponding to the degree of similarity between the vector representation of the input verbal utterance and each stored key. For example, the speech input "How do I get to Montreal?" may have a match score of 10 with "Find out how to get to Montreal" and a match score of 4 with "What is Montreal?"

[0082] At 208, the dialogue system identifies the particular key / value pair with the highest match score based on the match scores generated at 206. The dialogue system may compare the match scores generated at 206 to identify the key / value pair with the highest match score. For example, the utterance "How much is a hamburger?" has a match score of 70% with the key "How much is a ham sandwich," an 80% match score with the key "How much is a veggie burger," and a 100% match score with the key "How much is a hamburger?" Thus, a matching key / value pair with a match score of 100% with "How much is a hamburger?" is identified. Alternatively, in some embodiments, the dialogue system may assign match scores inversely proportional to the level of matching, in which case the key / value pair with the lowest match score may be selected.

[0083] At 210, the dialogue system outputs the value associated with the particular key / value pair as the value of the input verbal utterance. In some embodiments, the dialogue system selects a corresponding value from the key / value pair identified at 308 and generates and provides a response corresponding to the selected value.

[0084] The dialogue system may traverse the key / value pair with the highest match score selected in 208 to identify a value therein. This value may correspond to an intent, an answer, or other information used to execute dialogue processing. As a specific example, the intent matching subsystem may identify the intent mapped to the key “Find directions to Montreal” after determining that the key “Find directions to Montreal” has the highest match with the input utterance. The identified intent is DIRECTIONS_TO, which is stored in association with the stored key with the highest match score. As another example, the QnA matching subsystem may identify a URL that can be used to search for an answer to the question or the answer stored in association with the highest matching key (e.g., the QnA matching subsystem may identify the address of the nearest coffee shop as the value mapped to the highest-scoring key of “Where is the nearest coffee shop?”). In this case, the output value associated with the particular key / value pair may correspond to the selected answer, and the value is output as a response to the input verbal utterance.

[0085] The dialogue system may output the value to the user and / or another subsystem of the dialogue system. As an example of the latter, the intent matching subsystem 113 outputs the identified intent to the SP subsystem 114 for generating a logical form based in part on the identified intent. In some cases, the dialogue processing pipeline continues to generate output for the user. The output for the user may include the identified value and / or its derivatives as audio output and / or visual cues.

[0086] In some embodiments, the dialogue system generates a response that includes or is derived from the selected value. For example, the value is the intent "FIND_STORE." The dialogue system may use GPS or follow-up questions to identify the user's location and identify the closest store to the identified location. The dialogue system may then package the identified store as a user-friendly response (e.g., by adding words to form a complete sentence (e.g., "Jay's Party Supply is at 3215 Main Street. Want to get directions?"). In some embodiments, the selected value may include information that includes the response or part of a response (e.g., the QnA matching subsystem may identify a URL (e.g., one shown in Table 1 above)).

[0087] The dialogue system may provide the response as output (e.g., to the user who provided the verbal utterance input at 202) via audio output (e.g., by converting the text response generated by the dialogue manager subsystem 116 and natural language generator subsystem 118 into audio output via the text-to-speech subsystem 120). Alternatively or additionally, the dialogue system may provide the response by causing the text response to be displayed on a display component of the user device.

[0088] Values ​​may then be added to or removed from the stored key / value mappings. The dialogue system may process a second input verbal utterance without retraining the model to process the changed value. For example, the input verbal utterance received at 202 is the first input verbal utterance, and the value output at 210 is the first value. After determining the first value, the dialogue system updates the set of key / value pairs to include additional keys and / or values. For example, a customer may provide a set of additional questions and answers that they would like the call center chatbot to be able to process. The dialogue system's database is updated to reflect the updated set of key / value pairs (e.g., by storing the additional key / value pairs in a dataset). As a specific example, the dialogue system may add a key / value pair including a sample utterance "I want to return..." and the intent "RETURN_ITEM," a key / value pair including a sample utterance "How do I return something?" with the intent "RETURN_ITEM," and so on, along with multiple key / value pairs including various keys that correspond to the new value "RETURN_ITEM." The dialogue system then receives a second input verbal utterance. The second input verbal utterance may correspond to a recently added value (e.g., the second speech input is "I want to return the shirt," which matches the key mapped to the intent "RETURN_ITEM"). The dialogue system uses a match function (e.g., as described above with respect to 204-208) to determine a second value that corresponds to the second input verbal utterance. The dialogue system is not retrained between adding the additional key / value pair and processing the second input verbal utterance (e.g., the set of parameters of the match function does not change between determining the first value and determining the second value). As another example, values ​​can be removed on the fly from the set of possible values. One exemplary application of adding and removing intents on the fly is to allow each user to be associated with their own personalized set of intents.

[0089] FIG. 3 is a simplified flowchart 300 illustrating processing performed during a setup phase, according to certain embodiments. The processing illustrated in FIG. 3 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method illustrated in FIG. 3 and described below is intended to be exemplary and non-limiting. While FIG. 3 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, these steps may be performed in some different order, or some steps may be performed in parallel. In certain embodiments, the processing illustrated in FIG. 3 may be performed by a dialogue system as described above.

[0090] The process shown in Figure 3 may be used in a training step to configure the parameters of the match function. As described above with respect to Figure 2, the dialogue system may perform a training process that results in a set of parameters for the match function before receiving the input verbal utterance at 202. In some embodiments, this involves minimizing a loss function over a set of key / value pairs, which may be performed using the process shown in Figure 3.

[0091] 3, at 302, the dialogue system obtains training data. The dialogue system may obtain the training data from a variety of sources (from previous voice input received by the system, from customers, and / or by web scraping). For example, the training data may consist of thousands of pairs of voice input types and corresponding intents provided by the organization for which the dialogue system is configured.

[0092] At 304, the dialogue system identifies a loss function based on the positive and negative key pairs. A positive key pair (k, k′) is two keys that match the same value. For example, using the example above in Table 1, the keys “How can I open an account?” and “Create a new account for me” match the same value, http: / / xxx.com / open. A negative key pair (k, k″) is two keys that match different values. For example, again referring to Table 1, the keys “How can I open an account?” and “Can I close my account?” match different values.

[0093] In some embodiments, many of the values ​​appear with multiple keys in dataset D. That is, for many values ​​v, there are multiple keys k, k′, where k ≠ k′, and (k, v) ∈ D and (k′, v) ∈ D. The more values ​​with multiple keys, the more data is available and the more accurate the final result. For example, referring to Table 1 above, in D, the keys “How can I open an account?” and “Create me a new account” are both associated with the same value, “http: / / xxx.com / open.” These two keys will serve as a positive training pair in this procedure.

[0094]

number

[0095] Above is the set of positive training pairs for dataset D. The positive training pairs P can be extended into training triples T by adding a randomly chosen negative key k″ to each pair, where the negative key k″ is associated with a different value for the other keys.

[0096]

number

[0097] Alternatively, the training triples may be selected by another approach. The training data D = ((k1, v1),...,(k n ,v n )) consists of key / value pairs defining a set of triples T={(k,k′,k″)}, then:

[0098]

number

[0099] That is, BestKey(k,D) returns the key k' in data D that best matches key k. v = {(k′,v′):(k′,v′)∈D,v=v′} is a subset of D containing only pairs with value v. Then for each (k,v)∈D, T contains triples such that

[0100]

number

[0101] That is, for each key / value pair (k,v) in the training data, form a training triple, where the positive pair consists of (k,k′), and k′=BestKey(k,D v Let \{(k,v)}) be the BestMatch for k in the subset of D of other pairs with value v, the negative pair consists of (k,k″), and k″ = BestKey(k,D\{(k,v)}) be the BestMatch for k in all other pairs in D.

[0102] This procedure finds the best matching key in D. vIn some embodiments, this method of selecting training triples is combined with the method of generating all possible triples. One way to do this is to use one method for a portion (e.g., 50% of the training mini-batches) and the other method for a portion (e.g., 50% of the training mini-batches). This method of selecting training triples is particularly well suited for training probabilistic models, as described below with respect to FIG. 4.

[0103] Once the dialogue system has established the positive and negative key pairs, it may establish a loss function, Loss(), which penalizes incorrect guesses and can be generally expressed as:

[0104]

number

[0105] where θ is a vector of match function parameters to be established. In some embodiments, sentence embedding function parameters are added to θ. By adding the sentence embedding function parameters to θ, they can also be optimized together with the match function parameters. In some embodiments, the embedding function parameters are initialized with values ​​learned from a large external corpus (e.g., using fine-tuning / transfer learning methods).

[0106] An exemplary loss function suitable in this context includes a margin-based loss target.

[0107]

number

[0108] where c′ is the margin, i.e., a positive constant hyperparameter. Another example is a smoothed stochastic version of the margin-based objective that uses a sigmoid function σ().

[0109]

number

[0110] At 306, the dialogue system minimizes a loss function to identify parameters to be used for matching, such that a higher match score is assigned to a match of two different keys with the same associated value than to a match of two different keys with different associated values.

[0111] In some embodiments, the dialogue system optimizes the parameter θ using a loss function.

[0112]

number

[0113] A dialogue system may minimize a loss function with respect to θ. This is a training procedure, and the goal is to adjust the parameters θ so that for each triple (k, k′, k″) in the training triples T, the positive pair (k, k′) has a higher match score than the negative pair (k, k″). This is equivalent to ensuring that:

[0114]

number

[0115] Alternatively or additionally, calculations may be performed to ensure the following:

[0116]

number

[0117] This can make the parameter θ more robust by ensuring a "gap" between positive and negative paired Match() scores. These loss functions can be optimized using methods such as mini-batch stochastic gradient descent (SGD). Regularization techniques (such as dropout) may be useful for more complex Match() functions.

[0118] In some embodiments, multiple iterations may be used to fine-tune parameters. Embeddings in the match function may be recomputed over multiple iterations. For example, to fine-tune the embeddings, the embedding for each key in each training triple (k, k', k") may be recomputed at each iteration. In some cases, it may be sufficient to update the embeddings, for example, every 10 or 100 mini-batches, or not at all.

[0119] As described above with respect to FIG. 2, the dialogue system may incorporate the parameter θ identified at 306 into the match function used to identify the value.

[0120] FIG. 4 is a simplified flowchart 400 illustrating processing performed during a setup phase according to certain further embodiments. The processing shown in FIG. 4 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method illustrated in FIG. 4 and described below is intended to be exemplary and non-limiting. While FIG. 4 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, these steps may be performed in some different order, or some steps may be performed in parallel. In certain embodiments, the processing shown in FIG. 4 may be performed by a dialogue system as described above.

[0121] 4 illustrates a training method that may be suitable for a probabilistic model (e.g., using Equation 2 above). As described above with respect to FIG. 2, the dialog system may perform a training process that results in a set of parameters for the match function before receiving the input verbal utterance at 202. In some embodiments, this involves maximizing a likelihood function of a set of sample utterance and intent pairs to optimize the parameters of the match function, which may be performed using the process shown in FIG. 4.

[0122] The dialogue system obtains training data at 402. The dialogue system may obtain training data substantially as described above with respect to step 302 of Figure 3 .

[0123] At 404, the dialogue system identifies a likelihood function based on the positive and negative key pairs. A likelihood function is a function that outputs a value that increases with the likelihood that a set of keys corresponds to a positive key pair. As a specific example, a cross-validation-style leave-one-out approximation to log-likelihood can be used, which uses D as the set of possible key / value pairs and training data for θ. The following log-likelihood approximation assumes that the match function is symmetric, i.e., Match(k,k′)=Match(k′,k). The likelihood function can be approximated as follows:

[0124]

number

[0125] The log-likelihood sums over all key / value pairs in D. Before calculating the probability that value v is associated with key k, the key / value pair (k, v) can be removed from D to estimate the probability that k matches another key / value pair in D with value v.

[0126] To avoid overflows and underflows (which can occur with exponential functions), the following formula may be used when calculating the log-likelihood:

[0127]

number

[0128] Another example of a likelihood function is the following error-driven learning approximation:

[0129]

number

[0130] This approximation is fairly accurate when the largest value in S is much larger than the sum of all other values ​​in S. If this is the case for the match function values ​​in the log-likelihood, then we can approximate the log-likelihood as follows:

[0131]

number

[0132] There are further optimization opportunities in the error-driven learning approximation. For example, the embedding involved in computing A could be updated irregularly (e.g., once every 10 mini-batches or once per iteration), which should make computing A very fast. The actual training update only requires computing C. C can be computed relatively easily by comparing each training pair (k,v)∈D with the two data pairs identified in the computation of C.

[0133] At 406, the dialogue system maximizes the likelihood function to optimize a parameter θ of the match function, such that a higher match score is assigned to a match of two different keys with the same associated value than to a match of two different keys with different associated values. ℓ(θ) may be maximized using techniques such as stochastic gradient descent (SGD).

[0134]

number

[0135] As described above with respect to FIG. 2, the dialog system may incorporate the parameters identified at 406 into a match function used to identify values.

[0136] The techniques described herein have several advantages. Because key k is not "hardwired" into the Match() function as in a classifier (instead, it is provided to the Match() function as a parameter), the Match() function can evaluate how well a speech input q matches key k even if key k does not appear in the training data used to estimate the Match() function's parameters θ. This means that the set D of keys (example sentences) and values ​​(e.g., intents) can be changed without (re)training the parameters θ. Furthermore, the training data is easy for a non-expert to generate (i.e., multiple questions that should return the same answer) and may already exist in an existing dataset.

[0137] As a specific example of an advantageous application of these techniques, chatbot and digital assistant development tools can be provided that allow developers to add or remove pairs from set D (including adding pairs with new values ​​or removing all pairs for one or more values) (e.g., using the value recognition techniques described herein as a cloud service) and then immediately test and / or deploy the chatbot or digital assistant with the new, updated set D. There is no need to wait for the system to train a new set of parameters θ. For example, a chatbot or digital assistant developer could add the pair (“What's the weather going to be tomorrow?”, FIND_WEATHER) to set D. The system would then immediately recognize the voice input “What's the weather going to be tomorrow?” as belonging to the value “FIND_WEATHER.” In contrast, conventional techniques require frequent retraining as datasets grow. Reducing the amount of additional training required can significantly increase the speed at which value recognition can be performed. Furthermore, testing has demonstrated that the accuracy of value recognition using the matching-based techniques described herein can exceed the accuracy of value recognition using previous approaches (e.g., classification).

[0138] The above infrastructure can be implemented in a variety of different environments, including cloud environments (which may be various types of clouds, including private, public, and hybrid cloud environments), on-premise environments, hybrid environments, etc.

[0139] 5 is a simplified diagram of a distributed system 500 for implementing one embodiment. In the illustrated embodiment, the distributed system 500 includes one or more client computing devices 502, 504, 506, and 508, which are coupled to a server 512 via one or more communication networks 510. The client computing devices 502, 504, 506, and 508 may be configured to run one or more applications.

[0140] In various embodiments, the server 512 may be adapted to run one or more services or software applications that enable value recognition to be performed with reduced training.

[0141] In particular embodiments, server 512 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services (e.g., under a Software as a Service (SaaS) model) to users of client computing devices 502, 504, 506, and / or 508. Users operating client computing devices 502, 504, 506, and / or 508 may then utilize one or more client applications to interact with server 512 to utilize the services provided by these components.

[0142] 5, server 512 may include one or more components 518, 520, and 522 that implement the functions performed by server 512. These components may include software components, hardware components, or a combination thereof that may be executed by one or more processors. It should be understood that a variety of different system configurations are possible that may differ from distributed system 500. The embodiment illustrated in FIG. 5 is therefore an example of a distributed system for implementing the system of the embodiments and is not intended to be limiting.

[0143] A user can use client computing devices 502, 504, 506, and / or 508 in accordance with the teachings of this disclosure to more quickly and accurately determine values ​​based on received voice input. The client devices may provide an interface that allows a user of the client device to interact with the client device. The client devices may also output information to the user via this interface. While FIG. 5 shows only four client computing devices, any number of client computing devices may be supported.

[0144] Client devices may include various types of computing systems (e.g., portable handheld devices, general-purpose computers (e.g., personal computers and laptops), workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc.). These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems (e.g., Google Chrome™ OS)), including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, Palm OS®). Portable handheld devices may include mobile phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays and other devices. The gaming systems may include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Xbox® game consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.), etc. The client devices may be capable of running a variety of different applications (e.g., various Internet-related apps, communication applications (e.g., email applications, short message service (SMS) applications), etc.) and may use a variety of communication protocols.

[0145] Network 510 may be any type of network familiar to those skilled in the art that is capable of supporting data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk®, etc. By way of example only, network 510 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating under the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 suite of protocols, Bluetooth® and / or other wireless protocols), and / or any combination of these and / or other networks.

[0146] Servers 512 may consist of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable configurations and / or combinations. Servers 512 may include one or more virtual machines running a virtual operating system or other computing architectures that include virtualization (such as one or more flexible pools of logical storage devices that may be virtualized to maintain virtual storage devices for the servers). In various embodiments, servers 512 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0147] The computing systems in server 512 may run one or more operating systems, including any of those described above, and any commercially available server operating system. Server 512 may also run any of a variety of other server and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, JAVA servers, database servers, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle Corporation®, Microsoft Corporation®, Sybase®, IBM® (International Business Machines), etc.

[0148] In some implementations, server 512 may include one or more applications for parsing and consolidating data feeds and / or event updates received from users of client computing devices 502, 504, 506, and 508. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter feeds, Facebook updates, or real-time updates received from one or more third-party sources and continuous data streams, which may include real-time events related to sensor data applications, financial stock tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 512 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 502, 504, 506, and 508.

[0149] The distributed system 500 may also include one or more data repositories 514, 516. In particular embodiments, these data repositories may be used to store data and other information. For example, one or more of the data repositories 514, 516 may be used to store information (such as key / value pairs corresponding to speech input types and values). The data repositories 514, 516 may be in a variety of locations. For example, the data repository used by the server 512 may be local to the server 512 or may be remote from the server 512 and communicate with the server 512 via a network-based or dedicated connection. The data repositories 514, 516 may be of different types. In particular embodiments, the data repository used by the server 512 may be a database (e.g., a relational database, such as those provided by Oracle Corporation and other vendors). One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to SQL-formatted commands.

[0150] In particular embodiments, one or more of the data repositories 514, 516 may also be used by an application to store application data. The data repositories used by an application may be of different types (e.g., a key / value store repository, an object store repository, a general storage repository backed by a file system, etc.).

[0151] In certain embodiments, the value-awareness-related functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 6 is a simplified block diagram of a cloud-based system environment in which various value-awareness-related services may be provided as cloud services, according to certain embodiments. In the embodiment shown in FIG. 6, cloud infrastructure system 602 may provide one or more cloud services that may be requested by users using one or more client computing devices 604, 606, and 608. Cloud infrastructure system 602 may comprise one or more computers and / or servers, which may include those described above for server 612. The computers in cloud infrastructure system 602 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or other suitable configurations and / or combinations.

[0152] Network 610 may facilitate the communication and exchange of data between clients 604, 606, and 608 and cloud infrastructure system 602. Network 610 may include one or more networks. These networks may be of the same type or different types. Network 610 may support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0153] The embodiment shown in Figure 6 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in some other embodiments, cloud infrastructure system 602 may have more or fewer components than those shown in Figure 6, may combine two or more components, or may have a different configuration or arrangement of components. For example, while Figure 6 shows three client computing devices, any number of client computing devices may be supported in alternative embodiments.

[0154] The term cloud services is generally used to refer to services made available to users on demand via a communications network (such as the Internet) by a service provider's system (e.g., cloud infrastructure system 602). Generally, in a public cloud environment, the servers and systems that make up the cloud service provider's system are distinct from a customer's own on-premise servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can use these services without having to purchase separate licenses, support, or hardware and software resources for the cloud services offered by the cloud service provider. For example, the cloud service provider's system may host an application, and users can order and use this application on demand via the Internet without having to purchase infrastructure resources to run the application. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services (e.g., middleware services, database services, Java cloud services, etc.) are offered by Oracle Corporation of Redwood Shores, California.

[0155] In particular embodiments, cloud infrastructure system 602 may provide one or more cloud services using a variety of models, including hybrid service models (e.g., Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), etc.) Cloud infrastructure system 602 may include a set of applications, middleware, databases, and other resources that enable the delivery of various cloud services.

[0156] The SaaS model allows applications or software to be delivered to customers as a service over a communications network such as the Internet, without the customer having to purchase hardware or software for the underlying application. For example, the SaaS model may be used to provide customers with access to on-demand applications hosted by cloud infrastructure system 602. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human capital / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, etc.

[0157] The IaaS model is commonly used to provide infrastructure resources (e.g., servers, storage, hardware and networking resources) as a cloud service to customers to provide elastic compute and storage capabilities. Various IaaS services are offered by Oracle Corporation.

[0158] The PaaS model is commonly used to provide platform and environment resources as a service that enables customers to develop, run, and manage applications and services without the need for customers to procure, build, or maintain such resources. Examples of PaaS services offered by Oracle Corporation® include, but are not limited to, Oracle Java Cloud Services (JCS), Oracle Database Cloud Services (DBCS), data management cloud services, and various application development solution services.

[0159] Cloud services are generally provided in an on-demand self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by cloud infrastructure system 602 via a subscription order. Cloud infrastructure system 602 then executes processing to provide the services requested in the customer's subscription order. For example, a customer may provide a set of key / value pairs that the system uses to train a model to determine values ​​on the cloud based on a voice input type received by the customer. Cloud infrastructure system 602 may be configured to provide one or more cloud services.

[0160] Cloud infrastructure system 602 may provide cloud services through a variety of deployment models. In a public cloud model, cloud infrastructure system 602 may be owned by a third-party cloud service provider and cloud services may be offered to any public customer, which may be an individual or a business. In certain other embodiments, under a private cloud model, cloud infrastructure system 602 may be operated within an organization (e.g., within a corporate organization) and services may be offered to customers within that organization. For example, customers may be various departments of a company (such as human resources, payroll, etc.) or individuals within the company. In certain other embodiments, under a community cloud model, cloud infrastructure system 602 and the services offered may be shared by several organizations within an associated community. Various other models (such as hybrids of the above models) may also be used.

[0161] Client computing devices 604, 606, and 608 may be of different types (such as devices 602, 604, 606, and 608 shown in FIG. 6) and may be capable of running one or more client applications. Users may use the client devices to interact with cloud infrastructure system 602 (such as to request services provided by cloud infrastructure system 602). For example, users may use client devices to request value-aware related services described in this disclosure.

[0162] In some embodiments, the processing performed by cloud infrastructure system 602 to provide value recognition-related services may include big data analytics. This analytics may involve using, analyzing, and manipulating large data sets to detect and visualize various trends, behaviors, relationships, etc. within the data. This analytics may be performed by one or more processors, possibly processing the data in parallel or using the data to perform simulations. For example, big data analytics may be performed by cloud infrastructure system 602 for training purposes to identify parameters for a match function. The data used for this analytics may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0163] 6, cloud infrastructure system 602 may include infrastructure resources 630 utilized to facilitate the provision of various cloud services offered by cloud infrastructure system 602. Infrastructure resources 630 may include, for example, processing resources, storage or memory resources, networking resources, etc.

[0164] In particular embodiments, to facilitate efficient provisioning of these resources to support various cloud services offered by cloud infrastructure system 602 to various customers, resources may be bundled into sets of resources or resource modules (also referred to as “pods”). Each resource module or pod may comprise a pre-integrated and optimized combination of one or more types of resources. In particular embodiments, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, a second set of pods may be provisioned for Java services, and so on, which may include a different combination of resources than the pods in the first set of pods. For some services, resources allocated for provisioning of a service may be shared between services.

[0165] Cloud infrastructure system 602 itself may internally use services 632 that are shared by various components of cloud infrastructure system 602 to facilitate provisioning of services by cloud infrastructure system 602. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services to enable cloud support, email services, notification services, file transfer services, etc.

[0166] Cloud infrastructure system 602 may comprise multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in FIG. 6 , the subsystems may include a user interface subsystem 612 that allows users or customers of cloud infrastructure system 602 to interact with cloud infrastructure system 602. User interface subsystem 612 may include a variety of different interfaces, such as a web interface 614, an online store interface 616 (through which cloud services offered by cloud infrastructure system 602 are advertised and available for purchase by consumers), and other interfaces 618. For example, a customer may use a client device to request one or more services offered by cloud infrastructure system 602 (service request 634) using one or more of interfaces 614, 616, and 618. For example, a customer may access an online store to browse cloud services offered by cloud infrastructure system 602 and place a subscription order for one or more services offered by cloud infrastructure system 602 for which the customer wishes to subscribe. The service request may include information identifying the customer and the one or more services for which the customer wishes to subscribe. For example, a customer may place an order to subscribe to value recognition-related services provided by cloud infrastructure system 602. As part of the order, the customer may provide information identifying historical voice input types and values ​​and / or current voice input types.

[0167] In particular embodiments (such as the embodiment shown in FIG. 6 ), cloud infrastructure system 602 may include an order management subsystem (OMS) 620 configured to process new orders. As part of this processing, OMS 620 may be configured to create a customer account if not already done, receive billing and / or billing information from the customer that is used to issue a bill to the customer for providing the customer with the requested services, verify the customer information, confirm the customer's order upon verification, and orchestrate various workflows to prepare the order for provisioning.

[0168] Upon proper authentication, OMS 620 may then invoke Order Provisioning Subsystem (OPS) 624. Order Provisioning Subsystem (OPS) 624 is configured to provision resources (including processing, memory, and networking resources) for the order. Provisioning may include allocating resources for the order and configuring these resources to facilitate the service requested by the customer order. The manner in which resources for the order are provisioned and the type of resources provisioned may depend on the type of cloud service ordered by the customer. For example, according to one workflow, OPS 624 may be configured to determine that a specific cloud service is requested and identify the number of pods that would have been pre-configured for that specific cloud service. The number of pods allocated to the order may depend on the size / amount / level / scope of the requested service. For example, the number of pods allocated may be determined based on the number of users supported by the service, the duration for which the service is requested, etc. The allocated pods may then be customized to the particular requesting customer to provide the requested service.

[0169] Cloud infrastructure system 602 may send a response or notification 644 to the requesting customer indicating when the requested service will be ready for use. In some examples, information (e.g., a link) may be sent to the customer that enables the customer to begin using and utilizing the benefits of the requested service.

[0170] Cloud infrastructure system 602 may provide services to multiple customers. For each customer, cloud infrastructure system 602 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to these orders, and providing the requested services to the customer. Cloud infrastructure system 602 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and system up and down time. This usage information may be used to bill the customer. Billing may occur, for example, on a monthly cycle.

[0171] Cloud infrastructure system 602 may provide services to multiple customers concurrently. Cloud infrastructure system 602 may store information (including, in some cases, proprietary information) of these customers. In particular embodiments, cloud infrastructure system 602 includes an identity management subsystem (IMS) 628 configured to manage customer information and segregate the managed information so that information associated with one customer is not accessible by another customer. IMS 628 may be configured to provide various security-related services, such as identity services (information access management, authentication and authorization services, services for managing customer identities and roles and related functions, etc.).

[0172] 7 illustrates an exemplary computer system 700 that may be used to implement certain embodiments. For example, in some embodiments, the computer system 700 may be used to implement the dialogue system and any of the various servers and computer systems described above. As shown in FIG. 7 , the computer system 700 includes various subsystems, including a processing subsystem 704, which communicates with several other subsystems via a bus subsystem 702. These other subsystems may include a processing acceleration unit 706, an I / O subsystem 708, a storage subsystem 718, and a communication subsystem 724. The storage subsystem 718 may include a non-transitory computer-readable storage medium, including a storage medium 722 and a system memory 710.

[0173] Bus subsystem 702 provides a mechanism for allowing the various components and subsystems of computer system 700 to communicate with each other as intended. While bus subsystem 702 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 702 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus, etc., using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, etc.

[0174] The processing subsystem 704 controls the operation of the computer system 700 and may comprise one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). These processors may include single-core or multi-core processors. The processing resources of the computer system 700 may be organized into one or more processing units 732, 734, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some embodiments, the processing subsystem 704 may include one or more special-purpose coprocessors (such as a graphics processor, a digital signal processor (DSP)), etc.). In some embodiments, some or all of the processing units of the processing subsystem 704 may be implemented using customized circuitry (such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA)).

[0175] In some embodiments, the processing units in the processing subsystem 704 can execute instructions stored in the system memory 710 or on the computer-readable storage medium 722. In various embodiments, the processing units can execute various program or code instructions and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code being executed can be in the system memory 710 and / or on the computer-readable storage medium 722 (including, in some cases, on one or more storage devices). Through suitable programming, the processing subsystem 704 can provide the various functions described above. In examples in which the computer system 700 is running one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0176] In particular embodiments, processing acceleration unit 706 may be optionally provided to perform customized processing to accelerate the overall processing performed by computer system 700 or to offload portions of the processing performed by processing subsystem 704.

[0177] I / O subsystem 708 may include devices and mechanisms for inputting information into computer system 700 and / or outputting information from or through computer system 700. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information into computer system 700. User interface input devices may include, for example, keyboards, pointing devices (such as a mouse or trackball), touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion-sensing and / or gesture-recognition devices (such as a Microsoft Kinect® motion sensor) that allow a user to control and interact with the input device, a Microsoft Xbox® 360 game controller, and devices that provide an interface for receiving input using gestures and verbal commands. The user interface input devices may also include an eye gesture recognition device (such as a Google Glass® blink detector) that detects eye movements from the user (e.g., "blinks" while taking a picture and / or making a menu selection) and translates the eye gestures as input to the input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition sensing device that allows the user to interact with a voice recognition system (e.g., Siri® navigator) via voice commands.

[0178] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, as well as audio / visual devices (such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser range finders, and eye-tracking devices). Additionally, user interface input devices may include, for example, medical imaging input devices (such as computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices). User interface input devices may also include, for example, audio input devices (such as MIDI keyboards, digital musical instruments, etc.).

[0179] In general, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 700 to a user or to another computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays (such as audio output devices). Display subsystems may be cathode ray tubes (CRTs), flat panel devices (such as those using liquid crystal displays (LCDs) or plasma displays), projection devices, touch screens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / visual information (such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems).

[0180] The storage subsystem 718 provides a repository or data store for storing information and data used by the computer system 700. The storage subsystem 718 provides a tangible, non-transitory, computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. The storage subsystem 718 may store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 704, provide the above-described functionality. This software may be executed by one or more processing units of the processing subsystem 704. The storage subsystem 718 may also provide a repository for storing data used in accordance with the teachings of the present disclosure.

[0181] The storage subsystem 718 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 7, the storage subsystem 718 includes a system memory 710 and a computer-readable storage medium 722. The system memory 710 may include several memories, including volatile main random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory, in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS), containing basic routines that help transfer information between elements within the computer system 700, such as during start-up, may typically be stored in ROM. RAM typically contains data and / or program modules currently being operated on and executed by the processing subsystem 704. In some implementations, the system memory 710 may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0182] 7, system memory 710 may load running application programs 712 (which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc.), program data 714, and operating system 716. By way of example, operating system 716 may include various versions of Microsoft Windows, Apple Macintosh, and / or Linux operating systems, various commercially available UNIX or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome OS, etc.), and / or mobile operating systems (iOS, Windows Phone, Android OS, BlackBerry OS, Palm OS operating systems, etc.).

[0183] The computer-readable storage medium 722 may store programming and data structures that provide functionality of some embodiments. The computer-readable medium 722 may store computer-readable instructions, data structures, program modules, and other data for the computer system 700. Software (programs, code modules, instructions) that, when executed by the processing subsystem 704, provide the above-described functionality may be stored in the storage subsystem 718. By way of example, the computer-readable storage medium 722 may include non-volatile memory (such as a hard disk drive, a magnetic disk drive, an optical disk drive (such as a CD-ROM, DVD, or Blu-ray disc), or other optical media). The computer-readable storage medium 722 may include, but is not limited to, a Zip drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disc, a digital video tape, etc. The computer-readable storage medium 722 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROM), SSDs based on volatile memory (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM-based SSDs and flash memory-based SSDs.

[0184] In particular embodiments, storage subsystem 718 may also include a computer-readable storage medium reader 720 that may be further connected to a computer-readable storage medium 722. Reader 720 may be configured to receive and read data from a memory device (disk, flash drive, etc.).

[0185] Additionally, programming and / or data structures providing functionality of some embodiments may be provided by downloading to a computer (eg, computer system 700) over one or more networks.

[0186] In particular embodiments, computer system 700 may support virtualization technologies (including, but not limited to, virtualization of processing and memory resources). For example, computer system 700 may provide support for running one or more virtual machines. In particular embodiments, computer system 700 may execute a program (such as a hypervisor) that facilitates configuration and management of virtual machines. Each virtual machine may be assigned memory, computational (e.g., processors, cores), I / O, and networking resources. Each virtual machine generally operates independently from other virtual machines. A virtual machine generally runs its own operating system, which may be the same as or different from the operating systems run by other virtual machines executed by computer system 700. Thus, in some cases, multiple operating systems may be executed simultaneously by computer system 700.

[0187] The communications subsystem 724 provides an interface to other computer systems and networks. The communications subsystem 724 serves as an interface for sending and receiving data between other systems and the computer system 700. For example, the communications subsystem 724 may enable the computer system 700 to establish a communications channel with one or more client devices over the Internet for sending and receiving information to and from the client devices. For example, the communications subsystem may be used to receive voice input from the client device and send values ​​to the client device in response thereto.

[0188] The communications subsystem 724 may support both wired and / or wireless communications protocols. For example, in particular embodiments, the communications subsystem 724 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technology (such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution)), WiFi (IEEE 802.XX family of standards), or other mobile communications technology, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some embodiments, the communications subsystem 724 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0189] The communications subsystem 724 can send and receive data in a variety of formats. For example, in some embodiments, the communications subsystem 724 can receive incoming communications in the form of structured and / or unstructured data feeds 726, event streams 728, event updates 730, etc., among other formats. For example, the communications subsystem 724 can be configured to receive (or send) data feeds 726 in real time from users of social media networks and / or other communications services (such as Twitter feeds, Facebook updates, web feeds (such as Rich Site Summary (RSS) feeds), and / or real-time updates from one or more third-party sources).

[0190] In particular embodiments, the communications subsystem 724 may be configured to receive data in the form of a continuous data stream, which may include an event stream 728 of real-time events and / or event updates 730, which may be continuous or infinite in nature without a clear end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0191] Communications subsystem 724 may also be configured to communicate data from computer system 700 to other computer systems or networks. This data may be communicated in a variety of different formats (e.g., structured and / or unstructured data feeds 726, event streams 728, event updates 730, etc.) to one or more databases that may be in communication with one or more streaming data source computers coupled to computer system 700.

[0192] Computer system 700 can be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or other data processing system. Due to the ever-changing nature of computers and networks, the description of computer system 700 shown in FIG. 7 is intended only as an example. Many other configurations are possible, having more or fewer components than the system shown in FIG. 7. Based on the disclosure and teachings herein, one of ordinary skill in the art will appreciate other aspects and / or methods for implementing various embodiments.

[0193] While specific embodiments have been described, various modifications, variations, alternative constructions, and equivalents are possible. The embodiments are not limited to operation in any particular data processing environment, but may freely operate in multiple data processing environments. Furthermore, while specific embodiments have been described using a particular sequence of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be limiting. While some flowcharts describe operations as a sequential process, many of these operations may be performed in parallel or simultaneously. The order of operations may also be rearranged. A process may have additional steps not included in the figures. Various features and aspects of the above-described embodiments may be used individually or together.

[0194] Furthermore, while particular embodiments have been described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are possible. Particular embodiments may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor, or on any combination of different processors.

[0195] Although a device, system, component, or module has been described as being configured to perform particular operations or functions, such configuration may be achieved, for example, by designing an electronic circuit to perform the operations, by programming a programmable electronic circuit (such as a microprocessor) to perform the operations (such as executing computer instructions or code), or by a processor or core being programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques (including, but not limited to, conventional techniques for inter-process communication), and different process pairs may use different techniques, and the same process pair may use different techniques at different times.

[0196] Specific details are provided in this disclosure to provide a thorough understanding of the embodiments. However, the embodiments can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the embodiments. This description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of other embodiments. Rather, the above description of the embodiments will provide one skilled in the art with an enabling description for implementing various embodiments. Various changes may be made in the function and arrangement of elements.

[0197] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and alterations may be made therein without departing from the broader spirit and scope of the appended claims. Accordingly, although specific embodiments have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

Claims

1. 1. A computer-implemented method comprising: receiving an input verbal utterance by a dialogue system; the dialogue system generating a vector representation of the input verbal utterance; the dialogue system using a match function to generate a match score for each key / value pair in a plurality of key / value pairs, the match score generated for each key / value pair indicating a degree of matching between a key in the key / value pair and the vector representation of the input verbal utterance, each key in the plurality of key / value pairs being a multidimensional vector representation of a sentence, and the match function configured to generate a higher match score for a match of two different keys having the same associated value than for a match of two different keys having different associated values, the method further comprising: the dialogue system identifying a particular key / value pair from the plurality of key / value pairs having a highest match score based on the match scores generated for the plurality of key / value pairs; the dialogue system outputting a value associated with the particular key / value pair as a value for the input verbal utterance; The input spoken utterance is a first input spoken utterance, the value is a first value, and the match score is a first match score, and the method further comprises: after the dialogue system outputs the first value for the first input verbal utterance, updating the plurality of key / value pairs to include additional key / value pairs with new intents received as input to the dialogue system; receiving a second input spoken utterance by the dialogue system; the dialogue system using the match function to generate a second match score for each key / value pair in the updated plurality of key / value pairs; The match function does not change between generating the first match score and generating the second match score, and the method further comprises: the dialogue system identifying a second specific key / value pair from the updated plurality of key / value pairs having a highest match score; the dialogue system outputting a second value associated with the second particular key / value pair as a value for the input verbal utterance.

2. 10. The method of claim 1 , wherein the match function is trained by training using training data, the training comprising adjusting a set of parameters associated with the match function by minimizing a loss function.

3. 3. The method of claim 1, wherein the match function is trained by training using training data, the training comprising adjusting a set of parameters associated with the match function by maximizing a likelihood function.

4. The method of any one of claims 1 to 3, wherein the values ​​of the plurality of key / value pairs are intentions.

5. the key of the plurality of key / value pairs is a representation of a question; the values ​​of the plurality of key / value pairs being answers corresponding to respective questions; the outputted value associated with the particular key / value pair corresponds to a selected answer; The method of any one of claims 1 to 4, wherein the value is output as a response to the input verbal utterance.

6. The method of any one of claims 1 to 5, wherein the match function comprises one of a bilinear function, a single layer network, a multi-layer network, or a nearest neighbor function.

7. 1. A dialogue system, comprising: one or more processors; and a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform a method according to any one of claims 1 to 6.

8. 1. A training method for training a match function for use in a dialogue system, comprising: training the match function using training data stored in a database, the training data having a plurality of key / value pairs; The training step includes: calculating a match score for a key of a key / value pair among the plurality of key / value pairs against another key of another key / value pair using the match function; adjusting a set of parameters associated with the match function such that in the calculating step, a higher match score is calculated for a match of two different keys that have the same associated value than for a match of two different keys that have different associated values; A training method, wherein each key in the plurality of key / value pairs is a multidimensional vector representation of a sentence containing words derived from the value corresponding to that key, and the match score indicates the degree of matching of the multidimensional vector representations of two keys.

9. A training method for training the match function used in the dialogue system of claim 7, comprising: training the match function using training data stored in a database, the training data having a plurality of key / value pairs; The training step includes: using the match function, the computer calculates a match score for a key of a key / value pair among the plurality of key / value pairs against another key of another key / value pair; adjusting a set of parameters associated with the match function such that in the calculating step, a higher match score is calculated for a match of two different keys that have the same associated value than for a match of two different keys that have different associated values; A training method, wherein each key in the plurality of key / value pairs is a multidimensional vector representation of a sentence containing words derived from the value corresponding to that key, and the match score indicates the degree of matching of the multidimensional vector representations of two keys.

10. 10. The training method of claim 8 or 9, wherein the tuning step comprises adjusting a set of parameters associated with the match function by minimizing a loss function.

11. The training method of any one of claims 8 to 10, wherein the adjusting step comprises adjusting a set of parameters associated with the match function by maximizing a likelihood function.

12. The training method of any one of claims 8 to 11, wherein a value of the plurality of key / value pairs is an intention.

13. the key of the plurality of key / value pairs is a representation of a question; The training method of any one of claims 8 to 12, wherein the values ​​of the plurality of key / value pairs are answers corresponding to respective questions.

14. The training method of any one of claims 8 to 13, wherein the match function comprises one of a bilinear function, a single layer network, a multi-layer network, or a nearest neighbor function.

15. 1. A training apparatus for training a match function for use in a dialogue system, the training apparatus comprising a computer, the computer comprising means for training the match function using training data stored in a database having a plurality of key / value pairs; The means for training the match function comprises: means for calculating a match score for a key of a key / value pair among the plurality of key / value pairs against another key of another key / value pair using the match function; means for adjusting a set of parameters associated with the match function such that the calculating means calculates a higher match score for a match of two different keys that have the same associated value than for a match of two different keys that have different associated values; a training device, wherein each key in the plurality of key / value pairs is a multidimensional vector representation of a sentence containing words derived from the value corresponding to that key, and the match score indicates a degree of matching between the multidimensional vector representations of two keys.

16. A training apparatus having a computer for training the match function used in the dialogue system of claim 7, wherein the computer comprises means for training the match function using training data stored in a database having a plurality of key / value pairs; The means for training the match function comprises: means for calculating a match score for a key of a key / value pair among the plurality of key / value pairs against another key of another key / value pair using the match function; means for adjusting a set of parameters associated with the match function such that the calculating means calculates a higher match score for a match of two different keys that have the same associated value than for a match of two different keys that have different associated values; a training device, wherein each key in the plurality of key / value pairs is a multidimensional vector representation of a sentence containing words derived from the value corresponding to that key, and the match score indicates a degree of matching between the multidimensional vector representations of two keys.

17. 17. A training device according to claim 15 or 16, wherein the means for adjusting is configured to adjust a set of parameters associated with the match function by minimising a loss function.

18. A training device according to any one of claims 15 to 17, wherein the means for adjusting is configured to adjust a set of parameters associated with the match function by maximising a likelihood function.

19. The training device of any one of claims 15 to 18, wherein the values ​​of the plurality of key / value pairs are intentions.

20. the key of the plurality of key / value pairs is a representation of a question; The training device of any one of claims 15 to 19, wherein the values ​​of the plurality of key / value pairs are answers corresponding to respective questions.

21. A training device according to any one of claims 15 to 20, wherein the match function comprises one of a bilinear function, a single layer network, a multi-layer network, or a nearest neighbor function.

22. A program for causing a computer to execute the method according to any one of claims 1 to 6.

23. A program for causing a computer to execute the training method according to any one of claims 8 to 14.

Images