Neural Speech-to-Meaning

Abstract

To provide a unitary system for extracting meanings from utterance voices.SOLUTION: A neural speech-to-meaning system includes: an intent recognizer 401 which is a neural network-based model which captures a voice including utterances, and outputs a signal to call a request for a virtual assistant action; and a variable recognizer 402 which outputs a signal indicating that it has recognized a known enumeration value of variables in the captured voice, and a variable value provided as an argument for an API hit.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] FIELD OF THE INVENTION The present invention is in the field of natural language understanding using neural network machine learning. [Background technology]

[0002] background FIG. 1 shows the architecture of a conventional virtual assistant with a voice interface. The virtual assistant receives captured speech from a source such as a microphone or packets of voice information over a network. The virtual assistant performs automatic speech recognition (ASR) 101 on the captured speech to generate a transcription. Some virtual assistants generate a set of multiple transcription hypotheses, each with a probability score. The virtual assistant in FIG. 1 performs natural language understanding (NLU) 102 on the transcription, or often on the transcription hypotheses. NLU generates intents, which are data structures with information that the virtual assistant can act on to assist the user.

[0003] A virtual assistant makes a determination as to when the user has made a complete expression with a complete intent. We will call that a hit. Some virtual assistants determine that a hit has occurred when the user stops speaking for a sufficient amount of time. This is called end-of-utterance (EOU) detection. Some virtual assistants also, or instead, determine a hit based on the apparent completeness of the expression according to NLU.

[0004] After determining that a hit has occurred, the virtual assistant in FIG. 1 quickly does something 103 that might be useful to a user given the expressed intent. That something could be any type of functionality appropriate for responding to the intent. For example, the virtual assistant might execute a web application programming interface (API) call to a source of data 104 on the Internet, such as by using the HTTP protocol. This might be useful, for example, to look up the weather forecast. As another example, the virtual assistant might send a message with the words included in the intent or perform an action such as physically moving a robot. This functionality could be defined in a software development kit (SDK), where API access is an example of an appropriate function call.

[0005] The virtual assistant in Figure 1 generates a response to the user after doing something. This may include data from API access, confirmation of the action, and any other relevant information. The virtual assistant performs text-to-speech (TTS) 105 to synthesize the speech. The response may include an indication of prosody for the synthesized speech to emphasize certain words and convey a desired voice style. Traditional systems involve many complex techniques in the ASR and NLU components that receive the speech as a group and generate intent and hit signals.

[0006] Figure 2 shows traditional components within an ASR function. The acoustic model 201 receives speech, segments it into frames, and performs digital signal processing (DSP) to convert speech samples within the frames into spectral energy values. Older systems used hidden Markov models (HMMs) on the spectral data. Newer systems use neural network (NN) models 202. Other approaches are possible. The acoustic model generates phoneme sequence hypotheses and an acoustic score for each hypothesis.

[0007] During the tokenization 203 process, ASR for alphabetic languages ​​checks phoneme sequence hypotheses against a pronunciation lexicon 204, which contains one or more pronunciations per word. For full language transcription, the pronunciation lexicon may have over one million words. For application-specific systems, the lexicon may be smaller. Tokenization generates word sequence hypotheses made from sequences of words containing the hypothesized sequences of phonemes in order. Non-alphabetic languages, such as Chinese, assemble phonemes in order into character pronunciations instead of dictionary-based tokenization. The result, whether a character or a word, is a token sequence hypothesis.

[0008] A statistical language model (SLM) 205 calculates a probability score for each token sequence hypothesis using the token sequence hypotheses and the acoustic scores corresponding to the phoneme sequence hypotheses from which the token sequence hypotheses are derived. The probability score is based on how frequently the sequence of tokens is likely to appear in the language and is weighted by the probability of the token sequence according to the acoustic score. In systems such as systems for long-form dictation, the output of the ASR is the transcription hypothesis with the single highest SLM score. In virtual assistants that perform NLU, the ASR can output multiple transcription hypotheses and their corresponding scores, each of which can be processed to calculate the most probable intent. The SLM is trained on a corpus of language use examples. Older SLMs used n-gram models 206, which calculate the probability of a sequence of N tokens. Typically, N is 3 or 4, but can be fewer or more. More modern SLMs use neural network (NN) models, particularly models with long-term recursion, such as long short-term memory (LSTM) NNs. They learn to weight long leading tokens proportionally to their discriminatory value.

[0009] Figure 3 shows the components of a traditional NLU. It receives transcriptions. These can be multiple transcription hypotheses from an ASR or a single transcription, such as one input as text from a keyboard. The transcription hypotheses are compared to a grammar 301. The grammar includes phrasing and slots. A slot is a placeholder for information that can be filled from a list of possible values. For example, the name of a place is useful for a grammar about weather. Time can also be a slot value. A slot value can also be any arbitrary number, such as for a grammar that performs calculations.

[0010] Phrasing within a grammar can be expressed in a variety of specific formats, such as regular expression format or proprietary formats. Figure 3 shows the phrasing for "what's the weather [going to be] in <place>[on] <time>3 shows an example grammar with the phrasing "(What's the weather going to be in <location> [at] <time>)". It is written in a format that considers text within square brackets to be optional, and text within angle brackets to be a slot value. Thus, this example matches the transcription "what's the weather in Chicago on Wednesday" with "what's the weather going to be in Miami next week". Some virtual assistants group grammars based on the domains of knowledge they can address; however, domain grouping 302 is not necessary for proper functioning.

[0011] The grammar generates an intent and a score. For example, the grammar shown in Figure 3 would generate an intent with a weather API universal resource locator (URL) that includes arguments filled with PLACE and TIME slot values. The grammar may also provide a score based on the probability of a particular phrasing. For example, the English sentence "what's the weather going to be in Boston on yesterday" is improperly spoken and somewhat illogical because it uses the future tense marker "going to be" with the past tense time "yesterday." As a result, the grammar may provide a low score for the sentence. On the other hand, "what's the weather in Seattle currently" would match the grammar, with a meaningful intent and a high grammar score.

[0012] The virtual assistant NLU component selects the intent with the highest grammar score (303). The voice virtual assistant selects the intent with the highest grammar score as weighted by the SLM score weighted by the acoustic score. Other score components are possible in various virtual assistant designs.

[0013] Some virtual assistants store conversation state 304. Among other information, this includes slot values. As a result, a query such as "what's the weather in Seattle" stores Seattle as the value of the PLACE slot. A query using the pronoun "there," such as "how long would it take to drive there," fills the PLACE slot with the value Seattle because that was the most recently mentioned location. As a result, such virtual assistants can understand speech expressed in a natural style. Summary of the Invention [Problem to be solved by the invention]

[0014] problem Traditional systems have many components. Some components are designed by smart engineers using the best available research. While they are constantly improving, they are never perfect. Some components of traditional systems are trained from large corpora of data. The amount of data is finite, and the corpus used to train a system never perfectly matches the system's future use. Every component of the system has inaccuracies, which result in the loss of useful information throughout the pipeline. Much effort is put into improving each stage individually, and much effort is put into making the stages interact well with each other. However, these efforts need to be different and therefore replicated for each human language based on its characteristics. Furthermore, they need to be done differently based on the application, such as applications with fewer or more domains, larger or smaller vocabularies, power-sensitive or latency-critical applications, and other application-specific constraints.

[0015] Acoustic data is expensive. Text word statistics do not perfectly match speech. Corpora for domain-specific language models are difficult / impossible / expensive. Domain grammars require programming skills. The more powerful the grammar language, the more advanced skills are required. The more domain grammars a system has, the more fine-tuning is required to resolve ambiguities in natural human speech.

[0016] The salaries of engineers are astronomically expensive to spend on such tweaks. Prior art has demonstrated acceptable accuracy for "end-to-end" transcription without AM or LM. However, text is not the end for a virtual assistant. The end is an actionable representation of the voice intent, such as what an NLU grammar produces. [Means for solving the problem]

[0017] overview The present invention provides a unitary system for extracting meaning from spoken audio. The input is digitized speech. Some systems capture time-domain representations, some frequency-domain representations. The output is a data structure representing the intent. The data structure is in a format that provides the input needed for a function to perform an action or read a data value with little or no further processing. An optional output is a hit signal indicating that the speaker has completed an expression.

[0018] Some systems include a primary intent recognizer and one or more variable recognizers. An intent recognizer detects the completion of an expression with a certain intent. Variable recognizers are useful for systems that support complex expressions with entity values ​​such as place names or numbers. A variable recognizer takes in speech and outputs an indication of which enumerated values ​​of variables were recognized. A variable recognizer optionally outputs a signal indicating that an enumerated value of a variable was detected. Variable values ​​are useful in many systems as arguments for intent data structure output. Some intent recognizers may use the detection of an enumerated value of a variable as input to signal the detection of a complete intent.

[0019] Some systems may have multiple or many intent recognizers and multiple or many variable recognizers: the output of some types of variable recognizers can inform multiple intent recognizers, and some intent recognizers can observe the output of multiple variable recognizers.

[0020] Some systems treat a group of related intent recognizers as a domain, and such systems may optionally include a domain recognizer that takes in speech and outputs an indication of which of multiple domains the speech refers to.

[0021] The recognizer may be implemented based on a neural network trained on speech to generate hit and variable value outputs. The intent recognizer may be trained on multiple phrasings of the same intent, including phrasings in any number of human languages. To reduce data collection time and costs, the variable recognizer may be trained on generated speech.

[0022] Some systems offer recognizers for natural language understanding as a service through their platforms, either alone or as part of a virtual assistant capability.

[0023] Interaction between devices incorporating the speech and recognizer services may be via network connections, including wireless connections. Many systems are implemented using software instructions stored on a storage medium that, when executed by a processor, causes it to perform the methods described and claimed herein and function as a system. [Brief explanation of the drawings]

[0024] DESCRIPTION OF THE DRAWINGS [Figure 1] FIG. 1 is a diagram of a voice control system. [Figure 2] FIG. 1 is a diagram of the speech-to-text process. [Figure 3] FIG. 1 is a diagram of the natural language understanding process. [Figure 4] A black-box diagram of neural speech-to-meaning. [Figure 5A] FIG. 2 is a diagram showing an input voice waveform. [Figure 5B] FIG. 1 shows an input audio spectrogram. [Figure 6] FIG. 1 is a diagram of some basic recognizer elements. [Figure 7] FIG. 2 illustrates an example of an instance of several types of recognizers. [Figure 8] FIG. 1 illustrates an example of a neural network architecture. [Figure 9] 1 is a diagram of translation through the learned semantic space. [Figure 10] FIG. 1 illustrates an exemplary generative neural network for generating speech sounds. [Figure 11] FIG. 1 is a diagram of an exemplary platform ecosystem for a voice control system. [Figure 12A] FIG. 1 illustrates a rotating non-transitory computer readable medium. [Figure 12B] FIG. 1 illustrates a solid-state non-transitory computer-readable medium. [Figure 13A] A diagram showing a packaged system-on-chip device. [Figure 13B] FIG. 1 is a high-level functional block diagram of an exemplary system-on-chip. [Figure 14A] FIG. 1 illustrates a server blade system. [Figure 14B] FIG. 1 is a high-level functional block diagram of an exemplary server architecture. DETAILED DESCRIPTION OF THE INVENTION

[0025] Detailed Description The following describes various embodiments of the claimed invention. The described embodiments have various components. Some components have alternative design choices. Some components are optional. The following description is not necessarily explicit about whether each component is optional, whether the design choice has alternatives, and what all possible alternatives are. The following description provides examples from which one skilled in the art will understand the described invention and claims. The following description uses the word system to describe machines and methods, and at different levels so that one system can be a component of another system.

[0026] Intent An intent is a request that has a specific expected type of response. Many verbal phrasings can have the same intent. For example, "what's the time" and "what time is it" are two phrasings that have the same intent. A system can represent an intent as structured data. Structured data can be represented with a specific schema using a format such as JSON or XML. In a system that functions using API calls, an intent can be represented as (a) a set of arguments for the API request and (b) possibly a selection of what information from the API response is of interest to the user.

[0027] Generally, as a system's capabilities become more detailed, there will be fewer phrasings per intent. A time-telling system that provides only one type of response will provide its response for the phrasings "what time is it," "what hour is it," and "what minute is it." However, a time-telling system capable of reporting the full time, only the hour, or only the minute will have three intents, each of which responds to only one of those phrasings.

[0028] Some systems may respond to many intents that are the same but relate to one or a few pieces of variable information. For example, a system that can report the weather in any of many different cities may have many different intents. However, due to practical limitations on collecting data and training the system, it is more efficient to design it with one intent that includes variables, where the variables can have any of many specific values.

[0029] A general intent data structure is one that has placeholders for variables, if the intent has variables. A specific intent data structure is one that specifies specific values ​​for each variable.

[0030] API Example Many virtual assistants find information of interest to the user by accessing ("hitting") an API. Many modern APIs use a Representational State Transfer (REST) ​​style. This is typically done programmatically, such as through a command line URL (curl), or through a web framework such as Flask.

[0031] Here is an example request to an API to get weather information: The request has a URL, which has an argument q with a value of London.

[0032]

number

[0033] This is an example response to an API hit with the URL above.

[0034]

number

[0035] This response contains more information about the weather than a user would normally want, and the information is formatted differently. Thus, a useful virtual assistant responding to the user's question intent would provide a resulting verbal user response such as "the weather in London is 14 degrees Celsius with clear skies." Note that 286.67 degrees Kelvin is 13.52 degrees Celsius.

[0036] Many API hits involve accessing a data source through a network. Generally, the data is on a server, and the virtual assistant that executes the API hit is the client. The device containing the virtual assistant may include network client hardware and / or software functionality.

[0037] Outputs and Inputs Figure 4 shows an overview of the inputs and outputs of a neural speech-to-meaning system 400. One output is an intent data structure of a similar format to that described above. This output may effectively continuously change over time as the system operates. It may also register at a specific time when the system has assumed the user expression is complete. In such cases, there may be an additional signal indicating that a hit has occurred. The hit signal may change continuously, or it may emit a pulse when it has hypothesized a complete expression.

[0038] The input to Neural Speech-to-Meaning is speech data. It is possible to capture raw time-domain speech samples. Figure 5A shows the time-domain waveform of a speech sample.

[0039] Traditional ASR involves a first digital signal processing step that converts the time-domain samples into a spectral representation. Figure 5B shows a spectrogram of the speech ("never touch a snake with your bare hands"), showing the frequency content of the speech on the vertical axis as it changes from frame to frame over time on the horizontal axis.

[0040] Many conventional ASR systems compute Mel-frequency Cepstral Coefficients (MFCCs) over roughly 25 ms windows of speech samples at a frame rate of roughly 10 ms (100 frames per second is slightly faster than the fastest rate of human phoneme production expected to be recognized). Because the Mel scale is well designed for average human speech, little useful information is lost by converting raw time-domain speech samples to MFCC coefficients. However, neural networks excel at making inferences from raw data, and MFCC coefficient computation consumes unnecessary computational performance.

[0041] Two approaches to learning are possible. (1) Use windowing. This is an approach described in the paper "Learning the Speech Front-end With Raw Waveform CLDNNs" by Sainath et al. (2015) that attempts to train an acoustic model directly from the raw waveform and replaces windowing / FFT / log-mel filter banks with convolutional layers. However, this trained speech front-end still incorporates speech windows of size 25–50 ms with 10 ms shifts. This ends up training a filter bank similar to those traditionally used in speech recognition, which suggests that it is (near) optimal feature input. This kind of architecture is relatively common for end-to-end speech recognition systems, such as those described in the paper "End-to-End Speech Recognition From the Raw Waveform" by Zeghidour et al. (2018).

[0042] It is also possible to use non-overlapping windows. Tuske et al. describe such an approach using DNNs in their paper "Acoustic Modeling with Deep Neural Networks Using Raw Time Signal for LVCSR" (2014). The DNN can learn bandpass filters (again showing that melfb / gammatone looks pretty close to ideal) and achieve similar performance with sufficient training data as when using front-end features or raw non-overlapping input windows.

[0043] (2) Feed one sample at a time into the recurrent network. This approach does not require windowing. Acceptable performance requires having enough context to make an informed decision. For example, some vowels can be 200 ms long. At a sampling rate of 16 kHz, this requires storage for 3200 samples. This can be addressed in various ways, such as using time dilation in an RNN, such as that described by Chang et al. in their paper "Dilated Recurrent Neural Networks" (2017), or in a CNN, such as that used in the Google Wavenet speech generation system.

[0044] Improved performance is possible using a combination of the two approaches described above. Using context-independent targets can help by avoiding the need to have a long enough memory to see the previous and next phonemes.

[0045] Basic System Figure 6 shows a high-level diagram of the components of a simple neural speech-to-meaning system. It includes an intent recognizer 401 and a variable recognizer 402. The intent recognizer is a neural network-based model that takes in audio, which may include speech, and outputs a signal to invoke a request for a virtual assistant action. When the probability that the speech has an intent reaches a threshold, the intent recognizer issues a request. The threshold level may be fixed by the system design or may depend on the application, environment, time, or other contextual information.

[0046] The request can be a pulse or a continuously varying level. It can be programmed with a kind of hysteresis effect or hold-off timer so that the intent recognizer does not issue a second request until at least the minimum amount of time after issuing the request has passed before the user has made a second expression with the intent. Hitting a web API by making and sending a request over the internet is an example of a virtual assistant action. Sending a signal within a device to have it control a motor is another example.

[0047] Intent recognizers are trained on multiple voices for multiple phrasings of an intent, potentially in multiple human languages. A single-user system can be trained by having the user speak each phrasing several times. A system to support any voice may be functional if trained on 50 diverse voices, but a production-quality system should be trained on thousands of different voices.

[0048] In some systems, the intent recognizer runs continuously on speech, invoking an API hit each time an intent is recognized. Alternatively, the intent recognizer runs only after the phrase spotter detects a wake phrase and conditions the signaling of an API hit on an end-of-speech detection, such as detecting a period of silence for a specified duration.

[0049] The variable recognizer 402 takes in the audio and outputs a signal indicating that it recognizes a known enumerated value for a variable in the audio. The value of the variable provides the argument for the API hit.

[0050] The variable detection signal and the value of the variable may be used by the intent recognizer as inputs to inform intent detection. This is useful, for example, for receiving speech that says "I don't know the weather in Boston," which would cause the intent recognizer to check the weather because Boston is recognized as a known enumerated value for the city name variable, but for receiving speech that says "I don't know whether to stay or go," which would prevent the recognizer from executing a weather information request in response to the speech because the city name does not immediately precede or follow a key word that sounds similar to "weather." Thus, the output of a request for a virtual assistant action (e.g., an API hit) is conditional on the probability that the spoken speech has an enumerated variable value.

[0051] In some systems, the variable recognizer outputs both a signal that a variable was detected and a timestamp of when it was detected. An intent recognizer may be trained to use the timing of variable recognition relative to the currently captured speech to inform intent detection. Another approach is for a variable-dependent intent recognizer to be trained iteratively, or for an LSTM layer to calculate its intent probability according to time-delayed peaks in the probability values ​​from the variable recognizer. For example, "weather in Boston" and "Boston's weather" trigger a weather intent, even though the key information, "weather" and "Boston," come in different orders. However, "in Boston weather" and "weather's Boston" do not trigger the intent because the relative order of the variable name "Boston" to the timing of "in" or "'s" has been learned as not a hit for the intent.

[0052] Neural speech-to-meaning based virtual assistants typically contain either multiple discrete intent recognizer networks, each with its own API hit trigger, or a global intent recognizer network with multiple outputs, one intent recognizer per API hit trigger.

[0053] The system shown in Figure 6 differs from other "end-to-end" speech-to-text or natural language intent inference models in that there is no point in the system where a human-readable transcription of the speech exists: there is no lexical representation of the speech as input, as output, or that can be observed or extracted from within the system.

[0054] Figure 7 shows the architecture of a virtual assistant built using neural speech-to-meaning recognizers. Three domains are shown: one each for weather, business, and navigation. The weather domain has intent recognizers for current weather 401a, tomorrow's weather 401b, and 10-day forecast 401c. The business domain has a restaurant search intent recognizer 401d, and the navigation domain has a navigation request intent recognizer 401e. Each domain may have other intent recognizers.

[0055] The architecture also includes variable recognizers trained to recognize names of people 402a, names of places 402b, streets 402c, and numbers 402d. A virtual assistant may have other types of variable recognizers. A recognizer for people's names would be useful for a virtual assistant that supports sending messages to people. A variable recognizer for city names would be useful, along with appropriate intent recognizers for answering queries about the weather in a particular city at a particular time, for searching for restaurants, and for performing navigation to a particular location. A variable recognizer for numbers would also be useful for navigation requests, such as for navigation to a particular address.

[0056] The virtual assistant architecture of FIG. 7 also includes a domain recognizer 403, which is trained on speech recordings labeled by speech domain. Training data is easy to collect and label for the domain recognizer, which can provide scores useful for clarifying or weighting hit hypothesis scores in various domain-specific intent recognizers.

[0057] Some voice-based virtual assistant devices that perform intent recognition from input speech also provide responses to the user in the form of TTS synthesized speech. Some mobile devices and direct user devices include a local TTS engine that can generate speech from text received from a server. Some systems use a speech synthesis engine to perform speech synthesis for a server and provide speech output as digital audio transmitted over a network.

[0058] Conditional Calls For power-sensitive or performance-limited systems such as mobile devices and data centers, it can be beneficial to avoid running the recognizer unnecessarily. Thus, the recognizer can calculate a probability score and, depending on the score, invoke others as needed.

[0059] For example, an intent recognizer may operate independently of a variable recognizer at steady state, and when its score reaches a threshold, the intent recognizer may invoke one or more variable recognizers for variables potentially present in the intent.

[0060] For example, a domain recognizer may operate continuously, calculating a score for each of many possible domains or distinct intents. When the score reaches a threshold, the domain recognizer may invoke one or more intent recognizers. The domain recognizer may also invoke variable recognizers as appropriate.

[0061] For example, a variable recognizer may run continuously and calculate a probability score. When the score reaches a threshold, the variable recognizer may invoke one or more intent recognizers, which may depend on the variable. A variable recognizer may invoke an intent recognizer based on which of multiple enumerated variables is recognized. This is useful to avoid invoking an intent recognizer that does not understand all variable values. It may also be useful to enable training of variable recognizers that operate across multiple variable types. For example, a single variable recognizer may be trained for point of interest name type variables and street name type variables for London. An end of day intent would be invoked only for point of interest values, not street name values. Separate variable recognizers would be trained for points of interest and street names for Paris. Similarly, variable recognizers for multiple variable types may be trained for different spoken languages ​​used by a particular user or used in locations where a particular product is sold.

[0062] System construction Below are the steps involved in creating and enhancing a neural speech-to-meaning capable system.

[0063] Adding Intents - To add an intent to a system, designers may brainstorm phrasings for the intent, put those phrasings into the system in a random order, and obtain diverse speech data from readers expressing the intent. One public example of such a system is Mozilla Common Voice, in which participants read sentences aloud so that their phone or computer captures their voice. Such a system can be useful for collecting a diverse range of voices speaking different phrasings of a new intent.

[0064] Creating a variable recognizer - For each type of variable in the intent, you need to have a variable recognizer that can recognize all supported values ​​of the variable (for example, city names). You can train a variable recognizer from aligned transcribed audio recordings that cover all variable values. You can also collect recordings of specific variable values ​​through an audio recording collection system.

[0065] A new intent may use an existing variable recognizer of an appropriate type. However, if the intent can recognize values ​​for variables that are not recognized by the variable recognizer, it may be appropriate to collect new audio recordings for the new variable values. The audio collection system can do this by constructing sentences for the voice provider to read, replacing the values ​​of variable words in the sentence with values ​​for variables that require more audio data to improve training accuracy.

[0066] Variable recognizers can be trained independently of intent recognizers, but may achieve better performance if trained in the context of their use in the actual intent expressed, because people speak only words differently than in context. Intent recognizers can be trained independently using recordings of intent phrasing. The intent recognizer will learn to ignore information in speech recordings that express variable values ​​as long as it varies significantly from recording to recording. It may also be good to zero out or add random noise to speech segments with specific variable values. Joint training of intent recognizers and their supporting variable recognizers can achieve slightly faster training to the desired accuracy.

[0067] Once in operation, user spoken queries can be captured and used to retrain the recognizer, thereby improving future accuracy. One possibility is to have a human data labeler listen to the spoken query, view the response, and simply indicate a true or false signal as to whether the response was appropriate for the query. Another possibility is for the human labeler to listen to the spoken query and simply indicate which of a known set of domains the query addresses. Another possibility is for the human labeler to listen to the spoken query while viewing a display showing what APIs were hit with what arguments, indicate which were incorrect, and for those that were incorrect, indicate what the correct API hits or argument values ​​should be.

[0068] Another possibility is for the system to have a special intent recognizer trained to recognize indications that a previous virtual assistant response was dissatisfying. Words such as "no" or "I meant," or an annoyed tone of voice, could be such indicators. The dissatisfaction recognizer could be used to flag previous expressions as possibly incorrect and thus give them priority for human labeling or other automated procedures for labeling or refining the training data.

[0069] Creating a neural speech-to-meaning virtual assistant system is very difficult to start with, but gets better and easier as the system is used and as data is collected from its use.

[0070] Global and Hybrid Recognizers One possible approach is to have discrete domain-specific recognizers, which is useful for building modular, configurable virtual assistants. For example, a platform can support configuring a first virtual assistant for cars that supports the car control and weather domains but not the cooking domain, and a second virtual assistant for smart speakers that supports the weather and cooking domains but not the car control domain. Discrete recognizers also have the advantage of allowing developers to independently add, remove, and upgrade intent recognition without retraining the global model. It requires that each intent output a probability score that is normalized across intents and a final selection stage to select the intent with the highest probability for the natural language expression.

[0071] Another approach is to train a global cross-domain intent recognizer, which is useful for minimizing redundancy in application-specific systems. A global intent recognizer has the advantage that it is trained on all possible intents and therefore automatically selects the most probable intent.

[0072] Whether using a global or discrete approach, the intent recognizer has an output that triggers a reaction to the intent, which performs the function of an end-of-utterance detector in a traditional question-answering virtual assistant.

[0073] A hybrid approach is possible where the global intent recognizer is retrained by keeping the low-level features fixed and training the high-level features for new or improved intents, without backpropagating the retained low-level features to weights and biases.

[0074] Some approaches to implementing neural speech-to-meaning recognizers are CNN-LSTM-DNN or sequence-to-sequence or RNN transducer models with attention. Figure 8 shows an example with four layers that can be used for an intent recognizer. More or fewer layers are possible. It has the lowest input layer, a convolutional layer operating on frames or spectrograms of speech samples. It calculates a set of probabilities for Layer 1 features. The second layer is a recurrent layer. Recurrent nodes are indicated by double circles. The recurrent nodes may be of the long short-term memory (LSTM) type. Layer 2 features are input to a smaller third layer, which may also be a recurrent layer or an LSTM layer. Layer 3 features are used by a feedforward layer that takes input from an external recognizer, indicated by a circle with an X. The external recognizer may be from one or more variable recognizers and / or domain recognizers. In such an architecture, when used as an intent recognizer, the combination of the top-layer nodes produces a final output indicating that an API hit should occur.

[0075] This table shows what might be a good analogy in scientific terms and in terms of traditional ASR-NLU systems for each level of feature complexity for one embodiment of a four-layer neural speech-to-meaning network.

[0076] [Table 1]

[0077] Independent Variable Recognizer An architecture similar to that in Figure 8 can be used in a variable recognizer. A convolutional input layer, a recurrent layer, and several deep feedforward layers are appropriate. When used as a variable recognizer, there may be many (potentially thousands) output nodes, each representing an enumerated variable value. There may also be a final SoftMax layer that allows downstream systems to identify the single most probable variable value.

[0078] Some systems operate variable recognizers sequentially. They use multi-task learning, with lower layers recognizing common features such as speech formants and discarding irrelevant information such as gender- and age-specific acoustic information. Higher layers will tend to be those that learn to distinguish between specific variable values.

[0079] However, it is possible to train a single recognizer for all variables, which has the advantage of avoiding duplication of computations for low-level features. A configurable and upgradable system requires frequent retraining of variable recognizers as new values ​​become known, such as the names of new places or new celebrities. Some systems classify variable recognizers by type, such as place names, people's names, and business names. Some systems classify variable recognizers by geographic location, such as North America and Europe. Some systems classify variable recognizers by language, such as Chinese (People's Republic of China), Japanese, and English. Some systems classify variable recognizers by application, such as automotive, smart speakers, and retail.

[0080] Variable recognizers for people's common names are useful for intents that require access to a user's personal contact list. An API that handles such an intent needs to receive the recognized name, perform a phonetic match to a set of multiple known pronunciations of the name in the contact list, and return an error signal if the recognized name does not match a name in the address book. In some systems, separate variable recognizers for first and last names are appropriate.

[0081] Jointly trained variable recognizers It is possible to jointly train an intent recognizer and a variable recognizer, as in the paper by Bing Liu (Proceedings of SIGDIAL2016, pages 22-30, Los Angeles, USA, September 13-15, 2016). However, Liu's approach (a) has less coverage while performing beam search because the output space is exponential in the number of possible variable labels, and (b) does not have knowledge of the global intent while searching for variables.

[0082] The two-pass approach avoids the lack of knowledge of the global intent. The first pass looks a long amount of time to determine the intent / domain of the entire sentence. The second pass is conditional on the intent predicted in the first pass. The second pass predicts values ​​for variables. The two-pass approach has the disadvantage of requiring the first pass to process the complete sentence before the second pass can begin, which creates a large spike in processing power demands to run the second pass to meet real-time requirements.

[0083] The third approach addresses the problems of both of the previous approaches while requiring only one pass. The third approach separates the conditional probabilities of the variable value predictions from the word / acoustic signal as follows:

[0084]

number

[0085] Since the only concern is predicting the variable values ​​for the most probable domain and / or intent, this approach converts the sum in the above formula to an argmax, thereby yielding the most probable (domain, intent, variable) triplet according to the following formula:

[0086]

number

[0087] Assume we have D domains, I mean intents per domain, and S mean variables per intent. The above formula has a complexity of O(DIS), which can be in the thousands. Furthermore, we do this for all features in the input, thereby bringing the total complexity to O(TDIS), where T is the total number of input time steps. To reduce the complexity, we can perform a beam search over the domains / intents by considering a beamwidth of the W most probable intents / domains. This reduces the complexity to O(TWS). For practical cases, W can be in the approximate range of 5 to 20.

[0088] While this approach predicts the variables and their values ​​for the W most probable intents, the model continues to refine its intent probability distribution with each input feature acquisition. At the end of T time steps, the distribution P(Intent|x) is used as a rescoring mechanism to reweight the corresponding variables, thereby aligning the final output with the intent of the entire sentence.

[0089] With this approach, the coverage ratio is W / K T Rather, it is W / I, where W is the beamwidth, I is the number of distinct intents, K is the number of distinct variable labels from Bin, Liu et al. (2016), and T is the total number of input features. I< <K T Therefore, our method has exponentially better coverage than Bin, Liu et al. (2016) while keeping the conditional distributions of the variables computationally tractable.

[0090] ASR recognizer Some domains require arbitrary text, for example, the domain for sending dictated text messages requires a full vocabulary ASR. Text transcription is effectively a specialized variable recognizer.

[0091] Consider training an intent recognizer to send text messages. The system can learn phrasing where the text is separated by an initial "text" and a final "to." For example, it supports "text i love you to my mom."

[0092] However, supporting phrasing such as "text go to the store to my mom," where the word "to" appears in the ASR, benefits from the use of an attention-based model. Here, the model can look at both ends of the input while predicting the label. The labels could be, for example, MESSAGE_BEGIN, MESSAGE_END, TO_PERSON_BEGIN, and TO_PERSON_END. This requires having complete sentences and therefore cannot be done in an iterative manner by looking at words and predicting their labels.

[0093] Domain Recognizer A domain is a collection of intent recognizers, or each trained recognizer with multiple intent outputs, that relate to a common data source or similar topic.

[0094] To reduce inference time processing in systems that support multiple domains, domain recognizers are trained to select between domain-specific intent recognizers and activate only one or a few of the domain-specific intent recognizers that are likely to recognize speech.

[0095] multilingualism The advantage of these approaches is that they are language independent: differences in spoken language are simply differences in the learned phrasing of the intent. Neural translators are essentially encoder-decoders in the embedding space. For a reasonable number of intents, the dimensionality can be much smaller than that of an entire translator between human languages.

[0096] If the intent is essentially a common target language for translation, the system can use neural translation capabilities. Figure 9 shows the learned semantic space. Expressions in any spoken language can be mapped to expressions in another language through the shared semantic space. Similarly, expressions can be mapped to intents as if they were spoken languages.

[0097] This has the further surprising effect of enabling the creation of an inverse network that can generate grammatically correct audible speech in a human language from a set of intent arguments.

[0098] Note that there is already a lot of overlap in variable recognizers across languages ​​due to cross-linguistic borrowing: for example, the words "KFC" and "iPhone" are used in all languages, and lists of names of internationally known musicians are used across languages.

[0099] The intent recognizer can learn the phrasing of any language using existing semantic space language translator models. This is practical because virtual assistants only support a small number of intents compared to all possible sentence meanings of a language. The intent JSON is useful for forming API requests with its arguments and for selecting which API response data values ​​are appropriate to provide to the user.

[0100] Quantifying accuracy Traditional systems measure ASR word error rates. There are no industry-standard metrics for grammar interpretation accuracy. Whether an interpretation is correct can be subjective. A more practical measure of accuracy is overall user satisfaction. It is also the most meaningful way to quantify the accuracy of neural speech-to-meaning. Of course, accuracy increases with the amount of training data, and some data is more useful than others.

[0101] Most systems of intent recognizers recognize fewer possible intents than there are possible combinations of words in the spoken utterance. Most variable recognizers recognize fewer enumerated values ​​than there are possible combinations of words in the spoken utterance. Therefore, the prediction space is much smaller for intent recognizers and variable recognizers, and therefore the absolute accuracy required to achieve satisfactory performance is much lower.

[0102] Measuring the accuracy of an intent recognizer is comparable to measuring the accuracy of a phrase spotter in that it gives false assertions or lacks an actual expression. Using neural speech-to-meaning in a system with a phrase spotter and end-of-utterance detector can further increase accuracy by avoiding hit assertions during non-use times. Furthermore, after a wake phrase, upon end-of-utterance detection, a neural speech-to-meaning based system has a delimited segment of speech known to have a complete expression and operates on the complete expression to determine which intent was expressed.

[0103] Cost of obtaining data One of the biggest challenges to getting a neural speech-to-meaning system up and running and accurate is obtaining training data that is both large enough and diverse enough.

[0104] Launching a system with sufficient accuracy for user acceptance is relatively easy for systems that start with a small number of intents and for which users have reasonable expectations. Examples include a voice-based clock for the blind with one intent, a voice-controlled light bulb with two intents, on and off, or a virtual weather forecaster that supports 10-20 intents, such as a short or long forecast for general weather, temperature, or precipitation for today, tomorrow, or each of the next 10 days.

[0105] A virtual weather forecaster with 20 intents can be trained on approximately 100 carefully chosen phrasings per human language. Although there are 100 / 20 = 5 or more possible phrasings per intent in a human language, using an LSTM model with attention will recognize intents from keywords in various phrasings. For example, training on the phrasings "what is today's high temperature" and "tell me the high temperature for today" will recognize almost all other plausible user phrasings that have the speech segments "high temperature" and "today" in any order.

[0106] Many volunteers would read 100 phrases in exchange for, say, a $10 gift certificate. Quality data sources are available today that cost less than $10 per 100 phrases. To train a 20-intent recognizer with acceptable accuracy, 100-1000 voices are likely sufficient. This means the cost for training data is between $1000 and $10,000.

[0107] Readers viewing a sequence of phrases with identifiably similar intents will tend to use unnatural speech styles to avoid monotony in their task. The training system can randomize text samples for intents for different types of devices, APIs, or contexts to discourage unnatural reading styles. Additionally, the system can randomize variable values ​​within natural sentences to collect speech data for the variables.

[0108] Some methods for collecting audio recordings of "volunteer" readers are via downloadable phone apps, web pages with voice input capabilities such as Mozilla Common Voice, or by placing soundproof recording booths in public locations.

[0109] Some scatterbrained readers may say something other than the text they see. Traditional ASR can be used to check the voice against the text before using the data for training. If the reader's reading accuracy is less than a high percentage, such as 90%, the gift certificate will not be awarded.

[0110] A voice-based system for a diverse range of users needs to be trained on a diverse set of voices. To attract the types of voice-recording volunteers needed to diversify the training set, targeted advertising can be purchased, such as through Facebook or Google AdWords. For example, people of a certain gender are more likely to respond to a general notification of a voice-recording opportunity, so purchasing ads targeted to groups less likely to respond is one way to ensure a diverse set of training data.

[0111] A domain-rich virtual assistant might have 100 domains, each with an average of 100 generic intents, each with an average of 10 effective phrasings in each of 10 primary languages, each of which might require 1,000 voices for good accuracy. That would require about 1 billion recordings. If the average cost per recording is $0.01 per recording, the virtual assistant could be trained for only $10 million.

[0112] Domain-specific assistants can be much cheaper. Monolingual virtual assistants can be much cheaper.

[0113] Data Acquisition Training data can also be obtained from databases of diverse phrasings. Such databases exist as a result of storing speech utterance data from users of existing virtual assistant systems. These recordings are typically statistically relevant for training future neural speech-to-meaning virtual assistants. Such systems typically have speech recordings stored with intent data structures or with the ability to immediately play back intent data structures.

[0114] An ideal data source would include audio recordings where experts have labeled the correct intent for previously producing an incorrect intent. A method to facilitate efficient checking is to create a web-based interface for a human to simultaneously listen to the speech, see the virtual assistant's response, and type or click an indication of whether it is correct. A second-stage web page would see only the data checked as incorrect and provide the audio and a view of several possible correct intents to choose from, with an indication of whether any are incorrect or the option to enter the correct intent if possible.

[0115] From the database of checked, labeled, clean data, the training process can filter the database by intent to provide training data for each intent recognizer independently.

[0116] Compared with acquired clean data, acquired data tends to have more diverse phrasing. As a result, it can be more valuable for accurate model training. However, it can be relatively more expensive because it requires time for experts to check and label it. It also has a cold start problem in that data is only available after the system is in the field and users have used it.

[0117] TTS generation for training data Another method for generating or augmenting training datasets, particularly to overcome the cold start problem, is to use parametric TTS to generate speech sounds for discriminative phrasing using high parameter diversity. This is the least expensive method, but should be used for only a portion of the training dataset; otherwise, training will overfit the TTS synthesizer's attributes. First-generation TTS systems were concatenative; they synthesized speech from recordings of the voice provider. Modern TTS systems use a set of parameters to define the vocal sounds of the generated speech. The parameters are used by models such as generative neural networks to generate speech from text. Parametric TTS systems are becoming more sophisticated, capable of greater variation and realism in the synthesized speech. This allows TTS to generate more training data inexpensively without overfitting the trained model to the TTS's attributes.

[0118] Some domains require variable recognizers that support millions of variable values. This is the case for example with "show me <item>This occurs in applications with an intent to view items available at a Mall-Mart large retail store, supporting phrasing such as "(Show me <items>)" where: <item>(Item) is a variable that can refer to any one of millions of items in a store's inventory. If it is impractical to collect audio recordings of every item name being read, it is possible to train a variable recognizer using a dataset of speech generated by TTS.

[0119] FIG. 10 shows a diagram of a neural TTS generator 1001. It accepts variable-valued text as input. The system may accept words, characters, or phonemes as input. The system may accept a different number of inputs. The TTS generator 1001 uses a neural network with a set of weights to transform the input text and output spoken audio. Some hidden nodes in the neural TTS generator are recurrent nodes.

[0120] When designing a natural-sounding TTS generator, the primary goal is to learn parameter sets that human listeners rate as sounding natural and pleasant. It is not the primary goal for generating data to train a variable recognizer. In summary, it is important to find many diverse parameter sets that produce speech sounds that match the range of speakers who will be using the system.

[0121] One approach to achieving that is to create a corpus of diverse range of voices. The voices may all be saying a particular phrase, such as a wake phrase for a virtual assistant. The voices may all be saying different phrases. Then: 1. Using a speech analysis (e.g., i-vector fingerprinting) algorithm according to relevant features, calculate the centroid value for every audio recording and the range from the centroid that captures all natural speech. It is important that the algorithm ensures not only the presence of human speech characteristics (formant patterns or realistic voiced speech frequencies), but also the absence of non-human speech characteristics (e.g., noise or other artifacts that do not occur in human speech); 2. Use a basic parameter neural generative TTS engine to generate spoken audio for the phrase according to the parameter set; 3. Analyzing the produced speech sound to find its distance from the center of gravity; 4. Discard the parameter set if the generated speech is outside the natural range; 5. If not, calculate the vector from the centroid to the feature value of the generated speech, and save the parameter set and its vector value; 6. Apply a cost function to the selection of the next parameter set that prefers a large distance from the vector of saved parameter sets, but still within a natural range from the centroid; 7. Return to step 1 for many iterations; 8. Repeat the above steps for at least a few different specific phrases that cover a wide range of phonemes and diphones; 9. For each variable value that the variable recognizer needs to recognize, for each saved parameter set, generate a speech utterance as an initial training set for training the variable recognizer.

[0122] One approach to training neural speech-to-meaning from TTS is as follows: 1. Determine a number of words that may be values ​​for variables in the phrasing of the intent; 2. Determining a set of multiple parameters representing the voice of the virtual assistant user; 3. Synthesizing multiple speech segments for multiple words, the segments being synthesized according to multiple sets of parameters; and 4. The synthesized speech segments are used to train a variable recognizer that can calculate the probability of the presence of any of a number of words in the speech.

[0123] Furthermore, an intent recognizer can be trained on segments of speech in a phrasing, and the input to the intent recognizer is the probability output from the variable recognizer. Optionally, to synthesize speech, it is possible to do so with regard to adjacent phonetic information in the context of the variables in the phrasing, i.e., which words / phonemes come before or after the word. The pronunciation of a phoneme is influenced by which other phonemes are spoken immediately before or after it. This happens unintentionally in all languages, but in some languages ​​it is specifically due to linguistic rules; the liaison rules in French are one example.

[0124] It is also possible to do so optionally with regard to stress on variables within the phrasing. Phrases spoken in natural speech, especially commands, often contain stress. The stress may be on variable words to help the listener take in the most relevant information. For example, "how is the weather in AUSTIN( Austin In "how is the weather in Boston," AUSTIN is emphasized to avoid confusion with "how is the weather in Boston." Emphasis may also be on other words that carry important semantic information. For example, "book a flight FROM London TO Hong Kong." from Hong Kong to In "Book a flight," the emphasis is on FROM and TO.

[0125] One approach to determining multiple parameter sets that represent diverse user voices with minimal bias is as follows: 1. Obtain a large number of audio recordings of ordinary people representing a variety of voices; 2. Analyze the recording to compute the recorded speech vector in the speech feature embedding space; 3. Compute the region representing the range of recorded speech vectors in the embedding space; 4. Learn multiple speech synthesis parameter sets using gradient descent according to a loss function. The loss function is 4a. synthesizing a speech segment according to a parameter set in a plurality of parameter sets; 4b. Analyzing the synthesized speech segment to calculate the synthesized speech vector in space; 4c. It is calculated by computing a loss proportional to the clustering of the synthesized speech vectors in space.

[0126] The proportional clustering loss means that we prefer to learn a large number of parameter sets that produce a roughly uniform distribution of vocal sounds, which avoids bias towards certain speaker types and minimizes the amount of data needed to train an accurate model.

[0127] With a fully trained set of speech synthesis parameter sets, segments of speech of one or more enumerated words can be synthesized according to the speech synthesis parameter sets and used to train the variable recognizer, and the training data includes the synthesized segments of speech of the enumerated words. The training data may also include other data samples.

[0128] Embedding spaces are generally learned, but they can also be selected according to parameters that engineers or scientists know to be relevant.

[0129] When generating parameter sets, synthesizing speech, and analyzing it to calculate its embedding vectors, some parameter sets may produce speech that falls outside the range of natural speech. In such cases, it may be appropriate to simply discard such parameter sets.

[0130] To avoid training parameter sets that are too tightly constrained for the dataset of natural speech used to train the range of acceptable speech, it is reasonable to add some margin around the region. Parameter sets that synthesize speech segments that fall within the region containing the margin should not be discarded. They should be used to train a recognizer model that captures anomalous speech.

[0131] Hybrids of the above approaches to generating training data with TTS are possible, as are broadly similar approaches with application-specific variations.

[0132] Platform Infrastructure Some companies offer platforms that provide virtual assistant capabilities for a diverse range of devices. They also provide access to a diverse range of data source domains and action capability domains. As a platform supports more domains, it becomes more useful to device users, and as more device users connect to the platform, it becomes more profitable for third-party domain providers to provide content through the platform. Competing virtual assistant platform providers need to continuously grow to support more domain intents that can access more types of data through more types of devices.

[0133] Such a platform will typically provide built-in variable recognizers for variables that are likely to be used in many domain intents, such as numbers and place names. The platform will also host many more intent recognizers than variable recognizers, because there will typically be one or many intent recognizers per domain, and there may be thousands of third-party domain providers that offer their data and action capability services through the platform. It is also possible to have domain-specific variable recognizers, such as one that recognizes the names of products available at a particular retail store.

[0134] 11 shows an example of a very small platform used by one or more users 1101. The one or more users access the domain by voice command through a device such as a mobile phone 1102, a home device 1103, or an automobile 1104. The device can access the platform 1105 through a network 1106, such as the Internet or a 5G mobile device network. The platform 1105 may be provided by a company separate from the device manufacturer and network provider.

[0135] The device sends a digital audio recording or stream containing the speech to the platform, and the platform provider offers built-in common variable recognizers, such as for place names 1107, restaurant names 1108, addresses 1109, and numbers 1110.

[0136] Some platforms may use data in the user's contact list to build user-specific variable recognizers, such as a recognizer for friend names 1111, trained from recordings of the user's voice and known audio segments of names with the same lexical spelling.

[0137] Some platforms may offer domain-specific variable recognizers provided by the domain provider or trained by the platform provider using speech data for recognizable variable values ​​from the domain provider. Figure 11 does not show the domain-specific variable recognizer.

[0138] The platform 1105 also includes a domain recognizer 1112, which is trained on known or labeled domain-specific audio recordings. For incoming speech, it predicts which domain intent is correct for recognizing a query. The platform may use this to condition intent recognizer activation or assertion acceptance, or to rescore intent recognizer hypotheses. This improves the platform's accuracy and, therefore, the user experience, especially when the platform provides responses to immature intent recognizers for some third-party domains.

[0139] Domain providers provide intent recognizers for each domain. They train recognizer models, possibly in conjunction with platform providers, for the intents their domains can respond to. In the system of FIG. 11 , a weather information domain provider with weather data 1113 enables hits to its API when intent recognizer 1114 asserts a hit. The weather provider API hit receives a place name value from place name variable recognizer 1107. A CURL request to the API only occurs if the place name recognizer asserts a valid place name during the period when the audio that triggers the weather intent recognizer is received. Similarly, a food information provider with food data 1115 receives an API hit triggered by food intent recognizer 1116, which is conditional on a valid restaurant name 1108 from the restaurant name variable recognizer.

[0140] The provider of the smart light bulb 1117 enables the smart light bulb to be controlled by the speech-to-meaning platform when the light intent recognizer 1118 triggers an API hit. The light intent API and its recognizer do not require input from the variable recognizer because the functionality of the light bulb does not require any information other than the ability to turn it on and off and make the light brighter or dimmer.

[0141] A navigation command provider 1119 sends navigation control and information to user devices other than home devices 1103, such as a cell phone 1102 or a car 1104. Navigation domain API access is controlled by a navigation intent recognizer 1120, which requires only one of the restaurant name, address, or friend name variables.

[0142] The domains capable of sending messages 1121 are controlled by API hits triggered by a message intent recognizer 1122, which requires a valid friend name variable as the destination for the message. It also requires message text, which may optionally include numbers. It may also contain any arbitrary text that is recognized as a transcription using common automatic speech recognition functions. It is not shown in Figure 11.

[0143] Developers of client devices such as mobile phones, home devices, and automobiles can, through the platform, choose which domains they can access and, consequently, which intent recognizers should run for queries from their devices. This is useful because, for example, home devices cannot perform navigation and therefore do not need to recognize navigation intents.

[0144] Complementing grammar-based systems It is possible to bootstrap a neural-based approach using existing data from a grammar-based system by analyzing a large database of queries, parsing each one to find its common intents, and generating a list of intents ordered by frequency. Then, replace the variable values ​​with tags (e.g., CITY and NUMBER); Example: "What's the weather in <city>(What's the weather in <city>)" Example: "What's <number>plus <number>(What is <number> plus <number>)"

[0145] Next, for the most frequent common intents, we find the different phrasings that users used to generate the intent and use the query speech for that intent as training data for the intent recognizer.

[0146] For the variable recognizer, we search reliable transcriptions for instances that match the variable values ​​to be trained. We use cleaned and aligned speech segments to train each variable value.

[0147] Queries with reliable grammar interpretations are useful as training data. Queries that are known to be ungrammatical are also useful as negative training examples. As the neural speech-to-meaning recognizer is retrained on more and more data, it becomes able to accurately compute intent hypotheses for phrasing that does not appear in the grammar. This can be used to provide feedback to grammar authors and help improve their grammars. This is particularly useful for quickly adding new understanding capabilities to the system. The grammar developer does their best to anticipate phrasing, and after some time in the field, the system provides feedback to the grammar developer to improve the grammar.

[0148] A hybrid system using multiple weighted grammars corresponding to different intents can calculate intent hypothesis scores from the various grammars according to the weights to infer the most probable intent. At the same time, the system can calculate intent hypothesis scores using a neural model. The final intent decision can be based on a weighted average of the grammar-based score and the neural-based score.

[0149] It is also possible to use a grammar to generate sentences that match the grammar. Applying TTS to such sentences can generate training speech segments as described above. A diverse range of TTS parameters and generated sentence phrasing can create highly accurate neural speech-to-meaning-intent recognizers.

[0150] Conversation Status When a hit is triggered, the neural speech-to-meaning system may store a record of the variable types and their values ​​used in the API hit or other resulting action. There may be multiple stored variable types and values, such as place names, times, people's names, male names, and female names. The collection of such variables and values ​​is known as the conversation state.

[0151] Intent recognizers may be trained on speech containing pronouns or other linguistic indicators that refer to past semantic information. For example, in English, the word "there," such as in "Let's go there" following the previous sentence "Where is the nearest ice cream shop," refers to the aforementioned location. The word "then" refers to the aforementioned time. The words "him," "her," "he," and "she" refer to the aforementioned people.

[0152] Old conversation state information may be discarded some time later, after a person has perhaps forgotten about it during a conversation. Alternatively, conversation state information may be discarded after a certain number of conversational exchanges. Furthermore, whenever a new value of a variable type is recognized, previously stored values ​​for variables of that type must be discarded, because pronouns in human conversation refer only to the most recent value of any variable type.

[0153] The system may also replace the conversation state information with new variable values ​​recognized in the user's speech, as well as values ​​used to provide a response to the user from information in the API response. For example, if the virtual assistant provides a user response saying, "The nearest ice cream shop is just 3 blocks up the street," the system would store a PLACE variable value of "3 blocks up the street."

[0154] Some systems store a single conversation state history per user conversation, so a geography query "What is the population of New York?" followed by a weather query "What's the weather there?" would use the value "New York" for the location slot of the weather query.

[0155] Some systems store conversation state history per domain, so a query history like "Where's the nearest bagel shop?", "How many bagel shops are there in Pocatello?", or "Give me directions there" will tell you how to get to the nearest bagel shop, but not how to get to Pocatello.

[0156] One approach to recognizing when to use information stored in the conversation state rather than variable values ​​is to train a model on conversational speech labeled with pronouns and pointers to referents. While specific referents are not interesting, training to identify pronouns and pointers to arguments in specific intents of previous queries is important for learning features that indicate such lookup requirements. Training can include both specific previous intents and responses from the machine to the user, because pronouns can point to semantic information in responses as well as from previous queries.

[0157] Neural Turing machine approaches are also possible, such as that by Graves (arXiv:1410.5401v2 [cs.NE] 10 December 2014), which can store and retrieve information (probabilistically) in memory as dictated by an attention mechanism.

[0158] Linguistic Complexity The following considerations are not essential for a system to achieve high user satisfaction. The vast majority (99% or more) of queries to virtual assistants are linguistically simple and will not benefit from the following considerations. However, supporting linguistically complex queries is a selling point for some virtual assistants. Below are some difficult cases and how to handle them:

[0159] Negation Words - One approach is to train a variable recognizer for negation words (e.g., "not," "except," "without"). Based on the time of the word corresponding to the negation word immediately before (for some languages, such as English) or immediately after (for some other languages) the negation word relative to the verb feature of the query, the output intent gets a negation indicator in the intent. If supported by the API protocol, the negation indicator may be passed along with the API request. It may also be used to condition or filter the response from the API hit.

[0160] Another approach is to train using data samples with negations. Examples can be synthesized using negations from recorded human speech data or TTS production data.

[0161] Double negatives and mid-sentence corrections - This is rare, so it's not a big concern and the user will understand why the response doesn't make sense.

[0162] Compound queries - It is possible to train a query boundary recognizer from labeled speech or from synthesized single queries. In English, the word "and" will tend to appear as a highly weighted feature for distinguishing compound queries.

[0163] CRM 12A shows an exemplary non-transitory computer-readable medium, which is a rotating magnetic disk. Data centers typically use magnetic disks to store code and data, including instructions for server processors. The magnetic disks store code, including instructions, that, when executed by one or more computers, will cause the computers to perform the steps of the methods described herein. Rotating optical disks and other mechanically moving storage media are possible.

[0164] FIG. 12B illustrates an exemplary non-transitory computer-readable medium that is a flash random access memory (RAM) chip. Data centers typically use flash memory to store code and data for server processors. Mobile devices typically use flash memory to store code and data for processors in system-on-chip devices. Flash RAM chips store code containing instructions that, when executed by one or more computers, cause the computers to perform the method steps described herein. Other non-moving storage media packaged with leads or solder balls are possible.

[0165] Various types of computer-readable media are suitable for storing code including instructions according to various embodiments.

[0166] SoC 13A shows the bottom side of a packaged system-on-chip device 1300 with a ball grid array for surface mount soldering to a printed circuit board. Various package shapes and sizes are possible for various chip implementations. System-on-chip (SoC) devices control many embedded system embodiments such as those described herein.

[0167] FIG. 13B shows a block diagram of a system-on-chip 1300. It includes a multi-core cluster of computer processor (CPU) cores 1301 and a multi-core cluster of graphics processor (GPU) cores 1302. The processor connects through a network-on-chip 1303 to an off-chip dynamic random access memory (DRAM) interface 1304 for volatile storage of programs and data and a flash interface 1305 for non-volatile storage of computer program code in a non-transitory computer-readable medium, typically flash RAM. The SoC 1300 also includes a display interface 1306 for displaying GUIs, as required for different peripherals, and an I / O interface module 1307 for connecting to various I / O interface devices. The I / O interfaces enable, among other things, sensors such as touchscreen sensors, geolocation receivers, microphones, speakers, Bluetooth peripherals, and USB devices such as keyboards and mice. SoC 1300 also includes a network interface 1308 for enabling the processor to access the Internet through a wired or wireless connection, such as WiFi, 3G, 4G long-term evolution (LTE), 5G, and other air interface standard radios, and Ethernet connection hardware. By executing instructions stored in a RAM device through interface 1304 or stored in a flash device through interface 1305, CPU 1301 and GPU 1302 perform the steps of methods as described herein.

[0168] server Figure 14A illustrates a rack-mounted server blade multiprocessor server system 1400 according to some embodiments, which includes multiple networked computer processors that run software in parallel.

[0169] 14B shows a block diagram of a server system 1400. It includes a multi-core cluster of computer processor (CPU) cores 1401 and a multi-core cluster of graphics processor (GPU) cores 1402. The processors connect through board-level interconnects 1403 to random access memory (RAM) devices 1404 for program code and data storage. The server system 1400 also includes a network interface 1408 for enabling the processors to access the Internet. By executing instructions stored in the RAM devices 1404, the CPUs 1401 and GPUs 1402 perform steps of methods as described herein.

[0170] Special phrases The illustrated and described examples use a spoken language. Various embodiments will work similarly for other languages ​​or combinations of languages. The illustrated and described examples use a knowledgeable domain. Various embodiments will work similarly for other domains or combinations of domains.

[0171] Some embodiments are screenless, such as earphones without a display screen. Some embodiments are stationary, such as vending machines. Some embodiments are mobile, such as automobiles. Some embodiments are portable, such as cell phones. Some embodiments are intended for implantation in the human body. Some embodiments include a manual interface, such as a keyboard or touch screen. Some embodiments include a neural interface that uses human thought as a form of natural language expression.

[0172] Some embodiments function by running software on a general-purpose programmable processor (CPU), such as one with an ARM or x86 architecture. Some power-sensitive embodiments, and some embodiments requiring particularly high performance, such as for neural network training, use hardware optimization. Some embodiments use application-customizable processors with configurable instruction sets in specialized systems-on-chips, such as the ARC processor from Synopsys and the Xtensa processor from Cadence. Some embodiments use dedicated hardware blocks baked into field programmable gate arrays (FPGAs). Some embodiments use arrays of graphics processing units (GPUs). Some embodiments use application-specific-integrated circuits (ASICs) with customized logic to provide optimal performance. Some embodiments rely on hardware description language code, such as code written in the Verilog language.

[0173] Some embodiments of the physical machines described and claimed herein are programmable in many variables, the combinations of which provide an essentially infinite variety of operational behaviors. Some embodiments herein are configured by software tools that provide many parameters, the combinations of which provide an essentially infinite variety of physical machine embodiments of the described and claimed invention. Methods of using such software tools to configure hardware description language representations embody the described and claimed invention. A physical machine may embody the machines described and claimed herein, such as a semiconductor chip, a hardware description language representation of the logical or functional behavior of a machine according to the described and claimed invention, and one or more non-transitory computer-readable media arranged to store such hardware description language representations.

[0174] The hardware blocks, custom processor instructions, coprocessors, and hardware accelerators execute neural network processing, or portions of neural network processing algorithms, with particularly high performance and power efficiency, which provides longer battery life for battery-powered devices and reduces heat removal costs in data centers servicing many client devices simultaneously.

[0175] Common clichés Those skilled in the art will recognize many modifications and variations, including any relevant combination of the features disclosed.

[0176] Various embodiments are methods that use either human and machine behavior or a combination thereof. Some embodiments are systems of one or more non-transitory computer-readable media arranged to store such instructions for the methods described herein. Some embodiments are physical devices such as semiconductor chips, hardware description language representations of the logical or functional behavior of such devices, and one or more non-transitory computer-readable media arranged to store such hardware description language representations.

[0177] Statements herein reciting principles, features, and embodiments encompass both structural and functional equivalents thereof.< / number> < / number> < / city> < / item> < / item> < / time> < / place>

Claims

1. 1. A machine for recognizing intent in a speech utterance, said machine comprising: a variable recognizer that processes speech features, calculates the probability that the speech has one of a plurality of enumerated variable values, and outputs the enumerated variable value with the highest probability; An intent recognizer that processes speech features, calculates the probability that the speech has the intent, and generates a request for a virtual assistant action in response to the probability exceeding an intent threshold; a domain recognizer that processes speech features and calculates a probability that the speech refers to a particular domain; the intent recognizer is associated with the particular domain and is invoked in response to the probability that the speech refers to the particular domain exceeding a domain threshold; The intent recognizer invokes one or more variable recognizers for variables potentially present in the intent to calculate the probability that the speech has any of a plurality of enumerated variable values.

2. the variable recognizer indicates the probability that the speech has an enumerated variable value; The machine of claim 1 , wherein the intent recognizer conditions its output of a request for action on the probability that the speech has an enumerated variable value.

3. 3. The machine of claim 2, wherein conditioning is based on a delay in the indication of the probability that the speech utterance has an enumerated variable value.

4. A machine according to any one of claims 1 to 3, wherein no human-readable audio transcription is calculated.

5. further comprising a network client having access to the web API; In response to the intent recognizer generating a request for a virtual assistant action, the network client executes a request to the web API, and the request has as an argument the enumeration variable value output by the variable recognizer. The machine according to any one of claims 1 to 4.

6. further comprising a speech synthesis engine; In response to receiving a response from the web API, the speech synthesis engine synthesizes speech including information from the web API response and outputs the synthesized speech for a user of the virtual assistant. The machine of claim 5.

7. 1. A method for recognizing intent from speech by a computer system, the method comprising: obtaining speech; processing features of the utterance to calculate a probability that the utterance has any of a plurality of enumerated variable values; outputting the enumeration variable value with the highest probability; processing the features of the utterance to calculate a probability that the utterance has the intent; In response to the probability exceeding an intent threshold, outputting a request for a virtual assistant action; processing features of the speech to calculate a probability that the speech refers to a particular domain; the intent is associated with the particular domain, and the calculation of the probability that the utterance has the intent is performed in response to the probability that the utterance refers to the particular domain being above a threshold; 10. The method of claim 1, wherein calculating the probability that the speech has any of a plurality of enumerated variable values ​​comprises invoking one or more variable recognizers for variables potentially present in the intent.

8. The method of claim 7 , wherein the step of outputting a request is conditional on the probability of the variable having the highest probability.

9. The method of claim 8 , wherein conditioning is based on a delay in the indication of the probability that the speech utterance has an enumerated variable value.

10. The method of any one of claims 7 to 9, wherein no human-readable audio transcription is calculated.

11. The step of outputting a request for a virtual assistant action is performed by making a request to a web API, and the request has as an argument the enumeration variable value having the highest probability. The method according to any one of claims 7 to 10.

12. receiving a response from the web API; and synthesizing speech that includes information from the web API response.

13. A program that, when executed by at least one processor of one or more computers, causes the one or more computers to perform the method of any one of claims 7 to 12.

Images