Neural Speech to Meaning

Abstract

The present disclosure relates to neural speech-to-meaning. A neural speech-to-meaning system is trained on speech audio that expresses a specific intent. The system receives the speech audio and generates an indication of when the audio in the speech matches the intent. The intent can include variables, which can have a wide range of values, such as the name of a place. The neural speech-to-meaning system recognizes both enumerated values ​​of the variables and the overall intent. The recognized variable values ​​can be used as arguments for API requests made in response to the recognized intent. Thus, neural speech-to-meaning supports voice virtual assistants that provide services to users based on API hits.

Description

Technical Field

[0001] The present invention is in the field of natural language understanding using neural network machine learning. Background Art

[0002] Figure 1 The architecture of a conventional virtual assistant with a voice interface is shown. The virtual assistant receives captured speech audio from a source such as a microphone or receives packets of audio data over a network. The virtual assistant performs automatic speech recognition (ASR) 101 on the captured speech audio to produce a transcription. Some generate a set of multiple transcription hypotheses, each with a probability score. Figure 1 The virtual assistant performs natural language understanding (NLU) 102 on the transcription, or in multiple instances, on multiple transcription hypotheses. The NLU generates an intent. The intent is a data structure with information that the virtual assistant can act on to assist the user.

[0003] The virtual assistant determines when the user has made a complete utterance with a complete intent. We call this a hit. Some virtual assistants determine a hit occurs when the user stops speaking for a sufficient period of time. This is called end-of-utterance (EOU) detection. Some virtual assistants alternatively or also determine a hit based on the apparent completeness of the utterance according to NLU.

[0004] Figure 1 The virtual assistant, upon determining that a hit has occurred, quickly does something 103 that might be useful to the user given the expressed intent. The something is any kind of functionality suitable for responding to an intent. For example, the virtual assistant can perform a web application programming interface (API) call to a data source 104 on the internet, such as by using the HTTP protocol. This is useful for looking up a weather forecast, for example. For another example, the virtual assistant might perform an action such as sending a message with the words contained in the intent or causing a robot to physically move. The functionality can be defined in a software development kit (SDK), where an API call is an example of an appropriate function call.

[0005] Figure 1After completing a task, the virtual assistant 1 generates a response to the user. This can include data from the API call, confirmation of the action, and any other relevant information. The virtual assistant performs text-to-speech (TTS) 105 to synthesize speech audio. The response can include instructions for the prosody of the synthesized speech to emphasize certain words and convey a desired voice style. Conventional systems incorporate complex technologies in the ASR and NLU components, which together receive the speech audio and generate intent and hit signals.

[0006] Figure 2 The conventional components within the ASR function are shown. The acoustic model 201 receives audio, divides it into frames, and performs digital signal processing (DSP) to convert the audio samples into spectral energy values ​​within the frames. Older systems use hidden Markov models (HMMs) on the spectral data. Newer systems use neural network (NN) models 202. Other approaches are also possible. The acoustic model generates phoneme sequence hypotheses and an acoustic score for each hypothesis.

[0007] During tokenization 203, ASR for alphabetic languages ​​checks phoneme sequence hypotheses against a pronunciation dictionary 204 that includes one or more pronunciations for each word. For a complete language transcription, the pronunciation dictionary may have a million or more words. For application-specific systems, the dictionary can be smaller. Tokenization produces a word sequence hypothesis consisting of a sequence of words that, in order, include a hypothesized series of phonemes. Non-alphabetic languages ​​such as Chinese are not tokenized against a dictionary, but rather the phonemes are assembled in order into the pronunciation of a character. The result, whether a character or a word, is a token sequence hypothesis.

[0008] The statistical language model (SLM) 205 uses token sequence hypotheses and acoustic scores corresponding to the phoneme sequence hypotheses from which the token sequence hypotheses were derived, and calculates a probability score for each token sequence hypothesis. The probability score is based on the frequency with which a sequence of tokens is likely to occur in the language, weighted by the probability of the token sequence according to the acoustic score. In systems such as those for long-form dictation, the output of the ASR is the transcription hypothesis with the highest SLM score. In virtual assistants performing NLU, the ASR can output multiple transcription hypotheses and their corresponding scores, each of which can be processed to calculate the most likely intent. The SLM is trained on a corpus of language usage examples. Older SLMs use an n-gram model 206, which calculates the probability of a sequence of N tokens. Typically, N is 3 or 4, but can be fewer or more. More modern SLMs use NN models, particularly models with long-term recurrence, such as long short-term memory (LSTM) NNs. These models learn to assign weights to long-term priority tokens commensurate with their distinguishing value.

[0009] Figure 3 Components of a conventional NLU are shown. It receives transcriptions. These can be multiple transcription hypotheses from an ASR or a single transcription, for example, taking the form of one input such as text from a keyboard. The transcription hypotheses are compared to a grammar 301. A grammar consists of phrasing and slots. A slot is a placeholder for information that can be filled from a list of possible values. For example, the name of a place is useful for a grammar related to the weather. A time can be a slot value. For example, for a grammar that performs a calculation, a slot value can also be an arbitrary number.

[0010] The terms in the grammar may be expressed in various specific formats, such as a regular expression format or a proprietary format. Figure 3 shows an example grammar with the wording "what's the weather [going to be] in <place>[on] <time>” (“What will the weather be like in <place> [at] <time>”). Write the wording in a format that treats text in square brackets as optional and text in angle brackets as slot values. Thus, this example matches the transcriptions “What will the weather be like in Chicago on Wednesday” and “What will the weather be like in Miami next week.” Some virtual assistants group grammars according to the areas of knowledge they can address. However, domain grouping 302 is not necessary for normal functionality.

[0011] The grammar generates intents and scores. For example, Figure 3 The grammar shown will generate an intent with a weather API universal resource locator (URL) that includes arguments filled with place and time slot values. Grammars can also assign scores based on the probability of specific wording. For example, the English sentence "what's the weather going to be in Boston on yesterday" is an awkward way to speak, as the future tense indicator "will" is used with the past tense time "yesterday" and seems a bit ridiculous. Therefore, the grammar might give this sentence a low score, while "what's the current weather in Seattle" would give this grammar a meaningful intent and a high grammar score.

[0012] The virtual assistant NLU component selects 303 the intent with the highest grammatical score. The voice virtual assistant selects the intent with the highest grammatical score, which is weighted by the SLM score, which is itself weighted by the acoustic score. Other scoring components are also possible in various virtual assistant designs.

[0013] Some virtual assistants store a conversation state 304. This includes, among other information, slot values. Consequently, the query "What's the weather like in Seattle?" stores Seattle as the value for the location slot. Subsequent queries using the pronoun "there" (e.g., "How long does it take to drive there?") populate the location slot with the value Seattle, since Seattle was the most recently mentioned location. Thus, such virtual assistants can understand speech expressed in a natural style.

[0014] question

[0015] Conventional systems have many components. Some are designed by brilliant engineers using the best research findings. They are constantly improving, but never perfect. Some components of conventional systems are trained from large data corpora. However, the amount of data is limited, and the corpus used to train the system will never perfectly match the system's future use. Imperfections in every component of the system lead to the loss of useful information throughout the entire pipeline. Improving each stage individually requires significant effort, as does ensuring that the various stages interact well with each other. However, these efforts must be differentiated and replicated based on the unique characteristics of each human language. Furthermore, they must be tailored to the application, such as those with fewer or more domains, larger or smaller vocabularies, power-sensitive or latency-critical applications, and other application-specific constraints.

[0016] Acoustic data is expensive. Text word statistics don't fully match speech. Corpora for domain-specific language models are difficult / impossible / expensive. Domain grammars require programming skills. The more powerful the grammar, the more advanced the skills required. The more domain grammars a system has, the more fine-tuning it needs to decipher the ambiguity of natural human speech.

[0017] The salaries of engineers involved in this kind of fine-tuning are astronomically expensive.

[0018] Existing technologies have demonstrated acceptable accuracy for end-to-end transcription without automated method (AM) or language modeling (LM). However, text isn't the end point for a virtual assistant. The end point is an executable representation of the speech's intent, such as an NLU grammar product. Summary of the Invention

[0019] The present invention provides a unified system for extracting meaning from spoken audio. The input is digitized spoken audio. Some systems use a time-domain representation, while others use a frequency-domain representation. The output is a data structure representing the intent. This data structure is formatted to provide the input needed for functions to perform actions or read data values ​​with minimal further processing. An optional output is a hit signal, which indicates that the speaker has completed an utterance.

[0020] Some systems include an intent recognizer and one or more variable recognizers. The intent recognizer detects the completion of a statement with a certain type of intent. Variable recognizers are useful for systems that support complex statements with entity values ​​(e.g., place names or numbers). The variable recognizer receives speech audio and outputs an indication of which enumeration value of the variable was recognized. The variable recognizer optionally outputs a signal indicating that the enumeration value of the variable has been detected. The variable value can be used as an independent variable of the intent data structure output in many systems. Some intent recognizers can use the detection of the enumeration value of the variable as input to notify that a complete intent has been detected.

[0021] Some systems may have multiple or many intent recognizers and multiple or many variable recognizers. The output of some types of variable recognizers may inform multiple intent recognizers, while some intent recognizers may observe the output of multiple variable recognizers.

[0022] Some systems view groups of associated intent recognizers as domains.Such systems may optionally include a domain recognizer that receives speech audio and outputs an indication of which of multiple domains the speech relates to.

[0023] The recognizer can be implemented based on a neural network trained on speech audio to produce hit outputs and variable value outputs. The intent recognizer can be trained on multiple phrasings of the same intent, including phrasings in any number of human languages. To reduce data collection time and cost, the variable recognizer can be trained on generated speech audio.

[0024] Some systems provide recognizers for natural language understanding as a service, either through the platform alone or as part of the virtual assistant functionality.

[0025] The interaction between the device capturing the audio and the recognizer service can be carried out through a network connection (including a wireless connection).Many systems are implemented using software instructions stored on a storage medium so that when a processor executes the instructions, it performs the method and behaves as the system described and claimed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A diagram of a voice control system is shown.

[0027] Figure 2 A diagram showing the speech-to-text process.

[0028] Figure 3 A diagram showing the natural language understanding process.

[0029] Figure 4 A black box view of neural speech-to-meaning is shown.

[0030] Figure 5A The input audio waveform is shown.

[0031] Figure 5B An input audio spectrogram is shown.

[0032] Figure 6 A diagram showing some basic identifier elements is shown.

[0033] Figure 7 Examples of instances of certain types of recognizers are shown.

[0034] Figure 8 An example of a neural network architecture is shown.

[0035] Figure 9 A diagram showing translation through the learned semantic space.

[0036] Figure 10 An example generative neural network for producing speech audio is shown.

[0037] Figure 11 A diagram illustrating an example platform ecosystem for a voice control system is shown.

[0038] Figure 12A A rotating non-transitory computer readable medium is shown.

[0039] Figure 12B A solid-state, non-transitory computer readable medium is shown.

[0040] Figure 13A A packaged system-on-chip device is shown.

[0041] Figure 13B A high-level functional block diagram of an example system-on-chip is shown.

[0042] Figure 14A A server blade system is shown.

[0043] Figure 14B A high-level functional block diagram of an example server architecture is shown. DETAILED DESCRIPTION

[0044] Various embodiments of the claimed invention are described below. The described embodiments have various components. Some components have alternative design options. Some components are optional. The following description does not necessarily explicitly indicate whether each component is optional, whether a design option has alternatives, or what all possible alternatives are. The following description provides examples that will help a person of ordinary skill understand the described invention and claims. The following description uses the term "system" to describe machines and methods at different levels, so that one system may be a component of another system.

[0045] intention

[0046] An intent is a request with a specific expected response type. Many spoken phrases can have the same intent. For example, "What time is it now?" and "What time is it?" are two statements with the same intent. Systems can represent intents as structured data. Structured data can be represented using formats such as JSON or XML with a specific schema. In systems that operate using API calls, intents can be represented as (a) a set of arguments for the API request and (b) in some cases, the information the user is interested in selecting from the API response.

[0047] Typically, the more detailed the system's functionality, the fewer phrases each intent has. A time-telling system that only gives one type of response would respond to the phrases: "What time is it?", "What hour?", and "What minute?", but a time-telling system with the ability to report the full time, just the hours, and just the minutes would have three intents, each responding to just one of those phrases.

[0048] Some systems can respond to multiple intents that are identical except for one or a small number of variables. For example, a system that can report the weather in any of many different cities can have many different intents. However, due to practical limitations in collecting data and training the system, it is more efficient to design it with a single intent that includes a variable that can have any of many specific values.

[0049] If an intent has variables, the generic intent data structure is a structure with placeholders for the variables. The specific intent data structure is a structure that specifies specific values ​​for each variable.

[0050] API Examples

[0051] Many virtual assistants retrieve information of interest to the user by accessing ("hitting") an API. Many modern APIs use the Representational State Transfer (REST) ​​style. This is typically implemented programmatically (e.g., command-line URLs (curl)) or via a web framework (e.g., Flask).

[0052] This is an example request to the API for getting weather information. The request has a URL with an argument q whose value is London.

[0053] api.openweathermap.org / data / 2.5 / weather?q=London

[0054] Here is a sample response to an API hit with the above URL.

[0055] Temperature: 286.67

[0056] Minimum temperature: 281.556

[0057] Maximum temperature: 286.67

[0058] Pressure: 972.73

[0059] Humidity: 75

[0060] Description: "Clear sky"

[0061] Wind speed: 1.81

[0062] Wind direction: 247.501

[0063] dt_txt:"2017-02-16 12:00:00"

[0064] City: "London"

[0065] This response includes more information about the weather than the user would typically want, and the information is formatted differently. Therefore, a helpful virtual assistant, responsive to the intent of the user's question, would provide the resulting verbal user response, such as "The weather in London is 14 degrees Celsius with a clear sky." Note that 286.67 degrees Kelvin is 13.52 degrees Celsius.

[0066] Many API calls involve accessing a data source over a network. Generally speaking, the data resides on a server, and the virtual assistant executing the API call is the client. Devices containing virtual assistants may include network client hardware and / or software capabilities.

[0067] Output and Input

[0068] Figure 4 An overview of the inputs and outputs of the neural speech-to-meaning system 400 is shown. The output is an intent data structure, similar in form to that described above. The output may effectively change continuously over time as the system operates. It may also register a specific time at which the system assumes a user utterance is complete. In this case, there may be an additional signal to indicate a hit. The hit signal may change continuously or may pulse when the utterance is assumed to be complete.

[0069] The input of the neural speech-to-meaning method is speech audio data. Raw time-domain audio samples can be obtained. Figure 5A The time domain waveform of the speech sample is shown.

[0070] Conventional ASR includes a first digital signal processing step of converting time domain samples into a spectral representation. Figure 5B Shows a spectrogram of the speech ("never touch a snake with your bare hand"). It shows the frequency components of the speech audio vertically, as they change in time from frame to frame horizontally.

[0071] Many conventional ASR systems compute Mel-frequency Cepstral Coefficients (MFCCs) over approximately 25ms windows of audio samples at a frame rate of approximately 10ms (100 frames per second, slightly faster than the fastest human phoneme production rate expected to be recognized). The mel scale is carefully designed for typical human speech, so little useful information is lost by converting raw time-domain audio samples into MFCC coefficients. However, neural networks excel at inferring from raw data, and computing MFCC coefficients consumes unnecessary computational power.

[0072] There are two possible learning paths.

[0073] (1) Using windowing. This attempts to learn an acoustic model directly from the raw waveform, replacing the windowing / FFT / log-Mel filter bank with convolutional layers, much like the approach described in Sainath et al. (2015). However, this learned speech front-end still uses audio windows of size 25-50ms, with a 10ms offset. In the end, it learns a filter bank similar to the one traditionally used for speech recognition, suggesting that this is an (almost) optimal feature input. This architecture is relatively common in end-to-end speech recognition systems, such as those described in Zeghidour et al. (2018).

[0074] Non-overlapping windows can also be used. Tüske et al. describe this approach using a DNN in their paper "Acoustic Modeling with Deep Neural Networks Using Raw Time Signal for LVCSR" (2014). The DNN learns a bandpass filter (again, a Mel filter bank / gammatone seems very close to ideal), but with sufficient training data, it can achieve similar performance to using front-end features or unprocessed, non-overlapping input windows.

[0075] (2) Send one sample at a time to the recurrent network. This approach does not require windowing. Acceptable performance requires sufficient context to make intelligent decisions. For example, some vowels may be as long as 200 milliseconds. At a sampling rate of 16kHz, storage space for 3200 samples is required. This problem can be solved in various ways, for example, by using time dilation in RNNs as described in Chang et al. in "Dilated Recurrent Neural Networks" (2017) or in CNNs such as the one used in the Google Wavenet audio generation system.

[0076] Combining the two approaches above may improve performance. Using a context-free target may help by avoiding the need for long storage to look up the previous and next phonemes.

[0077] Basic System

[0078] Figure 6 A high-level view of the components of a simple neural speech-to-meaning system is shown. It includes an intent recognizer 401 and a variant recognizer 402. The intent recognizer is a neural network-based model that receives audio that may contain speech and outputs a signal to invoke a request for a virtual assistant action. The intent recognizer issues the request when the probability that the speech audio contains intent reaches a threshold. The threshold level can be fixed according to the system design, or it can depend on the application, environment, time, or other contextual information.

[0079] The request can be a pulse or a continuously varying level. It can be programmed with a hysteresis effect or a delay timer so that after issuing a request, the intent recognizer won't issue a second request until at least the minimum time the user might make a second expression of intent. Hitting a web API by sending a request over the internet is one example of a virtual assistant action. Sending a signal within a device to control an engine is another example.

[0080] The intent recognizer is trained on a large number of utterances representing multiple expressions of intent (possibly in multiple human languages). A single-user system can be trained by having the user speak each expression multiple times. If trained on 50 different utterances, a system that supports any utterance will likely work, but a mass-production quality system should be trained on tens of thousands of different utterances.

[0081] In some systems, intent recognizers run continuously on the audio and call API hits when an intent is recognized. Alternatively, intent recognizers run only after a wakeup phrase is detected by a phrase spotter, and they condition their API hit signals based on end-of-utterance detection (e.g., recognizing a period of non-speech of a specific duration).

[0082] The variable identifier 402 receives audio and outputs a signal indicating that it has identified a known enumerated value for a variable in the audio. The value of the variable provides the argument to the API hit.

[0083] The intent recognizer can use the signal that a variable has been detected and the value of the variable as input to inform intent detection. This can be used, for example, to cause the intent recognizer to look up the weather when receiving audio of "I don't know the weather in Boston" because Boston would be recognized as a known enumerated value for the city name variable, but prevent the recognizer from performing a request for weather information in response to "I don't know whether to stay or go" because no city name appears shortly before or after the key words that sound like "weather." Therefore, the outcome of the request for a virtual assistant action (e.g., an API hit) depends on the probability that the voice audio has the enumerated variable value.

[0084] In some systems, the variable recognizer outputs both a signal indicating the variable has been detected and a timestamp of when the variable was detected. The intent recognizer can be trained to use the timing of variable recognition relative to the currently captured speech to inform intent detection. Another approach is to calculate the intent probability of a variable-dependent intent recognizer, which will be trained using a recursive or LSTM layer, based on the time-delayed peaks in the probability values ​​from the variable recognizer. For example, despite the different ordering of the key information "weather" and "Boston," both "weather in Boston" and "Boston's weather" will trigger a weather intent. However, "in Boston weather" and "weather's Boston" will not trigger an intent because the relative order of the variable name "Boston" compared to the timing of "in" or "'s" is not considered a hit on the intent.

[0085] Neural speech-to-meaning based virtual assistants typically consist of multiple discrete intent recognizer networks (each with an API hit trigger) or a global intent recognizer network with multiple outputs (one intent recognizer per API hit trigger).

[0086] Figure 6 The system shown differs from other "end-to-end" speech-to-text models or natural language intent inference models in that there is no point in the system where a human-readable transcription of the speech exists. There is no lexical representation of the speech as input, as output, or that can be observed or extracted from within the system.

[0087] Figure 7 The architecture of a virtual assistant built with a neural speech-to-meaning recognizer is shown. Three domains are shown, one for weather, commerce, and navigation. The weather domain has an intent recognizer 401a for current weather, another for tomorrow's weather 401b, and another for a 10-day weather forecast 401c. The commerce domain has an intent recognizer 401d for restaurant search, and the navigation domain has an intent recognizer 401e for navigation request. Each domain may have additional intent recognizers.

[0088] The architecture also includes variable recognizers that are trained to recognize the names of people 402a, places 402b, streets 402c, and numbers 402d. The virtual assistant can have other types of variable recognizers. A person name recognizer is useful for supporting a virtual assistant that sends messages to people. A variable recognizer for a city name, combined with an appropriate intent recognizer, can be used to answer queries about the weather in a specific city at a specific time, search for restaurants, and perform navigation to a specific location. A variable recognizer for a number can also be used for navigation requests (e.g., navigating to a specific address).

[0089] Figure 7 The virtual assistant architecture also includes a domain identifier 403. This is trained on speech recordings labeled with the domain of the speech. Training data is easy to collect and label for the domain identifier. It can provide a score that can be used to disambiguate or weight the hypothetical scores of hits from various domain-specific intent recognizers.

[0090] Some voice-based virtual assistant devices perform intent recognition in input voice audio and provide responses to the user in the form of text-to-speech (TTS) synthesized voice audio. Some mobile and direct-user devices include a local TTS engine that can generate speech from text received from a server. Some systems use a speech synthesis engine to perform speech synthesis on the server and provide speech output as digital audio sent over the network.

[0091] Conditional Call

[0092] For power-sensitive or performance-constrained systems (e.g., mobile devices and data centers), it may be beneficial to avoid running the recognizer unnecessarily. Therefore, it is possible for the recognizer to calculate a probability score and call others based on the score as needed.

[0093] For example, the intent recognizer can operate independently of the variable recognizer in a steady state.When its score reaches a threshold, the intent recognizer can call one or more variable recognizers for the variables potentially present in the intent.

[0094] For example, a domain identifier can run continuously and calculate a score for each of many possible domains or individual intents. When the score reaches a threshold, the domain identifier can call one or more intent identifiers. The domain identifier can also call variable identifiers as appropriate.

[0095] For example, a variable identifier can run continuously and calculate a probability score. Once the score reaches a threshold, the variable identifier can call one or more intent identifiers that may depend on the variable. The variable identifier can call the intent identifier based on which variable among multiple enumerated variables is identified. This is useful for avoiding calling intent identifiers that do not understand all variable values. This is also useful for allowing training of variable identifiers that work across multiple variable types. For example, a single variable identifier can be trained for a point of interest name type variable and a street name type variable for London. Closing time of day intents will only be called for point of interest values, not street name values. For Paris, separate variable identifiers for points of interest and street names will be trained. Similarly, variable identifiers for multiple variable types can be trained for different spoken languages ​​used by specific users or in locations where specific products are sold.

[0096] System construction

[0097] Below is the process involved in creating and enhancing a system capable of neural speech-to-meaning conversion.

[0098] Adding intents—To add intents to the system, designers can brainstorm ideas for intent wording and place those wordings into the system in a random order to obtain a diverse set of voice data from readers expressing the intent. An open example of such a system is Mozilla Common Voice, in which participants read sentences aloud so their phones or computers can capture their voices. Such a system can be useful for collecting a diverse set of voices expressing various wordings of new intents.

[0099] Create a variable identifier—For each type of variable in an intent, you need a variable identifier. It must be able to identify all supported values ​​for that variable (e.g., the name of a city). It's possible to train a variable identifier based on aligned transcribed speech recordings covering all variable values. It's also possible to collect recordings of specific variable values ​​through a speech recording collection system.

[0100] New intents can use existing variable recognizers of the appropriate type. However, if an intent recognizes values ​​for a variable that the variable recognizer doesn't, it may be necessary to collect new voice recordings for the new variable values. The voice collection system can achieve this by composing sentences for voice contributors to read and replacing variable words in the sentences with the values ​​of the variables that require more voice data to improve training accuracy.

[0101] The variant recognizer can be trained independently of the intent recognizer, but better performance is achieved if it is trained using the context used in actual intent expressions, because people say different words in isolation than in context. The intent recognizer can be trained independently using recordings of intent phrasings. The intent recognizer will learn to ignore information in audio recordings that indicates the value of a variant, as long as it differs significantly between recordings. It is also beneficial to zero out or add random noise to portions of audio with specific variant values. Jointly training the intent recognizer and its supporting variant recognizer allows for faster training to reach the desired accuracy.

[0102] Once in use, user voice queries can be captured and used to retrain the recognizer, thereby improving future accuracy. One possibility is to have a human data labeler listen to the query audio, view the response, and simply indicate a correct or incorrect signal as to whether the response is appropriate for the query. Another possibility for the human labeler is to listen to the query audio and simply indicate which (if any) of the known domain sets the query addresses. Another possibility for the human labeler is to listen to the query audio while seeing a display that indicates which API was hit and with which arguments, and which arguments were wrong, and for the wrong arguments, what the correct API hit or argument value should be.

[0103] Another possibility is for the system to have a special intent recognizer that is trained to identify indicators that a previous virtual assistant response was unsatisfactory. Words such as "no" or "I meant" or an irritated tone can serve as such indicators. The dissatisfaction recognizer can be used to mark previous statements as potentially incorrect and prioritize them for human labeling or other automated processes used to label or improve training data.

[0104] Creating a neural speech-to-meaning virtual assistant system is difficult to start, but it will get better and easier as the system is used and data is collected from its use.

[0105] Global and hybrid recognizers

[0106] One possible approach is to use separate domain-specific recognizers. This is useful for building modular, configurable virtual assistants. For example, a platform may support configuring a first virtual assistant for a car that supports the car control and weather domains but not the cooking domain, and configuring a second virtual assistant for a smart speaker that supports the weather and cooking domains but not the car control domain. Separate recognizers also have the advantage of enabling developers to independently add, remove, and upgrade intent recognition without retraining the global model. It requires each intent to output a probability score (which will be normalized across intents), and a final selection stage for selecting the most likely intent for the natural language representation.

[0107] Another approach is to train a global, cross-domain intent recognizer. This is useful for minimizing redundancy in application-specific systems. The advantage of a global intent recognizer is that it can be trained for all possible intents, so the most likely intent can be automatically selected.

[0108] In both the global and discrete approaches, the intent recognizer has outputs that trigger a response to the intent. This enables the functionality of an end-of-utterance detector in a general question-answering virtual assistant.

[0109] A hybrid approach is possible where the global intent recognizer is retrained by keeping the low-level features fixed and training high-level features for new or improved intents without backpropagation to the retained low-level feature weights and biases.

[0110] Some approaches to implement neural speech-to-meaning recognizers are CNN-LSTM-DNN or sequence-to-sequence (seq-to-seq) or RNN transformer models including attention. Figure 8 An example is shown that has 4 layers and can be used for an intent recognizer. More or fewer layers are possible. It has a lowest input layer, which is a convolutional layer that operates on a frame of audio samples or its spectrum. It computes a collection of probabilities for the layer 1 features. The second layer is a recurrent layer. The recurrent nodes are shown with double circles. The recurrent can be of the long short-term memory (LSTM) type. The layer 2 features are input to a smaller third layer, which is also recurrent and can also be an LSTM layer. The layer 3 features are used by a feedforward layer, which takes input from an external recognizer, represented by a circle with an X. The external recognition can come from one or more variable recognizers and / or domain recognizers. In such an architecture, when used as an intent recognizer, the combination of the top-level nodes produces a final output that indicates that an API hit should occur.

[0111] For one embodiment of a 4-layer neural speech-to-meaning network, the table shows, for each feature, the level of complexity and possible appropriate analogies in scientific terms and in conventional ASR-NLU system terms.

[0112]

[0113] Independent variable identifier

[0114] and Figure 8 An architecture similar to that in

[15] might be used for a variable identifier. A convolutional input, recursion, and some deep feed-forward layers are suitable. When used as a variable identifier, there may be many (possibly thousands) output nodes, each representing an enumerated variable value. There may also be a final SoftMax layer that enables downstream systems to identify the single most likely variable value.

[0115] Some systems run variable identifiers continuously. They use multi-task learning with the bottom layer identifying common features (e.g., vocal formants) and discarding irrelevant information, such as acoustic information specific to gender and age. Higher layers will tend to be those that learn to distinguish between specific variable values.

[0116] But it is possible to train a single recognizer for all variables, which has the advantage of avoiding repeated computations for low-level features. Configurable and upgradeable systems require frequent retraining of variable recognizers as new values ​​(such as new place names or new celebrity names) become known. Some systems separate variable recognizers by type, such as by place names, person names, and company names. Some systems separate variable recognizers by geographic location (e.g., North America and Europe). Some systems separate variable recognizers by language (e.g., mainland Chinese, Japanese, and English). Some systems separate variable recognizers by application (e.g., cars, smart speakers, and retail businesses).

[0117] Variant recognizers for common person names can be used in intents requesting access to a user's personal contact list. The API handling this intent needs to receive the recognized name, perform a phonetic match against a set of multiple known pronunciations of names in the contact list, and return an error signal if the recognized name does not match any name in the address book. In some systems, separate variant recognizers are used for first and last names.

[0118] Jointly trained variant identifiers

[0119] It is possible to jointly train an intent identifier and a variable identifier, such as described in Bing Liu's paper (Proceedings of SIGDIAL 2016, pp. 22-30, Los Angeles, USA, September 13-15, 2016). However, Liu's approach (a) has less coverage when performing beam search because the output space is exponential in the number of possible variable labels; and (b) does not understand the global intent when determining the variables.

[0120] A 2-pass approach circumvents the lack of global intent knowledge. The first pass focuses on being long enough to determine the intent / scope of the entire sentence. The second pass is conditioned on the intent predicted in the first pass. The second pass predicts the value of (one or more) variables. A disadvantage of the 2-pass approach is that it requires the entire sentence to be processed in the first pass before the second pass can begin, which results in a large processing power demand spike to run the second pass in order to meet real-time requirements.

[0121] The third approach solves the problems of the first two approaches and only requires one pass. The third approach separates the conditional probability of variable value prediction from the word / acoustic signal as follows:

[0122]

[0123] Since the only thing we care about is predicting the most likely domain and / or intent variable value, the pipeline converts the sum in the above equation into an argmax, resulting in the most likely (domain, intent, variable) triplet according to the following equation:

[0124] Domain * , Intent * , VariabIe * =argmax Domain，Intent，variable P(Domain|x)P(Intent|Domain, x)P(variable|Domain, Intent, x)

[0125] Assume we have D domains, an average of I intents per domain, and an average of S variables per intent. The complexity of the equation above is O(DIS), which can be in the thousands. Furthermore, we perform this operation for every feature in the input, bringing the total complexity to O(TDIS), where T is the total number of input time steps. To reduce complexity, we can perform a beam search on domains / intents by considering the beam widths of the W most likely intents / domains. This reduces the complexity to O(TWS). For practical purposes, W can be approximately in the range of 5 to 20.

[0126] While this approach predicts the W most likely intent variables and their values, the model continuously refines its intent probability distribution by ingesting each input feature. At the end of T time steps, the distribution P(Intent|x) is used as a re-scoring mechanism to re-weight the corresponding variables so that the final output is consistent with the intent of the entire sentence. Figure 1 To.

[0127] With this approach, coverage is W / I rather than W / K T , where W is the beamwidth, I is the number of different intents, K is the number of different variable labels from Bing Liu et al. (2016), and T is the total number of input features. Since I < < K T , so our method has exponentially improved coverage compared to Bing Liu et al. (2016) while keeping the conditional distribution of the variables computationally tractable.

[0128] ASR recognizer

[0129] Some fields require arbitrary text. For example, sending command text messages requires full-vocabulary ASR. Text transcription is actually a special type of variable identifier.

[0130] Consider training an intent recognizer to send a text message. The system can learn phrasing where the text is delimited by "text" at the beginning and "to" at the end. For example, this supports "text i love you to my mom."

[0131] However, to support phrasing such as "text go to the store to my mom," where the word "to" appears in the ASR, an attention-based model is used, which is able to look at both ends of the input while predicting the label. The labels could be, for example, MESSAGE_BEGIN, MESSAGE_END, TO_PERSON_BEGIN, and TO_PERSON_END. To do this, you must have the complete sentence, so it's not possible to do it iteratively by looking at the words and predicting their labels.

[0132] Domain Identifier

[0133] A domain is a collection of intent recognizers related to a common data source or similar subject matter, or a single trained recognizer with multiple intent outputs.

[0134] To reduce inference time processing in systems supporting multiple domains, a domain recognizer is trained to select between domain-specific intent recognizers, and only one or a few domain-specific intent recognizers are run that are likely to recognize speech.

[0135] Multilingual

[0136] The benefit of these techniques is that they are language-agnostic. Different spoken languages ​​are simply different learned phrasings of intent. A neural translator is essentially an encoder-decoder in an embedding space. For a reasonable number of intents, the dimensionality can be much smaller than a full human-language-to-language translator.

[0137] The system can use neural translation capabilities, where the intent is essentially just another target language for the translation. Figure 9 The learned semantic space is shown. Through the shared semantic space, any spoken language expression can be mapped to an expression in another language. Similarly, expressions can be mapped to intentions as if they were spoken languages.

[0138] This has the additional surprising effect of allowing the creation of inverse networks that can generate grammatically correct audible speech in human language from a set of intent arguments.

[0139] Note that there is already a lot of overlap in variant identifiers across languages ​​due to borrowed words between them. For example, the words "KFC" and "iPhone" are used in all languages, and lists of internationally famous musicians are used across languages.

[0140] The intent recognizer can learn phrasing in any language using existing semantic space language translator models. This is practical because virtual assistants only support a very small number of intents compared to the meanings of all possible sentences in a language. The intent JSON can be used to form API requests with their parameters and select appropriate API response data values ​​to serve to the user.

[0141] Quantification accuracy

[0142] Conventional systems measure ASR's word error rate. There's no industry-standard metric for grammatical interpretation accuracy. Whether an interpretation is correct can be subjective. A more practical measure of accuracy is overall user satisfaction. This is also the most meaningful way to quantify the accuracy of neural speech-to-meaning translation. Of course, using a large amount of training data improves accuracy, and some data is more useful than others.

[0143] Most intent recognizer systems recognize fewer possible intents than the possible combinations of words in a spoken utterance. Most variant recognizers recognize fewer enumerated values ​​than the possible combinations of words in a spoken utterance. As a result, the prediction space for intent and variant recognizers is much smaller, and thus the absolute accuracy required to achieve satisfactory performance is much lower.

[0144] Measuring the accuracy of intent recognizers is comparable to measuring the accuracy of sentence detectors in terms of making false assertions or missing actual utterances. Using Neural Speech to Meaning in a system with a sentence detector and an end-of-utterance detector can further improve accuracy by avoiding hitting assertions when they are not used. Furthermore, after the wakeup sentence, at the time of end-of-utterance detection, a Neural Speech to Meaning based system has a defined speech segment that is known to have a complete utterance and can operate on it to determine the intent expressed (if any).

[0145] Cost of acquiring data

[0146] One of the biggest challenges in building neural speech-to-meaning systems and making them accurate is getting enough training data and enough diverse training data.

[0147] For systems that start with a small number of intents where the user has reasonable expectations, it is relatively easy to build a system with sufficient accuracy for the user to accept. For example: a voice-based clock for the blind has one intent; a voice-controlled light bulb has two intents: turn on and off; or a virtual meteorologist supports 10 to 20 intents, such as a short or long report each of a general weather report, a temperature report, or a precipitation report for today, tomorrow, or each of the next 10 days.

[0148] A virtual meteorologist can be trained on 20 intents using approximately 100 carefully selected utterances for each human language. While there are more than 100 / 20 = 5 utterances per intent in human language, using an LSTM model with attention will identify intents from keywords in a variety of utterances. For example, training on the utterances "what is today's high temperature" and "tell me the high temperature for today" will identify almost any other reasonable user utterance with the speech segments "high temperature" and "today."

[0149] Many volunteers will read, for example, 100 sentences in exchange for a $10 gift certificate. Today, high-quality data sources are available for less than $10 per 100 sentences. From 100 to 1,000 utterances might be sufficient to train a recognizer for 20 intents with acceptable accuracy. Training data can cost between $1,000 and $10,000.

[0150] Readers who see a series of sentences with identifiable similar intents will tend to use unnatural voice styles to avoid the monotony of their task. The training system can randomize text samples for intents targeting different types of devices, APIs, or contexts to prevent unnatural readings. Additionally, the system can randomize variable values ​​in natural sentences to collect variable voice data.

[0151] Some ways to collect voice recordings from "volunteer" readers are through downloadable phone apps, web pages with voice input capabilities (e.g., Mozilla Common Voice), or noise-isolating recording booths placed in public places.

[0152] Some dumb readers might say something other than what they see in the text. Before using the data for training, a regular ASR might be used to check the sound against the text. If the reader gives less than a high percentage (e.g., 90%) of correct readings, their gift certificate will be rejected.

[0153] A diverse collection of voices is needed to train a speech-based system for a diverse range of users. It's possible to purchase targeted advertising, such as through Facebook or Google AdWords, to attract the type of voice recording volunteers needed to diversify the training collection. For example, people of one gender are more likely to respond to general notifications about voice recording opportunities, so purchasing ads targeted to this group less likely to respond is one way to ensure a diverse training data collection.

[0154] A multi-domain virtual assistant might have 100 domains, each with an average of 100 common intents, each with an average of 10 valid phrasings in each of the 10 major languages, and 1,000 voices in each language to achieve good accuracy. This would require approximately 1 billion recordings. At an average per-record cost of $0.01 per recording, the virtual assistant would cost only $10 million to train.

[0155] Specialized assistants are likely to be much cheaper. Single-language virtual assistants are likely to be much cheaper.

[0156] Collect data

[0157] Training data can also be collected from databases of different utterances. Such databases exist as a result of storing audio utterance data from users of existing virtual assistant systems. These recordings are often statistically relevant for training future neural speech-to-meaning virtual assistants. Such systems typically have audio recordings stored with an intent data structure or with the ability to immediately reproduce the intent data structure.

[0158] The ideal data source would include recordings of voices that previously generated incorrect intents, which have been labeled with the correct intent by a human expert. One way to facilitate efficient review is to create a web-based interface that allows people to simultaneously listen to the voice audio, see the virtual assistant's response, and type or click an indication of whether it is correct. A second-stage webpage would focus solely on data reviewed as incorrect, providing a view of the audio and several possible correct intents to choose from, with the option to indicate whether none are correct or to enter the correct intent when possible.

[0159] Based on the database of inspected and labeled clean data, the training process can filter the database by intent to provide training data for each intent recognizer independently.

[0160] Compared to acquired clean data, collected data tends to have greater wording diversity. Therefore, it can be more valuable for accurate model training, but it can be relatively expensive because it requires professional time to review and label. It also suffers from the cold start problem, meaning that data becomes available only after the system has been deployed and is being used by users.

[0161] TTS generation of training data

[0162] Another way to generate or expand the training dataset (especially to overcome the cold start problem) is to use parametric TTS to generate speech audio for identifiable phrasing using high parameter diversity. This is the least expensive, but must be used only for a portion of the training dataset, otherwise the training will overfit the properties of the TTS synthesizer. The first generation of TTS systems were concatenated. They synthesized speech from recordings of voice donors. Modern TTS uses a set of parameters to define the sound of the generated speech. The model (e.g., a generated neural network) uses these parameters to generate speech from text. Parametric TTS systems are becoming increasingly sophisticated and have the ability to bring greater variation and realism to the synthesized speech. This allows TTS to generate more training data cheaply without causing the trained model to overfit the properties of TTS.

[0163] Some domains require variable identifiers that support millions of variable values. For example, this occurs in an application with the intent to view items available for purchase at a Mall-Mart big box retailer, which supports queries such as "show me <item>(Show me <product>)" <item>(<item>) is a variable that can refer to any of the millions of items in the store's inventory. When collecting voice recordings of each of these item names being read is impractical, a dataset of speech audio generated by TTS can be used to train a variable identifier.

[0164] Figure 10 A diagram of a neural TTS generator 1001 is shown. It accepts text of variable values ​​as input. The system can accept words, letters, or phonemes as input. The system can accept different numbers of inputs. TTS generator 1001 uses a neural network with a set of weights to convert the input text into output speech audio. Some hidden nodes of the neural TTS generator are recursive.

[0165] When designing a natural-sounding TTS generator, the primary goal is to learn a parameter set that listeners assess as natural-sounding and pleasing. This is not the primary goal of generating data to train a variant recognizer. The important goal is to find many different parameter sets that together produce speech audio that matches the range of speakers that will use the system.

[0166] One way to achieve this is to create a corpus of a diverse set of voices. These voices might all be saying a specific phrase, such as a wake-up phrase for a virtual assistant. They might also all be saying different phrases. Then:

[0167] 1. Based on the relevant features, use a sound analysis (e.g., i-vector fingerprinting) algorithm to calculate the centroid value and the distance from the centroid that captures all natural sounds for all sound recordings. Importantly, the algorithm must ensure not only the presence of human sound characteristics (formant patterns or true voiced speech frequencies), but also the absence of non-human sound characteristics (e.g., noise or other artifacts that do not occur in human speech).

[0168] 2. Use a basic parameterized neural generation TTS engine to generate speech audio for sentences based on the parameter set

[0169] 3. Analyze the generated speech audio to find its distance from the centroid

[0170] 4. If the generated audio is outside the natural range, discard the parameter set

[0171] 5. Otherwise, calculate the vector from the centroid to the eigenvalue of the generated speech audio and save the parameter set and its vector value

[0172] 6. Apply a cost function to select the next parameter set that supports a larger distance from the vector of the saved parameter set but is still within a natural distance from the centroid

[0173] 7. Return to step 1 and perform multiple iterations

[0174] 8. Repeat the above process for at least a few different specific sentences covering a wide range of phonemes and diphones

[0175] 9. For each variable value that the variable identifier needs to recognize, for each saved parameter set, generate speech audio as an initial training set for training the variable identifier.

[0176] One way to train neural speech from TTS to meaning is:

[0177] 1. Identify a number of words that can be values ​​of variables in the intended wording;

[0178] 2. determining a plurality of parameter sets representing the voice diversity of users of the virtual assistant;

[0179] 3. synthesizing a plurality of speech audio segments for a plurality of words, wherein the segments are synthesized based on a plurality of parameters; and

[0180] 4. Using the synthesized speech audio segment, a variant identifier is trained that can calculate the probability of any of a plurality of words in the speech audio.

[0181] It's also possible to train an intent recognizer on audio segments of spoken words, where its input is the probabilities output by the variable recognizer. Alternatively, to synthesize speech, it's possible to synthesize adjacent phonetic information within the context of the variables within the word. That is, what words / phonemes precede and follow a word. The pronunciation of a phoneme is influenced by other phonemes spoken just before or after it. This happens unintentionally in all languages, but in some languages ​​specifically, it happens according to linguistic rules. French legato rules are an example.

[0182] It's also possible to emphasize variables within a word. Spoken sentences in natural speech, especially commands, often contain emphasis. Emphasis can be placed on variable words to help listeners capture the most relevant information. For example, "how is the weather in Austin" so it's not confused with "how is the weather in Boston." Emphasis can also be placed on other words that carry key semantic information. For example, "book a flight from London to Hong Kong."

[0183] The following is one approach to determining multiple parameter sets that represent the diversity of users' voices with minimal deviation.

[0184] 1. Obtaining multiple audio recordings of natural person speech representing a variety of voices;

[0185] 2. Analyze these recordings to compute the recorded speech vectors within the embedding space of speech features;

[0186] 3. Compute the region representing the extent of the recorded speech vector within the embedding space;

[0187] 4. Learn multiple speech synthesis parameter sets using gradient descent based on the loss function. The loss function is calculated as follows:

[0188] 4a. A voice segment is formed based on the parameters in the plurality of parameter sets;

[0189] 4b. Analyzing the synthesized speech segment to calculate the synthesized speech vector in space; and

[0190] 4c. Compute a loss proportional to the clustering of the synthesized speech vectors in the space.

[0191] A loss proportional to the clustering means that we prefer to learn multiple sets of parameters that produce a roughly uniform distribution of vocal audibles. This avoids bias towards certain speaker types and minimizes the amount of data required to train an accurate model.

[0192] Using a well-trained set of speech synthesis parameter sets, it is possible to synthesize one or more speech segments of enumerated words according to the speech synthesis parameter sets; and use it to train a variant recognizer, wherein the training data includes the synthesized speech segments of the enumerated words. The training data may also include other data samples.

[0193] The embedding space is learned in the usual way. However, they can also be chosen based on parameters that are known to be relevant by the engineer or scientist.

[0194] When generating parameter sets, synthesizing audio, and analyzing it to compute its embedding vector, it is possible that some parameter sets will generate speech that falls outside the range of natural sounds. In such cases, it would be reasonable to simply discard such parameter sets.

[0195] To avoid the learned parameter sets being too strictly limited to the dataset of natural sounds used to learn the range of permissible sounds, it is reasonable to add some margin around this region. Parameter sets that synthesize audio segments that fall within this region (including the margin) should not be discarded. They should be used to train a recognizer model that captures unusual sounds.

[0196] Hybrids of the above approaches to generating training data via TTS are possible, as are generally similar approaches with application-specific variants.

[0197] Platform infrastructure

[0198] Several companies offer platforms that deliver virtual assistant capabilities across a wide variety of devices. They also provide access to a diverse range of data source domains and actionable domains. The more domains a platform supports, the more useful it is to device users. Furthermore, the more device users connect to the platform, the greater the value of content provided by third-party domain providers through the platform. Competitive virtual assistant platform providers must continually evolve to support a growing number of domain intents that access a growing number of data types across a growing number of device types.

[0199] Such platforms typically provide some built-in variable identifiers for variables that may be used for many domain intents (such as numbers and location names). The platform will also host more intent identifiers than variable identifiers, because each domain typically has one or more intent identifiers, and there can be thousands of third-party domain providers providing their data and actionable services through the platform. There may also be domain-specific variable identifiers, such as an identifier that identifies the names of products available at a specific retailer.

[0200] Figure 11 An example of a very small platform for use by one or more users 1101 is shown. One or more users access the domain through voice commands via devices such as mobile phones 1102, home appliances 1103, or cars 1104. These devices can access the platform 1105 via a network 1106 (e.g., the Internet or a 5G mobile device network). The platform 1105 can be provided by a company independent of the device manufacturers and network providers.

[0201] The device sends a digital audio stream or recording containing speech to the platform. The platform provider provides built-in common variable identifiers, such as identifiers for location names 1107, restaurant names 1108, addresses 1109, and numbers 1110.

[0202] Some platforms can build user-specific variant recognizers, for example, a recognizer for friend's name 1111, which uses data within the user's contact list and is trained from recordings of the user's voice and known audio segments of names with the same vocabulary spelling.

[0203] Some platforms may provide domain-specific variable identifiers, which are either provided by the domain provider or trained by the platform provider using sound data from the domain provider that can identify variable values. Figure 11 Domain-specific variable identifiers are not shown.

[0204] Platform 1105 also includes a domain identifier 1112. Domain identifier 1112 is trained on known or labeled domain-specific speech recordings. Given incoming speech audio, it predicts which domain intent is correct for recognizing the query. The platform can use this to adjust the intent recognizer's performance or assertion acceptance, or to re-score the intent recognizer's hypotheses. This improves the platform's accuracy, thereby improving the user experience, especially when the platform provides responses to poorly trained intent recognizers for certain third-party domains.

[0205] Domain providers provide domain-specific intent recognizers. Domain providers (possibly in conjunction with platform providers) train recognizer models for the intents that their domain can respond to. Figure 11 In the system, a weather information provider 1113 with weather data receives a hit to its API when intent recognizer 1114 asserts a hit. The weather provider API hit receives a place name value from location name variable recognizer 1107. A CURL request to the API occurs only if the location name recognizer asserts a valid place name within the timeframe of receiving the audio that triggered the weather intent recognizer. Similarly, a food information provider with food data 1115 receives an API hit triggered by food intent recognizer 1116 in the presence of a valid restaurant name from restaurant name variable recognizer 1108.

[0206] Providers 1117 of smart light bulbs enable them to be controlled by the voice-to-meaning platform when an API hit is triggered by a lighting intent recognizer 1118. The lighting intent API and its recognizer do not require input from the variable recognizer because the functionality of the light bulbs does not require any information other than the ability to turn on and off and make the light brighter or dimmer.

[0207] Navigation Command Provider 1119 sends navigation controls and information to the user device, e.g., mobile phone 1102 or car 1104, but not to the home device 1103. Navigation Domain API access is controlled by Navigation Intent Recognizer 1120, which requires exactly one of the restaurant name, address, or friend name variables.

[0208] The realm with the ability to send messages 1121 is controlled by an API hit triggered by a message intent recognizer 1122. It requires a valid friend name variable as the destination for the message. It also requires the message text, which can optionally include numbers. It can also include arbitrary text that is recognized as a transcription using general automatic speech recognition. This is not done in Figure 11 Shown in.

[0209] Developers of client devices such as mobile phones, home appliances, and cars can choose which domains they have access to through the platform, and thus which intent recognizers should be run to query their devices. This is useful because, for example, home devices cannot perform navigation and therefore do not need to recognize navigation intents.

[0210] Supplementing grammar-based systems

[0211] It is possible to use existing data from grammar-based systems to guide neural-based pathways by analyzing a large database of queries, parsing each query to find its general intent, and generating a list of intents sorted by frequency. Next, variable values ​​are replaced with labels (e.g., CITY and NUMBER).

[0212] Example: "what's the weather in <city>(What’s the weather like in <city>)”

[0213] Example "what's <number>plus <number>(<quantity> plus <quantity> is how much)"

[0214] Then, for the most common general intents, find the various wordings that users use to generate the intent, and then use the query audio for that intent as training data for the intent recognizer.

[0215] For the variant identifier, a trusted transcription is searched for instances matching the variable value to be learned. Each variable value is learned using the trimmed, aligned audio segments.

[0216] Queries with plausible grammatical interpretations can be used as training data. Queries known to be ungrammatical can also be used as negative training examples. As the neural speech-to-meaning recognizer is retrained on more data, it becomes more capable of accurately calculating intent hypotheses for phrasings not found in the grammar. This can be used to provide feedback to grammar authors to help them improve the grammar. This is particularly useful for quickly adding new understanding features to the system. Grammar developers do their best to anticipate phrasing, and after a period of use in the wild, the system provides feedback to the grammar developers to improve the grammar.

[0217] A hybrid system with multiple weighted grammars corresponding to different intents can calculate intent hypothesis scores from various grammars based on weights to guess the most likely intent. Simultaneously, the system can use a neural model to calculate intent hypothesis scores. The final intent decision can be based on a weighted average of the grammar-based and neural-based scores.

[0218] It is also possible to use a grammar to generate sentences that match the grammar. Applying TTS to such sentences can produce training audio segments as described above. A wide variety of TTS parameters and generated sentence wording can create a very accurate neural speech-to-meaning intent recognizer.

[0219] Session State

[0220] When a hit is triggered, the neural speech-to-meaning system can store a record of variable types and their values ​​for use in API hits or other result operations. There may be multiple stored variable types and values, such as location names, times, people's names, male names, and female names. The collection of variables and values ​​mentioned above is called session state.

[0221] The intent recognizer can be trained on speech that includes pronouns or other linguistic indicators that reference past semantic information. For example, in English, the word "there" refers to a place mentioned in the past, such as "Let's go there" following the previous sentence "Where is the nearest ice cream shop?" The word "then" can refer to a time mentioned in the past. The words "him," "her," "he," and "she" refer to people mentioned in the past.

[0222] Old session state information can be discarded after a period of time, after which a person may forget the old session state information in the session. Alternatively, session state information can be discarded after a certain number of session turns. In addition, whenever a new value of a variable type is identified, the previously stored value of that variable type must be discarded because pronouns in human conversation refer only to the most recent value of any variable type.

[0223] The system can replace session state information not only with new variable values ​​recognized from the user's speech, but also with values ​​used to provide responses to the user based on information in the API response. For example, if the virtual assistant provides the user response answer: "The nearest ice cream shop is just 3 blocks up the street.", the system will store the value of the PLACE variable "3 blocks up the street."

[0224] Some systems store a single session state history for each user session. Thus, a geo query "What is the population of New York?" followed by a weather query "What's the weather there?" would use the value "New York" for the location slot of the weather query.

[0225] Some systems store session state history for each domain. Thus, queries such as "Where's the nearest bagel shop?", "How many bagel shops are there in Pocatello?", and "Give me directions there." will give directions to the nearest bagel shop, not to Pocatello.

[0226] One approach to identifying when to use information stored in the conversation state instead of variable values ​​is to train a model on conversational audio labeled with pronouns and pointers to their referents. While not focusing on the specific referents, training to recognize pronouns and pointers to arguments in the specific intent of previous queries is important for learning features that indicate such lookup requirements. Training can include both previous specific intents and responses from the machine to the user, as pronouns can refer to semantic information in the response, not just from previous queries.

[0227] It is also possible to use a neural Turing machine approach, such as that given by Graves (arXiv:1410.5401v2[cs.NE], December 10, 2014). A neural Turing machine can store information in memory and retrieve it (according to probability), as dictated by an attention mechanism.

[0228] Linguistic complexity

[0229] The following considerations are not necessary for a system to achieve high user satisfaction. The vast majority of queries to virtual assistants (99% or more) are linguistically simple and do not benefit from the following considerations. However, for some virtual assistants, supporting linguistically complex queries is a selling point. Here are some challenging situations and how to deal with them.

[0230] NEGATIVES - One approach is to train a variant recognizer for negation terms (e.g., "not," "except," "without"). Based on the timing of the term relative to the verb of the query, the output intent gets a negation indicator in the intent just before (for some languages, such as English) or after (for some other languages) the negation term. If supported by the API protocol, the negation indicator can be passed along with the API request. It can also be used to condition or filter responses from API hits.

[0231] Another approach is to train data samples with negative meanings. It is possible to synthesize examples with negative meanings from human recorded speech data or TTS generated data.

[0232] DOUBLE NEGATIVES AND MID-SENTENCE CORRECTIONS — These are not a major issue because they are rare and the user will understand why the response is meaningless.

[0233] COMPOUND QUERIES - It is possible to train a query boundary identifier from labeled audio or synthesized single queries. In the case of English, the word "and" will tend to be a high-weight feature for identifying compound queries.

[0234] CRM

[0235] Figure 12A An example non-transitory computer-readable medium is shown, which is a rotating magnetic disk. Data centers often use magnetic disks to store data and code, including instructions for server processors. The magnetic disks store code including instructions that, if executed by one or more computers, will cause the computers to perform the steps of the methods described herein. Rotating optical disks and other mechanically moving storage media are also possible.

[0236] Figure 12B An example non-transitory computer-readable medium is shown, which is a flash random access memory (RAM) chip. Data centers often use flash memory to store data and code for server processors. Mobile devices often use flash memory to store data and code for processors in system-on-chip devices. The flash RAM chip stores code including instructions that, if executed by one or more computers, will cause the computers to perform the steps of the methods described herein. Other non-removable storage media may be packaged using leads or solder balls.

[0237] Various types of computer-readable media are suitable for storing code including instructions according to various embodiments.

[0238] SoC

[0239] Figure 13A The bottom side of a packaged system-on-chip device 1300 is shown, having a ball grid array for surface-mount soldering to a printed circuit board. Various package shapes and sizes are possible for various chip implementations. The system-on-chip (SoC) device controls many of the embedded system embodiments described herein.

[0240] Figure 13B A block diagram of a system on chip 1300 is shown. It includes a multi-core cluster of computer processor (CPU) cores 1301 and a multi-core cluster of graphics processor (GPU) cores 1302. The processor is connected to an off-chip dynamic random access memory (DRAM) interface 1304 for volatile program and data storage and a flash interface 1305 for non-volatile storage of computer program code in flash RAM non-transitory computer readable media via an on-chip network 1303. The SoC 1300 also has a display interface 1306 for displaying a GUI and an I / O interface module 1307 for connecting to various I / O interface devices (such as those required by different peripheral devices). The I / O interface supports sensors such as touch screen sensors, geolocation receivers, microphones, speakers, Bluetooth peripherals, and USB devices (e.g., keyboards and mice). The SoC 1300 also includes a network interface 1308 to allow the processor to access the Internet via a wired or wireless connection (e.g., WiFi, 3G, 4G Long Term Evolution (LTE), 5G, and other wireless interface standard radios and Ethernet connection hardware). By executing instructions stored in a RAM device via interface 1304 or stored in a RAM device via interface 1305 , CPU 1301 and GPU 1302 perform the steps of the methods described herein.

[0241] server

[0242] Figure 14A A rack-mounted server blade multi-processor server system 1400 is shown according to some embodiments. It includes multiple network-connected computer processors running software in parallel.

[0243] Figure 14B A block diagram of a server system 1400 is shown. It includes a multi-core cluster 1401 of computer processor (CPU) cores and a multi-core cluster 1402 of graphics processing unit (GPU) cores. The processors are connected to a random access memory (RAM) device 1404 for program and data storage via a board-level interconnect 1403. Server system 1400 also includes a network interface 1408 to allow the processors to access the Internet. By executing instructions stored in RAM device 1404, CPU 1401 and GPU 1402 perform the steps of the methods described herein.

[0244] Special samples

[0245] The examples shown and described use certain colloquial languages. Various embodiments operate similarly for other languages ​​or combinations of languages. The examples shown and described use certain domains of knowledge. Various embodiments operate similarly for other domains or combinations of domains.

[0246] Some embodiments are screenless, such as a handset without a display. Some embodiments are stationary, such as a vending machine. Some embodiments are mobile, such as a car. Some embodiments are portable, such as a mobile phone. Some embodiments are intended to be implanted in the human body. Some embodiments include artificial interfaces such as a keyboard or touch screen. Some embodiments include neural interfaces that use a person's thoughts as a form of natural language representation.

[0247] Some embodiments function by running software on a general-purpose programmable processor (CPU) (e.g., a processor with an ARM or x86 architecture). Some power-sensitive embodiments and some embodiments requiring particularly high performance (e.g., for neural network training) use hardware optimization. Some embodiments use application-customizable processors with configurable instruction sets in dedicated on-chip systems, such as the ARC processors from Synopsys and the Xtensa processors from Cadence. Some embodiments use dedicated hardware blocks burned into field programmable gate arrays (FPGAs). Some embodiments use arrays of graphics processing units (GPUs). Some embodiments use application-specific integrated circuits (ASICs) with custom logic to give optimal performance. Some embodiments employ hardware description language code, such as code written in the Verilog language.

[0248] Some embodiments of the physical machines described and claimed herein are programmable in a number of variables, the combinations of which provide a substantially infinite variety of operational behaviors. Some embodiments herein are configured by software tools that provide a large number of parameters, the combinations of which provide a substantially infinite variety of physical machine embodiments of the invention described and claimed. Methods of using such software tools to configure hardware description language representations embody the invention described and claimed. A physical machine may embody the machine described and claimed herein, for example: a semiconductor chip; a hardware description language representation of the logical or functional behavior of the machine according to the invention described and claimed; and one or more non-transitory computer-readable media arranged to store such hardware description language representations.

[0249] Hardware blocks, custom processor instructions, coprocessors, and hardware accelerators perform neural network processing, or portions of neural network processing algorithms, with exceptionally high performance and energy efficiency. This provides long battery life for battery-powered devices and reduces cooling costs for data centers serving many client devices simultaneously.

[0250] General template

[0251] Those skilled in the art will recognize numerous modifications and variations, including any relevant combination of the disclosed features.

[0252] Various embodiments are methods that utilize the actions of either humans or machines, or a combination thereof. Some embodiments are systems of one or more non-transitory computer-readable media arranged to store such instructions for the methods described herein. Some embodiments are physical devices, such as semiconductor chips; hardware description language representations of the logical or functional behavior of such devices; and one or more non-transitory computer-readable media arranged to store such hardware description language representations.

[0253] The description herein reciting the principles, features, and embodiments encompasses both structural and functional equivalents thereof.< / number> < / number> < / city> < / item> < / item> < / time> < / place>

Claims

1. A machine for identifying intent in speech audio, the machine comprising: a variable identifier that processes speech audio features, calculates a probability that the speech audio has any one of a plurality of enumerated variable values, and outputs the variable value having the highest probability; as well as an intent recognizer that processes speech audio features, calculates a probability that the speech audio has the intent, and generates a request for a virtual assistant action in response to the probability that the speech audio has the intent being greater than an intent threshold, wherein: The variable identifier indicates a probability of the speech audio having an enumerated variable value; and The intent recognizer adjusts its output of a request for a virtual assistant action based on a probability that the speech audio has an enumerated variable value.

2. The machine according to claim 1, wherein The adjustment is based on a delayed indication of a probability of the speech audio having a value of an enumerated variable.

3. The machine according to claim 1, wherein: The intent recognizer adjusts its output of the request for a virtual assistant action based on which variable value has the highest probability.

4. The machine according to claim 3, wherein: The adjustment is based on a delayed indication of which variable value has the highest probability.

5. The machine according to claim 1, wherein One of the variable identifier and the intent identifier generates a score, and the other identifier is invoked in response to the score being above a score threshold.

6. The machine of claim 1 , further comprising a domain identifier that processes speech audio features and calculates a probability that the speech audio relates to a specific domain, wherein the intent identifier is associated with the domain and is invoked in response to the probability that the speech audio relates to a specific domain being above a domain threshold.

7. The machine according to claim 1, wherein: No human-readable speech transcription is computed.

8. The machine according to claim 1, further comprising a network client having a capability of accessing a web API, wherein In response to the intent recognizer generating a request for a virtual assistant action, the network client executes a request to the web API, the request having the variable value output by the variable recognizer as an argument.

9. The machine of claim 8, further comprising a speech synthesis engine, wherein In response to receiving a response from the web API, the speech synthesis engine synthesizes speech audio containing information from the web API response and outputs the synthesized speech audio for a user of the virtual assistant.

10. A method for identifying intent from speech audio by a computer system, the method comprising: Get voice audio; processing features of the speech audio to calculate a probability that the speech audio has any one of a plurality of enumerated variable values; Output the variable value with the highest probability; processing features of the speech audio to calculate a probability that the speech audio has the intent; and In response to the probability that the speech audio has the intent being greater than an intent threshold, outputting a request for a virtual assistant action, wherein outputting the request is adjusted based on a probability of the variable value having the highest probability.

11. The method according to claim 10, wherein: The probability is delayed.

12. The method according to claim 10, wherein: Outputting the request is conditioned based on which variable value has the highest probability.

13. The method according to claim 12, wherein: Indication of the value of the variable is delayed.

14. The method according to claim 10, wherein: One of the probability calculations is performed in response to the other probability calculation having a result above a threshold.

15. The method according to claim 10, further comprising: Processing features of the speech audio to calculate a probability that the speech audio relates to a specific domain, wherein the intent is associated with the domain, and calculating the probability that the speech audio has the intent is performed in response to the probability that the speech audio relates to the domain being higher than a threshold.

16. The method according to claim 10, wherein No human-readable speech transcription is computed.

17. The method according to claim 10, wherein Outputting a request for a virtual assistant action is performed by making a request to a web API, the request having as an argument the variable value having the highest probability.

18. The method according to claim 17, further comprising: receiving a response from the web API; and synthesizing voice audio containing information of the response from the web API.

19. A non-transitory computer-readable medium storing code capable of causing one or more computer processors to recognize intent from speech audio by: Get voice audio; processing features of the speech audio to calculate a probability that the speech audio has any one of a plurality of enumerated variable values; Output the variable value with the highest probability; processing features of the speech audio to calculate a probability that the speech audio has the intent; and In response to the probability that the speech audio has the intent being greater than an intent threshold, outputting a request for a virtual assistant action, wherein Outputting the request is adjusted based on the probability of the variable value having the highest probability.

20. The medium according to claim 19, wherein The probability is delayed.

21. The medium according to claim 19, wherein Outputting the request is conditioned based on which variable value has the highest probability.

22. The medium according to claim 21, wherein Indication of the value of the variable is delayed.

23. The medium of claim 19, wherein One of the probability calculations is performed in response to the other probability calculation having a result above a threshold.

24. A medium according to claim 19, wherein the code causes the one or more computer processors to further identify the intent from the speech audio by processing features of the speech audio to calculate a probability that the speech audio relates to a specific domain, wherein the intent is associated with the domain, and calculating the probability that the speech audio has the intent is performed in response to the probability that the speech audio relates to the domain being higher than a threshold.

25. The medium of claim 19, wherein No human-readable speech transcription is computed.

26. The medium of claim 19, wherein Outputting a request for a virtual assistant action is performed by making a request to a web API, the request having as an argument the variable value having the highest probability.

27. The medium of claim 26, wherein the code causes the one or more computer processors to further identify the intent from the voice audio by: receiving a response from the web API; and synthesizing voice audio that includes information of the response from the web API.

Images