Determining a language of speech recognition of a spoken utterance received via an automated assistant interface

Abstract

The present disclosure relates to determining a language of speech recognition of a spoken utterance received via an automated assistant interface. In various implementations, audio data indicative of a voice input includes a natural language request from a user, which can be applied as input across multiple speech-to-text (“STT”) machine learning models to generate a plurality of candidate speech recognition outputs. Each STT machine learning model can be trained in a particular language. For each respective STT machine learning model of the plurality of STT models, the plurality of candidate speech recognition outputs can be analyzed to determine an entropy score for the respective STT machine learning model. Based on the entropy scores, a target language associated with at least one STT machine learning model of the plurality of STT machine learning models can be selected. An automated assistant can respond to the request using the target language.

Description

[0001] Divisional

[0002] This application is a continuation of Chinese Patent Application No. 201980039764.2, filed January 8, 2019, which is a divisional of Chinese Patent Application No. 201980039764.2, filed January 8, 2019. TECHNICAL FIELD

[0003] This application relates to determining a language of speech recognition of a spoken utterance received via an automated assistant interface. BACKGROUND

[0004] Humans can engage in human-to-computer dialog with interactive software applications referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, humans (which when they interact with automated assistants can be referred to as "users") can provide commands and / or requests to an automated assistant using spoken natural language input (i.e., utterances) that can be converted into text and then processed, and / or by providing textual (e.g., typed) natural language input. The automated assistant responds to the requests by providing responsive user interface output that can include audible user interface output and / or visible user interface output.

[0005] As mentioned above, an automated assistant can convert audio data corresponding to a spoken utterance of a user into corresponding text (or other semantic representation). For example, the audio data can be generated based on detection of a spoken utterance of a user via one or more microphones of a client device that includes an assistant interface for enabling the user to interact with the automated assistant. The automated assistant can include a speech recognition engine that attempts to recognize various characteristics of the spoken utterance captured in the audio data, such as sounds (e.g., phonemes) produced by the spoken utterance, an order of the sounds produced, a cadence of speech, intonation, and so on. In addition, the speech recognition engine can identify text words or phrases represented by such characteristics. This text can then be further processed by the automated assistant (e.g., using a natural language understanding (NLU) engine and / or a dialog state engine) to determine responsive content for the spoken utterance. The speech recognition engine can be implemented by the client device and / or through one or more automated assistant components that can be remote from the client device but in network communication with the client device.

[0006] However, many speech recognition engines are configured to recognize speech in only a single language. For users and / or occupants that use multiple languages, such single-language speech recognition engines can be unsatisfactory and can result in automated assistants failing and / or providing incorrect output when receiving spoken utterances in additional languages that are not the single language supported by the speech recognition engine. This can render automated assistants unavailable and / or cause excessive use of computing resources and / or network resources. The excessive use of computing resources and / or network resources can be due to users needing to provide further spoken utterances (i.e., in the supported single language) when automated assistants fail or provide incorrect output. Such additional spoken utterances must be additionally processed by the corresponding client device and / or one or more remote automated assistant components, causing additional use of various resources.

[0007] Other speech recognition engines can be configured to recognize speech in multiple languages, but require a user to explicitly specify which of the multiple languages should be utilized for speech recognition at a given time. For example, some of the other speech recognition engines can require a user to manually specify a default language to be utilized in speech recognition of all spoken utterances received at a particular client device. To change the default language to another language, the user can be required to interact with a graphical interface and / or an audible interface to explicitly change the default language. Such interactions can result in excessive use of computing resources and / or network resources in rendering of the interface, processing of user input provided via the interface, etc. Furthermore, it can often be the case that a user forgets to change the default language before providing a spoken utterance that is not the current default language. As described above, this can render automated assistants unavailable and / or cause excessive use of computing resources and / or network resources. SUMMARY

[0008] Embodiments described herein relate to systems, methods, and apparatuses for automatically determining a language for speech recognition of spoken utterances received via an automated assistant interface. Some embodiments described herein relate to identifying a language of a user request, processing the request using the language, and / or providing a response in the language. In various embodiments, a request submitted by a first user to an automated assistant can be in a first language. Processing such a request can require the automated assistant to determine which language the first user is speaking. Additionally, the first user can prefer that the automated assistant also provide its replies in the first language. The same scenario can be repeated in one or more additional languages different from the first language and can be spoken by the first user or by one or more additional users. Techniques are described herein for identifying a language of a user request, processing the request using the language, and / or replying to the request in the language.

[0009] To identify the language of the user’s request, the automated assistant can provide data indicative of the audio recording of the user’s request (e.g., the audio recording itself, a dimensionally reduced embedding, etc.) to a plurality of machine learning models configured to generate speech-to-text (“STT”) output. Each machine learning model can be designed to perform STT processing in a particular language. To this end, each of these machine learning models can analyze the audio recording of the user’s request (or data indicative thereof) to provide textual data of a plurality of candidate phrases or “hypotheses” corresponding to potential meanings of the audio recording in the respective language of the model.

[0010] In various implementations, it can be determined which model should be used, and ultimately which language should be used to process the request and / or provide the response output. In some implementations, for each machine learning model, an entropy score can be computed, where an entropy score indicative of low entropy represents a good candidate for the user’s language, and an entropy score indicative of high entropy represents a poor candidate for the user’s language. As used herein, an entropy score generally refers to a level of entropy or diversity among the plurality of candidate STT phrases. In some implementations, the entropy score can be based on an average and / or length-normalized edit distance between each candidate phrase of each machine learning model corresponding to a particular language and each of the remaining additional phrases. This can be computed based on a character-by-character difference and / or a word-by-word difference between any two given phrases. Additionally or alternatively, the candidate STT phrases can be embedded into a dimensionally reduced space, and a semantic distance between them can be computed, e.g., based on a Euclidean distance. An average distance computed from this distance or some other value can be used to determine the entropy score for that particular STT machine learning model. The automated assistant can then select the target language based on the entropy score of each model. In some implementations, the automated assistant can further consider other factors, such as a confidence score for each candidate phrase, a language that the assistant has used in the past, and a context of the interaction between the user and the assistant, such as location.

[0011] As an example, assume that Dave and Alice both wish to use an automated assistant. Assume that Dave speaks English, while Alice speaks Spanish. Dave and Alice wish to take turns using the automated assistant and respond in their native language without having to manually change the assistant’s settings. Assume that Alice comes first and requests that the assistant set an alarm for 8:00 AM in Spanish. The assistant will feed an audio recording of her request (or other data of the indicator) into multiple machine learning models to generate candidate phrases in multiple languages corresponding to her request, generate an entropy score for each of the various language models, and select a language based on several factors, including based on the language model with the lowest entropy score, which in this case would be Spanish. The assistant will then process Alice’s request in Spanish and reply in Spanish to inform her that the alarm has been set. Next, Dave can speak with the assistant in English and ask it to dim the lights. The same process can be repeated, except in this case the language model with the lowest entropy score will be English, and the request will be processed in English, and the assistant will reply in English to let Dave know that the lights have been dimmed.

[0012] The techniques described herein give rise to a variety of technical advantages. By processing user requests to determine the language they give in, automated assistants can understand and respond in a multilingual situation, such as a multilingual household or a destination with international travelers, without requiring multiple users to manually switch the assistant’s operating language in the assistant’s settings. This can save time for the users, and save computing resources and user overhead needed to manually switch the assistant’s language to communicate with it.

[0013] In some implementations, a method implemented by one or more processors is set forth, the method comprising: receiving, at a microphone of a computing device operated by a user to engage with an automated assistant, a voice input from the user, wherein the voice input includes a request from the user; applying, as input, data indicative of an audio recording of the voice input to a plurality of STT machine learning models to generate a plurality of candidate speech recognition outputs, wherein each of the plurality of STT machine learning models is trained in a particular language; analyzing, for each respective STT machine learning model of the plurality of STT models, the plurality of candidate speech recognition outputs to determine an entropy score for the respective STT machine learning model; selecting, based on the entropy scores associated with the plurality of STT machine learning models, the language associated with at least one of the plurality of STT machine learning models as a target language; and causing the automated assistant to respond to the request from the user using the target language.

[0014] In various implementations, the entropy score for the respective STT machine learning model can be determined based on a count of the plurality of candidate speech recognition outputs for the respective STT model. In various implementations, the entropy score for the respective STT machine learning model can be determined based on a difference between the plurality of candidate speech recognition outputs for the respective STT model. In various implementations, the entropy score for the respective STT machine learning model can be determined based on a semantic distance between the plurality of candidate speech recognition outputs for the respective STT model in a latent space.

[0015] In various implementations, the target language can be selected further based on one or more of: a historical context of the computing device, a current context of the computing device, or a confidence score computed for each candidate speech recognition output. In various implementations, the entropy score for each respective STT machine learning model can be computed using the candidate speech recognition outputs generated by the respective STT machine learning model having a confidence score above a particular threshold.

[0016] In various implementations, the method can further include causing the automated assistant to prompt the user for confirmation prior to performing any task based on the user's request if the entropy score for the selected STT machine learning model satisfies a predetermined threshold. In various implementations, the causing can include causing the automated assistant to provide a natural language response to the user's request as an audio output, wherein the natural language response is provided in a target language.

[0017] In various implementations, the plurality of STT machine learning models can include a subset of a superset of STT machine learning models, and the method can further include selecting the subset from the superset based on one or more of: a historical context of the computing device, a current context of the computing device, or a setting previously set by the user for the automated assistant.

[0018] Other implementations can include a non-transitory computer readable storage medium storing instructions executable by one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and / or tensor processing unit(s) (TPU(s))) to perform a method such as one or more of the methods described below and / or elsewhere herein. Yet other implementations can include a system of one or more computers and / or one or more robots that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described below and / or elsewhere herein.

[0019] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the inventive subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A system for selecting a language for an automated assistant to interact with a user is illustrated.

[0021] Figure 2 A state machine that can be implemented by an automated assistant is illustrated in accordance with various implementations.

[0022] Figure 3A And 3B An example scenario in which the techniques described herein can be employed is illustrated in accordance with various embodiments.

[0023] Figure 4 An example method for practicing selected aspects of the present disclosure is depicted in accordance with various embodiments.

[0024] Figure 5 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0025] Figure 1FIG. 1 illustrates a system 100 for selecting a language for an automated assistant 104 to interact with a user 130. The automated assistant 104 can operate in part via an automated assistant 126 provided at one or more client devices, such as a client computing device 118 (e.g., a portable computing device 132), and in part via one or more remote computing devices 112, such as a server device 102 (e.g., that can form part of what is commonly referred to as a "cloud infrastructure" or simply "the cloud"). When "automated assistant 104" is used herein, it can refer to one or both of 104 and 126. A user 130 can interact with the automated assistant 104 via an assistant interface 128 of the client computing device 118. The assistant interface 128 includes user interface input devices and user interface output devices for use by the automated assistant 126 in interacting with the user 130.

[0026] The assistant interface 128 accepts user interface input directed to the user 130 of the automated assistant 104 and renders content from the automated assistant 104 for presentation to the user 130. The assistant interface 128 can include a microphone, a speaker, a display panel, a camera, a touch screen display, and / or any other user interface devices of the client computing device 118. The assistant interface 128 can also include a display, a projector, a speaker, and / or any other user interface output devices of the client computing device 118 that can be used to render content from the automated assistant 104. A user can initiate the automated assistant 104 by providing spoken, textual, or graphical input to the assistant interface 128 to cause the automated assistant 104 to perform a function (e.g., provide data, control a peripheral device, access an agent, etc.). In some implementations, the client computing device 118 can include a display device that can be a display panel that includes a touch interface for receiving touch input and / or gestures to allow a user to control applications of the client computing device 118 via the touch interface. In some implementations, the client computing device 118 can lack a display device, providing audible user interface output without providing graphical user interface output. Further, the client computing device 118 can provide user interface input devices, such as microphone(s), for receiving spoken natural language input from the user 130 (and from additional, not-illustrated users).

[0027] The client computing device 118 can communicate with the remote computing device 112 over one or more networks 114, such as the Internet. The client computing device 118 can offload computing tasks to the remote computing device 112, for example, to conserve computing resources at the client device 118 and / or to utilize more robust resources available at the remote computing device 112. For example, the remote computing device 112 can host the automated assistant 104, and the client computing device 118 can send inputs received at one or more assistant interfaces to the remote computing device 112. However, in some implementations, the automated assistant 104 can be hosted by an automated assistant 126 at the client computing device 118. In various implementations, all or less than all aspects of the automated assistant 104 can be implemented at the client computing device 118 by the automated assistant 126. In some of such implementations, aspects of the automated assistant 104 are implemented via the local automated assistant 126 of the client computing device 118 and interface with the remote computing device 112, which enables other aspects of the automated assistant 104.

[0028] The remote computing device 112 can optionally serve multiple users and their associated assistant applications via multiple threads. In implementations where all or less than all aspects of the automated assistant 104 are implemented via the local automated assistant 126 of the client computing device 118, the local automated assistant 126 can be an application separate from (e.g., installed "on top of") the operating system of the client device 118 - or can be implemented directly by the operating system of the client device 118 (e.g., considered an application of the operating system, but integrated with the operating system).

[0029] In some implementations, the remote computing device 112 can include a speech recognition engine 134 that can process audio data received at the assistant interface 128 to determine text and / or other semantic representations of spoken utterances embodied in the audio data. The speech recognition engine 134 can utilize one or more speech recognition models 136 in determining text and / or other semantic representations of spoken utterances embodied in audio data, at least some of which can be referred to as “speech-to-text” or “STT” models. As described herein, multiple speech recognition models 136 can be provided, and each speech recognition model can be for a corresponding language. For example, a first speech recognition model can be for English, a second speech recognition model can be for French, a third speech recognition model for Spanish, a fourth speech recognition model for Mandarin, a fifth speech recognition model for Japanese, and so on. In some implementations, a language model can include or refer to an acoustic model, a speech recognition model, an invocation phrase model, a speech-to-text model, a speech-to-semantic representation model, a text-to-semantic representation model, and / or any other model that can be used to transform natural language input into data that can be processed by an application or device.

[0030] In some implementations, the speech recognition models 136 each include one or more machine learning models (e.g., neural network models) and / or statistical models for determining text (or other semantic representations) that correspond to spoken utterances embodied in audio data. In some implementations, the speech recognition engine 134 can utilize one of the speech recognition models 136 to determine phonemes included in audio data for a corresponding language, and then generate text for the corresponding language based on the determined phonemes. In some implementations, a speech recognition model receives an audio recording of a voice input, e.g., in the form of digital audio data, and converts the digital audio data into one or more text tokens (e.g., STT processing). The one or more models used by such functionality together can be considered a speech recognition model generally model the relationship between audio signals and phonetic units in a language, as well as word sequences in a language. In some implementations, a speech recognition model can be an acoustic model, a language model, a pronunciation model, and so on, as well as models that incorporate functionality of one or more such models. For example, in some implementations, a speech recognition model can be implemented as a finite state decoding graph that includes multiple paths or passageways.

[0031] Furthermore, as described herein, multiple speech recognition models 136 for multiple different languages can be utilized in the processing of audio data to generate multiple candidate semantic and / or textual representations for each of the multiple different languages. For example, in some of those implementations, each speech recognition model can generate multiple candidate speech recognition outputs or "hypotheses." Each candidate speech recognition output / hypothesis for a given language can constitute a "guess" or "prediction" about what was said by the user. If the language used by the user is a different language than the language for which a particular model was trained, there is a possibility that the multiple candidate speech recognition outputs generated by that model can vary greatly from one another in terms of semantics and / or syntax because the particular model was not trained to handle that language. Conversely, if the language used by the user is the language for which a particular model was trained, there is a possibility that the multiple candidate speech recognition outputs will be relatively similar in terms of semantics and / or syntax. In various implementations, the semantic and / or syntactic variation between the multiple speech recognition outputs of a language model can be quantified in a manner that is referred to herein as an "entropy" score or a "diversity" score.

[0032] In some implementations, the entropy score for a particular language can be determined using an equation such as the following:

[0033]

[0034] Here, N is a positive integer that represents the total number of hypotheses generated by the model under consideration by the STT when calculating the entropy score, h i represents the i-th hypothesis generated for the language, d(h i ,h j ) represents the edit distance between the i-th hypothesis and the j-th hypothesis (i and j are both positive integers), and |h i | represents the number of characters in the i-th hypothesis h i . The edit distance d can be a measure of how different two hypotheses are from one another, and in some cases, it can range from zero (i.e., no difference) to the sum of the number of characters in each hypothesis. Thus, in implementations that employ the above equation, the entropy or diversity score can be the average of the length-normalized edit distance between any two pairs of hypotheses, and can be considered a measure of the uncertainty of the STT results. The value output by this equation can be inversely related to the actual cross-entropy of the hypotheses produced by the STT model in question. For example, the lower the value output by this equation, the greater the entropy, and vice versa. However, this is not meant to be limiting. Different formulas can be used for the entropy approximation.

[0035] Note that in some implementations, N need not necessarily equal the number of hypotheses generated by a particular speech recognition model 136. For example, a given speech recognition model 136 can generate a large number of hypotheses. In some such cases, the actual number of hypotheses considered in determining the entropy score can be limited to N, e.g., based on an analysis of the lattice posteriors (e.g., posterior probabilities resulting from decoding the recognition graph) for each hypothesis. In some implementations, the lattice posterior for each hypothesis h can be incorporated in the above equation, e.g., as a scaling term to the edit distance d(). In some implementations, the actual number of hypotheses considered in determining the entropy score can be limited to a value of N (a positive integer). In some implementations, the value of N can be constant across all queries. In some implementations, the value of N can be dynamic, e.g., changing from query to query based on comparing the probability of a hypothesis to a threshold.

[0036] Other equations are also contemplated herein. For example, in some implementations, in addition to or instead of edit distance, the distance between two embeddings of two hypotheses can be considered, e.g., in a latent semantic (or syntactic) space. In some such embodiments, the average of the pairwise distances between all hypotheses (or their latent embeddings) can be used as the entropy score for the speech recognition model.

[0037] Referring back to Figure 1 , the automated assistant 104 can operate concurrently according to one or more language models in order to respond to natural language input from the user 130 and / or provide responsive content to the user 130. In some implementations, the automated assistant 104 can operate concurrently using invocation phrase models and using speech recognition models associated with a particular language. In this way, the automated assistant can process audio data embodying an invocation phrase and one or more commands provided in a particular language, and respond to both the invocation phrase and the one or more commands. Text converted from the audio data and / or semantic representations of the text can be parsed by the text parser engine 110 and made available to the automated assistant 104 as text data or semantic data, which can be used to generate and / or identify command phrases from the user 130 and / or third party applications.

[0038] When the user 130 communicates with the automated assistant 126 at the client computing device 118, the user 130 can provide spoken natural language input to the assistant interface 128 of the client computing device 118. The spoken natural language input can be converted to audio data, which can be processed by the client language model 124, such as for an invocation phrase model to identify whether the audio data embodies an invocation phrase for invoking the automated assistant 126. In some implementations, the invocation phrase model can be employed at the client computing device 118 to determine whether the user 130 intends to invoke the automated assistant 104. When the user provides natural language input to the assistant interface 128 and the natural language input includes an invocation phrase for invoking the automated assistant 104, the client computing device 118 can cause the automated assistant 104 at the server device 102 to receive the natural language input and / or subsequent natural language input from the user 130.

[0039] For example, in response to determining that the user 130 intends to invoke the automated assistant 104 at the client computing device 118, one or more communication channels can be established between the client computing device 118 and the server device 102. Thereafter, when the user continues to provide natural language input to the assistant interface 128, the natural language input will be converted to data, which is then transmitted over the network 114 and processed by the server device 102. Thereafter, the plurality of speech recognition models 136 can be employed to process each of the natural language inputs described herein. Based on the entropy / diversity scores, one or more speech recognition models 136 can be selected for each natural language input as the appropriate model. In some implementations, the candidate speech recognition having the highest confidence score generated by the model can be processed by the automated assistant 104 in order to determine which responsive action to perform.

[0040] Figure 2 The figure illustrates how audio data 260 indicative of spoken natural language input captured at one or more microphones can be processed to determine a language model for downstream processing. The audio data 260 can alternatively take the form of a recording of the spoken natural language input, an embedding generated from such a recording, a feature vector having features extracted from the recording and / or embedding, etc.

[0041] First, the audio data 260 is processed as a recording across multiple STT models 1361-136 NThe input to the speech recognition models (also referred to herein as "speech recognition models") is applied. The number N of models employed can be selected based on various factors. For example, a user who lives in a multilingual household or works in a multilingual workplace can operate a graphical user interface or a voice interface of the automated assistant 104 to select as many languages as the user desires to use in processing. For example, if a user lives in a bilingual household that speaks English and Spanish, the user can select both languages, and two corresponding STT models 136 can be used to process the speech input. In some implementations, there can be multiple dialects for a particular language. In this case, the user can select one or more of those dialects, such as Castilian Spanish and Catalan Spanish.

[0042] Figure 2 Each of the STT models 136 generates at least one hypothesis, and in most cases, multiple hypotheses. For example, the STT model 136i generates a first hypothesis 1A, a second hypothesis 1B, the STT model 1362generates a first hypothesis 1A, a second hypothesis 1B, and so on. For each STT model 136, an entropy score is computed based on the multiple hypotheses generated by that STT model 136, e.g., using the equation set forth earlier or another equation. The entropy scores can then be provided to the speech recognition engine 134. The target language 270 can be selected based at least in part on the entropy scores. For example, the language for which the lowest entropy score is generated can be selected as the target language 270. In various implementations, the target language 270 can be used in downstream processing, such as in parsing, natural language processing, semantic processing, natural language generation, etc., by the dialog state engine. For example, if Italian is selected based on having the lowest entropy score, then Italian natural language processing models and / or semantic processing models trained in Italian can be employed downstream. Natural language output provided by the automated assistant 104 in response to natural language input embodied in the audio data 260 can be in Italian.

[0043] In some implementations, in addition to the entropy score, the speech recognition engine 134 can use one or more other signals 265 to select the target language 270. Signals such as confidence scores for individual hypotheses, audio-based speaker recognition, visual-based speaker recognition (e.g., face recognition, recognition of badges or other markers, detection of wireless signals emitted by a device carried by the speaker, etc.), time of day, languages that the automated assistant 104 has used in the past, context of the interaction between the user and the automated assistant 104 such as the user's location as determined from location coordinate sensors (e.g., GPS, wireless triangulation), can be considered. In some implementations, third-party applications that fulfill the user's request through automated assistant interactions (e.g., car sharing services, food ordering services, etc.) can also be considered. Additionally or alternatively, in some implementations, information from the user's online profile 120 (see Figure 1 ) can be employed, which can include one or more of the above-mentioned signals or additional signals. For example, the user's personal profile 120 can indicate the language she speaks, where she is going, her preferences, etc.

[0044] In some such implementations, these other factors can be used as a tiebreaker for two STT models 136 that have similar entropy scores. For example, assume that the audio data 260 contains a request in Italian. Because both Italian and Spanish are Romance languages, the entropy scores computed based on their respective hypotheses can be similar. In this case, the fact that the user is located in Italy rather than a Spanish-speaking country such as Spain or Mexico can be used to break the tie. As another example, assume that the user has an online personal profile that indicates that he or she speaks Spanish. Such a signal can be used as a tiebreaker to decide that the target language selected should be Spanish rather than Italian.

[0045] Figure 3A and 3B An example is illustrated in which the techniques described herein are implemented to facilitate conversations between two users 101 A and 101 B and an automated assistant 104 operating at least in part on a client computing device 318. In this example, the client computing device 318 takes the form of a standalone interactive speaker, but this is not limiting. As previously noted, the client computing device 318 can take other forms (including assistant devices) such as a vehicle computing system, a smart television, a "dumb" television equipped with a networked key that can be used to engage with the automated assistant 104, a smart watch, a smartphone, a smart appliance equipped with an assistant, etc.

[0046] In Figure 3AIn the scenario where the first user 101A asks, "OK Assistant, how do I enable fingerprint authentication on my smartwatch?", the automated assistant 104 may or may not use the techniques described herein to determine if the first user 101A speaks English. For example, the automated assistant 104 can process each request using the native language set by the first user 101A (English in this example), while simultaneously using the techniques described herein to identify other languages ​​the speaker may have used. If the confidence metric of the English STT output is greater than some threshold, or sufficiently greater than the highest confidence metric of the language selected using the techniques described herein, then English can be selected. Alternatively, in some embodiments, the techniques described herein can be used to process the audio data captured for each utterance. Or, in some embodiments, one or more coarse signals 265 (e.g., location, speaker identity determined using speaker identification, etc.) can be used to determine whether a default language should be used (e.g., Figure 3A (English in Chinese), whether the techniques described herein should be used alone to select the target language, or whether the default language should be used in parallel with the techniques described herein.

[0047] Figure 3B Examples illustrating how the dialogue could continue. Figure 3B In the example, another user, 101B, asked in Chinese, “So what about two-factor authentication?” This translates to “What about two-factor authentication?” in English. In some implementations that employ the English STT model and techniques described herein in parallel, the output from the English STT model may have a low confidence metric (e.g., below a threshold). In contrast, the Chinese STT model selected using the techniques described herein—that is, the model that may have the lowest entropy score—may have a higher confidence metric. Therefore, the output of the Chinese STT model can be used alternatively, and downstream processing may or may not use the Chinese model and / or elements in the same way. Figure 3B As shown, the automated assistant 104 responds with "Same menu, but select 'two-factor'". This can be translated into English as "Same menu, but select 'two-factor'".

[0048] In some implementations, the automated assistant 104 can select various attributes of the natural language output based on the selected language. For example, assume that a question such as "How far from Louisville Kentucky is Frankfort Kentucky?" is posed in Chinese. China is a country that typically employs the metric system. Thus, in various implementations, the answer can be provided in kilometers (and in Chinese). On the other hand, if the same question is posed in English, then the automated assistant 104 can instead respond in English with units of the imperial system.

[0049] Figure 4 An example method 400 for practicing selected aspects of the present disclosure in accordance with various embodiments is depicted. The operations of this flow diagram are described with reference to a system that performs the operations (e.g., the system 100 of FIG. 1). The system can include various components of various computer systems, including the components depicted in FIG. 1. Furthermore, although this method 400 is illustrated as a particular order, this is not meant to be limiting. One or more operations can be reordered, omitted, or added. Figure 1

[0050] At block 402, the system can receive speech input from a user, e.g., at a microphone (e.g., 118) of a computing device operated by the user to engage with an automated assistant (e.g., 104) or at another microphone communicatively coupled with the computing device. The speech input can include a request from the user, such as a request for information (e.g., weather), a request to perform some responsive action (e.g., call a ride-share vehicle, start a timer, operate a smart device, etc.), and so on.

[0051] At block 404, the system can apply data indicative of an audio recording of the speech input (such as the audio recording itself, an embedding, a feature vector, and so on) as input to a plurality of STT machine learning models (e.g., 136) to generate a plurality of candidate speech recognition outputs (or "hypotheses"). Each of the plurality of STT machine learning models can be trained with a particular language. As previously noted, these models can take various forms, such as neural networks, hidden Markov models, and so on.

[0052] ​At block 406, the system can analyze the plurality of candidate speech recognition outputs for each respective STT machine learning model of the plurality of STT models to determine an entropy score for the respective STT machine learning model. Various equations and / or heuristics can be employed, such as the equation described above. For example, the equation described above relies on a normalized edit distance (i.e., the difference between hypotheses). Other implementations can rely on a distance of the dimensionally reduced embeddings of the plurality of candidate speech recognition outputs in a latent space. Additionally or alternatively, other factors that can be considered in computing the entropy score include, but are not limited to, a count of hypotheses generated by a given STT model, a count of hypotheses generated by a given STT model that satisfy some criteria (e.g., a confidence threshold), a semantic difference between hypotheses (e.g., which can be determined by embedding them into a semantic latent space), and so forth.

[0053] As previously described, for many different languages, there can be many different STT models. From a computational resource perspective, it can be wasteful to apply all models in all cases. Thus, in various implementations, various factors such as one or more of the signals 265 can be used to select which STT model to use. For example, in some implementations, the plurality of STT machine learning models that are ultimately used can be a subset of a superset of STT machine learning models. The subset of STT models that are actually used can be selected from the superset based on various signals such as a historical context of the computing device, a current context of the computing device, settings previously set by the user for the automated assistant, and so forth, a time of day, historical usage, a profile 120 of the user (e.g., which can be consulted when the user is recognized by visual appearance or speech), and so forth.

[0054] At block 408, the system can select, based at least in part on the entropy scores associated with the plurality of STT machine learning models, a language (e.g., the language used for training) associated with at least one STT machine learning model of the plurality of STT machine learning models as a target language. As previously described, in some implementations, other signals (e.g., 265) can also be considered as, for example, a tiebreaker. The output of the selected STT model, which in some embodiments can be the one or more hypotheses with the highest confidence measure, can be used for downstream processing. In some implementations, the entropy score itself can be used to determine a confidence measure for the STT output. Assume that a particular STT model has the lowest entropy score, and thus is selected. However, the entropy score for that STT model can still be relatively large, for example, when the speaker did not clearly articulate the request. In some implementations, this relatively large entropy score can affect the confidence scores computed for each hypothesis generated from that STT model. If those confidences do not satisfy some threshold, the automated assistant 104 can seek further confirmation from the user and / or disambiguation.

[0055] At block 410, the system can cause the automated assistant to use the target language to correspond to the request from the user. For example, in some implementations, downstream components such as natural language processors, semantic processors, dialog state engines, natural language generators, etc. can be tailored to the selected target language to process the user’s request. Additionally or alternatively, in some implementations, the user’s request can be embedded into a semantic latent space in which the syntax of the request is more or less ignored or discarded, thereby facilitating a semantically rich feature representation. In some such implementations, at least some of the downstream components used to process the user’s request can be language-agnostic. In some cases, responding to the user’s request can include generating and providing natural language output, which can be presented audibly (e.g., through a speech synthesizer) and / or visually. In some such cases, the language used to generate the natural language output can be the selected target language, and a speech synthesizer can be selected based on the target language, as applicable.

[0056] Figure 5 FIG. 5 is a block diagram illustrating physical components of a computing device 500 with a

[0057] User interface input devices 522 can include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information to computer system 510 or to communicate that information to another computer system.

[0058] User interface output devices 520 can include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem can include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem can also provide non-visual display such as via audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information from computer system 510 to the user or to another machine or computer system.

[0059] Storage subsystem 524 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 can include the logic to perform selected aspects of method 400, and / or to implement one or more of the servers devices 102, client computing devices 118, portable computing devices 132, and / or any other device or operation discussed herein.

[0060] These software modules are generally executed by processor 514 alone or in combination with other processors. Memory 525 used in the storage subsystem 524 can include a number of memories including a main random access memory (RAM) 530 for storage of instructions and data during program execution and a read only memory (ROM) 532 in which fixed instructions are stored. A file storage subsystem 526 can provide persistent (nonvolatile) storage for program and data files, and can include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable memory cards. The modules implementing the functionality of certain implementations can be stored by file storage subsystem 526 in the storage subsystem 524, or in other machines accessible by the processor(s) 514.

[0061] Bus subsystem 512 provides a mechanism for letting the various components and subsystems of computer system 510 communicate with each other as intended. Although bus subsystem 512 is shown schematically as one bus, alternative implementations of the bus subsystem can use multiple busses.

[0062] Computer system 510 can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of computer system 510 depicted in Figure 5 The computer system 510 depicted in

[0063] In situations in which the systems described herein collect personal information about users (or in situations in which personal information can be used), users are provided with an opportunity to control whether programs or features collect user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current geographic location), or to control whether and / or how to receive content from the content server that can be more relevant to the user. In addition, certain data can be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity can be treated so that no personally identifiable information can be determined for the user, or a user's geographic location can be generalized where geographic location information is obtained (such as to a city, postal code, or state level), so that a particular geographic location of a user cannot be determined. Therefore, users can have control over how information is collected about them and / or used.

[0064] While several embodiments have been described and illustrated herein, a variety of other means and / or structures for performing the functions and / or obtaining the results described herein can be utilized and / or developed that are not specifically enumerated herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which a particular implementation of the teachings is used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. It is, therefore, to be understood that the foregoing embodiments are presented by way of example only and that other embodiments may be developed without departing from the scope of the appended claims. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

Claims

1. A method for determining the language of spoken utterance received via an automated assistant interface, comprising: Voice input from the user is received at the microphone of a computing device operated by the user to interact with an automated assistant, wherein the voice input includes requests from the user; The application instructs the audio recording data of the voice input to be used as input across multiple speech-to-text (STT) machine learning models to generate multiple candidate speech recognition outputs for each STT machine learning model, wherein each of the multiple STT machine learning models is trained to generate speech recognition outputs in different languages. For each of the multiple STT machine learning models: The content of one candidate speech recognition output generated based on the corresponding STT machine learning model is compared with the content of at least another candidate speech recognition output generated based on the corresponding STT machine learning model. Based on the comparison, the diversity score of the corresponding STT machine learning model is determined; Based on multiple diversity scores associated with multiple different STT machine learning models, the language associated with at least one of the multiple STT machine learning models is selected as the target language; and This enables the automated assistant to respond to the user's request using the target language.

2. The method according to claim 1, wherein, The diversity score for the corresponding STT machine learning model is determined based on the difference between the corresponding multiple candidate speech recognition outputs of the corresponding STT machine learning model.

3. The method according to claim 1, wherein, The diversity score of the corresponding STT machine learning model is determined based on the semantic distance between the embeddings of the corresponding multiple candidate speech recognition outputs generated by the corresponding STT machine learning model in the latent space.

4. The method according to claim 1, wherein, The selection of the target language is further based on one or more of the following: the historical context of the computing device, the current context of the computing device, or the confidence score calculated for each candidate speech recognition output.

5. The method according to claim 1, wherein, The diversity score for each corresponding STT machine learning model is calculated using two or more candidate speech recognition outputs from the corresponding plurality of candidate speech recognition outputs that have a confidence score above a certain threshold generated by the corresponding STT machine learning model.

6. The method of claim 1, further comprising: If the diversity score for the selected STT machine learning model meets a predetermined threshold, the automated assistant prompts the user for confirmation based on the user's request before performing any task.

7. The method according to claim 1, wherein, The phrase "makes" includes causing the automated assistant to provide a natural language response to the user's request as audio output, wherein the natural language response is provided in the target language.

8. The method according to claim 1, wherein, The plurality of STT machine learning models include a subset of a superset of STT machine learning models, and the method further includes selecting the subset from the superset based on one or more of the following: the historical context of the computing device, the current context of the computing device, or a setting previously set by the user for the automated assistant.

9. A system for determining the language of spoken utterance received via an automated assistant interface, comprising one or more processors and a memory operatively coupled to the one or more processors, wherein, The memory stores instructions, the instructions being responsive to execution by one or more processors, such that the one or more processors: Voice input from the user is received at the microphone of a computing device operated by the user to interact with an automated assistant, wherein the voice input includes requests from the user; The application instructs the audio recording data of the voice input to be used as input across multiple speech-to-text (STT) machine learning models to generate multiple candidate speech recognition outputs for each STT machine learning model, wherein each of the multiple STT machine learning models is trained to generate speech recognition outputs in different languages. For each of the multiple STT machine learning models: The content of one candidate speech recognition output generated based on the corresponding STT machine learning model is compared with the content of at least another candidate speech recognition output generated based on the corresponding STT machine learning model. Based on the comparison, the diversity score of the corresponding STT machine learning model is determined; Based on multiple diversity scores associated with multiple different STT machine learning models, the language associated with at least one of the multiple STT machine learning models is selected as the target language; and This enables the automated assistant to respond to the user's request using the target language.

10. The system according to claim 9, wherein, The diversity score for the corresponding STT machine learning model is determined based on the difference between the corresponding multiple candidate speech recognition outputs of the corresponding STT machine learning model.

11. The system according to claim 9, wherein, The diversity score of the corresponding STT machine learning model is determined based on the semantic distance between the corresponding multiple candidate speech recognition outputs of the corresponding STT machine learning model in the latent space.

12. The system according to claim 9, wherein, The selection of the target language is further based on one or more of the following: the historical context of the computing device, the current context of the computing device, or the confidence score calculated for each candidate speech recognition output.

13. The system according to claim 9, wherein, The diversity score for each corresponding STT machine learning model is calculated using one or more candidate speech recognition outputs from the corresponding plurality of candidate speech recognition outputs that have a confidence score above a certain threshold generated by the corresponding STT machine learning model.

14. The system of claim 9, further comprising: If the diversity score for the selected STT machine learning model meets a predetermined threshold, the automated assistant prompts the user for confirmation based on the user's request before performing any task.

15. The system according to claim 9, wherein, The phrase "makes" includes causing the automated assistant to provide a natural language response to the user's request as audio output, wherein the natural language response is provided in the target language.

16. At least one non-transitory computer-readable storage medium comprising instructions, the instructions being responsive to execution by one or more processors, such that the one or more processors: Voice input from the user is received at the microphone of a computing device operated by the user to interact with an automated assistant, wherein... The voice input includes requests from the user; The application instructs the audio recording data of the voice input to be used as input across multiple speech-to-text (STT) machine learning models to generate multiple candidate speech recognition outputs for each STT machine learning model, wherein each of the multiple STT machine learning models is trained to generate speech recognition outputs in different languages. For each of the multiple STT machine learning models: The content of one candidate speech recognition output generated based on the corresponding STT machine learning model is compared with the content of at least another candidate speech recognition output generated based on the corresponding STT machine learning model. Based on the comparison, the diversity score of the corresponding STT machine learning model is determined; Based on multiple diversity scores associated with multiple different STT machine learning models, the language associated with at least one of the multiple STT machine learning models is selected as the target language; and This enables the automated assistant to respond to the user's request using the target language.

17. The at least one non-transitory computer-readable medium according to claim 16, wherein, The diversity score of the corresponding STT machine learning model is determined based on the semantic distance between the corresponding multiple candidate speech recognition outputs of the corresponding STT machine learning model in the latent space.

Images