Speech processing
By analyzing and biasing the language model, using common terms and phrases in the transcription log of the source language, the transcription of the target language is generated, and the problem of degradation of the recognition quality of speech recognition system in different domains is solved, achieving more accurate and rapid recognition Effect.
Patent Information
- Application Number
- CN202410994336.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-11-30
- Filing Date
- 2019-11-26
- Publication Date
- 2024-11-08
AI Technical Summary
When existing speech recognition systems are used in different domains, it is difficult to achieve high accuracy and rapid recognition, especially in the absence of specific domain data for the target language, resulting in a degradation of recognition quality.
By analyzing and biasing the language model, the common terms and phrases in the transcription log of the source language are used to generate transcription of the target language, thereby improving the accuracy and delay of the speech recognition system. The method includes receiving transcription of the source language, translating into the target language, analyzing and biasing the language model, making it more likely to generate terms and phrases common in a particular domain.
It realizes more accurate and fast speech recognition in new languages and new domains, and even when there is a lack of specific domain data, it can improve the system's recognition performance by using transcription data from other languages.
Smart Images

Figure CN118918885A_ABST
Abstract
Description
[0001] Description of the case
[0002] This application is a divisional application of Chinese invention patent application No. 201980078166.6, with a filing date of November 26, 2019.
[0003] CROSS-REFERENCE TO RELATED APPLICATIONS
[0004] This application claims the benefit of U.S. Application No. 62 / 773,361, filed November 30, 2018, which is incorporated herein by reference. Technical Field
[0005] The present specification generally relates to systems and methods for processing speech input. Background Art
[0006] There is an increasing desire to enable interaction with computers to be performed using speech input. This requires developments in input processing, particularly how to program computers to process and analyze natural language data. Such processing may involve speech recognition, which is a field of computational linguistics that enables spoken language to be recognized and translated into text by computers. Summary of the invention
[0007] In order to obtain optimal speech recognition accuracy, a general purpose speech recognition system may need to be adapted to excel at domain-specific speech recognition. The domain may be related to the type of device receiving the speech data, the application running on the device receiving the speech data, and / or the context of the device receiving the speech data. For example, if the speech recognition system is adapted for use in a vehicle, the speech recognition quality may improve the speech recognition system in the vehicle by being able to more quickly and accurately transcribe speech that includes words or phrases that a user may say in the vehicle.
[0008] One technique used to adapt a speech recognition system to a specific domain may be to analyze a transcription log received by a speech recognition system operating in the specific domain. The adaptation process may include identifying words or phrases that are more common in the transcription log for the specific domain than in the transcription log for a general speech recognition system. The language model may be biased so that the language model is more likely to generate a candidate transcription that includes words or phrases that are more common in the transcription log for the specific domain.
[0009] If there are no domain-specific transcription logs for the target language, the technology may not be practical for launching a speech recognition system for a specific domain of a new target language. In this instance, it may be helpful to utilize domain-specific transcription logs in another language. The language model adaptation system can translate the domain-specific transcription logs into the target language. The language model adaptation system can analyze the translated transcription logs to identify words and phrases that seem more common to users speaking to devices operating in a specific domain. In the event that common words are identified, the language model adaptation system can bias the general language model for the target language so that when processing audio data that may sound similar to other words in the target language, the biased language model may be more likely to generate transcriptions that include common words.
[0010] Using the biased language model, it may be possible to launch a domain-specific speech recognition system in a target language and benefit from improved speech recognition with improved accuracy and latency. A user may benefit from speech recognition that is configured at system launch to more quickly and accurately recognize words and phrases that a user is more likely to say when speaking to a system operating in a specific domain.
[0011] According to innovative aspects of the subject matter described in the present application, a method for processing speech input includes the following actions: receiving, by a computing device, a transcription of an utterance, which is received by a computing device operating in a domain and is in a source language; generating, by the computing device, a translated transcription of the transcription of the utterance in a target language; receiving, by the computing device, a language model for the target language; biasing, by the computing device, the language model for the target language by increasing the likelihood that the language model selects terms included in the translated transcription; and generating, by the computing device, a transcription of the utterance in the target language using the biased language model when operating in the domain.
[0012] These and other embodiments may each optionally include one or more of the following features. The action of generating a transcription of an utterance in a target language using a biased language model includes: when the computing device is operating in the domain, receiving audio data of the utterance by the computing device; providing the audio data as an input to an acoustic model by the computing device, the acoustic model being configured to identify phonemes of the utterance; based on the computing device operating in the domain, providing the phonemes of the utterance as input to the biased language model by the computing device; and, based on providing the phonemes of the utterance as input to the biased language model, generating a transcription of the utterance by the computing device. The language model for the target language is a universal language model. The biased language for the target language is a language model specific to the computing device operating in the domain. The action includes: identifying, by the computing device, n-grams that appear in the translated transcription and the frequency of occurrence of each n-gram; and identifying, by the computing device, a subset of n-grams that appear in the translated transcription with a frequency greater than a threshold.
[0013] The computing device biases the language model by increasing the likelihood that the language model selects a subset of n-grams. The action of increasing the likelihood that the language model selects a subset of n-grams includes: for each n-gram in the subset of n-grams, based on a larger difference between the frequency of occurrence of the n-gram and the threshold frequency of occurrence, increasing the likelihood by a larger amount. The action includes: receiving, by a computing device, audio data of an utterance, the utterance being received by a computing device operating in a domain and being in a source language. The action of receiving, by a computing device operating in a domain and being a transcription of an utterance being received by a computing device operating in a domain and being in the source language includes: generating, by the computing device, a transcription of the utterance, the utterance being received by a computing device operating in a domain other than the domain and being in the source language. The action includes: receiving, by the computing device, an additional transcription of an additional utterance, the additional utterance being received by a computing device operating in a domain other than the domain and being in the source language; generating, by the computing device, an additional translated transcription of the additional transcription; identifying terms having a higher frequency of occurrence in the translated transcription than in the additional translated transcription.
[0014] The act of biasing the language model for the target language by increasing the likelihood that the language model selects a term included in the translated transcription includes: biasing the language model for the target language by increasing the likelihood that the language model selects a term that has a higher frequency of occurrence in the translated transcription than in the transcription after the translation is appended. The act includes: receiving, by a computing device, a grammar of the target language. The act of biasing the language model for the target language by increasing the likelihood that the language model selects a term included in the translated transcription includes: biasing the language model for the target language by increasing the likelihood that the language model selects a grammar that includes the term included in the translated transcription.
[0015] Other embodiments of this aspect include corresponding systems, apparatus, and computer programs recorded on a computer storage device, each configured to perform the operations of the method.
[0016] Particular embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. It may be possible to launch a speech recognition system for a new language and a new domain that is configured to more accurately and quickly recognize common phrases spoken by users in the domain. It may not be necessary to collect any speech data for the domain in the new language.
[0017] The details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and the description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 An example system is illustrated that uses translated training data to bias a language model for use in a particular domain.
[0019] Figure 2 An example system is illustrated that uses translated training data and a pre-selected grammar to bias a language model for use in a particular domain.
[0020] Figure 3 is a flow chart of an example process for adapting a language model for use in a particular domain.
[0021] Figure 4 are examples of computing devices and mobile computing devices.
[0022] Like reference numbers and designations throughout the various drawings refer to like elements. DETAILED DESCRIPTION
[0023] Figure 1 An example system 100 is illustrated that uses translated training data 110 to bias a language model 105 for use in a particular domain. In short, and as described in more detail below, the system 100 examines the translated training data 110 to identify common words and phrases that are likely to be used by speakers in a particular domain. The system 100 biases the language model to favor those common words and phrases.
[0024] The smart speaker 120 may use the biased language model 115 during the automatic speech recognition process. The user 125 speaks a query to the smart speaker 120, and the smart speaker 120 generates a transcription of the query using the biased language model 115. Based on the query, the smart speaker 120 may generate a response 135 to the query.
[0025] Utilize many different languages all over the world, design a kind of speech recognizer that can accurately identify all languages and may not be practical.For example, the same speech recognizer may not be able to identify both English and Turkish.On the contrary, speech recognizer can use English acoustic model and English language model trained to identify English phonemes, and this English language model is configured to identify English words and phrases, which can be called n-gram (such as unigram is a single word, bigram is a phrase of two words etc.).Another speech recognizer can use Turkish acoustic model and Turkish language model trained to identify Turkish phonemes, and this Turkish language model is configured to identify Turkish words and phrases.When the user speaks the language that speech recognizer is not configured to identify, the transcription generated by speech recognizer may be inaccurate.Additionally, the computing resources required for the speech recognizer of accessing multiple acoustic models and multiple language models may be too high, which will cause the time delay of the speaker to be unacceptable.
[0026] Speech recognizers can be adapted for use in different types of situations. For example, a speech recognizer can be adapted for use in a vehicle, by a smart speaker without a display, by a smart speaker with a display, by a digital assistant application running on a mobile phone, by a digital assistant application running on a laptop computer, or by any other similar application or situation. The application or situation may be referred to as a domain. A speech recognizer configured to recognize a specific language can be adapted for use in different domains. For example, an English speech recognizer can be adapted for use by a digital assistant application running on a mobile phone, or for use by a smart speaker with a display. A user interacting with a digital assistant application running on a mobile phone may be more likely to speak a first subset of n-grams, and a user interacting with a smart speaker with a display may be more likely to speak a second subset of n-grams, which may or may not include some of the n-grams in the first set. Therefore, developers may bias the language model of the speech recognizer for a digital assistant application running on a mobile phone domain to be more likely to recognize the first set of n-grams, and bias the language model of the speech recognizer for a smart speaker with a display domain to be more likely to recognize the second set of n-grams.
[0027] Identifying n-grams to bias a language model can be accomplished by examining transcriptions of utterances received by a computing device operating in a particular domain. For example, to identify popular n-grams for a smart speaker without a display domain, the system 100 can examine transcriptions of utterances received by a smart speaker without a display. The system 100 can identify the most common n-grams and bias the language model to favor those n-grams. The smart speaker without a display can then perform an automatic speech recognition process using the biased language model.
[0028] This biasing process for a particular language may not be practical for situations where transcriptions of utterances received by a computing device operating in a particular domain are not available in a particular language. Since smart speakers without displays may not be available for purchase in countries where the population speaks a particular language, transcriptions may not be available. For example, smart speakers without displays may not be available in Turkey. Therefore, there may not be any transcriptions of people speaking Turkish to smart speakers without displays.
[0029] Despite the lack of such transcription data, the system 100 may be able to leverage data from other languages to bias the language model for the target language. For example, the system 100 may be able to leverage English transcription data 140 to bias the Turkish language model 115. Doing so may allow an application or product to launch with a Turkish language model that is biased or adapted for a specific domain, such as a smart speaker without a display.
[0030] like Figure 1 As illustrated, the system 100 accesses English transcription data 140 received by a computing device that operates in a home assistant domain. Devices operating in the home assistant domain may include devices such as smart speakers without a display. Other devices may operate in the home assistant domain, and transcriptions collected from these devices may be added to the English transcription data 140. As an example, the English transcription data 140 may include transcriptions 145 such as "what is the weather", "play jazz music", "volume up", "lights on", and other utterances received by devices operating in the home assistant domain.
[0031] In some embodiments, the system 100 can generate English transcription data 140 by performing automatic speech recognition on audio data of an utterance received by a device operating in the home assistant domain. In some embodiments, the system 100 can receive English transcription data 140 from a human transcriptionist who transcribes the audio data of an utterance received by a device operating in the home assistant domain. In some embodiments, the English transcription data 140 can include a mix of both automatic transcription and manual transcription.
[0032] The system 100 provides the English transcription data 140 as input to the English to Turkish translator 150. The English to Turkish translator 150 may be a machine translator that automatically translates the English transcription data 140 into the Turkish transcription data 110. In some implementations, the English to Turkish translator 150 may be a human translator.
[0033] The Turkish transcription data 110 may include information such as and Transcription 155 may include additional translated transcriptions of English transcription data 140, which may be Turkish transcriptions of "what is the weather", "play jazz music", "volume up", and "lights on".
[0034] In some embodiments, the English to Turkish translator 150 may make errors when translating the English transcription data 140. In this case, the system 100 may translate additional English transcription data transcribed from other utterances received in various domains and / or general domains. The system 100 may provide general English transcription data as input to the English to Turkish translator 150. The system 100 may compare the Turkish transcription data 110 with the general Turkish transcription data to identify n-grams that may be more common in the home assistant domain. Using both the domain-specific translated transcription and the general translated transcription may allow the system 100 to remove some errors that occur during translation. Since some errors may occur when translating domain-specific transcriptions and when translating general transcriptions, these errors may cancel out, and the system may not be able to identify n-grams associated with errors because they are more likely to occur in a specific domain.
[0035] The system 100 provides the Turkish transcription data 110 and the Turkish language model 105 as input to the language model adapter 160. The language model adapter 160 can be configured to bias the Turkish language model 105 to generate a Turkish language model 170 adapted for the home assistant domain. The Turkish language model 105 can be a general Turkish language model that is not biased to any specific n-gram 165.
[0036] The language model adapter 160 can adapt the Turkish language model 105 for the home assistant domain by increasing the weight of the n-grams of the Turkish transcription data 110. For example, the language model adapter 160 can increase the weight of the n-grams for the Turkish transcription data 110. weights, so that if a Turkish speaker says something similar to Even if the general Turkish language model 105 does not select, the Turkish language model 170 may still be more likely to select As a transcription.
[0037] The language model adapter 160 may adapt the Turkish language model 170 to be more likely to recognize n-grams 175, such as and other n-grams included in the Turkish transcription 110. In some implementations, the language model adapter 160 can adapt the Turkish language model 170 based on the frequency of occurrence of n-grams in the Turkish transcription 110. For example, if compared to, appears more frequently in Turkish transcription 110, then In contrast, the language model adapter 160 may make the Turkish language model 170 more biased towards In some implementations, the language model adapter 160 can bias the Turkish language model 170 toward different terms for each n-gram 175. For example, the language model adapter 160 can bias the Turkish language model 170 toward different terms for each n-gram 175. Both. In some instances, the preference for "hava" and In contrast, the language model adapter 160 may be more inclined to
[0038] The smart speaker 120 accesses the Turkish language model 170 during processing of the utterance 130. The smart speaker 120 may operate in the home assistant domain and therefore accesses the Turkish language model 170 adapted for the home assistant domain. Figure 1 As shown, the smart speaker 120 may output The smart speaker 120 may prompt the user 125 in response to the user 125 touching the smart speaker 120, saying a predetermined hot word (e.g., "ok computer" or a similar Turkish phrase), or any other similar action. The user 125 may use speech 130 In some implementations, user 125 may say the hot word following utterance 130 without a smart speaker outputting prompt 180 .
[0039] The smart speaker 120 receives the utterance 130 and performs automatic speech recognition using a Turkish acoustic model and a Turkish language model 170 adapted for the home assistant domain. The smart speaker 120 initiates an action based on the utterance 130. In this case, the action may be to generate a query for the weather in the location of the smart speaker 120. The smart speaker 120 may receive a response to the query and generate a response 135.
[0040] Figure 2 An example system 200 is illustrated that uses both translated training data 204 and a pre-selected grammar 206 to bias a language model 202 for use in a particular domain. In short, and as described in more detail below, the system 200 compares the pre-selected grammar 206 with the translated training data 204 to identify common words and phrases that are likely to be used by speakers in the particular domain. The system 200 biases the language model to favor terms in the grammar 206 and the translated training data.
[0041] A vehicle equipped with an automated digital assistant 208 can use the biased language model 210 during an automated speech recognition process. A user 212 speaks a query to the automated digital assistant 208, and the automated digital assistant 208 generates a transcription of the query using the biased language model 210. Based on the query, the automated digital assistant 208 can generate a response 216 to the query.
[0042] Similar to Figure 1 In the system 100, the system 200 may be configured to generate a biased language model 210 for a language for which the system 200 does not have any access to training data for that language. Figure 2 In the example of , despite having access only to English transcriptions 218 received by a device operating in the vehicle assistant domain, the system 200 generates a biased Basque language model 210 for the vehicle assistant domain.
[0043] In addition to utilizing the English transcription 218 received by the device operating in the vehicle assistant domain, the system 200 also accesses pre-programmed grammars 206. To ensure that speech recognition products for new languages can function correctly, developers can include a set of grammars 206 that a speech recognizer can access to assist in determining the action requested by the user. In addition to using only a general language model to initiate a speech recognition product in a new language, developers can include a set of grammars 206 to ensure that the user can interact with the device. The device can indicate the syntax of the grammar 206 and the action for each grammar by displaying the syntax on a display, outputting audio of an example, or any other similar way of outputting data.
[0044] A grammar may be different from a vocabulary from which a language model selects words. A grammar is associated with the structure of a transcription of an utterance. If a transcription matches the structure of a grammar, the system 200 performs an action corresponding to the grammar. A grammar may be any word structure that can be described using common annotation techniques, such as Bakus-Naur form. Each grammar may correspond to a specific user intent. For example, a user intent may be to issue a home automation command or a media playback command. An example of a grammar may include a grammar for an alarm. An alarm grammar may define a number as 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 0 using the annotation $DIGIT=(0|1|2|3|4|5|6|7|8|9). An alarm grammar may define a time using the annotation $TIME=$DIGIT$DIGIT:$DIGIT$DIGIT(am|pm), which indicates that a time includes two digits, followed by a colon, then two digits, followed by "morning" or "afternoon". An alarm grammar may define the mode of an alarm using the annotation $MODE=(alarm|timer), which indicates whether the alarm should be in alarm mode or timer mode. Finally, the alarm grammar may define the alarm syntax as $ALARM=set$MODE for $TIME, which indicates that the user can say "set alarm for 6:00am" or "set timer for 20:00." The speech recognition system uses the grammar to parse the transcription of the utterance or typed command and identify the action for the device to perform.
[0045] In some instances, grammar 206 may not indicate which grammar 206 or which word included in grammar 206 may be something a speaker is more likely to say to a device operating in a particular domain. For example, grammar 206 may be a set of grammars for a vehicle assistant domain. In addition to grammar examples 220 including "egungo trafikoa," "$LOCATION jarraibideak," "$NUMBERtenperatura ezarri," and "ezarriirratia$STATION," grammar examples 220 may also include the $ALARM grammar. However, in the vehicle assistant domain, a speaker may be more likely to say commands for controlling a vehicle, requesting traffic information, requesting route guidance, and other vehicle-related commands. Although it may be important for a vehicle to be able to recognize and execute a command to set an alarm, alarm commands may not be common in the vehicle assistant domain.
[0046] against Figure 2 In the illustrated example, a developer may wish to make the vehicle assistant technology available to Basque speakers. If the developer accesses transcriptions of Basque speakers using the vehicle assistant technology, then the developer will be able to use these transcriptions to adapt the Basque language model for the vehicle assistant domain. In the absence of transcriptions of Basque speakers using the vehicle assistant technology, the developer can use a generic Basque language model 202. Using a generic Basque language model 202 may result in a poor experience for the speaker because the Basque language model 202 may not recognize that the speaker is saying "egungo trafikoa" if the user is not speaking clearly or there are other Basque phrases that sound similar to "egungo trafikoa". In instances where the generic Basque language model 202 may not recognize, a Basque language model that is biased towards phrases that the speaker is more likely to say in the vehicle assistant domain may be more likely to recognize that the speaker is saying "egungo trafikoa". In the absence of a transcription of a Basque speaker using the vehicle assistant technology and adapting the Basque language model for the vehicle assistant domain, the included grammar 206 may be a grammar that a developer includes in the vehicle assistant technology when launching the technology for Basque speakers.
[0047] System 200 illustrates a component for adapting a Basque language model 202 using a grammar 206 and an English transcription 218 and generating a Basque language model 210 adapted for a vehicle assistant domain. System 200 accesses English transcription data 218 received by a computing device operating in a vehicle assistant domain. Devices operating in the vehicle assistant domain may include devices such as cars equipped with a voice interaction system. Other devices may operate in the vehicle assistant domain, and transcriptions collected from these devices may be added to the English transcription data 218. As an example, the English transcription data 218 may include transcriptions 145 such as "current traffic", "directions to home", "open garagedoor", "lower fan speed" and other utterances received by devices operating in the vehicle assistant domain.
[0048] In some embodiments, the system 200 can generate English transcription data 218 by performing automatic speech recognition on audio data of an utterance received by a device operating in the vehicle assistant domain. In some embodiments, the system 200 can receive English transcription data 218 from a human transcriptionist who transcribes the audio data of an utterance received by a device operating in the vehicle assistant domain. In some embodiments, the English transcription data 218 can include a mix of both automatic transcription and manual transcription.
[0049] The system 200 provides the English transcription data 218 as input to an English to Basque translator 224. The English to Basque translator 224 may be a machine translator that automatically translates the English transcription data 218 into the Basque transcription data 204. In some implementations, the English to Basque translator 224 may be a human translator.
[0050] The Basque transcription data 204 may include transcriptions 226 such as “egungo trafikoa”, “etxerakojarraibideak”, “garaje ate irekia”, and “behe-fanaren abiadura”, which may be Basque translations of “current traffic”, “directions to home”, “open garage door”, and “lower fanspeed”, respectively. The transcriptions 226 may include additional translated transcriptions of the English transcription data 218.
[0051] In some embodiments, the English to Basque translator 224 may make mistakes when translating the English transcription data 140. In this case, the system 200 can translate additional English transcription data transcribed from other utterances received in various domains and / or universal domains. The system 200 can provide universal English transcription data as input to the English to Basque translator 224. The system 200 can compare the Basque transcription data 204 with the universal Basque transcription data to identify n-grams that may be more common in the vehicle assistant domain. Using both the domain-specific translated transcription and the universal translated transcription can allow the system 200 to remove some errors that occur during translation. Since some errors may occur when translating both the domain-specific transcription and the universal transcription, these errors may cancel out, and the system may not be able to identify n-grams associated with the errors because they are more likely to occur in a specific domain.
[0052] The system 200 provides the Basque transcription data 204 and the Basque grammar 206 as inputs to a grammar weighter 228. The grammar weighter 228 can be configured to generate a weighted grammar 230 by comparing the Basque transcription data 204 to the Basque grammar 206. The grammar weighter 228 can identify grammars included in the Basque grammar 206 that parse words or phrases stored in the Basque transcription data 204. For example, the Basque grammar "$LOCATION jarraibideak" can parse the Basque transcription "Etxerako jarraibideak" because "etxerako" may be included as a lexical item that can occupy the $LOCATION variable. The Basque grammar "ezarriirratia$STATION" may not parse any transcription in the Basque transcription data 204. As illustrated by weighted grammar 232 , grammar weighter 228 may identify both “egungo trafikoa” and “$LOCATION jarraibideak” as grammars that are more likely to be spoken by a Basque-speaking user interacting with the vehicle assistant system.
[0053] In some embodiments, the grammar weighter 228 can assign numerical weights to the Basque grammar 206. For example, the grammar weighter 228 can assign a weight of 0.1 to a grammar that does not parse any transcription in the Basque transcription data 204. A weight of 0.1 may not indicate that the vehicle assistant system may not be able to identify the grammar as a parsed transcription, but it may reduce the likelihood that the vehicle assistant system will select the grammar and perform a corresponding action when there are other grammars with higher weights that also parse the transcription. The grammar weighter 228 can assign higher weights based on the frequency with which the parsable transcription occurs in the Basque transcription data 204. For example, if the Basque transcription data 204 includes seven transcriptions that can be parsed by the grammar "ezarriirratia$STATION", the grammar weighter 228 can assign a weight of 0.3. Some of the seven transcriptions may be identical while other transcriptions may include different terms for the $STATION variable. If the Basque transcription data 204 includes eleven transcriptions that can be parsed by the grammar "$NUMBER tenperatura ezarri", the grammar weighter 228 can assign a weight of 0.4. In some implementations, the grammar weighter 228 can assign the same weight to grammars that parse at least one transcription, and assign different weights to grammars that do not parse the transcription.
[0054] The system 200 provides a weighted Basque grammar 230 and a Basque language model 202 as input to a language model adapter 234. The language model adapter 234 can be configured to bias the Basque language model 202 to generate a Basque language model 210 adapted for the vehicle assistant domain. The Basque language model 202 can be a general Basque language model that is not biased to any particular terms, phrases, words, or n-grams.
[0055] The language model adapter 234 can adapt the Basque language model 202 for the vehicle assistant domain by increasing the weights of the n-grams of the weighted Basque grammar 230. For example, the language model adapter 234 can increase the weight for the n-gram "egungo trafikoa" so that if a Basque speaker says something that sounds similar to "egungotrafikoa", the Basque language model 210 may be more likely to select "egungo trafikoa" as a transcription even if the generic Basque language model 202 does not.
[0056] The language model adapter 234 can adapt the Basque language model 210 to be more likely to recognize n-grams 232, such as "egungo trafikoa," "$LOCATION jarraibideak," and other n-grams included in the weighted Basque grammar 230. In some implementations, the language model adapter 234 can adapt the Basque language model 210 according to the weights assigned to each of the grammars in the grammar. For example, if the weight of "egungo trafikoa" is 0.3 and the weight of "$LOCATION jarraibideak" is 0.5, the language model adapter 234 may bias the Basque language model 210 so that the Basque language model 210 may be more likely to select "etxerako jarraibideak" over a similar phonetic transcription when receiving an utterance. Upon receiving the utterance, the Basque language model 210 may still likely select "egungo trafikoa" over a similar phonetic transcription, but the Basque language model 210 bias may be less than "etxerako jarraibideak". In some implementations, the language model adapter 234 may bias the Basque language model 210 toward different terms in each weighted grammar 230. For example, the language model adapter 234 may bias "egungo trafikoa", "trafikoa", and "egungo". In some instances, the language model adapter 234 may bias "egungo trafikoa" more than "trafikoa" and "egungo" individually. In some implementations, the language model adapter 234 may identify "trafikoa" as a more preferred n-gram than "egungo" because "trafikoa" may appear in a heavier weighted grammar included in the weighted grammar 230.
[0057] In some implementations, the language model adapter 234 can bias the Basque language model 210 toward n-grams that are parsable by the weighted grammar 230, in addition to every term that can be used for a variable in the grammar. For example, the language model adapter 234 can bias the Basque language model 210 toward terms that can occupy the $LOCATION variable in addition to "$LOCATION jarraibideak" and "jarraibideak".
[0058] The vehicle may be equipped with a vehicle assistant system. The vehicle assistant system may access the Basque language model 210 during the processing of the utterance 214. Since the vehicle assistant system is a digital assistant system located in the vehicle, it operates in the vehicle assistant domain.
[0059] like Figure 2 As illustrated, the vehicle assistant system may listen for voice commands from user 212. The vehicle assistant system may respond to user 212 saying a predetermined hot word (e.g., "ok computer" or a similar Basque phrase), user 212 pressing a button on the steering wheel, or any other similar action to begin processing the received audio. The user may say utterance 214 "Nola trafikoa da?".
[0060] The vehicle assistant system receives the utterance 214 and performs automatic speech recognition using a Basque acoustic model and a Basque language model 210 adapted for the vehicle assistant domain. The vehicle assistant system initiates an action based on the utterance 214. In this case, the action may be to generate a query for traffic for the location of the vehicle. The vehicle assistant system may receive a response to the query and generate a response 216 "Trafikoa argi dago". The Basque language model 210 may not be biased towards the exact phrase of the utterance 214, but the Basque language model 210 may be biased towards the keyword "trafikoa". Biasing towards n-grams spoken by the user may assist the vehicle assistant system in identifying an accurate transcription.
[0061] Figure 3 3 is a flow chart of an example process 300 for adapting a language model for use in a particular domain. In general, process 300 uses translated training data to adapt a language model for use in a particular domain. Process 300 may be useful if no training data for the target language exists in the particular domain. Process 300 will be described as being performed by a computer system (e.g., Figure 1 System 100 or Figure 2 The system 100 or each of the components of the system 100 may be included on a single computing device or distributed across multiple computing devices.
[0062] The system receives a transcription of an utterance that is received by other systems operating in the domain and is in a source language (310). In some embodiments, the system can receive audio data of a source language utterance received by the other systems when the other systems are operating in the domain. In this instance, the system can generate a transcription of the audio data using an automatic speech recognizer that is configured to recognize the source language. The domain can be related to the type of system receiving the utterance, such as a smart speaker with a display. Other domains can include smart speakers without displays, vehicle assistant domains, map application domains, domains that depend on location (e.g., a park or restaurant), home assistant domains, and any other similar type of domains.
[0063] The system generates a translated transcript in a target language of the transcript of the utterance (320). The system may use machine translation to translate the transcript. For example, the system may use a machine translator configured to translate English into Turkish. In some embodiments, the system may receive and translate transcripts received by a system operating in a different domain. For example, the system may translate transcripts received in a map application domain. The system may also translate transcripts received in other domains, which may include multiple domains, such as a general domain or a combination of other domains that may or may not include a map application domain.
[0064] The system receives a language model for a target language (330). In some embodiments, the target language model is a general language model that may not be biased toward any particular word, phrase, and / or n-gram. For example, the target language model may be a general Turkish language model.
[0065] The system biases the language model for the target language by increasing the likelihood that the language model selects terms included in the translated transcription (340). The system identifies words, phrases, and / or n-grams that appear in the translated transcription. The system biases the target language model so that in instances where the user's speech sounds similar to other words, phrases, and / or n-grams, the biased language model may be more likely to generate a transcription that includes the words, phrases, and / or n-grams that appear in the translated transcription.
[0066] In some embodiments, the system can compare the translated transcription from a specific domain with the translated transcription from other domains. The system can identify those words, phrases and / or n-grams that are more common in the translated transcription from a specific domain. The system can bias the language model to more likely generate a transcription including these words, phrases and / or n-grams. By using additional translations from other domains, the system can be allowed to remove noise from the translation process. Because the translation process may be a machine translation process, the translation process may have some inaccuracies. These inaccuracies may appear in the translated transcription from a specific domain, just as they appear in the translated transcription from other domains. By comparing the two groups of translated transcriptions, the system may be able to offset words that may not reflect accurate translations, and the system biases the language model to more likely generate a transcription that includes words, phrases and / or n-grams of the translated transcription from a specific domain, without any words, phrases and / or n-grams from translation noise. Translating transcriptions from other domains can also help the system identify words, phrases, and / or n-grams that may be closely related to a specific domain, rather than words, phrases, and / or n-grams that may generally be common words.
[0067] In some embodiments, the system can bias the language model based on how often the words, phrases, and / or n-grams appear in the translated transcription. For example, if the Turkish word "hava," which is related to the English word "weather," appears more frequently in the translated transcription than the Turkish word "caz," which is related to the English word "jazz," the system may bias the language model more toward "hava" than toward "caz." In some embodiments, the system can bias the language model equally for words, phrases, and / or n-grams that appear in the translated transcription.
[0068] In some embodiments, the system can access a set of grammars for the target language that a device operating in a particular domain can use to parse candidate transcriptions. The system can compare n-grams included in the grammar to n-grams in the translated transcription. The system can weight the grammars based on the frequency with which n-grams appear in the translated transcription that the n-grams appear in the grammar or can be parsed by the gram. The system can adapt the language model using weighted grammars by biasing the language model when generating the transcription to make n-grams more likely to be selected in the grammar and the translated transcription.
[0069] In some embodiments, when the system or another system operates in a domain for which the system adapts a language model, the system or another system can use the adapted language model during the automatic speech recognition process. The system or other system can use an acoustic model that is trained to identify phonemes for the target language and for the adapted language model. If the system or other system has access to a set of grammars, the system can use these grammars to parse candidate transcriptions.
[0070] Figure 4 Examples of computing device 400 and mobile computing device 450 that can be used to implement the techniques described herein are shown. Computing device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other appropriate computers. Mobile computing device 450 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be examples only, and are not intended to be limiting.
[0071] The computing device 400 includes a processor 402, a memory 404, a storage device 406, a high-speed interface 408 connected to the memory 404 and a plurality of high-speed expansion ports 410, and a low-speed interface 412 connected to a low-speed expansion port 414 and the storage device 406. Each of the processor 402, the memory 404, the storage device 406, the high-speed interface 408, the high-speed expansion port 410, and the low-speed interface 412 is interconnected using various buses and can be installed on a common motherboard or otherwise appropriately installed. The processor 402 can process instructions for execution within the computing device 400, including instructions stored in the memory 404 or on the storage device 406, to display graphical information for a GUI on an external input / output device (such as a display 416 coupled to the high-speed interface 408). In other embodiments, multiple processors and / or multiple buses and multiple memories and memory types can be used appropriately. Moreover, multiple computing devices can be connected, each of which provides a portion of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system).
[0072] Memory 404 stores information within computing device 400. In some implementations, memory 404 is one or more volatile memory units. In some implementations, memory 404 is one or more non-volatile memory units. Memory 404 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.
[0073] Storage device 406 can provide mass storage for computing device 400. In some embodiments, storage device 406 can be or include computer readable media, such as floppy disk device, hard disk device, optical disk device or tape device, flash memory or other similar solid-state memory device or array of devices (including devices or other configurations in storage area network). Instructions can be stored in information carriers. When executed by one or more processing devices (such as processor 402), instructions execute one or more methods, such as the above method. Instructions can also be stored by one or more storage devices (such as memory 404, storage device 406 or memory on processor 402) such as computer or machine readable media.
[0074] The high-speed interface 408 manages bandwidth-intensive operations for the computing device 400, while the low-speed interface 412 manages less bandwidth-intensive operations. This allocation of functions is merely an example. In some embodiments, the high-speed interface 408 is coupled to the memory 404, the display 416 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 410, which can accept various expansion cards (not shown). In an embodiment, the low-speed interface 412 is coupled to the storage device 406 and the low-speed expansion port 414. The low-speed expansion port 414, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device (such as a switch or router), for example, through a network adapter.
[0075] As shown in the figures, computing device 400 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 420, or multiple times in a group of such servers. In addition, it can be implemented in a personal computer (such as a laptop computer 422). It can also be implemented as part of a rack server system 424. Alternatively, components from computing device 400 can be combined with other components in a mobile device (not shown) (such as mobile computing device 450). Each such device can contain one or more of computing device 400 and mobile computing device 450, and the entire system can be composed of multiple computing devices that communicate with each other.
[0076] The mobile computing device 450 includes other components such as a processor 452, a memory 464, an input / output device (such as a display 454), a communication interface 466, and a transceiver 468. The mobile computing device 450 may also be provided with a storage device, such as a micro hard drive or other device, to provide additional storage. Each of the processor 452, the memory 464, the display 454, the communication interface 466, and the transceiver 468 are interconnected using various buses, and multiple components may be mounted on a common motherboard or otherwise appropriately mounted.
[0077] Processor 452 may execute instructions within mobile computing device 450, including instructions stored in memory 464. Processor 452 may be implemented as a chipset of chips that include individual or multiple analog and digital processors. For example, processor 452 may provide coordination for other components of mobile computing device 450, such as control of a user interface, applications executed by mobile computing device 450, and wireless communications performed by mobile computing device 450.
[0078] Processor 452 can communicate with the user through control interface 458 and display interface 456 coupled to display 454. Display 454 can be, for example, a TFT (thin film transistor liquid crystal display) display or an OLED (organic light emitting diode) display or other appropriate display technology. Display interface 456 may include appropriate circuit systems to drive display 454 to present graphics and other information to the user. Control interface 458 can receive commands from the user and convert them for submission to processor 452. In addition, external interface 462 can provide communication with processor 452 so that mobile computing device 450 can communicate with other devices in the vicinity. In some embodiments, external interface 462 can provide, for example, wired communication, or in other embodiments, wireless communication can be provided, and multiple interfaces can also be used.
[0079] The memory 464 stores information within the mobile computing device 450. The memory 464 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 474 may also be provided and connected to the mobile computing device 450 via an expansion interface 472, which may include, for example, a SIMM (single line memory module) card interface. The expansion memory 474 may provide additional storage space for the mobile computing device 450, or may also store applications or other information for the mobile computing device 450. Specifically, the expansion memory 474 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Thus, for example, the expansion memory 474 may be provided as a security module for the mobile computing device 450, and may be programmed with instructions that allow the mobile computing device 450 to be used securely. In addition, secure applications may be provided via a SIMM card together with additional information (such as placing identification information on the SIMM card in an inviolable manner).
[0080] The memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), as discussed below. In some embodiments, the instructions are stored in an information carrier. When executed by one or more processing devices (e.g., processor 452), the instructions perform one or more methods, such as the above methods. The instructions may also be stored by one or more storage devices, such as one or more computer or machine readable media (e.g., memory 464, expansion memory 474, or memory on processor 452). In some embodiments, the instructions may be received in a propagated signal, for example, via transceiver 468 or external interface 462.
[0081] The mobile computing device 450 may communicate wirelessly via a communication interface 466, which may include digital signal processing circuitry if desired. The communication interface 466 may provide for communication in various modes or protocols, such as GSM voice calls (Global System for Mobile Communications), SMS (Short Message Service), EMS (Enhanced Messaging Service) or MMS messaging (Multimedia Messaging Service), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), etc. For example, such communication may be performed using radio frequencies via a transceiver 468. In addition, short-range communications may be performed such as using Bluetooth, WiFi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 470 may provide additional navigation and location-related wireless data to the mobile computing device 450, which may be used appropriately by applications running on the mobile computing device 450.
[0082] Mobile computing device 450 may also communicate audibly using audio codec 460, which may receive verbal information from a user and convert it into usable digital information. Audio codec 460 may likewise generate audible sounds for the user, such as through a speaker in, for example, an earpiece of mobile computing device 450. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on mobile computing device 450.
[0083] As shown in the drawings, mobile computing device 450 can be implemented in many different forms. For example, it can be implemented as a cellular phone 480. It can also be implemented as a part of a smart phone 482, a personal digital assistant, or other similar mobile devices.
[0084] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor that may be special or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.
[0085] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented using high-level procedural and / or object-oriented programming languages and / or assembly / machine languages. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) that is used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal that is used to provide machine instructions and / or data to a programmable processor.
[0086] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input from the user can be received in any form (including acoustic input, verbal input, or tactile input).
[0087] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.
[0088] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
[0089] Although some embodiments have been described in detail above, other modifications are possible. For example, although the client application is described as accessing (multiple) delegates, in other embodiments, (multiple) delegates may be adopted by other applications that are implemented by one or more processors, such as applications executed on one or more servers. In addition, the logic flows depicted in the accompanying drawings do not require the specific order or sequential order shown to achieve the desired results. In addition, other actions may be provided, or actions may be eliminated from the described processes, and other components may be added to or removed from the described systems. Therefore, other embodiments are within the scope of the following claims.
[0090] The claims are as stated above.
Claims
1. A computer-implemented method, which, when executed on data processing hardware, causes the data processing hardware to perform operations, the operations comprising: receiving a transcription of an utterance in a source language received by a computing device operating in a particular domain; using the received transcription of the utterance in the source language, adapting a speech recognizer for a target language for use by a digital assistant system operating in the specific domain to increase the likelihood that the speech recognizer generates a transcription that includes common words or phrases identified for the specific domain, wherein the digital assistant system does not have any access to training data for the target language; receiving audio data corresponding to an utterance spoken by a user in the target language, the utterance being directed to the digital assistant operating in the particular domain; as well as A transcription of the utterance spoken by the user in the target language is generated using the adapted speech recognizer.
2. The computer-implemented method of claim 1 , wherein: The operations further include: generating a translated transcript of the transcript of the utterance in the target language; and The common words or phrases for the particular domain are identified from the translated transcript.
3. The computer-implemented method of claim 2, wherein: Common words or phrases that are identified include: identifying n-grams that appear in the translated transcript and the frequency of occurrence of each n-gram; and A subset of the n-grams that occur in the translated transcript more than a threshold frequency of occurrence is identified.
4. The computer-implemented method of claim 3, wherein: The operations further include, for each n-gram in the subset of n-grams, increasing the likelihood that the language model selects the subset of n-grams by a larger amount based on a larger difference between the frequency of occurrence of the n-gram and the threshold frequency of occurrence.
5. The computer-implemented method of claim 2, wherein: The operations further include: receiving additional transcriptions of additional utterances, the additional utterances being received by a computing device operating in a domain other than the domain and being in the source language; generating additional translated transcripts of the additional transcripts; and identifying words or phrases that occur more frequently in the translated transcript than in the appended translated transcript, Wherein adapting the speech recognizer comprises biasing the language model for the target language by increasing the likelihood that the language model selects words or phrases that have a higher frequency of occurrence in the translated transcription than in the additional translated transcription.
6. The computer-implemented method of claim 2, wherein: The operations further include: receiving a grammar of the target language, Wherein adapting the speech recognizer comprises biasing the language model for the target language by increasing the likelihood that the language model selects a grammar that includes the word or phrase identified from the translated transcription.
7. The computer-implemented method of claim 1 , wherein: The speech recognizer for the target language includes an acoustic model and a language model for the target language; as well as Adapting the speech recognizer includes biasing the language model for the target language by increasing the likelihood that the language model selects words or phrases that are common for the particular domain identification.
8. The computer-implemented method of claim 6, wherein: Generating the transcription of the utterance spoken by the user in the target language comprises: identifying phonemes of the utterance spoken by the user in the target language using the acoustic model; and The transcription of the utterance is generated based on the identified phonemes of the utterance using the biased language model.
9. The computer-implemented method of claim 6, wherein: The language model for the target language comprises a universal language model; and The biased language model for the target language comprises a language model specific to the digital assistant system operating in the particular domain.
10. The computer-implemented method of claim 1, wherein: The data processing hardware resides on a user device or a server.
11. A system comprising: Data processing hardware; as well as Memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising: receiving a transcription of an utterance in a source language received by a computing device operating in a particular domain; using the received transcription of the utterance in the source language, adapting a speech recognizer for a target language for use by a digital assistant system operating in the specific domain to increase the likelihood that the speech recognizer generates a transcription that includes common words or phrases identified for the specific domain, wherein the digital assistant system does not have any access to training data for the target language; receiving audio data corresponding to an utterance spoken by a user in the target language, the utterance being directed to the digital assistant operating in the particular domain; and A transcription of the utterance spoken by the user in the target language is generated using the adapted speech recognizer.
12. The system according to claim 11, wherein: The operations further include: generating a translated transcript of the transcript of the utterance in the target language; and The common words or phrases for the particular domain are identified from the translated transcript.
13. The system according to claim 12, wherein: Common words or phrases that are identified include: identifying n-grams that appear in the translated transcript and the frequency of occurrence of each n-gram; and A subset of the n-grams that occur in the translated transcript more than a threshold frequency of occurrence is identified.
14. The system according to claim 13, wherein: The operations further include, for each n-gram in the subset of n-grams, increasing the likelihood that the language model selects the subset of n-grams by a larger amount based on a larger difference between the frequency of occurrence of the n-gram and the threshold frequency of occurrence.
15. The system of claim 12, wherein: The operations further include: receiving additional transcriptions of additional utterances, the additional utterances being received by the computing device operating in a domain other than the domain and being in the source language; generating additional translated transcripts of the additional transcripts; and identifying words or phrases that occur more frequently in the translated transcript than in the appended translated transcript, Wherein adapting the speech recognizer comprises biasing the language model for the target language by increasing the likelihood that the language model selects words or phrases that have a higher frequency of occurrence in the translated transcription than in the additional translated transcription.
16. The system of claim 12, wherein: The operations further include: receiving a grammar of the target language, Wherein adapting the speech recognizer comprises biasing the language model for the target language by increasing the likelihood that the language model selects a grammar that includes the word or phrase identified from the translated transcription.
17. The system of claim 11, wherein: The speech recognizer for the target language includes an acoustic model and a language model for the target language; and Adapting the speech recognizer includes biasing the language model for the target language by increasing the likelihood that the language model selects words or phrases that are common for the particular domain identification.
18. The system of claim 16, wherein: Generating the transcription of the utterance spoken by the user in the target language comprises: identifying phonemes of the utterance spoken by the user in the target language using the acoustic model; and The transcription of the utterance is generated based on the identified phonemes of the utterance using the biased language model.
19. The system of claim 16, wherein: The language model for the target language comprises a universal language model; and The biased language model for the target language comprises a language model specific to the digital assistant system operating in the particular domain.
20. The system of claim 11, wherein: The data processing hardware resides on a user device or a server.
