Updating automatic speech recognition system using anti-contextual examples

Abstract

A method (400) for personalizing a speech recognition model (132) using anti-contextual examples includes receiving audio data (104) corresponding to an utterance (102) spoken by a user (10), and processing the audio data using the speech recognition model to generate a transcription (106) of the utterance. The transcription includes a misrecognized phrase (144) that is misrecognized in the transcription by the speech recognition model. The method also includes receiving user-corrected text (141), the user-corrected text including a corrected phrase (146) that replaces the misrecognized phrase that is misrecognized in the transcription. Based on the misrecognized phrase, the method includes generating an anti-contextual example (305), the anti-contextual example including an anti-contextual text (310) containing the misrecognized phrase paired with text-to-speech (TTS) audio data (315) corresponding to a synthetic speech representation of the anti-contextual text. The method also includes personalizing the speech recognition model based on the anti-contextual example.

Description

Technical Field

[0001] The present disclosure relates to generating and using anti-context examples to update automatic speech recognition (ASR) systems. Background Art

[0002] ASR systems provide technology that is commonly used in mobile devices and / or other devices. Generally speaking, ASR systems attempt to provide an accurate transcription of what a user says to the device. However, in some cases, the ASR system generates a transcription that may not match what the user intended or actually said. In these cases, the user can correct the transcription by providing user input to correct the transcription. Summary of the invention

[0003] One aspect of the present disclosure provides a method for updating an ASR system using anti-context examples, which when executed on data processing hardware causes the data processing hardware to perform operations. The operations include receiving audio data corresponding to an utterance spoken by a user, and processing the audio data using a speech recognition model to generate a transcription of the utterance. Here, the transcription includes an incorrectly identified phrase that is incorrectly identified by the speech recognition model in the transcription. The operations also include receiving a user-corrected text, which includes a corrected phrase that replaces the incorrectly identified phrase that is incorrectly identified in the transcription. The operations further include generating anti-context examples based on the incorrectly identified phrases. Here, the anti-context examples include anti-context text, which includes an incorrectly identified phrase paired with text-to-speech (TTS) audio data corresponding to a synthetic speech representation of the anti-context text. The operations also include personalizing the speech recognition model based on the anti-context examples.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation further includes displaying the transcription on a graphical user interface of a user device. In some examples, receiving user-corrected text includes: receiving user input indicating selection of an incorrectly recognized phrase in the transcription displayed on the graphical user interface, and receiving input of the user-corrected text from the user. In these examples, receiving input of user-corrected text includes receiving text input of user-corrected text provided by the user. Alternatively, receiving input of user-corrected text includes receiving streaming audio captured by the user device corresponding to one or more letters of the user speaking the corrected phrase.

[0005] In some implementations, generating anti-context examples includes determining anti-context text containing text corrected by the user using a language model based on the text corrected by the user. In these implementations, the operation further includes providing the anti-context text to the TTS system. Here, the TTS system is configured to convert the anti-context text into TTS audio data including a synthetic speech representation of the anti-context text. In some examples, the operation also includes determining the domain of the utterance spoken by the user. Here, the language model is trained on training text utterances associated with the domain of the utterance spoken by the user. In these examples, the domain of the utterance includes a long form speech domain, and the training text utterance is sampled from at least one of an input method editor (IME) text source or a dictation text source. Alternatively, the domain of the utterance includes a query domain; and the training text utterance is sampled from a query log.

[0006] In some implementations, personalizing the speech recognition model includes training the speech recognition model on anti-context examples by teaching the speech recognition model to learn how to predict anti-context text from TTS audio data. In some examples, the operation further includes personalizing the speech recognition model by training the speech recognition model on positive training examples including user-corrected text paired with audio data to teach the speech recognition model to learn how to predict user-corrected text from audio data corresponding to utterances spoken by the user. In some implementations, personalizing the speech recognition model includes executing an evaluation routine to test the performance of the speech recognition model by: processing the TTS audio data using the speech recognition model to generate a speech recognition result; and determining whether the speech recognition result meets the acceptance criteria based on the anti-context text; and one of the following: accepting the speech recognition model when the speech recognition result meets the acceptance criteria, or rejecting the speech recognition model when the speech recognition result fails to meet the acceptance criteria.

[0007] Another aspect of the present disclosure provides a system for updating an ASR system using anti-context examples. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations comprising: receiving audio data corresponding to an utterance spoken by a user; and processing the audio data using a speech recognition model to generate a transcription of the utterance. Here, the transcription includes an incorrectly identified phrase that is incorrectly identified in the transcription by the speech recognition model. The operation also includes receiving a user-corrected text, the user-corrected text including a corrected phrase that replaces the incorrectly identified phrase in the transcription. The operation further includes generating an anti-context example based on the incorrectly identified phrase. Here, the anti-context example includes an anti-context text that contains an incorrectly identified phrase paired with text-to-speech (TTS) audio data corresponding to a synthetic speech representation of the anti-context text. The operation also includes personalizing the speech recognition model based on the anti-context example.

[0008] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation further includes displaying the transcription on a graphical user interface of a user device. In some examples, receiving user-corrected text includes: receiving user input indicating user input selecting an incorrectly recognized phrase in the transcription displayed on the graphical user interface, and receiving input of the user-corrected text from the user. In these examples, receiving input of user-corrected text includes receiving text input of user-corrected text provided by the user. Alternatively, receiving input of user-corrected text includes receiving streaming audio captured by the user device corresponding to one or more letters of the user speaking the corrected phrase.

[0009] In some implementations, generating anti-context examples includes determining anti-context text containing text corrected by the user using a language model based on the text corrected by the user. In these implementations, the operation further includes providing the anti-context text to the TTS system. Here, the TTS system is configured to convert the anti-context text into TTS audio data including a synthetic speech representation of the anti-context text. In some examples, the operation also includes determining the domain of the utterance spoken by the user. Here, the language model is trained on training text utterances associated with the domain of the utterance spoken by the user. In these examples, the domain of the utterance includes a long form speech domain, and the training text utterance is sampled from at least one of an input method editor (IME) text source or a dictation text source. Alternatively, the domain of the utterance includes a query domain; and the training text utterance is sampled from a query log.

[0010] In some implementations, personalizing the speech recognition model includes training the speech recognition model on anti-context examples by teaching the speech recognition model to learn how to predict anti-context text from TTS audio data. In some examples, the operation further includes personalizing the speech recognition model by training the speech recognition model on positive training examples including user-corrected text paired with audio data to teach the speech recognition model to learn how to predict user-corrected text from audio data corresponding to utterances spoken by the user. In some implementations, personalizing the speech recognition model includes executing an evaluation routine to test the performance of the speech recognition model by: processing the TTS audio data using the speech recognition model to generate a speech recognition result; and determining whether the speech recognition result meets the acceptance criteria based on the anti-context text; and one of the following: accepting the speech recognition model when the speech recognition result meets the acceptance criteria, or rejecting the speech recognition model when the speech recognition result fails to meet the acceptance criteria.

[0011] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram of an example system for updating an automatic speech recognition (ASR) system using anti-context examples.

[0013] FIG. 2A to FIG. 2C is a schematic diagram of a user providing input to correct a transcription.

[0014] Figure 3 is a schematic diagram depicting an anti-context example generator for generating anti-context examples.

[0015] Figure 4 is a flow diagram of an exemplary operational arrangement of a method of generating and using anti-context examples to update a speech recognition system.

[0016] Figure 5 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0017] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0018] As automatic speech recognition (ASR) systems continue to provide more accurate transcriptions of what users say, ASR systems are becoming more and more popular in client devices. In addition, in some cases, ASR systems can generate inaccurate transcriptions when they misidentify what the user actually or expected to say. This may usually happen when words are acoustically similar or when the user says a unique, uncommon, or rare word that the ASR system does not know. For example, a user may say a correct name, such as "Khe Chai", but the ASR system may not recognize it because the correct name does not exist in the training data used to train the ASR system. Therefore, the ASR system may mistakenly transcribe what the user said into another word or phrase that is acoustically similar to "Khe Chai" (e.g., "kitchen"). In some examples, the user uses a client device to correct the original transcription (e.g., inputting the corrected text via a keyboard, microphone, etc. of the client device). For example, the client device may display the transcription on a graphical user interface, and the user may select an incorrectly recognized phrase (e.g., "kitchen") in the original transcription (e.g., "My name is kitchen") displayed on the graphical user interface, and thereafter provide user-corrected text including the corrected phrase (e.g., "Khe Chai") to be used to replace the incorrectly recognized phrase in the corrected transcription (e.g., "My name is Khe Chai") displayed on the graphical user interface.

[0019] A particular difficulty for ASR systems is how to use these user corrections to generate more accurate transcriptions for subsequent utterances. For example, if a user repeatedly says the correct name "Khe Chai" in subsequent utterances, causing the ASR system to repeatedly misidentify the correct name as "kitchen", the user may lose trust in the ASR system. Therefore, in some examples, the speech recognition model can be updated using training examples containing corrected transcriptions or at least corrected phrases and captured audio data representing what the user said, so as to better personalize the speech recognition model to recognize proper nouns spoken by the user, so that the speech recognition model can learn to recognize or better recognize corrected phrases (e.g., proper nouns). Such training examples are referred to as "positive examples" in this article because they actively train or strengthen the ability of the speech recognition model to correctly recognize corrected phrases.

[0020] However, personalizing a speech recognition model based on user corrections of incorrectly transcribed utterances may have the unintended consequence of "overlearning" the speech recognition model, in which case the speech recognition model loses the ability to correctly transcribe a spoken utterance that actually includes a common phrase (e.g., "kitchen") that was previously incorrectly recognized and then corrected and replaced with a corrected phrase (e.g., "Khe Chai"). For example, personalizing a speech recognition model to accurately recognize an utterance uttered by a user that contains the proper noun "Khe Chai" instead of the acoustically similar phrase "kitchen" may cause the speech recognition model to incorrectly recognize the phrase / word "kitchen" as "Khe Chai", even though the user actually uttered the phrase "kitchen". That is, just because a user intended to convey "Khe Chai" in some utterance does not mean that the user will never intend to convey an acoustically similar term, such as "kitchen", in another utterance later.

[0021] The implementation of this paper involves preventing a speech recognition model from over-learning from a text corrected by a user by utilizing an anti-context example containing an incorrectly recognized phrase (e.g., "kitchen") and a text-to-speech (TTS) audio data corresponding to a synthetic speech representation of the incorrectly recognized phrase. It is apparent that the speech recognition model can also be updated on the TTS audio data paired with the anti-context text containing the incorrectly recognized phrase to help reduce the possibility that the speech recognition model erroneously transcribes the speech that the user actually says containing the incorrectly recognized phrase. In some cases, the text for such an anti-context example (e.g., a longer phrase including an incorrectly recognized phrase, such as "I am in the kitchen") does not need to be related to the context, domain, meaning, intent, etc. of the original utterance (e.g., "My name is Khe Chai"), so such text is referred to herein as "anti-context text". In addition, training examples based on such "anti-context texts" based on incorrectly recognized phrases are correspondingly referred to herein as "anti-context examples" to distinguish them from positive training examples based on text corrected by the user.

[0022] The implementation of this article is more specifically related to a system and method for generating and using anti-context examples to prevent a speech recognition model from over-biasing the recognition of terms / phrases corrected by a user in a transcription of a previously spoken utterance by the user. Specifically, a speech recognition model executed on a computing device processes audio data corresponding to an utterance spoken by the user to generate a transcription including a phrase incorrectly recognized by the speech recognition model. The computing device can display the transcription including the incorrectly recognized phrase on a graphical user interface, and then receive a user-corrected text, which includes a corrected phrase that replaces the incorrectly recognized phrase, to provide a corrected transcription now containing the corrected phrase to be displayed in the graphical user interface. Although the user-corrected text and corresponding audio data can be used to personalize the speech recognition model to accurately transcribe subsequent utterances containing the corrected phrase, the computing device also further personalizes the speech recognition model on one or more anti-context examples to mitigate the possibility that the speech recognition model is over-biased in recognizing the corrected phrase in subsequent utterances spoken by the user after the phrase that was previously incorrectly recognized as the incorrectly recognized phrase is actually spoken. Here, when a user provides user-corrected text to replace an incorrectly identified phrase that was incorrectly identified in a transcription of an utterance spoken by the user, a computing device generates a corresponding anti-context example based on the incorrectly identified phrase, wherein the anti-context example includes an anti-context text that contains the incorrectly identified phrase paired with TTS audio data corresponding to a synthetic speech representation of the anti-context text. As used herein, personalizing a speech recognition model based on anti-context examples may include training the speech recognition model on anti-context examples by teaching the speech recognition model to learn how to predict anti-context text from TTS audio. Additionally or alternatively, personalizing a speech recognition model based on anti-context examples may include using the anti-context examples to evaluate the performance of the speech recognition model to determine whether the speech recognition model can accurately transcribe the TTS audio data corresponding to the synthetic speech representation of the anti-context text.

[0023] Figure 1 An example of a system 100 for performing ASR on recorded audio data 104 corresponding to an utterance 102 (e.g., a query, a command, etc.) spoken by a user 10 is shown. The system 100 includes a client device 110. In some examples, the client device 110 communicates with a computing system 120 via a network 115. The computing system 120 can be a distributed system with scalable elastic resources (e.g., a cloud computing environment). The resources include computing resources 122 (e.g., data processing hardware) and / or storage resources 124 (e.g., memory hardware). The network 115 can be wired, wireless, or a combination thereof, and can include private networks and / or public networks, such as the Internet.

[0024] In some examples, computing system 120 receives or otherwise obtains audio data 104 from client device 110 , and computing system 120 processes audio data 104 using ASR to generate an original transcription 106 of utterance 102 based on audio data 104 .

[0025] Figure 1 Operations (A) to (F) are shown to illustrate data flow. As described herein, computing system 120 performs operations (B) to (F). However, it should be understood that, in addition to or in lieu of computing system 120 performing operations (B) to (F), client device 110 may also perform one of the operations. In some examples, client device 110 performs a first portion of the operations (e.g., operations (A), (B), and (C)) and computing system 120 performs a second portion of the operations (e.g., operations (D) to (F)), or vice versa. Furthermore, in some examples, another computing system (not shown for clarity of illustration) different from client device 110 and computing system 120 performs operation (F).

[0026] The client device 110 includes data processing hardware 112 and memory hardware 113. The client device 110 may include one or more audio capture devices (e.g., microphones) 114 for capturing utterances 102 from the user 10 and converting them into audio data 104 (e.g., digital data or electrical signals). In some examples, the microphone 114 is separate from the client device 110 and communicates with the client device 110 to provide the utterances 102 to the client device 110. The client device 110 may be any computing device capable of communicating with the computing system 120 via a network 115. The client device 110 includes, but is not limited to, desktop computing devices and mobile computing devices such as laptop computers, tablet computers, smart phones, smart keyboards, digital assistants, smart speakers / displays, smart appliances, in-vehicle infotainment systems, Internet of Things (IoT) devices, and wearable computing devices (e.g., headphones and / or watches).

[0027] exist Figure 1 In the example of FIG. 1 , during operation (A), user 10 speaks utterance 102, and microphone 114 of client device 110 captures spoken utterance 102. In this example, utterance 102 includes user 10 saying “My name is Khe Chai”. In some examples, client device 110 transmits audio data 104 corresponding to utterance 102 captured by microphone 114 to computing system 120 via network 115. In other examples, in addition to or instead of transmitting audio data 104 to computing system 120, client device 110 processes audio data 104 locally.

[0028] During operation (B), the computing system 120 (or the client device 110) processes the audio data 104 to generate an original transcription 106 of the utterance 102. For example, the computing system 120 can execute the speech recognizer 130 (e.g., using the speech recognition model 132) to produce the original transcription 106 (e.g., "My name is kitchen"). Notably, the original transcription 106 includes an incorrectly recognized phrase (e.g., "kitchen") that was incorrectly recognized by the speech recognizer 130, rather than the phrase actually spoken by the user 10 ("Khe Chai").

[0029] In some implementations, the speech recognizer 130 includes an end-to-end (E2E) speech recognition model configured to receive the audio data 104 and generate a word grid. Specifically, the E2E speech recognition model processes the audio data 104 to generate a corresponding likelihood score for each of a plurality of candidate hypotheses in the word grid. In some examples, the speech recognizer 130 includes a separate acoustic model, a language model, and / or a pronunciation model. The speech recognizer 130 may share an acoustic model and a language model with an additional hypothesis scorer (e.g., an acoustic model and a language model), or have an independent acoustic model and a language model. In some examples, the speech recognizer 130 includes an acoustic model and / or a language model to generate a word grid based on the audio data 104 or otherwise generate a plurality of candidate hypotheses of the utterance 102. Here, the likelihood scores of the plurality of candidate hypotheses may include a combination of an acoustic modeling score from an acoustic model and / or a priori likelihood score from a language model. In other words, the likelihood score includes at least one of an acoustic modeling score output by the acoustic model and / or a priori likelihood score output by the language model. The speech recognizer 130 may identify the highest ranked candidate hypothesis from the plurality of candidate hypotheses in the lattice as the original transcription 106. As used herein, the terms "transcription" and "transcript" may be used interchangeably.

[0030] During operation (C), the computing system 120 (or the client device 110) executes a correction module 140 that generates a corrected transcription 108 in response to one or more user correction inputs 142 indicating selection or identification of an incorrectly recognized phrase 144 (e.g., "kitchen") of the original transcription 106 and a user-corrected text 141 that includes a corrected phrase 146 (e.g., "Khe Chai") to replace the incorrectly recognized phrase 144. The incorrectly recognized phrase 144 may include one or more corresponding words, word fragments, characters / graphemes, numbers, punctuation marks, etc. Similarly, the corrected phrase 146 may include one or more corresponding words, word fragments, characters / graphemes, numbers, punctuation marks, etc. In some examples, the correction module 140 generates the corrected transcription 108 by replacing more than one incorrectly recognized phrase 144 incorrectly recognized by the speech recognizer 130 in the original transcription 106 with the corresponding corrected phrase 146.

[0031] FIG. 2A to FIG. 2C An example is shown in which a user corrects an original transcription 106 containing an incorrectly identified phrase 144 to produce a corrected transcription 108 that includes the corrected phrase 146 instead of the incorrectly identified phrase 144 (see Figure 1 In some implementations, the speech recognizer 130 generates a transcription 106 that includes misrecognized phrases 144 of the utterance 102 spoken by the user 10 .

[0032] Figure 2A Schematic diagram 200a of FIG. 1 shows that microphone 114 of client device 110 captures user 10 speaking utterance 102 “My name is Khe Chai”. Client device 110 converts utterance 102 into audio data 104 and transmits or otherwise provides audio data 104 to speech recognizer 130. Speech recognizer 130 processes audio data 104 to generate raw transcription 106 (e.g., “My name is kitchen”) corresponding to audio data 104. In the example shown, raw transcription 106 represents or includes an erroneous recognition of utterance 102 spoken by user 10. As shown, client device 110 displays raw transcription 106 to user 10 via graphical user interface (GUI) 116. In other examples, client device 110 displays raw transcription 106 to user 10 via data processing hardware 112 ( Figure 1 ) to locally execute a speech recognizer 130 to process the audio data 104 and generate a transcription 106.

[0033] Reference now Figure 2B200b, the user 10 may recognize that the original transcription 106 displayed on the GUI 116 does not match the utterance 102 because the transcription 106 includes the misidentified phrase 144 spoken by the user 10 in the utterance 102. Therefore, the user 10 may provide one or more inputs 142 to the GUI 116 of the client device 110, the one or more inputs indicating the selection or identification of the misidentified phrase 144 in the transcription 106 that was misidentified by the speech recognizer 130. In some examples, the input 142 includes the user 10 providing a touch input to the GUI 116, the touch input selecting the misidentified phrase 144 (e.g., "kitchen") from the transcription 106. The misidentified phrase 144 may include the entire transcription 106 or a portion thereof. In the example shown, the misidentified phrase 144 includes only a portion of the transcription 106. As shown, the client device 110 transmits or otherwise provides the misidentified phrase 144 to the anti-context example generator 300.

[0034] Reference now Figure 2C 200c, the user 10 may replace the incorrectly identified phrase 144 in the original transcription 106 with the user-corrected text including the corrected phrase 146 (e.g., "KheChai") to form the corrected transcription 108. In some examples, the user 10 uses a physical or virtual keyboard 118 of the client device 110 to provide the user-corrected text 141 including the corrected phrase 146. The keyboard 118 may optionally be displayed in response to the client device 110 receiving an input indication from the user 10 ( Figure 2B ). In these examples, the user 10 may type the user-corrected text including the corrected phrase 146 using a physical or virtual keyboard 118 of the client device 110. In other examples, the user 10 enters the user-corrected text of the corrected phrase 146 by speaking to the client device 110. That is, the user 10 may speak each letter of the user-corrected text of the corrected phrase 146 (e.g., "KHE space CHAI"). The client device 110 may receive the user's 10 speech as streaming audio captured by the client device 110, and process the streaming audio using, for example, speech recognition to recognize one or more spoken letters of the user-corrected text. After receiving the user-corrected text 141 including the corrected phrase 146, the client device 110 may replace the incorrectly recognized phrase 144 with the corrected phrase 146 to generate a corrected transcription 108 representing an accurate transcription of the speech 102.

[0035] Return to reference Figure 1During operations (D) and (E), the computing system 120 executes the anti-context example generator 300 to generate one or more anti-context examples 305, 305a-n based on the misrecognized phrases 144 in the original transcription 106. Each anti-context example 305 includes a corresponding anti-context text 310, 310a-n based on the misrecognized phrases 144 paired with corresponding TTS audio data 315, 315a-n corresponding to a synthesized speech representation of the corresponding anti-context text 310, 310a-n.

[0036] In the example shown, the client device 110 and / or the computing system 120 may store the generated anti-context examples 305 on one or more local or remote storage resources 150 (e.g., residing on the memory hardware 113 of the client device 110 and / or the memory hardware 124 of the computing system 120) for subsequent retrieval and use by the model updater 160 to personalize, update, adjust, train, etc., a speech recognition model (e.g., the speech recognition model 132) during operation (F). In some examples, the anti-context example model updater 160 uses the anti-context examples 305 to update the speech recognition model 132 in real time during operation (F).

[0037] In some examples, the model updater 160 performs an evaluation routine to test the performance of the personalized speech recognition model 132 by processing the TTS audio 315 of the anti-context example 305 using the speech recognition model 132 to generate one or more speech recognition results. The model updater 160 can then determine whether the speech recognition results meet the acceptance criteria based on the anti-context text. When the speech recognition results meet the acceptance criteria, the module updater 160 accepts the personalized speech recognition model 132. When the speech recognition results do not meet the acceptance criteria, the module updater 160 can reject the personalized speech recognizer model 132.

[0038] In some examples, the client device 110 or the computing system 120 generates a positive training example 170 that includes the recorded audio data 104 and the corrected transcription 108 or a portion thereof (e.g., the corrected phrase 146). Similar to the anti-context example 305, the client device 110 and / or the computing system 120 can store the positive training example 170 on one or more storage resources 150 for subsequent retrieval and use by the model updater 160 to personalize, update, adjust, train, etc. a speech recognition model (e.g., the speech recognition model 132). In some examples, the model updater 160 uses the positive training example 170 to update the speech recognition model 132 in real time.

[0039] As referenced below Figure 3Described in more detail, the anti-context example generator 300 includes a text generator module 320 for generating an anti-context text 310 at operation (D), and a TTS system 335 for generating a corresponding TTS audio 315 at operation (E). The client device 110 can execute the text generator module 320 locally to generate the anti-context text 310, and then transmit the anti-context text 310 to the TTS system 335 executed on the computing system 120 to generate the TTS audio 315. However, both the text generator module 320 and the TTS system 335 can be executed locally on the client device 110, or remotely on the computing system 120, without departing from the scope of the present disclosure.

[0040] Reference now Figure 3 , during operation (D), the text generator module 320 generates an anti-context text 310 (e.g., “I am in the kitchen”) based on the misrecognized phrase 144 (e.g., “kitchen”) extracted from the original transcription 106. The text generator module 320 can utilize a language model 330 that receives the misrecognized phrase 144 and generates an anti-context text 310 that includes the misrecognized phrase 144. Notably, the anti-context text 310 output from the language model 330 includes a text utterance that includes the misrecognized phrase 144. Although the illustrated example depicts the text generator module 320 generating only one instance of the anti-context text 310 for simplicity, the text generator module 320 can generate multiple instances of the anti-context text 310, 310a-n, each instance including a respective sentence that includes the misrecognized phrase 144 (e.g., “I am in the kitchen” and “The stove is in the kitchen”).

[0041] In some implementations, the text generator module 320 generates the anti-context text 310 based on another misrecognized phrase (e.g., "keychain") extracted from another speech recognition hypothesis in the lattice of speech recognition hypotheses predicted by the speech recognition model 132 for the input audio data 104 representing the utterance (e.g., "My name is Khe Chai"). Each hypothesis in the lattice corresponds to a possible transcription of the utterance and can be assigned a confidence score by the speech recognizer 130. For example, Figure 1The original transcription 106 depicted in FIG. 1 with the misrecognized phrase 144 (“kitchen”) may include the speech recognition hypothesis with the highest score / confidence in the speech recognition hypothesis lattice, while one or more other hypotheses in the lattice with lower scores / confidence may include other possible misrecognized phrases. Thus, the text generator module 320 may generate the anti-context text 310 based on the misrecognized phrase extracted from any speech recognition hypothesis in the lattice.

[0042] In some examples, the text generator module 320 generates the anti-context text 310 based on context information 325 (e.g., an application identifier, a device identifier, a user identifier, etc.), which indicates a domain associated with the utterance 102 (e.g., a query, a command, etc.). In these examples, the text generator module 320 can select a language model 330 associated with the domain indicated by the context information 325 from a plurality of language models 330, 330a-n, each associated with a different corresponding domain, for use in generating the anti-context text 310. Thus, the anti-context text 310 includes a text utterance of a sentence / query / command containing an incorrectly recognized phrase 144, which is associated with a domain used by the speech recognition model 132 to better personalize the speech recognition model 132 and prevent it from over-learning. For example, the text generator module 320 can determine the domain based on an application identifier of an application (e.g., a digital assistant) to which the recognition utterance 102 relates. For example, the utterance 102 may be “Hey Google, call Khe Chai on mobile,” instructing the user 10 to invoke a digital assistant application, or the utterance 102 may be “send the following message to Mom [contents of message].” The context information 325 may also indicate the length of the original utterance 102 for use by the text generator module 320 to distinguish between generating anti-context text 310 associated with a long-form utterance (i.e., the long-form voice domain) or a short query utterance (e.g., the query domain).

[0043] The language model 330 can be trained on corresponding training text utterances associated with different domains, contexts, etc. For example, the language model 330 can be trained using training text utterances sampled from at least one of an input method editor (IME) text source, a dictation text source (e.g., a text or email message, a free-form dictation, a reminder, etc.), or a query log (e.g., a query entered into a digital assistant or voice search engine, such as "What is the temperature", or a query entered into a navigation application, etc.). In some implementations, the language model 330 is anonymously trained on training text utterances sampled from a source that does not include any data extracted from or otherwise associated with a user.

[0044] In other implementations, at least one language model 330 is trained on training text utterances sampled from query logs (voice commands) or other typing history (search engine queries) entered by the user 10. In these implementations, the user 10 explicitly consents to share personal data for use by the language model 330 to generate anti-context text 310 to better personalize the speech recognition model 132 for the user. The user 10 can revoke consent to share personal data at any time. In some examples, when the client device 110 determines that the text generator module 320 and the TTS system 335 are executed entirely on the device (as well as the model updater 160 and the speech recognition model 132), the client device 110 allows the text generator module 320 to utilize the language model 330 trained on the user's personal training text utterances. In this way, neither the anti-context text 310 nor the resulting TTS audio 315 are shared over the network, but remain entirely on the device so that all data of the user 10 remains private and secure.

[0045] During operation (E), the anti-context example generator 300 executes the TTS system 335 to generate TTS audio data 315 corresponding to the synthesized speech representation of the anti-context text 310 generated by the text generator module 320 during operation (D). That is, the TTS system 335 can convert the anti-context text 310 into TTS audio data 315. In some examples, the anti-context text 310 includes a phoneme sequence input to the TTS system 335 for conversion into TTS audio data 315. In some examples, the TTS module 335 is conditioned on the speaker embedding 340 associated with the user 10 to allow the TTS system 335 to generate TTS audio data 315 with speaker characteristics associated with the user 10. In these examples, the TTS system 335 can use the context information 325 (e.g., application identifier, device identifier, user identifier, etc.) to uniquely identify the user 10 and obtain the speaker embedding 340 of the user 10.

[0046] Figure 4 4 is a flow chart of an exemplary arrangement of operations for a method 400 of generating and using anti-context examples to personalize a speech recognition model. Figure 1 The data processing hardware 122 of the computing system 120 of the embodiment of the present invention can perform the operations of the method 400 by executing instructions stored on the memory hardware 520 (e.g., the memory hardware 113, 124). At operation 402, the method 400 includes receiving audio data 104 corresponding to the utterance 102 spoken by the user 10. At operation 404, the method 400 includes processing the audio data 104 using the speech recognition model 132 to generate the original transcription 106 of the utterance 102.

[0047] At operation 406, the method 400 includes receiving user-corrected text including a corrected phrase 146 that replaces an incorrectly recognized phrase 144 that was incorrectly recognized in the transcription 106. Here, the method 400 may receive one or more user inputs 142 ( Figure 1 and Figure 2A 2D ), and provide user-corrected text that includes a corrected phrase 146 that replaces the misidentified phrase 144 in the corrected transcription 108 of the utterance 102 .

[0048] At operation 408, the method 400 includes generating one or more anti-context examples 305 based on the misrecognized phrase 144. Here, each anti-context example 305 includes anti-context text 310 generated based on the misrecognized phrase 144 paired with TTS audio data 315 corresponding to a synthesized speech representation of the anti-context text 310.

[0049] At operation 410, method 400 includes personalizing speech recognition model 132 based on anti-context example 305. In some examples, personalizing speech recognition model 132 includes model updater 160 ( Figure 1) The speech recognition model 132 is trained on one or more anti-context examples 305 by teaching the speech recognition model 132 to learn how to predict the anti-context text 310 from the TTS audio data 315. For example, the anti-context text 310 can be used as a ground truth for the ASR results predicted by the speech recognition model 132 based on processing the TTS audio data 315, whereby the model updater 160 can update the parameters of the speech recognition model 132 based on the anti-context text 310 and the predicted ASR results using supervised learning techniques, such as stochastic gradient descent via back-propagation of the training loss. Thus, the model updater 160 can update the parameters of the speech recognition model 132 based on the anti-context examples 305 to mitigate over-learning of the speech recognition model 132, which can occur when the model 132 is updated based on the user-corrected text 142 replacing a previously misrecognized phrase 144 in the original transcription 106 with a corrected phrase 146. Additionally, personalizing the speech recognition model 132 may include training the speech recognition model 132 on positive training examples that include user-corrected text paired with the audio data 104 to teach the speech recognition model 132 to learn how to predict the user-corrected text from the audio data corresponding to the utterance 102 spoken by the user 10.

[0050] In some additional examples, personalizing the speech recognition model 132 includes executing an evaluation routine to test the performance of the speech recognition model 132 by: processing the TTS audio data 315 using the speech recognition model 132 to generate a speech recognition result; and determining whether the speech recognition result meets the acceptance criteria based on the anti-context text 310. Here, the anti-context text 310 may include a baseline truth of the speech recognition result output by the speech recognition model 132 based on processing the TTS audio data 315, so that the word error rate can be determined and compared with the acceptance criteria corresponding to the word error rate threshold. In these examples, when the speech recognition result meets the acceptance criteria, the evaluation routine accepts the speech recognition model 132. Here, the speech recognition model 132 can generate an accurate speech recognition result from the TTS audio data 315 matched with the anti-context text 310 to indicate that the acceptance criteria are met, thereby indicating that the speech recognition model 132 does not lose performance due to over-learning when recognizing the speech including the incorrectly recognized phrase 144. On the other hand, when the speech recognition result fails to meet the acceptance criteria, the evaluation routine rejects the speech recognition model 132. For example, when the speech recognition model 132 fails to recognize the incorrectly recognized phrase 144 in the TTS audio data 315, the speech recognition result may fail to meet the acceptance criteria, thereby indicating that the performance of the speech recognition model 132 has declined due to overlearning. In a scenario where the evaluation routine rejects the speech recognition model, the model updater 160 can train / update the parameters of the speech recognition model 132 based on the anti-context examples 305 as described above. For example, the rejection of the speech recognition model 132 by the evaluation routine may trigger the anti-context example generator 300 to generate additional anti-context examples 305 based on the incorrectly recognized phrase 144 for the model updater 160 to use to update / train the speech recognition model 132 to learn (or relearn) how to predict the anti-context text containing the incorrectly recognized phrase 144 from the corresponding TTS audio data 144. Figure 5 is a schematic diagram of an example computing device 500 that may be used to implement the systems and methods described in this document. For example, computing device 500 may be used to implement client device 110 and / or computing system 120. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0051] The computing device 500 includes a processor 510 that can be used to implement the data processing hardware 112 and / or 122, a memory 520 that can be used to implement the memory hardware 113 and / or 124, a storage device 530 that can be used to implement the memory hardware 113 and / or 124, a high-speed interface / controller 540 connected to the memory 520 and the high-speed expansion port 550, and a low-speed interface / controller 560 connected to the low-speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and can be mounted on a common motherboard or otherwise installed as appropriate. The processor 510 can process instructions for execution within the computing device 500, including instructions stored in the memory 520 or on the storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 580 coupled to the high-speed interface 540). In other implementations, multiple processors and / or multiple buses and multiple memories and types of memories may be used as appropriate. Also, multiple computing devices 500 may be connected, with each device providing portions of the necessary operations (e.g., as a server group, blade server group, or multi-processor system).

[0052] Memory 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 520 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0053] Storage device 530 is capable of providing mass storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various implementations, storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as memory 520, storage device 530, or a memory on processor 510.

[0054] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or a router, for example, through a network adapter.

[0055] The computing device 500 may be implemented in a variety of different forms, as shown. For example, the computing device may be implemented as a standard server 500a or multiple times in a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0056] Various implementations of the systems and techniques described herein may be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that may be executed and / or interpreted on a programmable system that includes at least one programmable processor that may be either special purpose or general purpose and that may be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0057] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0058] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, a processor suitable for executing a computer program includes both a general-purpose microprocessor and a special-purpose microprocessor, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CDROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0059] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and possibly a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0060] Unless expressly stated to the contrary, "or" refers to an inclusive or and not an exclusive or. For example, "A, B, or C" refers to any combination or subset of A, B, and C, such as: (1) A alone; (2) B alone; (3) C alone; (4) A and B; (5) A and C; (6) B and C; and (7) A and B and C. Similarly, the phrase "at least one of A or B" is intended to refer to any combination or subset of A and B, such as: (1) at least one A; (2) at least one B; and (3) at least one A and at least one B. As used herein, the phrase "at least one of A and B" is intended to refer to any combination or subset of A and B, such as: (1) at least one A; (2) at least one B; and (3) at least one A and at least one B.

[0061] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method (400) executed on data processing hardware (510) causing the data processing hardware (510) to perform operations comprising: Receiving audio data (104) corresponding to an utterance (102) spoken by a user (10); processing the audio data (104) using a speech recognition model (132) to generate a transcription (106) of the utterance (102), the transcription (106) including misrecognized phrases (144) misrecognized in the transcription (106) by the speech recognition model (132); receiving user-corrected text (141), the user-corrected text comprising a corrected phrase (146) replacing the misidentified phrase (144) misidentified in the transcription (106); generating, based on the misidentified phrase (144), a counter-context example (305), the counter-context example (305) comprising a counter-context text (310) comprising the misidentified phrase (144) paired with text-to-speech (TTS) audio data (315) corresponding to a synthesized speech representation of the counter-context text (310); and The speech recognition model (132) is personalized based on the anti-context examples (305).

2. The computer-implemented method (400) of claim 1, wherein the operations further comprise: displaying the transcription (106) on a graphical user interface of a user device (110), The receiving of the user-corrected text (141) comprises: receiving user input indicating selection of the misidentified phrase (144) in the transcription (106) displayed on the graphical user interface (116); and An input of the user-corrected text (141) is received from the user (10).

3. A computer-implemented method (400) as described in claim 2, wherein receiving the input of the user-corrected text (141) includes receiving text input of the user-corrected text (141) provided by the user (10).

4. The computer-implemented method (400) of claim 2, wherein receiving the input of the user-corrected text (141) comprises receiving streaming audio captured by the user device (110) corresponding to one or more letters of the user (10) speaking the corrected phrase (146).

5. The computer-implemented method (400) of any one of claims 1 to 4, wherein generating the anti-context example (305) comprises: Based on the user-corrected text (141), using a language model (330), determining the anti-context text (310) that includes the user-corrected text (141); and The anti-context text (310) is provided to a TTS system (335), which is configured to convert the anti-context text (310) into the TTS audio data (315) including the synthesized speech representation of the anti-context text (310).

6. The computer-implemented method (400) of claim 5, wherein the operations further comprise: determining the domain of the utterance (102) uttered by the user (10), The language model (330) is trained on training text utterances associated with the domain of the utterances (102) spoken by the user (10).

7. The computer-implemented method (400) of claim 6, wherein: The domain of the utterance (102) includes a long-form speech domain; and The training text utterances are sampled from at least one of an input method (400) editor (IME) text source or a dictation text source.

8. The computer-implemented method (400) of claim 6, wherein: The domain of the utterance (102) includes a query domain; and The training text utterances are sampled from query logs.

9. A computer-implemented method (400) as described in any one of claims 1 to 8, wherein personalizing the speech recognition model (132) includes training the speech recognition model (132) on the anti-context examples (305) by teaching the speech recognition model (132) to learn how to predict the anti-context text (310) from the TTS audio data (315).

10. A computer-implemented method (400) as described in any one of claims 1 to 9, wherein the operation further includes personalizing the speech recognition model (132) by training the speech recognition model (132) on positive training examples (170) including the user-corrected text (141) paired with the audio data (104) to teach the speech recognition model (132) to learn how to predict the user-corrected text (141) from the audio data (104) corresponding to the utterance (102) spoken by the user (10).

11. The computer-implemented method (400) of any one of claims 1 to 10, wherein personalizing the speech recognition model (132) comprises executing an evaluation routine to test the performance of the speech recognition model (132) by: Processing the TTS audio data (315) using the speech recognition model (132) to generate a speech recognition result; determining whether the speech recognition result meets acceptance criteria based on the anti-context text (310); as well as One of the following: When the speech recognition result meets the acceptance criteria, accepting the speech recognition model (132); or When the speech recognition results fail to meet the acceptance criteria, the speech recognition model is rejected (132).

12. A system (100), comprising: Data processing hardware (510); as well as Memory hardware (520) in communication with the data processing hardware (510) and storing instructions that, when executed on the data processing hardware (510), cause the data processing hardware (510) to perform operations comprising: Receiving audio data (104) corresponding to an utterance (102) spoken by a user (10); processing the audio data (104) using a speech recognition model (132) to generate a transcription (106) of the utterance (102), the transcription (106) including misrecognized phrases (144) misrecognized in the transcription (106) by the speech recognition model (132); receiving user-corrected text (141), the user-corrected text comprising a corrected phrase (146) replacing the misidentified phrase (144) misidentified in the transcription (106); generating a counter-context example (305) based on the misidentified phrase (144), the counter-context example (305) comprising a counter-context text (310) comprising the misidentified phrase (144) paired with text-to-speech (TTS) audio data (104) corresponding to a synthesized speech representation of the counter-context text (310); and The speech recognition model (132) is personalized based on the anti-context examples (305).

13. The system (100) of claim 12, wherein the operations further comprise: displaying the transcription (106) on a graphical user interface (116) of a user device (110), The receiving of the user-corrected text (141) comprises: receiving user (10) input indicating selection of the misidentified phrase (144) in the transcription (106) displayed on the graphical user interface (116); and An input of the user-corrected text (141) is received from the user (10).

14. The system (100) of claim 13, wherein receiving the input of the user-corrected text (141) comprises receiving text input of the user-corrected text (141) provided by the user (10).

15. The system (100) of claim 13, wherein receiving the input of the user-corrected text (141) comprises receiving streaming audio captured by the user device (110) corresponding to one or more letters of the user (10) speaking the corrected phrase (146).

16. The system (100) of any one of claims 12 to 15, wherein generating the anti-context example comprises: Based on the user-corrected text (141), using a language model (330), determining the counter-context text containing the user-corrected text (141); and The anti-context text is provided to a TTS system (335), which is configured to convert the anti-context text into the TTS audio data (315) including the synthesized speech representation of the anti-context text.

17. The system (100) of claim 16, wherein the operations further comprise: determining the domain of the utterance (102) uttered by the user (10), The language model (330) is trained on training text utterances associated with the domain of the utterances (102) spoken by the user (10).

18. The system (100) of claim 17, wherein: The domain of the utterance (102) includes a long-form speech domain; and The training text utterances are sampled from at least one of an input method (400) editor (IME) text source or a dictation text source.

19. The system (100) of claim 17, wherein: The domain of the utterance (102) includes a query domain; and The training text utterances are sampled from query logs.

20. The system (100) of any one of claims 12 to 19, wherein personalizing the speech recognition model (132) comprises training the speech recognition model (132) on the anti-context examples by teaching the speech recognition model (132) to learn how to predict the anti-context text from the TTS audio data (315).

21. A system (100) as described in any one of claims 12 to 20, wherein the operation further includes personalizing the speech recognition model (132) by training the speech recognition model (132) on positive training examples (170) including the user-corrected text (141) paired with the audio data (104) to teach the speech recognition model (132) to learn how to predict the user-corrected text (141) from the audio data (104) corresponding to the utterance (102) spoken by the user (10).

22. The system (100) of any one of claims 12 to 21, wherein personalizing the speech recognition model (132) comprises executing an evaluation routine to test the performance of the speech recognition model (132) by: Processing the TTS audio data (315) using the speech recognition model (132) to generate a speech recognition result; determining whether the speech recognition result meets acceptance criteria based on the anti-context text; as well as One of the following: When the speech recognition result meets the acceptance criteria, accepting the speech recognition model (132); or When the speech recognition results fail to meet the acceptance criteria, the speech recognition model is rejected (132).

Images