Lattice Audio Correction

Abstract

The method (500) includes receiving audio data (112) corresponding to a query, and processing the audio data to generate a plurality of candidate hypotheses (135), each represented by a respective sequence of hypothesis terms. For each candidate hypothesis, the method includes determining whether the sequence of hypothesis terms includes a source phrase (126) from a list of phrase correction pairs (125). Each phrase correction pair includes a corresponding misrecognized source phrase and a corresponding target phrase (128) that replaces the source phrase. When the respective sequence of hypothesis terms includes the source phrase, the method includes generating a corresponding additional candidate hypothesis (145) that replaces the source phrase. The method also includes generating a transcription (175) of the query by ranking the plurality of candidate hypotheses and each corresponding additional candidate hypothesis, and selecting a highest ranking one of the plurality of candidate hypotheses and each additional candidate hypothesis.

Description

[Technical field]

[0001] This disclosure relates to lattice audio correction. [Background technology]

[0002] Automatic speech recognition (ASR) systems provide technology that is commonly used on mobile and other devices. In general, ASR systems attempt to provide an accurate transcription of what a user speaks into a mobile device. In some instances, the ASR system generates an inaccurate transcription that does not match what the user spoke. In these instances, the user may correct the inaccurate transcription by providing user input on the mobile device that corrects the transcription. One of the challenges for ASR systems includes how to best utilize user input to correct the inaccurate transcription to generate accurate transcriptions in the future. Summary of the Invention

[0003] One aspect of the disclosure provides a computer implementation that, when executed on the data processing hardware, causes the data processing hardware to perform operations including receiving audio data corresponding to a query spoken by a user and using a speech recognizer to process the audio data to generate a plurality of candidate hypotheses. Each candidate hypothesis corresponds to a candidate transcription of the query and is represented by a respective sequence of hypothesis terms. For each candidate hypothesis, the method also includes determining whether the respective sequence of hypothesis terms includes a source phrase from the list of phrase correction pairs, and when the respective sequence of hypothesis terms includes the source phrase, generating a corresponding additional candidate hypothesis that replaces the source phrase in the respective sequence of hypothesis terms with the corresponding target phrase. Each phrase correction pair in the list of phrase correction pairs includes a corresponding source phrase that was misrecognized in a corresponding previous transcription transcribed by the speech recognizer of a previous utterance made by the user, and a corresponding target phrase that matches the user correction that replaces the misrecognized source phrase in the corresponding previous transcription transcribed by the speech recognizer. The method also includes generating a transcription of the query spoken by the user by ranking the plurality of candidate hypotheses and each generated corresponding additional candidate hypothesis, and selecting one of the highest ranking among the plurality of candidate hypotheses and each generated corresponding additional candidate hypothesis.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations further include, for each phrase correction pair in the list of phrase correction pairs, identifying an n-gram of the language model that includes a corresponding source phrase, cloning a corresponding new n-gram that replaces the corresponding source phrase in the identified n-gram with the corresponding target phrase, modifying the language model, and determining a corresponding prior likelihood score for each additional candidate hypothesis using a modified language model configured to receive each additional candidate hypothesis as an input, where modifying the language model includes adding to the language model the new n-gram that replaces the source phrase with the corresponding target phrase, and conditioning the language model to determine that a prior likelihood score of a unigram of the target phrase is higher than a unigram of the source phrase. In these implementations, ranking the multiple candidate hypotheses and each additional candidate hypothesis is based on the corresponding prior likelihood score determined for each additional candidate hypothesis.

[0005] In an additional embodiment, the operations further include, for each phrase correction pair in the list of phrase correction pairs, obtaining an original sequence of n-grams representing a corresponding prior transcription that misrecognized the corresponding source phrase, obtaining a corrected sequence of n-grams that replaces the source phrase in the original sequence of n-grams with the corresponding target phrase to modify the language model, and determining a corresponding prior likelihood score for each additional candidate hypothesis using the modified language model configured to receive each additional candidate hypothesis as an input, where the original sequence of n-grams includes the corresponding source phrase and one or more other terms that precede and / or follow the corresponding source phrase in the corresponding prior transcription, and the corrected sequence of n-grams includes the target phrase and the same one or more other terms that precede and / or follow the corresponding source phrase in the corresponding prior transcription. Here, modifying the language model includes adding the original sequence of n-grams and the corrected sequence of n-grams to the language model and conditioning the language model to determine that a prior likelihood score of a number of n-grams from the corrected sequence of n-grams containing the target phrase is higher than the same number of n-grams from the original sequence of n-grams containing the source phrase. In these embodiments, ranking the plurality of candidate hypotheses and each additional candidate hypothesis is based on the corresponding prior likelihood score determined for each additional candidate hypothesis. Optionally, the original sequence of n-grams and the corrected sequence of n-grams each may further include n-grams representing sentence boundaries of a corresponding previous transcription transcribed by the speech recognizer for a previous utterance made by the user.

[0006] In the aforementioned implementation, the margin between the prior likelihood scores determined by the language model for a number of n-grams from the corrected sequence of n-grams and the same number of n-grams from the original sequence of n-grams may increase as the numbers of n-grams from the corrected sequence and the original sequence of n-grams increase. Additionally or alternatively, conditioning the language model may further include conditioning the language model to determine that a prior likelihood score of a first number of n-grams from the sequence of n-grams including the target phrase will be lower than a higher second number of n-grams from the sequence of n-grams including the target phrase.

[0007] In some examples, the operations further include, for each of the plurality of candidate hypotheses generated by the speech recognizer, obtaining a corresponding likelihood score assigned by the speech recognizer to the corresponding candidate hypothesis, and after generating each additional candidate hypothesis, determining a corresponding likelihood score for each generated additional candidate hypothesis using an additional hypothesis scorer, where the ranking of the plurality of candidate hypotheses and each generated corresponding additional candidate hypothesis is based on the corresponding likelihood scores assigned by the speech recognizer to the plurality of candidate hypotheses and the corresponding likelihood score determined for each additional candidate hypothesis using the additional hypothesis scorer. In these examples, the additional hypothesis scorer includes at least one of an acoustic model configured to process the audio data to determine an acoustic modeling score for a portion of the audio data including either the source phrase or the target phrase, or a language model configured to receive each additional candidate hypothesis as an input and determine a corresponding prior likelihood score for each additional candidate hypothesis. Here, the corresponding likelihood score determined for each additional candidate hypothesis may be based on at least one of the acoustic modeling score or the corresponding prior likelihood score determined for the additional candidate hypothesis.

[0008] The language model may include an auxiliary language model external to the speech recognizer or an internal language model integrated with the speech recognizer. Additionally, the speech recognizer may include an end-to-end speech recognition model configured to generate a corresponding likelihood score for each of the plurality of candidate hypotheses. Alternatively, the speech recognizer may include an acoustic model and a language model, and the corresponding likelihood score assigned by the speech recognizer to each of the plurality of candidate hypotheses may be based on at least one of an acoustic modeling score output by the acoustic model or a corresponding prior likelihood score output by the language model.

[0009] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving audio data corresponding to a query spoken by a user and processing the audio data using a speech recognizer to generate a plurality of candidate hypotheses. Each candidate hypothesis corresponds to a candidate transcription of the query and is represented by a respective sequence of hypothesis terms. For each candidate hypothesis, the method also includes determining whether the respective sequence of hypothesis terms includes a source phrase from the list of phrase correction pairs and, when the respective sequence of hypothesis terms includes the source phrase, generating a corresponding additional candidate hypothesis that replaces the source phrase in the respective sequence of hypothesis terms with the corresponding target phrase. Each phrase correction pair in the list of phrase correction pairs includes a corresponding source phrase that was misrecognized in a corresponding previous transcription transcribed by the speech recognizer of a previous utterance made by the user and a corresponding target phrase that matches the user correction to replace the misrecognized source phrase in the corresponding previous transcription transcribed by the speech recognizer. The method also includes generating a transcription of the query spoken by the user by ranking the plurality of candidate hypotheses and each corresponding additional candidate hypothesis generated, and selecting a highest ranking one of the plurality of candidate hypotheses and each corresponding additional candidate hypothesis generated.

[0010] This aspect may include one or more of the following optional features. In some implementations, the operations further include, for each phrase correction pair in the list of phrase correction pairs, identifying an n-gram of the language model that includes a corresponding source phrase, cloning a corresponding new n-gram that replaces the corresponding source phrase in the identified n-gram with the corresponding target phrase, modifying the language model, and determining a corresponding prior likelihood score for each additional candidate hypothesis using a modified language model configured to receive each additional candidate hypothesis as an input. Here, modifying the language model includes adding to the language model the new n-gram that replaces the source phrase with the corresponding target phrase, and conditioning the language model to determine that a prior likelihood score of a unigram of the target phrase is higher than a unigram of the source phrase. In these implementations, ranking the multiple candidate hypotheses and each additional candidate hypothesis is based on the corresponding prior likelihood score determined for each additional candidate hypothesis.

[0011] In an additional embodiment, the operations further include, for each phrase correction pair in the list of phrase correction pairs, obtaining an original sequence of n-grams representing a corresponding prior transcription that misrecognized the corresponding source phrase, obtaining a corrected sequence of n-grams that replaces the source phrase in the original sequence of n-grams with the corresponding target phrase to modify the language model, and determining a corresponding prior likelihood score for each additional candidate hypothesis using the modified language model configured to receive each additional candidate hypothesis as an input, where the original sequence of n-grams includes the corresponding source phrase and one or more other terms that precede and / or follow the corresponding source phrase in the corresponding prior transcription, and the corrected sequence of n-grams includes the target phrase and the same one or more other terms that precede and / or follow the corresponding source phrase in the corresponding prior transcription. Here, modifying the language model includes adding the original sequence of n-grams and the corrected sequence of n-grams to the language model and conditioning the language model to determine that a prior likelihood score of a number of n-grams from the corrected sequence of n-grams containing the target phrase is higher than the same number of n-grams from the original sequence of n-grams containing the source phrase. In these embodiments, ranking the plurality of candidate hypotheses and each additional candidate hypothesis is based on the corresponding prior likelihood score determined for each additional candidate hypothesis. Optionally, the original sequence of n-grams and the corrected sequence of n-grams each may further include n-grams representing sentence boundaries of a corresponding previous transcription transcribed by the speech recognizer for a previous utterance made by the user.

[0012] In the aforementioned implementation, the margin between the prior likelihood scores determined by the language model for a number of n-grams from the corrected sequence of n-grams and the same number of n-grams from the original sequence of n-grams may increase as the numbers of n-grams from the corrected sequence and the original sequence of n-grams increase. Additionally or alternatively, conditioning the language model may further include conditioning the language model to determine that a prior likelihood score of a first number of n-grams from the sequence of n-grams including the target phrase will be lower than a higher second number of n-grams from the sequence of n-grams including the target phrase.

[0013] In some examples, the operations further include, for each of the plurality of candidate hypotheses generated by the speech recognizer, obtaining a corresponding likelihood score assigned by the speech recognizer to the corresponding candidate hypothesis, and after generating each additional candidate hypothesis, determining a corresponding likelihood score for each generated additional candidate hypothesis using an additional hypothesis scorer, where the ranking of the plurality of candidate hypotheses and each generated corresponding additional candidate hypothesis is based on the corresponding likelihood scores assigned by the speech recognizer to the plurality of candidate hypotheses and the corresponding likelihood score determined for each additional candidate hypothesis using the additional hypothesis scorer. In these examples, the additional hypothesis scorer includes at least one of an acoustic model configured to process the audio data to determine an acoustic modeling score for a portion of the audio data including either the source phrase or the target phrase, or a language model configured to receive each additional candidate hypothesis as an input and determine a corresponding prior likelihood score for each additional candidate hypothesis. Here, the corresponding likelihood score determined for each additional candidate hypothesis may be based on at least one of the acoustic modeling score or the corresponding prior likelihood score determined for the additional candidate hypothesis.

[0014] The language model may include an auxiliary language model external to the speech recognizer or an internal language model integrated with the speech recognizer. Additionally, the speech recognizer may include an end-to-end speech recognition model configured to generate a corresponding likelihood score for each of the plurality of candidate hypotheses. Alternatively, the speech recognizer may include an acoustic model and a language model, and the corresponding likelihood score assigned by the speech recognizer to each of the plurality of candidate hypotheses may be based on at least one of an acoustic modeling score output by the acoustic model or a corresponding prior likelihood score output by the language model.

[0015] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0016] [Figure 1] 1 is a schematic diagram of an exemplary system for automatic speech recognition using phrase correction pairs. [Figure 2A] FIG. 2 is a schematic diagram showing an example of a word lattice. [Figure 2B] FIG. 2 is a schematic diagram showing an example of a word lattice. [Diagram 3] FIG. 2 is a schematic diagram illustrating a language model for scoring candidate hypotheses. [Figure 4A] FIG. 13 is a schematic diagram of a user providing an input indication to correct a misrecognized previous transcription. [Figure 4B] FIG. 13 is a schematic diagram of a user providing an input indication to correct a misrecognized previous transcription. [Figure 4C] FIG. 13 is a schematic diagram of a user providing an input indication to correct a misrecognized previous transcription. [Diagram 5] 4 is a flow chart of an exemplary arrangement of operations of a method of lattice audio correction. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Like reference symbols in the various drawings indicate like elements.

[0018] As automatic speech recognition (ASR) systems continue to provide more accurate transcriptions of what users say, they are becoming more prevalent on client devices. Yet, in some instances, ASR systems generate inaccurate transcriptions that misrecognize what a user actually said. This often occurs when words are acoustically similar or when a user speaks unique words that are unknown to the ASR system. For example, "I say" and "Ice age" sound very similar, and it can be difficult for an ASR system to disambiguate which phrase the user intended to convey. In some instances, if a client device inaccurately transcribes "I say" when the user actually intended to convey "Ice age," the user can correct the transcription by using the client device (e.g., by entering the correct transcription using the client device's keyboard).

[0019] One of the particular challenges for ASR systems is how to leverage these user corrections to generate more accurate transcriptions for the user's subsequent utterances. For example, an ASR system cannot simply transcribe "I say" as "Ice age" every time an utterance made by a user is similar to "I say" and "Ice age." That is, just because a user intended to say "Ice age" previously does not mean that the user will never intend to say "I say" later. Furthermore, if the user continues to say "Ice age," but the ASR system repeatedly generates the inaccurate transcription "I say," the user may lose trust in the ASR system.

[0020] Embodiments herein are directed to a system and method for lattice speech correction. In particular, a computing system receives audio data of an utterance made by a user, and the computing system processes the audio data to generate candidate hypotheses. The computing system then determines whether any words or phrases in the candidate hypotheses contain a source phrase (e.g., a previously misrecognized phrase) transcribed by the computing system. When the computing system determines that the candidate hypotheses contain the source phrase, the computing system generates additional candidate hypotheses that replace the source phrase with a correction phrase (e.g., a target phrase). The computing system then generates a likelihood score for each of the candidate hypotheses and the additional candidate hypotheses, where the likelihood score indicates a probability that the hypothesis is a correct transcription of the utterance made by the user. The computing system generates a transcription of the utterance made by the user by selecting the highest likelihood score of either the candidate hypothesis or the additional candidate hypothesis.

[0021] 1 illustrates an example of a system 100 for automatic speech recognition (ASR) of an utterance 101 made by a user 10 using audio data 112 corresponding to the utterance (i.e., query) 101. The system 100 includes a client device 110, a computing system 120, and a network 118. The computing system 120 may be a distributed system (e.g., a cloud computing environment) with scalable elastic resources. The resources include computing resources 122 (e.g., data processing hardware) and / or storage resources 124 (e.g., memory hardware). The network 118 may be wired, wireless, or a combination thereof, and may include a private network and / or a public network such as the Internet.

[0022] In some examples, the computing system 120 receives the audio data 112 from the client device 110, and the computing system 120 processes the audio data 112 to generate a plurality of candidate hypotheses 135 for the utterance 101 based on the audio data 112, where each candidate hypothesis corresponds to a candidate transcription of the utterance 101 and is represented by a respective sequence of hypothesis terms. As described in more detail below, for each candidate hypothesis 135, the computing system 120 is configured to determine whether the respective sequence of hypothesis terms includes a source phrase 126 from the correction phrase list 125. When the respective sequence of hypothesis terms includes a source phrase 126 from the correction phrase list 125, the computing system 120 is configured to generate an additional candidate hypothesis 145 that replaces the source phrase 126 with a corresponding target phrase 128. The computing system 120 then generates the transcription 175 by selecting a highest-ranked hypothesis from the plurality of candidate hypotheses 135 or the additional candidate hypothesis 145.

[0023] 1 illustrates operations (A)-(G) showing the flow of data. As described herein, computing system 120 performs operations (B)-(G). However, it is understood that client device 110 may also perform operations (B)-(G) in addition to or instead of computing system 120 performing the operations. In some examples, client device 110 performs a first portion of the operations (e.g., operations (B)-(D)) and computing system 120 performs a second portion of the operations (e.g., operations (E)-(G)), or vice versa.

[0024] The client device 110 includes data processing hardware 114 and memory hardware 116. The client device 110 may include one or more audio capture devices (e.g., microphone(s)) 103 for capturing speech 101 from the user 10 and converting it into audio data 112 (e.g., electrical signals). In some examples, the microphone 103 is separate from the client device 110 and communicates with the client device 110 to provide the recorded speech 101 to the client device 110. The client device 110 may be any computing device capable of communicating with the computing system 120 via the network 118. The client device 110 includes desktop and mobile computing devices, such as, but not limited to, laptops, tablets, smartphones, smart speakers / displays, smart appliances, Internet of Things (IoT) devices, and wearable computing devices (e.g., headsets and / or watches).

[0025] In the example of FIG. 1, during stage (A), a user 10 makes an utterance 101, and a microphone 103 of a client device 110 records the utterance 101. In this example, the utterance 101 includes the user 10 saying "call Beshad on mobile". The client device 110 transmits audio data 112 corresponding to the utterance 101 recorded by the microphone 103 to a computing system 120 via a network 118. During stage (B), the computing system 120 processes the audio data 112 to generate a plurality of candidate hypotheses 135, 135a-n, where each candidate hypothesis 135 corresponds to a candidate transcription of the utterance 101 and is represented by a respective sequence of hypothesis terms. For example, the computing system 120 may execute a speech recognizer module 130 (e.g., an automatic speech recognition (ASR) module) to produce a word lattice 200 indicating a plurality of possible candidate hypothesis transcriptions 135 for the utterance 101 based on the audio data 112. The speech recognizer module 130 may evaluate potential paths through the word lattice 200 to determine a number of candidate hypotheses 135 .

[0026] Figure 2A shows an example of a word lattice 200, 200a that may be provided by the speech recognizer module 130 of Figure 1. The word lattice 200a represents multiple possible combinations of words that may form different candidate hypotheses 135 for the utterance 101.

[0027] The word lattice 200a includes one or more nodes 202a-g corresponding to possible boundaries between words. The word lattice 200a includes a number of edges 204a-l of possible words of candidate hypotheses arising from the word lattice 200a. Additionally, each of the edges 204a-l may have one or more weights or probabilities that the edge is a correct edge from the corresponding node. The weights are determined by the speech recognizer module 130 and may be based, for example, on the confidence of the match between the speech data and the word of that edge and the degree to which the word fits grammatically and / or lexically with other words in the word lattice 200a.

[0028] For example, initially, the most likely path through the word lattice 200a (e.g., the most likely candidate hypothesis 135) may include edges 204c, 204e, 204i, 204k, with the text "we're coming about 11:30". The second best path through the word lattice 200a (e.g., the second best candidate hypothesis 135) may include edges 204d, 204h, 204j, 204l, with the text "deer hunting scouts 7:30".

[0029] Each pair of nodes may have one or more paths that correspond to alternative words in the various candidate hypotheses 135. For example, the initial most likely path between the pair of nodes starting at node 202a and ending at node 202c is edge 204c "we're", which has alternative paths including edges 204a, 204b "we are" and edge 204d "deer".

[0030] Figure 2B is an example of a hierarchical word lattice 200, 200b that may be provided by the speech recognizer module 130 of Figure 1. The word lattice 200b includes nodes 252a-l that represent words that make up the various candidate hypotheses 135 for the utterance 101. The edges between the nodes 252a-l indicate that the possible candidate hypotheses 135 are (1) nodes 252c, 252e, 252i, and 252k, “we're coming about 11:30”, (2) nodes 252a, 252b, 252e, 252i, and 252k, “we are coming about 11:30”, (3) nodes 252a, 252b, 252f, 252g, 252i, and 252k, “we are come at about 11:30”, (4) nodes 252d, 252f, 252g, 252i, and 252k, “deer come at about 11:30”, and (5) nodes 252d, 252h, 252j, and 252k, “deer hunting scouts”. 11:30", and (6) nodes 252d, 252h, 252j, and 252l "deer hunting scouts 7:30".

[0031] Again, edges between nodes 242a-l may have associated weights or probabilities based on confidence of the speech recognition (e.g., candidate hypotheses) and grammar / lexical analysis of the resulting text. In this example, "we are coming about 11:30" may be the current best hypothesis, and "deer hunting scouts 7:30" may be the next best hypothesis. One or more divisions 354a-d that group together words and their alternatives may be created in the word lattice 200b. For example, division 254a includes the word "we're" and the alternatives "we are" and "deer". Division 252b includes the word "coming" and the alternatives "come at" and "hunting". Division 254c includes the word "about" and the alternative "Scouts", and division 254d includes the word "11:30" and the alternative "7:30".

[0032] Referring again to FIG. 1 , the speech recognizer module 130 may generate a plurality of candidate hypotheses 135 from the word lattice 200. That is, the speech recognizer module 130 generates a likelihood score 155 for each of the candidate hypotheses 135 in the word lattice 200. Each likelihood score 155 indicates a probability that the candidate hypothesis 135 is correct (e.g., matches the utterance 101). In some implementations, the speech recognizer module 130 includes an end-to-end (E2E) speech recognition model configured to receive the audio data 112 and generate the word lattice 200. In particular, the E2E speech recognition model processes the audio data 112 to generate a corresponding likelihood score 155 for each of the plurality of candidate hypotheses 135 from the word lattice 200. In some examples, the speech recognizer module 130 includes separate acoustic models, language models, and / or pronunciation models. The speech recognizer module 130 may share acoustic and language models (eg, acoustic model 301 and language model 302) with the additional hypothesis scorer 300 or may have standalone acoustic and language models.

[0033] In some examples, the speech recognizer module 130 includes an acoustic model and / or a language model that generates a word lattice 200 or otherwise generates a plurality of candidate hypotheses 135 for the utterance 101 based on the audio data 112. Here, the likelihood scores 155 of the plurality of candidate hypotheses 135 may include a combination of acoustic modeling scores from the acoustic model and / or prior likelihood scores from the language model. In other words, the likelihood scores 155 include at least one of the acoustic modeling scores output by the acoustic model and / or the prior likelihood scores output by the language model.

[0034] During stage (C), the computing system 120 identifies a set of highest-ranking candidate hypotheses 135 from the plurality of candidate hypotheses 135 in the word lattice 200. For example, using the likelihood scores 155 from the speech recognizer module 130, the computing system 120 selects the n candidate hypotheses 135 with the highest likelihood scores 155, where n is an integer. In some examples, the computing system 120 selects the candidate hypotheses 135 with likelihood scores 155 that meet a likelihood score threshold. Optionally, the speech recognizer module 130 may use the likelihood scores 155 to rank the set of highest-ranking candidate hypotheses 135.

[0035] In the illustrated example, the speech recognizer module 130 generates candidate hypotheses 135 for the utterance 101 "call Beshad on mobile" made by the user 10. In this example, the top two candidate transcriptions (e.g., the two most likely to be correct) are selected as a set of highest-ranking candidate hypotheses 135. The highest-ranking candidate hypotheses 135 include a first candidate hypothesis 135 "call best shot on mobile" with a likelihood score 155 of 0.8 and a second candidate hypothesis 135 "goal best shot immobile" with a likelihood score 155 of 0.2. Here, a higher likelihood score 155 indicates a higher confidence that the candidate hypothesis 135 is correct. Notably, none of the highest-ranking candidate hypotheses 135 includes the utterance 101 (e.g., "call Beshad on mobile") actually made by the user 10. Thus, if the computing system 120 were to select any of the highest-ranking candidate hypotheses 135, the transcription 175 output to the user 10 would be inaccurate.

[0036] During stage (D), the computing system 120 executes a lattice augmenter 140 that determines, for each candidate hypothesis 135, whether the respective sequence of hypothesis terms includes a source phrase 126 from a correction phrase list 125. The correction phrase list 125 includes a list of phrase correction pairs. Each phrase correction pair in the correction phrase list 125 includes a source phrase 126 and a corresponding target phrase 128. Thus, the correction phrase list (e.g., a list of phrase correction pairs) 125 includes a plurality of source phrases 126, 126a-n and a plurality of corresponding target phrases 128, 128a-n. In particular, the source phrase 126 represents a misrecognition transcription previously generated by the speech recognizer module 130, and the target phrase 128 represents a correction transcription for the misrecognition transcription.

[0037] 4A-4C illustrate correcting the misrecognized transcription and storing the corrected transcription to generate the correction phrase list 125 (FIG. 1). In some implementations, the speech recognizer module 130 generates a misrecognized transcription 475 that misrecognizes a previous utterance 401 made by the user 10. As used herein, the previous utterance 401 and the misrecognized transcription 475 are received at the client device 110 at a time prior to receiving the transcription 175 and the utterance 101 (FIG. 1). As will become apparent, the misrecognized transcription 475 and the previous utterance 401 can be used to generate the correction phrase list 125 (FIG. 1) to improve speech recognition accuracy when the client device 110 later receives the utterance 101 (FIG. 1).

[0038] 4A shows that the microphone 103 of the client device 110 records the user 10 speaking the previous utterance 401 "call Beshad on mobile". The client device 110 converts the previous utterance 401 into audio data 412 and transmits the audio data 412 to the speech recognizer module 130. The speech recognizer module 130 processes the audio data 412 to generate a misrecognition transcription 475 corresponding to the audio data 412. In the illustrated example, the speech recognizer module 130 generates the misrecognition transcription 475 "call best shot", which is a misrecognition of the previous utterance 401 spoken by the user 10. The client device 110 displays the previous transcription 475 to the user 10 via a graphical user interface (GUI) 111.

[0039] 4B, the user 10 may identify that the misrecognized transcription 475 displayed on the GUI 111 does not match the previous utterance 401. Thus, the user 10 provides an input indication to the GUI 111 of the client device 110 indicating a selection of the misrecognized phrase 425 of the misrecognized transcription 475. In some examples, the input indication includes the user 10 providing a touch input to the GUI 111 selecting the misrecognized phrase 425 from the misrecognized transcription 475. The misrecognized phrase 425 may include the entirety of the misrecognized transcription 475 or a portion thereof. In the example shown, the misrecognized phrase 425 "best shot" includes only the inaccurate portion of the previous transcription 475.

[0040] FIG. 4C shows the user 10 replacing the misrecognized phrase 425 of the misrecognized transcription 475 with the correction phrase 430. In some examples, the user 10 uses the keyboard 119 of the client device 110 to input text to provide the correction phrase 430. Optionally, the keyboard 119 may display in response to the client device 110 receiving an input indication from the user 10 (FIG. 4B). In these examples, the user 10 may type the correction phrase (e.g., "Beshad") using the keyboard 118 of the client device 110. In other examples, the user 10 inputs the correction phrase 430 by speaking to the client device 110. That is, the user 10 may speak each character of the correction phrase 430 (e.g., "BESHAD"). After receiving the correction phrase 430, the client device 110 replaces the misrecognized phrase 425 with the correction phrase 430 to generate a previous correction transcription 476 representing an accurate transcription of the previous utterance 401.

[0041] Thus, the computing system 120 may store the misrecognized phrase 425 and the correction phrase 430 as a phrase correction pair in the correction phrase list 125 as the source phrase 126 and the target phrase 128, respectively. In some examples, the computing system stores the misrecognized transcription 475 and the previous correction transcription 476 as a phrase correction pair in the correction phrase list 125 as the source phrase and the target phrase 128, respectively. The phrase correction pairs may be stored in the memory hardware 116 of the client device 110 and / or in the memory hardware 124 of the computing system 120. The phrase correction pairs may also be stored in the correction history data store 320 (FIG. 3). As will become apparent, the computing system 120 uses the phrase correction pairs to generate an accurate transcription 175 when the computing system 120 detects the source phrase 126 (e.g., best shot) again.

[0042] 1, in some implementations, lattice augmentor 140 is configured to use candidate hypotheses 135 and correction phrase list 125 to generate additional candidate hypotheses 145, 145a-n. In particular, lattice augmentor 140 receives candidate hypotheses 135 and correction phrase list 125 and determines, for each candidate hypothesis 135, whether a respective sequence of hypothesis terms includes one of source phrases 126 from correction phrase list 125. When a respective sequence of hypothesis terms includes one of source phrases 126, lattice augmentor 140 generates additional candidate hypotheses 145 that replace source phrases 126 in each sequence of hypothesis terms with corresponding target phrases 128.

[0043] Continuing with the illustrated example, the lattice augmentor 140 determines whether the first candidate hypothesis 135 "call best shot on mobile" and the second candidate hypothesis "goal best shot immobile" both contain the source phrase "best shot" from the correction phrase list 125. Thus, the lattice augmentor 140 generates a first additional candidate hypothesis 145 "call Beshad on mobile" and a second additional candidate hypothesis 145 "goal Beshad immobile." Here, the lattice augmentor 140 replaces the source phrase 126 "best shot" with the target phrase 128 "Beshad" in each of the additional candidate hypotheses 145.

[0044] In some implementations, the computing system 120 executes an additional hypothesis scorer 300 (also referred to simply as "hypothesis scorer 300"), which is configured to determine a likelihood score 155 for each additional candidate hypothesis 145 generated by the lattice augmentor 140. That is, simply generating an additional candidate hypothesis 145 does not mean that the additional candidate hypothesis 145 contains a correct transcription 175 of the utterance 101. For example, the lattice augmentor 140 may generate an additional candidate hypothesis 145 "What is Hungary's Beshad at the Euro cup" from the candidate hypothesis 135 of "What is Hungary's best shot at the Euro cup" (e.g., replacing the source phrase 126 "best shot" with the target phrase "Beshad"). In this example, the additional candidate hypothesis 145 is the most likely hypothesis to be nonsensical and should have a lower likelihood score 155 than the candidate hypothesis 135. Thus, the computing system 120 selects the hypothesis with the highest likelihood from the candidate hypotheses 135 and the additional candidate hypotheses 145 to maximize the probability of an accurate transcription 175.

[0045] However, in some examples, the hypothesis scorer 300 may not be able to generate an accurate likelihood score 155 for the additional candidate hypotheses 145. That is, because the computing system 120 receives the additional candidate hypotheses 145 from user input, the additional candidate hypotheses 145 may be out-of-domain, out-of-vocabulary (OOV), and / or may include only sparse training data in the computing system 120. Thus, the hypothesis scorer 300 needs to generate a likelihood score 155 that accurately reflects the probability that the additional candidate hypotheses 145 are correct or not.

[0046] The hypothesis scorer 300 includes at least one of an acoustic model 301 and / or a language model 302. In some examples, the language model 302 includes an auxiliary language model external to the speech recognizer module 130. In other examples, the language model 302 includes an internal language model integrated with the speech recognizer module 130. The acoustic model 301 is configured to process the audio data 112 to determine an acoustic modeling score for a portion of the audio data 112 that includes either the source phrase 126 or the target phrase 128. In some examples, the acoustic model 301 compares the acoustic similarity between the portion of the audio data 112 that includes the source phrase 126 and the portion of the candidate hypothesis 135 that includes the source phrase 126. In other examples, the acoustic model compares the acoustic similarity between the portion of the audio data 112 that includes the target phrase 128 and the portion of the additional candidate hypothesis 145 that includes the target phrase 128.

[0047] Thus, the acoustic modeling score may indicate whether the portion of the audio data 112 containing the source phrase 126 or the target phrase 128 is acoustically similar to the candidate hypothesis 135 or the additional candidate hypothesis 145. In some examples, the hypothesis scorer 300 cannot determine the acoustic modeling score of the additional candidate hypothesis 145. For example, when the target phrase 128 of the additional candidate hypothesis 145 is an OOV phrase, the hypothesis scorer 300 cannot compare the portion of the audio data 112 to the OOV phrase. In these examples, the hypothesis scorer 300 assumes that the source phrase 126 and the target phrase 128 are acoustically similar and relies solely on the language model 302.

[0048] In some implementations, the language model 302 of the hypothesis scorer 300 is configured to receive as input each additional candidate hypothesis 145 and determine a prior likelihood score 304 corresponding to each additional candidate hypothesis 145. The prior likelihood score 304 may indicate the probability of the additional candidate hypothesis 145 based on probability data received by the language model 302 120 indicating the frequency with which different sequences of terms occur. For example, the language model 302 may assign a higher prior likelihood score 304 to the additional candidate hypothesis "three blind mice" than to "tree blind mice" because the former linguistic sequence occurs more frequently in the corpus of utterances 101 than the latter.

[0049] In some implementations, the target phrase 128 is an OOV phrase of the language model 301. Thus, for each phrase correction pair in the correction phrase list 125, the language model 302 identifies an n-gram in the language model 302 that includes the corresponding source phrase 126. Here, the language model 302 clones a corresponding new n-gram that replaces the corresponding source phrase 126 in the identified n-gram with the target phrase 128. That is, the language model 302 creates a new n-gram for the target phrase 128 using existing n-grams in the language model 302. In these implementations, the computing system 120 modifies the language model 302 by adding the new n-gram of the target phrase 128 to the language model 302 and conditioning the language model 302 to determine that the prior likelihood score of the n-gram of the target phrase 128 is higher than the n-gram of the source phrase 126.

[0050] For example, the computing system 120 adds an n-gram for the target phrase 128 "Beshad" to the language model 302 that contains a slightly higher probability than the existing n-gram for the source phrase "best shot." The n-gram for the target phrase 128 contains a slightly higher probability due to a user input indication of replacing the source phrase 126 with the target phrase 128. However, in some examples, the language model 302 cannot determine whether the intent of the user 10 was to speak "Beshad" or "best shot" by simply analyzing only the source phrase 126 and the target phrase 128. For example, "Beshad" can be either a standalone name or a unique name (e.g., a movie name) such as "Call Beshad." When "Call Beshad" is a unique name, "Beshad" itself may not have any particular significance. Thus, the language model 302 may add additional adjacent context (e.g., additional n-grams) to the target phrase 128 to increase the conditional probability of predicting whether the user 10 intended the source phrase 126 or the target phrase 128. In simple terms, if the target phrase 128 contains adjacent terms that are generally adjacent to the target phrase 128, then the probability that the user 10 intended the target phrase 128 is greater than the source phrase 126.

[0051] 3, in some examples, the language model 302 includes an n-gram language model that uses conditional probabilities to predict successive terms in a sequence of n-grams based on the n or n-1 terms preceding and / or the n or n+1 terms following the target phrase 128. In the example shown, the language model includes increased conditional probabilities 310, where the probability that the user 10 intends to convey the target phrase 128 increases as the number of n-grams increases. For example, a sequence with one n-gram is a unigram (e.g., one word), a sequence with two n-grams is a bigram (e.g., two words), and a sequence with three n-grams is a trigram (e.g., three words).

[0052] In the illustrated example, the increased conditional probability 310 includes a prior likelihood score 304 of a unigram with the target phrase 128 (e.g., “Beshad”) that has a slightly higher probability than a prior likelihood score 304 of a unigram with the source phrase 126 (e.g., “best shot”). The probability is only slightly higher because the language model 302 does not know the context (e.g., other words) around the target phrase 128 and source phrase 126.

[0053] The increased conditional probability 310 also includes a prior likelihood score 304 of a bigram with the target phrase 128 (e.g., “call Beshad”) that has a higher probability than a prior likelihood score 304 of a bigram with the source phrase 126 (e.g., “call best shot”). Here, the margin between the prior likelihood score 304 of the target phrase 128 and the source phrase 126 of the bigram (e.g., higher probability) is larger than the margin between the prior likelihood score 304 of the target phrase 128 and the source phrase 126 of the unigram (e.g., slightly higher probability). This is because the number of n-grams following the target phrase 128 and the source phrase 126 increases. The increased number of n-grams provides increased context to the language model 302, which can more accurately predict whether the user 10 intended the target phrase 128 or the source phrase 126. Simply put, because there are words in additional candidate hypotheses 145 that commonly contain the term "Beshad," there is a greater likelihood that user 10 intended to convey "Beshad" rather than "best shot."

[0054] Furthermore, the increased conditional probability 310 also increases the likelihood of a target phrase 128 (e.g., " <s>The trigram prior likelihood scores 304 with the source phrase 126 (e.g., <s>In this example, the probability of the trigram with "Call Beshad" is much higher than the prior likelihood score 304. <s>indicates the beginning of a sentence to provide additional context. Here, the margin between the prior likelihood score 304 of the target phrase 128 and the source phrase 126 of the trigram (e.g., much higher probability) is larger than the margin between the prior likelihood score 304 of the target phrase 128 and the source phrase 126 of the bigram and unigram. This is because the number of n-grams following the target phrase 128 and the source phrase 126 increases. That is, the trigrams provide more context to the language model 302, allowing the language model 302 to more accurately predict the probability that the user 10 intends to convey the target phrase 128 or the source phrase 126.

[0055] As shown in FIG. 3, in some implementations, the language model 302 uses an original sequence of n-grams 326 and a corrected sequence of n-grams 328 in addition to the new n-grams. That is, for each phrase correction pair in the correction phrase list 125, the language model 302 may obtain a sequence of n-grams 325 from the correction history data store 320. The correction history data store 320 stores a previous transcription 475 and a previous corrected transcription 476, as illustrated in FIG. 4. The sequence of n-grams 325 includes the original sequence of n-grams 326 representing a corresponding previous transcription 475 (FIG. 4) that misrecognized the corresponding source phrase 126, and a corrected sequence of n-grams 328 (e.g., a previous corrected transcription 476 (FIG. 4)) that replaces the source phrase 126 in the original sequence of n-grams 326 with the corresponding target phrase 128. Here, the original sequence of n-grams 326 includes a source phrase 126 and one or more other terms that precede and / or follow the source phrase 126 in the previous transcription 475 (FIG. 4). The corrected sequence of n-grams includes a target phrase 128 and the same one or more other terms that precede and / or follow the corresponding source phrase 126 in the previous transcription 475 (FIG. 4).

[0056] In the example shown, the original sequence of n-grams 326 is <s> Call the best shot< / s> " and the corrected sequence of n-grams 328 is " <s> Call Beshad< / s> Here, both sequences of n-grams contain the term "call" that precedes either the source phrase 126 or the target phrase 128. In this example, the original sequence 326 and the corrected sequence 328 of n-grams further contain n-grams that represent sentence boundaries of a previous transcription 475 transcribed by the speech recognizer module 130 of a previous utterance 401 spoken by the user 10, i.e. <s> represents the boundary token at the beginning of a sentence,< / s> represents the boundary token at the end of a sentence.

[0057] The computing system 120 modifies the language model 302 by adding the original sequence of n-grams 326 and the corrected sequence of n-grams 328 to the language model 302 and conditioning the language model 302 to determine that the prior likelihood score 304 of the number of n-grams from the corrected sequence of n-grams 328 will be higher than the same number of n-grams from the original sequence of n-grams 326. That is, if the original sequence of n-grams 326 and the corrected sequence of n-grams 328 are both bigrams, the computing system 120 conditions the language model 302 to determine that the prior likelihood score 304 of the corrected sequence of n-grams 328 will be higher. For example, the language model 320 determines that the likelihood score 155 of the corrected sequence of n-grams 328 “call Beshad” will be higher than the original sequence of n-grams 326 “call best shot.”

[0058] In some examples, the computing system 120 conditions the language model 302 to determine that a prior likelihood score 304 of a first number of n-grams from a sequence of n-grams that includes the target phrase 128 will be lower than a larger second number of n-grams from the sequence of n-grams that includes the target phrase 128. For example, "call Beshad on mobile" should include a higher prior likelihood score 304 because it has a larger number of n-grams than "call Beshad." These additional n-grams adjacent to the target phrase 128 provide more confidence that the target phrase 128 is a correct transcription. That is, a correct transcription is more likely to include the target phrase 128 in this context because the target phrase 128 is more likely to be found in these other words or phrases.

[0059] In some implementations, when language model 302 receives candidate hypothesis 135 "What is Hungary's best shot at the Euro cup," language model 302 determines that the prior likelihood score 304 is high because candidate hypothesis 135 includes a common sequence of n-grams. However, language model 302 may also receive additional candidate hypothesis 145, which replaces source phrase 126 with target phrase 128 to produce "What is Hungary's Beshad at the Euro cup." In this example, language model 302 determines that this sequence of n-grams has not been spoken before by user 10. Thus, language model 302 reduces the number of n-grams that lead to "Beshad" and determines the probability of the reduced sequence of n-grams.

[0060] For example, language model 302 may determine that the sequence of n-grams "What is Hungary's Beshad" has also not been spoken before or has been spoken very rarely by user 10, and thus language model 302 removes another n-gram. For example, language model 302 may determine that the sequence of n-grams "Hungary's Beshad" has also not been spoken before or has been spoken very rarely by user 10, and thus language model 302 removes yet another n-gram. Continuing with the example, language model 302 may determine that target phrase 128 "Beshad" is a new unigram added by user 10 due to a misrecognized transcription. Thus, even though the prior likelihood score 304 of the unigram "Beshad" has a slightly higher probability than the prior likelihood score 304 of the unigram "best shot," the word history of the n-gram sequence of "best shot" in this scenario may mean that "What is Hungary's best shot at the Euro cup" is the most likely transcription.

[0061] 1, the corresponding likelihood score 155 determined for each additional candidate hypothesis 145 is based on at least one of the acoustic modeling score and / or the corresponding prior likelihood score 304 determined for the additional candidate hypothesis 145. During stage (F), the reranker 160 receives the likelihood scores 155 for the plurality of candidate hypotheses 135 from the speech recognizer module 130 and receives the likelihood scores 155 for the additional candidate hypothesis 145 from the hypothesis scorer 300. The reranker 160 is configured to output a reranked result 165 including a ranking of the plurality of candidate hypotheses 135 and the additional candidate hypothesis 145 based on the likelihood scores. In the illustrated example, the reranked result 165 includes the additional candidate hypothesis 145, with a likelihood score 155 of 0.9, considered as the most likely correct transcription 175.

[0062] In stage (G), the computing system 120 is configured to generate a transcription 175 of the utterance 101 made by the user 10 by selecting the highest ranking candidate in the reranked results 165. Thus, the computing system 120 determines whether the candidate hypothesis 135 including the source phrase 126 or the additional candidate hypothesis 145 including the target phrase 128 has the highest likelihood score 155. In the illustrated example, the computing system 120 selects the additional candidate hypothesis 145 "call Beshad on mobile" since it has the highest likelihood score 155 of 0.9. The computing system 120 transmits the transcription 175 to the client device 110 via the network 118 to provide the transcription 175 to the user 10.

[0063] 5 is a flowchart of an exemplary arrangement of operations of a method 500 for performing lattice speech correction. The data processing hardware 114 (FIG. 1) of the client device 110 and / or the data processing hardware 122 (FIG. 1) of the computing system 120 may perform the operations of the method 500 by executing instructions stored in the memory hardware 116, 124. At operation 502, the method 500 includes receiving audio data 112 corresponding to a query (i.e., an utterance) 101 spoken by the user 10. At operation 504, the method 500 includes processing the audio data 112 using the speech recognizer module 130 to generate a plurality of candidate hypotheses 135. Each candidate hypothesis 135 corresponds to a candidate transcription of the query 101 and is represented by a respective sequence of hypothesis terms.

[0064] For each candidate hypothesis 135, the method 500 performs operations 506 and 508. In operation 506, the method 500 includes determining whether the respective sequence of hypothesis terms includes a source phrase 126 from a list of phrase correction pairs (i.e., a correction phrase list) 125. Each phrase correction pair in the list of phrase correction pairs 125 includes a corresponding source phrase 126 that was misrecognized in a corresponding previous transcription 475 transcribed by the speech recognizer module 130 for a previous utterance 401 made by the user 10. Here, each phrase correction pair in the list of phrase correction pairs 125 also includes a corresponding target phrase 128 that matches the user correction replacing the misrecognized source phrase 126 in the corresponding previous transcription 475 transcribed by the speech recognizer module 130. At operation 508, when the respective sequence of hypothesis terms includes a source phrase 126, the method 500 includes generating a corresponding additional candidate hypothesis 145 that replaces the source phrase 126 in the respective sequence of hypothesis terms with the corresponding target phrase 128.

[0065] At operation 510, the method 500 includes ranking the plurality of candidate hypotheses 135 and each corresponding additional candidate hypothesis 145. At operation 512, the method 500 includes generating a transcription 175 of the query 101 spoken by the user 10 by selecting one of the highest rankings of the plurality of candidate hypotheses 135 and each corresponding additional candidate hypothesis 145.

[0066] 6 is a schematic diagram of an example computing device 600 that may be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are intended to be exemplary only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0067] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or otherwise mounted as desired. Processor 610 can process instructions for execution within computing device 600, including instructions stored in memory 620 or on storage device 630, and display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 coupled to high-speed interface 640. Other implementations may use multiple processors and / or multiple buses, as desired, along with multiple memories and multiple types of memories. Also, multiple computing devices 600 may be connected, with each device providing a portion of the required operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0068] The memory 620 stores information non-transiently within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory, read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0069] The storage device 630 is capable of providing mass storage to the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory, or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0070] The high-speed controller 640 manages the bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages the less bandwidth-intensive operations. Such role assignments are merely examples. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., via a graphics processor or accelerator), and the high-speed expansion port 650, which may accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and the low-speed expansion port 690. The low-speed expansion port 690, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled, for example, via a network adapter, to one or more input / output devices, such as a keyboard, pointing device, scanner, or a networking device, such as a switch or router.

[0071] The computing device 600 may be implemented in a variety of forms, as shown in the figure, such as a standard server 600a, or a group of such servers 600a multiple times, a laptop computer 600b, or as part of a rack server system 600c.

[0072] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or readable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0073] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural programming languages, and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0074] The processes and logic flows described herein are performed by one or more programmable processors (also called data processing hardware) that execute one or more computer programs, and can perform functions by manipulating input data to generate output. The processes and logic flows can also be performed by special purpose logic circuitry, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). Processors suitable for executing computer programs can include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The essential elements of a computer include a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or is operatively coupled to a mass storage device to receive data from or transfer data to the mass storage device, or both. However, a computer need not have such devices. Suitable computer-readable media for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0075] To provide interaction with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to a user, and optionally a keyboard and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Similarly, other types of devices can be used to provide interaction with a user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input. Furthermore, the computer can interact with a user by sending documents to and receiving documents from a device used by the user. For example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0076] Although several embodiments have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.< / s> < / s> < / s>

Claims

1. A computer-implemented method (500) that, when executed by data processing hardware (122), causes the data processing hardware (122) to perform operations, the operations including: receiving audio data (112) corresponding to a query spoken by a user; processing the audio data (112) using a speech recognizer to generate a plurality of candidate hypotheses (135), each candidate hypothesis (135) corresponding to a candidate transcription (175) for the query and represented by a respective sequence of hypothesis terms; For each candidate hypothesis (135), determining whether each sequence of hypothesis terms includes a source phrase (126) from a list of phrase correction pairs (125), each phrase correction pair in the list of phrase correction pairs including: a corresponding source phrase (126) that was misrecognized in a corresponding previous transcription (475) transcribed by the speech recognizer of a previous utterance (401) made by the user; and a corresponding target phrase (128) that matches a user correction replacing the misrecognized source phrase (126) in the corresponding previous transcription (475) transcribed by the speech recognizer; generating corresponding additional candidate hypotheses (145) that replace the source phrase (126) in each sequence of hypothesis terms with the corresponding target phrase (128) when the respective sequence of hypothesis terms includes the source phrase (126); ranking the plurality of candidate hypotheses (135) and each corresponding additional candidate hypothesis (145) generated; generating a transcription (175) of the query spoken by the user by selecting one of the highest rankings from the plurality of candidate hypotheses (135) and each corresponding additional candidate hypothesis (145) generated; The computer-implemented method (500).

2. The operations further comprise the steps of: for each phrase correction pair in the list of phrase correction pairs (125): identifying an n-gram of a language model (302) that includes the corresponding source phrase (126); cloning corresponding new n-grams that replace the corresponding source phrases (126) in the identified n-grams with the corresponding target phrases (128); The language model (302) adding to the language model (302) the new n-grams replacing the source phrases (126) with the corresponding target phrases (128); modifying the language model (302) by conditioning the language model (302) to determine that a prior likelihood score (304) of a unigram of the target phrase (128) is higher than a unigram of the source phrase (126); determining a corresponding prior likelihood score (304) for each additional candidate hypothesis (145) using the modified language model (302) configured to receive each additional candidate hypothesis (145) as an input; 2. The computer-implemented method of claim 1, wherein ranking the plurality of candidate hypotheses and each additional candidate hypothesis is based on the corresponding prior likelihood score determined for each additional candidate hypothesis.

3. The operations further comprise the steps of: for each phrase correction pair in the list of phrase correction pairs (125): obtaining an original sequence of n-grams (326) representing the corresponding previous transcription (475) that misrecognized the corresponding source phrase (126), the original sequence of n-grams (326) including the corresponding source phrase (126) and one or more other terms that precede and / or follow the corresponding source phrase (126) in the corresponding previous transcription (475); obtaining a corrected sequence of n-grams (328) that replaces the source phrase (126) in the original sequence of n-grams (326) with the corresponding target phrase (128), the corrected sequence of n-grams (328) including the target phrase (128) and the same one or more other terms that precede and / or follow the corresponding source phrase (126) in the corresponding previous transcription (475); The language model (302) adding the original sequence of n-grams (326) and the corrected sequence of n-grams (328) to the language model (302); conditioning the language model (302) to determine that a prior likelihood score (304) for a number of n-grams from the corrected sequence of n-grams (328) containing the target phrase (128) is higher than the same number of n-grams from the original sequence of n-grams (326) containing the source phrase (126); determining a corresponding prior likelihood score (304) for each additional candidate hypothesis (145) using the modified language model (302) configured to receive each additional candidate hypothesis (145) as an input; 2. The computer-implemented method of claim 1, wherein ranking the plurality of candidate hypotheses and each additional candidate hypothesis is based on the corresponding prior likelihood score determined for each additional candidate hypothesis.

4. 4. The computer-implemented method of claim 3, wherein a margin between the prior likelihood scores (304) determined by the language model (302) for a number of n-grams from the corrected sequence of n-grams (328) and the same number of n-grams from the original sequence of n-grams (326) increases as the numbers of n-grams from the corrected sequence and the original sequence of n-grams increase.

5. 5. The computer-implemented method of claim 3 or 4, wherein conditioning the language model (302) further comprises conditioning the language model (302) to determine that a prior likelihood score (304) of a first number of n-grams from the sequence of n-grams that includes the target phrase (128) is lower than a higher second number of n-grams from the sequence of n-grams that includes the target phrase (128).

6. 6. The computer-implemented method (500) of any of claims 3 to 5, wherein the original and corrected sequences of n-grams each further comprise n-grams representing sentence boundaries of the corresponding previous transcription (475) transcribed by the speech recognizer for the previous utterance (401) made by the user.

7. The operation further comprises: obtaining, for each of the plurality of candidate hypotheses (135) generated by the speech recognizer, a corresponding likelihood score (155) that the speech recognizer assigned to the corresponding candidate hypothesis (135); and after generating each additional candidate hypothesis (145), determining a corresponding likelihood score (155) for each generated additional candidate hypothesis (145) using an additional hypothesis scorer (300); 7. The computer-implemented method (500) of any of claims 1 to 6, wherein ranking the plurality of candidate hypotheses (135) and each generated corresponding additional candidate hypothesis (145) is based on the corresponding likelihood scores (155) assigned to the plurality of candidate hypotheses (135) by the speech recognizer and the corresponding likelihood scores (155) determined for each additional candidate hypothesis (145) using the additional hypothesis scorer (300).

8. The additional hypothesis scorer (300) an acoustic model (301) configured to process audio data (112) to determine an acoustic modeling score for a portion of the audio data (112) that includes either the source phrase (126) or the target phrase (128); or a language model (302) configured to receive as input each additional candidate hypothesis (145) and determine a corresponding prior likelihood score (304) for each additional candidate hypothesis (145); 8. The computer-implemented method of claim 7, wherein the corresponding likelihood score determined for each additional candidate hypothesis is based on at least one of the acoustic modeling score or the corresponding prior likelihood score determined for the additional candidate hypothesis.

9. 9. The computer-implemented method of claim 8, wherein the language model comprises an auxiliary language model external to the speech recognizer or an internal language model integrated with the speech recognizer.

10. 10. The computer-implemented method (500) of claim 8 or 9, wherein the speech recognizer includes an end-to-end speech recognition model configured to generate, for each of the plurality of candidate hypotheses (135), the corresponding likelihood score (155).

11. The speech recognizer includes an acoustic model (301) and a language model (302); 10. The computer-implemented method (500) of any of claims 7 to 9, wherein the corresponding likelihood score (155) assigned by the speech recognizer to each of the plurality of candidate hypotheses (135) is based on at least one of an acoustic modeling score output by the acoustic model (301) or a corresponding prior likelihood score (304) output by the language model (302).

12. A system (100), comprising: Data processing hardware (122); memory hardware (124) in communication with the data processing hardware (122) for storing instructions that, when executed by the data processing hardware (122), cause the data processing hardware (122) to perform operations, the operations including: receiving audio data (112) corresponding to a query spoken by a user; processing the audio data (112) using a speech recognizer to generate a plurality of candidate hypotheses (135), each candidate hypothesis (135) corresponding to a candidate transcription (175) for the query and represented by a respective sequence of hypothesis terms; For each candidate hypothesis (135), determining whether each sequence of hypothesis terms includes a source phrase (126) from a list of phrase correction pairs (125), each phrase correction pair in the list of phrase correction pairs including: a corresponding source phrase (126) that was misrecognized in a corresponding previous transcription (475) transcribed by the speech recognizer of a previous utterance (401) made by the user; and a corresponding target phrase (128) that matches a user correction replacing the misrecognized source phrase (126) in the corresponding previous transcription (475) transcribed by the speech recognizer; generating corresponding additional candidate hypotheses (145) that replace the source phrase (126) in each sequence of hypothesis terms with the corresponding target phrase (128) when the respective sequence of hypothesis terms includes the source phrase (126); ranking the plurality of candidate hypotheses (135) and each corresponding additional candidate hypothesis (145) generated; generating a transcription (175) of the query spoken by the user by selecting one of the highest rankings from the plurality of candidate hypotheses (135) and each corresponding additional candidate hypothesis (145) generated; The system (100).

13. The operations further comprise the steps of: for each phrase correction pair in the list of phrase correction pairs (125): identifying an n-gram of a language model (302) that includes the corresponding source phrase (126); cloning corresponding new n-grams that replace the corresponding source phrases (126) in the identified n-grams with the corresponding target phrases (128); The language model (302) adding to the language model (302) the new n-grams replacing the source phrases (126) with the corresponding target phrases (128); modifying the language model (302) by conditioning the language model (302) to determine that a prior likelihood score (304) of a unigram of the target phrase (128) is higher than a unigram of the source phrase (126); determining a corresponding prior likelihood score (304) for each additional candidate hypothesis (145) using the modified language model (302) configured to receive each additional candidate hypothesis (145) as an input; 13. The system (100) of claim 12, wherein ranking the plurality of candidate hypotheses (135) and each additional candidate hypothesis (145) is based on the corresponding prior likelihood score (304) determined for each additional candidate hypothesis (145).

14. The operations further comprise the steps of: for each phrase correction pair in the list of phrase correction pairs (125): obtaining an original sequence of n-grams (326) representing the corresponding previous transcription (475) that misrecognized the corresponding source phrase (126), the original sequence of n-grams (326) including the corresponding source phrase (126) and one or more other terms that precede and / or follow the corresponding source phrase (126) in the corresponding previous transcription (475); obtaining a corrected sequence of n-grams (328) that replaces the source phrase (126) in the original sequence of n-grams (326) with the corresponding target phrase (128), the corrected sequence of n-grams (328) including the target phrase (128) and the same one or more other terms that precede and / or follow the corresponding source phrase (126) in the corresponding previous transcription (475); The language model (302) adding the original sequence of n-grams and the corrected sequence of n-grams to the language model (302); conditioning the language model (302) to determine that a prior likelihood score (304) of a number of n-grams from the corrected sequence of n-grams containing the target phrase (128) is higher than the same number of n-grams from the original sequence of n-grams containing the source phrase (126); determining a corresponding prior likelihood score (304) for each additional candidate hypothesis (145) using the modified language model (302) configured to receive each additional candidate hypothesis (145) as an input; 13. The system (100) of claim 12, wherein ranking the plurality of candidate hypotheses (135) and each additional candidate hypothesis (145) is based on the corresponding prior likelihood score (304) determined for each additional candidate hypothesis (145).

15. 15. The system of claim 14, wherein a margin between the prior likelihood scores determined by the language model for a number of n-grams from the corrected sequence of n-grams and the same number of n-grams from the original sequence of n-grams increases as the numbers of n-grams from the corrected sequence and the original sequence of n-grams increase.

16. 16. The system of claim 14 or 15, wherein conditioning the language model further comprises conditioning the language model to determine that a prior likelihood score of a first number of n-grams from the sequence of n-grams that includes the target phrase is lower than a higher second number of n-grams from the sequence of n-grams that includes the target phrase.

17. The system (100) of any of claims 14 to 16, wherein the original and corrected sequences of n-grams each further comprise n-grams representing sentence boundaries of the corresponding previous transcription (475) transcribed by the speech recognizer for the previous utterance (401) made by the user.

18. The operation further comprises: obtaining, for each of the plurality of candidate hypotheses (135) generated by the speech recognizer, a corresponding likelihood score (155) that the speech recognizer assigned to the corresponding candidate hypothesis (135); and after generating each additional candidate hypothesis (145), determining a corresponding likelihood score (155) for each generated additional candidate hypothesis (145) using an additional hypothesis scorer (300); 18. The system (100) of claim 12, wherein ranking the plurality of candidate hypotheses (135) and each generated corresponding additional candidate hypothesis (145) is based on the corresponding likelihood scores (155) assigned to the plurality of candidate hypotheses (135) by the speech recognizer and the corresponding likelihood scores (155) determined for each additional candidate hypothesis (145) using the additional hypothesis scorer (300).

19. The additional hypothesis scorer (300) an acoustic model (301) configured to process audio data (112) to determine an acoustic modeling score for a portion of the audio data (112) that includes either the source phrase (126) or the target phrase (128); or a language model (302) configured to receive as input each additional candidate hypothesis (145) and determine a corresponding prior likelihood score (304) for each additional candidate hypothesis (145); 20. The system of claim 18, wherein the corresponding likelihood score determined for each additional candidate hypothesis is based on at least one of the acoustic modeling score or the corresponding prior likelihood score determined for the additional candidate hypothesis.

20. 20. The system of claim 19, wherein the language model includes an auxiliary language model that is external to the speech recognizer, or an internal language model that is integrated with the speech recognizer.

21. 21. The system (100) of claim 19 or 20, wherein the speech recognizer includes an end-to-end speech recognition model configured to generate, for each of the plurality of candidate hypotheses (135), the corresponding likelihood score (155).

22. The speech recognizer includes an acoustic model (301) and a language model (302); 21. The system (100) of claim 18, wherein the corresponding likelihood score (155) assigned by the speech recognizer to each of the plurality of candidate hypotheses (135) is based on at least one of an acoustic modeling score output by the acoustic model (301) or a corresponding prior likelihood score (304) output by the language model (302).

Images