Lattice Audio Correction

Abstract

The method (500) includes receiving audio data (112) corresponding to a query, and processing the audio data to generate a plurality of candidate hypotheses (135), each represented by a respective sequence of hypothesis terms. For each candidate hypothesis, the method includes determining whether the sequence of hypothesis terms includes a source phrase (126) from a list of phrase correction pairs (125). Each phrase correction pair includes a corresponding misrecognized source phrase and a corresponding target phrase (128) that replaces the source phrase. When the respective sequence of hypothesis terms includes the source phrase, the method includes generating a corresponding additional candidate hypothesis (145) that replaces the source phrase. The method also includes generating a transcription (175) of the query by ranking the plurality of candidate hypotheses and each corresponding additional candidate hypothesis, and selecting a highest ranking one of the plurality of candidate hypotheses and each additional candidate hypothesis.

Description

Technical Field

[0001] This disclosure relates to lattice voice correction.

Background Art

[0002] Automatic speech recognition (ASR) systems provide a technology commonly used in mobile devices and other devices. Generally, an ASR system attempts to provide an accurate transcription of what a user says into a mobile device. In some instances, the ASR system generates an inaccurate transcription that does not match what the user said. In these instances, the user may correct the inaccurate transcription by providing user input to correct the transcription on the mobile device. One of the challenges of an ASR system includes how to make the best use of user input to correct inaccurate transcriptions to generate accurate transcriptions in the future.

Summary of the Invention

[0003] One aspect of the present disclosure provides a computer implementation that, when executed on data processing hardware, causes data processing hardware to perform an operation, the operation including receiving audio data corresponding to a query spoken by a user, and processing the audio data using a speech recognition system to generate a number of candidate hypotheses. Each candidate hypothesis corresponds to a candidate transcription of the query and is represented by a sequence of hypothetical terms. For each candidate hypothesis, the method also includes determining whether each sequence of hypothetical terms contains a source phrase from a list of phrase correction pairs, and, if each sequence of hypothetical terms contains a source phrase, generating a corresponding additional candidate hypothesis that replaces the source phrase in each sequence of hypothetical terms with a corresponding target phrase. Each phrase correction pair in the list of phrase correction pairs includes a corresponding source phrase that was misrecognized in a corresponding previous transcription transcribed by a speech recognition system for a previous utterance made by a user, and a corresponding target phrase that matches a user correction that replaces the source phrase that was misrecognized in the corresponding previous transcription transcribed by a speech recognition system. This method also includes ranking multiple candidate hypotheses and each corresponding additional candidate hypothesis generated, and generating a transcript of the query spoken by the user by selecting the highest-ranking of the multiple candidate hypotheses and each corresponding additional candidate hypothesis generated.

[0004] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the operation further includes: identifying an n-gram of the language model containing the corresponding source phrase for each phrase correction pair in a list of phrase correction pairs; cloning a corresponding new n-gram that replaces the corresponding source phrase in the identified n-gram with the corresponding target phrase; modifying the language model; and determining the corresponding prior likelihood score for each additional candidate hypothesis using the modified language model configured to receive each additional candidate hypothesis as input. Here, modifying the language model includes adding a new n-gram to the language model that replaces the source phrase with the corresponding target phrase; and conditioning the language model to determine that the prior likelihood score of the unigram of the target phrase is higher than that of the unigram of the source phrase. In these embodiments, ranking of the multiple candidate hypotheses and each additional candidate hypothesis is based on the corresponding prior likelihood score determined for each additional candidate hypothesis.

[0005] In an additional embodiment, the operation further includes: for each phrase correction pair in a list of phrase correction pairs, obtaining an original sequence of n-grams representing the corresponding previous transcription that misidentified the corresponding source phrase; obtaining a correction sequence of n-grams that replaces the source phrase in the original sequence of n-grams with the corresponding target phrase; and determining the corresponding prior likelihood score for each additional candidate hypothesis using the modified language model configured to receive each additional candidate hypothesis as input. The original sequence of n-grams includes the corresponding source phrase and one or more other terms that precede and / or follow the corresponding source phrase in the corresponding previous transcription, and the correction sequence of n-grams includes the target phrase and the same one or more other terms that precede and / or follow the corresponding source phrase in the corresponding previous transcription. Here, modifying the language model includes adding an original sequence of n-grams and a corrected sequence of n-grams to the language model and conditioning the language model to determine that the prior likelihood score for the number of n-grams from the corrected sequence of n-grams containing the target phrase is higher than the same number of n-grams from the original sequence of n-grams containing the source phrase. In these embodiments, ranking the multiple candidate hypotheses and each additional candidate hypothesis is based on the corresponding prior likelihood score determined for each additional candidate hypothesis. Optionally, the original sequence of n-grams and the corrected sequence may each further include n-grams representing sentence boundaries of the corresponding previous transcription transcribed by the speech recognition system for a previous utterance made by the user.

[0006] In the embodiments described above, the margin between the prior likelihood scores determined by the language model for the number of n-grams from the corrected sequence of n-grams and the same number of n-grams from the original sequence of n-grams may increase as the number of n-grams from the corrected sequence of n-grams and the original sequence increases. Additionally or alternatively, conditioning the language model may further include conditioning the language model to determine that the prior likelihood score for a first number of n-grams from the sequence of n-grams containing the target phrase is lower than a larger second number of n-grams from the sequence of n-grams containing the target phrase.

[0007] In some examples, the operation further includes obtaining the corresponding likelihood score that the speech recognition system assigned to each of the multiple candidate hypotheses generated by the speech recognition system, and, after generating each additional candidate hypothesis, using an additional hypothesis scorer to determine the corresponding likelihood score for each generated additional candidate hypothesis. Here, ranking the multiple candidate hypotheses and each generated corresponding additional candidate hypothesis is based on the corresponding likelihood scores assigned to the multiple candidate hypotheses by the speech recognition system and the corresponding likelihood score determined for each additional candidate hypothesis using the additional hypothesis scorer. In these examples, the additional hypothesis scorer includes at least one of the following: an acoustic model configured to process audio data and determine an acoustic modeling score for the portion of the audio data containing either a source phrase or a target phrase, or a language model configured to receive each additional candidate hypothesis as input and determine the corresponding prior likelihood score for each additional candidate hypothesis. Here, the corresponding likelihood score determined for each additional candidate hypothesis may be based on at least one of the acoustic modeling score or the corresponding prior likelihood score determined for the additional candidate hypothesis.

[0008] The language model may include an auxiliary language model located outside the speech recognition system, or an internal language model integrated with the speech recognition system. Furthermore, the speech recognition system may include an end-to-end speech recognition model configured to generate corresponding likelihood scores for each of several candidate hypotheses. Alternatively, the speech recognition system may include an acoustic model and a language model, where the corresponding likelihood scores assigned by the speech recognition system to each of the several candidate hypotheses may be based on at least one of either acoustic modeling scores output by the acoustic model or corresponding prior likelihood scores output by the language model.

[0009] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that cause the data processing hardware to perform actions when executed by the data processing hardware, the actions including receiving audio data corresponding to a query spoken by a user, and processing the audio data using a speech recognition system to generate a number of candidate hypotheses. Each candidate hypothesis corresponds to a candidate transcription of the query and is represented by a sequence of hypothetical terms. For each candidate hypothesis, the method also includes determining whether each sequence of hypothetical terms contains a source phrase from a list of phrase correction pairs, and, if each sequence of hypothetical terms contains a source phrase, generating a corresponding additional candidate hypothesis that replaces the source phrase in each sequence of hypothetical terms with a corresponding target phrase. Each phrase correction pair in the list of phrase correction pairs includes a corresponding source phrase that was misrecognized in a corresponding previous transcription transcribed by a speech recognition system for a previous utterance made by the user, and a corresponding target phrase that matches a user correction that replaces the misrecognized source phrase in the corresponding previous transcription transcribed by the speech recognition system. The method also includes ranking multiple candidate hypotheses and each corresponding additional candidate hypothesis generated, and generating a transcription of a query spoken by the user by selecting one of the highest-ranking candidates and each corresponding additional candidate hypothesis generated.

[0010] This embodiment may include one or more of the following optional features. In some embodiments, the operation further includes: identifying an n-gram of the language model containing the corresponding source phrase for each phrase correction pair in the list of phrase correction pairs; cloning a corresponding new n-gram that replaces the corresponding source phrase in the identified n-gram with the corresponding target phrase; modifying the language model; and determining the corresponding prior likelihood score for each additional candidate hypothesis using the modified language model configured to receive each additional candidate hypothesis as input. Here, modifying the language model includes adding a new n-gram to the language model that replaces the source phrase with the corresponding target phrase; and conditioning the language model to determine that the prior likelihood score of the unigram of the target phrase is higher than that of the unigram of the source phrase. In these embodiments, ranking of the multiple candidate hypotheses and each additional candidate hypothesis is based on the corresponding prior likelihood score determined for each additional candidate hypothesis.

[0011] In an additional embodiment, the operation further includes: for each phrase correction pair in a list of phrase correction pairs, obtaining an original sequence of n-grams representing the corresponding previous transcription that misidentified the corresponding source phrase; obtaining a correction sequence of n-grams that replaces the source phrase in the original sequence of n-grams with the corresponding target phrase; and determining the corresponding prior likelihood score for each additional candidate hypothesis using the modified language model configured to receive each additional candidate hypothesis as input. The original sequence of n-grams includes the corresponding source phrase and one or more other terms that precede and / or follow the corresponding source phrase in the corresponding previous transcription, and the correction sequence of n-grams includes the target phrase and the same one or more other terms that precede and / or follow the corresponding source phrase in the corresponding previous transcription. Here, modifying the language model includes adding an original sequence of n-grams and a corrected sequence of n-grams to the language model and conditioning the language model to determine that the prior likelihood score for the number of n-grams from the corrected sequence of n-grams containing the target phrase is higher than the same number of n-grams from the original sequence of n-grams containing the source phrase. In these embodiments, ranking the multiple candidate hypotheses and each additional candidate hypothesis is based on the corresponding prior likelihood score determined for each additional candidate hypothesis. Optionally, the original sequence of n-grams and the corrected sequence may each further include n-grams representing sentence boundaries of the corresponding previous transcription transcribed by the speech recognition system for a previous utterance made by the user.

[0012] In the embodiments described above, the margin between the prior likelihood scores determined by the language model for the number of n-grams from the corrected sequence of n-grams and the same number of n-grams from the original sequence of n-grams may increase as the number of n-grams from the corrected sequence of n-grams and the original sequence increases. Additionally or alternatively, conditioning the language model may further include conditioning the language model to determine that the prior likelihood score for a first number of n-grams from the sequence of n-grams containing the target phrase is lower than a larger second number of n-grams from the sequence of n-grams containing the target phrase.

[0013] In some examples, the operation further includes obtaining the corresponding likelihood score that the speech recognition system assigned to each of the multiple candidate hypotheses generated by the speech recognition system, and, after generating each additional candidate hypothesis, using an additional hypothesis scorer to determine the corresponding likelihood score for each generated additional candidate hypothesis. Here, ranking the multiple candidate hypotheses and each generated corresponding additional candidate hypothesis is based on the corresponding likelihood scores assigned to the multiple candidate hypotheses by the speech recognition system and the corresponding likelihood score determined for each additional candidate hypothesis using the additional hypothesis scorer. In these examples, the additional hypothesis scorer includes at least one of the following: an acoustic model configured to process audio data and determine an acoustic modeling score for the portion of the audio data containing either a source phrase or a target phrase, or a language model configured to receive each additional candidate hypothesis as input and determine the corresponding prior likelihood score for each additional candidate hypothesis. Here, the corresponding likelihood score determined for each additional candidate hypothesis may be based on at least one of the acoustic modeling score or the corresponding prior likelihood score determined for the additional candidate hypothesis.

[0014] The language model may include an auxiliary language model located outside the speech recognition system, or an internal language model integrated with the speech recognition system. Furthermore, the speech recognition system may include an end-to-end speech recognition model configured to generate corresponding likelihood scores for each of several candidate hypotheses. Alternatively, the speech recognition system may include an acoustic model and a language model, where the corresponding likelihood scores assigned by the speech recognition system to each of the several candidate hypotheses may be based on at least one of either acoustic modeling scores output by the acoustic model or corresponding prior likelihood scores output by the language model.

[0015] Details of one or more embodiments of this disclosure are described in the accompanying drawings and the following description. Other embodiments, features, and advantages will become apparent from the description and drawings and the claims. [Brief explanation of the drawing]

[0016] [Figure 1] This is a schematic diagram of an exemplary system for automatic speech recognition using phrase correction pairs. [Figure 2A] This is a schematic diagram illustrating an example of a word lattice. [Figure 2B] This is a schematic diagram illustrating an example of a word lattice. [Figure 3] This is a schematic diagram showing a language model for scoring candidate hypotheses. [Figure 4A] This is a schematic diagram of a user providing input indications to correct previously misrecognized transcriptions. [Figure 4B] This is a schematic diagram of a user providing input indications to correct previously misrecognized transcriptions. [Figure 4C] This is a schematic diagram of a user providing input indications to correct previously misrecognized transcriptions. [Figure 5] This is a flowchart illustrating the operational configuration of the lattice audio correction method. [Figure 6] This is a schematic diagram of an exemplary computing device that may be used to implement the systems and methods described herein. [Modes for carrying out the invention]

[0017] Similar reference symbols in various drawings refer to the same elements.

[0018] Automatic speech recognition (ASR) systems are becoming increasingly prevalent in client devices as they continue to provide more accurate transcriptions of what users say. Nevertheless, in some cases, ASR systems produce inaccurate transcriptions that misinterpret what the user actually said. This often occurs when words are acoustically similar or when the user speaks a unique word unfamiliar to the ASR system. For example, "I say" and "Ice age" sound very similar, making it difficult for the ASR system to eliminate ambiguity about which phrase the user intended to convey. In some cases, when a user actually intended to say "Ice age," but the client device inaccurately transcribes it as "I say," the user can correct the transcription by using the client device (for example, by typing the correct transcription using the client device's keyboard).

[0019] Among the particularly difficult aspects of an ASR system is how to utilize these user corrections to generate a more accurate transcription for subsequent user utterances. For example, when an utterance made by a user is similar to "I say" and "Ice age", the ASR system cannot simply transcribe it as "Ice age" every time. That is, just because the user previously intended to convey "Ice age" does not mean that the user will never intend to convey "I say" later. Furthermore, if the ASR system repeatedly generates an inaccurate transcription of "I say" even when the user continues to convey "Ice age", the user may lose trust in the ASR system.

[0020] Embodiments of the present specification are directed to systems and methods for performing lattice speech correction. In particular, a computing system receives audio data of an utterance made by a user, and the computing system processes the audio data to generate candidate hypotheses. Next, the computing system determines whether any word or phrase in a candidate hypothesis contains a source phrase (e.g., a previously misrecognized phrase) transcribed by the computing system. When the computing system determines that a candidate hypothesis contains a source phrase, the computing system generates an additional candidate hypothesis by replacing the source phrase with a correction phrase (e.g., a target phrase). Thereafter, the computing system generates a likelihood score for each of the candidate hypothesis and the additional candidate hypothesis. Here, the likelihood score indicates the probability that the hypothesis is the correct transcription of the utterance made by the user. The computing system generates a transcription of the utterance made by the user by selecting the highest likelihood score of either the candidate hypothesis or the additional candidate hypothesis.

[0021] Figure 1 shows an example of a system 100 for automatic speech recognition (ASR) of an utterance (i.e., query) 101 made by a user 10, using audio data 112 corresponding to the utterance 101. The system 100 includes a client device 110, a computing system 120, and a network 118. The computing system 120 may be a distributed system with scalable, elastic resources (e.g., a cloud computing environment). The resources include computing resources 122 (e.g., data processing hardware) and / or storage resources 124 (e.g., memory hardware). The network 118 may be wired, wireless, or a combination thereof, and may include a private network and / or a public network such as the Internet.

[0022] In some examples, the computing system 120 receives audio data 112 from a client device 110, processes the audio data 112, and generates several candidate hypotheses 135 for an utterance 101 based on the audio data 112. Here, each candidate hypothesis corresponds to a candidate transcription of the utterance 101 and is represented by a sequence of hypothetical terms. For each candidate hypothesis 135, as will be described in more detail below, the computing system 120 is configured to determine whether each sequence of hypothetical terms contains a source phrase 126 from a correction phrase list 125. When each sequence of hypothetical terms contains a source phrase 126 from the correction phrase list 125, the computing system 120 is configured to generate an additional candidate hypothesis 145 that replaces the source phrase 126 with a corresponding target phrase 128. The computing system 120 then generates a transcription 175 by selecting the highest-ranking hypothesis from the multiple candidate hypotheses 135 or additional candidate hypotheses 145.

[0023] FIG. 1 shows operations (A)-(G) indicating the flow of data. As described in this specification, computing system 120 performs operations (B)-(G). However, it is understood that client device 110 may also perform operations (B)-(G) in addition to or instead of computing system 120 performing the operations. In some examples, client device 110 performs the first part of the operations (e.g., operations (B)-(D)) and computing system 120 performs the second part of the operations (e.g., operations (E)-(G)), or vice versa.

[0024] Client device 110 includes data processing hardware 114 and memory hardware 116. Client device 110 may include one or more audio capture devices (e.g., a microphone(s)) 103 for capturing utterance 101 from user 10 and converting it to audio data 112 (e.g., an electrical signal). In some examples, microphone 103 is separate from client device 110 and communicates with client device 110 to provide the recorded utterance 101 to client device 110. Client device 110 can be any computing device capable of communicating with computing system 120 via network 118. Client device 110 includes desktop computing devices and mobile computing devices such as, but not limited to, laptops, tablets, smartphones, smart speakers / displays, smart appliances, Internet of Things (IoT) devices, and wearable computing devices (e.g., headsets and / or watches).

[0025] In the example in Figure 1, during stage (A), user 10 makes an utterance 101, and the microphone 103 of the client device 110 records the utterance 101. In this example, the utterance 101 includes user 10 saying "call Beshad on mobile". The client device 110 transmits audio data 112 corresponding to the utterance 101 recorded by the microphone 103 to the computing system 120 via the network 118. During stage (B), the computing system 120 processes the audio data 112 to generate several candidate hypotheses 135, 135a-n. Here, each candidate hypothesis 135 corresponds to a candidate transcription of the utterance 101 and is represented by a sequence of hypothetical terms. For example, the computing system 120 may run a speech recognition module 130 (e.g., an automatic speech recognition (ASR) module) to create a word lattice 200 showing several possible candidate hypothesis transcriptions 135 for the utterance 101 based on the audio data 112. The speech recognition module 130 can evaluate potential paths through the word lattice 200 to determine multiple candidate hypotheses 135.

[0026] Figure 2A shows an example of word lattices 200, 200a that may be provided by the speech recognition module 130 of Figure 1. The word lattice 200a represents multiple possible combinations of words that may form different candidate hypotheses 135 for an utterance 101.

[0027] The word lattice 200a includes one or more nodes 202a-g corresponding to possible boundaries between words. The word lattice 200a includes multiple edges 204a-l of possible words of candidate hypotheses arising from the word lattice 200a. Furthermore, each of the edges 204a-l may have one or more weights or probabilities that the edge is a correct edge from the corresponding node. The weights are determined by the speech recognition module 130 and may be based, for example, on the reliability of the match between the speech data and the word at the edge, and the degree to which the word is grammatically and / or lexically compatible with other words in the word lattice 200a.

[0028] For example, initially, the highest-probability path through word lattice 200a (e.g., the highest-probability candidate hypothesis 135) may include edges 204c, 204e, 204i, and 204k, which have the text "we're coming about 11:30". A second best path through word lattice 200a (e.g., the second best candidate hypothesis 135) may include edges 204d, 204h, 204j, and 204l, which have the text "deer hunting scouts 7:30".

[0029] Each pair of nodes may have one or more paths corresponding to alternative words in various candidate hypotheses 135. For example, the initial highest-probability path between node pairs starting at node 202a and ending at node 202c is edge 204c "we're". This path has alternative paths including edges 204a, 204b "we are" and edge 204d "deer".

[0030] Figure 2B shows an example of a hierarchical word lattice 200, 200b that may be provided by the speech recognition module 130 of Figure 1. The word lattice 200b includes nodes 252a-l representing words that constitute various candidate hypotheses 135 for an utterance 101. The edges between nodes 252a-l are the possible candidate hypotheses 135: (1) nodes 252c, 252e, 252i, 252k "we're coming about 11:30", (2) nodes 252a, 252b, 252e, 252i, 252k "we are coming about 11:30", (3) nodes 252a, 252b, 252f, 252g, 252i, 252k "we are come at about 11:30", (4) nodes 252d, 252f, 252g, 252i, 252k "deer come at about 11:30", (5) nodes 252d, 252h, 252j, 252k "deer hunting scouts (6) Includes nodes 252d, 252h, 252j, and 252l, "deer hunting scouts 7:30".

[0031] In this case as well, the edges between nodes 242a-l may have associated weights or probabilities based on the reliability of speech recognition (e.g., candidate hypotheses) and grammatical / lexical analysis of the resulting text. In this example, "we are coming about 11:30" could be the best hypothesis at this point, and "deer hunting scouts 7:30" could be the next best hypothesis. One or more divisions 354a-d that group words and their alternatives together can be made in word lattice 200b. For example, division 254a contains the word "we're" and the alternatives "we are" and "deer". Division 252b contains the word "comming" and the alternatives "come at" and "hunting". Division 254c contains the word "about" and the alternative "Scouts", and division 254d contains the word "11:30" and the alternative "7:30".

[0032] Referring again to Figure 1, the speech recognition module 130 may generate multiple candidate hypotheses 135 from the word lattice 200. That is, the speech recognition module 130 generates a likelihood score 155 for each of the candidate hypotheses 135 of the word lattice 200. Each likelihood score 155 indicates the probability that the candidate hypothesis 135 is correct (e.g., matches utterance 101). In some embodiments, the speech recognition module 130 includes an end-to-end (E2E) speech recognition model configured to receive audio data 112 and generate the word lattice 200. In particular, the E2E speech recognition model processes the audio data 112 to generate a corresponding likelihood score 155 for each of the multiple candidate hypotheses 135 from the word lattice 200. In some examples, the speech recognition module 130 includes separate acoustic models, language models, and / or pronunciation models. The speech recognition module 130 may share an acoustic model and a language model (e.g., acoustic model 301 and language model 302) with the additional hypothesis scorer 300, or it may have independent acoustic and language models.

[0033] In some examples, the speech recognition module 130 includes an acoustic model and / or a language model, which, based on audio data 112, generate a word lattice 200, or otherwise, a set of candidate hypotheses 135 for an utterance 101. Here, the likelihood score 155 of the candidate hypotheses 135 may include a combination of the acoustic modeling score from the acoustic model and / or the prior likelihood score from the language model. In other words, the likelihood score 155 includes at least one of the acoustic modeling score output by the acoustic model and / or the prior likelihood score output by the language model.

[0034] During Stage (C), the computing system 120 identifies a set of the highest-ranking candidate hypotheses 135 from among multiple candidate hypotheses 135 in the word lattice 200. For example, using likelihood scores 155 from the speech recognition module 130, the computing system 120 selects n candidate hypotheses 135 with the highest likelihood scores 155 (where n is an integer). In some examples, the computing system 120 selects candidate hypotheses 135 with likelihood scores 155 that satisfy a likelihood score threshold. Optionally, the speech recognition module 130 may use the likelihood scores 155 to rank the set of the highest-ranking candidate hypotheses 135.

[0035] In the example shown, the speech recognition module 130 generates candidate hypotheses 135 for the utterance 101 "call Beshad on mobile" made by user 10. In this example, the top two candidate transcriptions (e.g., the two most likely to be correct) are selected as the highest-ranked candidate hypothesis 135 set. The highest-ranked candidate hypotheses 135 include a first candidate hypothesis 135 "call best shot on mobile" with a likelihood score 155 of 0.8, and a second candidate hypothesis 135 "goal best shot immobile" with a likelihood score 155 of 0.2. Here, we show that a higher likelihood score 155 indicates greater confidence that the candidate hypothesis 135 is correct. In particular, none of the highest-ranked candidate hypotheses 135 do not contain the utterance 101 (e.g., "call Beshad on mobile") actually made by user 10. Therefore, if the computing system 120 selects any of the highest-ranked candidate hypotheses 135, the transcription 175 output to user 10 will be inaccurate.

[0036] During Stage (D), the computing system 120 runs a lattice augmenter 140 for each candidate hypothesis 135, determining whether each sequence of hypothetical terms contains a source phrase 126 from the correction phrase list 125. The correction phrase list 125 contains a list of phrase correction pairs. Each phrase correction pair in the correction phrase list 125 contains a source phrase 126 and a corresponding target phrase 128. Thus, the correction phrase list (e.g., a list of phrase correction pairs) 125 contains multiple source phrases 126, 126a-n and multiple corresponding target phrases 128, 128a-n. In particular, the source phrase 126 represents a misrecognized transcription previously generated by the speech recognition module 130, and the target phrase 128 represents a corrected transcription for the misrecognized transcription.

[0037] Figures 4A to 4C illustrate how to correct a misrecognized transcription, store the corrected transcription, and generate a corrected phrase list 125 (Figure 1). In some embodiments, the speech recognition module 130 generates a misrecognized transcription 475 that misrecognizes a previous utterance 401 made by user 10. As used herein, the previous utterance 401 and the misrecognized transcription 475 are received by the client device 110 before the transcription 175 and utterance 101 (Figure 1) are received. As will become clear, the misrecognized transcription 475 and the previous utterance 401 can be used to generate a corrected phrase list 125 (Figure 1) to improve speech recognition accuracy when the client device 110 later receives utterance 101 (Figure 1).

[0038] Figure 4A shows that the microphone 103 of the client device 110 records that user 10 speaks the previous utterance 401, "call Beshad on mobile." The client device 110 converts the previous utterance 401 into audio data 412 and transmits the audio data 412 to the speech recognition module 130. The speech recognition module 130 processes the audio data 412 to generate a misrecognition transcription 475 corresponding to the audio data 412. In the example shown, the speech recognition module 130 generates a misrecognition transcription 475, "call best shot," which is a misrecognition of the previous utterance 401 spoken by user 10. The client device 110 displays the previous transcription 475 to user 10 via the graphical user interface (GUI) 111.

[0039] Referring here to Figure 4B, user 10 may identify that the misrecognized transcription 475 displayed on GUI 111 does not match the previous utterance 401. Therefore, user 10 provides the GUI 111 of the client device 110 with an input indication showing the selection of a misrecognized phrase 425 from the misrecognized transcription 475. In some examples, the input indication includes providing touch input to GUI 111 for user 10 to select a misrecognized phrase 425 from the misrecognized transcription 475. The misrecognized phrase 425 may contain all or part of the misrecognized transcription 475. In the example shown, the misrecognized phrase 425 "best shot" contains only the inaccurate portion of the previous transcription 475.

[0040] Figure 4C shows that user 10 replaces the misrecognized phrase 425 in the misrecognized transcription 475 with the corrective phrase 430. In some examples, user 10 uses the keyboard 119 of the client device 110 to type text to provide the corrective phrase 430. Optionally, the keyboard 119 may appear in response to the client device 110 receiving an input indication from user 10 (Figure 4B). In these examples, user 10 may use the keyboard 118 of the client device 110 to type the corrective phrase (e.g., "Beshad"). In other examples, user 10 inputs the corrective phrase 430 by speaking to the client device 110; that is, user 10 may speak each character of the corrective phrase 430 (e.g., "BESHAD"). After receiving the corrective phrase 430, the client device 110 replaces the misrecognized phrase 425 with the corrective phrase 430 to generate the previous corrective transcription 476, which represents the accurate transcription of the previous utterance 401.

[0041] Therefore, the computing system 120 may store the misrecognized phrase 425 and the corrected phrase 430 as phrase correction pairs in the correction phrase list 125, with the source phrase 126 and the target phrase 128, respectively. In some examples, the computing system stores the misrecognized transcription 475 and the previous corrected transcription 476 as phrase correction pairs in the correction phrase list 125, with the source phrase and the target phrase 128, respectively. The phrase correction pairs may be stored in the memory hardware 116 of the client device 110 and / or in the memory hardware 124 of the computing system 120. The phrase correction pairs may also be stored in the correction history data store 320 (Figure 3). As will become clear, the computing system 120 uses the phrase correction pairs to generate the correct transcription 175 when the computing system 120 detects the source phrase 126 (e.g., best shot) again.

[0042] Referring again to Figure 1, in some embodiments, the lattice augmenter 140 is configured to generate additional candidate hypotheses 145, 145a-n using the candidate hypotheses 135 and the correction phrase list 125. Specifically, the lattice augmenter 140 receives the candidate hypotheses 135 and the correction phrase list 125 and determines for each candidate hypothesis 135 whether each sequence of hypothetical terms contains one of the source phrases 126 from the correction phrase list 125. When each sequence of hypothetical terms contains one of the source phrases 126, the lattice augmenter 140 generates an additional candidate hypothesis 145 that replaces the source phrase 126 in each sequence of hypothetical terms with the corresponding target phrase 128.

[0043] To continue illustrating the example, the lattice augmenter 140 determines whether both the first candidate hypothesis 135 "call best shot on mobile" and the second candidate hypothesis "goal best shot immobile" contain the source phrase "best shot" from the correction phrase list 125. Therefore, the lattice augmenter 140 generates the first additional candidate hypothesis 145 "call Beshad on mobile" and the second additional candidate hypothesis 145 "goal Beshad immobile". Here, in each of the additional candidate hypotheses 145, the lattice augmenter 140 replaces the source phrase 126 "best shot" with the target phrase 128 "Beshad".

[0044] In some embodiments, the computing system 120 runs an additional hypothesis scorer 300 (also simply called the “hypothesis scorer 300”), which is configured to determine a likelihood score 155 for each additional candidate hypothesis 145 generated by the lattice augmenter 140. That is, simply generating an additional candidate hypothesis 145 does not mean that the additional candidate hypothesis 145 contains the correct transcription 175 of the utterance 101. For example, the lattice augmenter 140 may generate an additional candidate hypothesis 145 “What is Hungary’s Beshad at the Euro cup” from a candidate hypothesis 135 “What is Hungary’s best shot at the Euro cup” (e.g., by replacing the source phrase 126 “best shot” with the target phrase “Beshad”). In this example, the additional candidate hypothesis 145 is the hypothesis most likely to be meaningless and should contain a lower likelihood score 155 than candidate hypothesis 135. Therefore, the computing system 120 selects the hypothesis with the highest likelihood from the candidate hypothesis 135 and the additional candidate hypothesis 145 to maximize the probability of an accurate transcription 175.

[0045] However, in some cases, the hypothesis scorer 300 may not be able to generate an accurate likelihood score 155 for the additional candidate hypothesis 145. That is, since the computing system 120 receives the additional candidate hypothesis 145 from user input, the additional candidate hypothesis 145 may be out-of-domain, out-of-lexical (OOV), and / or may only include sparse training data in the computing system 120. Therefore, the hypothesis scorer 300 needs to generate a likelihood score 155 that accurately reflects the probability of whether the additional candidate hypothesis 145 is true or false.

[0046] The hypothesis scorer 300 includes at least one of the acoustic model 301 and / or language model 302. In some examples, the language model 302 includes an external auxiliary language model of the speech recognition module 130. In other examples, the language model 302 includes an internal language model integrated with the speech recognition module 130. The acoustic model 301 is configured to process the audio data 112 and determine an acoustic modeling score for portions of the audio data 112 that contain either the source phrase 126 or the target phrase 128. In some examples, the acoustic model 301 compares the acoustic similarity between a portion of the audio data 112 containing the source phrase 126 and a portion of a candidate hypothesis 135 containing the source phrase 126. In other examples, the acoustic model compares the acoustic similarity between a portion of the audio data 112 containing the target phrase 128 and a portion of an additional candidate hypothesis 145 containing the target phrase 128.

[0047] Therefore, the acoustic modeling score may indicate whether a portion of the audio data 112 containing source phrase 126 or target phrase 128 is acoustically similar to candidate hypothesis 135 or additional candidate hypothesis 145. In some examples, the hypothesis scorer 300 cannot determine the acoustic modeling score for additional candidate hypothesis 145. For example, when the target phrase 128 of additional candidate hypothesis 145 is an OOV phrase, the hypothesis scorer 300 cannot compare the portion of the audio data 112 with the OOV phrase. In these examples, the hypothesis scorer 300 assumes that source phrase 126 and target phrase 128 are acoustically similar and depend solely on the language model 302.

[0048] In some embodiments, the hypothesis scorer 300's language model 302 is configured to receive each additional candidate hypothesis 145 as input and to determine a prior likelihood score 304 corresponding to each additional candidate hypothesis 145. The prior likelihood score 304 may indicate the probability of an additional candidate hypothesis 145 based on probability data received by the language model 302 120, which shows the frequency of different sequences of terms occurring. For example, the language model 302 may assign a higher prior likelihood score 304 to the additional candidate hypothesis "three blind mice" than to "tree blind mice" because the former language sequence occurs more frequently in the corpus of utterances 101 than the latter.

[0049] In some embodiments, the target phrase 128 is an OOV phrase in the language model 301. Thus, for each phrase correction pair in the correction phrase list 125, the language model 302 identifies an n-gram in the language model 302 that includes the corresponding source phrase 126. Here, the language model 302 clones a corresponding new n-gram that replaces the corresponding source phrase 126 in the identified n-gram with the target phrase 128. That is, the language model 302 creates a new n-gram for the target phrase 128 using an existing n-gram in the language model 302. In these embodiments, the computing system 120 modifies the language model 302 by adding the new n-gram of the target phrase 128 to the language model 302 and conditioning the language model 302 to determine that the prior likelihood score of the n-gram for the target phrase 128 is higher than that of the n-gram for the source phrase 126.

[0050] For example, computing system 120 adds an n-gram for target phrase 128 "Beshad" to language model 302, which has a slightly higher probability than the existing n-gram for source phrase "best shot". The n-gram for target phrase 128 has a slightly higher probability due to a user input indication that replaces source phrase 126 with target phrase 128. However, in some examples, language model 302 cannot determine whether user 10's intent was to say "Beshad" or "best shot" by simply analyzing only source phrase 126 and target phrase 128. For example, "Beshad" could be either an independent name or a specific name such as "Call Beshad" (e.g., a movie title). When "Call Beshad" is a specific name, "Beshad" itself may not have any particularly significant meaning. Therefore, the language model 302 may add additional context (e.g., additional n-grams) adjacent to the target phrase 128, increasing the conditional probability of predicting whether user 10 intended source phrase 126 or target phrase 128. In short, if the target phrase 128 contains adjacent terms that are generally adjacent to it, the probability that user 10 intended target phrase 128 is greater than that that of source phrase 126.

[0051] Referring here to Figure 3, in some examples, the language model 302 includes an n-gram language model that uses conditional probabilities to predict terms in a sequence of n-grams sequentially based on terms that are n or n-1 positions before and / or n or n+1 positions after the target phrase 128. In the example shown, the language model includes an increased conditional probability 310 such that the probability that user 10 intends to convey the target phrase 128 increases as the number of n-grams increases. For example, a sequence with one n-gram is a unigram (e.g., one word), a sequence with two n-grams is a bigram (e.g., two words), and a sequence with three n-grams is a trigram (e.g., three words).

[0052] In the example shown, the increased conditional probability 310 includes a prior likelihood score 304 for the unigram with target phrase 128 (e.g., "Beshad"), which has a slightly higher probability than the prior likelihood score 304 for the unigram with source phrase 126 (e.g., "best shot"). The probability is only slightly higher because the language model 302 does not know the context (e.g., other words) around target phrase 128 and source phrase 126.

[0053] Furthermore, the increased conditional probability 310 includes a prior likelihood score 304 for the bigram with target phrase 128 (e.g., "call Beshad"), which has a higher probability than the prior likelihood score 304 for the bigram with source phrase 126 (e.g., "call best shot"). Here, the margin between the prior likelihood score 304 for target phrase 128 and the source phrase 126 with the bigram (e.g., higher probability) is greater than the margin between the prior likelihood score 304 for target phrase 128 and the source phrase 126 with the unigram (e.g., slightly higher probability). This is because the number of n-grams following target phrase 128 and source phrase 126 increases. The increase in the number of n-grams provides the language model 302 with increased context, allowing it to more accurately predict whether user 10 intended target phrase 128 or source phrase 126. Simply put, because the additional candidate hypothesis 145 includes words that generally contain the term "Beshad," it is more likely that user 10 intended to communicate "Beshad" rather than "best shot."

[0054] Furthermore, the increased conditional probability 310 also applies to the target phrase 128 (for example, " <s>The trigram includes a prior likelihood score of 304 with the phrase "call Beshad", and that score is derived from source phrase 126 (for example, <s>The trigram with "Call Beshad" has a much higher probability than the prior likelihood score of 304. In this example, <s>The beginning of the sentence is shown to provide additional context. Here, the margin between the prior likelihood score 304 of the target phrase 128 and the source phrase 126 of the trigram (e.g., with a much higher probability) is greater than the margin between the prior likelihood score 304 of the target phrase 128 and the source phrase 126 of the bigram and unigram. This is because the number of n-grams following the target phrase 128 and source phrase 126 increases. In other words, by providing the language model 302 with more context, the language model 302 can more accurately predict the probability that user 10 intends to convey either the target phrase 128 or the source phrase 126.

[0055] As shown in Figure 3, in some embodiments, the language model 302 uses an original sequence 326 of the n-gram and a corrected sequence 328 of the n-gram, in addition to the new n-gram. That is, for each phrase correction pair in the corrected phrase list 125, the language model 302 may retrieve the n-gram sequence 325 from the correction history data store 320. The correction history data store 320 stores previous transcriptions 475 and previous corrected transcriptions 476, as described in Figure 4. The n-gram sequence 325 includes an original sequence 326 of the n-gram representing a corresponding previous transcription 475 (Figure 4) that misidentified the corresponding source phrase 126, and a corrected sequence 328 of the n-gram that replaces the source phrase 126 in the original sequence 326 of the n-gram with the corresponding target phrase 128 (e.g., a previous corrected transcription 476 (Figure 4)). Here, the original sequence 326 of the n-gram includes the source phrase 126 and one or more other terms that precede and / or follow the source phrase 126 in the previous transcription 475 (Figure 4). The corrected sequence of the n-gram includes the target phrase 128 and the same one or more other terms that precede and / or follow the corresponding source phrase 126 in the previous transcription 475 (Figure 4).

[0056] In the example shown, the original sequence 326 of n grams is " <s> Call Best Shot< / s> The correction sequence 328, which includes "", is " <s> Call BeShad< / s> This includes the term "call" which precedes either the source phrase 126 or the target phrase 128. In this example, the original sequence 326 and corrected sequence 328 of the n-grams further include n-grams representing the sentence boundaries of the previous transcription 475 transcribed by the speech recognition module 130 for the previous utterance 401 spoken by user 10. That is, <s> This represents the boundary token at the beginning of the sentence.< / s> represents the boundary token at the end of a sentence.

[0057] The computing system 120 modifies the language model 302 by adding the original n-gram sequence 326 and the corrected n-gram sequence 328 to the language model 302 and conditioning the language model 302 to determine that the prior likelihood score 304 for the number of n-grams from the corrected n-gram sequence 328 is higher than the same number of n-grams from the original n-gram sequence 326. That is, if both the original n-gram sequence 326 and the corrected n-gram sequence 328 are bigrams, the computing system 120 conditions the language model 302 to determine that the prior likelihood score 304 for the corrected n-gram sequence 328 is higher. For example, the language model 320 determines that the likelihood score 155 for the corrected n-gram sequence 328 "call Beshad" is higher than that for the original n-gram sequence 326 "call best shot".

[0058] In some examples, computing system 120 conditions language model 302 to determine that the prior likelihood score 304 for a first number of n-grams from a sequence of n-grams containing the target phrase 128 is lower than a larger second number of n-grams from the same sequence. For example, "call Beshad on mobile" should have a higher prior likelihood score 304 than "call Beshad" because it has a larger number of n-grams. These additional n-grams adjacent to the target phrase 128 provide greater confidence that the target phrase 128 is the correct transcription. That is, the correct transcription is likely to contain the target phrase 128 in this context because the target phrase 128 is likely to be found in these other words or phrases.

[0059] In some embodiments, when the language model 302 receives candidate hypothesis 135, "What is Hungary's best shot at the Euro cup," the language model 302 determines that the prior likelihood score 304 is high. This is because candidate hypothesis 135 contains a common sequence of n grams. However, the language model 302 may also receive an additional candidate hypothesis 145, which replaces source phrase 126 with target phrase 128 to produce "What is Hungary's Beshad at the Euro cup." In this example, the language model 302 determines that this sequence of n grams has not been spoken before by user 10. Therefore, the language model 302 reduces the number of n grams leading to "Beshad" and determines the probability of the reduced sequence of n grams.

[0060] For example, language model 302 determines that the n-gram sequence "What is Hungary's Beshad" has also not been spoken before by user 10, or has been spoken very rarely, and therefore language model 302 removes another n-gram. For example, language model 302 determines that the n-gram sequence "Hungary's Beshad" has also not been spoken before by user 10, or has been spoken very rarely, and therefore language model 302 removes yet another n-gram. To continue the example, language model 302 determines that target phrase 128 "Beshad" is a new unigram added by user 10 due to misrecognition transcription. Therefore, even if the prior likelihood score of unigram "Beshad" is 304, which is slightly higher than the prior likelihood score of unigram "best shot", the word history of the n-gram sequence of "best shot" in this scenario may mean that "What is Hungary's best shot at the Euro cup" is the most likely transcription.

[0061] Referring again to Figure 1, the corresponding likelihood score 155 determined for each additional candidate hypothesis 145 is based on at least one of the acoustic modeling score and / or the corresponding prior likelihood score 304 determined for the additional candidate hypothesis 145. During Stage (F), the reranker 160 receives likelihood scores 155 for multiple candidate hypotheses 135 from the speech recognition module 130 and likelihood scores 155 for the additional candidate hypothesis 145 from the hypothesis scorer 300. The reranker 160 is configured to output a reranked result 165 that includes rankings of the multiple candidate hypotheses 135 and the additional candidate hypothesis 145 based on the likelihood scores. In the example shown, the reranked result 165 includes the additional candidate hypothesis 145, with a likelihood score 155 of 0.9, which is considered the most likely correct transcription 175.

[0062] In stage (G), the computing system 120 is configured to generate a transcription 175 of the utterance 101 made by user 10 by selecting the highest-ranking candidate in the re-ranked results 165. Thus, the computing system 120 determines whether a candidate hypothesis 135 containing the source phrase 126 or an additional candidate hypothesis 145 containing the target phrase 128 has a highest likelihood score 155. In the example shown, the computing system 120 selects the additional candidate hypothesis 145 "call Beshad on mobile" because it has a highest likelihood score 155 of 0.9. The computing system 120 provides the transcription 175 to user 10 by transmitting it to the client device 110 via the network 118.

[0063] Figure 5 is a flowchart illustrating an exemplary configuration of the operation of method 500 for performing lattice speech correction. The data processing hardware 114 (Figure 1) of the client device 110 and / or the data processing hardware 122 (Figure 1) of the computing system 120 can perform the operation of method 500 by executing instructions stored in memory hardware 116, 124. In operation 502, method 500 includes receiving audio data 112 corresponding to a query (i.e., utterance) 101 spoken by user 10. In operation 504, method 500 includes processing the audio data 112 using a speech recognition module 130 to generate a number of candidate hypotheses 135. Each candidate hypothesis 135 corresponds to a candidate transcription of the query 101 and is represented by a sequence of hypothetical terms.

[0064] For each candidate hypothesis 135, Method 500 performs operations 506 and 508. In operation 506, Method 500 determines whether each sequence of hypothetical terms contains a source phrase 126 from a list of phrase correction pairs (i.e., a correction phrase list) 125. Each phrase correction pair in the list of phrase correction pairs 125 contains a corresponding source phrase 126 that was misrecognized in a corresponding previous transcription 475 transcribed by the speech recognition module 130 for a previous utterance 401 made by the user 10. Here, each phrase correction pair in the list of phrase correction pairs 125 also contains a corresponding target phrase 128 that matches a user correction that replaces the source phrase 126 that was misrecognized in the corresponding previous transcription 475 transcribed by the speech recognition module 130. In operation 508, when each sequence of hypothetical terms includes a source phrase 126, method 500 includes generating a corresponding additional candidate hypothesis 145 in which the source phrase 126 in each sequence of hypothetical terms is replaced with a corresponding target phrase 128.

[0065] In operation 510, method 500 includes ranking multiple candidate hypotheses 135 and each corresponding additional candidate hypothesis 145. In operation 512, method 500 includes generating a transcription 175 of the query 101 spoken by user 10 by selecting one of the highest-ranking candidates among the multiple candidate hypotheses 135 and each corresponding additional candidate hypothesis 145.

[0066] Figure 6 is a schematic diagram of an example of a computing device 600 that may be used to carry out the systems and methods described in this document. The computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown herein, their connections and relationships, and their functions are for illustrative purposes only and are not intended to limit the embodiments of the invention described and / or claimed in this document.

[0067] The computing device 600 includes a processor 610, memory 620, storage device 630, a high-speed interface / controller 640 connected to memory 620 and high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and can be mounted on a common motherboard or in other ways as needed. The processor 610 processes instructions for execution within the computing device 600, including instructions stored in memory 620 or on storage device 630, and can display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 680 coupled to the high-speed interface 640. In other embodiments, multiple processors and / or multiple buses may be used as needed, along with multiple memories and multiple types of memory. Furthermore, multiple computing devices 600 may be connected, each providing some of the necessary functionality (for example, as a server bank, a group of blade servers, or a multiprocessor system).

[0068] Memory 620 stores information non-temporarily within the computing device 600. Memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporarily stored memory 620 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) temporarily or permanently for use in the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory, read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and disk or tape.

[0069] The storage device 630 can provide mass storage to the computing device 600. In some embodiments, the storage device 630 is a computer-readable medium. In various different embodiments, the storage device 630 may be an array of devices including a floppy disk device, a hard disk device, an optical disk device, or a tape device, flash memory, or other similar solid-state memory devices, or devices in a storage area network or other configuration. In additional embodiments, the computer program product is tangibly embodied within an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or machine-readable medium, such as memory 620, the storage device 630, or memory on the processor 610.

[0070] The high-speed controller 640 manages bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages lower bandwidth-intensive operations. Such role assignments are merely examples. In some embodiments, the high-speed controller 640 is coupled to memory 620, a display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some embodiments, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled, for example, via a network adapter to one or more input / output devices such as a keyboard, pointing device, scanner, or networking devices such as a switch or router.

[0071] The computing device 600 can be implemented in various forms, as shown in the figure. For example, it can be implemented as a standard server 600a, or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0072] Various embodiments of the systems and technologies described herein can be realized in digital electronic circuits and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include embodiments in one or more computer programs executable and / or decipherable in a programmable system comprising at least one programmable processor, the at least one programmable processor may be dedicated or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0073] These computer programs (also known as programs, software, software applications, or code) contain machine instructions to a programmable processor and can be implemented in high-level procedural programming languages ​​and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-temporary computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor, including machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0074] The processes and logic flows described herein are performed by one or more programmable processors (also called data processing hardware) that execute one or more computer programs, and can perform their functions by manipulating input data to produce outputs. Alternatively, the processes and logic flows can be performed by dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Suitable processors for executing computer programs may include, for example, both general-purpose and dedicated microprocessors, as well as any one or more processors in any type of digital computer. Generally, processors receive instructions and data from read-only memory, random-access memory, or both. Essential components of a computer include a processor for issuing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or is operably coupled to mass storage devices to receive data from or transfer data to or both. However, a computer is not required to have such devices. Suitable computer-readable media for storing computer program instructions and data include, for example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory can be complemented by or incorporated into dedicated logic circuits.

[0075] To provide user interaction, one or more aspects of this disclosure can be implemented in a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, and optionally a keyboard and pointing device, such as a mouse or trackball, to which the user can provide input to the computer. Similarly, user interaction can be provided using other types of devices. For example, feedback provided to the user may be any form of perceptual feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, speech input, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from a device used by the user. For example, it can interact by sending a web page to the web browser on the user's client device in response to a request received from a web browser.

[0076] Several embodiments have been described. Nevertheless, it is understood that various modifications can be made without departing from the spirit and scope of this disclosure. Accordingly, other embodiments are within the scope of the following claims.< / s> < / s> < / s>

Claims

1. A method (500) performed by a computer that causes the data processing hardware (122) to perform an operation when the data processing hardware (122) is executed, wherein the operation is Receiving audio data (112) corresponding to the query spoken by the user, The process involves using a speech recognition system to process the audio data (112) and generate a plurality of candidate hypotheses (135), each of which corresponds to a candidate transcription (175) for the query and is represented by a sequence of hypothetical terms. For each candidate hypothesis (135), Each sequence of the aforementioned hypothetical terms is to determine whether it contains a source phrase (126) from the list of phrase correction pairs (125), and each phrase correction pair in the list of phrase correction pairs (125) is The corresponding source phrase (126) that was misrecognized in the corresponding previous transcription (475) transcribed by the speech recognition device for the previous utterance (401) made by the user, The determination includes a corresponding target phrase (128) that matches a user correction that replaces the source phrase (126) that was misrecognized in the corresponding previous transcription (475) transcribed by the speech recognition device, When each sequence of the hypothetical terms includes the source phrase (126), the process involves generating a corresponding additional candidate hypothesis (145) which replaces the source phrase (126) in each sequence of the hypothetical terms with the corresponding target phrase (128), and the process of generating the hypothesis. Ranking the aforementioned multiple candidate hypotheses (135) and each corresponding additional candidate hypothesis (145) that was generated, The process involves generating a transcription (175) of the query spoken by the user by selecting one of the highest-ranking candidates (135) and each of the generated corresponding additional candidate hypotheses (145), Includes, The above operation further applies to each phrase correction pair in the list of phrase correction pairs (125): The acquisition of an original sequence (326) of an n-gram of a language model (302) representing the corresponding previous transcription (475) that misrecognized the corresponding source phrase (126), wherein the original sequence (326) of the n-gram includes the corresponding source phrase (126) and one or more other terms that precede and / or follow the corresponding source phrase (126) in the corresponding previous transcription (475). The acquisition of an n-gram correction sequence (328) which replaces the source phrase (126) in the n-gram original sequence (326) with the corresponding target phrase (128), wherein the n-gram correction sequence (328) includes the target phrase (128) and one or more other terms that precede and / or follow the corresponding source phrase (126) in the corresponding previous transcription (475), The aforementioned language model (302) The original sequence of n-grams (326) and the correction sequence of n-grams (328) are added to the language model (302), The language model (302) is modified by conditioning it to determine that the prior likelihood score (304) of the number of n-grams from the corrected n-gram sequence (328) containing the target phrase (128) is higher than the same number of n-grams from the original n-gram sequence (326) containing the source phrase (126), This includes determining a corresponding prior likelihood score (304) for each additional candidate hypothesis (145) using the modified language model (302) configured to receive each additional candidate hypothesis (145) as input, The computer-based method (500) ranks the aforementioned multiple candidate hypotheses (135) and each additional candidate hypothesis (145) based on the corresponding prior likelihood score (304) determined for each additional candidate hypothesis (145).

2. The computer-implemented method (500) according to claim 1, wherein the margin between the number of n-grams from the n-gram correction sequence (328) and the prior likelihood score (304) determined by the language model (302) for the same number of n-grams from the n-gram original sequence (326) increases as the number of n-grams from the n-gram correction sequence and the original sequence increases.

3. The computer-based method (500) according to claim 1 or 2, further comprising conditioning the language model (302) to determine that a prior likelihood score (304) of a first number of n grams from the sequence of n grams containing the target phrase (128) is lower than a larger second number of n grams from the sequence of n grams containing the target phrase (128).

4. The computer-based method (500) according to any one of claims 1 to 3, wherein the original sequence and corrected sequence of the n-grams each further include n-grams representing sentence boundaries of the corresponding previous transcription (475) transcribed by the speech recognition device for the previous utterance (401) made by the user.

5. The aforementioned operation further, For each of the multiple candidate hypotheses (135) generated by the speech recognition device, the corresponding likelihood score (155) assigned by the speech recognition device to the corresponding candidate hypothesis (135) is obtained. This includes generating each additional candidate hypothesis (145), and then using an additional hypothesis scorer (300) to determine the corresponding likelihood score (155) for each generated additional candidate hypothesis (145), A computer-based method (500) according to any one of claims 1 to 4, wherein ranking the plurality of candidate hypotheses (135) and each corresponding additional candidate hypothesis (145) generated is based on the corresponding likelihood scores (155) assigned to the plurality of candidate hypotheses (135) by the speech recognition system and the corresponding likelihood scores (155) determined for each additional candidate hypothesis (145) using the additional hypothesis scorer (300).

6. The aforementioned additional hypothesis scorer (300) is, An acoustic model (301) configured to process audio data (112) and determine an acoustic modeling score for a portion of the audio data (112) that includes either the source phrase (126) or the target phrase (128), or The language model (302) includes at least one of the following: which receives each additional candidate hypothesis (145) as input and is configured to determine the corresponding prior likelihood score (304) for each additional candidate hypothesis (145). The computer-based method (500) according to claim 5, wherein the corresponding likelihood score (155) determined for each additional candidate hypothesis (145) is based on at least one of the acoustic modeling score or the corresponding prior likelihood score (304) determined for each additional candidate hypothesis (145).

7. The computer-based method (500) according to claim 6, wherein the language model (302) includes an auxiliary language model (302) located outside the speech recognition device, or an internal language model (302) integrated with the speech recognition device.

8. The computer-based method (500) according to claim 6 or 7, wherein the speech recognition device includes an end-to-end speech recognition model configured to generate the corresponding likelihood score (155) for each of the plurality of candidate hypotheses (135).

9. The aforementioned speech recognition device includes an acoustic model (301) and a language model (302). The computer-based method (500) according to any one of claims 5 to 7, wherein the corresponding likelihood score (155) assigned by the speech recognition device to each of the plurality of candidate hypotheses (135) is based on at least one of the acoustic modeling score output by the acoustic model (301) or the corresponding prior likelihood score (304) output by the language model (302).

10. System (100), Data processing hardware (122) and The system includes memory hardware (124) that communicates with the data processing hardware (122) and stores instructions that cause the data processing hardware (122) to perform an action when executed by the data processing hardware (122), and the action is Receiving audio data (112) corresponding to the query spoken by the user, The process involves using a speech recognition system to process the audio data (112) and generate a plurality of candidate hypotheses (135), each of which corresponds to a candidate transcription (175) for the query and is represented by a sequence of hypothetical terms. For each candidate hypothesis (135), Each sequence of the aforementioned hypothetical terms is to determine whether it contains a source phrase (126) from the list of phrase correction pairs (125), and each phrase correction pair in the list of phrase correction pairs (125) is The corresponding source phrase (126) that was misrecognized in the corresponding previous transcription (475) transcribed by the speech recognition device for the previous utterance (401) made by the user, The determination includes a corresponding target phrase (128) that matches a user correction that replaces the source phrase (126) that was misrecognized in the corresponding previous transcription (475) transcribed by the speech recognition device, When each sequence of the hypothetical terms includes the source phrase (126), the process involves generating a corresponding additional candidate hypothesis (145) which replaces the source phrase (126) in each sequence of the hypothetical terms with the corresponding target phrase (128), and the process of generating the hypothesis. Ranking the aforementioned multiple candidate hypotheses (135) and each corresponding additional candidate hypothesis (145) that was generated, The process involves generating a transcription (175) of the query spoken by the user by selecting one of the highest-ranking candidates (135) and each of the generated corresponding additional candidate hypotheses (145), Includes, The above operation further applies to each phrase correction pair in the list of phrase correction pairs (125): The acquisition of an n-gram original sequence (326) of a language model (302) representing the corresponding previous transcription (475) that misrecognized the corresponding source phrase (126), wherein the n-gram original sequence (326) includes the corresponding source phrase (126) and one or more other terms that precede and / or follow the corresponding source phrase (126) in the corresponding previous transcription (475), The acquisition of an n-gram correction sequence (328) which replaces the source phrase (126) in the n-gram original sequence (326) with the corresponding target phrase (128), wherein the n-gram correction sequence (328) includes the target phrase (128) and one or more other terms that precede and / or follow the corresponding source phrase (126) in the corresponding previous transcription (475), The aforementioned language model (302) The original sequence of the n-grams and the corrected sequence of the n-grams are added to the language model (302), The language model (302) is modified by conditioning it to determine that the prior likelihood score (304) of the number of n-grams from the corrected sequence of n-grams containing the target phrase (128) is higher than the same number of n-grams from the original sequence of n-grams containing the source phrase (126), This includes determining a corresponding prior likelihood score (304) for each additional candidate hypothesis (145) using the modified language model (302) configured to receive each additional candidate hypothesis (145) as input, The system (100) ranks the aforementioned multiple candidate hypotheses (135) and each additional candidate hypothesis (145) based on the corresponding prior likelihood score (304) determined for each additional candidate hypothesis (145).

11. The system (100) according to claim 10, wherein the margin between the number of n-grams from the n-gram correction sequence and the prior likelihood score (304) determined by the language model (302) for the same number of n-grams from the original sequence of n-grams increases as the number of n-grams from the n-gram correction sequence and the original sequence increases.

12. The system (100) according to claim 10 or 11, wherein conditioning the language model (302) further includes conditioning the language model (302) to determine that a prior likelihood score (304) of a first number of n grams from the sequence of n grams containing the target phrase (128) is lower than a larger second number of n grams from the sequence of n grams containing the target phrase (128).

13. The system (100) according to any one of claims 10 to 12, wherein the original sequence and corrected sequence of the n-grams each further include n-grams representing sentence boundaries of the corresponding previous transcription (475) transcribed by the speech recognition device for the previous utterance (401) made by the user.

14. The aforementioned operation further, For each of the multiple candidate hypotheses (135) generated by the speech recognition device, the corresponding likelihood score (155) assigned by the speech recognition device to the corresponding candidate hypothesis (135) is obtained. This includes generating each additional candidate hypothesis (145), and then using an additional hypothesis scorer (300) to determine the corresponding likelihood score (155) for each generated additional candidate hypothesis (145), The system (100) according to any one of claims 10 to 13, wherein ranking the plurality of candidate hypotheses (135) and each corresponding additional candidate hypothesis (145) generated is based on the corresponding likelihood score (155) assigned to the plurality of candidate hypotheses (135) by the speech recognition system and the corresponding likelihood score (155) determined for each additional candidate hypothesis (145) using the additional hypothesis scorer (300).

15. The aforementioned additional hypothesis scorer (300) is, An acoustic model (301) configured to process audio data (112) and determine an acoustic modeling score for a portion of the audio data (112) that includes either the source phrase (126) or the target phrase (128), or The language model (302) includes at least one of the following: which receives each additional candidate hypothesis (145) as input and is configured to determine the corresponding prior likelihood score (304) for each additional candidate hypothesis (145). The system (100) according to claim 14, wherein the corresponding likelihood score (155) determined for each additional candidate hypothesis (145) is based on at least one of the acoustic modeling score or the corresponding prior likelihood score (304) determined for each additional candidate hypothesis (145).

16. The system (100) according to claim 15, wherein the language model (302) includes an auxiliary language model (302) located outside the speech recognition device, or an internal language model (302) integrated with the speech recognition device.

17. The system (100) according to claim 15 or 16, wherein the speech recognition device includes an end-to-end speech recognition model configured to generate the corresponding likelihood score (155) for each of the plurality of candidate hypotheses (135).

18. The aforementioned speech recognition device includes an acoustic model (301) and a language model (302). The system (100) according to any one of claims 14 to 16, wherein the corresponding likelihood score (155) assigned by the speech recognition device to each of the plurality of candidate hypotheses (135) is based on at least one of the acoustic modeling score output by the acoustic model (301) or the corresponding prior likelihood score (304) output by the language model (302).

Images