Multimodal LLM that learns to self-correct, with a focus on ASR

Abstract

Systems and methods are disclosed that are designed for an assistive large language model (LLM) with self-correction. An assistive LLM can operate autoregressively and perform language-related tasks such as automatic speech recognition, speech translation, and / or machine translation.

Description

TECHNICAL AREA

[0001] This revelation refers to a multimodal LLM that learns to self-correct, with a focus on ASR. GENERAL STATE OF THE ART

[0002] Multimodal large language models (LLMs) are capable of handling a variety of tasks, such as performing actions and / or answering questions based on text, images, audio recordings, and / or video content. Some LLMs include specialized models or modules for tasks involving automatic speech recognition (ASR), speaker diarization, and automatic speech translation (AST). Typically, these models use an autoregressive decoding mechanism that generates one or more output tokens based on previous tokens and coded hidden representations. However, autoregressive output token generation can cause the LLM to ignore the future context provided by subsequent input frames. Ignoring future context can negatively impact the LLM's accuracy. DESCRIPTION OF THE DRAWINGS Fig. 1A and Fig. Figure 1B shows schematic views of an example of a digital assistance system. Fig. Figure 2 is a schematic view of an example training data generation process. Fig. Figure 3 is a schematic view of an example of training an assisting large language model. Fig. Figure 4 is a flowchart of an exemplary sequence of operations for a computer-implemented procedure for an assisting large language model that corrects itself. Fig. Figure 5 is a schematic view of an exemplary computing device that can be used to implement the systems and procedures described here.

[0003] Identical reference symbols in the different drawings refer to the same elements. DETAILED DESCRIPTION

[0004] Multimodal large language models (LLMs) are capable of handling a variety of tasks, such as performing actions and / or answering questions based on text, images, audio recordings, and / or video content. Some LLMs include specialized models or modules for tasks involving automatic speech recognition (ASR), speaker diarization, and automatic speech translation (AST). Typically, these models employ an autoregressive decoding mechanism that generates one or more output tokens based on previous tokens and encoded hidden representations.

[0005] A significant drawback of autoregressive decoding is the inability to revise past predictions. That is, once a particular token has been selected or output in a specific output step, all future predictions of output tokens are conditioned on that token, even if that token is later found to have been incorrectly identified. Thus, the inability to revise past predictions with autoregressive decoding can lead to cascading errors in the generation of subsequent output tokens, contributing to problems such as hallucinations and other misidentifications. In ASR tasks, for example, hallucinations might manifest as word or phrase repetitions, incorrect completions, or even responses that were not present in the audio input.

[0006] Existing approaches to mitigating hallucinations and other recognition errors include improved loss functions and expanded training data for these models. However, these approaches never teach the models how to detect and correct such errors as they occur. This failure to correct these errors is particularly problematic for streaming models, such as those providing streaming ASR, speech translation, and machine translation, which generate output based on incrementally received or otherwise incomplete information. This means that streaming models may not benefit from processing future acoustic frames to obtain the most accurate results, as streaming models prioritize minimal latency over accuracy.Thus, streaming models may initially produce partially incorrect results to minimize latency and, due to the nature of autoregressive decoding, may not be able to correct previous incorrect predictions.

[0007] Accordingly, the implementations contained herein are designed for a self-correcting assisted large language model (LLM). As you will see, the assisted LLM can operate autoregressively and perform language-related tasks such as automatic speech recognition, speech translation, and / or machine translation. The assisted LLM receives a prompt and generates a sequence of output tokens based on this prompt. For example, the prompt might contain a sequence of acoustic images corresponding to a spoken utterance, with the sequence of output tokens generated by the assisted LLM containing a speech recognition result for the spoken utterance. In another example, the prompt might contain a text token representation in a first language, with the sequence of output tokens generated by the assisted LLM containing a translated text token representation in a second language.The output token sequence includes a sequence of text tokens and one or more revision tokens. The text sequence contains one or more correct text characters and one or more incorrect text characters. Each revision token specifies a corresponding N number of incorrect text tokens generated before that revision token and corresponding replacement text tokens generated after that revision token to replace those incorrect text tokens. An output layer then generates a revised sequence of output tokens for the prompt based on this output token sequence. In some examples, the output layer can be integrated with the assisting LLM, so that the assisting LLM generates the revised output token sequence using the output layer.In other examples, the output layer resides outside the assistant LLM, so the assistant LLM passes the output token sequence to the output layer and instructs the output layer to generate the revised token sequence. The output layer might, for example, reside in a downstream application of a user device.

[0008] Generating one or more revision tokens allows the assisting LLM to revise past predictions after processing subsequent acoustic frames while operating autoregressively. That is, the revision tokens indicate that one or more of the predicted text tokens are erroneous, and one or more replacement text tokens replace the erroneous predictions. Thus, the revision tokens enable the models to revise previous predictions while continuing to autoregressively generate the sequence of output tokens. This revision of past predictions can be applied to models operating in streaming mode, non-streaming mode, or a combination thereof. The assisting LLM can thereby advantageously operate in streaming mode to reduce latency and benefit from the additional context of future acoustic frames.

[0009] Fig. Figure 1 shows an example system 100 that includes a digital assistance system 105. Generally, the user 10 inputs a prompt 116 via a user device 110, which is directed to an assisting large language model (LLM) 150. In some examples, the prompt 116 can specify an action for the assisting LLM 150 to perform. As will be shown, the assisting LLM 150 generates a sequence of output tokens 152 based on processing the prompt 116. In some examples, the user 10 speaks an utterance 106 or a natural language request, which serves as the prompt 116 to the assisting LLM 150. The prompt 116 can include audio data 102 of the natural language request or utterance 106 spoken by the user 10. Additionally or alternatively, the prompt 116 can include a textual representation 104 of a request or utterance in natural language, which the user 10 provides as text input (e.g.(via the keyboard or a graphical user interface). The assisting LLM 150 can be a multimodal LLM configured to process various types of input and output, such as audio, text, video, and / or images. Therefore, depending on the type and content of Prompt 116, the assisting LLM 150 can perform different tasks when processing Prompt 116.

[0010] For example, if prompt 116 contains the audio data 102 of an utterance 106 spoken by user 10, the assisting LLM 150 can perform automatic speech recognition (ASR) on the audio data 102 to generate a speech recognition result (i.e., a log) of the spoken utterance 106. Utterances 106 spoken by user 10 can include voice commands, such as queries to which the assisting LLM 150 should respond, or commands that instruct the assisting LLM 150 to perform a specific action. For example, the query might be "What is the weather like today?", to which the assisting LLM 150 responds with an answer about the weather, or the command might be "Schedule a meeting for tomorrow," to which the assisting LLM 150 schedules the meeting.In another example, prompt 116 can contain audio data 102 of an utterance 106 spoken by user 10 in a first language and a request (e.g., spoken by user 10 or provided as text input) to translate the utterance 106 into a second language, so that the assisting LLM 150 performs automatic speech translation (AST) on the utterance 106 to generate synthetic speech in the second language. The synthetic speech in the second language is a translation of the utterance 106 spoken by user 10 in the first language. In yet another example, prompt 116 can contain a text representation 104 entered by user 10 in a first language and a request to translate the text representation 104 into a second language, so that the assisting LLM 150 performs machine translation of the text representation to generate a translation of the text representation 104 into the second language.The textual representation 104 can be provided by a user 10 as text or be a record of an utterance 106 spoken by the user 10.

[0011] In some examples, the assisting LLM 150 incorporates an encoder-decoder architecture. For instance, the assisting LLM 150 might include a recurrent neural network-transducer (RNN-T) architecture. In this case, the encoder of the assisting LLM 150 generates encodings based on Prompt 116, and the decoder of the assisting LLM 150 decodes these encodings to produce an output. The encoder might include a stack of multi-headed self-assertion layers (e.g., conformers or transformer layers). In other examples, the assisting LLM 150 incorporates a decoder-only architecture. In these examples, the assisting LLM 150 omits the encoder, allowing the decoder of the assisting LLM 150 to directly process Prompt 116 to generate the outputs. In some examples, a language model is used instead of the assisting LLM 150 (not shown).For example, a speech recognition model can process the audio data 102 of an utterance 106 to generate a speech recognition result of the utterance, or a translation model can process the textual representation 104 to generate a translation of the textual representation 104.

[0012] The system includes the user device 110, a remote computing device system 120, and a network 130. The user device 110 includes data processing hardware 113 and storage hardware 114. The user device 110 may include or communicate with an audio recording device 115 (e.g., by means of one or more microphones) to convert spoken utterances 106 or natural language queries by the user 10 into corresponding audio data (e.g., a sequence of acoustic still images) 102. In addition to or instead of spoken input, the user 10 may input a textual representation 104 of the query in natural language via a user interface running on the user device 110.

[0013] The user device 110 can be any computing device capable of communicating with the remote computing system 120 over the network 130. The user device 110 includes, but is not limited to, desktop computing devices and mobile computing devices such as laptops, tablets, smartphones, smart speakers / displays, digital assistive devices, smart devices, Internet of Things (IoT) devices, infotainment systems, vehicle infotainment systems, and wearable computing devices (e.g., headsets, smart glasses, and / or watches). The remote computing system 120 can be a distributed system (e.g., a cloud computing environment) with scalable, elastic resources. These resources include computing resources 123 (e.g., data processing hardware) and / or storage resources 124 (e.g., storage hardware). Additionally or alternatively, the remote computing system 120 can be a centralized system.The network 130 can be wired, wireless or a combination thereof and can include private networks and / or public networks such as the Internet.

[0014] The assisting LLM 150 is configured to process prompt 116 to generate the sequence of output tokens 152. In some configurations, the assisting LLM 150 is a streaming model that operates autoregressively. More precisely, the assisting LLM 150 contains a decoder that generates a corresponding output token 152 at each of a multitude of output steps, based on previously generated output tokens 152. That is, the assisting LLM 150 predicts the next output token 152 in the sequence based on the output tokens 152 generated before it. Thus, each output token 152 is generated individually, with each new output token 152 dependent on the previously generated output tokens.In other words, at each of a multitude of output steps, the assisting LLM 150 generates a corresponding output token 152 or a corresponding acoustic frame 102 within the sequence of acoustic frames 102. Consequently, when generating each new output token 152, the assisting LLM 150 cannot benefit from the context provided by the subsequent output tokens 152. Here, the context can refer to the linguistic context, which provides additional information for disambiguating words and phrases. The linguistic context can help the assisting LLM 150 anticipate upcoming words, enabling it to more accurately predict and interpret the current input, particularly in cases involving homophones or ambiguous phrases.Thus, the linguistic context enables the system to better understand the overall meaning and intention of the speech, leading to more accurate transcripts. As will be shown, future context can help correct errors by re-evaluating previous words in light of new information.

[0015] The sequence of output token 152 contains a sequence of text tokens 154 and one or more revision tokens 156. Each text token 154 in the sequence of text tokens 154 contains a text unit, such as a word, a word part, a grapheme, etc. The sequence of text tokens 154 contains one or more correct text tokens 154, 154a, and one or more erroneous text tokens 154, 154b. The correct text tokens 154a represent the correctly generated text tokens 154 with respect to the subsequently generated text tokens 154. Each correct text token 154a is a text token 154 that is initially generated by the assisting LLM 150 and correctly corresponds to prompt 116. This means that the correct text token 154a represents a correctly generated text token 154, which is not subsequently replaced by another text token. For example, in a speech recognition context, a correct text token 154a refers to a text token 154 that exactly corresponds to utterance 106.In another example, a correct text token 154a in the machine translation context refers to a text token 154 that exactly corresponds to a translation of the textual representation 104.

[0016] On the other hand, erroneous text tokens 154b represent inaccurate text tokens 154 with respect to subsequently generated text tokens 154. That is, the erroneous text token 154b represents an inaccurately generated text token 154, which can then be replaced by another text token. For example, an erroneous text token 154b in the speech recognition context refers to a text token 154 that inaccurately corresponds to utterance 106. In another example, an erroneous text token 154b in the machine translation context refers to a text token 154 that inaccurately corresponds to a translation of textual representation 104.

[0017] If the assisting LLM 150 operates autoregressively, it can generate a corresponding text token 154 based on previously generated text tokens 154, thus leading the assisting LLM 150 to consider this text token 154 as correct based on the previously generated text tokens 154. Subsequently, the assisting LLM 150 generates one or more text tokens 154 after the respective text token 154, thereby providing the assisting LLM 150 with further context. Based on this additional context provided by the one or more text tokens 154 generated after the respective text token 154, the assisting LLM 150 may no longer consider the previously generated text token 154 to be correct.

[0018] For this purpose, the sequence of output tokens 152 can include one or more revision tokens 156. Each revision token 156 identifies a corresponding N number of erroneous text tokens 154b generated before that revision token 156, and corresponding replacement text tokens 154 and 154c generated after that revision token 156 to replace those N numbers of erroneous text tokens 154b. In simpler terms, each revision token 156 indicates that the assisting LLM 150 has determined that one or more of the previously output text tokens 154 are erroneous in light of the additional context, and also specifies the corresponding replacement text tokens 154c to replace the erroneous text tokens 154b.The N number of erroneous text tokens 154b identified by each revision token 156 can specify all erroneous text tokens 154 that were generated before the respective revision token 156, but after the revision token 156 that precedes that revision token 156. For example, a second revision token 156 in the sequence of output tokens 152 can specify all erroneous text tokens 154b that were generated before the second revision token 156, but after a first revision token 156.

[0019] In some implementations, the N number of erroneous text tokens 154b identified by each revision token 156 are located immediately before that revision token 156. For example, a revision token 156 might identify three erroneous text tokens 154b generated before it, with the three erroneous text tokens 154b being the three text tokens 154 immediately preceding it. In other implementations, the N number of erroneous text tokens 154b identified by each revision token 156 are located an offset number of output tokens 152 away from that revision token 156.For example, a revision token 156 can identify three erroneous text tokens 154b that were created before the revision token 156, with one or more correct text tokens 154a located between the revision token 156 and the three erroneous text tokens 154b.

[0020] Fig. Figure 1A shows a first example system 100, 100a, where the prompt 116 contains audio data (e.g., the sequence of acoustic frames) 102 that corresponds to an utterance 106 spoken by the user 10, "I want to buy the red car," and the assisting LLM 150 aims to generate a speech recognition result 162 that corresponds to the utterance 106. The assisting LLM 150 processes the audio data 102 of the prompt 116 to generate, based on the processing of the sequence of acoustic frames 102, the sequence of output tokens 152 that corresponds to the statement "I would like to [Revise 2] buy a [Revise_1] the red car." Here, the text tokens 154 corresponding to "I", "buy", and "red car" represent the correct text tokens 154a. The text tokens 154 corresponding to "want" and "a" represent erroneous text tokens 154b and are indicated by the dashed boxes.The erroneous text tokens 154b represent text tokens 154 that the assisting LLM 150 believed to be correct at the corresponding output step when they were generated, but which the assisting LLM 150 subsequently believes to be inaccurate based on later-generated text tokens 154 that provide additional context. Furthermore, "[Revise_2]" represents a revision token 156 to replace the two previous erroneous text tokens 154b of "möchte" with the replacement text tokens 154c of "würde gerne," and "[Revise_1]" represents a revision token 156 to replace the one previous erroneous text token 154b of "ein" with the replacement text token 154c of "das." Here, the replacement text tokens 154c are indicated by the filled boxes.

[0021] Output layer 160 is configured to process the sequence of output tokens 152 to generate a revised sequence of output tokens 152, 152R, which may include revised speech recognition results 162 for utterance 106. In other words, output layer 160 is configured to make one or more revisions (if any) to the sequence of output tokens 152 based on the presence of revision tokens 156. More precisely, for each revision token 156 of the one or more revision tokens 156, output layer 160 identifies the corresponding N number of erroneous text tokens 154b specified by that revision token 156 and replaces that N number of erroneous text tokens 154b with the corresponding replacement text token 154c specified by that revision token 156.Thus, the speech recognition result 162 generated by the output layer 160 includes the one or more correct text tokens 154a from the sequence of output tokens 152 and excludes the one or more erroneous text tokens 154b, which are replaced by the corresponding substitute text tokens 154c.

[0022] As in the example shown, output layer 160 receives the sequence of output token 152 corresponding to "I would like [Revise_2] would like to buy a [Revise_1] the red car" and generates the revised sequence of output token 152R, including the revised speech recognition result 162 of "I would like to buy the red car". Based on the "[Revise_2]" revision token 156, output layer 160 replaces the erroneous text tokens 154b of "would like" with the corresponding replacement text tokens 154c of "would like" and replaces the erroneous text token 154b of "a" with the corresponding replacement text token 154c of "the".

[0023] Fig. Figure 1B shows a second example system 100, 100b, in which the prompt 116 contains audio data (e.g., the sequence of acoustic images) 102, corresponding to an utterance 106 spoken by a user 10: “Translate this into Spanish: What will the weather be like tomorrow?” The assisting LLM 150 processes a textual representation 104 of the utterance 106 “What will the weather be like tomorrow?” The textual representation 104 can be transcribed based on spoken speech input or provided directly by the user 10 as a textual representation 104. In this example, the assisting LLM 150 is to translate the sentence “What will the weather be like tomorrow?” from English into Spanish. For this purpose, the assisting LLM 150 processes the textual representation 104 of the prompt 116 to generate the sequence of output tokens 152 that corresponds to “¿Cómo estará el pluma [Revise_1] tiempo mañana?”.Here, the text tokens 154 corresponding to "I", "buy", and "red car" represent the correct text tokens 154a. The text token 154 corresponding to "pluma" represents an erroneous text token 154b and is indicated by the dashed border. The erroneous text token 154b represents a text token 154 that the assisting LLM 150 believed to be accurate at the corresponding output step when it was generated, but afterward, the assisting LLM 150 believes it to be inaccurate based on later-generated text tokens 154 that provide additional context. Additionally, "[Revise_1]" represents a revision token 156 to replace a previous erroneous text token 154b of "pluma" with the replacement text token 154c of "tiempo". Here, the replacement text tokens 154c are indicated by the filled boxes.

[0024] Output layer 160 is configured to process the sequence of output tokens 152 to generate a revised sequence of output tokens 152, 152R, which may include a revised translated text representation 164 for text representation 104. In other words, output layer 160 is configured to make one or more revisions (if any) to the sequence of output tokens 152 based on the presence of revision tokens 156. More precisely, for each revision token 156 of the one or more revision tokens 156, output layer 160 identifies the corresponding N number of erroneous text tokens 154b specified by that revision token 156 and replaces that N number of erroneous text tokens 154b with the corresponding replacement text token 154c specified by that revision token 156.Thus, the revised translated text representation 164, generated by the output layer 160, includes the one or more correct text tokens 154a from the sequence of output tokens 152 and excludes the one or more erroneous text tokens 154b, which are replaced by the corresponding replacement text tokens 154c.

[0025] To continue with the example shown, output layer 160 receives the sequence of output token 152 corresponding to “¿Cómo estará el pluma [Revise_1] tiempo mañana?” and generates the revised sequence of output token 152R, including the revised translated text representation 164 of “¿Cómo estará el tiempo mañana? Ich würde gerne das rote Auto kaufen”. Based on the “[Revise_1]” revision token 156, output layer 160 replaces the erroneous text tokens 154b of “pluma” with the corresponding replacement text tokens 154c of “tiempo”.

[0026] The assisting LLM 150 determines at each output step of the multitude of output steps whether the next text token 154 in the sequence of text tokens 154 or a revision token 156 should be output. That is, at each output step, the assisting LLM 150 can determine, based on the subsequently generated output tokens 152, whether any of the previous text tokens 154 are faulty. If the assisting LLM 150 determines that one or more of the previous text tokens 154 are faulty at a given output step, it generates a revision token 156 instead of a text token 154 at that output step. Conversely, if the assisting LLM 150 does not determine that one or more of the previous text tokens 154 are faulty at a given output step, it outputs a different text token 154 at that output step.

[0027] In some implementations, output layer 160 is integrated with the assisting LLM 150, so the output generated by output layer 160 is the output of the assisting LLM 150. Here, the assisting LLM 150 generates the revised sequence of output tokens 152R with the help of output layer 160. In other implementations, output layer 160 is located outside of the assisting LLM 150 and resides in one or more downstream applications of the user device 110. Here, the assisting LLM 150 can transfer the sequence of output tokens 152 to output layer 160, causing output layer 160 to generate the revised sequence of output tokens 152R based on the sequence of output tokens 152.

[0028] Fig. Figure 2 illustrates the training data generation process. In some cases, the assisting LLM is prone to a continuation error when performing the ASR. The assisting LLM can be trained or fine-tuned on a corpus of pure text training examples so that it learns to predict the next likely text token based on previously generated text tokens. Therefore, the assisting LLM may inadvertently predict continuation text words that are likely continuations of previously generated text words, even though the audio data does not contain such a continuation. For example, during training with the pure training examples, the assisting LLM may learn that a likely sequence is "How are you? I'm fine, thank you." Thus, the assisting LLM might incorrectly predict the speech recognition result "How are you?" as "I'm fine, thank you."Where were you last week?”, when you hear utterance 106 “How are you? Where were you last week?” That is, the assisting LLM 150 might mistakenly assume that the sentence “I’m fine, thank you” is included in utterance 106 due to pure text training, even though no such sentence was spoken.

[0029] For this purpose, the training data generation process 200 is configured to generate a variety of extended training protocols 232, which the assisting LLM 150 uses for training. The training data generation process 200 obtains a variety of training samples 310. Each corresponding training sample 310 contains audio data 204 that characterizes a spoken utterance and is paired with a corresponding protocol (i.e., a training protocol) 202. The training data generation process includes a prompt generator 210 that receives a variety of training protocols 202. The training protocols 202 contain plain text data that can optionally be paired with corresponding audio data. For each training protocol 202, the prompt generator 210 generates a corresponding continuation prompt 212. The continuation prompt 212 can request a likely text continuation for the training protocol 202 or a section thereof.Therefore, the prompt generator 210 can generate the continuation prompt 212 by extracting a section of the training protocol 202 and generating the continuation prompt 212 based on the extracted section. For example, for the training protocol 202 of "How are you? Where have you been last week?", the prompt generator 210 can extract "How are you?" and generate the continuation prompt 212 "Please continue with the following text. How are you?".

[0030] Next, an auxiliary LLM 220, distinct from the assisting LLM 150, receives the continuation prompt 212 and generates a continuation output 222 based on it. The continuation output 222 represents a likely continuation of the text that would follow the training protocol 202 or the extracted portion of the training protocol 202. Continuing the example above, for the continuation prompt 212, which represents a likely continuation after "How are you?", the LLM 220 can generate the output 222 "I'm fine, thank you". Finally, an extension module 230 generates the extended training protocol 232 based on the training protocol 202 and the corresponding continuation output 222 generated by the auxiliary LLM 220. In particular, the extension module 230 can insert or append the continuation issue 222 to the training protocol 202 and a corresponding revision token 156.That is, since the continuation output 222 inserted into the training protocol 202 represents a simulated error, the extension module 230 also inserts the revision token 156 to indicate that the continuation output 222 should be deleted.

[0031] To continue with the example shown, the extension module 230 generates the extended training protocol 232 "How are you? I'm fine, thank you. [Revise_4] Where were you last week?" Here, the extension module 230 has inserted the output token 222 "I'm fine, thank you." after "How are you?" and the revision token 156 "[Revise_4]" after output token 222 to indicate that "I'm fine, thank you." should be deleted. Revision token 156 simply indicates that "I'm fine, thank you" should be deleted without the replacement text token 154c.

[0032] In some implementations, training protocol 202 includes a ground-truth protocol and an incorrectly detected protocol. For example, training protocol 202 might contain the ground-truth protocol of "How are you today?" (e.g., the correct protocol) and the incorrectly detected protocol of "How are you about?". Here, extension module 230 can directly receive training protocol 202 and generate the extended training protocol 232 based on the ground-truth protocol and the incorrectly detected protocol. Referring to the example above, extension module 230 might, for instance, generate the extended training protocol 232 of "How are you about? [Revise_2] today?"

[0033] Fig. Figure 3 shows a training process 300 for training the assisting LLM 150. The training process 300 obtains the multitude of training samples 310. Each training sample contains audio data 204 that identifies a spoken utterance and is paired with a corresponding protocol (i.e., a training protocol) 202. Each protocol 202 can be generated by the training data generation process 200 ( Fig. 2) are extended so that each training sample 310 includes a corresponding extended training protocol 232, which serves as the base protocol. For each training sample 310 of the plurality of training samples 310, the assisting LLM 150 generates a corresponding sequence of output tokens 152 based on the audio data 204 or the training protocol 202 of the respective training sample 310. The assisting LLM 150 can, for example, generate the sequence of output tokens 152, which represents a protocol of the audio data 204 or a translated text token representation of the training protocol 202. The sequence of output tokens 152 can include a sequence of text tokens 154 and one or more revision tokens 156.

[0034] A loss module 320 then receives the sequence of output tokens 152 generated for each corresponding training sample 310 and determines a loss 322 by comparing the sequence of output tokens 152 with the corresponding extended training protocol 232. The training process 300 can train the assisting LLM 150 based on the loss 322 determined for each of the multitude of training samples 310. In some implementations, the assisting LLM 150 determines an N-best list of sequences of output tokens 152 for each training sample 310. For example, the assisting LLM 150 can determine ten sequences of output tokens 152 for each training sample. Here, the loss module 320 determines the loss 322 by comparing each sequence of output token 152 from the N-best list of sequences of output token 152 with the extended training sample 232 to determine the loss 322.

[0035] Fig. Figure 4 is a flowchart of an exemplary sequence of operations for a computer-implemented procedure 400 for use with an assisting LLM 150 with self-correction. The procedure 400 can be run on the data processing hardware 510 ( Fig. 5) are executed based on instructions that are based on the memory hardware 520 ( Fig. 5) are stored. In some examples, the data processing hardware 510 includes the data processing hardware 113 of the user device 110, and the storage hardware 520 includes the storage hardware 114 of the user device 110. In other examples, the data processing hardware 510 includes the data processing hardware 123 of the remote computing system 120, and the storage hardware 520 includes the data processing hardware 124 of the remote computing system 120.

[0036] In process 402, procedure 400 involves receiving a prompt 116 directed to the assisting LLM 150. In process 404, procedure 400 involves generating a sequence of output tokens 152 based on the prompt 116 using the assisting LLM 150. The sequence of output tokens 152 can include a speech recognition result if the prompt 116 contains a sequence of acoustic frames 102 corresponding to a user-spoken utterance 106. Alternatively, the sequence of output tokens 152 can include a translated text token representation in a translated language if the prompt 116 contains a text token representation 104 in a source language. The sequence of output token 152 includes a sequence of text token 154 and one or more revision tokens 156. The sequence of text token 154 includes one or more correct text tokens 154a and one or more erroneous text tokens 154b.Each revision token specifies a corresponding N number of erroneous text tokens 154b generated before the respective revision token 156, and corresponding replacement text tokens 154c generated after the respective revision token 156 to replace the corresponding N number of erroneous text tokens 154b. In operation 406, procedure 400 involves generating a revised sequence of output tokens 152R for the prompt based on the sequence of output tokens 152. The revised sequence of output tokens 152 may include a revised speech recognition result 162 if the assisting LLM 150 performs speech recognition, or a revised translated text representation 164 if the assisting LLM 150 performs machine translation.

[0037] Fig.Figure 5 is a schematic view of an exemplary computing device 500 that can be used to implement the systems and methods described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functions serve only as examples and are not intended to limit the implementation of the inventions described and / or claimed in this publication.

[0038] The computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and high-speed expansion ports 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and can be mounted on a common mainboard or otherwise, as required.The processor 510 can process instructions for execution within the computing device 500, including instructions stored in the memory 520 or on the storage device 530 to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 connected to the high-speed interface 540. Other implementations can utilize multiple processors and / or multiple buses, along with multiple memories and memory types, as needed. Furthermore, multiple computing devices 500 can be connected, with each device providing parts of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0039] The memory 520 stores information non-transiently within the computing device 500. The memory 520 can be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 can be physical devices used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program status information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs).Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and floppy disks or tapes.

[0040] The storage device 530 is capable of providing mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a floppy disk device, a hard disk device, an optical disk device or tape device, flash memory or other similar semiconductor storage device, or an array of devices, including devices in a memory area network or other configurations. In additional implementations, a computer program product is physically executed on an information carrier. The computer program product contains instructions that, when executed, perform one or more procedures, such as those described above.The information carrier is a computer- or machine-readable medium, such as the memory 520, the storage device 530, or the memory on the processor 510.

[0041] The high-speed controller 540 manages bandwidth-intensive operations for the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. Such task assignment is merely an example. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and the high-speed expansion ports 550, which can accommodate various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the memory device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which provides various communication connections (e.g.,USB, Bluetooth, Ethernet, wireless Ethernet) may be included, and may be connected to one or more input / output devices, such as a keyboard, pointing device, scanner or network device, such as a switch or router, e.g. via a network adapter.

[0042] The computing device 500 can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server 500a or multiple servers 500a together, as a laptop computer 500b, or as part of a rack server system 500c.

[0043] Various implementations of the systems and techniques described here can be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include one or more computer programs that are executable and / or interpretable on a programmable system with at least one programmable processor, which may be a specialized or general-purpose processor and is coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to these devices.

[0044] These computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a higher-level procedural and / or object-oriented programming language and / or in assembly / machine language. In this context, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, any device, and / or any apparatus (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0045] The processes and logic flows described in this specification can be performed by one or more programmable processors, also known as data processing hardware, which execute one or more computer programs to perform functions by processing input data and producing outputs. The processes and logic flows can also be performed by specialized logic circuits, such as FPGAs (field-programmable gate arrays) or ASICs (application-specific integrated circuits). Processors suitable for executing a computer program can include, for example, general-purpose and specialized microprocessors, any one or more processors, or any type of digital computer. Generally, a processor receives instructions and data from read-only memory, random-access memory, or both.The essential elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical media, or is functionally coupled to one or more of these devices to receive data from them, transmit data to them, or both. However, a computer does not necessarily have to include such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.The processor and memory can be supplemented or integrated with special logic circuits.

[0046] To provide interaction with a user, one or more aspects of the disclosure may be implemented on a computer that has a display device, such as a monitor with a CRT (cathode ray tube), LCD (liquid crystal display), or touchscreen for displaying information to the user, and optionally a keyboard and pointing device (such as a mouse or trackball) with which the user can provide input to the computer. Furthermore, other types of devices may be used to interact with a user; for example, feedback provided to the user may be any form of sensory feedback (such as visual, auditory, or tactile feedback); and input from the user may be received in any form, including acoustic, verbal, or tactile input.Furthermore, a computer can interact with a user by sending and receiving documents to and from a device used by the user. For example, in response to requests received from the web browser, it can send web pages to a web browser on a user's client device.

[0047] Several implementations have been described. However, it is understood that various modifications can be made without altering the scope of protection and the nature of the disclosure. Accordingly, other implementations fall within the scope of protection of the following claims.

Claims

[1] Computer-implemented procedure that is executed on data processing hardware and causes the data processing hardware to perform operations that include: Receiving a prompt directed to an assisting large language model (LLM); Generating a sequence of output tokens based on the prompt using the assisting LLM, where the sequence of output tokens includes the following: a sequence of text tokens comprising one or more valid text tokens and one or more invalid text tokens; and one or more revision tokens, each revision token specifying a corresponding N number of erroneous text tokens generated before the respective revision token, and corresponding replacement text tokens generated after the respective revision token to replace the corresponding N number of erroneous text tokens; and Generate a revised sequence of output tokens for the prompt, based on the sequence of output tokens. [2] Computer-implemented method according to claim 1, wherein generating the sequence of output tokens comprises autoregressive generation of each respective output token in the sequence of output tokens. [3] Computer-implemented method according to claim 2, wherein the generation of each respective output token in the sequence of output tokens comprises autoregressive conditioning of the respective output token to one or more output tokens in the sequence of output tokens that were generated prior to the respective output token. [4] Computer-implemented method according to any one of claims 1 to 3, wherein: The prompt comprises a sequence of acoustic images that correspond to an utterance spoken by a user; and The sequence of output tokens and the revised sequence of output tokens each include a corresponding speech recognition result for the utterance. [5] Computer-implemented method according to any one of claims 1 to 3, wherein: The prompt comprises a textual representation in a first language; a sequence of acoustic images corresponding to an utterance spoken by a user; and The sequence of output tokens and the revised sequence of output tokens each comprise a corresponding translated textual representation in a second language that differs from the first language for the textual representation. [6] Computer-implemented method according to any one of claims 1 to 5, wherein generating the revised sequence of output tokens for each respective revision token of the one or more revision tokens comprises: Identifying the corresponding N number of erroneous text tokens, as specified by the respective revision token; and Replacing the corresponding N number of faulty text tokens with the corresponding replacement text tokens specified by the respective revision token. [7] Computer-implemented method according to any one of claims 1 to 6, wherein the operations further comprise determining, for each respective faulty text token, using the assisting LLM, that the respective faulty text token is inaccurate after the respective faulty text token has been generated based on one or more text tokens in the sequence of text tokens generated after the respective faulty text token. [8] Computer-implemented method according to any one of claims 1 to 7, wherein the corresponding N number of defective text tokens is located immediately before the respective revision token in the sequence of text tokens. [9] Computer-implemented method according to any one of claims 1 to 8, wherein: Each revision token further specifies an offset number of text tokens between the respective revision token and the corresponding N number of erroneous text tokens; and The corresponding N number of faulty text tokens is the offset number of text tokens away from the revision token. [10] Computer-implemented method according to any one of claims 1 to 9, wherein the assisting LLM comprises an encoder-decoder architecture. [11] Computer-implemented method according to any one of claims 1 to 10, wherein the assisting LLM comprises a decoder-only architecture. [12] Computer-implemented method according to any one of claims 1 to 11, wherein the operations further comprise: Obtaining a large number of training samples, each training sample comprising audio data that characterizes a spoken utterance and is paired with a corresponding protocol; Expanding the number of training scans for each corresponding protocol of each respective training scan by: Inserting one or more incorrect terms into the relevant protocol; and Inserting a training revision token that identifies a number of the one or more erroneous terms inserted into the corresponding protocol, and training replacement text tokens to replace the one or more erroneous terms; and Training the assisting LLM based on the expanded variety of training scans. [13] Computer-implemented method according to any one of claims 1 to 11, wherein the operations further comprise: Obtaining a large number of training samples, each training sample comprising audio data that characterizes a spoken utterance and is paired with a corresponding protocol; Expanding the number of training scans for each corresponding protocol of each respective training scan by: Requesting an auxiliary LLM to generate a continuation output based on the appropriate protocol; Appending the continuation issue to the corresponding protocol; and Inserting a training revision token between the relevant protocol and the continuation output, wherein the training revision token identifies a number of terms in the continuation output that are appended to the relevant protocol, and training replacement text tokens to replace the one or more erroneous text tokens; and Training the assisting LLM based on the expanded variety of training scans. [14] System, encompassing: Data processing hardware; and Storage hardware in communication with data processing hardware, wherein the storage hardware stores instructions which, when executed on the data processing hardware, cause the data processing hardware to perform operations that include the following: Receiving a prompt addressed to an assisting large language model (LLM); Generating a sequence of output tokens based on the prompt using the assisting LLM, where the sequence of output tokens includes the following: a sequence of text tokens comprising one or more valid text tokens and one or more invalid text tokens; and one or more revision tokens, each revision token specifying a corresponding N number of erroneous text tokens generated before the respective revision token, and corresponding replacement text tokens generated after the respective revision token to replace the corresponding N number of erroneous text tokens; and Generate a revised sequence of output tokens for the prompt, based on the sequence of output tokens. [15] System according to claim 14, wherein generating the sequence of output tokens comprises autoregressive generation of each respective output token in the sequence of output tokens. [16] System according to claim 15, wherein the generation of each respective output token in the sequence of output tokens comprises autoregressive conditioning of the respective output token to one or more output tokens in the sequence of output tokens that were generated prior to the respective output token. [17] System according to any one of claims 14 to 16, wherein: The prompt comprises a sequence of acoustic images that correspond to an utterance spoken by a user; and The sequence of output tokens and the revised sequence of output tokens each include a corresponding speech recognition result for the utterance. [18] System according to any one of claims 14 to 16, wherein: The prompt comprises a textual representation in a first language; a sequence of acoustic images corresponding to an utterance spoken by a user; and The sequence of output tokens and the revised sequence of output tokens each comprise a corresponding translated textual representation in a second language that differs from the first language for the textual representation. [19] System according to any one of claims 14 to 18, wherein generating the revised sequence of output tokens for each respective revision token of the one or more revision tokens comprises: Identifying the corresponding N number of erroneous text tokens, as specified by the respective revision token; and Replacing the corresponding N number of faulty text tokens with the corresponding replacement text tokens specified by the respective revision token. [20] System according to any one of claims 14 to 19, wherein the operations further comprise determining, for each respective faulty text token, using the assisting LLM, that the respective faulty text token is inaccurate after the respective faulty text token has been generated based on one or more text tokens in the sequence of text tokens generated after the respective faulty text token. [21] System according to one of claims 14 to 20, wherein the corresponding N number of faulty text tokens is located immediately before the respective revision token in the sequence of text tokens. [22] System according to any one of claims 14 to 21, wherein: Each revision token further specifies an offset number of text tokens between the respective revision token and the corresponding N number of erroneous text tokens; and The corresponding N number of faulty text tokens is the offset number of text tokens away from the revision token. [23] System according to any one of claims 14 to 22, wherein the assisting LLM comprises an encoder-decoder architecture. [24] System according to any one of claims 14 to 23, wherein the assisting LLM comprises a decoder-only architecture. [25] System according to any one of claims 14 to 24, wherein the processes further comprise: Obtaining a large number of training samples, each training sample comprising audio data that characterizes a spoken utterance and is paired with a corresponding protocol; Expanding the number of training scans for each corresponding protocol of each respective training scan by: Inserting one or more incorrect terms into the relevant protocol; and Inserting a training revision token that identifies a number of the one or more erroneous terms inserted into the corresponding protocol, and training replacement text tokens to replace the one or more erroneous terms; and Training the assisting LLM based on the expanded variety of training scans. [26] System according to any one of claims 14 to 24, wherein the processes further comprise: Obtaining a large number of training samples, each training sample comprising audio data that characterizes a spoken utterance and is paired with a corresponding protocol; Expanding the number of training scans for each corresponding protocol of each respective training scan by: Requesting an auxiliary LLM to generate a continuation output based on the appropriate protocol; Appending the continuation issue to the corresponding protocol; and Inserting a training revision token between the relevant protocol and the continuation output, wherein the training revision token identifies a number of terms in the continuation output that are appended to the relevant protocol, and training replacement text tokens to replace the one or more erroneous text tokens; and Training the assisting LLM based on the expanded variety of training scans. [27] Computer program comprising instructions which, when executed on data processing hardware, cause the data processing hardware to perform the method according to any one of claims 1 to 13.

Images