Voice dictation with a large language model for audio

Abstract

A procedure involves receiving audio data that identifies an utterance spoken by a user. The procedure also involves processing the audio data to generate a transcription of the utterance using a multimodal large language model (LLM). The transcription comprises a sequence of expressions. The procedure also involves processing the audio data and the transcription in parallel using the multimodal LLM to identify one or more revision expressions within the sequence. The one or more revision expressions specify a revision action to be performed on at least one other expression within the sequence. The procedure also involves modifying the transcription based on the one or more revision expressions.

Description

TECHNICAL AREA

[0001] This revelation refers to speech dictation using an audio large-scale language model. GENERAL STATE OF THE ART

[0002] Automatic speech recognition (ASR) aims to convert speech into text. End-to-end speech recognition models integrate multiple components into a single model, thereby improving the performance (e.g., word error rate (WER) and latency) of speech-to-text transcription. Some systems contain multiple cascaded models capable of performing ASR in several different languages. Recently, speech recognition models have benefited from being improved through training on both audio and text data. However, using audio and text data presents certain challenges due to the different modalities of audio and text. Many current approaches use multiple models to process audio and text, which is expensive and difficult to maintain, as each of these models uses different data sources, training processes, and evaluation metrics. DESCRIPTION OF THE DRAWINGS Fig. 1A and Fig. Figure 1B shows schematic views of an exemplary speech recognition system. Fig. Figure 2 is a schematic view of a sample transcription with an inserted revision token. Fig. Figure 3 is a schematic view of an exemplary training process for training a multimodal large language model. Fig. Figure 4 is a flowchart of an exemplary sequence of processes for a computer-implemented procedure for conducting a speech dictation using a large language model. Fig. Figure 5 is a schematic view of an exemplary computing device that can be used to implement the systems and procedures described here.

[0003] Identical reference symbols in different drawings indicate identical elements. SUMMARY

[0004] One aspect of the revelation provides a computer-implemented procedure, executed on data processing hardware, that causes the data processing hardware to perform operations for conducting speech dictation using a large language model. These operations involve receiving audio data that identifies an utterance spoken by a user. They also involve processing the audio data to generate a transcription of the utterance using a multimodal large language model (LLM). The transcription includes a sequence of expressions. Furthermore, the operations involve processing the audio data and the transcription in parallel to identify one or more revision expressions within the sequence of expressions, again using the multimodal LLM.The one or more revision expressions specify a revision action to be performed on at least one other expression in the sequence of expressions. These operations also include modifying the transcription based on the one or more revision expressions.

[0005] Implementations of the disclosure may include one or more of the following optional features. In some implementations, the operations further involve determining the user's corresponding intention when speaking the respective expression, based on the parallel processing of the audio data and the transcription. The identification of one or more revision expressions within the sequence of expressions is based on the corresponding intention determined for each expression within the sequence.In some examples, the parallel processing of the audio data and the transcript for each expression in the sequence of expressions involves determining the corresponding language features based on the processing of the audio data, determining the corresponding linguistic context based on the processing of the transcript, and correlating the corresponding language features of each expression with the corresponding linguistic context of each expression. It may happen that the corresponding language features of each expression are not represented in the transcript, and the corresponding linguistic context of each expression is not represented in the audio data. In these examples, the corresponding language features may include at least one of the following: pitch information, tone information, or prosody information.

[0006] In some implementations, the operations further involve inserting a revision token into the sequence of expressions based on one or more revision expressions. The revision token specifies a corresponding N number of expressions within the at least one other expression and corresponding replacement expressions to replace those N number of expressions. In these implementations, the transcription modification is also based on the revision token inserted into the sequence of expressions. The operations may further involve retrieving context data associated with the user who spoke the utterance and conditioning the multimodal LLM to that context data.

[0007] In some examples, the processes further include determining a training prompt for a multimodal helper LLM, generating a multitude of training samples based on the training prompt using the multimodal helper LLM, and training the multimodal LLM using the multitude of training samples. The training prompt includes a transcription editing task and a multitude of training samples. Each training sample includes a corresponding training transcript paired with a corresponding modified training transcript. The revision action can include at least one replacement action, one deletion action, or one spelling action.

[0008] Another aspect of the revelation provides a system for operating an influencer rating model. The system includes data processing hardware and storage hardware that communicate with the data processing hardware. The storage hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. These operations include receiving audio data that identifies an utterance spoken by a user. The operations also include processing the audio data to generate a transcription of the utterance using a multimodal large language model (LLM). The transcription includes a sequence of expressions. The operations also include parallel processing of the audio data and the transcription to identify one or more revision expressions within the sequence of expressions, using the multimodal LLM.The one or more revision expressions specify a revision action to be performed on at least one other expression in the sequence of expressions. These operations also include modifying the transcription based on the one or more revision expressions.

[0009] Implementations of the disclosure may include one or more of the following optional features. In some implementations, the operations further involve determining the user's corresponding intention when speaking the respective expression, based on the parallel processing of the audio data and the transcription. The identification of one or more revision expressions within the sequence of expressions is based on the corresponding intention determined for each expression within the sequence.In some examples, the parallel processing of the audio data and the transcript for each expression in the sequence of expressions involves determining the corresponding language features based on the processing of the audio data, determining the corresponding linguistic context based on the processing of the transcript, and correlating the corresponding language features of each expression with the corresponding linguistic context of each expression. It may happen that the corresponding language features of each expression are not represented in the transcript, and the corresponding linguistic context of each expression is not represented in the audio data. In these examples, the corresponding language features may include at least one of the following: pitch information, tone information, or prosody information.

[0010] In some implementations, the operations further involve inserting a revision token into the sequence of expressions based on one or more revision expressions. The revision token specifies a corresponding N number of expressions within the at least one other expression, and corresponding replacement expressions to replace these N number of expressions within the at least one other expression. In these implementations, the transcription modification is also based on the revision token inserted into the sequence of expressions. The operations may further involve retrieving context data associated with the user who spoke the utterance and conditioning the multimodal LLM to this context data.

[0011] In some examples, the processes further include determining a training prompt for a multimodal helper LLM, generating a multitude of training samples based on the training prompt using the multimodal helper LLM, and training the multimodal LLM using the multitude of training samples. The training prompt includes a transcription editing task and a multitude of training samples. Each training sample includes a corresponding training transcript paired with a corresponding modified training transcript. The revision action can include at least one replacement action, one deletion action, or one spelling action.

[0012] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Further aspects, features, and advantages will become apparent from the description, the drawings, and the claims. DETAILED DESCRIPTION

[0013] Automatic speech recognition (ASR) is the process of converting spoken utterances into text. ASR can be used to recognize spoken commands executed by a digital assistant or to recognize spoken requests answered by the digital assistant. It can also be used for dictation. For example, a user might speak a long utterance (e.g., minutes or hours of continuous speech) that is transcribed by the ASR model. However, when users speak such long utterances, they often want to correct one or more phrases they have spoken earlier.For example, a user might want to speak a sequence of expressions that corresponds to a list of entries, and then rearrange one or more entries in the list, add another entry or remove one or more entries from the list, specify a particular text formatting (e.g., punctuation, capitalization), or have a list of entries and then rearrange, add, or remove one or more entries from the list.

[0014] The implementations contained herein relate to procedures and systems for conducting speech dictation using a large language model. The procedure specifically involves receiving audio data that identifies an utterance spoken by a user. It also involves processing the audio data to generate a transcription of the utterance using a multimodal large language model (LLM). The transcription comprises a sequence of expressions. Furthermore, the procedure involves processing the audio data and the transcription in parallel using the multimodal LLM to identify one or more revision expressions within the sequence. The one or more revision expressions specify a revision action to be performed on at least one other expression within the sequence.The procedure also involves modifying the transcription based on one or more revision expressions. As will be shown, multimodal LLM offers the advantage that the audio data and the text data (i.e., the transcription) can be processed in parallel, compared to using different audio and text models for processing audio and text, respectively.

[0015] Fig. 1A and Fig. Figure 1B shows an example system 100 that includes a speech recognition system 105. In general, the user 10 speaks an utterance 106 via a user device 110, which is directed to a multimodal LLM 150. The speech recognition system 105 includes the user device 110, a remote computing system 120, and a network 130. The user device 110 includes data processing hardware 113 and storage hardware 114. The user device 110 can include or communicate with an audio recording device 115 (e.g., an array of one or more microphones) to convert utterances 106 or natural language queries spoken by the user 10 into corresponding audio data (e.g., a sequence of acoustic still images) 102.In addition to or instead of spoken input, the user 10 can enter a textual representation of the query in natural language via a user interface running on the user device 110.

[0016] The user device 110 can be any computing device capable of communicating with the remote computing system 120 over the network 130. The user device 110 includes, but is not limited to, desktop computing devices and mobile computing devices such as laptops, tablets, smartphones, smart speakers / displays, digital assistive devices, smart devices, Internet of Things (IoT) devices, infotainment systems, vehicle infotainment systems, and wearable computing devices (e.g., headsets, smart glasses, and / or watches). The remote computing system 120 can be a distributed system (e.g., a cloud computing environment) with scalable, elastic resources. These resources include computing resources 123 (e.g., data processing hardware) and / or storage resources 124 (e.g., storage hardware). Additionally or alternatively, the remote computing system 120 can be a centralized system.The network 130 can be wired, wireless or a combination thereof and can include private networks and / or public networks such as the Internet.

[0017] The multimodal LLM 150 is configured to process the audio data 102 to generate a transcription 151 of the utterance 106 spoken by user 10. Alternatively, a separate ASR model can process the audio data 102 to generate the transcription 151, with the multimodal LLM 150 then processing the audio data 102 and the transcription 151 output by the ASR model in parallel to identify one or more revision expressions 152, 152R in the sequence of expressions 152. The transcription 151 contains a sequence of expressions 152, each representing a corresponding word or expression in the utterance 106 spoken by user 10. In some scenarios, user 10 may want to edit or revise one or more words in the utterance 106 previously spoken by user 10. Simply put, the user can revise words they have previously spoken (e.g., edit, delete, etc.).), so that the output of the multimodal LLM 150 reflects the revision. For this purpose, the multimodal LLM 150 is configured to process the audio data 102 and the transcription 151 in parallel in order to identify one or more revision expressions 152, 152R in the sequence of expressions 152. The one or more revision expressions 152R can specify a revision action to be performed on at least one other expression 152 in the sequence of expressions 152. The revision action can include any action that edits or modifies the transcription 151 or the at least one other expression 152 in any way.The revision action can include, for example, adding one or more expressions 152 between expressions already spoken by the user 10, deleting a user-spoken expression, and / or rearranging one or more user-spoken expressions. In some examples, the revision action may include a text formatting action, such as correcting a speech error, adding punctuation to the text, summarizing the text, etc. The revision action can include at least one replacement action, one deletion action, or one spelling action (e.g., rewriting).

[0018] In particular, some utterances contain revision expressions 152R, while other utterances do not. Thus, the multimodal LLM can determine whether the spoken utterance contains such revision expressions 152R. The revision expressions 152R are not predefined expressions that trigger the revision action. For example, if the expression "change" is uttered in one context, it may mean a change to the transcription 151, while the same expression "change" in another context may be a word that the user 10 wants transcribed. Thus, the multimodal LLM determines whether revision expressions 152R are present within each transcription 151 by determining the context of each expression 152 within the transcription 151.

[0019] User 10 may determine that one or more previously spoken words need to be changed or revised in some way. User 10 may then speak one or more other words that describe and / or explain the revision of transcription 151. For example, User 10 may speak utterance 106, "I go home at 4 p.m. Sorry, not 4 p.m., I mean 5 p.m.," for which the multimodal LLM 150 generates transcription 151, which corresponds to utterance 106. In this example, "4 p.m." represents expression 152, which User 10 wishes to change or modify, so User 10 speaks the revision expressions 152R: "Sorry, not 4 p.m., I mean 5 p.m." Here, the revision expressions 152R specify the revision action of replacing expression 152 "4 p.m." with "5 p.m." In this example, the 152R expressions do not explicitly state which revision action (e.g.,"Replacement") is to be performed. Therefore, the multimodal LLM 150 can infer the revision action from the transcription 151 based on the context of the revision expressions 152R in the transcription 151 by performing a semantic interpretation of the transcription 151. On the other hand, the revision action to be performed can be explicitly stated in the revision expressions 152R. For example, user 10 can explicitly state their intention to replace a particular expression with another expression.

[0020] Not all utterances 106 spoken by a user 10 include the revision expressions 152R. In some implementations, the multimodal LLM 150 determines, for each expression 152 in the sequence of expressions 152, a corresponding intention 154 of the user 10 when speaking that expression 152, based on the parallel processing of the audio data 102 and the transcription 151. The intention 154 indicates whether, at the time of speaking the expression 152, the user 10 intended to transcribe the expression 152 or to specify a revision action to revise another of the expressions 152. Thus, the multimodal LLM 150 identifies the revision expression(s) 152R based on the corresponding intention 154 determined for each expression 152 in the sequence of expressions 152.To continue with the example above: Even if User 10 accidentally said the expression "4 p.m.", the multimodal LLM 150 would determine that User 10 intended to transcribe the expression "4 p.m." at the time they said it. That is, in this example, User 10 only realized that the expression "4 p.m." was an error after they had spoken it. Thus, User 10 may initially intend to transcribe a particular expression and then later decide that the expression 152 needs to be revised in some way. Furthermore, in some examples, User 10 does not explicitly specify a revision action but wants certain actions to be performed on the transcription 151.For example, user 10 might speak an utterance 106 and want the transcription to include punctuation, specific text formatting, a summary of utterance 160, etc., without actually speaking such a revision. Similarly, user 10 might pause after speaking an expression 152 and add a comma after it without explicitly mentioning it, or want to add an exclamation mark after an expression.

[0021] Fig. Figure 1A illustrates a first example system 100, 100a, in which the user 10 speaks the utterance 106 “Buy some tomatoes and bananas. Trade tomatoes for potatoes.” and the user’s device 110 converts the utterance 106 into corresponding audio data 102. Here, the multimodal LLM 150 processes the audio data 102 to generate the transcription 151 of “Buy some tomatoes and bananas. Trade tomatoes for potatoes.” In particular, in this example, the user 10 does not intend for the multimodal LLM 150 to output a final transcription that includes tomatoes. Therefore, the multimodal LLM 150 processes the transcription 151 and the audio data 102 in parallel to identify the revision expression(s) 152R.To continue with the example shown, the multimodal LLM 150 identifies the expressions 152R of “Swap tomatoes for potatoes”, which specify the revision action of replacing or changing tomatoes with potatoes.

[0022] In some examples, the assisting LLM 150 processes the audio data 102 and the transcription 151 in parallel by determining, for each expression 152 in the sequence of expressions 152, the corresponding language features 156 of the respective expression 122, determining a corresponding linguistic context 158 ​​of the respective expression 122, and correlating the corresponding language features 156 of the respective expression 122 with the corresponding linguistic context 158 ​​of the respective expression 122. This parallel processing of the audio data 102 and the transcription 151 contrasts with sequential processing, in which the audio data 102 is first processed to generate the transcription 151 and determine the language features 156, and then the transcription 151 is processed to determine the linguistic context 158.Simply put, sequential processing first processes the audio data 102 and then the transcription 151, without correlating the language features 156 (e.g. determined by the processing of the audio data 102) and the linguistic context 158 ​​(e.g. determined by the processing of the transcription 151).

[0023] Advantageously, parallel processing enables such a correlation between the linguistic context 158 ​​and the language features 156, allowing the multimodal LLM 150 to better recognize the user's intention 10 when speaking each expression, which would otherwise not be apparent from the audio data 102 or the transcription 151 alone. The corresponding language features 156 of the respective expression 152 are not represented in the transcription 151, and the corresponding linguistic context 158 ​​of the respective expression is not represented in the audio data 102. By correlating the linguistic context 158 ​​and the language features 158 together (e.g., by parallel processing of the transcription 151 and the audio data 102), the multimodal LLM 150 can correlate the language features 156 of each expression with the corresponding linguistic context 158 ​​of the same expression.For example, speaking an expression 152 with a rising or falling pitch, paired with the linguistic context of expression 152 within the transcription 151, can inform the multimodal LLM 150 that expression 152 should be transcribed with a punctuation mark (e.g., comma, exclamation mark, etc.). In this example, the linguistic context 158 ​​of the expression alone may not be sufficient to inform the multimodal LLM 150 of the user's intention 10 (e.g., the intention to use or not use punctuation), whereas the correlation of the language features 156 (e.g., the rising pitch of the user's voice while speaking the expression) with the linguistic context 158 ​​sufficiently informs the multimodal LLM 150 that the expression should be transcribed with punctuation.

[0024] The language features 156 generally represent how the utterance 106 was spoken. The language features 156 may, for example, include at least one of the pitch, tone, and / or prosody information of each expression 152 in the transcription 151. On the other hand, the linguistic context 158 ​​provides semantic meaning for each expression 152 within the transcriptions 151. In other words, the linguistic context 158 ​​provides the semantic meaning for each expression 152 in the transcription 151 in relation to one or more other expressions 152 in the transcription 151. In some scenarios, processing either the language features 156 or the linguistic context 158 ​​alone is not sufficient to identify the revision expressions 152R.If, for example, utterance 106 contains certain speech errors, such as long pauses, word repetitions, or stuttering, the linguistic context 158 ​​of transcription 151 alone may not be sufficient to determine whether some expressions 152 were intended for transcription 151. However, parallel processing of the language features 156 and the linguistic context 158 ​​can provide such insights. For example, repeated words in transcription 151, paired with a lower pitch, can tell the multimodal LLM 150 that the repeated words were not intended for transcription. In another example, speaking an expression with a rising pitch can tell the multimodal LLM 150 that an exclamation mark should be added after that expression.

[0025] Fig. Figure 1B illustrates a second system 100, 100b, in which user 10 speaks the utterance 106, "I sent you a shopping list. Add the apples to the list." Here, the multimodal LLM 150 processes the audio data 102 to create the transcription 151 of "I sent you a shopping list. Add the apples to the list." In this example, user 10 intends to transcribe the entire utterance 106 because none of the expressions 152 match the revision expressions 152R. The multimodal LLM 150 processes the transcription 151 and the audio data 102 in parallel to identify whether revision expressions 152R are present in the transcription 151. In the example shown, the multimodal LLM 150 does not identify any revision expressions 152R, so the output layer 160 outputs the same transcription 151 that was generated by the multimodal LLM 150.In this particular example, the multimodal LLM 150 has determined that the expression "add" in the context of this specific utterance 106 is not a revision expression 152R. That is, instead of simply assuming that the expression "add" corresponds to a certain action and performing such an action on the transcription 151, the multimodal LLM 150 processes the audio data 102 and the transcription 151 in parallel to determine that the expression "add" should be transcribed in this scenario, rather than specifying a revision action.

[0026] Referring again to Fig. 1A and Fig. In some implementations, the multimodal LLM 150 obtains context data 104 associated with the user 10 who spoke the utterance 106, thus making the multimodal LLM 150 dependent on this context data 104. The context data 104 can specify at least one of the following: a user profile (e.g., contact name), device information (e.g., location, text displayed on screen, etc.), previously spoken utterances 106, and / or operating system information (e.g., date, time, etc.). By being conditioned on the context data 104, the conditioned multimodal LLM 150 can generate transcriptions 151 that are more specifically tailored to the individual user 10. For example, the multimodal LLM 150 can obtain context data 104, specifying the contact names of a user 106 who uttered the utterance 106 “Call Christyne”.Thus, the multimodal LLM 150 can first generate the transcription 151 of “Ruf Christine” and then, based on the context data 104, generate the modified transcription 151, 151M of “Ruf Christyne”.

[0027] In some implementations, the multimodal LLM 150 operates in a streaming mode, so that each expression 152 from the transcription 151 is displayed on a screen of the user device 110 as soon as it is recognized. This allows the user 10 to see the transcription 151 of each expression 151 in real time while speaking the utterance 106. Furthermore, the user 10 can thus detect if any of the expressions they previously spoke were incorrectly recognized while speaking the utterance 106. For example, the user 10 might speak the utterance 106 "Remind me to call Christyne, change Christine to CHRI STYNE". Here, the multimodal LLM 150 can generate and display the transcription 151 of "Remind me to call Christine" before the user 10 speaks the expression "tomorrow".Thus, user 10 can notice the incorrect spelling of "Christyne" and instruct the multimodal LLM 150 to correct the spelling by saying "change Christine to CHRISTYNE", with the multimodal LLM 150 identifying the revision expressions 152R of "change Christine to CHRISTYNE" and changing transcription 151 to the modified transcription 151M of "remind me to call Christyne" before user 10 says the expression "tomorrow".

[0028] Fig. Figure 2 shows a schematic view 200 of the multimodal LLM 150, which inserts one or more revision tokens 202 into the sequence of expressions 152 of the transcription 151 based on the one or more revision expressions 152R. Each revision token 202 specifies a corresponding N number of expressions 152 in the at least one other expression 152 and corresponding replacement expressions (if any) to replace the corresponding N number of expressions 152 in the at least one other expression. The revision token 202 tells the output layer 160 which expressions 152 need to be changed or revised and can also specify the replacement expressions that are to replace these expressions 152. In the Fig. In the example shown, the multimodal LLM 150 receives audio data 102 for an utterance corresponding to the sentence "How are you, um? [Revise_1] today?". The multimodal LLM 150 can then process the audio data 102 to generate a transcription 151 containing a sequence of expressions 152 and a revision token 202. In the example shown, the transcription 151 contains "How are you, um? [Revise_1] today?" In this example, "[Revise_1]" specifies the revision token 202, where the revision action indicates deleting the expression "um?". More precisely, "1" from the revision token 202 specifies a number of expressions for revision, and since the revision action is deletion, there are no replacement expressions. In other words, the revision token 202 in this example indicates that the earlier expression 152 of "um?" should be deleted.In other examples, the revision token 202 can also specify one or more replacement expressions that replace the expression "um?" instead of simply deleting it. Consequently, output layer 160 can process transcription 151, which includes revision token 202 and the replacement expressions (if any), to produce the modified transcription 151M. In the example shown, output layer 160 would produce the modified transcription 151M of "How are you today?". The modified transcription 151M contains a sequence of modified expressions 152 and 152M. That is, the modified expressions 152M contain the same expressions from the sequence of expressions 152 with modified expressions 152M specified by the revision action.

[0029] Fig. Figure 3 illustrates an exemplary training process 300 for training the multimodal LLM 150. The training process 300 includes an input prompt generator 310, a multimodal auxiliary LLM 320, and a loss module 330. The input prompt generator 310 receives a request 302 and determines a training input prompt 312 for the multimodal auxiliary LLM 320. The training input prompt 312 includes a transcription editing task and a variety of training samples. For example, the transcription editing task might request the multimodal auxiliary LLM 320 to generate training samples that include one or more examples of speech errors, examples of speech addition, examples of speech deletion, and / or examples of spelling errors. The speech deletion examples might include, for example,The training sample includes speech spoken by User 10, and then User 10 decides to delete one or more of the previous expressions. Each training sample includes a corresponding training transcription paired with a corresponding modified transcription. In some examples, the training prompt 312 involves generating paired speech and text samples that correspond to a user's wish to delete a previously spoken expression from a transcription. Here are a few examples for reference: [sample_1], [sample_2], and [sample_3]. The multimodal auxiliary LLM 320 differs from the multimodal LLM 150 and generates a variety of training samples 322 based on the training prompt 312. Each training sample 322 includes a training transcription 324 paired with a corresponding synthetic training audio 326.The auxiliary LLM 150 then receives each training example 322 and generates a corresponding transcription 151 based on the training example 322. More precisely, the auxiliary LLM 150 generates the corresponding transcription 151 based on processing the trained synthetic audio data 326. In some examples, the auxiliary LLM 150 generates the transcription 151 based on each training example 322. In other examples, the auxiliary LLM 150 generates the modified transcription 151M based on each training example 322.

[0030] The loss module 330 receives each transcription 151 generated by the auxiliary LLM 150 and determines a corresponding loss 332 based on the transcription 151 and the training transcription 324. That is, the loss module 330 compares the transcription 151 generated by the auxiliary LLM 320 with the corresponding training transcription 324 to determine the loss 332 for each training example 322. The training process 300 trains the multimodal LLM 150 based on each corresponding loss 332 determined for each training example 322. Advantageously, the training process 300 can utilize the multimodal capability of the auxiliary LLM 320 to generate various training examples 322 that are used to train the multimodal LLM 150.

[0031] Fig. Figure 4 is a flowchart of an exemplary sequence of operations for a computer-implemented procedure 400 for conducting speech dictation using a large language model. The procedure 400 can be run on the data processing hardware 510 ( Fig. 5) are executed based on instructions that are based on the memory hardware 520 ( Fig. 5) are stored. In some examples, the data processing hardware 510 includes the data processing hardware 113 of the user device 110, and the storage hardware 520 includes the storage hardware 114 of the user device 110. In other examples, the data processing hardware 510 includes the data processing hardware 123 of the remote computing system 120, and the storage hardware 420 includes the storage hardware 124 of the remote computing system 120.

[0032] In process 402, procedure 400 involves receiving audio data 102 that identifies an utterance 106 spoken by a user 10. In process 404, procedure 400 involves processing the audio data 102 to generate a transcription 151 of the utterance 106 using a multimodal large language model (LLM) 150. The transcription 151 contains a sequence of expressions 152. In process 406, procedure 400 involves processing the audio data 102 and the transcription 151 in parallel using the multimodal LLM 150 to identify one or more revision expressions 152R in the sequence of expressions 151. The one or more revision expressions 152R specify a revision action to be performed on at least one other expression 152 in the sequence of expressions 152.In process 308, procedure 300 involves modifying the transcription 151 based on one or more revision expressions 152R.

[0033] Fig. Figure 5 is a schematic view of an exemplary computing device 500 that can be used to implement the systems and methods described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functions serve only as examples and are not intended to limit the implementation of the inventions described and / or claimed in this publication.

[0034] The computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and high-speed expansion ports 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and can be mounted on a common mainboard or otherwise, as required.The processor 510 can process instructions for execution within the computing device 500, including instructions stored in the memory 520 or on the storage device 530 to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 connected to the high-speed interface 540. Other implementations can utilize multiple processors and / or multiple buses, along with multiple memories and memory types, as needed. Furthermore, multiple computing devices 500 can be connected, with each device providing parts of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0035] The memory 520 stores information non-transiently within the computing device 500. The memory 520 can be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 can be physical devices used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program status information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs).Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and floppy disks or tapes.

[0036] The storage device 530 is capable of providing mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a floppy disk device, a hard disk device, an optical disk device or tape device, flash memory or other similar semiconductor storage device, or an array of devices, including devices in a memory area network or other configurations. In additional implementations, a computer program product is physically executed on an information carrier. The computer program product contains instructions that, when executed, perform one or more procedures, such as those described above.The information carrier is a computer- or machine-readable medium, such as the memory 520, the storage device 530, or the memory on the processor 510.

[0037] The high-speed controller 540 manages bandwidth-intensive operations for the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. Such task assignment is merely an example. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and the high-speed expansion ports 550, which can accommodate various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the memory device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which supports various communication ports (e.g.,USB, Bluetooth, Ethernet, wireless Ethernet) may be included, and may be connected to one or more input / output devices, such as a keyboard, pointing device, scanner or network device, such as a switch or router, e.g. via a network adapter.

[0038] The computing device 500 can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server 500a or multiple servers 500a together, as a laptop computer 500b, or as part of a rack server system 500c.

[0039] Various implementations of the systems and techniques described here can be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include one or more computer programs that are executable and / or interpretable on a programmable system with at least one programmable processor, which may be a specialized or general-purpose processor and is coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to these devices.

[0040] These computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a higher-level procedural and / or object-oriented programming language and / or in assembly / machine language. In this context, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, any device, and / or any apparatus (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0041] The processes and logic flows described in this specification can be performed by one or more programmable processors, also known as data processing hardware, which execute one or more computer programs to perform functions by processing input data and producing outputs. The processes and logic flows can also be performed by specialized logic circuits, such as FPGAs (field-programmable gate arrays) or ASICs (application-specific integrated circuits). Processors suitable for executing a computer program can include, for example, general-purpose and specialized microprocessors, any one or more processors, or any type of digital computer. Generally, a processor receives instructions and data from read-only memory, random-access memory, or both.The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical media, or is functionally coupled to one or more of these devices to receive data from them, transmit data to them, or both. However, a computer does not necessarily have to include such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.The processor and memory can be supplemented or integrated with special logic circuits.

[0042] To provide interaction with a user, one or more aspects of the disclosure may be implemented on a computer that has a display device, such as a monitor with a CRT (cathode ray tube), LCD (liquid crystal display), or touchscreen for displaying information to the user, and optionally a keyboard and a pointing device (such as a mouse or trackball) with which the user can provide input to the computer. Furthermore, other types of devices may be used to interact with a user; for example, feedback provided to the user may be any form of sensory feedback (such as visual, auditory, or tactile feedback); and input from the user may be received in any form, including auditory, verbal, or tactile input.Furthermore, a computer can interact with a user by sending and receiving documents to and from a device used by the user. For example, in response to requests received from the web browser, it can send web pages to a web browser on a user's client device.

[0043] Several implementations have been described. However, it is understood that various modifications can be made without altering the scope of protection and the nature of the disclosure. Accordingly, other implementations fall within the scope of protection of the following claims.

Claims

[1] Computer-implemented procedure that is executed on data processing hardware and causes the data processing hardware to perform operations that include: Receiving audio data that identifies an utterance spoken by a user; Processing the audio data to generate a transcription of the utterance, the transcription comprising a sequence of expressions; Parallel processing of the audio data and the transcription to identify one or more revision expressions in the sequence of expressions, using a multimodal large language model (LLM), wherein the one or more revision expressions specify a revision action to be performed on at least one other expression in the sequence of expressions; and Modifying the transcription based on one or more revision expressions. [2] Computer-implemented method according to claim 1, wherein the operations further comprise: For each individual expression in the sequence of expressions, determining the corresponding intention of the user when speaking the respective expression, based on the parallel processing of the audio data and the transcription, where the identification of one or more revision expressions in the sequence of expressions is based on the corresponding intention determined for each respective expression in the sequence of expressions. [3] Computer-implemented method according to claim 1 or 2, wherein the parallel processing of the audio data and the transcription for each respective expression in the sequence of expressions comprises the following: based on processing the audio data, determining corresponding language features of the respective expression; based on processing the transcription, determining a corresponding linguistic context for the respective expression; and Correlating the relevant linguistic features of the respective expression with the relevant linguistic context of the respective expression. [4] Computer-implemented method according to claim 3, wherein: the corresponding linguistic features of the respective expression are not reproduced in the transcription; and The corresponding linguistic context of the respective expression is not reproduced in the audio data. [5] Computer-implemented method according to claim 3 or 4, wherein the corresponding language features comprise at least one of the following: Pitch information; Audio information; or Prosody information. [6] Computer-implemented method according to any one of claims 1 to 5, wherein the operations further comprise inserting a revision token into the sequence of expressions based on one or more revision expressions, wherein the revision token specifies a corresponding N number of expressions in the at least one other expression and corresponding replacement expressions to replace the corresponding N number of expressions in the at least one other expression. [7] Computer-implemented method according to claim 6, wherein the modification of the transcription is further based on the revision token inserted into the sequence of expressions. [8] Computer-implemented method according to any one of claims 1 to 7, wherein the operations further comprise: Obtaining contextual data associated with the user who made the utterance; and Conditioning the multimodal LLM using context data. [9] Computer-implemented method according to any one of claims 1 to 8, wherein the operations further comprise: Determining a training prompt for a multimodal assistive LLM, wherein the training prompt comprises a transcription editing task and a plurality of training samples, each of which comprises a corresponding training transcription paired with a corresponding modified training transcription; and Generating a variety of training examples based on the training prompt using the multimodal helper LLM; and Training in multimodal LLM using a variety of training examples. [10] Computer-implemented method according to any one of claims 1 to 9, wherein the revision action comprises at least one of the following: a replacement action; a deletion operation; or a spelling challenge. [11] System, encompassing: Data processing hardware; and Storage hardware in communication with data processing hardware, wherein the storage hardware stores instructions which, when executed on the data processing hardware, cause the data processing hardware to perform operations that include the following: Receiving audio data that identifies an utterance spoken by a user; Processing the audio data to generate a transcription of the utterance, the transcription comprising a sequence of expressions; Parallel processing of the audio data and the transcription to identify one or more revision expressions in the sequence of expressions, using a multimodal large language model (LLM), wherein the one or more revision expressions specify a revision action to be performed on at least one other expression in the sequence of expressions; and Modifying the transcription based on the identified one or more revision terms. [12] System according to claim 11, wherein the processes further comprise: For each individual expression in the sequence of expressions, determining the corresponding intention of the user when speaking the respective expression, based on the parallel processing of the audio data and the transcription, where the identification of one or more revision expressions in the sequence of expressions is based on the corresponding intention determined for each respective expression in the sequence of expressions. [13] System according to claim 11 or 12, wherein the parallel processing of the audio data and the transcription for each respective expression in the sequence of expressions comprises the following: based on processing the audio data, determining corresponding language features of the respective expression; based on processing the transcription, determining a corresponding linguistic context for the respective expression; and Correlating the relevant linguistic features of the respective expression with the relevant linguistic context of the respective expression. [14] System according to claim 13, wherein: the corresponding linguistic features of the respective expression are not reproduced in the transcription; and The corresponding linguistic context of the respective expression is not reproduced in the audio data. [15] System according to claim 13 or 14, wherein the corresponding language features comprise at least one of the following: Pitch information; Audio information; or Prosody information. [16] System according to any one of claims 11 to 15, wherein the operations further comprise, based on one or more revision expressions, inserting a revision token into the sequence of expressions, wherein the revision token specifies a corresponding N number of expressions in the at least one other expression and corresponding replacement expressions to replace the corresponding N number of expressions in the at least one other expression. [17] System according to claim 16, wherein the modification of the transcription is further based on the revision token inserted into the sequence of expressions. [18] System according to any one of claims 11 to 17, wherein the processes further comprise: Obtaining contextual data associated with the user who made the utterance; and Conditioning the multimodal LLM to the context data. [19] System according to any one of claims 11 to 18, wherein the processes further comprise: Determining a training prompt for a multimodal assistive LLM, wherein the training prompt comprises a transcription editing task and a plurality of training samples, each of which comprises a corresponding training transcription paired with a corresponding modified training transcription; and Generating a variety of training examples based on the training prompt using the multimodal helper LLM; and Training in multimodal LLM using a variety of training examples. [20] System according to any one of claims 11 to 19, wherein the revision action comprises at least one of the following: a replacement action; a deletion operation; or a spelling challenge. [21] Computer program comprising instructions which, when executed on data processing hardware, cause the data processing hardware to execute the method according to any one of claims 1 to 10.

Images