Maintaining voice hypotheses across computing devices and / or dialog sessions

Abstract

The invention discloses maintaining voice hypotheses across computing devices and / or dialog sessions. Implementations can receive, at a computing device, audio data corresponding to a spoken utterance of a user, process the audio data to generate a plurality of speech hypotheses for one or more portions of the spoken utterance, select a given one of the speech hypotheses, a given one of the speech hypotheses is incorporated as part of a transcription associated with the software application, and a plurality of speech hypotheses are stored. In some implementations, multiple speech hypotheses can be loaded at an additional computing device when transcription is accessed at the additional computing device. In additional or alternative implementations, multiple speech hypotheses can be loaded into memory of a computing device when a software application is reactivated and / or when a subsequent dialog session associated with a transcription is initiated.

Description

Description of the case

[0001] This application is a divisional application of Chinese invention patent application 202080100904.5, with a filing date of December 14, 2020. Technical Field

[0002] The present disclosure relates to maintaining speech hypotheses across computing devices and / or conversational sessions. Background Art

[0003] Humans may participate in human-computer conversations using interactive software applications referred to herein as "automated assistants" (also referred to as "chatbots," "interactive personal assistants," "intelligent personal assistants," "personal voice assistants," "conversational agents," etc.). For example, humans (who may be referred to as "users" when they interact with automated assistants) may provide spoken natural language input (i.e., spoken utterances) to the automated assistant, which may in some cases be converted to text and then processed, and / or by providing textual (e.g., typed) natural language input. The automated assistant typically responds to the spoken utterances by dictating the spoken utterances, providing responsive user interface outputs (e.g., audible and / or visual user interface outputs), controlling smart devices, and / or performing other actions.

[0004] Automated assistants typically rely on a pipeline of components when interpreting and responding to user requests. For example, an automatic speech recognition (ASR) engine can be used to process audio data corresponding to spoken utterances and generate a transcription (i.e., a sequence of words and / or other tokens) of the spoken utterances. However, when performing ASR, certain words may be misrecognized. As a result, the transcription may include words that the user did not intend. This can cause the user to repeat the same spoken utterance (which may be misrecognized again) or cause the user to manually edit misrecognitions, thereby prolonging the human-computer dialogue and / or causing the user to consume additional computing resources outside of the human-computer dialogue. Summary of the invention

[0005] Implementations disclosed herein relate to maintaining speech hypotheses across computing devices and / or conversation sessions. For example, in response to receiving a spoken utterance of "text Jane and tell her I like the new house" from a user of a computing device, speech recognition can be performed on the spoken utterance. When performing speech recognition, multiple term hypotheses can be generated for one or more portions of the spoken utterance. This in turn causes multiple transcription hypotheses to be generated. In this example, assume that multiple term hypotheses are generated for portions of speech corresponding to "house", such as "blouse", "house", and "mouse". This produces multiple transcription hypotheses of text information, such as "I like the new blouse", "I like the new house", etc. Among the multiple transcription hypotheses, one or more techniques can be used to select a given transcription hypothesis as corresponding to the spoken utterance. For example, assume that the selected transcription speech hypothesis corresponds to "I like the new blouse" (rather than "house" as the user intended). The selection of the term hypothesis "blouse" over the term hypothesis "house" in this example represents a misidentification of a given transcription hypothesis in speech recognition. As a result of the misidentification, the transcription rendered at the computing device based on the transcription hypothesis does not correspond to the spoken utterance. Nevertheless, the term hypotheses and / or the transcription hypotheses can be stored in a memory accessible by the computing device. The term hypotheses and / or the transcription hypotheses can be subsequently loaded at the computing device or an additional computing device of the user.

[0006] As an example, it is further assumed that additional user input is received at an additional computing device that is distinct from the computing device that received the original spoken utterance. In this example, in response to receiving the additional user input at the additional computing device, and optionally in response to determining that the additional user input is directed to a software application associated with the transcription and / or in response to determining that the additional user input is directed to the transcription itself, the term hypothesis and / or the transcription hypothesis can be subsequently loaded at the additional computing device. As another example, it is further assumed that the additional user input is received at the computing device that received the original spoken utterance. In this example, in response to receiving the additional user input at the computing device as part of a subsequent conversation session, and optionally in response to determining that the additional user input is directed to a software application associated with the transcription and / or in response to determining that the additional user input is directed to the transcription itself, the term hypothesis and / or the transcription hypothesis can be subsequently loaded at the computing device. In some of these examples, a potentially misidentified portion of the spoken utterance can optionally be graphically defined in the transcription to indicate that an alternative speech hypothesis can be used for the potentially misidentified portion of the spoken utterance. In some other of these examples, the potentially misidentified portion of the spoken utterance may not be provided to the user until the user interacts with the potentially misidentified portion of the transcribed portion (e.g., via a long press at a computing device including a touch screen, via a right click or hover action at a computing device including a stylus or mouse, via a virtual keyboard interaction). Additionally, the potentially misidentified portion of the spoken utterance can be automatically modified based on further user input detected at the computing device or an additional computing device.

[0007] Thus, multiple speech hypotheses generated for the original spoken utterance can be maintained across computing devices and / or across conversation sessions. By maintaining multiple speech hypotheses across computing devices and / or across conversation sessions, potentially misidentified portions of the original spoken utterance can be marked for the user of the computing device and / or automatically corrected across computing devices and / or conversation sessions (and optionally in response to the user's partial interaction with the transcription including the potential misidentification). In these and other ways, misidentification can be identified and corrected in a manner that reduces the length of the human-computer dialogue between the user and one or more computing devices (or the automated assistant implemented thereon), thereby saving computing resources and / or network resources. In addition, the amount of user input received at one or more computing devices can be reduced because one or more computing devices (or the automated assistant implemented thereon) can automatically correct these potential misidentifications or notify the user of these potential identifications, and the user does not need to manually correct any potential misidentifications (and optionally in response to the user's partial interaction with the transcription including the potential misidentification). For example, absent these techniques, a user would have to manually delete portions of the transcription and manually add new portions to the transcription, whereas the techniques described herein enable "one-click" corrections for misidentified portions of the transcription. Additionally, in implementations where the transcription is part of a message (e.g., an email message, a text message, and / or other type of message) transmitted over one or more networks to other users, computing resources and / or network resources can be conserved because additional messages to correct misidentifications in the original message do not need to be generated by one or more computing devices or transmitted to the other users.

[0008] In some implementations, audio data capturing spoken utterances can be processed using an automatic speech recognition (ASR) model at a computing device that detects the audio data (e.g., via a microphone of the computing device). In some additional and / or alternative implementations, the audio data (or features derived therefrom) can be transmitted to a remote computing system (e.g., a server) over one or more networks and processed remotely using the ASR model at the remote computing system. While processing the audio data, a speech recognition model can be used to generate a plurality of speech hypotheses (including lexical hypotheses and / or transcription hypotheses) (e.g., as direct output from processing using the speech recognition model and / or through post-processing of such output (e.g., using a finite state decoding graph)), and each of the plurality of speech hypotheses can be associated with a corresponding confidence level corresponding to the likelihood that a given speech hypothesis corresponds to the spoken utterance or a portion thereof. A given speech hypothesis from among a plurality of speech hypotheses can be selected as corresponding to a spoken utterance based on a corresponding confidence level (e.g., determined directly based on the corresponding confidence level and / or based on a corresponding confidence level associated with a finite state decoding graph) and incorporated as part of a transcription associated with a software application (e.g., an automated assistant application, an email application, a text messaging application, and / or other software application accessible by at least a computing device).

[0009] In addition, multiple speech hypotheses for a given spoken utterance can be stored in a memory accessible by one or more computing devices of the user. For example, assume that a spoken utterance of "I like the new house" is received in a text messaging application for dictation. In this example, a first transcription hypothesis of "I like the new blouse" (including a first term hypothesis of "blouse") can be generated along with a first corresponding confidence level (e.g., associated with the first transcription hypothesis and / or the first term hypothesis), a second transcription hypothesis of "I like the new house" (including a second term hypothesis of "house") can be generated along with a second corresponding confidence level (e.g., associated with the second transcription hypothesis and / or the second term hypothesis), and so on. Further assume that a transcription hypothesis of "I like the new blouse" (e.g., including an incorrectly recognized term "blouse") is selected as corresponding to the spoken utterance, and a transcription is generated for the text messaging application including the incorrectly recognized term.

[0010] In some implementations, multiple speech hypotheses can be stored in a memory accessible by at least a computing device. The memory accessible by the computing device can be, for example, on-device memory (e.g., RAM or ROM) of one or more computing devices, and / or remote memory (e.g., cloud-based memory) of one or more remote systems. Multiple speech hypotheses can be stored in the memory in association with corresponding confidence levels and / or a finite state decoding graph of the spoken utterance, the finite state decoding graph including corresponding confidence levels of terms and / or phrases predicted to correspond to the spoken utterance. In an implementation utilizing an automated assistant implemented by one or more computing devices of a user, multiple speech hypotheses can persist in a memory associated with the automated assistant. In an implementation omitting an automated assistant (e.g., a user dictates directly to a software application without utilizing an automated assistant, or the techniques described herein are implemented by a virtual keyboard and / or dictation application), multiple speech hypotheses can be stored in the memory as metadata associated with the software application, and the software application can include an application programming interface (API) that can generate metadata and transmit it to the memory, and can subsequently receive metadata including multiple speech hypotheses.

[0011] In some versions of those implementations, the plurality of speech hypotheses can be stored in memory in response to determining that one or more conditions are satisfied. The one or more conditions can include, for example, determining that a plurality of corresponding confidence levels for a plurality of the plurality of speech hypotheses are within a threshold range of each other, determining that the corresponding confidence levels do not satisfy a threshold confidence level, determining that a plurality of speech hypotheses were generated for the spoken utterance, determining that a software application associated with the transcription is deactivated (e.g., closed or operating in the background of the computing device), and / or determining that a conversational session between a user of the computing device and the automated assistant is terminated. For example, if for the portion of the spoken utterance corresponding to “house” a first term hypothesis of “blouse” is associated with a first confidence level (e.g., a probability of 0.50), and for the portion of the spoken utterance corresponding to “house” a second term hypothesis of “house” is associated with a second confidence level (e.g., a probability of 0.45), and the confidence level is within a threshold range of confidence levels (e.g., within 0.10), then the term hypothesis for the portion of the spoken utterance corresponding to “house” can be stored in memory. As another example, if a first confidence level associated with a first term hypothesis for “blouse” (e.g., a probability of 0.50) and a second confidence level associated with a second term hypothesis for “house” (e.g., a probability of 0.45) both fail to meet a threshold confidence level (e.g., a probability of 0.65), then the term hypothesis for the portion of the spoken utterance corresponding to “house” can be stored in memory. As yet another example, multiple speech hypotheses can be stored in memory in response to determining that a spoken utterance received at a computing device is complete (e.g., determined using an endpoint model). In some further versions of those implementations, each of the multiple speech hypotheses can be stored in memory, whereas in other further implementations, only one or more alternative speech hypotheses (e.g., excluding a given speech hypothesis that was selected) can be stored in memory.

[0012] In some implementations, a given speech hypothesis corresponding to a spoken utterance or one or more portions thereof can be graphically defined in a transcription. In some versions of those implementations, a given speech hypothesis can be graphically defined based on a corresponding confidence level associated with the given speech hypothesis, such as when one or more conditions described above regarding storing multiple speech hypotheses are met. Continuing with the above example, assume that "blouse" is associated with a corresponding probability of 0.50 and "house" is associated with a corresponding probability of 0.45, thereby satisfying one or more conditions. In other words, the spoken utterance includes the term "house", but it is incorrectly predicted to correspond to "blouse" (e.g., misidentified). Although the term "house" is misidentified, the portion of the transcription that includes "blouse" can be graphically defined to indicate that it is a potential misidentification and / or to indicate that there are one or more alternative speech hypotheses. For example, in this transcription, the term "blouse" can be highlighted, underlined, italicized, associated with a graphical element that causes one or more alternative speech hypotheses to be presented when selected, and / or graphically defined in other ways.

[0013] In some implementations, a plurality of speech hypotheses (or one or more alternative speech hypotheses) can be loaded at the computing device and / or the additional computing device and from a memory accessible by at least the computing device. The plurality of speech hypotheses (or one or more alternative speech hypotheses) can be loaded at the computing device and / or the additional computing device based on user interactions detected at the computing device and / or the additional computing device. The user interactions can include, for example, receiving additional user input (e.g., typed or spoken) directed to the software application and / or the transcription at the computing device or the additional computing device, accessing the software application at the computing device or the additional computing device, interacting with the transcription at the computing device or the additional computing device, and / or any other user interaction with the computing device or the additional computing device associated with the software application or the transcription.

[0014] For example, assume that an original spoken utterance of “Hey assistant, text message Jane I like the new house” is received at a user's standalone speaker device and “house” is misidentified as “blouse” as described above, a speech hypothesis is stored in a memory accessible by at least the computing device, and the user subsequently accesses a transcription of the text message via a text messaging application at the user's mobile device. In this example, multiple speech hypotheses can be loaded at the mobile device (e.g., for the text messaging application), and the transcription of the text message can be graphically bounded in a similar manner as described above to indicate that multiple speech hypotheses are maintained across the computing device. As another example, assume that an original spoken utterance of “Hey assistant, textmessage Jane I like the new house” is received at a user's standalone speaker device and “house” is misidentified as “blouse” as described above, a speech hypothesis is stored in a memory accessible by at least the computing device, and an additional spoken utterance of “Hey assistant, add to the text message to Jane that the backyard is huge” is received at the user's standalone speaker device. In this example, multiple speech hypotheses can be loaded at a standalone speaker device (e.g., for an automated assistant application interacting with a text messaging application), and a transcription of the text message can be graphically delineated in a similar manner as described above to indicate that multiple speech hypotheses are maintained across the conversation session.

[0015] In various implementations, an automated assistant implemented by a computing device and / or an additional computing device can be utilized when performing the techniques described herein. In implementations utilizing an automated assistant, the automated assistant can interact with various software applications installed on the computing device and / or accessible by the computing device. In addition, the automated assistant can facilitate the operations described herein. In additional or alternative implementations, the automated assistant can be omitted. In implementations omitting the automated assistant, the computing device can receive input from the user that allows the user to provide spoken input (e.g., via a graphical interface element). Although the automated assistant is not utilized in these implementations, the client device can still facilitate the operations described herein. In some of these implementations, portions of the user interaction can utilize the automated assistant, whereas other portions of the user interaction can omit the automated assistant.

[0016] In various implementations, a given speech hypothesis can be modified based on additional user input associated with the transcription. In some versions of those implementations, a given speech hypothesis (or a portion thereof) can be automatically modified with an alternative speech hypothesis based on processing the additional user input. Continuing with the above example, it is further assumed that an additional user input (e.g., typed or spoken) of "The backyard is huge" is incorporated into the transcription for the text information (e.g., producing a transcription of "I love the new blouse. The backyard is huge". In this example, a given speech hypothesis of "blouse" can be replaced with an alternative speech hypothesis of "house" in the transcription based on the additional user input including "backyard". In some additional or alternative versions of those implementations, one or more prompts can be generated and rendered at a computing device or an additional computing device to solicit user input (e.g., spoken, touched and / or typed) to modify a given speech hypothesis (or a portion thereof) with an alternative speech hypothesis. The prompt can be generated and rendered at the computing device 110 in response to processing the additional user input. Continuing with the above example, a prompt can ask the user whether to replace "blouse" with "house". In some versions of these implementations, an indication can be rendered (e.g., visually and / or audibly) to indicate that a given speech hypothesis that was originally predicted to correspond to a portion of a spoken utterance is modified with an alternative speech hypothesis. In some versions of those implementations, one or more semantic models or rules can be utilized in determining whether to modify a given speech hypothesis with an alternative speech hypothesis. For example, a semantic model or rule can process a transcription and / or multiple speech hypotheses (or one or more alternative speech hypotheses) and determine that "house" and "backyard" are more semantically meaningful than "blouse" and "backyard". Modification of the given speech hypothesis can be performed based on this semantic determination.

[0017] The above description is provided as an overview of only some of the implementations disclosed herein. Those implementations and other implementations are described in additional detail herein. It should be understood that the technology disclosed herein can be implemented locally at a computing device, remotely at a server that communicates with the computing device via one or more networks, or in a distributed manner at both locations.

[0018] Various implementations can include a non-transitory computer-readable storage medium storing instructions that can be executed by a processor to perform a method such as one or more of the methods described herein. However, still other various implementations can include a system that includes a memory and one or more hardware processors that are operable to execute instructions stored in the memory to perform a method such as one or more of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Depicts a block diagram of an example environment that demonstrates various aspects of the disclosure and in which implementations disclosed herein may be implemented.

[0020] Figure 2 Illustrations depicting utilizing speech hypotheses when maintaining speech hypotheses across computing devices and / or conversation sessions in accordance with various implementations Figure 1 Flow of example implementations of various components.

[0021] Figure 3A is a flow chart illustrating an example method of maintaining speech hypotheses across computing devices in accordance with various implementations.

[0022] Figure 3B is a flow chart illustrating an example method of maintaining speech hypotheses across a conversation session in accordance with various implementations.

[0023] Figure 4 It is a diagram according to various implementation methods. Figure 3A and Figure 3B Flowchart of an example method for correcting speech misrecognition based on speech hypotheses maintained across computing devices and / or conversation sessions.

[0024] Figure 5A , Figure 5B and Figure 5C Various non-limiting examples of maintaining speech hypotheses across computing devices in accordance with various implementations are described.

[0025] Fig. 6A and Figure 6B Various non-limiting examples of maintaining speech hypotheses across dialog sessions in accordance with various implementations are described.

[0026] Fig. 7A and Figure 7B Additional non-limiting examples of maintaining speech hypotheses across dialog sessions in accordance with various implementations are described.

[0027] Figure 8 Depicted are example architectures for computing devices in accordance with various implementations. DETAILED DESCRIPTION

[0028] Now turn to Figure 1 , depicts a block diagram of an example environment that demonstrates various aspects of the present disclosure. The example environment includes multiple computing devices 110 of a user. 1-N In various implementations, computing device 110 1-N Including corresponding user input engine 111 1-N , corresponding rendering engine 112 1-N , corresponding speech recognition engine 120A1 1-N , corresponding speech synthesis engine 130A1 1-N , corresponding natural language understanding (NLU) engine 140A1 1-N and corresponding transcription engine 113 1-N Unless otherwise indicated, computing device 110 1-N Also referred to herein simply as computing device 110. Computing device 110 may be, for example, any combination of a mobile device, a standalone speaker device with a display, a standalone speaker device without any display, a laptop, a desktop computer, a home automation device, and / or any other device.

[0029] A speech hypothesis maintenance system 170 is also shown in FIG. Figure 1 , and in various implementations, includes a speech recognition engine 120A2, a speech synthesis engine 130A2, an NLU engine 140A2, a speech hypothesis maintenance engine 150, and a word detection engine 160. In some implementations, the speech hypothesis maintenance system 170 can be implemented by a server in communication with the computing device 110, and as shown in FIG. Figure 1 The system 170 is shown communicating via a network 190. The network 190 can include, for example, Wi-Fi, Bluetooth, near field communication, a local area network, a wide area network, and / or other networks. In other implementations, the speech hypothesis maintenance system 170 can be implemented locally at the computing device 110. Each computing device 110 can implement what is, from the user's perspective, a logical instance of an automated assistant with which the user can participate in a human-computer conversation session. Such an automated assistant 115 1-N Two instances of Figure 1 Automated assistant 115 can be implemented locally at computing device 110, remotely at a server implementing speech hypothesis maintenance system 170, and / or in a distributed manner at both (e.g., as in Figure 1In implementations utilizing automated assistant 115, it can facilitate operations performed by computing devices 110 and / or speech hypothesis maintenance system 170. In implementations omitting automated assistant 115, computing devices 110 can communicate directly with each other (e.g., computing device 1101 and computing device 110 N ) and / or directly with speech hypothesis maintenance system 170 (eg, via network 190).

[0030] Corresponding user input engine 111 1-N (also referred to herein simply as "user input engine 111") can detect user input at computing device 110. The user input detected at computing device 110 can include spoken utterances detected via a corresponding microphone of the corresponding computing device 110. The microphone can generate audio data based on the spoken utterances detected via the corresponding microphone of computing device 110. For example, user input engine 111 can detect spoken utterances of a user of computing device 110. A corresponding speech recognition engine 120A1 of computing device 110 1-N The speech recognition engine 120A2 of the speech hypothesis maintenance engine 170 (also referred to herein simply as “speech recognition engine 120A1”) and / or speech hypothesis maintenance engine 170 processes the spoken utterance using an automatic speech recognition (ASR) model (e.g., stored locally at the computing device 110 and / or stored remotely at a server in communication with the computing device 110) to generate a plurality of speech hypotheses. Although the techniques described herein generally relate to processing spoken utterances, it should be understood that the user input detected via the user input engine 111 can also include touch input and / or typing input.

[0031] In some implementations, a speech recognition engine 120A1 that is local only to the computing device 110 may be utilized when processing the spoken utterance to generate the plurality of speech hypotheses. In other implementations, a speech recognition engine 120A2 that is remote from the computing device 110 may be utilized when processing the spoken utterance to generate the plurality of speech hypotheses. In some additional and / or alternative implementations, the speech recognition engine 120A1 and the speech recognition engine 120A2 may be utilized to generate the plurality of speech hypotheses, such that the plurality of speech hypotheses include speech hypotheses generated by both the speech recognition engines 120A1 and / or 120A2. The plurality of speech hypotheses can be generated, for example, based on values ​​generated using an ASR model when processing the spoken utterance.

[0032] In some implementations, the ASR model is an end-to-end speech recognition model, so that the speech recognition engine 120A1 and / or 120A2 can generate multiple speech hypotheses for spoken utterances directly based on the values ​​generated using the ASR model. For example, the ASR model can be used to generate multiple speech hypotheses on a character-by-character basis (or a word-by-word basis). A non-limiting example of such an end-to-end model for generating multiple speech hypotheses on a character-by-character basis is a recurrent neural network transformer (RNN-T) model. The RNN-T model is a form of a sequence-to-sequence model that does not use an attention mechanism. Unlike most sequence-to-sequence models that typically need to process an entire input sequence (e.g., an audio data waveform, Mel-frequency cepstral coefficients (MFCCs) and / or other representations) to generate a predicted output, the RNN-T model can be used to continuously process input samples and stream output symbols (e.g., characters of an alphabet). In addition, each of the multiple speech hypotheses generated using the ASR model can be associated with a corresponding probability (or confidence level) that the corresponding speech hypothesis corresponds to the spoken utterance (or portion thereof), and a given speech hypothesis from among the multiple speech hypotheses can be predicted to correspond to the spoken utterance (or portion thereof) based on the corresponding probability.

[0033] In other implementations, when the ASR model is not an end-to-end speech recognition model, the speech recognition engine 120A1 and / or 120A2 can generate predicted phonemes (and / or other representations) instead. For example, the phonemes (and / or other representations) predicted by such a model are then used by the speech recognition engine 120A1 and / or 120A2 to determine multiple speech hypotheses that conform to the predicted phonemes based on the values ​​associated therewith. In doing so, the speech recognition engine 120A1 and / or 120A2 can optionally employ a finite state decoding graph, a lexicon, and / or other resources, and utilize the values ​​associated therewith. For example, an ASR model can be used to generate a finite state decoding graph for a spoken utterance, and the finite state decoding graph maps predicted phonemes to multiple terms and / or phrases (various combinations of which form multiple speech hypotheses), and each of the terms or phrases in the finite state decoding graph can be associated with a corresponding probability (or confidence level) that the terms or phrases correspond to the spoken utterance (or a portion thereof), and a given speech hypothesis (e.g., including a subset of terms or phrases) from among the multiple speech hypotheses (e.g., different combinations of terms or phrases) can be predicted to correspond to the spoken utterance (or a portion thereof) based on the corresponding probability.

[0034] The plurality of speech hypotheses can include, for example, a plurality of term hypotheses and / or a plurality of transcription hypotheses. For example, assume that the computing device 110 receives a spoken utterance of "tell Jane that I like the new house" detected via the user input engine 1111, and the microphone of the computing device 1101 generates audio data corresponding to the spoken utterance. The speech recognition engine 120A11 and / or 120A2 can use the ASR model to process the audio data corresponding to the spoken utterance to generate a plurality of speech hypotheses. In this example, the speech recognition engine 120A11 and / or 120A2 can generate term hypotheses corresponding to each term included in the spoken utterance - "tell", "Jane", "that", "I", "like", "the", "new", "house". For example, the term hypotheses corresponding to "house" of the spoken utterance can include term hypotheses of "house" (e.g., associated with a first value), "blouse" (e.g., associated with a second value), "mouse" (e.g., associated with a value), etc. Assume further that automated assistant 180 is highly confident in the term hypotheses for the spoken utterances "tell," "Jane," "that," "I," "like," "the," and "new," but is not highly confident in a given term hypothesis corresponding to the spoken utterance "house." Multiple transcription hypotheses can be generated based on the multiple term hypotheses corresponding to the spoken utterance "house," resulting in transcription hypotheses for: "tell Jane that I like the new house" (e.g., associated with a first value), "tell Jane that I like the new blouse" (e.g., associated with a second value), "tell Jane that I like the new mouse" (e.g., associated with a third value), etc. Automated assistant 1151 (e.g., when utilizing automated assistant 1151) and / or computing device 1101 (e.g., when a user is dictating without utilizing automated assistant 1151) is able to select, based on the corresponding confidence level, a given term hypothesis from among multiple term hypotheses that corresponds to “blouse” of the spoken utterance, thereby also selecting a given transcription hypothesis from among multiple transcription hypotheses that includes the given term hypothesis.

[0035] In implementations that include automated assistant 115, it can determine an action to perform based on a given speech hypothesis (e.g., a given term hypothesis and / or a given transcription hypothesis) selected from among a plurality of speech hypotheses. The action can be determined based on, for example, an intent and slot values ​​for parameters associated with the intent included in the spoken utterance. For example, automated assistant 115 can cause natural language understanding ("NLU") engine 140A1 to perform a task. 1-N (also referred to herein simply as "NLU engine 1401A1") and / or 140A2 use an NLU model (e.g., stored locally at computing device 110 and / or remotely at a server in communication with computing device 110) to process a given speech hypothesis to determine intent. Continuing with the above example, assume that automated assistant 115 selects "tell Jane that I like the new blouse" as the transcription hypothesis corresponding to the spoken utterance (rather than "house" as intended by the user). In this example, the intent of the action may be "send a message" (e.g., a text, email, voice, social media, and / or other type of message), and the slot value for the "recipient" parameter may be "Jane", while the slot value for the "message content" may be "I like the new blouse". In some versions of those implementations, automated assistant 115 can cause rendering engine 112 of computing device 110 to perform a transcription of the speech hypothesis corresponding to the spoken utterance. 1-N (also referred to herein simply as “rendering engine 112”) audibly renders a spoken response based on a given transcription hypothesis via a respective speaker of computing device 110 and / or visually renders a transcription of the spoken utterance via a respective display of computing device 110 (e.g., using transcription engine 113). 1-N Continuing with the above example, automated assistant 115 can cause computing device 110 to audibly render, via a corresponding speaker of computing device 110, a spoken response of "Ok, sending a message to Jane" indicating an action to be performed by automated assistant 115. In some implementations, speech synthesis engine 130A1 1-N(also referred to herein simply as “speech synthesis engine 1301A1”) and / or 130A2 can process a given speech hypothesis using a speech synthesis model (e.g., stored locally at computing device 110 and / or remotely at a server in communication with computing device 110) to generate a spoken response that is audibly rendered via a speaker of computing device 110. In implementations where automated assistant 115 is omitted, a transcription corresponding to the spoken utterance (e.g., generated using transcription engine 113) can be visually rendered via a corresponding display of computing device 110.

[0036] In some implementations, user input detected via the user input engine 110 can be directed to a software application from among a plurality of software applications 195 accessible by the computing device 110 (e.g., via the network 190) and / or installed on the computing device 110. The transcription generated using the transcription engine 113 can be associated with one or more software applications 195. The software applications 195 can include, for example, an automated assistant application associated with the automated assistant 115, a text messaging application, an email application, a note application, a dictation application, a reminder application, a social media application, a document editing application, a presentation application, a spreadsheet application, a calendar application, a browser application, a document application, and / or any other application capable of transcribing spoken utterances. In addition, the software applications 195 can optionally include an application programming interface (API) for storing a plurality of speech hypotheses and / or loading a plurality of stored speech hypotheses (e.g., as described with respect to Figure 2In some versions of those implementations, the user may utilize the automated assistant 115 to interface with the software application 195. For example, assume that a spoken utterance of "send Jane a text message that says I like your house" is detected via the user input engine 1111 of the computing device 1101. In this example, the automated assistant 1151 is able to utilize resources of a text messaging application accessible by and / or installed on the computing device 1101 to generate and send a text message to an additional user ("Jane") on behalf of the user using the techniques described above with respect to the speech recognition engine 120A1 and the NLU engine 140A1. In some additional or alternative versions of those implementations, the user may still be able to provide the spoken utterance, but without utilizing the automated assistant 115. For example, assume that a user provides a touch input at computing device 1101 that is directed to a graphical element that is associated with a text messaging application and that, when selected, enables the user to provide a spoken utterance (e.g., by activating speech recognition engine 120A1) without interacting with automated assistant 1151. In this example, the user may provide the spoken utterance "I like your house" to generate a text message (e.g., using speech recognition engine 120A11).

[0037] In various implementations, and as Figure 1 As depicted in , speech hypothesis maintenance engine 150 may include confidence engine 151, modification engine 152, and / or bias engine. Although speech hypothesis maintenance engine 150 is described as being remotely implemented at a server in communication with computing device 110, it should be understood that this is for the sake of example and is not intended to be limiting. As noted above, it is possible to perform a speech hypothesis maintenance operation on computing device 1101 and / or computing device 110. N A speech hypothesis maintenance system 170 including a speech hypothesis maintenance engine 150 is implemented locally.

[0038] In various implementations, computing device 110 and / or automated assistant 115 can cause the plurality of speech hypotheses to be stored in memory 199 accessible by computing device 110. Memory 199 can be, for example, on-device memory of computing device 1101, computing device 110 NThe plurality of speech hypotheses stored in the memory 199 can include on-device memory (e.g., RAM or ROM), memory remote to the computing device 110 (e.g., memory of a server), and / or memory associated with the software application 195. In some implementations, the plurality of speech hypotheses stored in the memory 199 can include a plurality of lexical item hypotheses and / or transcription hypotheses generated based on the spoken utterance (and optionally values ​​and / or finite state decoding graphs associated therewith). In some additional or alternative implementations, the plurality of speech hypotheses stored in the memory 199 can include raw output and / or intermediate output from an ASR model. For example, the plurality of speech hypotheses stored in the memory can include a probability distribution sequence (or a subset thereof) generated when processing a spoken utterance or some other intermediate output generated using an ASR model.

[0039] In some implementations, the plurality of speech hypotheses can be stored in memory 199 in response to determining that one or more conditions are satisfied. In some versions of those implementations, confidence engine 151 can determine whether to store the plurality of speech hypotheses based on a value generated by an ASR model when processing a spoken utterance (or portion thereof) detected at computing device 110. For example, assume that the value generated by the ASR model includes a respective confidence level associated with each of the plurality of speech hypotheses (or portions thereof). In some examples, the plurality of speech hypotheses can be stored in memory 199 in response to determining that the plurality of respective confidence levels are within a threshold range of each other. For example, if a first term hypothesis for “house” for a portion of spoken utterance corresponding to “house” is associated with a first confidence level (e.g., a probability of 0.50), and a second term hypothesis for “blouse” for a portion of spoken utterance corresponding to “house” is associated with a second confidence level (e.g., a probability of 0.45) within a threshold range of confidence levels (e.g., within a probability of 0.10), the speech hypothesis for the portion of spoken utterance corresponding to “house” can be stored in memory 199. In some additional or alternative examples, multiple speech hypotheses can be stored in memory 199 in response to determining that the corresponding confidence levels do not meet a threshold confidence level. For example, if a first term hypothesis for "house" for the portion of the spoken utterance corresponding to "house" is associated with a first confidence level (e.g., a probability of 0.50) and a second term hypothesis for "blouse" for the portion of the spoken utterance corresponding to "house" is associated with a second confidence level (e.g., a probability of 0.45) that fails to meet the threshold confidence level (e.g., associated with a probability of 0.65), the speech hypotheses for the portion of the spoken utterance corresponding to "house" can be stored in memory 199.

[0040] In some additional or alternative versions of those implementations, the confidence engine 151 can determine whether to store the multiple speech hypotheses based on the capabilities of the computing device 110 that received the spoken utterance. For example, if the computing device 110 that received the spoken utterance is a stand-alone speaker device without a display (e.g., not capable of visually rendering a transcription of the spoken utterance), the multiple speech hypotheses can be stored in the memory 199 regardless of any determination made by the confidence engine 151. As another example, if the computing device 110 that received the spoken utterance is a stand-alone speaker device with a display, but the user who provided the spoken utterance is not close to the computing device 110 to view the transcription (e.g., determined based on a proximity sensor of the computing device 110), the multiple speech hypotheses can be stored in the memory 199 regardless of any determination made by the confidence engine 151.

[0041] By storing multiple speech hypotheses in memory 199 accessible by computing device 110, multiple speech hypotheses for a spoken utterance (or portions thereof) can be maintained across computing devices and / or conversation sessions (which optionally utilize the functionality of automated assistant 115). For example, and with reference to Figure 2 , depicts a process flow for maintaining speech hypotheses across computing devices and / or conversation sessions. In some implementations, Figure 2 200A may represent user interaction with computing device 1101, and Figure 2 The frame 200B may represent the user and the computing device 110 N Interactions between users and computing devices 110 1-N In some additional or alternative implementations, Figure 2 Frame 200A may represent a conversation session with computing device 1101 (optionally including automated assistant 115), and Figure 2 Block 200B may represent the computing device 110 N A subsequent dialog session (optionally including automated assistant 115), wherein the plurality of speech hypotheses are maintained across the dialog session.

[0042] With respect to box 200A, assume that audio data 201 corresponding to a spoken utterance of "tell Jane that I like the newhouse" is received at computing device 1101. Speech recognition engine 120A11 and / or 120A2 can process audio data 201 using ASR model 120A to generate a plurality of speech hypotheses. Assume further that for the portion of the spoken utterance corresponding to "house", a given speech hypothesis of "blouse" is selected from among the plurality of speech hypotheses based on a respective confidence level associated with each of the plurality of speech hypotheses for the portion of the spoken utterance corresponding to "house". The plurality of speech hypotheses for the portion of the spoken utterance corresponding to "house" can be stored in memory 199 (and optionally in response to confidence engine 151 determining that one or more conditions are satisfied).

[0043] In an implementation utilizing the automated assistant 115, the automated assistant 115 can cause the NLU engine 140A11 and / or 140A2 to process a given speech hypothesis using the NLU model 140A to generate NLU data 203. The NLU data 203 can include, for example, an intent (e.g., send a message) and slot values ​​for parameters associated with the intent (e.g., the slot value for the "recipient" parameter is "Jane", and the slot value for the "message content" parameter is "I like the new blouse"). In addition, the automated assistant 115 can cause the fulfillment engine 221 to process the NLU data 203 using a fulfillment model or rule (not depicted) to generate fulfillment data 204. The fulfillment data 204 can include, for example, a request to establish communication with a software application (e.g., one or more software applications 195) via an API of the software application, a request to be transmitted to a server or other computing device, and / or other fulfillment data. The automated assistant can then cause the fulfillment data 204 to be transmitted to the rendering engine 1121. In addition, automated assistant 115 can cause transcription engine 1131 to process given speech hypothesis 202 to generate transcription 205 corresponding thereto (e.g., “I like the new blouse”), and can cause transcription 205 to likewise be transmitted to rendering engine 1121. Rendering engine 1121 can process fulfillment data 24 and transcription 205 to cause rendered transcription 206 to be rendered at a software application (e.g., a text messaging application, an email application, a social media application, and / or any other software application for communicating messages).

[0044] In implementations that omit utilizing automated assistant 115, NLU engine 140A11 and / or 140A2 and fulfillment engine 221 may be omitted. In these implementations, a user of computing device 1101 may simply direct input to a graphical element that, when selected, enables the user of computing device 1101 to provide spoken utterances (e.g., such as a description of a spoken word) without utilizing automated assistant 115. Fig. 7A and Figure 7B , as described above). The graphical element may be displayed along with an interface associated with the software application. As a result, because the automated assistant 115 is not invoked, the audio data 201 may correspond to the spoken utterance of “I like the new house” (e.g., omitting “tell Jane that …”), thereby avoiding the need for the NLU engines 140A11 and / or 140A2 and the fulfillment engine 221. Multiple speech hypotheses may be stored in the memory 199 in the same manner as described above (and optionally in response to the confidence engine 151 determining that one or more conditions are satisfied). Thus, for the portion of the spoken utterance corresponding to “house”, a given speech hypothesis 202 of “blouse” may be selected from among the multiple speech hypotheses as described above and transmitted to the transcription engine 1131 to generate a transcription 205 (e.g., “I like the new blouse”). The computing device 1101 is capable of causing the transcription 205 to be transmitted to the rendering engine 1121, and the rendering engine 1121 is capable of processing the transcription 205 so that a rendered transcription 206 is rendered at the software application.

[0045] In various implementations, the rendering engine 1121 can further enable, when one or more conditions are met (e.g., as determined by the confidence engine 151), to generate a corresponding value associated with a given speech hypothesis 202 (e.g., as determined by the confidence engine 151). Figures 5A-5C121 graphically delimits the rendered transcription 206. For example, assume that the values ​​associated with the plurality of speech hypotheses are respective probabilities indicating a confidence level that each of the plurality of speech hypotheses corresponds to the spoken utterance (or a portion thereof). In this example, assume that "blouse" is associated with a respective probability of 0.5 and "house" is associated with a respective probability of 0.4, thereby satisfying one or more of the conditions described above with respect to the confidence engine 151. In this example, a given speech hypothesis 202 corresponding to the "house" portion of the spoken utterance may correspond to "blouse" (e.g., selected based on the value), but the rendering engine 1121 may cause the portion of the transcription corresponding to "blouse" to be graphically delimited (e.g., highlighted, underlined, italicized, associated with a graphical element that, when selected, causes alternative speech hypotheses to be presented for the given speech hypothesis 202, and / or otherwise graphically delimited) to indicate that the computing device 1101 has no confidence in the portion of the transcription.

[0046] Assume further that the user refrains from further interaction with computing device 1101 at block 200A and / or terminates the conversation session at block 200A without sending a message to “Jane” that “I like the new blouse”. For example, the user may deactivate the software application (e.g., close the software application or leave the software application working in the background of computing device 1101), affirmatively terminate the conversation session, inferentially terminate the conversation session by no longer providing spoken utterances or touch inputs directed to computing device 1101, and / or computing device 1101 and / or automated assistant 115 may determine that the spoken utterance is complete. As to block 200B, assume further that at computing device 110 1-N A user interaction 207 is detected at one of the locations (e.g., via user input engine 111). User interaction 207 can include, for example, a user subsequently activating the software application (e.g., relaunching the software application or the software application operating in the foreground of computing device 1101), a user interacting with previously rendered transcript 206, a user initiating a subsequent conversation session with automated assistant 115 related to previously rendered transcript 206, and / or continuing a previous conversation session.

[0047] In response to the computing device 110 1-NWhen a user interaction 207 is detected at one of the locations, one or more alternative speech hypotheses 208 can be loaded at a software application and / or an automated assistant interface associated with the automated assistant 115 and from a memory 199 accessible by the computing device 110. The one or more alternative speech hypotheses can include a subset of the plurality of speech hypotheses initially stored in the memory 199, wherein the subset of the plurality of speech hypotheses excludes the given speech hypothesis 202. The subset of the plurality of speech hypotheses initially stored in the memory 199 can also be limited to those associated with values ​​within a threshold range of the given speech hypothesis selected as corresponding to the portion of the spoken utterance. In addition, the rendering engine 1121 and / or 112 N The alternative speech hypothesis can be processed to modify the rendered transcript 206, thereby producing a modified transcript 210. In other words, the rendered transcript 206 can persist in the memory of the software application, but when the software application is deactivated or the conversation session is terminated, the alternative speech hypothesis can be discarded and the rendering engine 1121 and / or 112 N The alternative speech hypothesis can be made to appear from the perspective of a user of computing device 110 as if the alternative speech hypothesis is also persisted in the memory of the software application (e.g., graphically defined). In some implementations, and prior to rendering modified transcription 210, when the transcription is not persisted in the memory of the software application and / or automated assistant when utilized, alternative speech hypothesis 208 can be generated by transcription engine 1131 and / or 113 N Processing to generate a transcription 209. In these implementations, rendering engine 1121 and / or 112 N The alternative speech hypotheses 208 and the transcription 209 can be processed to generate a modified transcription 210 .

[0048] Return to reference Figure 1, and subsequent to loading the plurality of speech hypotheses (or one or more alternative speech hypotheses) from memory 199 accessible by computing device 110, modification engine 152 can determine whether to modify the transcription of the spoken utterance (or portion thereof) based on additional user input received at computing device 110. The additional user input can be, for example, additional spoken utterance, typed input (e.g., via a keyboard of computing device 110), and / or touch input (e.g., via a display of computing device 110). For example, assume that a transcription of “I like the new blouse” was previously generated based on a spoken utterance of “I like the new house”, where a portion “house” of the spoken utterance was misidentified as “blouse”, and assume that a plurality of speech hypotheses for the portion “house” of the spoken utterance are stored in memory 199. Assume further that an additional spoken utterance of “add to my message to Jane that the backyard is huge” (or the additional spoken utterance “the backyard is huge” and / or a typed input of “the backyard is huge” when automated assistant 115 is not utilized) is correctly recognized and incorporated into the transcription based on selecting a given additional speech hypothesis from among a plurality of additional speech hypotheses based on an additional value associated with processing the additional spoken utterance, and produces the transcription “I like the new blouse, the backyard is huge” (e.g., where “blouse” is an incorrect recognition).

[0049] In this example, the modification engine 152 can determine whether to modify the original given speech hypothesis of "blouse" selected based on the additional given speech hypothesis (or its lexical item hypothesis) of "the backyard is huge" and / or based on the value associated with the original given speech hypothesis. For example, the modification engine 152 can determine that the original given speech hypothesis of "blouse" is a misrecognition based on the additional given speech hypothesis including the lexical item "backyard" and / or based on the value associated with "house" when processing the original spoken utterance, and that the alternative speech hypothesis of "house" is the correct speech hypothesis for the original portion of the spoken utterance. In other words, the modification engine 152 can automatically modify the portion of the transcription including "blouse" by replacing "blouse" with "house".

[0050] In some implementations, the modification engine 152 can automatically replace a given speech hypothesis (or a portion thereof) with an alternative speech hypothesis in response to selecting an additional given speech hypothesis for an additional spoken utterance or in response to receiving additional typed input. The modification engine 152 can automatically replace a given speech hypothesis (or a portion thereof) with an alternative speech hypothesis in response to determining that a value associated with the alternative speech hypothesis (and optionally in conjunction with a value associated with the additional given speech hypothesis) satisfies a threshold. Continuing with the above example, the modification engine 152 can automatically replace "blouse" with "house" in the transcription in response to selecting an additional speech hypothesis including "backyard". In some versions of those implementations, the modification engine 152 can cause the rendering engine 112 to render an indication (e.g., visually and / or audibly) that a given speech hypothesis that was originally predicted to correspond to a portion of the spoken utterance is modified. In some additional or alternative implementations, the modification engine 152 can generate a prompt soliciting user input (e.g., spoken, touched, and / or typed) to modify a given speech hypothesis (or a portion thereof) with an alternative speech hypothesis. The modification engine 152 can generate a prompt in response to determining that the value associated with the alternative speech hypothesis (and optionally in combination with the value associated with the additional given speech hypothesis) fails to meet the threshold, instead of automatically replacing the given speech hypothesis (or part thereof) with the alternative speech hypothesis. The prompt can be generated and rendered at the computing device 110 in response to selecting the additional given speech hypothesis for the additional spoken utterance or in response to receiving the additional typing input. Continuing with the above example, the modification engine 152 can generate and render a prompt asking the user whether to replace "blouse" with "house". The modification engine 152 can optionally utilize one or more semantic models or rules when determining whether to modify the given speech hypothesis with the alternative speech hypothesis. For example, the semantic model or rule can process the transcription and / or multiple speech hypotheses (or one or more alternative speech hypotheses) and determine that "house" and "backyard" are more semantically meaningful than "blouse" and "backyard". The modification engine 152 can determine to modify the given speech hypothesis based on this semantic determination.

[0051] In implementations where the ASR model is an end-to-end model and output generated using the ASR model includes a plurality of speech hypotheses and a corresponding confidence level associated with each of the plurality of speech hypotheses, the modification engine 152 can modify a given speech hypothesis (e.g., exclude a subset of the plurality of speech hypotheses that were originally predicted to correspond to the spoken utterance (or a portion thereof)) using the corresponding probabilities of the one or more alternative speech hypotheses. Continuing with the above example, assume that a given speech hypothesis of "blouse" is associated with a first corresponding confidence level of 0.50 and is selected to correspond to the portion of the spoken utterance corresponding to "house", and an alternative speech hypothesis of "house" is associated with a second corresponding confidence level of 0.45 and is not selected to correspond to the portion of the spoken utterance, but is stored in the memory 199 (and optionally stored together with the second confidence level). In this example, the modification engine 152 can replace "blouse" with "house" after loading the plurality of speech hypotheses (or one or more alternative speech hypotheses) from the memory 199 and based on the second corresponding confidence level associated with "house".

[0052] In implementations where the ASR model is not an end-to-end model and the output generated using the ASR model includes a finite state decoding graph that includes corresponding confidence levels associated with terms and / or phrases predicted to correspond to the original spoken utterance, the modification engine 152 is capable of re-decoding the finite state decoding graph to modify the given speech hypothesis. Continuing with the above example, assume that a given speech hypothesis of "blouse" in the finite state decoding graph is associated with a first corresponding confidence level of 0.50 and is selected to correspond to the portion of the spoken utterance corresponding to "house", and an alternative speech hypothesis of "house" in the finite state decoding graph is associated with a second corresponding confidence level of 0.45, is not selected to correspond to the portion of the spoken utterance, but is stored in the memory 199 as part of the finite state decoding graph (and optionally stored together with the second confidence level). In this example, the modification engine 152 is capable of replacing "blouse" with "house" after loading the plurality of speech hypotheses (or one or more alternative speech hypotheses) from the memory 199 and based on re-decoding the finite state decoding graph in response to receiving additional user input. Modification engine 151 may optionally utilize a finite state decoding graph that includes terms and / or phrases based on the original spoken input and additional user input.

[0053] In various implementations, the bias engine 153 can bias subsequent speech recognition of subsequent spoken utterances on any modification of speech hypotheses. The bias engine 153 can identify misrecognized terms and / or terms that replace or substitute for misrecognized terms, and can store these terms in the memory 199. Continuing with the above example, the bias engine 143 can cause the terms "house" and "blouse" to be stored as biased terms. Further assume that the computing device 110 receives a subsequent spoken utterance of "the new house is beautiful". In this example, the bias engine 153 can cause the automated assistant to bias speech recognition of subsequent spoken utterances toward the term hypothesis of "house" and / or away from the term hypothesis of "blouse". In some implementations, the bias engine 153 can bias speech recognition of subsequent spoken utterances toward and / or away from the biased terms for only the current human-computer dialogue session. In some additional and / or alternative implementations, the bias engine 153 can bias the speech recognition of subsequent spoken utterances toward and / or away from the biased terms for subsequent human-machine dialogue sessions. For example, the bias can be limited to a specific user interaction and / or a specific duration. By limiting the bias, the speech hypothesis maintenance system 170 can combat excessive bias toward and / or away from the biased terms.

[0054] In various implementations, the word detection engine 160 can process the audio data using various word detection models to determine whether the corresponding spoken utterance and / or additional spoken utterances include a particular term or phrase, whether the spoken utterance is complete, and / or other word processing models. The word detection model can include, for example, a hot word detection model that is trained to detect a particular term or phrase (e.g., "Assistant," "Hey Assistant," "Okay Assistant," etc.) that invokes the automated assistant 115 and activates one or more components of the computing device 110 and / or speech error recognition system 160 (e.g., speech recognition engines 120A1 and / or 120A2). The word detection model can additionally or alternatively include, for example, an endpoint model that is trained to detect when a spoken utterance is complete (e.g., based on acoustic features of the terms and / or audio data included in the spoken utterance).

[0055] Therefore, it is possible to cross the computing device 110 1-N and / or the conversation session maintains multiple speech hypotheses generated for the original spoken utterance. 1-N and / or maintain multiple speech hypotheses across a conversation session, capable of performing the same operation on the computing device 110 1-N to mark and / or target a user of computing device 110 1-NThe user of the computing device 110 can automatically correct the potentially misrecognized portion of the original spoken utterance. In this way, it is possible to reduce the time it takes for the user to interact with the one or more computing devices 110. 1-N 110 (or an automated assistant implemented thereon) in a manner that identifies and corrects misidentifications in a manner that preserves computing resources and / or network resources. In addition, it is possible to reduce the time required for one or more computing devices 110 to communicate with one another. 1-N The amount of user input received at one or more computing devices 110 1-N (or an automated assistant implemented thereon 115 1-N ) can automatically correct these potential misidentifications or notify the user of these potential misidentifications, and the user does not need to manually correct any potential misidentifications. In addition, in implementations where the transcription is part of a message (e.g., an email message, a text message, and / or other type of message) transmitted over network 190 to other users, computing resources and / or network resources can be conserved because additional messages to correct misidentifications in the original message do not need to be processed by one or more computing devices 110. 1-N Generate or transmit to other users.

[0056] Now turn to Figure 3A , Figure 3B and Figure 4 , depicts a flow chart of an example method 300A, 300B, 400 for maintaining speech hypotheses across computing devices and / or conversational sessions and correcting misrecognitions based thereon. For convenience, the operations of the methods 300A, 300B, 400 are described with reference to a system that performs the operations. The system of the methods 300A, 300B, 400 includes one or more processors and / or other components of a computing device. For example, the system of the methods 300A, 300B, 400 may be Figure 1 , Figures 5A-5C , Figure 6A-6B and Figure 7A-7B computing device 110, Figure 8 The computing device 810, one or more servers, other computing devices, and / or any combination thereof. In addition, although the operations of methods 300A, 300B, and 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0057] Initial reference Figure 3A, and with respect to maintaining multiple speech hypotheses across computing devices, at block 352A, the system receives audio data corresponding to a spoken utterance of a user (e.g., "tell Jane that I like the new house") via a microphone of the computing device of the user. At block 354A, the system processes the audio data corresponding to the spoken utterance using an automatic speech recognition (ASR) model to generate multiple speech hypotheses for a portion of the spoken utterance based on values ​​generated using the ASR model. The multiple speech hypotheses can include, for example, term hypotheses for each term included in the spoken utterance (e.g., "house," "blouse," "mouse," etc. for a portion of the spoken utterance including "house") and / or transcription hypotheses including various combinations of term hypotheses (e.g., "I like the new house," "I like the new blouse," etc.). Each of the multiple utterance hypotheses can be associated with a corresponding confidence level. For example, each term hypothesis can be associated with a corresponding confidence level, and each transcription hypothesis can be associated with a distinct corresponding confidence level.

[0058] At box 356A, the system selects a given speech hypothesis predicted to correspond to a portion of the spoken utterance from among a plurality of speech hypotheses based on the value. In some implementations, the value generated using the ASR model may include a corresponding confidence level. In some additional or alternative implementations, the value generated using the ASR model may be based on a generated finite state decoding graph that includes terms and / or phrases predicted to correspond to the spoken utterance (or portion thereof) and corresponding confidence levels for the terms and / or phrases. A given speech hypothesis can be selected based on the corresponding confidence level (e.g., selecting a term hypothesis for "blouse" instead of "house" as desired by the user).

[0059] At box 358A, the system causes a given speech hypothesis to be incorporated as part of a transcription that is associated with a software application. The transcription associated with the software application can be, for example, text for an automated assistant application, a text message for a text messaging application, an email for an email application, a note for a note application, text for a dictation application, a reminder for a reminder application, a social media post or message for a social media application, a document for a document editing application, a presentation for a presentation application, an electronic form for an electronic form application, a calendar entry for a calendar application, a query for a browser application, and / or any other application capable of transcribing spoken utterances. The transcription can be visually rendered at a user interface associated with an automated assistant (if utilized) and / or a user interface associated with a software application at a computing device (if it includes a display). In addition, the transcription can optionally include an additional portion prior to the portion incorporated into the transcription.

[0060] In some implementations, block 358A may include sub-block 358A. If included, at sub-block 358A, the system graphically defines portions of the transcription. The transcription may be graphically defined based on values ​​associated with a given speech hypothesis when one or more conditions are met (e.g., as described above with respect to Figure 1 For example, assume that the values ​​associated with multiple speech hypotheses for the spoken utterance "I like the new house" are respective probabilities indicating a confidence level that each of the multiple speech hypotheses corresponds to the spoken utterance (or a portion thereof). In this example, assume that "Blouse" is associated with a respective probability of 0.5 and "house" is associated with a respective probability of 0.4. A given speech hypothesis corresponding to the "house" portion of the spoken utterance in this example may correspond to "blouse" (e.g., selected based on the value), but the system may cause the portion of the transcription corresponding to "blouse" to be graphically delimited (e.g., highlighted, underlined, italicized, associated with a graphical element that, when selected, causes alternative speech hypotheses for the given speech hypothesis 202 to be presented, and / or otherwise graphically delimited) to indicate that the computing device has no confidence in the portion of the transcription (e.g., as described above with respect to Figure 1 as described by the confidence engine 151).

[0061] At block 360A, the system stores the plurality of speech hypotheses in a memory accessible by at least the computing device. The memory may be, for example, on-device memory local to the computing device and / or remote memory at a server in communication with the computing device. In implementations where the speech hypotheses are transmitted over a network, the speech hypotheses can be encrypted to maintain the privacy of the user. In some implementations, the system may store the plurality of speech hypotheses in a memory accessible by the computing device only in response to determining that one or more conditions are satisfied (e.g., as described above with respect to the confidence engine 151).

[0062] At box 352A, the system determines whether to transmit multiple speech hypotheses to the user's additional computing device. The system can transmit multiple speech hypotheses to the additional computing device in response to determining that one or more conditions are met. The one or more conditions for transmitting multiple speech hypotheses to the additional computing device can include, for example, determining that the software application is accessed at the additional computing device, determining that the user has interacted with the transcription, determining that there are multiple speech hypotheses of the transcription associated with the software application, and / or other conditions for transmitting multiple speech hypotheses. The system can transmit multiple speech hypotheses (and optionally corresponding confidence levels and / or corresponding finite state decoding graphs of the spoken utterance) from the memory to the additional computing device (e.g., in a point-to-point manner from the computing device to the additional computing device or from a remote system to the additional computing device). If, at the iteration of box 362A, the system determines that one or more conditions are not met at box 362A, the system continues to monitor for the satisfaction of one or more conditions at box 362A. If, at the iteration of box 362A, the system determines that one or more conditions are met, the system transmits multiple speech hypotheses to the additional computing device and proceeds to box 364A.

[0063] At block 364A, the system causes the plurality of speech hypotheses to be loaded at the additional computing device. In implementations where the software application is associated with an automated assistant, the plurality of speech hypotheses can be loaded at the automated assistant application. In implementations where the software application is not associated with an automated assistant, the software application can include an API capable of receiving the plurality of speech hypotheses. The system can proceed to Figure 4 462. Described in more detail below Figure 4 Box 462.

[0064] Reference now Figure 3B , and regarding maintaining multiple speech hypotheses across a conversation session, the system is able to perform the above regarding Figure 3A The operations of blocks 352B, 354B, 356B, and 358B (optionally including sub-block 358B1 ) are performed in the same or similar manner as the operations of blocks 352A, 354A, 356A, and 358A (optionally including sub-block 358A1 ) described respectively.

[0065] However, at block 360B, when the system stores the plurality of speech hypotheses in a memory accessible by at least the computing device, the storage can be based on the operations of subblock 360B1 and / or the operations of subblock 360B2. In some implementations, and at block 360B1, the system can store the plurality of speech hypotheses in response to determining that the spoken utterance is complete. The system can use an endpoint model (e.g., as described above with respect to Figure 1 The system may determine that the spoken utterance is complete by performing a word detection engine 160 described in the example of FIG. 360B. In some additional or alternative implementations, and at block 360B2, the system may store a plurality of speech hypotheses in response to determining that the software application has been deactivated. The system may determine that the software application is deactivated if the software application is closed by a user of the computing device or the software application is only working in the background of the computing device.

[0066] Additionally, at box 362B, the system loads multiple speech hypotheses at the computing device based on the operations of subbox 362B1 and / or the operations of subbox 362B2. In some implementations, and at box 362B1, the system can load multiple speech hypotheses at the computing device in response to receiving additional spoken utterances via a microphone of the computing device. In some versions of those implementations, multiple speech hypotheses can be loaded only if the additional spoken utterances are related to the software application and / or the transcription. In some additional or alternative implementations, and at box 362B2, the system can load multiple speech hypotheses at the computing device in response to determining that the software application has been subsequently activated. If the software application is restarted by a user of the computing device or the software application is working in the foreground of the computing device, the system can determine that the software application is subsequently activated. The system can proceed to Figure 4 462. Described in more detail below Figure 4 Box 462.

[0067] Reference now Figure 4 , and with respect to automatically correcting speech misrecognition, at box 452, the system receives additional audio data user input at the computing device or the additional computing device via a microphone of the computing device or the additional computing device (e.g., "also tell Jane that the backyard is huge"). In some implementations, when the additional spoken utterance is received at box 452, multiple speech hypotheses may already be loaded at the computing device or the additional computing device. In some additional or alternative implementations, multiple speech hypotheses may be loaded at the computing device and / or the additional computing device in response to receiving the additional spoken utterance at box 452. Although described herein with respect to receiving the additional spoken utterance, Figure 4, but it should be understood that this is for the sake of example and is not intended to be limiting. For example, a user can access the transcription and begin adding terms or phrases to the transcription, and prior misidentifications can still be corrected using the techniques described herein (e.g., with respect to blocks 460, 462, 464, and 466) and blocks 454, 456, and 458 can be skipped.

[0068] At box 454, the system uses the ASR model to process the additional audio data corresponding to the additional spoken utterance to generate a plurality of additional speech hypotheses based on the additional values ​​generated using the ASR model. The plurality of additional speech hypotheses can also include, for example, term hypotheses for each term included in the spoken utterance and / or transcription hypotheses including various combinations of term hypotheses. Each of the plurality of speech hypotheses can be associated with a corresponding confidence level. For example, each term hypothesis can be associated with a corresponding confidence level, and each transcription hypothesis can be associated with a distinct corresponding confidence level.

[0069] At box 456, the system selects an additional given speech hypothesis predicted to correspond to the additional spoken utterance from among the plurality of additional speech hypotheses based on the additional value. In some implementations, the additional value generated using the ASR model may include a corresponding confidence level. In some additional or alternative implementations, the additional value generated using the ASR model may be based on a generated finite state decoding graph that includes terms and / or phrases predicted to correspond to the spoken utterance (or portions thereof) and corresponding confidence levels for the terms and / or phrases. A given speech hypothesis can be selected based on the corresponding confidence level (e.g., selecting a term hypothesis of "backyard" as desired by the user).

[0070] At box 458, the system causes the additional given speech hypothesis to be incorporated as an additional portion of the transcription. For example, the additional portion of the transcription can be appended to the portion of the transcription from box 358A and / or box 368B. At box 460, the system determines whether to modify the given speech hypothesis. The system can determine whether to modify the given speech hypothesis (e.g., selected as corresponding to the spoken utterance (or portion thereof) at box 356A or 356B) based on the additional given speech hypothesis incorporated as an additional portion of the transcription and / or based on the value associated with multiple speech hypotheses loaded at the computing device or additional computing device (e.g., at box 364A and / or box 362B). If, at an iteration of box 460, the system determines not to modify the given speech hypothesis, the system can return to box 452. If, at an iteration of box 460, the system determines to modify the given speech hypothesis, the system can proceed to box 462. At box 462, the system selects an alternative speech hypothesis from among the multiple speech hypotheses generated at box 354A based on at least the additional given speech hypothesis. For example, assume that an original spoken utterance of "I like the new house" is received, but "house" is misidentified as "blouse" and incorporated into the transcription. Assume further that an additional spoken utterance of "the backyard is huge" is received (e.g., at an additional computing device and / or during a subsequent conversation session) and correctly identified. In this example, the system can determine to modify it by replacing or substituting the word "blouse" in the transcription with the word "house" based on the additional given speech hypothesis being related to the alternative speech hypothesis for "house" (e.g., referencing "backyard").

[0071] At box 464, the system replaces (or substitutes) the given speech hypothesis with the alternative speech hypothesis in the transcription. In some implementations, the system automatically replaces the given speech hypothesis with the alternative speech hypothesis in response to selecting the alternative speech hypothesis at box 462. In some additional or alternative implementations, the system can generate a prompt soliciting user input to modify the given speech hypothesis with the alternative speech hypothesis. For example, the system can generate a notification that includes a first selectable interface element that, when selected, causes the given speech hypothesis to be replaced with the alternative speech hypothesis, and includes a second selectable interface element that, when selected, prevents the given speech hypothesis from being replaced with the alternative speech hypothesis. In this document (e.g., with respect to Figure 1 , Figures 5A-5C , Figure 6A-6B and Figure 7A-7B) describes modifying a given speech hypothesis in more detail. At block 466, the system causes the computing device or an additional computing device to visually render one or more graphical elements indicating that the given speech hypothesis is modified. For example, the system can optionally render an indication that a given speech hypothesis (e.g., "blouse") is replaced with an alternate speech hypothesis (e.g., "house").

[0072] Now turn to Figure 5A , Figure 5B and Figure 5C , depicting computing devices 110 across users 101 1-N Various non-limiting examples of maintaining speech hypotheses. As shown, Figure 5A The computing device 1101 depicted in is a standalone computing device having a microphone for generating audio data based on spoken utterances and / or other audible input, a speaker for audibly rendering synthesized speech and / or other audible output, and a display 1801 for visually rendering transcriptions and / or other visual output. In addition, Figure 5B and Figure 5C The computing device 110 depicted in N is a mobile device having: a microphone for generating audio data based on spoken utterances and / or other audible input; a speaker for audibly rendering synthesized speech and / or other audible output; and a display 180 N , for visually rendering the transcription and / or other visual output. However, it should be understood that this is for the sake of example and is not intended to be limiting. For example, computing device 110 1-N This can be any combination of: mobile devices, stand-alone speaker devices with a display, stand-alone speaker devices without a display, home automation devices, vehicle systems, laptops, desktop computers, tablet computers, and / or any other computing device with which user 101 is able to interact.

[0073] In some implementations, one or more computing devices 110 1-N can be on the computing device 110 1-N Locally and / or over a network (e.g. Figure 1 network 190) and computing device 110 1-N The automated assistant is implemented remotely at one or more servers in communication with the Figure 1In some versions of those implementations, the automated assistant 115 may be activated based on detecting one or more terms or phrases included in the spoken utterance (e.g., “Assistant,” “Hey Assistant,” and other terms or phrases detected via the hotword detection model), based on detecting that the user 101 is gazing at one of the computing devices and beginning to provide the spoken utterance (e.g., via the gaze detection model), and / or providing any input that otherwise invokes the automated assistant (e.g., a command to the computing device 110 to initiate a spoken utterance). 1-N One of the squeeze, 180 on the display 1-N selection of a graphical element on the computing device 110 1-N actuation of one of the hardware buttons) to invoke the automated attendant.

[0074] In some additional or alternative implementations, the automated assistant may be omitted, and the user 101 may be able to interact with one or more graphical elements or hardware buttons to provide spoken utterances without interacting with the automated assistant. Figure 5B and Figure 5C The computing device 110 is shown N Includes voice input interface element 185 that user 101 may select to provide spoken utterance N , and can be transmitted via the computing device 110 N The microphone generates audio data corresponding to the spoken utterance. In some implementations, user 101 can select the voice reply interface element 185 without selecting the voice reply interface element 185. N For example, active monitoring for spoken utterances may occur to avoid the user selecting a voice reply interface element 185 N In some additional or alternative implementations, the voice reply interface element 185 may be omitted. N .also, Figure 5B and Figure 5C The computing device 110 is shown N Also included is a text entry interface element 184 that the user 101 can select to provide typing input via a virtual keyboard or other touch and / or typing input. N , and the text corresponding to the typed input can be read by the computing device 110 N In some implementations, the text reply interface element 184 may additionally and / or alternatively be omitted. N (For example, user 101 may only provide spoken utterances). Figure 5B and Figure 5C Display 180 N Also includes system interface elements 181 N , 182 N and 183 N, these system interface elements can be interacted with by the user 101 to enable the computing device 110 N Perform one or more actions.

[0075] Specifically refer to Figure 5A , assume that the computing device 1101 receives a spoken utterance 552A of "Hey assistant, draft a text message to Jane Doe that says I just bought new shoes" from the user 101. The automated assistant can cause the computing device 1101 (and / or a remote system) to process the audio data corresponding to the spoken utterance 552A using an ASR model to generate a plurality of speech hypotheses and corresponding values ​​associated with each of the plurality of speech hypotheses. The plurality of speech hypotheses can include, for example, transcription hypotheses and / or term hypotheses. For example, assume that the automated assistant is highly confident in the term hypotheses corresponding to "I", "just", "bought", and "new" based on the values ​​generated using the ASR model. However, further assume that the automated assistant is not confident in the term hypothesis corresponding to "shoes" based on the values ​​generated using the ASR model. For example, table 520A includes a table of term hypotheses corresponding to "shoes" of spoken utterance 552A, and a corresponding confidence level associated with each term hypothesis as a value generated using the ASR model. Table 520A includes a first term hypothesis of "juice" associated with a first probability of 0.50, a second term hypothesis of "shoes" associated with a second probability of 0.45, and a third term hypothesis of "screws" associated with a third probability of 0.05. Although particular term hypotheses are depicted in table 520A as being associated with particular probabilities, it should be understood that table 520A is provided for purposes of example and is not intended to be limiting. Further, assume that the automated assistant determines, based on the corresponding values ​​associated with each of the plurality of term hypotheses, that the portion of spoken utterance 552A that includes "shoes" corresponds to the term hypothesis "juice" as indicated by table 520A.

[0076] Assume further that the automated assistant generates and audibly renders a spoken response 554A1 of "Okay, drafting a text message to Jane Doe" via a speaker of computing device 1101, and generates a transcription of "I just bought new juice" as indicated by 554A2. In this example, the transcription can be generated based on a selected transcription hypothesis of the misrecognized term hypothesis including "juice" (e.g., "I just bought new juice"). A plurality of speech hypotheses (including term hypotheses as shown in table 520A and / or transcription hypotheses based thereon) can be stored in a memory accessible to computing device 1101 as indicated by 554A3. In some implementations, the plurality of speech hypotheses can be stored in the memory only in response to determining that one or more conditions are satisfied. The one or more conditions can include, for example, determining that multiple of the respective confidence levels for multiple of the multiple speech hypotheses are within a threshold range of each other (e.g., a respective confidence level of 0.50 for “juice” is within 0.15 of a respective confidence level of 0.45 for “shoes”), determining that the respective confidence levels do not satisfy a threshold confidence level (e.g., the respective confidence levels for both “juice” and “shoes” fail to satisfy a threshold of 0.70), determining that multiple speech hypotheses were generated for the spoken utterance (e.g., the transcription itself was generated), determining that a software application associated with the transcription is deactivated (e.g., closed or working in the background of computing device 1101), and / or determining that a conversational session between user 101 of computing device 1101 and the automated assistant is terminated. Assume further that the conversational session between user 101 and the automated assistant implemented by computing device 1101 is terminated without sending a text message.

[0077] Specifically refer to Figure 5B , assuming that user 101 is on computing device 110 N The text messaging application is accessed at the address indicated by graphical element 522 to complete the text message to Jane Doe. Figure 5A In this example, the text message can be completed during the conversation session of the computing device 110 in response to determining that the user 101 has accessed the computing device 110 N accessed text messaging application, in response to determining that user 101 is in a state that can be accessed by computing device 110 N The accessed text messaging application accesses the transcript 552B2 associated with Jane Doe, and / or based on other users' interactions with the computing device 110 NThe interaction is performed to load multiple speech hypotheses (e.g., from Figure 5A Table 520A). In some implementations, and as Figure 5B As shown, it is possible to graphically define portions of transcript 552B2 (e.g., "* juice *") to indicate that a portion of the transcription 552B2 may include a potential misidentification. In some additional or alternative implementations, the transcription 552B2 may not be graphically delimited. However, in some versions of those implementations, the portion of the transcription 552B2 may be graphically delimited in response to a user interaction directed to the portion of the transcription 552B2 that includes the potential misidentification. For example, the user 101 can long-press (e.g., hold for a threshold duration, such as two or three seconds) the portion of the transcription 552B2, hover over or right-click the portion of the transcription 552B2 (e.g., if the computing device includes a stylus or is a laptop, desktop, or tablet computer), or select to have multiple speech hypotheses be presented to the user for viewing on the computing device 101 when selected. N As another example, user 101 can use computing device 110 to select a graphical element. N Display 180 N The virtual keyboard can be provided on the computing device 110 N Multiple speech hypotheses are loaded at .

[0078] Furthermore, in various implementations, and prior to receiving any additional user input, computing device 110 N Enable notification 579B to be rendered on computing device 110 N at (eg, via display 180 N Or via Figure 5B , a separate interface shown). Notification 579B can solicit additional user input asking user 101 whether a given speech hypothesis (e.g., "juice") is correct or whether user 101 wants an alternative speech hypothesis. For example, notification 579B can include one or more selectable graphical elements that, when selected, cause the given speech hypothesis to be maintained in a portion of transcription 552B2 (e.g., selection of selectable graphical element 579B1 for "juice") or cause the given speech hypothesis to be replaced or substituted in a portion of transcription 552B2 with an alternative speech hypothesis associated with the selectable graphical element (e.g., selection of selectable graphical element 579B2 for "shoes"). In some implementations, notification 579B may not be generated and / or rendered until additional user input directed to transcription 552B is received (e.g., as described in connection with a description of the alternative speech hypothesis). Figure 5C described).

[0079] Specifically refer to Figure 5C , and similar to Figure 5B , assuming that the user is on computing device 110 N The text messaging application is accessed at the address indicated by graphical element 522 to complete the text message to Jane Doe. Figure 5A In this example, the text message can be completed during the conversation session of the user 101. In response to determining that the user 101 has accessed the text messaging application accessible by the computing device 110N, in response to determining that the user 101 is in the text messaging application accessible by the computing device 110N, N The accessed text messaging application accesses the transcript 552C2 associated with Jane Doe, and / or based on other users' interactions with the computing device 110 N The interaction comes as indicated by 552C1 on the computing device 110 N Load multiple speech hypotheses (e.g., from Figure 5A 520A). It is further assumed that before a given speech hypothesis is modified (automatically or in response to further user input being received in response to any prompt soliciting further user input) at computing device 110 N Additional user input 552C3 (eg, typed or spoken) of “They're really comfortable” is received at the transcribed location, resulting in a combined transcription of “I just bought new juice. They're really comfortable”.

[0080] In various implementations, the given phonetic hypothesis for "juice" can be modified based on the additional user input 552C3 associated with the combined transcription. In some versions of those implementations, the given phonetic hypothesis for "juice" can be automatically modified with an alternative phonetic hypothesis for "shoes" based on processing the additional user input 552C3 (e.g., as Figure 5C In some additional or alternative versions of those implementations, user input soliciting modification of a given speech hypothesis for “juice” with an alternative speech hypothesis (e.g., regarding Figure 5BIn some versions of those implementations, one or more semantic models or rules can be utilized in determining whether to modify a given speech hypothesis of “juice” with an alternative speech hypothesis of “shoes.” For example, a semantic model or rule can process a transcription and / or multiple speech hypotheses (or one or more alternative speech hypotheses) and determine that “shoes” and “comfortable” are more semantically meaningful than “juice” and “comfortable.” Modification of the given speech hypothesis can be performed based on this semantic determination.

[0081] although Figures 5A-5C Described herein as utilizing a particular computing device 110 1-N , but it should be understood that this is for the sake of example and is not intended to be limiting. For example, computing device 110 1-N The present invention may be any combination of a mobile device, a stand-alone speaker device with a display, a stand-alone speaker device without a display, a home automation device, a vehicle system, a laptop, a desktop computer, a tablet computer, and / or any other computing device with which the user 101 is able to interact as indicated above. In addition, although the description herein is about storing and loading multiple speech hypotheses for a given word item only, the present invention may be any combination of a mobile device, a stand-alone speaker device with a display, a home automation device, a vehicle system, a laptop, a desktop computer, a tablet computer, and / or any other computing device with which the user 101 is able to interact as indicated above. Figures 5A-5C , but it should also be understood that this is for the sake of example and is not intended to be limiting. For example, it is possible to cross computing device 110 1-N A plurality of corresponding phonetic hypotheses for any combination of term hypotheses and / or transcription hypotheses are stored and / or loaded.

[0082] Now turn to Fig. 6A and Figure 6B , depicting various non-limiting examples of maintaining speech hypotheses across a conversation session of user 101. As shown, Fig. 6A and Figure 6B The computing device 1101 depicted in FIG. 1 is a computer having Figure 5A A standalone computing device having the same functionality as described. It is noteworthy that Fig. 6A depicting a conversational session with an automated assistant implemented by computing device 1101 (e.g., as indicated by 652A1), and Figure 6B Depicted is a subsequent conversation session (eg, as represented by 652B1) with an automated assistant implemented by computing device 1101. Despite the distinct conversation session with the automated assistant, multiple speech hypotheses can be maintained across the conversation session.

[0083] Specifically refer to Fig. 6A, assume that the computing device 1101 receives a spoken utterance 652A2 of "Hey assistant, draftan email to Scott Smith that says" from the user 101 to initiate a conversation session as indicated by 652B1 for drafting a transcription of an email associated with an email application. The automated assistant can cause the computing device 1101 (and / or a remote system) to process audio data corresponding to the spoken utterance 652A2 using an ASR model to generate a plurality of speech hypotheses and corresponding values ​​associated with each of the plurality of speech hypotheses. The plurality of speech hypotheses can include, for example, transcription hypotheses and / or term hypotheses. For example, assume that the automated assistant is highly confident in term hypotheses corresponding to "the", "attorneys", "at", "the", "meeting", "understand", and "networks" based on values ​​generated using the ASR model. However, further assume that the automated assistant is not confident in term hypotheses corresponding to "neural" based on values ​​generated using the ASR model. For example, table 620A includes a table of term hypotheses corresponding to "neural" for spoken utterance 652A2 and corresponding confidence levels associated with each term hypothesis as values ​​generated using an ASR model. Table 620A includes a first term hypothesis of "rural" associated with a first probability of 0.45, a second term hypothesis of "neural" associated with a second probability of 0.40, and a third term hypothesis of "mural" associated with a third probability of 0.15. Although particular term hypotheses are described in table 620A as being associated with particular probabilities, it should be understood that table 620A is provided for purposes of example and is not intended to be limiting. Further, assume that the automated assistant determines, based on the corresponding values ​​associated with each of the plurality of term hypotheses, that the portion of spoken utterance 652A2 that includes "neural" corresponds to the term hypothesis "rural" as indicated by table 620A.

[0084] Assume further that the automated assistant generates and audibly renders via a speaker of computing device 1101 a spoken response 654A1 of “Okay, drafting an email to Scott Smith,” and generates a transcription of “the attorneys at the meeting understood rural networks” as indicated by 654A2 (which can be rendered and interacted with via display 1801 of computing device 1101). In this example, a transcription including an incorrectly recognized term hypothesis of “rural” can be generated based on a selected transcription hypothesis (e.g., “the attorneys at the meeting understood rural networks”). A plurality of speech hypotheses (including term hypotheses as shown in table 620A and / or transcription hypotheses based thereon) can be stored in a memory accessible by computing device 1101 as indicated by 654A3. In some implementations, the plurality of speech hypotheses can be stored in the memory only in response to determining that one or more conditions are satisfied. The one or more conditions can include, for example, determining that multiple of the respective confidence levels for multiple of the multiple speech hypotheses are within a threshold range of each other (e.g., a respective confidence level of 0.45 for “rural” is within 0.08 of a respective confidence level of 0.40 for “neural”), determining that the respective confidence levels do not satisfy a threshold confidence level (e.g., the respective confidence levels for both “rural” and “neural” fail to satisfy a threshold of 0.51), determining that multiple speech hypotheses were generated for the spoken utterance (e.g., the transcription itself was generated), determining that a software application associated with the transcription is deactivated (e.g., closed or working in the background of the computing device 1101), and / or determining that a conversational session between a user 101 of the computing device 1101 and the automated assistant is terminated. Assume further that the conversational session between the user 101 and the automated assistant implemented by the computing device 1101 is terminated without sending an email.

[0085] Specifically refer to Figure 6B, assume that computing device 110 receives an additional spoken utterance 652A2 from user 101 of “Hey assistant, add to my email to Scott Smith that I'm surprised they were able to grasp machine learning technology” to initiate a subsequent conversation session as indicated by 652B1. In some implementations, Figure 6B The subsequent conversation session is timed with Fig. 6A For example, Figure 6B Subsequent conversations can take place at Fig. 6A hours, days, or weeks after the initial conversation session. In some additional or alternative implementations, Figure 6B Subsequent conversations can follow in time Fig. 6A For example, Figure 6B A subsequent conversation session can be followed by Fig. 6A In other words, the conversation session in this example can be characterized based on user input and responses thereto, and the subsequent conversation session in this example can be characterized based on subsequent user input and subsequent responses thereto. As another example, each time a specific word or phrase (e.g., detected using a hot word model) is utilized to invoke the automated assistant, the interactions following the specific word or phrase can be characterized as a conversation session.

[0086] In response to receiving the additional spoken utterance 652B2, the automated assistant can cause the computing device 1101 (and / or a remote system) to process the audio data corresponding to the additional spoken utterance 652B2 using the ASR model to generate a plurality of speech hypotheses and a corresponding value associated with each of the plurality of speech hypotheses. Fig. 6A As described, the plurality of speech hypotheses can include, for example, transcription hypotheses and / or term hypotheses. In this example, assume that the additional spoken utterance 652B2 is correctly identified based on the value generated using the ASR model when processing the additional spoken utterance 652B2. Further assume that the automated assistant generates and audibly renders a spoken response 654B1 of “Okay, adding to the draft email to Scott Smith” via a speaker of the computing device 1101. In this example, the automated assistant can determine that the additional spoken utterance 652B1 is directed to Fig. 6AAs a result, the automated assistant is able to load multiple speech hypotheses for the transcription as indicated by 654B2. For example, table 620B includes a table of lexical hypotheses. Notably, Figure 6B Table 620B with Fig. 6A 620A is similar to , but omits the term hypothesis that "rural" was selected as corresponding to "neural" in the original spoken utterance 652A2. In other words, Figure 6B Table 620B includes only the alternative term hypotheses that were not selected for inclusion in the transcription.

[0087] In addition, the automated assistant can update the transcription for the email as indicated by 654B3 to include additional portions that were determined and correctly identified based on processing the additional spoken utterance 652B2 using the ASR model, thereby producing a transcription of “Theattorneys at the meeting understood rural networks. I'm surprised they wereable to grasp machine learning technology.” In some implementations, the automated assistant can modify portions of the transcription based on multiple speech hypotheses loaded at computing device 1101 and / or based on the additional spoken utterance 652B2. For example, the automated assistant can modify a given speech hypothesis for “rural” with an alternative speech hypothesis (e.g., from table 620B) in response to updating the transcription as indicated by 654B3. In some versions of those implementations, one or more semantic models or rules can be utilized in determining whether to modify a given speech hypothesis of “rural” with an alternative speech hypothesis of “neural”, thereby producing a transcription corresponding to spoken utterance 652A2 and additional spoken utterance 652B2, regardless of an initial misrecognition in an initial conversation session. For example, a semantic model or rule can process the transcription and / or multiple speech hypotheses (or one or more alternative speech hypotheses) and determine that “neural networks” and “machine learning technology” are more semantically meaningful than “rural networks” and “machine learning technology”. Modifications to a given speech hypothesis can be performed based on this semantic determination and even without the knowledge of user 101, as the user is only engaged in voice communications with the automated assistant.

[0088] although Fig. 6A and Figure 6BIt is described herein as utilizing an automated assistant to maintain speech hypotheses in a conversational session, but it should be understood that this is for the sake of example and is not intended to be limiting. For example, and now with reference to Fig. 7A and Figure 7B , depicts various non-limiting examples of maintaining speech hypotheses across a conversation session of user 101. As shown, Fig. 7A and Figure 7B The computing device 110 depicted in N Is about Figure 5B and Figure 5C A mobile device having the same functionality as described. It is noteworthy that Fig. 7A and Figure 7B Describe as about Fig. 6A and Figure 6B The same user interactions (eg, the same draft email to Scott Smith as indicated by 722 ) are described across conversation sessions (eg, as indicated by 752A1 and 752A2 ).

[0089] However, and Fig. 6A and Figure 6B In contrast, when the email software is applied on the computing device 110 N When the computing device 110 is active, N The voice input interface element 185 may be selected in response to detecting input for providing a spoken utterance and without invoking an automated assistant (eg, selecting the voice input interface element 185 N ) receives spoken utterance 752A2 of user 101. For example, and about Fig. 7A , assuming that computing device 110 N Spoken utterance 752A2 is received, and user 101 then deactivates the email application without sending a transcription of the email to Scott Smith. Multiple speech hypotheses (e.g., from Fig. 6A Table 620A) is stored in a table that can be accessed by computing device 110 N However, when the user subsequently activates the email application to begin a subsequent conversation session as indicated by 752B1, the plurality of speech hypotheses can be loaded at the email application as indicated by 752B2 along with a transcription of the draft email as indicated by 752B3. Assume further that at computing device 110 N Additional user input 752B4 (e.g., typed and / or spoken) is received, processed, and incorporated into the transcription. In some implementations, alternative speech hypotheses can be presented for selection by user 101 based on values ​​associated with the alternative speech hypotheses. For example, and as Figure 7BAs shown, a first selectable graphical element 779B1 for "rural" and a second selectable graphical element 779B2 for "neural" can be presented for selection by the user. The selection may be made in response to receiving additional user input 752B4, in response to receiving user input directed to a virtual keyboard, in response to user input directed to a portion of the transcription that includes a potential misrecognition (e.g., "rural"), and / or in response to the text (e.g., regarding Figure 1 These graphical elements are presented to the user through other user interactions described by the modification engine 152 of the embodiment of the present invention. In other implementations, the transcription can be automatically modified based on the alternative speech hypothesis in response to receiving additional user input (e.g., as described in relation to Figure 6B described).

[0090] although Figures 6A-7B Described herein as utilizing a particular computing device 110 1-N , but it should be understood that this is for the sake of example and is not intended to be limiting. For example, computing device 110 1-N The present invention may be any combination of a mobile device, a stand-alone speaker device with a display, a stand-alone speaker device without a display, a home automation device, a vehicle system, a laptop, a desktop computer, a tablet computer, and / or any other computing device with which the user 101 is able to interact as indicated above. In addition, although the description herein is about storing and loading only multiple speech hypotheses for a given word item, the present invention may be any combination of a mobile device, a stand-alone speaker device with a display, a home automation device, a vehicle system, a laptop, a desktop computer, a tablet computer, and / or any other computing device with which the user 101 is able to interact as indicated above. Figures 6A-7B , but it should also be understood that this is for the sake of example and is not intended to be limiting. For example, it is possible to cross computing device 110 1-N A plurality of corresponding phonetic hypotheses for any combination of term hypotheses and / or transcription hypotheses are stored and / or loaded.

[0091] Now turn to Figure 8 , depicts a block diagram of an example computing device 810 that can optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a cloud-based automated assistant component, and / or other components can include one or more components of the example computing device 810.

[0092] The computing device 810 typically includes at least one processor 814 that communicates with a number of peripheral devices via a bus subsystem 812. These peripheral devices may include a storage subsystem 824 (including, for example, a memory subsystem 825 and a file storage subsystem 826), a user interface output device 820, a user interface input device 822, and a network interface subsystem 816. The input and output devices allow a user to interact with the computing device 810. The network interface subsystem 816 provides an interface to an external network and couples to corresponding interface devices in other computing devices.

[0093] The user interface input devices 822 may include a keyboard, a pointing device such as a mouse, trackball, touchpad or graphic tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 810 or onto a communication network.

[0094] The user interface output device 820 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, for example, via an audio output device. In general, the use of the term "output device" is intended to include devices and methods for outputting information from the computing device 810 to a user or to another machine or computing device of all possible types.

[0095] The storage subsystem 824 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 824 may include a computer program for executing selected aspects of the methods disclosed herein and a computer program for implementing the methods disclosed herein. Figure 1 The logic of the various components is depicted in .

[0096] These software modules are typically run by the processor 814 alone or in combination with other processors. The memory 825 used in the storage subsystem 824 can include many memories, including a main random access memory (RAM) 830 for storing instructions and data during program execution and a read-only memory (ROM) 832 in which fixed instructions are stored. The file storage subsystem 826 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and related removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations can be stored in the storage subsystem 824 by the file storage subsystem 826, or in other machines that can be accessed by the processor 814.

[0097] The bus subsystem 812 provides a mechanism for the various components and subsystems of the computing device 810 to communicate with each other as intended. Although the bus subsystem 812 is shown schematically as a single bus, alternative implementations of the bus subsystem 812 may use multiple busses.

[0098] The computing device 810 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 8 The description of the computing device 810 depicted in FIG. 8 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of the computing device 810 may have more Figure 8 The computing devices depicted may have greater or fewer components.

[0099] In situations where the systems described herein collect or otherwise monitor personal information about a user or can utilize personal and / or monitored information, the user may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about a user's social network, social actions or activities, occupation, user's preferences, or the user's current geographic location) or to control whether and / or how to receive content that may be more relevant to the user from a content server. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be generalized (such as to a city, zip code, or state level) in the case of obtaining geographic location information so that the user's specific geographic location cannot be determined. Therefore, the user may control how information is collected and / or used about the user.

[0100] In some implementations, a method implemented by one or more processors is provided, and the method includes: receiving audio data corresponding to a spoken utterance of a user via one or more microphones of a computing device of the user; processing the audio data corresponding to the spoken utterance using an automatic speech recognition (ASR) model to generate a plurality of speech hypotheses for one or more portions of the spoken utterance based on values ​​generated using the ASR model; selecting a given speech hypothesis from among the plurality of speech hypotheses, the given speech hypothesis being predicted to correspond to one or more of the portions of the spoken utterance based on the values; causing the given speech hypothesis to be incorporated as part of a transcription associated with a software application accessible by at least the computing device, and the transcription to be visually rendered at a user interface of the computing device of the user; storing the plurality of speech hypotheses in a memory accessible by at least the computing device; and transmitting the plurality of speech hypotheses. Transmitting the plurality of speech hypotheses causes the plurality of speech hypotheses to be loaded at an additional computing device of the user when the transcription associated with the software application is accessed at the additional computing device, the additional computing device being supplemental to the computing device.

[0101] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0102] In some implementations, the method may further include determining, for one or more of the portions of the spoken utterance, a corresponding confidence level associated with each of the plurality of speech hypotheses based on the value generated using the ASR model. Selecting the given speech hypothesis predicted to correspond to one or more of the portions of the spoken utterance from among the plurality of speech hypotheses may be based on the corresponding confidence level associated with each of the plurality of speech hypotheses.

[0103] In some versions of those implementations, storing the plurality of speech hypotheses in the memory accessible by at least the computing device can be in response to determining that, for one or more of the portions of the spoken utterance, the corresponding confidence levels of two or more of the plurality of speech hypotheses are within a threshold range of confidence levels.

[0104] In some additional or alternative versions of those implementations, storing the plurality of speech hypotheses in the memory accessible by at least the computing device can be in response to determining that, for the portion of the spoken utterance, the corresponding confidence level for each of the plurality of speech hypotheses fails to satisfy a threshold confidence level. In some further versions of those implementations, the method can further include graphically defining the portion of the transcription that includes the portion of the spoken utterance corresponding to the given speech hypothesis. Graphically defining the portion of the transcription can be in response to determining that, for the portion of the spoken utterance, the corresponding confidence level for each of the plurality of speech hypotheses fails to satisfy a threshold confidence level. In still further versions of those implementations, graphically delimiting the portion of the transcription that includes the portion of the spoken utterance corresponding to the given speech hypothesis includes one or more of: highlighting the portion of the transcription, underlining the portion of the transcription, italicizing the portion of the transcription, or providing a selectable graphical element that, when selected, causes one or more additional speech hypotheses from among the plurality of speech hypotheses and in addition to the given speech hypothesis to be visually rendered along with the portion of the transcription.

[0105] In some additional or alternative versions of those implementations, storing the plurality of speech hypotheses in the memory accessible by at least the computing device may include storing each of the plurality of speech hypotheses in the memory accessible by at least the computing device in association with the corresponding confidence level.

[0106] In some additional or alternative versions of those implementations, the method may further include: receiving additional audio data corresponding to additional spoken utterances of the user via one or more additional microphones of the additional computing device; processing the additional audio data corresponding to the additional spoken utterances using the ASR model to generate a plurality of additional speech hypotheses for additional portions of the additional spoken utterances based on additional values ​​generated using the ASR model; and modifying, for the portions of the spoken utterances, the given speech hypothesis incorporated as the portion of the transcription based on the plurality of additional speech hypotheses. In some further versions of those implementations, modifying the given speech hypothesis incorporated as the portion of the transcription based on the plurality of additional speech hypotheses may include selecting an alternative speech hypothesis from among the plurality of speech hypotheses based on the corresponding confidence level associated with each of the plurality of speech hypotheses and based on the plurality of additional speech hypotheses; and replacing the given speech hypothesis with the alternative speech hypothesis for one or more of the portions of the spoken utterances in the transcription. In still further versions of those implementations, the method may further include: selecting an additional given speech hypothesis from one or more of the additional speech hypotheses, the additional given speech hypothesis being predicted to correspond to one or more of the additional portions of the additional spoken utterance; and causing the additional given speech hypothesis to be incorporated as an additional portion of the transcription, wherein the additional portion of the transcription positionally follows the portion of the transcription.

[0107] In some implementations, the method may further include generating a finite state decoding graph based on the values ​​generated using the ASR model, the finite state decoding graph including a corresponding confidence level associated with each of the plurality of speech hypotheses. Selecting the given speech hypothesis from among the plurality of speech hypotheses is based on the finite state decoding graph.

[0108] In some versions of those implementations storing the plurality of speech hypotheses in the memory accessible by at least the computing device may include storing the finite state decoding graph in the memory accessible by at least the computing device.

[0109] In some additional or alternative versions of those implementations, the method may further include: receiving additional audio data corresponding to additional spoken utterances of the user via one or more additional microphones of the additional computing device; processing the additional audio data corresponding to the additional spoken utterances using the ASR model to generate one or more additional speech hypotheses based on additional values ​​generated using the ASR model; and, for one or more of the portions of the spoken utterance, modifying the given speech hypothesis incorporated as the portion of the transcription based on one or more of the additional speech hypotheses.

[0110] In some further versions of those implementations, modifying the given speech hypothesis incorporated as part of the transcription based on one or more of the additional speech hypotheses can include: adapting the finite state decoding graph to select an alternative speech hypothesis from among the multiple speech hypotheses based on one or more of the additional speech hypotheses; and replacing the given speech hypothesis with the alternative speech hypothesis for one or more of the portions of the spoken utterance in the transcription.

[0111] In some additional or alternative versions of those further implementations, the method may further include: selecting an additional given speech hypothesis from one or more of the additional speech hypotheses, the additional given speech hypothesis being predicted to correspond to an additional portion of the additional spoken utterance; and causing the additional given speech hypothesis to be incorporated as an additional portion of the transcription, wherein the additional portion of the transcription positionally follows the portion of the transcription.

[0112] In some additional or alternative versions of those further implementations, the method may further include causing the computing device to visually render one or more graphical elements indicating that the given speech hypothesis is modified for one or more of the portions of the spoken utterance.

[0113] In some implementations, the software application can include one or more of: an automated assistant application, a text messaging application, an email application, a note taking application, a dictation application, a reminder application, a document editing application, a presentation application, a spreadsheet application, a social media application, a browser application, or a calendar application.

[0114] In some implementations, transmitting the plurality of speech hypotheses may include, subsequent to causing the given speech hypothesis to be incorporated as the portion of the transcription associated with the software application: determining that the transcription associated with the software application is accessed at the additional computing device; and for one or more of the portions of the spoken utterance, causing the plurality of speech hypotheses to be transmitted from the memory accessible by at least the computing device to the additional computing device. In some versions of those implementations, the software application may be associated with a third-party system, and causing the plurality of speech hypotheses to be transmitted to the additional computing device may include transmitting the plurality of speech hypotheses to the third-party system.

[0115] In some implementations, memory accessible by at least a computing device may include one or more of: on-device memory of the computing device or remote memory of a remote system.

[0116] In some implementations, a method implemented by one or more processors is provided, and the method includes: receiving audio data corresponding to a spoken utterance of a user via one or more microphones of a computing device of the user; processing the audio data corresponding to the spoken utterance using an automatic speech recognition (ASR) model to generate a plurality of speech hypotheses for one or more parts of the spoken utterance based on values ​​generated using the ASR model; selecting a given speech hypothesis from among the plurality of speech hypotheses, the given speech hypothesis being predicted to correspond to one or more of the parts of the spoken utterance based on the values; causing the given speech hypothesis to be incorporated as part of a transcription, the transcription being visually rendered at a user interface of the computing device of the user; determining that the spoken utterance is complete; in response to determining that the spoken utterance is complete, is complete, storing one or more alternative speech hypotheses in a memory accessible by the computing device, the one or more alternative speech hypotheses comprising a subset of the plurality of speech hypotheses excluding at least the given speech hypothesis; receiving additional audio data corresponding to additional spoken utterances of the user via one or more of the microphones of the computing device; in response to receiving the additional audio data, loading one or more of the alternative speech hypotheses from the memory accessible by the computing device; causing the additional given speech hypotheses to be incorporated as additional portions of the transcription; and modifying the portion of the transcription predicted to correspond to one or more of the portions of the spoken utterance based on the additional given speech hypothesis to include a given alternative speech hypothesis from among the one or more alternative speech hypotheses. The additional given speech hypothesis is selected from among the one or more additional speech hypotheses predicted to correspond to one or more additional portions of the additional spoken utterance

[0117] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0118] In some implementations, the method may further include: processing the additional audio data corresponding to the additional spoken utterance using the ASR model to generate one or more of the additional speech hypotheses for one or more of the additional parts of the additional spoken utterance based on additional values ​​generated using the ASR model; and selecting the additional given speech hypothesis from one or more of the additional speech hypotheses, the additional given speech hypothesis being predicted to correspond to one or more of the additional parts of the additional spoken utterance.

[0119] In some implementations, a method implemented by one or more processors is provided, and the method includes: receiving audio data corresponding to a spoken utterance of a user via one or more microphones of a computing device of the user; processing the audio data corresponding to the spoken utterance using an automatic speech recognition (ASR) model to generate a plurality of speech hypotheses for one or more portions of the spoken utterance based on values ​​generated using the ASR model; selecting a given speech hypothesis from among the plurality of speech hypotheses, the given speech hypothesis being predicted to correspond to one or more of the portions of the spoken utterance based on the values; causing the given speech hypothesis to be incorporated as part of a transcription, the transcription being associated with a software application accessible by at least the computing device, and the transcription being visually rendered at a user interface of the computing device of the user; and storing the plurality of speech hypotheses in a memory accessible by at least the computing device. Storing the plurality of speech hypotheses in a memory accessible by at least the computing device is in response to the software application being deactivated and subsequently activated at the computing device of the user, causing the software application to load the plurality of speech hypotheses.

[0120] In addition, some implementations include one or more processors (e.g., central processing units (CPUs), graphics processing units (GPUs), and / or tensor processing units (TPUs)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memories, and wherein the instructions are configured to cause the execution of any of the foregoing methods. Some implementations also include one or more non-transitory computer-readable storage media storing computer instructions that can be executed by one or more processors to perform any of the foregoing methods. Some implementations also include a computer program product that includes instructions that can be executed by one or more processors to perform any of the foregoing methods.

[0121] It should be appreciated that all combinations of the foregoing concepts and the additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

Claims

1. A method implemented by one or more processors, the method comprising: receiving, via one or more microphones of a computing device of a user, audio data corresponding to spoken utterances of the user; processing the audio data corresponding to the spoken utterance using an automatic speech recognition (ASR) model to generate a plurality of speech hypotheses for one or more portions of the spoken utterance based on values ​​generated using the ASR model; selecting a given speech hypothesis from among the plurality of speech hypotheses, the given speech hypothesis predicted to correspond to one or more of the portions of the spoken utterance based on the value; causing the given speech hypothesis to be incorporated as part of a transcription, the transcription being associated with a software application accessible by at least the computing device, and the transcription to be visually rendered at a user interface of the computing device of the user; as well as storing the plurality of speech hypotheses in a memory accessible by at least the computing device, Wherein storing the plurality of speech hypotheses in a memory accessible by at least the computing device is in response to the software application being deactivated at the computing device.

2. The method according to claim 1, further comprising: determining, for one or more of the portions of the spoken utterance, a respective confidence level associated with each of the plurality of speech hypotheses based on the values ​​generated using the ASR model, wherein selecting the given speech hypothesis predicted to correspond to one or more of the portions of the spoken utterance from among the plurality of speech hypotheses is based on the respective confidence level associated with each of the plurality of speech hypotheses.

3. The method according to claim 2, wherein: Storing the plurality of speech hypotheses in the memory accessible by at least the computing device is further responsive to determining that, for one or more of the portions of the spoken utterance, the corresponding confidence levels of two or more of the plurality of speech hypotheses are within a threshold range of confidence levels.

4. The method according to claim 2, wherein: Storing the plurality of speech hypotheses in the memory accessible by at least the computing device is further responsive to determining that, for one or more of the portions of the spoken utterance, the corresponding confidence level for each of the plurality of speech hypotheses fails to satisfy a threshold confidence level.

5. The method according to claim 2, wherein: Storing the plurality of speech hypotheses in the memory accessible by at least the computing device includes storing each of the plurality of speech hypotheses in the memory accessible by at least the computing device in association with the corresponding confidence level.

6. The method according to claim 1, further comprising: Graphically defining the portion of the transcription that includes one or more of the portions of the spoken utterance corresponding to the given speech hypotheses, wherein graphically defining the portion of the transcription is responsive to determining that the corresponding confidence level for each of the plurality of speech hypotheses fails to satisfy a threshold confidence level for one or more of the portions of the spoken utterance.

7. The method according to claim 6, wherein: Graphically delimiting the portion of the transcription that includes one or more of the portions of the spoken utterance corresponding to the given speech hypothesis includes one or more of: highlighting the portion of the transcription, underlining the portion of the transcription, italicizing the portion of the transcription, and providing a selectable graphical element that, when selected, causes one or more alternative speech hypotheses from among the plurality of speech hypotheses and other than the given speech hypothesis to be visually rendered along with the portion of the transcription.

8. The method according to claim 1, further comprising: In response to the software application being activated at the computing device: loading the plurality of speech hypotheses; receiving, via one or more additional microphones of the additional computing device, additional audio data corresponding to additional spoken utterances of the user; processing the additional audio data corresponding to the additional spoken utterance using the ASR model to generate one or more additional speech hypotheses for one or more additional portions of the additional spoken utterance based on additional values ​​generated using the ASR model; as well as For one or more of the portions of the spoken utterance, the given speech hypothesis incorporated as the portion of the transcription is modified based on one or more of the additional speech hypotheses.

9. The method according to claim 8, wherein: Modifying the given speech hypothesis incorporated as part of the transcription based on one or more of the additional speech hypotheses comprises: selecting an alternative speech hypothesis from among the plurality of speech hypotheses based on the respective confidence levels associated with each of the plurality of speech hypotheses and based on one or more of the additional speech hypotheses; and The given speech hypothesis is replaced with the alternative speech hypothesis for one or more of the portions of the spoken utterance in the transcription.

10. The method according to claim 9, further comprising: selecting an additional given speech hypothesis from among one or more of the additional speech hypotheses, the additional given speech hypothesis being predicted to correspond to one or more of the additional portions of the additional spoken utterance; and The additional given speech hypothesis is caused to be incorporated as an additional portion of the transcription, wherein the additional portion of the transcription positionally follows the portion of the transcription.

11. The method according to claim 1, further comprising: generating a finite state decoding graph based on the values ​​generated using the ASR model, the finite state decoding graph comprising a respective confidence level associated with each of the plurality of speech hypotheses, Wherein, selecting the given speech hypothesis from among the plurality of speech hypotheses is based on the finite state decoding graph.

12. The method according to claim 11, wherein: Storing the plurality of speech hypotheses in the memory accessible by at least the computing device includes storing the finite state decoding graph in the memory accessible by at least the computing device.

13. The method according to claim 11, further comprising: In response to the software application being activated at the computing device: loading the plurality of speech hypotheses; receiving, via one or more additional microphones of the additional computing device, additional audio data corresponding to additional spoken utterances of the user; processing the additional audio data corresponding to the additional spoken utterance using the ASR model to generate one or more additional speech hypotheses based on additional values ​​generated using the ASR model; as well as For one or more of the portions of the spoken utterance, the given speech hypothesis incorporated as the portion of the transcription is modified based on one or more of the additional speech hypotheses.

14. The method according to claim 13, wherein: Modifying the given speech hypothesis incorporated as part of the transcription based on one or more of the additional speech hypotheses comprises: adapting the finite state decoding graph to select an alternative speech hypothesis from among the plurality of speech hypotheses based on one or more of the additional speech hypotheses; and The given speech hypothesis is replaced with the alternative speech hypothesis for one or more of the portions of the spoken utterance in the transcription.

15. The method according to claim 13, further comprising: selecting an additional given speech hypothesis from among one or more of the additional speech hypotheses, the additional given speech hypothesis being predicted to correspond to an additional portion of the additional spoken utterance; as well as The additional given speech hypothesis is caused to be incorporated as an additional portion of the transcription, wherein the additional portion of the transcription positionally follows the portion of the transcription.

16. The method according to claim 13, further comprising: The computing device is caused to visually render one or more graphical elements indicating that, for one or more of the portions of the spoken utterance, the given speech hypothesis is modified.

17. The method according to claim 1, wherein: The software applications include one or more of: an automated assistant application, a text messaging application, an email application, a note taking application, a dictation application, a reminder application, a document editing application, a presentation application, a spreadsheet application, a social media application, a browser application, and a calendar application.

18. The method according to claim 1, wherein: The memory accessible by at least the computing device includes one or more of: on-device memory of the computing device and remote memory of a remote system.

19. A system comprising: at least one processor; as well as A memory storing instructions which, when executed, cause the at least one processor to be operable to perform a method according to any one of claims 1 to 18.

20. A non-transitory computer-readable storage medium storing instructions which, when executed by at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 18.

Images