Distributed ASR error identification and correction

Abstract

Implementations described herein identify and correct automatic speech recognition (ASR) misrecognitions. For example, on-device processor(s) of a client device may generate predicted text segments predicted to correspond to a voice utterance of a user of the client device and may receive further input to modify the predicted text segments into alternative text segments. Furthermore, the on-device processor(s) may store these text segments as candidate correction pairs in on-device storage and transmit the candidate correction pairs to a remote system. Furthermore, the remote processor(s) of the remote system may determine that the candidate correction pairs are actual correction pairs and cause the client device to generate an update of a global ASR model of the candidate correction pairs. Furthermore, the remote processor(s) may distribute the global ASR model to the client device and / or additional client devices.

Description

[Background technology]

[0001] Voice-based user interfaces are increasingly being used to control computers and other electronic devices. Voice-based user interfaces have evolved from early, basic interfaces that could only understand simple, direct commands to more advanced interfaces that respond to natural language requests, understand context, and manage two-way dialogue or conversations with the user. Many voice-based user interfaces perform speech recognition (e.g., using an automatic speech recognition (ASR) model) on a spoken utterance to generate corresponding text, perform a semantic analysis of that text to determine the meaning of the spoken utterance, and perform one or more actions based on the identified meaning.

[0002] Although speech recognition performance continues to improve, inaccurate speech recognition can still occur in many situations. As a non-limiting example, inaccurate speech recognition can occur for new terms and / or relatively infrequent (or non-existent) terms in the training corpus on which the ASR model is trained. In an attempt to effectively recognize new terms and / or infrequent terms, techniques have been proposed that generate additional speech recognition hypotheses in addition to an initial hypothesis (or multiple initial hypotheses) and consider the additional speech recognition hypotheses as candidates for speech recognition. However, such techniques require additional post-processing and may still not lead to effective recognition of many terms in many situations, such as when the initial hypothesis / hypotheses are too far off the mark and / or when the word list of the additional hypotheses does not include a particular term.

[0003] Furthermore, inaccurate speech recognition may be more pronounced when speech recognition is performed on-device (i.e., on the client device). This may be because, for example, on-device ASR models are less robust than cloud-based global ASR models, on-device memory and / or processor resources are more constrained than cloud-based resources, and / or word lists for generating additional hypotheses are more limited on-device. However, in many situations, technical considerations, which may include performance, data security, and network usage, make it beneficial to perform speech recognition on-device rather than remotely.

[0004] Although several techniques have been proposed for identifying and correcting inaccurate speech recognition, these techniques have drawbacks. For example, some techniques may monitor user interaction with a transcript that includes corresponding text generated based on processing audio data capturing the user's speech utterance. In these examples, the user interaction may correct the corresponding text to alternative text, and these corrections may be considered corrections. However, the user may change their mind when providing the speech utterance and subsequently correcting the corresponding text. Therefore, it may not be appropriate to assume that these corrections are corrections for inaccurate speech recognition. Furthermore, some techniques also consider phonetic similarity and / or Levenshtein edit distance between the corresponding text and these corrections. While these techniques may provide a better indication of whether these corrections are corrections for inaccurate speech recognition, they are generally applied on a per-user basis and cannot take into account whether other users make the same corrections. As a result, there is a need in the art for improved techniques for identifying inaccurate speech recognition and learning from it, and doing so in a privacy-conscious manner. Summary of the Invention [Means for solving the problem]

[0005]

[0003] Implementations described herein are directed to identifying and correcting automatic speech recognition (ASR) misrecognitions in a distributed manner. For example, on-device processor(s) of a client device may receive audio data capturing speech utterances of a user of the client device, process the audio data using an on-device ASR model (e.g., stored in on-device storage of the client device) to generate predictive text segments predicted to correspond to the speech utterances captured in the audio data, and the predictive text segments may be visually rendered for presentation to the user via a display of the client device. Additionally, the on-device processor(s) may receive further user interface input (e.g., additional speech utterances, touch input, typing, and / or other input) that modifies portions of the predictive text segments into alternative text segments. In response to receiving further user interface input modifying a portion of the predictive text segment into an alternative text segment, the on-device processor(s) may store the modified portion of the predictive text segment and the alternative text segment as corresponding candidate correction pairs (e.g., in on-device storage of the client device) and may store (and optionally associate with) the audio data processed to generate the predictive text segment (e.g., in on-device storage of the client device). Further, the on-device processor(s) may send the corresponding candidate correction pairs to a remote system without sending the audio data to the remote system. In some implementations, the on-device processor(s) may send the corresponding candidate correction pairs to a remote system only if the number of occurrences of the candidate correction pairs detected at the client device reaches a threshold amount. In additional or alternative implementations, the on-device processor(s) may send the corresponding candidate correction pairs to a remote system only if the candidate correction pairs are included in a list of candidate correction pairs. This list of candidate correction pairs is received from a remote system and includes candidate correction pairs that have been detected at another client device.Corresponding additional on-device processor(s) of multiple additional client devices (e.g., those added to the client device) may perform the same or similar operations as above and may send additional corresponding candidate correction pairs to the remote system in the same or similar circumstances as described above.

[0006] Further, the remote processor(s) of the remote system may determine whether a given corresponding candidate correction pair received from the client device and / or one or more of the multiple client devices is actually an actual correction pair (i.e., correcting a portion of the predicted text segment into an alternative text segment corrects an ASR misrecognition by the on-device ASR model(s) of the client device and the multiple additional client devices). For example, the remote processor(s) may determine that a given corresponding candidate correction pair is an actual candidate pair based on a threshold amount of occurrences of the corresponding candidate correction pair received from the client device and / or one or more of the multiple additional client devices. Additionally or alternatively, the remote processor(s) may determine that the corresponding candidate correction pair is an actual correction based on query activity associated with the multiple client devices. This determination may be based, for example, on query activity indicating that a threshold amount of queries including the alternative text segment have been submitted by users of the client device and / or one or more of the multiple additional client devices, and optionally, over a threshold period of time and / or within a particular geographic region (i.e., a spike in queries). In particular, any corrections to the query activity may also be transmitted to the remote system and from the client device that detects these corrections in the query activity.

[0007] In various embodiments, in response to determining that a given corresponding candidate correction pair corresponds to an actual correction, the remote processor(s) may identify a subset of the client devices from among the client device and / or one or more of the multiple additional client devices that provided the given corresponding candidate correction pair determined to correspond to the actual correction pair, and may update a global ASR model, which is a global-based counterpart of the corresponding ASR model(s) of the client device and / or one or more of the multiple additional client devices and is stored in a remote memory of the remote system, in a distributed manner. For example, the remote processor(s) may send instructions to update the global ASR model based on the given corresponding candidate correction pair in a non-distributed manner.

[0008] Further, the on-device processor(s) of a given client device identified for inclusion in the subset may generate corresponding updates to the global ASR model and for a given corresponding candidate correction pair. For example, the on-device processor(s) may retrieve (e.g., from) corresponding audio data that was previously stored in (e.g., from) given on-device storage of the given client device. Further, the on-device processor(s) may generate corresponding updates to the global ASR model based on processing the corresponding audio data and using the corresponding on-device ASR model. For example, the on-device processor(s) may process the corresponding audio data using the corresponding on-device ASR model to generate corresponding additional predictive text segments, and may generate corresponding updates based on comparing the corresponding additional predictive text segments with alternative text segments previously stored in association with the given corresponding candidate correction pair locally at the given client device. Additionally or alternatively, the on-device processor(s) may process the corresponding audio data using only a subset of the machine learning (ML) layers of the corresponding on-device ASR model (e.g., using the input layer(s) and / or encoding layer(s) of the corresponding on-device ASR model but not the combination layer(s), decoding layer(s), and / or output layer(s) of the corresponding on-device ASR model; using the input layer(s) and combination layer(s) of the corresponding on-device ASR model but not the output layer(s) of the corresponding on-device ASR model, etc.) to generate a corresponding representation of the corresponding audio data, and may perform various forward and / or backward passes over the subset of the ML layers of the corresponding on-device ASR model.In these examples, a subset of ML layers may be updated based on the processing, while other layers are fixed, and the difference between the subset of ML layers before and after the update (or the difference between one or more on-device weights associated with the subset of ML layers before and after the update) may correspond to a corresponding update to the global ASR model. In these examples, only a subset of ML layers may be updated in this manner to prioritize conserving computational resources on a given client device.

[0009] Further, the remote processor(s) may receive corresponding updates to the global ASR model from each of the client devices identified for inclusion in the subset. Further, the remote processor(s) may update the global ASR model based on the corresponding updates received from each of the client devices identified for inclusion in the subset. Further, the remote processor(s) may distribute the updated global ASR model to the multiple client devices (e.g., to the client devices identified for inclusion in the subset, the additional client devices, and / or further additional client devices), thereby causing the multiple client devices to update their corresponding on-device ASR models.

[0010] Thus, based on the techniques described herein, not only is a global ASR model updated in a distributed manner to address ASR misrecognition, but the ASR misrecognition itself is also identified in a distributed manner. In other words, the techniques described herein enable the identification of ASR misrecognition based on modifications across a population of client devices, rather than considering modifications at a single client device (e.g., using phonetic similarity, Levenshtein edit distance, etc.) to distinguish between scenarios in which a given user has changed their mind and / or where the modifications may not be due to an ASR misrecognition. Furthermore, the techniques described herein enable the correction of these ASR misrecognitions by having client devices across the population generate updates to the global ASR model. The global ASR model is then redistributed across the population of client devices to replace corresponding on-device ASR models, thereby eliminating and / or mitigating future occurrences of these ASR misrecognitions in on-device speech processing using the corresponding on-device ASR models. As a result, the corresponding on-device ASR model may be more robust to these ASR misrecognitions and may more easily recognize new terms and / or phrases that are newly added to the client device user's word list or that the corresponding on-device ASR model was previously trained to recognize.

[0011] For example, assume that a given user of a given client device provides the speech utterance, "I will miss the meeting because I tested positive for COVID." However, further assume that, when processing the audio data that captured the speech utterance, a corresponding on-device ASR model stored locally in the on-device storage of the given client device misrecognized "COVID" as "covet." Therefore, the given user may provide a further user interface input that corrects "covet" to "COVID," as intended by the given user when providing the speech utterance. In this case, part of the predicted text segment may be "covet," the alternative text segment may be "COVID," and the corresponding candidate correction pair [covet, COVID] may be stored in the on-device storage of the client device and, optionally, in association with the audio data that captured the speech utterance. Further, in this example, the corresponding candidate correction pair [covet,COVID] may be transmitted to the remote system, and optionally may be transmitted in response to determining that the number of occurrences of the candidate correction pair [covet,COVID] detected at a given client device has reached a threshold amount, and / or may be transmitted in response to determining that the candidate correction pair [covet,COVID] is included in a list of candidate correction pairs received from the remote system (e.g., indicating that another client device is also detecting occurrences of the candidate correction pair [covet,COVID]).

[0012] In this case, further assume that the remote system receives the corresponding candidate corrected pair [covet, COVID] from a given client device and receives multiple additional occurrences of the corresponding candidate corrected pair [covet, COVID] from multiple additional devices in addition to the given client device. In some of these examples, the remote system may determine that the corresponding candidate corrected pair [covet, COVID] is the actual candidate corrected pair based on the number of occurrences of the corresponding candidate corrected pair [covet, COVID] received from the given client device and / or multiple additional client devices reaching a threshold amount. In additional or alternative examples, the remote system may determine that the corresponding candidate corrected pair [covet, COVID] is the actual corrected pair based on the corresponding user of the given client device and / or multiple additional client devices submitting a query including the term “COVID” (and optionally based on the term “COVID” being modified in that query activity). Thus, the remote system may, in fact, determine that the corresponding candidate corrected pair [covet, COVID] is the actual corrected pair. As a result, the remote system can instruct the given client device, and any other client devices that provided the corresponding correction pair [covet,COVID], to generate corresponding updates to the global ASR model, thereby preventing and / or mitigating ASR misidentification of "COVID" as "covet."

[0013] This example further assumes that the given client device receives an instruction to generate a corresponding update of the global ASR model. The given client device may retrieve, from a given on-device storage of the given client device, at least a portion of the corresponding candidate correction pairs [covet, COVID] and audio data capturing a speech utterance stored in the on-device storage of the given client device (e.g., audio data capturing the speech utterance, "I'm going to miss the meeting because I tested positive for COVID"). In some of these examples, the given client device may process the audio data using the corresponding on-device ASR model to generate additional predictive text segments predicted to correspond to the speech utterance and compare the additional predictive text segments with the alternative text segments to generate a corresponding update of the global ASR model. In other words, the given client device may utilize the alternative text segments (e.g., the alternative text segment for "COVID") as a training signal when generating the corresponding update of the global ASR model. In additional or alternative examples, the given client device may process the audio data using a subset of the ML layers of the corresponding on-device ASR model to generate a corresponding representation of the audio data. Various forward and / or backward passes may then be performed across the subset of ML layers of the corresponding on-device ASR model, e.g., the subset of ML layers may be updated based on one or more gradients generated based on performing various forward and / or backward passes across the subset of ML layers of the corresponding on-device ASR model. Another client device that provided the corresponding candidate correction pair [covet,COVID] to the remote system may generate corresponding updates in the same or similar manner, and the remote system may update the global ASR model based on the corresponding updates received from multiple client devices.

[0014] In particular, in the above example, if further user input provided by a given user corrects a portion of the predictive text segment from “covet” to “an illness,” the corresponding candidate correction pair [covet, an illness] may be stored in on-device storage of the given client device and optionally stored in association with the audio data capturing the speech utterance, and the corresponding candidate correction pair [covet, an illness] may be transmitted to the server. However, in this case, the remote system is unlikely to receive many corresponding candidate correction pairs [covet, an illness] (e.g., compared to the corresponding candidate correction pair [covet, COVID]). This is because they represent cases where a given user has changed their mind (e.g., specifying “an illness” instead of “COVID”), and different users may change their minds in different ways. Therefore, only corresponding candidate correction pairs that actually indicate an ASR misrecognition should be identified by the remote system. Nevertheless, in various implementations, the techniques described herein may be utilized in conjunction with another technique for identifying and / or verifying ASR misrecognition (e.g., phonetic similarity, Levenshtein edit distance, etc.).

[0015] By implementing the techniques described herein, one or more technical advantages may be achieved. As one non-limiting example, the techniques described herein enable the rapid and efficient identification of actual ASR misrecognitions by considering potential ASR misrecognitions across a population of client devices. By considering potential ASR misrecognitions across a population of client devices rather than on a device-by-device basis, actual ASR misrecognitions can be quickly and efficiently distinguished from situations in which the users of each of the client devices simply change their minds. As another non-limiting example, the techniques described herein enable the ASR model to be updated in a manner that ensures the security of user data because audio data (e.g., underlying the ASR misrecognition) remains only on the client device, but can still be utilized in generating updates to the ASR model as described herein. As a result, the ASR model described herein prevents and / or mitigates the occurrence of ASR misrecognitions in a manner that maintains the security of user data.

[0016] The above description is provided as a summary of only some of the embodiments disclosed herein. These and other embodiments are described in more detail herein. [Brief explanation of the drawings]

[0017] [Figure 1A] 1 shows exemplary process flows demonstrating various aspects of the present disclosure, according to various embodiments. [Figure 1B] 1B shows a block diagram of an exemplary environment including various components of FIG. 1A and in which embodiments disclosed herein may be implemented. [Figure 2] 1 shows a flowchart illustrating an exemplary method for identifying locally corresponding candidate compensation pairs at a client device, according to various implementations. [Figure 3]A flowchart illustrating an exemplary method for remotely determining whether a given corresponding candidate correction pair corresponds to an actual correction pair at a remote system, according to various embodiments, and updating a global automatic speech recognition (ASR) model based on determining that the given corresponding candidate correction pair corresponds to an actual correction pair. [Figure 4] 1 shows a flowchart illustrating an exemplary method for generating corresponding updates of a global automatic speech recognition (ASR) model locally on a client device, according to various embodiments. [Figure 5] 1 shows a flowchart illustrating an exemplary method for remotely updating a global automatic speech recognition (ASR) model in a remote system, according to various embodiments. [Figure 6A] 1 illustrates various user interfaces showing exemplary user interactions with a client device, according to various implementations. [Figure 6B] 1 illustrates various user interfaces showing exemplary user interactions with a client device, according to various implementations. [Figure 7] 1 illustrates an exemplary architecture of a computing device, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0018] Referring now to FIG. 1A, an exemplary process flow illustrating various aspects of the present disclosure is shown. Client device 110 is shown in FIG. 1A and includes components enclosed within a box in FIG. 1A representing client device 110. An on-device automatic speech recognition (ASR) engine 122 may receive audio data 110A, which corresponds to a voice utterance of a user of client device 110 and is generated via one or more microphones of client device 110. Using an on-device ASR model 122A stored in on-device storage 111 (e.g., random access memory (RAM) and / or other type of volatile memory or storage device) of client device 110, on-device ASR engine 122 may process audio data 110A to generate one or more predicted outputs 122B. Furthermore, on-device ASR engine 122 may generate one or more predicted text segments 122C based on the one or more predicted outputs 122B.

[0019] For example, when the on-device ASR model 122A is an end-to-end speech recognition model, the on-device ASR engine 122 may use the on-device ASR model 122A to generate one or more predicted text segments 122C (e.g., one or more predicted outputs 122B may correspond to one or more predicted text segments 122C). For example, the on-device ASR model 122A may be an end-to-end model used to generate one or more predicted text segments 122C for each string (or for each separate token) as one or more predicted outputs 122B. One non-limiting example of such an end-to-end model used to generate one or more predicted text segments 122C for each character is a recurrent neural network transducer (RNN-T) model. The RNN-T model is a type of sequence-to-sequence model that does not use an attention mechanism. Unlike most sequence-to-sequence models, which typically require processing the entire input sequence (e.g., audio data waveforms, Mel-Frequency Cepstral Coefficients (MFCCs), or other representations of the audio data 110A) to generate one or more predicted text segments 122C, an RNN-T model can be used to process input samples sequentially and stream output symbols (e.g., letters of the alphabet). Also, for example, if the on-device ASR model 122A is not an end-to-end speech recognition model, the on-device ASR engine 122 may instead generate one or more predicted outputs 122B and / or other representations, such as one or more predicted phonemes, and may generate one or more predicted outputs 122C based on the one or more predicted outputs 122B. For example, using such a model, the one or more predicted phonemes and / or other representations may be utilized by the on-device ASR engine 122 to determine one or more predicted text segments 122C that match the one or more predicted phonemes. In doing so, the on-device ASR engine 122 may optionally use a decoding graph, a glossary, and / or other resource(s).

[0020] Further, rendering engine 124 can visually render one or more predictive text segments 122C for presentation to a user of client device 110 via a display, and optionally audibly render one or more predictive text segments 122C for presentation to a user of client device 110 via one or more speakers. In various implementations, in addition to the voice utterances initially provided by the user of client device 110, further input 110B can be received at client device 110. Further input 110B can include, for example, touch input and / or typing input on a display of client device 110 and / or additional voice utterances captured in additional audio data generated by one or more microphones of client device 110. In these implementations, correction engine 126 can analyze further input 110B to determine whether further input 110B modifies portions of one or more predictive text segments 122C into one or more alternative text segments 126A. Further, assuming that further input 110B corrects portions of one or more predictive text segments 122C into one or more alternative text segments 126A, correction pair engine 128 may generate correction pair 128A including at least portions of the corrected one or more predictive text segments 122C and one or more alternative text segments 126A. Further, correction pair engine 128 may store correction pair 128A in on-device storage 111 of client device 110, and audio data 110A may be stored in on-device storage 111 of client device 110 (and optionally stored in association with correction pair 128A). Further, correction pair engine 128 may cause correction pair 128A to be transmitted to remote system 160, without audio data 110A being transmitted to the remote system (and optionally in response to determining that the number of occurrences of correction pair 128 detected at client device 110 has reached a threshold amount).However, even though further input 110B modifies portions of one or more predicted text segments 122C into one or more alternative text segments 126A, correction pair 128A may not yet be considered an actual correction pair and may be referred to as a candidate correction pair.

[0021] In various implementations, correction identification engine 162 may store correction pair 128A and one or more additional correction pairs 180A received from one or more additional client devices 180 in correction pair database 162A, along with an indication of the client device 110 that provided correction pair 128A (e.g., a list of candidate correction pairs may be stored in correction pair database 162A and indexed based on the different text segments included in the correction pairs, which may be distributed to multiple client devices). Additionally, correction identification engine 162 may analyze correction pair 128A and one or more additional correction pairs 180A to determine whether any of the correction pairs (e.g., those stored in correction pair database 162A) correspond to an actual correction pair. In some variations of those implementations, correction identification engine 162 may determine that correction pair 128A is an actual correction pair based on the number of occurrences of correction pair 128A received from client device 110 and other client devices (e.g., from one or more of additional client devices 180) reaching a threshold amount. For example, as described in more detail with respect to FIG. 6A , assume that correction pair 128A includes a portion of a predicted text segment corresponding to “covet,” which is subsequently corrected to an alternative text segment of “COVID” (e.g., the correction pair [covet, COVID]). This example further assumes that thousands of correction pairs indicating “covet” being corrected to “COVID” have been identified by correction identification engine 162 (e.g., from among correction pair 128A and one or more correction pairs 180A), and that the number of occurrences of these thousands meets a threshold amount of occurrences for considering the correction pair [covet, COVID] as an actual correction pair. Thus, in this example, correction identification engine 162 may consider the correction pair [covet, COVID] to be an actual correction pair indicative of a common ASR misrecognition across a population of client devices.

[0022] In contrast, as described in more detail with respect to FIG. 6B , assume that correction pair 128A includes a portion of a predicted text segment corresponding to “covet,” which is subsequently corrected to an alternative text segment of “an illness” (e.g., the correction pair [covet, an illness]). In this example, further assume that only a few hundred correction pairs indicating “covet” being corrected to “an illness” (e.g., from among correction pair 128A and one or more correction pairs 180A) were identified by correction identification engine 162, but the number of occurrences of these few hundred did not meet the threshold amount of occurrences for considering the correction pair [covet, an illness] as an actual correction pair. Thus, in this example, correction identification engine 162 may not consider the correction pair [covet, an illness] as an actual correction pair indicative of a common ASR misrecognition across a population of client devices. Rather, in this example, the correction pair [covet, an illness] may simply indicate that a user has changed their mind about the speech utterance they originally provided, and this correction may be based on further input for reasons other than an ASR misrecognition.

[0023] In additional or alternative variations of these embodiments, correction identification engine 162 may operate in conjunction with query activity engine 164 to determine whether correction pair 128A is an actual correction pair based on query activity (e.g., accessible via query activity database 164A). For example, assume again that correction pair 128A includes a portion of a predicted text segment corresponding to “covet,” which has subsequently been modified to an alternative text segment of “COVID” (e.g., the corrected pair [covet,COVID]). In this example, further assume that thousands of queries including the alternative text segment of “COVID” have been received within a threshold period, and that these queries represent a spike in queries including “COVID.” Thus, in this example, correction identification engine 162 may utilize the spike in queries detected by query activity engine 164 to determine that the alternative text segment of “COVID” may be a newly introduced term to the population of users of client devices that provided the correction pair [covet,COVID] to remote system 160. In various embodiments, any client device (e.g., one or more of client device 110 and / or additional client devices 180) may transmit any corresponding candidate correction pairs determined based on queries submitted at each client device for inclusion in the subset.

[0024] In various implementations, assuming correction identification engine 162 determines that correction pair 128A indeed corresponds to an actual correction pair, client device identification engine 166 may identify a subset of client devices that provided occurrences of correction pair 128A. For example, client device identification engine 166 may access correction pair database 162A to identify any client devices (if any) that previously provided correction pair 128A for inclusion in the subset. In particular, client device identification engine 166 may make this identification when one or more client devices check in to remote system 160 in a distributed manner and periodically (e.g., once a day, once a week, etc.) to train. The subset of client devices that provided occurrences of correction pair 128A may be utilized by remote system 160 in updating global ASR model 168A, which is a remote-based counterpart of on-device ASR model 122A (and one or more corresponding on-device ASR models among additional client devices 180). The remote training engine 168 may send instructions to each of the client devices in the subset, causing each of the client devices to generate updates to the global ASR model 168A and for the correction pair 128A. For example, assume that the client device 110 is identified for inclusion in the subset based on providing an instance of the correction pair 128A, and further assume that the remote system 160 determines that the correction pair 128A corresponds to an actual correction pair. As a result, the remote training engine 168 may generate and send instructions to the client device 110 that cause the client device 110 to generate updates to the global ASR model 168A and for the correction pair 128A.

[0025] At client device 110, update engine 130 may receive instructions to generate updates to global ASR model 168A and for correction pair 128A. Update engine 130 may retrieve at least a portion of correction pair 128A and audio data 110A processed locally at the client device from on-device storage 111 of client device 110 to generate a portion of one or more text segments included in correction pair 128A. Further, update engine 130 may again process audio data 110A using on-device ASR model 122A and generate update 130A based on the processing of audio data 110A. Further, update engine 130 may transmit update 120A to remote system 160 for utilization by remote training engine 168 in updating global ASR model 168A. One or more of the additional client devices 180 similarly identified for inclusion in the subset may generate corresponding updates 180B in the same or similar manner and may transmit the corresponding updates 180B to the remote system 160 for use by the remote training engine 168 when updating the global ASR model 168A.

[0026] In some implementations, in generating update 130A, update engine 130 may process audio data 110A using on-device ASR model 122A to generate one or more additional predictive text segments in the same or similar manner as described above (e.g., with respect to on-device ASR engine 122). However, rather than the one or more additional predictive text segments being at least visually rendered for presentation to a user of a client device (e.g., via rendering engine 124, as described with respect to one or more predictive text segments), update engine 130 may compare a portion of the one or more additional predictive text segments generated based on subsequent processing of audio data 110A with one or more alternative text segments of correction pair 128A to generate gradients, for example, using supervised learning techniques. In some variations of those implementations, the gradients may correspond to update 130A being sent to remote system 160. In additional or alternative variations of these embodiments, the update engine 130 may update the on-device ASR model locally at the client device 110 and based on the gradient, and may utilize one or more updated on-device weights of the updated on-device ASR model 122A as update 130A.

[0027] In additional or alternative embodiments, in generating update 130A, update engine 130 may process audio data 110A using a subset of machine learning (ML) layers of on-device ASR model 122A to generate a representation of audio 110A, and may perform various forward and / or backward passes across the subset of ML layers of on-device ASR model 122A. The ML layers of on-device ASR model 122A may include, for example, one or more input layers, one or more combining layers, one or more encoding layers, one or more combining layers, one or more output layers, and / or other layers. Thus, in processing audio data using a subset of ML layers of on-device ASR model 122A, update engine 130 may generate only representations across one or more of the input layers or across one or more of the combining or encoding layers, without generating anything across one or more of the decoding or output layers. In these embodiments, a subset of ML layers may be updated based on processing while other layers are fixed (e.g., other ML layers of on-device ASR model 122A that were not used in processing audio data 110A are fixed), and the difference between the subset of ML layers before the update and the subset of ML layers after the update (or the difference between one or more on-device weights associated with the subset of ML layers before the update and the subset of ML layers after the update) may be used as update 130A.

[0028] In various implementations, in response to receiving update 130A from client device 110 and one or more corresponding updates 180A from one or more of additional client devices 180, remote training engine 162 may use these updates to update global ASR model 168A with respect to correction pair 128. Specifically, in updating global ASR model 168A (e.g., using backpropagation or another technique for updating the ASR model(s)), it updates one or more global weights of global speech recognition model 168A. Furthermore, as indicated by 170A, update distribution engine 170 can provide one or more of the updated global weights of the updated global ASR model and / or the updated global ASR model to client device 110 and / or one or more of additional client devices 180 in response to one or more conditions being met. The one or more conditions may include, for example, a threshold period and / or a threshold amount of updates of the global ASR model 168A since any updated weight(s) and / or updated global ASR model were last distributed, a measured improvement of the updated global ASR model, and / or the lapse of a threshold period of time since any updated weight(s) and / or updated global ASR model were last distributed, and / or other conditions. When one or more of the updated global weights and / or the updated global ASR model of the updated global ASR model are distributed to the client device 110, the client device 110 can replace the on-device ASR model 122A (or its on-device ASR model(s)) with the updated global ASR model (or its updated global weight(s)).

[0029] 1B, client device 110 is shown, in this embodiment, including on-device ASR engine 122 of FIG. 1A as part of (or in communication with) automated assistant client 140. Also, on-device ASR model 122A is shown interfacing with on-device ASR engine 122. Other components of client device 250 are not shown in FIG. 1B for simplicity. FIG. 1B illustrates an example of how on-device ASR engine 122 and on-device ASR model 122A can be used to generate predictive text segments used by automated assistant client 140 in performing various actions.

[0030] Client device 110 in FIG. 1B is shown with one or more microphones 151, one or more speakers 152, one or more camera(s) and / or other vision component(s) 153, and display(s) 154 (e.g., a touch-sensitive display). Client device 110 at least selectively executes automated assistant client 140, which, in the example of FIG. 1B, includes on-device ASR engine 122, on-device natural language understanding (NLU) engine 144, and on-device fulfillment engine 145. Automated assistant client 140 further includes audio capture engine 141 and visual capture engine 142. Automated assistant client 140 may include additional and / or alternative engines, such as a voice activity detector (VAD), an endpoint detector, a hotword detector, and / or other engine(s).

[0031] One or more cloud-based automated assistant components 191 can optionally be implemented on one or more computing systems (collectively referred to as "cloud" computing systems), which are communicatively connected to client device 110 via one or more local area networks and / or wide area networks (e.g., the Internet), generally designated as 190. Cloud-based automated assistant component 191 can be implemented, for example, via a high-performance server or cluster of high-performance servers. In various implementations, automated assistant client instance 140, and optionally through interaction with one or more cloud-based automated assistant components 191, can form what appears from the user's perspective as a logical instance of automated assistant 195, with which the user can engage in human-computer interaction (e.g., voice-based interaction, gesture-based interaction, and / or touch-based interaction).

[0032] Client device 110 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in a user's vehicle (e.g., an in-car communication system, an in-car entertainment system, an in-car navigation system), a standalone interactive speaker, a smart appliance such as a smart television (or a standard television with a networked dongle with automated assistant functionality), and / or a user wearable device that includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual reality or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0033] Vision component(s) 153 may take various forms, such as a monographic camera, a stereographic camera, a LIDAR component (or other laser-based component(s)), a radar component, etc. One or more vision components 153 may be used, for example, by visual capture engine 142 to capture vision frames (e.g., image frames, laser-based vision frames) of the environment in which client device 110 is deployed. In some implementations, such vision frame(s) may be utilized to determine whether a user is present near client device 110 and / or the distance of the user (e.g., the user's face) relative to the client device. Such determination(s) may be utilized, for example, in determining whether to activate on-device ML engine 122.

[0034] The voice capture engine 141 can be configured to capture user speech and / or other audio data captured via microphone(s) 151. As described herein, such audio data can be utilized (optionally after preprocessing) by the on-device ASR engine 122. For example, the on-device ASR engine 122 can utilize the on-device ASR model 122A to process the audio data capturing the speech utterance and generate predictive text segments predicted to correspond to the speech utterance. The on-device NLU engine 144 performs on-device natural language understanding on the predictive text segments to generate the NLU data. The on-device NLU engine 144 can optionally utilize one or more on-device NLU models (not shown in FIG. 1B for simplicity) in generating the NLU data. The NLU data can include, for example, intent(s) corresponding to the speech utterance and, optionally, parameter(s) (e.g., slot values) of the intent(s). Additionally, the on-device fulfillment engine 145 uses the NLU data to generate fulfillment data. The on-device fulfillment engine 145 can optionally utilize one or more on-device fulfillment models and / or rules (not shown in FIG. 1B for simplicity) in generating the fulfillment data. The fulfillment data can define local and / or remote responses (e.g., answers) to the voice utterance, interaction(s) with locally installed application(s) based on the voice utterance, command(s) to send to Internet of Things (IoT) device(s) (directly or via corresponding remote system(s)), and / or other resolution action(s) to take based on the voice utterance. The fulfillment data is then provided for local and / or remote performance / execution of the action(s) determined to resolve the voice utterance.Execution may include, for example, rendering local and / or remote responses (e.g., visually and / or audibly (optionally utilizing a text-to-speech module)), interacting with locally installed applications, sending command(s) to IoT device(s), and / or other action(s).

[0035] Display(s) 154 can be utilized to visually render the streaming predictive text segments generated based on the prediction output from the on-device ASR engine 122. Display(s) 154 can also be one of the user interface output component(s) on which the visual portion(s) of the response are rendered from the automated assistant client 140.

[0036] In some implementations, cloud-based automated assistant component(s) 191 may include a remote ASR engine 192 that performs speech recognition, a remote NLU engine 193 that performs natural language understanding, and / or a remote fulfillment engine 194 that generates fulfillment data. It may also optionally include a remote execution module that performs remote execution based on locally or remotely determined fulfillment data. Additional and / or alternative remote engines may be included. As described herein, in various implementations, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be preferred when resolving a voice utterance because they may at least reduce latency and / or network usage (because client-server round trip(s) are not required to resolve the voice utterance). However, one or more cloud-based automated assistant component(s) 191 may be utilized, at least selectively. For example, such component(s) can be utilized in parallel with on-device component(s), and output from such component(s) can be utilized when processing of the local component(s) fails. For example, the on-device fulfillment engine 145 can fail in certain circumstances (e.g., due to relatively limited resources on the client device 110), and in such circumstances, the remote fulfillment engine 194 can utilize the more robust resources of the cloud to generate fulfillment data. The remote fulfillment engine 194 can operate in parallel with the on-device fulfillment engine 145, and its results can be utilized when on-device fulfillment fails or can be invoked in response to the on-device fulfillment engine 145 determining that it has failed.

[0037] In various implementations, NLU engine(s) 144 and / or 193 may generate annotated output that includes one or more annotations of the recognized text and one or more (e.g., all) of the terms of the natural language input. In some implementations, NLU engine(s) 144 and / or 193 may be configured to identify and annotate various types of grammatical information in the natural language input. For example, NLU engine(s) 144 and / or 193 may include a morpheme module, which may separate individual words into morphemes and / or annotate morphemes with, for example, their classes. NLU engine(s) 144 and / or 193 may also include a part-of-speech tagger configured to annotate terms with their grammatical roles. Also, for example, in some implementations, NLU engine(s) 144 and / or 193 may additionally and / or alternatively include a dependency parser configured to determine syntactic relationships between terms in the natural language input.

[0038] In some implementations, NLU engine(s) 144 and / or 193 may additionally and / or alternatively include an entity tagger configured to annotate entity references in one or more segments, such as references to people (e.g., including literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional locations), etc. In some implementations, NLU engine(s) 144 and / or 193 may additionally and / or alternatively include a coreference resolver (not shown) configured to group or “cluster” references to the same entity based on one or more contextual cues. In some implementations, one or more components of NLU engine(s) 144 and / or 193 may rely on annotations from one or more other components of NLU engine(s) 144 and / or 193.

[0039] In some embodiments, NLU engine(s) 144 and / or 193 may also include an intent matcher configured to determine a user's intent to interact with automated assistant client 195. The intent matcher can determine the user's intent using various techniques. In some embodiments, the intent matcher may have access to one or more local and / or remote data structures, for example, that include multiple mappings between grammars and response intents. For example, the grammars included in the mappings may be selected and / or learned over time to represent typical user intents. For example, one grammar, "play <artist>," may be mapped to an intent that invokes a response action in which music by <artist> is played on client device 110. Another grammar, "[weather|forecast] today," may be able to match user queries such as "What's the weather today?" and "What's the forecast today?" In addition to or instead of grammars, in some embodiments, the intent matcher may use one or more trained machine learning models, alone or in combination with one or more grammars. These trained machine learning models can be trained to identify intents, for example, by embedding recognized text from a speech utterance into a low-dimensional space and then determining which alternative embedding (and therefore intent) is closest, using techniques such as Euclidean distance, cosine similarity, etc. As seen in the example grammar "play <artist>" above, some grammars have slots (e.g., <artist>) that can be filled with slot values ​​(or "parameters"). Slot values ​​can be determined in a variety of ways. Often, a user actively provides a slot value. For example, for the grammar "order <topping> pizza," a user can speak the phrase "order sausage pizza," in which case the slot <topping> is automatically filled. For example, alternative slot value(s) can be inferred based on the user's location, currently rendered content, the user's preferences, and / or other cue(s).

[0040] In some implementations, fulfillment engine(s) 145 may be configured to receive the predicted / inferred intent and any associated slot values ​​output by NLU engine(s) 144 and / or 193 and fulfill (or “resolve”) the intent. In various implementations, fulfillment (or “resolve”) of the user's intent may involve various fulfillment information (also referred to as fulfillment data) being generated / obtained, for example, by the fulfillment engine. This may include determining a local and / or remote response (e.g., answer) to the voice utterance, interaction(s) with locally installed application(s) based on the voice utterance, command(s) to send to Internet of Things (IoT) device(s) (directly or via corresponding remote system(s)), and / or other resolution action(s) based on the voice utterance. The on-device fulfillment engine 145 may then initiate local and / or remote performance / execution of the determined action(s) to resolve the voice utterance.

[0041] 2, a flowchart illustrating an example method 200 for identifying locally corresponding candidate compensation pairs at a client device is shown. For convenience, the operations of method 200 are described with reference to a system on which the operations are performed. The system of method 200 includes one or more processors and / or other component(s) of a client device (e.g., client device 110 of FIGS. 1A and / or 1B, client device 610 of FIGS. 6A and / or 6B, computing device 710 of FIG. 7, and / or other client devices). Furthermore, while the operations of method 200 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0042] At block 252, the system receives audio data capturing a voice utterance of a user of the client device via one or more microphones of the client device. At block 254, the system processes the audio data using an on-device ASR model stored in on-device storage of the client device to generate a predicted text segment that is predicted to correspond to the voice utterance. At block 256, the system visually renders the predicted text segment for presentation to the user on the display of the client device. The system returns to block 252 and continues to receive additional audio data, if any, and continue to process any additional audio data capturing any additional voice utterances of the user using the on-device ASR model, and continue to visually render any additional predictive text segments for presentation to the user on the display of the client device in a streaming manner.

[0043] At block 258, the system determines whether further user interface input has been received that includes modifying the portion of the predictive text segment to an alternative text segment. In some implementations, the further user interface input may be a typed or touched input to the portion of the predictive text segment that is visually rendered on the client device's display (e.g., as described with respect to FIGS. 6A and 6B). In additional or alternative implementations, the further user interface input may be an additional voice utterance (e.g., determined using the various components shown in FIG. 1B) that includes a command to the system to modify the portion of the predictive text segment to an alternative text segment. If, in a repetition of block 258, the system determines that further user interface input has not been received that includes modifying the portion of the predictive text segment to an alternative text segment, then at block 258, the system may continue to monitor for further user interface input. In particular, the system may monitor for modifications to any text segments that are visually rendered for presentation to the user at block 258.

[0044] If, in an iteration of block 258, the system determines that it has received further user interface input including modifying a portion of the predicted text segment to an alternative text segment, the system may proceed to block 260. In block 260, the system stores (1) the portion of the predicted text segment and the alternative text segment that are considered to be corresponding candidate correction pairs, and (2) the audio data, in on-device storage of the client device. In block 262, the system transmits the corresponding candidate correction pairs, but not the audio data, to the remote system. When transmitting the corresponding candidate correction pairs to the remote system, the system may optionally use one or more techniques to obfuscate the corresponding candidate correction pairs for privacy purposes. For example, the system may tokenize the corresponding candidate correction pairs in an opaque manner, thereby making them interpretable by the remote system but not to a user associated with the remote system. In some implementations, the system may transmit the corresponding candidate correction pairs to the remote system only if the number of occurrences of the candidate correction pairs detected at the client device reaches a threshold amount. In additional or alternative embodiments, the system may transmit a corresponding candidate correction pair to a remote system only if the candidate correction pair is included in a list of candidate correction pairs received from the remote system, the list including candidate correction pairs that have been detected on another client device.

[0045] 3, a flowchart illustrating an exemplary method 300 is shown in which a remote system determines whether a given corresponding candidate correction pair corresponds to an actual correction pair, and updates a global automatic speech recognition (ASR) model based on determining that the given corresponding candidate correction pair corresponds to the actual correction pair. For convenience, operations of method 300 are described with reference to a system that performs the operations. The system of method 300 includes one or more processors and / or other component(s) of a remote system (e.g., remote system 160 of FIG. 1A , cloud-based automated assistant component(s) 191 of FIG. 1B , computing device 710 of FIG. 7 , one or more high-performance servers, and / or other computing devices). Furthermore, while the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0046] In block 352, the system receives corresponding candidate correction pairs from a plurality of client devices, each of the corresponding candidate correction pairs including (1) a corresponding portion of a predicted text segment generated based on processing corresponding audio data generated locally at a corresponding one of the plurality of client devices and using a corresponding on-device ASR model, and (2) a corresponding alternative text segment generated locally at a corresponding one of the plurality of client devices based on corresponding modifications to the portion of the predicted text segment. Further, the system may store the corresponding candidate correction pairs received from the plurality of client devices in one or more remote storage devices accessible by the system. The system may index the corresponding candidate correction pairs and identify client devices that previously provided the corresponding candidate correction pairs.

[0047] In block 354, the system determines whether a given corresponding candidate correction pair corresponds to an actual correction pair (e.g., as described with respect to correction identification engine 162 of FIG. 1A). In some implementations, the system may determine whether a given corresponding candidate correction pair corresponds to an actual correction pair based on whether a threshold amount of occurrences of the given corresponding candidate correction pair has been received from multiple client devices. In additional or alternative implementations, the system may determine whether a given corresponding candidate correction pair corresponds to an actual correction pair based on query activity associated with multiple client devices. In some variations of these implementations, query activity at a given client device among the multiple client devices may be utilized in determining whether a given corresponding candidate correction pair corresponds to an actual correction pair even if the given client device does not provide occurrences of the given corresponding candidate correction pair to the system. In some implementations, in response to determining that a given corresponding candidate correction pair corresponds to an actual correction pair, the system may utilize one or more techniques to verify that the given corresponding candidate correction pair does, in fact, correspond to an actual correction pair (e.g., based on the degree of acoustic similarity between the corresponding portions of the predicted text segments and the alternative text segments, based on the Levenshtein edit distance between the corresponding portions of the predicted text segments and the alternative text segments, and / or using other techniques).

[0048] If, in an iteration of block 354, the system determines that the given corresponding candidate correction pair does not correspond to an actual correction pair, the system may return to block 352 and continue receiving corresponding candidate correction pairs from the multiple client devices. If, in an iteration of block 354, the system determines that the given corresponding candidate correction pair does correspond to an actual correction pair, the system may proceed to block 356. In block 356, the system identifies, from among the multiple client devices, a subset of the multiple client devices that provided the given corresponding candidate correction pair (e.g., as described with respect to client device identification engine 166 of FIG. 1 ).

[0049] At block 358, the system causes a global ASR model, which is a global-based counterpart of the corresponding on-device ASR models of multiple client devices, to be updated in a distributed manner utilizing the subset of client devices that provided a given corresponding candidate correction. In other words, for method 200 of FIG. 2, an exemplary technique for generating corresponding candidate correction pairs is described. Furthermore, for method 300 of FIG. 3, an exemplary technique for determining whether a given corresponding candidate correction pair actually corresponds to an actual correction pair that exhibits ASR misrecognition across a population of client devices, rather than simply considering corresponding candidate correction pairs generated on a single client device, is described.

[0050] 4, a flowchart illustrating an example method 400 for generating corresponding updates of a global automatic speech recognition (ASR) model locally at a client device is shown. For convenience, the operations of method 400 are described with reference to a system on which the operations are performed. The system of method 400 includes one or more processors and / or other component(s) of a client device (e.g., client device 110 of FIGS. 1A and / or 1B, client device 610 of FIGS. 6A and / or 6B, computing device 710 of FIG. 7, and / or other client devices). Furthermore, while the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0051] At block 452, the system receives, at a given client device of a given user, an instruction from a remote system to update, in a distributed manner, a global ASR model, which is a global-based counterpart of a corresponding on-device ASR model stored in a given on-device storage of the given client device, based on the corresponding candidate correction pairs stored in the on-device storage of the given client device. At block 454, the system retrieves at least a portion of the corresponding candidate correction pairs (e.g., alternative text segment(s)) from the given on-device storage of the given client device. At block 456, the system retrieves corresponding audio data stored in association with the corresponding candidate correction pairs from the given on-device storage of the given client device.

[0052] At block 458, the system processes the corresponding audio data associated with the corresponding candidate correction pair using the corresponding on-device ASR model stored locally in the given on-device storage of the given client device. At block 460, the system generates a corresponding update of the global ASR model based on processing the corresponding audio data (e.g., as described with respect to update engine 130 of FIG. 1A).

[0053] In block 462, the system determines whether additional corresponding candidate correction pairs are stored in the given on-device storage of the given client device. If, in a repetition of block 462, the system determines that additional corresponding candidate correction pairs are stored in the given on-device storage of the given client device, the system may return to block 454 and generate additional corresponding updates for the additional corresponding candidate correction pairs. In other words, the system may generate corresponding updates for each instance of the corresponding correction stored locally in the given on-device storage of the given client device. If, in a repetition of block 462, the system determines that additional corresponding candidate correction pairs are not stored in the given on-device storage of the given client device, the system may proceed to block 464.

[0054] At block 464, the system transmits one or more corresponding updates generated locally at the given client device to the remote system. In other words, for method 200 of Figure 2, an exemplary technique for generating corresponding candidate correction pairs is described. Further, for method 300 of Figure 3, an exemplary technique is described for determining whether a given corresponding candidate correction pair actually corresponds to an actual correction pair that exhibits ASR misrecognition across a population of client devices, rather than simply considering corresponding candidate correction pairs generated at a single client device. Further, for method 400 of Figure 4, an exemplary method is shown in which a given client device that provided a given corresponding candidate correction pair to the system generates a corresponding update for the given corresponding candidate correction pair.

[0055] 5, a flowchart illustrating an example method 500 for remotely updating a global automatic speech recognition (ASR) model at a remote system is shown. For convenience, the operations of method 500 are described with reference to the system on which the operations are performed. The system of method 500 includes one or more processors and / or other component(s) of the remote system (e.g., remote system 160 of FIG. 1A , cloud-based automated assistant component(s) 191 of FIG. 1B , computing device 710 of FIG. 7 , one or more high performance servers, and / or other computing devices). Furthermore, while the operations of method 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0056] At block 552, the system receives corresponding updates from the multiple client devices to be utilized in updating the global ASR model with respect to the actual correction pairs. At block 554, the system updates the global ASR model based on the corresponding updates (e.g., as described with respect to remote training engine 168 of FIG. 1A). In some implementations, the system may wait until a corresponding update is received from each of the multiple client devices (e.g., those identified for inclusion in the subset at block 456 of method 300 of FIG. 3) before updating the global ASR model based on the corresponding update. In additional or alternative implementations, the system may update the global ASR model based on the corresponding updates once the updates are received from the multiple client devices.

[0057] At block 556, the system determines whether one or more conditions are satisfied. If, at the iteration of block 556, the system determines that one or more conditions are not satisfied, the system may continue to monitor for one or more conditions to be satisfied at block 556. In particular, the system may continue to update the global ASR model while monitoring for one or more conditions to be satisfied at block 556. If, at the iteration of block 556, the system determines that one or more conditions are satisfied, the system may proceed to block 558. At block 558, the system transmits one or more updated weights of the updated global ASR model and / or the updated global ASR model to multiple client devices.

[0058] In some embodiments, the one or more conditions may be specific to the global ASR model and / or the remote system. In these embodiments, the one or more conditions may include, for example, whether the global ASR model has been updated based on reaching a threshold amount of corresponding updates, whether the global ASR model has been updated for a threshold period of time, whether a threshold period of time has elapsed since any of the updated global ASR models were last distributed, whether improvement of the updated global ASR model has been measured, and / or other conditions. In additional or alternative embodiments, the one or more conditions may be specific to one or more updated weights of the updated global ASR model and / or to each of multiple client devices that receive the updated global ASR model. In these implementations, the one or more conditions may include, for example, whether a given client device (e.g., of a plurality of client devices) is charging and / or has reached a threshold charge state, whether a given client device is being held or utilized by a given user of the given client device, whether a given time associated with a given location of a given client device (e.g., determined using the location sensor(s) of the given client device) is within a particular time range, and / or other conditions.

[0059] In other words, for method 200 of Figure 2, an exemplary technique for generating corresponding candidate correction pairs is described. Furthermore, for method 300 of Figure 3, an exemplary technique for determining whether a given corresponding candidate correction pair actually corresponds to an actual correction pair that exhibits an ASR misidentification across a population of client devices, rather than simply considering corresponding candidate correction pairs generated at a single client device, is described. Furthermore, for method 400 of Figure 4, an exemplary method is shown in which a given client device that provided a given corresponding candidate correction pair to the system generates a corresponding update for the given corresponding candidate correction pair. Furthermore, for method 500 of Figure 5, an exemplary method is shown in which a global ASR model is updated and distributed to prevent and / or mitigate future occurrences of an ASR misidentification characterized by a given corresponding candidate correction pair.

[0060] 6A and 6B, various user interfaces illustrating exemplary user interactions with a client device 610 (e.g., an instance of client device 110 of FIGS. 1A and 1B) are shown. Client device 610 of FIGS. 6A and 6B includes a touch-sensitive display screen 640 that displays and / or streams (i.e., in real time) predictive text segments corresponding to voice utterances provided by a user of client device 610, according to implementations disclosed herein. For convenience, actions performed by client device 610 are described with reference to an automated assistant (e.g., automated assistant 195 of FIG. 1B) performing the actions.

[0061] Display screen 640 includes text response and / or editing elements 684 that allow a user to provide user input (e.g., touch input or type input) to create, modify, delete, and / or replace term(s) via a virtual keyboard. Additionally, display screen 640 also includes a voice interface element 685 that, when activated, allows a user to provide user input (e.g., voice input) to confirm an action taken by client device 610, cancel an action taken by client device 610, and / or provide a voice utterance or additional voice utterance via one or more microphones. In some implementations, audio data corresponding to the voice utterance can be captured via one or more of the microphones, a predictive text segment can be generated and visually rendered on display screen 640 of client device 610, and the user input that amends the predictive text segment to an alternative text segment can be touch input to the predictive text segment(s) included in text response and / or editing elements 684 on display screen 640 of client device 610. In additional and / or alternative embodiments, the user input that amends the predictive text segment to the alternative text segment may be a voice input. In some variations of these embodiments, the voice input is received in response to touch input to the voice interface element 685, or the voice input is received within a threshold period of user input while one or more microphones are activated without touch input to the voice interface element 685 and / or other voice input activation methods. Furthermore, in some embodiments, the display screen 640 also includes system interface elements 681, 682, 683, which may interact with the user to cause the computing device 610 to perform one or more actions.

[0062] Additionally, in some implementations, the voice utterance may include an action to be performed by an automated assistant using the client device 610. As some non-limiting examples, the action may be an action such as calling or dialing a phone number, sending a text or SMS message (e.g., as shown in FIGS. 6A and 6B), sending an email, looking up contact information, requesting navigation information, sending a calendar invite, controlling one or more IoT devices, and / or any other action capable of being performed by an automated assistant operating on the client device 610.

[0063] 6A , assume that a user of client device 610 provides a speech utterance 652, “I tested positive for COVID, so I will miss tomorrow's meeting,” and the automated assistant visually renders a predicted text segment 654 (e.g., generated using on-device ASR engine 122 of FIGS. 1A and 1B ), “I tested positive for covet, so I will miss tomorrow's meeting.” In particular, predicted text segment 654 includes an ASR misrecognition, i.e., “covet,” instead of the user's intended “COVID.” Therefore, assume that the user provides further user interface input, such as a touch input, to client device 610, which corrects a portion of predicted text segment 654 (e.g., “covet”) to an alternative text segment (e.g., “COVID”), as shown by 656A. This results in the user's intended result: “I tested positive for COVID, so I will miss tomorrow's meeting.” In this example, the automated assistant may generate a corresponding candidate correction pair [covet, COVID] based on further user interface input that modifies a portion of predictive text segment 654 (e.g., "covet") to an alternative text segment (e.g., "COVID"), as shown by 656A. Further, the automated assistant may store the candidate correction pair [covet, COVID] in on-device memory of client device 610, and optionally in association with the audio data capturing speech utterance 652. Further, the automated assistant may transmit the corresponding candidate correction pair to a remote system (e.g., remote system 160 of FIG. 1A) without transmitting the audio data capturing speech utterance 652.This enables the remote system to determine whether the corresponding candidate correction pair corresponds to an actual correction pair, and if the remote system determines that the corresponding candidate correction pair [covet,COVID] corresponds to an actual correction pair (e.g., as described with respect to the correction identification engine 162 of FIG. 1A), it causes the client device 610 to generate a corresponding update that can be utilized in updating the global ASR model.

[0064] In some implementations, client device 610 may transmit the corresponding candidate correction pair [covet,COVID] to the remote system only if the number of occurrences of the candidate correction pair [covet,COVID] detected at client device 610 reaches a threshold amount. In additional or alternative implementations, client device 610 may transmit the corresponding candidate correction pair [covet,COVID] to the remote system only if the candidate correction pair [covet,COVID] is included in a list of candidate correction pairs received from the remote system and including candidate correction pairs [covet,COVID] that have been detected at another client device.

[0065] In some implementations, the predictive text segment 654, "I tested positive for covet, so I'm going to miss tomorrow's meeting," may be automatically populated into the text response and / or edit element 684 for editing by the user. The user may provide input to the text response and / or edit element 684, as also indicated in 656A (e.g., a cursor identifier), to, for example, modify "covet" to "COVID." In additional or alternative implementations, the automated assistant may visually render a send selectable graphical element 661, an edit selectable graphical element 662, and / or a cancel selectable graphical element 663. In some variations of these implementations, some user input is provided to the edit selectable graphical element 662, thereby populating the text response and / or edit element 684, to populate the predictive text segment 654, "I tested positive for covet, so I'm going to miss tomorrow's meeting," into the text response and / or edit element 684 for editing by the user.

[0066] In particular, the correction from "covet" to "COVID" for part of the predictive text segment 654 may only be utilized in generating a corresponding update to the global ASR model in response to the remote system determining that the correction is indeed a correction targeted to the performance of the on-device ASR model of the client device 610. Furthermore, the correction identified in FIG. 6A is targeted to the performance of the on-device ASR model (e.g., on-device ASR model 122A of FIGS. 1A and 1B). The determination that the correction is targeted to the performance of the on-device ASR model may be based, at the remote system, on a threshold amount of occurrences of a given corresponding correction pair (e.g., the corresponding correction pair [covet, COVID]) received from a population of client devices (e.g., client device 610 and other client devices) and / or a spike in query activity across the population of client devices associated with the alternative text segment 656A (e.g., a spike in queries associated with "COVID" in the example of FIG. 6A). Additionally, verifying that the modification is a correction aimed at the performance of the on-device ASR model may be based on, for example, the similarity between a portion of the predicted text segment 654 (e.g., "covet") and the alternative text segment 656A (e.g., "COVID"), which may be determined based on acoustic similarity, Levenshtein edit distance, and / or using other techniques.

[0067] In contrast, and with particular reference to FIG. 6B , suppose the user instead provides further user interface input, such as a touch input, to client device 610, which causes client device 610 to modify a portion of predicted text segment 654 (e.g., “covet”) to an alternative text segment (e.g., “an illness”), as indicated by 656B. This results in the result “I will miss tomorrow's meeting because I tested positive for illness.” Notably, in this example, the user of client device 610 simply changes his or her mind about speech utterance 652 (e.g., by changing “COVID” to “an illness”) despite the ASR misrecognition. Nevertheless, the automated assistant may generate a corresponding candidate correction pair [covet, an illness] based on the further user interface input that modifies a portion of predicted text segment 654 (e.g., “covet”) to an alternative text segment (e.g., “an illness”), as indicated by 656B. Additionally, the automated assistant may store the candidate correction pair [covet, an illness] in on-device memory of client device 610, and optionally in association with the audio data capturing speech utterance 652. Additionally, the automated assistant may transmit the corresponding candidate correction pair to a remote system (e.g., remote system 160 of FIG. 1A ) without transmitting the audio data capturing speech utterance 652. This enables the remote system to determine whether the corresponding candidate correction pair corresponds to an actual correction pair, and if the remote system determines that the corresponding candidate correction pair [covet, an illness] does correspond to an actual correction pair (e.g., as described with respect to correction identification engine 162 of FIG. 1A ), it may cause client device 610 to generate a corresponding update that can be utilized in updating the global ASR model.

[0068] In some implementations, client device 610 may transmit the corresponding candidate correction pair [covet, an illness] to the remote system only if the number of occurrences of the candidate correction pair [covet, an illness] detected at client device 610 reaches a threshold amount. In additional or alternative implementations, client device 610 may transmit the corresponding candidate correction pair [covet, an illness] to the remote system only if the candidate correction pair [covet, an illness] is included in a list of candidate correction pairs received from the remote system and including candidate correction pairs [covet, an illness] that have been detected at another client device.

[0069] However, in this example, based on, for example, a threshold amount of occurrences of a given corresponding correction pair (e.g., the corresponding correction pair [covet, an illness]) received at the remote system and from a population of client devices (e.g., client device 610 and other client devices) and / or a spike in query activity across the population of client devices associated with alternative text segment 656A (e.g., a spike in queries associated with “COVID” in the example of FIG. 6A ), it is unlikely that the remote system will determine that the candidate correction pair [covet, an illness] corresponds to the actual correction pair. In other words, because users often change their minds in various ways, it is unlikely that the threshold amount of occurrences of the candidate correction pair [covet, an illness] received by the remote system will be reached. Also, even if the threshold amount of occurrences of the candidate correction pair [covet, an illness] received by the remote system were reached, it is unlikely that this correction would be validated as a correction for purposes of on-device ASR performance. This is due, for example, to a lack of acoustic similarity between some of the predicted text segments 654 (eg, "covet") and the alternative text segment 656B (eg, "an illness").

[0070] 6A and 6B illustrate particular examples, it should be understood that these examples are provided for illustrative purposes and are not intended to be limiting. Rather, it should be understood that Figures 6A and 6B are provided to illustrate various user interactions that may result in corresponding candidate correction pairs being generated locally at client device 610.

[0071] 7, shown is a block diagram of an exemplary computing device 710 that may optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, cloud-based automated assistant component(s), and / or other component(s) may include one or more components of exemplary computing device 710.

[0072] The computing device 710 typically includes at least one processor 714 that communicates with several peripheral devices via a bus subsystem 712. These peripheral devices may include, for example, a storage subsystem 724, including a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices enable user interaction with the computing device 710. The network interface subsystem 716 provides an interface to an external network and is coupled to a corresponding interface device of another computing device.

[0073] The user interface input devices 722 may include keyboards, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, scanners, touchscreens integrated into displays, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 710 or over a communications network.

[0074] The user interface output devices 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 710 to a user or other machine or computing device.

[0075] Storage subsystem 724 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include logic for performing selected aspects of the methods disclosed herein or for implementing the various components shown in Figures 1A and 1B.

[0076] These software modules generally execute on the processor 714 alone or in combination with other processors. The memory 725 used by the storage subsystem 724 may include several memories, including a main random access memory (RAM) 730 for storing instructions and data during program execution and a read-only memory (ROM) 732 in which fixed instructions are stored. The file storage subsystem 726 may provide persistent storage for program files and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of particular embodiments may be stored by the file storage subsystem 726 in the storage subsystem 724, or may be stored on another machine accessible by the processor(s) 714.

[0077] The bus subsystem 712 provides a mechanism for allowing the various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0078] Computing device 710 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 710 shown in Figure 7 is intended only as a specific example for purposes of describing some implementations. Many alternative configurations of computing device 710 can have more or fewer components than the computing device shown in Figure 7.

[0079] In situations where the systems described herein may collect or otherwise monitor personal information about users or utilize personal and / or monitored information, users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social behavior or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how it receives content from content servers that may be more relevant to the user. Certain data may also be processed in one or more ways to remove personally identifiable information before it is stored or used. For example, a user's identity may be processed so that information that can identify the user cannot be determined, or if geographic location information (such as to the city, zip code, or state level) is obtained, the user's geographic location may be generalized so that the user's specific geographic location cannot be determined. Thus, users may control how information about them is collected and / or used.

[0080] In some implementations, a method performed by one or more processors of a remote system is provided herein, comprising receiving corresponding candidate correction pairs from a plurality of client devices, each of the corresponding candidate correction pairs comprising a corresponding portion of a corresponding predicted text segment generated based on processing corresponding audio data locally at a corresponding one of the plurality of client devices and using a corresponding on-device automatic speech recognition (ASR) model, and a corresponding alternative text segment generated locally at the corresponding one of the plurality of client devices and based on corresponding modifications to the corresponding portion of the corresponding predicted text segment that resulted in the corresponding alternative text segment. The method further includes determining whether a given corresponding candidate correction pair among the corresponding candidate correction pairs is a corresponding actual correction pair based on a threshold amount of occurrences of the given corresponding candidate correction pair received from one or more of the plurality of client devices; and, in response to determining that the given corresponding candidate correction pair is the corresponding actual correction pair, identifying a subset of the plurality of client devices that provided the given corresponding candidate correction pair from among the plurality of client devices; and causing a global ASR model, which is a global-based counterpart of the corresponding on-device ASR model, to be updated in a distributed manner using the subset of the plurality of client devices.

[0081] These and other implementations of the technology may include one or more of the following features.

[0082] In some implementations, causing the global ASR model to be updated in a distributed manner using a subset of the plurality of client devices that provided a given corresponding candidate correction pair may include sending, to each client device included in the subset of the plurality of client devices that provided a given corresponding candidate correction pair, a corresponding instruction to update the global ASR model in a distributed manner. Sending, to a given client device included in the subset of the plurality of client devices, a corresponding instruction to update the global ASR model in a distributed manner may cause the given client device to retrieve, from on-device storage of the given client device, corresponding audio data that was previously processed to generate a corresponding predictive text segment, generate a corresponding update of the global ASR model based on processing the corresponding audio data and using the corresponding on-device ASR model, and transmit the corresponding update to a remote system.

[0083] In some variations of those embodiments, causing the given client device to generate the corresponding update based on processing the corresponding audio data and using the corresponding on-device ASR model may cause the given client device to process the corresponding audio data using the corresponding on-device ASR model stored locally on the client device's on-device storage to generate corresponding additional predictive text segments, and to generate the corresponding update based on comparing at least a portion of the corresponding additional predictive text segments with corresponding alternative text segments.

[0084] In some variations of these embodiments, the corresponding on-device ASR model may include multiple layers, and the corresponding updates may be generated for a subset of the multiple layers of the corresponding on-device ASR model.

[0085] In some variations of those embodiments, the method may further include receiving a corresponding update from the given client device, and updating the global ASR model based on the corresponding update received from the given client device and corresponding additional updates of the global ASR model received from one or more additional client devices of the plurality of client devices that are also included in the subset of the plurality of client devices.

[0086] In some further variations of those embodiments, the method may further include transmitting the updated global ASR model to a plurality of client devices. Transmitting the updated global ASR model to a given client device may cause the given client device to replace a corresponding on-device ASR model with the updated global ASR model in the given client device's on-device storage.

[0087] In additional or alternative further variations of those embodiments, the method may further include transmitting one or more updated global weights of the updated global ASR model to a plurality of client devices. Transmitting the one or more updated global weights of the updated global ASR model to a given client device may cause on-device storage of the client device to replace one or more on-device weights of the corresponding on-device ASR model with the one or more updated global weights of the updated global ASR model.

[0088] In a further variation of these embodiments, either additionally or alternatively, the updated global ASR model may bias subsequent speech processing toward the corresponding alternative text segments.

[0089] In additional or alternative further variations of these embodiments, the updated global ASR model may bias subsequent speech processing away from some of the corresponding predicted text segments.

[0090] In some embodiments, the method may further include, in response to determining that the given corresponding candidate correction pair is not the corresponding actual correction pair, refraining from updating the global ASR model in a distributed manner using one or more of the multiple client devices that provided the given corresponding candidate correction pair.

[0091] In some implementations, corresponding audio data that is processed using a corresponding on-device ASR model locally at a corresponding one of the multiple client devices may not be received at a remote system.

[0092] In some implementations, the corresponding audio data processed using the corresponding on-device ASR model locally at the corresponding one of the multiple client devices may be stored in on-device storage of the corresponding one of the multiple client devices in response to the corresponding alternative text segment being generated.

[0093] In some implementations, determining whether a given corresponding candidate correction pair is a corresponding actual correction pair may further be based on corresponding query activity at multiple client devices.

[0094] In some implementations, a method performed by one or more processors of a client device is provided herein, the method including receiving, via one or more microphones of the client device, audio data capturing a voice utterance of a user of the client device; processing the audio data using an on-device automatic speech recognition (ASR) model stored locally in on-device storage of the client device to generate a predictive text segment predicted to correspond to the voice utterance; causing the predictive text segment to be visually rendered for presentation to the user on a display of the client device; and in response to the visual rendering of the predictive text segment, receiving a further user interface input that modifies at least a portion of the predictive text segment into an alternative text segment; and in response to receiving the further user interface input that modifies a portion of the predictive text segment into the alternative text segment, storing, in on-device storage of the client device, a portion of the predicted text segment and the alternative text segment as corresponding candidate correction pairs; transmitting the corresponding candidate correction pairs to a remote system; storing, in on-device storage of the client device, audio data capturing the user's voice utterance; receiving, from the remote system, an instruction to update, in a distributed manner, a global ASR model, which is a global-based counterpart of the corresponding on-device ASR model, based on the corresponding correction pairs; in response to receiving the instruction to update, in a distributed manner, the global ASR model based on the corresponding correction pairs and based on the audio data, generating an update for the global ASR model to be utilized by the remote system in updating the global ASR model; and transmitting the update to the remote system so that the update is utilized in updating the global ASR model.

[0095] These and other implementations of the technology may include one or more of the following features.

[0096] In some implementations, the method may further include refraining from transmitting audio data captured from the user's voice utterances to the remote system.

[0097] In some implementations, generating an update for use by the remote system in updating the global ASR model based on the corresponding candidate correction pairs and based on the audio data may include processing the audio data using an on-device ASR model stored locally in on-device storage of the client device to generate an additional predictive text segment predicted to correspond to the speech utterance, and generating the update based on comparing the additional predictive text segment and the corresponding alternative text segment.

[0098] In some embodiments, sending the update to the remote system may cause the remote system to update the global ASR model based on the update provided by the client device and multiple additional corresponding updates provided by multiple additional client devices that provided additional instances of the corresponding candidate correction pairs.

[0099] In some variations of those embodiments, the method may further include receiving an updated global ASR model from the remote system and replacing the on-device ASR model with the updated global ASR model in on-device storage of the client device.

[0100] In some variations of those embodiments, the method may further include receiving, from the remote system, one or more updated global weights of the updated global ASR model; and replacing, in on-device storage of the client device, the one or more on-device weights of the on-device ASR model with the one or more updated global weights of the updated global ASR model.

[0101] In some embodiments, the instruction to update the global ASR model based on the corresponding correction pairs in a distributed manner may be received in response to the remote system determining that the number of occurrences of the corresponding correction pairs received from the client device and / or multiple additional client devices has reached a threshold amount.

[0102] In some implementations, the further user interface input may be a touch input by a user of the client device to a portion of the predictive text segment that is being visually rendered for presentation to the user on a display of the client device.

[0103] In some implementations, the further user interface input may be an additional utterance by the user in response to a portion of the predictive text segment being visually rendered for presentation to the user on the display of the client device.

[0104] In some implementations, the on-device ASR model may include multiple layers, and updates may be generated for a subset of the multiple layers of the on-device ASR model.

[0105] In some implementations, transmitting the corresponding candidate correction pairs to the remote system may be in response to determining that the number of occurrences of the corresponding candidate correction pairs detected at the client device has reached a threshold amount.

[0106] In some embodiments, transmitting the corresponding candidate correction pair to the remote system may be in response to determining that the corresponding candidate correction pair is included in a list of candidate correction pairs received from the remote system.

[0107] In some implementations, a system of computing devices is provided, including a given client device of a given user, the given client device including at least one client device processor and a client device memory storing instructions that, when executed by the at least one client device processor, cause the at least one client device processor to: receive audio data capturing a voice utterance of the given user via one or more microphones of the given client device; process the audio data using an on-device automatic speech recognition (ASR) model stored locally at the given client device to generate predictive text segments predicted to correspond to the voice utterance; The predictive text segment is visually rendered for presentation to a given user on a display of a given client device; in response to the visual rendering of the predictive text segment, a further user interface input is received that modifies at least a portion of the predictive text segment into an alternative text segment; in response to receiving the further user interface input that modifies the predictive text segment into the alternative text segment, the portion of the predictive text segment and the alternative text segment are stored locally at the given client device as corresponding candidate correction pairs; and audio data capturing the user's voice utterance is stored locally at the given client device.The system of computing devices further includes a remote server communicatively coupled to the given client device, the remote server including at least one remote processor and a remote memory that stores instructions that, when executed by the at least one remote processor, cause the at least one remote processor to: receive corresponding candidate correction pairs from the given client device; determine whether the corresponding candidate correction pairs are corresponding actual correction pairs based on a threshold amount of occurrences of the corresponding candidate correction pairs received from the given client device and / or multiple additional client devices in addition to the given client; and, in response to determining that the given corresponding candidate correction pair is the corresponding actual correction pair, cause a global ASR model, which is a global-based counterpart of the corresponding on-device ASR model, to be updated in a distributed manner using the given client device and / or multiple additional client devices.

[0108] Various embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., central processing unit (CPU)(ies), graphics processing unit (GPU)(ies), digital signal processor (DSP)(ies), and / or tensor processing unit (TPU)(ies)) to perform methods such as one or more of the methods described herein. Another embodiment may include an automated assistant client device (e.g., a client device including at least an automated assistant interface for interfacing with cloud-based automated assistant component(s)) including processor(s) operable to execute stored instructions to perform methods such as one or more of the methods described herein. Yet another embodiment may include a system of one or more servers including one or more processors operable to execute stored instructions to perform methods such as one or more of the methods described herein.

Claims

1. 1. A method implemented by one or more processors of a remote system, comprising: receiving corresponding candidate correction pairs from a plurality of client devices, each of the corresponding candidate correction pairs comprising: a corresponding portion of a corresponding predictive text segment generated based on processing the corresponding audio data locally at a corresponding one of the plurality of client devices and using a corresponding on-device automatic speech recognition (ASR) model; receiving, locally at the corresponding one of the plurality of client devices, a corresponding alternative text segment generated based on a corresponding modification to the corresponding portion of the corresponding predictive text segment from which the corresponding alternative text segment was derived; determining whether a given one of the corresponding candidate correction pairs is a corresponding actual correction pair based on a threshold amount of occurrences of the given corresponding candidate correction pair received from one or more of the plurality of client devices; In response to determining that the given corresponding candidate correction pair is a corresponding actual correction pair, identifying a subset of the plurality of client devices that provided the given corresponding candidate correction pair; causing a global ASR model, which is a global-based counterpart of the corresponding on-device ASR model, to be updated using the subset of the plurality of client devices in a distributed manner; A method comprising:

2. Causing the global ASR model to be updated in the distributed manner using the subset of the plurality of client devices that provided the given corresponding candidate correction pair may include: transmitting, to each client device in the subset of the plurality of client devices that provided the given corresponding candidate correction pair, a corresponding instruction to update the global ASR model in the distributed manner, wherein transmitting the corresponding instruction to a given client device in the subset of the plurality of client devices to update the global ASR model in the distributed manner includes instructing the given client device: retrieving, from on-device storage of the given client device, the corresponding audio data that was previously processed to generate the corresponding predictive text segment; generating a corresponding update of the global ASR model based on processing the corresponding audio data and using the corresponding on-device ASR model; and transmitting the corresponding update to the remote system; The method of claim 1 , wherein

3. Having the given client device generate the corresponding update based on processing the corresponding audio data and using the corresponding on-device ASR model includes causing the given client device to: processing the corresponding audio data using the corresponding on-device ASR model stored locally in the on-device storage of the client device to generate corresponding additional predictive text segments; generating the corresponding update based on comparing at least a portion of the corresponding additional predictive text segment with the corresponding alternative text segment; The method of claim 2, wherein

4. The method of claim 2 or claim 3, wherein the corresponding on-device ASR model includes multiple layers, and the corresponding update is generated for a subset of the multiple layers of the corresponding on-device ASR model.

5. receiving the corresponding update from the given client device; updating the global ASR model based on the corresponding update received from the given client device and corresponding additional updates of the global ASR model received from one or more additional client devices of the plurality of client devices that are also included in the subset of the plurality of client devices; The method of any one of claims 2 to 4, further comprising:

6. and transmitting the updated global ASR model to the plurality of client devices, wherein transmitting the updated global ASR model to the given client device includes transmitting to the given client device: The method of claim 5 , further comprising causing the corresponding on-device ASR model to be replaced with the updated global ASR model in the on-device storage of the given client device.

7. and transmitting one or more updated global weights of the updated global ASR model to the plurality of client devices, wherein transmitting the one or more updated global weights of the updated global ASR model to the given client device includes transmitting to the given client device:

6. The method of claim 5, further comprising causing one or more on-device weights of the corresponding on-device ASR model to be replaced with the one or more updated global weights of the updated global ASR model in the on-device storage of the given client device.

8. The method of any one of claims 5 to 7, wherein the updated global ASR model biases subsequent speech processing towards the corresponding alternative text segments.

9. The method of any one of claims 5 to 7, wherein the updated global ASR model biases subsequent speech processing to avoid some of the corresponding predicted text segments.

10. In response to determining that the given corresponding candidate correction pair is not a corresponding actual correction pair, 10. The method of claim 1, further comprising refraining from updating the global ASR model in the distributed manner using the one or more of the plurality of client devices that provided the given corresponding candidate correction pair.

11. 11. The method of claim 1, wherein the corresponding audio data that is processed locally at the corresponding one of the plurality of client devices using the corresponding on-device ASR model is not received at the remote system.

12. 12. The method of claim 1, wherein the corresponding audio data processed using the corresponding on-device ASR model locally at the corresponding one of the plurality of client devices is stored in on-device storage of the corresponding one of the plurality of client devices in response to the corresponding alternative text segment being generated.

13. The method of any one of claims 1 to 12, wherein determining whether the given corresponding candidate correction pair is a corresponding actual correction pair is further based on corresponding query activity at the plurality of client devices.

14. 1. A method implemented by one or more processors of a client device, comprising: receiving, via one or more microphones of a client device, audio data capturing a voice utterance of a user of said client device; processing the audio data using an on-device automatic speech recognition (ASR) model stored locally in on-device storage of the client device to generate a predictive text segment that is predicted to correspond to the voice utterance; causing the predictive text segments to be visually rendered for presentation to the user on a display of the client device; and receiving a further user interface input in response to the predictive text segment being visually rendered, the further user interface input modifying at least a portion of the predictive text segment into an alternative text segment; in response to receiving the further user interface input modifying the portion of the predictive text segment to the alternative text segment; storing the portion of the predictive text segment and the alternative text segment as corresponding candidate correction pairs in the on-device storage of the client device; transmitting the corresponding candidate correction pairs to a remote system; storing the audio data capturing the voice utterance of the user in the on-device storage of the client device; receiving an instruction from the remote system to update a global ASR model, which is a global-based counterpart of a corresponding on-device ASR model, in a distributed manner based on the corresponding correction pair; in response to receiving the instruction to update the global ASR model based on the corresponding correction pairs in the distributed manner; generating an update to the global ASR model based on the corresponding candidate correction pairs and based on the audio data, the update being utilized by the remote system in updating the global ASR model; transmitting the update to the remote system so that the update is utilized in updating the global ASR model; A method comprising:

15. The method of claim 14 , further comprising refraining from transmitting the audio data capturing the voice utterance of the user to the remote system.

16. generating the updates to be utilized by the remote system in updating the global ASR model based on the corresponding candidate correction pairs and based on the audio data, processing the audio data using the on-device ASR model stored locally on the on-device storage of the client device to generate additional predictive text segments predicted to correspond to the voice utterance; generating the update based on comparing the additional predictive text segment and the corresponding alternative text segment; 16. The method of claim 14 or 15, comprising:

17. Transmitting the update to the remote system includes transmitting the update to the remote system.

17. The method of claim 14, further comprising updating the global ASR model based on the update provided by the client device and a plurality of additional corresponding updates provided by a plurality of additional client devices that provided additional instances of the corresponding candidate correction pairs.

18. receiving the updated global ASR model from the remote system; replacing the on-device ASR model with the updated global ASR model in the on-device storage of the client device; 20. The method of claim 17, further comprising:

19. receiving one or more updated global weights of the updated global ASR model from the remote system; replacing, in the on-device storage of the client device, one or more on-device weights of the on-device ASR model with the one or more updated global weights of the updated global ASR model; 20. The method of claim 17, further comprising:

20. 20. The method of claim 14, wherein the instruction to update the global ASR model in the distributed manner based on the corresponding correction pairs is received in response to the remote system determining that a threshold amount of occurrences of the corresponding correction pairs received from the client device and / or a plurality of additional client devices has been reached.

21. 21. The method of claim 14, wherein the further user interface input is a touch input by the user of the client device to the portion of the predictive text segment being visually rendered for presentation to the user on the display of the client device.

22. 22. The method of any one of claims 14 to 21, wherein the further user interface input is an additional utterance of the user in response to the portion of the predictive text segment being visually rendered for presentation to the user on the display of the client device.

23. The method of any one of claims 14 to 22, wherein the on-device ASR model includes multiple layers, and the update is generated for a subset of the multiple layers of the on-device ASR model.

24. 24. The method of claim 14, wherein transmitting the corresponding candidate correction pairs to the remote system is in response to determining that a number of occurrences of the corresponding candidate correction pairs detected at the client device has reached a threshold amount.

25. 25. The method of claim 14, wherein transmitting the corresponding candidate correction pair to the remote system is in response to determining that the corresponding candidate correction pair is included in a list of candidate correction pairs received from the remote system.

26. 1. A system of computing devices, comprising: A given client device of a given user, the given client device including at least one client device processor and a client device memory storing instructions that, when executed by the at least one client device processor, cause the at least one client device processor to: receiving audio data capturing a voice utterance of the given user via one or more microphones of the given client device; processing the audio data using an on-device automatic speech recognition (ASR) model stored locally at the given client device to generate predictive text segments that are predicted to correspond to the voice utterance; causing the predictive text segments to be visually rendered for presentation to the given user on a display of the given client device; receiving a further user interface input in response to the predictive text segment being visually rendered, the further user interface input modifying at least a portion of the predictive text segment into an alternative text segment; in response to receiving the further user interface input modifying the portion of the predictive text segment into an alternative text segment; causing the portion of the predictive text segment and the alternative text segment to be stored locally at the given client device as a corresponding candidate correction pair; causing the audio data capturing the voice utterance of the user to be stored locally at the given client device; a remote server communicatively coupled to the given client device, the remote server including at least one remote processor and a remote memory storing instructions that, when executed by the at least one remote processor, cause the at least one remote processor to: receiving the corresponding candidate correction pairs from the given client device; determining whether the corresponding candidate correction pairs are corresponding actual correction pairs based on a threshold amount of occurrences of the corresponding candidate correction pairs received from the given client device and / or a plurality of additional client devices in addition to the given client device; In response to determining that the given corresponding candidate correction pair is a corresponding actual correction pair, a remote server causing a global ASR model, which is a global-based counterpart of the corresponding on-device ASR model, to be updated in a distributed manner using the given client device and / or the plurality of additional client devices; 1. A system of computing devices comprising:

27. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to perform operations in accordance with the method of any one of claims 1 to 26.

Images