Identification and correction of misrecognition in distributed automatic speech recognition (ASR).

Abstract

Implementations described herein identify and correct automatic speech recognition (ASR) misrecognitions. For example, on-device processor(s) of a client device may generate predicted text segments predicted to correspond to a voice utterance of a user of the client device and may receive further input to modify the predicted text segments into alternative text segments. Furthermore, the on-device processor(s) may store these text segments as candidate correction pairs in on-device storage and transmit the candidate correction pairs to a remote system. Furthermore, the remote processor(s) of the remote system may determine that the candidate correction pairs are actual correction pairs and cause the client device to generate an update of a global ASR model of the candidate correction pairs. Furthermore, the remote processor(s) may distribute the global ASR model to the client device and / or additional client devices.

Description

Background Art

[0001] Voice-based user interfaces are becoming increasingly used in the control of computers and other electronic devices. Voice-based user interfaces have evolved from initial basic interfaces that could only understand simple and direct commands to more advanced interfaces that can respond to natural language requests, understand context, and manage two-way conversations or interactions with users. Many voice-based user interfaces perform speech recognition on voice utterances (e.g., using an automatic speech recognition (ASR) model) to generate corresponding text, perform semantic analysis of that text to determine the meaning of the voice utterance, and execute one or more actions based on the identified meaning.

[0002] Although the performance of speech recognition continues to improve, inaccurate speech recognition can still occur in many situations. As a non-limiting example, in the training corpus on which an ASR model is trained, inaccurate speech recognition can occur for new terms and / or terms that are relatively infrequent (or non-existent). In attempts to effectively recognize new terms and / or infrequent terms, techniques have been proposed to generate additional speech recognition hypotheses in addition to an initial hypothesis (or multiple initial hypotheses) and consider the additional speech recognition hypotheses as candidates for speech recognition. However, such techniques require additional post-processing and may not result in effective recognition of many terms in many situations, such as when the first hypothesis / multiple hypotheses are too far off and / or when a particular term is not included in the word list of the additional hypotheses.

[0003] Furthermore, when speech recognition is performed on-device (i.e., on the client device), inaccurate speech recognition may be more pronounced. This may be due, for example, to on-device ASR models being less robust than cloud-based global ASR models, to on-device memory and / or processor resources being more constrained than cloud-based resources, and / or to the more limited word lists for additional hypothesis generation being on-device. However, in many situations, performing speech recognition on-device rather than remotely is beneficial, considering the technical issues that may include performance, data security, and network usage.

[0004] Several techniques have been proposed to identify and correct inaccurate speech recognition, but these techniques have drawbacks. For example, some techniques can monitor user interaction with a transcript containing corresponding text generated based on processing audio data captured from a user's speech utterance. In these examples, user interaction may modify the corresponding text to alternative text, and these modifications may be considered corrections. However, a user's thinking may change between providing a speech utterance and subsequently modifying its corresponding text. Therefore, it may not be appropriate to assume that these modifications are corrections for inaccurate speech recognition. Furthermore, some techniques also consider speech similarity and / or Levenshtein edit distance between the corresponding text and these modifications. While these techniques may provide a better indicator of whether these modifications are corrections for inaccurate speech recognition, they are generally applied on a per-user basis and cannot consider whether other users would make the same modifications. Consequently, there is a need in the art for improvements in techniques to identify, learn from, and do so in a privacy-conscious manner for inaccurate speech recognition. [Overview of the project] [Means for solving the problem]

[0005] The embodiments described herein relate to identifying and correcting misrecognitions in automated speech recognition (ASR) in a distributed manner. For example, an on-device processor(s) of a client device may receive audio data capturing the speech utterances of the client device's user, process the audio data using an on-device ASR model (e.g., one stored in the client device's on-device storage) to generate predictive text segments that are expected to correspond to the speech utterances captured in the audio data, and these predictive text segments may be visually rendered for presentation to the user via the client device's display. Furthermore, the on-device processor(s) may receive additional user interface inputs (e.g., additional speech utterances, touch inputs, type inputs, and / or other inputs) that modify portions of the predictive text segments into alternative text segments. In response to receiving further user interface input to modify a portion of a predicted text segment into an alternative text segment, the on-device processor(s) may store the modified portion of the predicted text segment and the alternative text segment as a corresponding candidate correction pair (e.g., in the client device's on-device storage), and (optionally, in association with the candidate correction pair) the audio data processed to generate the predicted text segment (e.g., in the client device's on-device storage). Furthermore, the on-device processor(s) may send the corresponding candidate correction pair to a remote system without sending the audio data to the remote system. In some embodiments, the on-device processor(s) may send the corresponding candidate correction pair to a remote system only if the number of occurrences of the candidate correction pair detected on the client device reaches a threshold amount. In additional or alternative embodiments, the on-device processor(s) may send the corresponding candidate correction pair to a remote system only if the candidate correction pair is included in a list of candidate correction pairs. This list of candidate correction pairs includes candidate correction pairs received from a remote system and detected on another client device.The corresponding additional on-device processors of multiple additional client devices (e.g., those added to a client device) may perform the same or similar operations as described above and may send additional corresponding candidate correction pairs to the remote system in the same or similar circumstances as described above.

[0006] Furthermore, a remote processor(s) of a remote system may determine whether a given corresponding candidate correction pair received from a client device and / or one or more of the client devices is actually a corrected pair (i.e., correcting a portion of a predicted text segment to an alternative text segment corrects an ASR misrecognition by the on-device ASR model(s) of the client device and / or additional client devices). For example, a remote processor(s) may determine that a given corresponding candidate correction pair is an actual candidate pair based on the number of occurrences of the corresponding candidate correction pair received from a client device and / or one or more of the additional client devices reaching a threshold amount. Additionally or alternatively, a remote processor(s) may determine that a corresponding candidate correction pair is an actual correction based on query activity associated with multiple client devices. This determination may be based, for example, on query activity indicating that a threshold amount of queries containing alternative text segments are being submitted by one or more users of the client device and / or additional client devices, and optionally, over a threshold period and / or within a specific geographical area (i.e., a spike in queries). In particular, any corrections to query activity may also be sent to a remote system and from a client device that detects these corrections in the query activity.

[0007] In various embodiments, in response to a determination that a given corresponding candidate correction pair corresponds to an actual correction, a remote processor(s) may identify a subset of client devices from among the client device and / or a subset of additional client devices that provided the given corresponding candidate correction pair determined to correspond to the actual correction pair, and may update the global ASR model in a distributed manner using the subset of client devices. This global ASR model is a global-based correspondence of the corresponding ASR models(s) from the client device and / or a subset of additional client devices, and is stored in the remote memory of the remote system. For example, a remote processor(s) may send an instruction in a non-distributed manner to update the global ASR model based on a given corresponding candidate correction pair.

[0008] Furthermore, an on-device processor(s) of a given client device identified for inclusion in a subset may generate a corresponding update to the global ASR model with respect to a given corresponding candidate correction pair. For example, an on-device processor(s) may retrieve corresponding audio data previously stored in a given on-device storage of a given client device (e.g., from a given on-device storage of a given client device). Furthermore, an on-device processor(s) may generate a corresponding update to the global ASR model based on processing the corresponding audio data and using the corresponding on-device ASR model. For example, an on-device processor(s) may use the corresponding on-device ASR model to process the corresponding audio data to generate a corresponding additional predictive text segment, and then generate a corresponding update based on comparing the corresponding additional predictive text segment with an alternative text segment previously stored locally on a given client device in association with a given corresponding candidate correction pair. Additionally or alternatively, an on-device processor(s) may process the corresponding audio data to generate a corresponding representation of the corresponding audio data using only a subset of the machine learning (ML) layers of the corresponding on-device ASR model (for example, using only the input(or) and / or encoding(or) layers of the corresponding on-device ASR model, but not the coupling(or), decoding(or), and / or output(or) layers of the corresponding on-device ASR model; or using only the input(or) and coupling(or) layers of the corresponding on-device ASR model, but not the output(or) layers of the corresponding on-device ASR model, etc.), and may perform various forward and / or backward passes across the subset of the ML layers of the corresponding on-device ASR model.In these examples, a subset of the ML layer may be updated based on its processing, while another layer remains fixed. The difference between the pre-update subset of the ML layer and the updated subset (or the difference between one or more on-device weights associated with the pre-update and updated subsets of the ML layer) may correspond to a corresponding update in the global ASR model. In these examples, only a subset of the ML layer may be updated in this manner to prioritize saving computational resources on a given client device.

[0009] Furthermore, the remote processor(s) may receive corresponding updates to the global ASR model from each of the client devices identified to be included in the subset. Additionally, the remote processor(s) may update the global ASR model based on the corresponding updates received from each of the client devices identified to be included in the subset. Furthermore, the remote processor(s) may distribute the updated global ASR model to multiple client devices (e.g., client devices identified to be included in the subset, additional client devices, and / or further additional client devices), thereby causing multiple client devices to update their corresponding on-device ASR models.

[0010] Therefore, based on the techniques described herein, not only is the global ASR model updated in a distributed manner to address ASR misrecognition, but the ASR misrecognition itself is also identified in a distributed manner. In other words, the techniques described herein enable the identification of ASR misrecognition based on modifications across a population of client devices, rather than considering modifications on a single client device (e.g., using speech similarity, Levenshtein edit distance, etc.), in order to distinguish between scenarios where a given user has changed their mind and / or where the modification may not be due to an ASR misrecognition. Furthermore, the techniques described herein enable the correction of these ASR misrecognitions by causing the client devices across the entire population to generate updates to the global ASR model. The global ASR model is then redistributed across the entire population of client devices, replacing the corresponding on-device ASR model, thereby eliminating and / or mitigating the future occurrence of these ASR misrecognitions in on-device speech processing using the corresponding on-device ASR model. As a result, the corresponding on-device ASR model becomes more robust against these ASR misrecognitions and can more easily recognize new terms and / or phrases that have been newly added to the client device user's word list, or new terms and / or phrases that the corresponding on-device ASR model has previously been trained to recognize.

[0011] For example, suppose a given user of a given client device provides the voice utterance, "I tested positive for COVID, so I will be absent from the meeting." However, suppose that when processing the audio data capturing the voice utterance, the corresponding on-device ASR model stored locally in the client device's on-device storage misinterprets "COVID" as "covet." Therefore, the given user may provide further user interface input to correct "covet" to "COVID," as the user intended when providing the voice utterance. In this case, part of the predicted text segment could be "covet," the alternative text segment could be "COVID," and the corresponding candidate correction pair [covet,COVID] could be stored in the client device's on-device storage and optionally associated with the audio data capturing the voice utterance. Furthermore, in this example, the corresponding candidate correction pair [covet,COVID] may be sent to the remote system, and optionally, in response to a determination that the number of occurrences of the candidate correction pair [covet,COVID] detected by a given client device has reached a threshold amount, and / or in response to a determination that the candidate correction pair [covet,COVID] is included in a list of candidate correction pairs received from the remote system (for example, indicating that another client device has also detected occurrences of the candidate correction pair [covet,COVID]).

[0012] In this case, we further assume that the remote system receives the corresponding candidate correction pair [covet,COVID] from a given client device, and receives multiple additional occurrences of the corresponding candidate correction pair [covet,COVID] from multiple additional devices in addition to the given client device. In some of these examples, the remote system may determine that the corresponding candidate correction pair [covet,COVID] is an actual candidate pair based on the number of occurrences of the corresponding candidate correction pair [covet,COVID] received from the given client device and / or the additional client devices reaching a threshold amount. In additional or alternative examples, the remote system may determine that the corresponding candidate correction pair [covet,COVID] is an actual correction pair based on the corresponding user of the given client device and / or the additional client devices submitting a query containing the term "COVID" (and optionally, based on the term "COVID" being modified in that query activity). Thus, the remote system may indeed determine that the corresponding candidate correction pair [covet,COVID] is an actual correction pair. As a result, the remote system can instruct a given client device, and any other client devices that have provided the corresponding correction pair [covet,COVID], to generate corresponding updates to the global ASR model, thereby preventing and / or mitigating ASR misidentification where "COVID" is mistaken for "covet".

[0013] In this example, we further assume that a given client device receives instructions to generate a corresponding update to a global ASR model. The given client device may retrieve from its given on-device storage at least a portion of the corresponding candidate correction pairs [covet,COVID] and audio data capturing a speech utterance stored in the given client device's on-device storage (e.g., audio data capturing the speech utterance, "I tested positive for COVID, so I will miss the meeting"). In some of these examples, the given client device may use the corresponding on-device ASR model to process the audio data and generate additional predictive text segments that are expected to correspond to the speech utterance, and compare these additional predictive text segments to alternative text segments to generate a corresponding update to the global ASR model. In other words, the given client device may use the alternative text segments (e.g., the alternative text segment for "COVID") as training signals when generating a corresponding update to the global ASR model. In additional or alternative examples, the given client device may use a subset of the ML layers of the corresponding on-device ASR model to process the audio data and generate a corresponding representation of the audio data. Then, various forward and / or backward passes can be performed across a subset of the ML layers of the corresponding on-device ASR model, and for example, a subset of the ML layers can be updated based on one or more gradients generated based on the execution of various forward and / or backward passes across a subset of the ML layers of the corresponding on-device ASR model. Another client device that has provided the corresponding candidate correction pair [covet,COVID] to the remote system can generate a corresponding update in the same or similar manner, and the remote system can update the global ASR model based on the corresponding updates received from multiple client devices.

[0014] In particular, in the above example, if further user input provided by a given user modifies part of the predicted text segment from "covet" to "an illness," the corresponding candidate correction pair [covet,an illness] may be stored in the on-device storage of a given client device and optionally associated with audio data capturing the speech utterance, and the corresponding candidate correction pair [covet,an illness] may be sent to the server. However, in this case, the remote system is unlikely to receive many of the corresponding candidate correction pairs [covet,an illness] (for example, compared to the corresponding candidate correction pair [covet,COVID]). This is because it represents a case where a given user has changed their mind (for example, specifying "an illness" instead of "COVID"), and different users may change their minds in different ways. Therefore, only the corresponding candidate correction pairs that actually indicate an ASR misrecognition should be identified by the remote system. Nevertheless, in various embodiments, the techniques described herein may be used in conjunction with other techniques to identify and / or verify ASR misrecognitions (e.g., speech similarity, Levenshtein edit distance, etc.).

[0015] By implementing the techniques described herein, one or more technical advantages can be achieved. As a non-limiting example, the techniques described herein enable the rapid and efficient identification of actual ASR misidentifications by considering candidate ASR misidentifications across an entire population of client devices. By considering candidate ASR misidentifications across an entire population of client devices rather than on a device-by-device basis, actual ASR misidentifications can be rapidly and efficiently distinguished from situations where each user of a client device has simply changed their mind. As another non-limiting example, the techniques described herein enable updating the ASR model in a manner that ensures the security of user data, since audio data (e.g., the basis of the ASR misidentification) remains only on the client device, but can still be utilized when generating updates to the ASR model, as described herein. As a result, the ASR model described herein prevents and / or mitigates the occurrence of ASR misidentifications in a manner that maintains the security of user data.

[0016] The above description is provided as an overview of only some of the embodiments disclosed herein. Those embodiments and other embodiments are described in further detail herein. [Brief explanation of the drawing]

[0017] [Figure 1A] This document presents exemplary process flows demonstrating various aspects of the disclosure through various embodiments. [Figure 1B] Figure 1A shows a block diagram of an exemplary environment that may implement the embodiments disclosed herein, including various components. [Figure 2] A flowchart illustrates exemplary methods for identifying locally corresponding candidate correction pairs on a client device, according to various embodiments. [Figure 3]A flowchart illustrates exemplary methods, in various embodiments, for remote systems to remotely determine whether a given corresponding candidate correction pair corresponds to an actual correction pair, and for updating a global automatic speech recognition (ASR) model based on the determination that the given corresponding candidate correction pair corresponds to an actual correction pair. [Figure 4] A flowchart illustrates exemplary methods for generating corresponding updates to a global automatic speech recognition (ASR) model locally on a client device, using various embodiments. [Figure 5] A flowchart illustrates exemplary methods for remotely updating a global automatic speech recognition (ASR) model in a remote system, using various embodiments. [Figure 6A] This document presents various user interfaces illustrating exemplary user interactions with client devices in various embodiments. [Figure 6B] This document presents various user interfaces illustrating exemplary user interactions with client devices in various embodiments. [Figure 7] This document illustrates exemplary architectures of computing devices in various embodiments. [Modes for carrying out the invention]

[0018] Referring here to Figure 1A, an exemplary process flow illustrating various aspects of the present disclosure is shown. The client device 110 is shown in Figure 1A and includes components enclosed within the frame of Figure 1A representing the client device 110. The on-device automatic speech recognition (ASR) engine 122 may receive audio data 110A, which corresponds to the voice utterance of the user of the client device 110 and is generated via one or more microphones of the client device 110. The on-device ASR engine 122 may process the audio data 110A using an on-device ASR model 122A stored in the on-device storage 111 of the client device 110 (e.g., random access memory (RAM) and / or other types of volatile memory or storage devices) to generate one or more predictive outputs 122B. Furthermore, the on-device ASR engine 122 may generate one or more predictive text segments 122C based on one or more predictive outputs 122B.

[0019] For example, when the on-device ASR model 122A is an end-to-end speech recognition model, the on-device ASR engine 122 may use the on-device ASR model 122A to generate one or more predicted text segments 122C (for example, one or more predicted outputs 122B may correspond to one or more predicted text segments 122C). For example, the on-device ASR model 122A may be an end-to-end model used to generate one or more predicted text segments 122C per string (or per different token) as one or more predicted outputs 122B. One non-restrictive example of such an end-to-end model used to generate one or more predicted text segments 122C per character is a recurrent neural network transducer (RNN-T) model. The RNN-T model is a type of sequence-to-sequence model that does not use an attention mechanism. Unlike most inter-sequence models that typically require processing the entire input sequence (e.g., audio data waveform, Mel-frequency cepstrum coefficients (MFCCs), or other representations of audio data 110A) to generate one or more predicted text segments 122C, an RNN-T model can be used to process input samples sequentially and stream output symbols (e.g., alphabetic characters). Also, for example, if the on-device ASR model 122A is not an end-to-end speech recognition model, the on-device ASR engine 122 may instead generate one or more predicted outputs 122B and / or other representations, such as one or more predicted phonemes, and based on one or more predicted outputs 122B, generate one or more predicted outputs 122C. For example, using such a model, one or more predicted phonemes and / or other representations may be utilized by the on-device ASR engine 122 to determine one or more predicted text segments 122C that fit one or more predicted phonemes. In doing so, the on-device ASR engine 122 may optionally use a decoding graph, glossary, and / or other resources.

[0020] Furthermore, the rendering engine 124 can visually render one or more predictive text segments 122C for presentation to the user of the client device 110 via the display, and optionally, it can audibly render one or more predictive text segments 122C for presentation to the user of the client device 110 via one or more speakers. In various embodiments, in addition to the speech utterance initially provided by the user of the client device 110, further input 110B may be received by the client device 110. Further input 110B may include, for example, touch input and / or type input to the display of the client device 110, and / or additional speech utterances captured in additional audio data generated by one or more microphones of the client device 110. In these embodiments, the correction engine 126 may analyze the further input 110B to determine whether the further input 110B corrects a portion of one or more predictive text segments 122C to one or more alternative text segments 126A. Furthermore, assuming that an additional input 110B corrects part of one or more predicted text segments 122C to one or more alternative text segments 126A, the correction pair engine 128 may generate a correction pair 128A that includes at least part of the corrected one or more predicted text segments 122C and one or more alternative text segments 126A. Furthermore, the correction pair engine 128 may store the correction pair 128A in the on-device storage 111 of the client device 110, and the audio data 110A may be stored in the on-device storage 111 of the client device 110 (and optionally stored in association with the correction pair 128A). Furthermore, the correction pair engine 128 may cause the correction pair 128A to be sent to a remote system 160, in which case the audio data 110A is not sent to the remote system (and optionally sent in response to the client device 110 determining that the number of occurrences of the correction pair 128 detected has reached a threshold).However, even though a part of one or more predicted text segments 122C is modified to one or more alternative text segments 126A by further input 110B, the correction pair 128A may not yet be regarded as an actual correction pair and may be referred to as a candidate correction pair.

[0021] In various embodiments, the correction identification engine 162 may store correction pair 128A and one or more additional correction pairs 180A received from one or more additional client devices 180 in the correction pair database 162A, along with instructions from the client device 110 that provided the correction pair 128A (for example, a list of candidate correction pairs is stored in the correction pair database 162A and indexed based on different text segments contained in correction pairs that can be distributed to multiple client devices). Furthermore, the correction identification engine 162 may analyze correction pair 128A and one or more additional correction pairs 180A to determine whether any of the correction pairs (e.g., those stored in the correction pair database 162A) correspond to an actual correction pair. In some variations of those embodiments, the correction identification engine 162 may determine that correction pair 128A is an actual correction pair based on the number of occurrences of correction pair 128A received from client device 110 and other client devices (e.g., from one or more of the additional client devices 180) reaching a threshold amount. For example, as will be explained in more detail with respect to Figure 6A, suppose that correction pair 128A contains a portion of the predicted text segment corresponding to “covet,” and that “covet” is subsequently corrected to the alternative text segment for “COVID” (e.g., correction pair [covet,COVID]). In this example, we further assume that thousands of correction pairs (e.g., from correction pair 128A and one or more correction pairs 180A) indicating that “covet” has been corrected to “COVID” are identified by the correction identification engine 162, and that these thousands of occurrences satisfy the threshold amount of occurrences required to consider correction pair [covet,COVID] as an actual correction pair. Thus, in this example, the correction identification engine 162 may consider correction pair [covet,COVID] as an actual correction pair indicating a common ASR misrecognition across a population of client devices.

[0022] In contrast, as will be explained in more detail with respect to Figure 6B, assume that correction pair 128A contains a portion of the predicted text segment corresponding to “covet,” and that “covet” is subsequently corrected to the alternative text segment for “an illness” (e.g., correction pair [covet,an illness]). In this example, further assume that only a few hundred correction pairs (e.g., from correction pair 128A and one or more correction pairs 180A) indicating that “covet” is corrected to “an illness” are identified by the correction identification engine 162, but that the number of occurrences of these hundreds did not meet the threshold amount for considering correction pair [covet,an illness] to be an actual correction pair. Therefore, in this example, the correction identification engine 162 may not consider correction pair [covet,an illness] to be an actual correction pair indicating a common ASR misrecognition across the entire population of client devices. Rather, in this example, correction pair [covet,an illness] may simply indicate that the user has changed their mind about the speech utterance initially provided, and this correction may be based on further input for reasons other than ASR misrecognition.

[0023] In additional or alternative variations of those embodiments, the correction identification engine 162 may operate in conjunction with the query activity engine 164 to determine whether a correction pair 128A is an actual correction pair based on query activity (e.g., accessible via a query activity database 164A). For example, assume again that correction pair 128A includes a portion of a predicted text segment corresponding to "covet" and that "covet" was subsequently modified to an alternative text segment of "COVID" (e.g., correction pair [covet, COVID]). In this example, further assume that thousands of queries including the alternative text segment of "COVID" were received within a threshold period and that these queries indicate a spike in queries including "COVID". Thus, in this example, the correction identification engine 162 may utilize the query spike detected by the query activity engine 164 to determine that the alternative text segment of "COVID" may be a newly introduced term to the population of users of the client device that provided the correction pair [covet, COVID] to the remote system 160. In various embodiments, any client device (e.g., one or more of client devices 110 and / or additional client devices 180) may transmit any corresponding candidate correction pair determined based on queries submitted at each client device for inclusion in a subset.

[0024] In various embodiments, assuming that the correction identification engine 162 has determined that correction pair 128A actually corresponds to an actual correction pair, the client device identification engine 166 may identify a subset of client devices that provided the occurrence of correction pair 128A. For example, the client device identification engine 166 may access the correction pair database 162A to identify any client devices (if any) that previously provided correction pair 128A to be included in the subset. In particular, the client device identification engine 166 may perform this identification when one or more client devices check in to the remote system 160 in a distributed and periodic manner (e.g., once a day, once a week, etc.) for training. The subset of client devices that provided the occurrence of correction pair 128A may be used by the remote system 160 to update the global ASR model 168A, which is a remote-based counterpart of the on-device ASR model 122A (and one or more corresponding on-device ASR models among the additional client devices 180). The remote training engine 168 can send instructions to each of the subset of client devices to cause each client device to generate an update for the global ASR model 168A and regarding the correction pair 128A. For example, suppose client device 110 is identified to be included in the subset based on providing an instance of the correction pair 128A, and the remote system 160 determines that the correction pair 128A corresponds to the actual correction pair. As a result, the remote training engine 168 can generate and send instructions to client device 110 to generate an update for the global ASR model 168A and regarding the correction pair 128A.

[0025] In the client device 110, the update engine 130 may receive instructions to generate an update for the global ASR model 168A and for the correction pair 128A. The update engine 130 may retrieve at least a portion of the correction pair 128A and locally processed audio data 110A from the on-device storage 111 of the client device 110 to generate a portion of one or more text segments contained in the correction pair 128A. Furthermore, the update engine 130 may use the on-device ASR model 122A to process the audio data 110A again and generate an update 130A based on the processing of the audio data 110A. In addition, the update engine 130 may send an update 120A to the remote system 160 for use by the remote training engine 168 when updating the global ASR model 168A. Similarly, one or more additional client devices 180 identified for inclusion in the subset may generate a corresponding update 180B in the same or similar manner and send the corresponding update 180B to the remote system 160 for use by the remote training engine 168 when updating the global ASR model 168A.

[0026] In some embodiments, when generating update 130A, the update engine 130 may process audio data 110A using an on-device ASR model 122A to generate one or more additional predictive text segments in the same or similar manner as described above (for example, with respect to the on-device ASR engine 122). However, instead of the one or more additional predictive text segments being at least visually rendered for presentation to the user of the client device (for example, via the rendering engine 124, as described with respect to one or more predictive text segments), the update engine 130 may compare some of the one or more additional predictive text segments generated based on subsequent processing of the audio data 110A with one or more alternative text segments of the correction pair 128A to generate a gradient, for example, using supervised learning techniques. In some variations of those embodiments, the gradient may correspond to update 130A sent to a remote system 160. In additional or alternative variations of these embodiments, the update engine 130 may update the on-device ASR model locally and on a gradient basis at the client device 110, and may utilize one or more updated on-device weights of the updated on-device ASR model 122A as update 130A.

[0027] In additional or alternative embodiments, when generating update 130A, the update engine 130 may use a subset of the machine learning (ML) layers of the on-device ASR model 122A to process audio data 110A and generate a representation of audio 110A, performing various forward and / or backward passes across the subset of the ML layers of the on-device ASR model 122A. The ML layers of the on-device ASR model 122A may include, for example, one or more input layers, one or more coupling layers, one or more encoding layers, one or more coupling layers, one or more output layers, and / or other layers. Thus, when processing audio data using a subset of the ML layers of the on-device ASR model 122A, the update engine 130 may generate a representation across one or more of the input layers, or a representation across one or more of the coupling or encoding layers, without generating anything across one or more of the decoding or output layers. In these embodiments, a subset of ML layers may be updated based on processing, while other layers remain fixed (for example, another ML layer of the on-device ASR model 122A that was not used when processing audio data 110A is fixed), and the difference between the subset of ML layers before updating and the subset of ML layers after updating (or the difference between one or more on-device weights associated with the subset of ML layers before updating and the subset of ML layers after updating) may be used as update 130A.

[0028] In various embodiments, in response to receiving update 130A from client device 110 and one or more corresponding updates 180A from one or more of the additional client devices 180, the remote training engine 162 may use these updates to update the global ASR model 168A with respect to the correction pair 128. Specifically, when updating the global ASR model 168A (for example, by using backpropagation or another technique to update the ASR model(s)), it updates one or more global weights of the global speech recognition model 168A. Furthermore, as indicated by 170A, the update distribution engine 170 may, in response to one or more conditions being met, provide client device 110 and / or one or more of the additional client devices 180 with one or more updated global weights of the updated global ASR model and / or the updated global ASR model. One or more conditions may include, for example, a threshold period and / or threshold amount for updates to the global ASR model 168A since any updated weight(s) and / or the updated global ASR model was last distributed, the measured improvement of the updated global ASR model, and / or the passage of a threshold period since any updated weight(s) and / or the updated global ASR model was last distributed, and / or other conditions. When one or more updated global weights of the updated global ASR model and / or the updated global ASR model are distributed to the client device 110, the client device 110 may replace the on-device ASR model 122A (or its on-device ASR model(s)) with the updated global ASR model (or its updated global weight(s)).

[0029] Referring now to Figure 1B, the client device 110 is shown, in this embodiment the on-device ASR engine 122 of Figure 1A is included as part of (or communicating with) the automated assistant client 140. The on-device ASR model 122A is also shown to interface with the on-device ASR engine 122. Another component of the client device 250 is not shown in Figure 1B for simplification. Figure 1B shows an example of how the on-device ASR engine 122 and the on-device ASR model 122A can be used to generate predictive text segments that are used by the automated assistant client 140 when performing various actions.

[0030] The client device 110 in Figure 1B is shown with one or more microphones 151, one or more speakers 152, one or more cameras and / or other vision components 153, and one or more displays 154 (e.g., touch-sensitive displays). The client device 110 selectively runs at least the Automated Assistant Client 140. In the example in Figure 1B, the Automated Assistant Client 140 includes an on-device ASR engine 122, an on-device Natural Language Understanding (NLU) engine 144, and an on-device Fulfillment engine 145. The Automated Assistant Client 140 further includes a voice capture engine 141 and a visual capture engine 142. The Automated Assistant Client 140 may include additional and / or alternative engines such as a voice activity detector (VAD), an endpoint detector, a hotword detector, and / or other engines.

[0031] One or more cloud-based automated assistant components 191 may optionally be implemented on one or more computing systems (collectively referred to as “cloud” computing systems), which are communicably connected to the client device 110 via one or more local area networks and / or wide area networks (e.g., the Internet), generally indicated as 190. The cloud-based automated assistant components 191 may be implemented, for example, via high-performance servers or clusters of high-performance servers. In various embodiments, an instance 140 of the automated assistant client may also optionally form something like a logical instance of the automated assistant 195 from the user's perspective through interaction with one or more cloud-based automated assistant components 191, and using this automated assistance, the user can perform human-computer interactions (e.g., voice interactions, gesture-based interactions, and / or touch-based interactions).

[0032] The client device 110 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in the user's vehicle (e.g., an in-car communication system, an in-car entertainment system, an in-car navigation system), a standalone interactive speaker, a smart home appliance such as a smart TV (or a standard TV with a network-connected dongle with an automatic assistant function), and / or a user's wearable device including a computing device (e.g., a user's wristwatch with a computing device, a user's glasses with a computing device, a virtual reality or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0033] The vision component(s) 153 can take various forms, such as a monographic camera, a stereographic camera, a LiDAR component (or other laser-based component(s)), or a radar component. One or more vision components 153 may be used, for example, by the visual capture engine 142 to capture vision frames (e.g., image frames, laser-based vision frames) of the environment in which the client device 110 is deployed. In some embodiments, such vision frames(s) can be used to determine whether a user is near the client device 110 and / or the distance of the user (e.g., the user's face) to the client device. Such determination(s) can be used, for example, to determine whether to activate the on-device ML engine 122.

[0034] The speech capture engine 141 can be configured to capture user speech and / or other audio data captured via microphones 151. As described herein, such audio data can be made available (optionally after preprocessing) by the on-device ASR engine 122. For example, the on-device ASR engine 122 can use the on-device ASR model 122A to process the audio data capturing speech utterances to generate predicted text segments that are expected to correspond to the speech utterances. The on-device NLU engine 144 performs on-device natural language understanding on the predicted text segments to generate NLU data. The on-device NLU engine 144 can optionally use one or more on-device NLU models (not shown in Figure 1B for simplicity) when generating NLU data. The NLU data may include, for example, intents (optional) corresponding to the speech utterances, and optionally, parameters (optional) of the intents (optional) (e.g., slot values). Furthermore, the on-device fulfillment engine 145 generates fulfillment data using the NLU data. The on-device fulfillment engine 145 may optionally utilize one or more on-device fulfillment models and / or rules (not shown in Figure 1B for simplicity) when generating fulfillment data. The fulfillment data can define local and / or remote responses (e.g., answers) to voice utterances, voice utterance-based interactions with locally installed applications, voice utterance-based commands to be sent to Internet of Things (IoT) devices (directly or via corresponding remote systems), and / or other resolution actions to be performed based on voice utterances. The fulfillment data is then provided for local and / or remote performance / execution of the actions determined to resolve the voice utterances.Execution may include, for example, rendering local and / or remote responses (e.g., visually and / or audibly, using an optional text-to-speech module), interacting with locally installed applications, sending commands to IoT devices, and / or other actions.

[0035] The display(s) 154 can be used to visually render streaming predictive text segments generated based on predictive output from the on-device ASR engine 122. The display(s) 154 may also be one of the user interface output components(s) to which the visual(s) portion of the response from the automated assistant client 140 is rendered.

[0036] In some embodiments, the cloud-based automated assistant component(s) 191 may include a remote ASR engine 192 for speech recognition, a remote NLU engine 193 for natural language understanding, and / or a remote fulfillment engine 194 for generating fulfillment data. It may also optionally include a remote execution module, which performs remote execution based on locally or remotely determined fulfillment data. Additional and / or alternative remote engines may be included. As described herein, in various embodiments, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be preferred because they enable at least a reduction in latency and / or network usage when resolving speech utterances (because client-server round trips are not required to resolve speech utterances). However, one or more cloud-based automated assistant components(s) 191 may be used, at least selectively. For example, such components(s) can be used in parallel with on-device components(s), and their output can be used when processing by local components(s) fails. For example, the on-device fulfillment engine 145 may fail in certain circumstances (for example, because the resources of the client device 110 are relatively limited), in which case the remote fulfillment engine 194 can use the more robust resources of the cloud to generate the fulfillment data. The remote fulfillment engine 194 can operate in parallel with the on-device fulfillment engine 145, and its results can be used when on-device fulfillment fails, or invoked in response to a determination that the on-device fulfillment engine 145 has failed.

[0037] In various embodiments, the NLU engines 144 and / or 193 may generate an annotated output that includes one or more annotations of the recognized text and one or more (e.g., all) of the terms of the natural language input. In some embodiments, the NLU engines 144 and / or 193 may be configured to identify and annotate various types of grammatical information in the natural language input. For example, the NLU engines 144 and / or 193 may include a morpheme module that can divide individual words into morphemes and / or annotate the morphemes with, for example, their class. The NLU engines 144 and / or 193 may also include part-of-speech taggers configured to annotate terms with their grammatical roles. Additionally and / or alternatively, in some embodiments, the NLU engines 144 and / or 193 may include dependent parsers configured to determine syntactic relationships between terms in the natural language input.

[0038] In some embodiments, the NLU engines 144 and / or 193 may additionally and / or alternatively include entity taggers configured to annotate entity references within one or more segments, such as references to people (e.g., characters in literary works, celebrities, public figures, etc.), organizations, locations (actual and fictional locations), etc. In some implementations, the NLU engines 144 and / or 193 may additionally and / or alternatively include coreference resolvers (not shown) configured to group, or “cluster,” references to the same entity based on one or more context queues. In some embodiments, one or more components of the NLU engines 144 and / or 193 may depend on annotations from one or more other components of the NLU engines 144 and / or 193.

[0039] In some embodiments, the NLU engines 144 and / or 193 may also include an intent matcher configured to determine the intent of a user interacting with the automated assistant client 195. The intent matcher can determine the user's intent using a variety of techniques. In some embodiments, the intent matcher may have access to one or more local and / or remote data structures, for example, multiple mappings between grammars and response intents. For example, the grammars included in the mappings may be selectable and / or learned over time and may represent the user's typical intent. For example, one grammar, "Play <artist>", may map to the intent to invoke a response action in which music by <artist> is played on the client device 110. Another grammar, "[weather|forecast] today", may match user queries such as "What's the weather like today?" and "What's the forecast for today?". In addition to grammars, or instead, in some embodiments, the intent matcher may use one or more trained machine learning models, either alone or in combination with one or more grammars. These trained machine learning models can be trained to identify intent by, for example, embedding recognized text from speech utterances into a low-dimensional space, and then determining which alternative embedding (and therefore intent) is the closest approximation using techniques such as Euclidean distance or cosine similarity. Some grammars have slots (e.g., <artist>) that can be filled with slot values ​​(or "parameters"), as seen in the exemplary grammar "Play <artist>" above. Slot values ​​can be determined in various ways. Often, users actively provide slot values. For example, in the grammar "Order a <topping> pizza," a user might say the phrase "Order a sausage pizza," in which case the slot <topping> is automatically filled. Another slot value(s) can be estimated based on, for example, the user's location, the content currently being rendered, the user's preferences, and / or other cues(s).

[0040] In some embodiments, the fulfillment engine(s) 145 may be configured to receive predicted / estimated intents output by the NLU engine(s) 144 and / or 193, as well as any associated slot values, and to perform (or "resolve") the intent. In various embodiments, the fulfillment (or "resolve") of a user's intent may include various fulfillment information (also called fulfillment data) that may be generated / obtained by the fulfillment engine, for example. This may include determining local and / or remote responses (e.g., answers) to a speech utterance, interactions with locally installed applications(s) based on the speech utterance, commands based on the speech utterance to be sent (directly or via corresponding remote systems(s)) to Internet of Things (IoT) devices(s), and / or other resolution actions(s) based on the speech utterance. The on-device fulfillment engine(s) 145 can then initiate local and / or remote performance / execution of the determined actions(s) to resolve the speech utterance.

[0041] Referring here to Figure 2, a flowchart illustrating an exemplary method 200 for identifying locally corresponding candidate correction pairs on a client device is shown. For convenience, the operation of method 200 is described with reference to the system in which it operates. The system of method 200 includes one or more processors and / or other components of a client device (e.g., client device 110 in Figure 1A and / or Figure 1B, client device 610 in Figure 6A and / or Figure 6B, computing device 710 in Figure 7, and / or other client devices). Furthermore, the operation of method 200 is shown in a particular order, but this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0042] In block 252, the system receives audio data capturing the user's speech utterances on the client device via one or more microphones on the client device. In block 254, the system processes the audio data using an on-device ASR model stored in the client device's on-device storage to generate predictive text segments that are expected to correspond to the speech utterances. In block 256, the system visually renders the predictive text segments for presentation to the user on the client device's display. The system returns to block 252, continuing to receive additional audio data, if any, and processing any additional audio data capturing any additional speech utterances from the user using the on-device ASR model, and continuing to visually render any additional predictive text segments for presentation to the user on the client device's display in a streaming manner.

[0043] In block 258, the system determines whether it has received further user interface input, including a modification of part of the predictive text segment to an alternative text segment. In some embodiments, the further user interface input may be typed or touched input to part of the predictive text segment that is visually rendered on the client device's display (as described, for example, with respect to Figures 6A and 6B). In additional or alternative embodiments, the further user interface input may be additional voice utterances containing commands to the system to modify part of the predictive text segment to an alternative text segment (as determined, for example, using the various components shown in Figure 1B). If, in an iteration of block 258, the system determines that it has not received further user interface input, including a modification of part of the predictive text segment to an alternative text segment, then in block 258, the system may continue to monitor for further user interface input. In particular, in block 258, the system may monitor for modifications to any text segment that is visually rendered for presentation to the user.

[0044] In an iteration of block 258, if the system determines that it has received further user interface input, including a modification of part of the predicted text segment to an alternative text segment, the system may proceed to block 260. In block 260, the system stores in the client device's on-device storage (1) part of the predicted text segment and alternative text segment that are considered to be the corresponding candidate correction pair, and (2) audio data. In block 262, the system transmits the corresponding candidate correction pair to the remote system, but does not transmit the audio data. When transmitting the corresponding candidate correction pair to the remote system, the system may optionally use one or more techniques to obfuscate the corresponding candidate correction pair for privacy reasons. For example, the system may tokenize the corresponding candidate correction pair in an opaque manner so that the corresponding candidate correction pair is interpretable by the remote system but not by the user associated with the remote system. In some embodiments, the system may transmit the corresponding candidate correction pair to the remote system only if the number of occurrences of the candidate correction pair detected on the client device reaches a threshold amount. In additional or alternative embodiments, the system may send a corresponding candidate correction pair to a remote system only if the candidate correction pair is included in the list of candidate correction pairs. This list of candidate correction pairs includes candidate correction pairs received from the remote system and detected by another client device.

[0045] Referring here to Figure 3, a flowchart illustrating an exemplary method 300 is shown, in which a remote system remotely determines whether a given corresponding candidate correction pair corresponds to an actual correction pair, and the global automatic speech recognition (ASR) model is updated based on the determination that the given corresponding candidate correction pair corresponds to an actual correction pair. For convenience, the operation of method 300 is described with reference to the system in which the operation is performed. The system of method 300 includes one or more processors and / or other components of a remote system (e.g., remote system 160 in Figure 1A, cloud-based automated assistant component(s) 191 in Figure 1B, computing device 710 in Figure 7, one or more high-performance servers, and / or other computing devices). Furthermore, the operation of method 300 is shown in a particular order, but this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0046] In block 352, the system receives corresponding candidate correction pairs from multiple client devices, each of which includes (1) a corresponding portion of a predicted text segment generated using a corresponding on-device ASR model based on processing corresponding audio data locally generated on one of the multiple client devices, and (2) a corresponding alternative text segment generated based on corresponding modifications to a portion of the predicted text segment locally on one of the multiple client devices. Furthermore, the system may store the corresponding candidate correction pairs received from multiple client devices on one or more remote storage devices accessible by the system. The system may index the corresponding candidate correction pairs and identify the client devices that previously provided the corresponding candidate correction pairs.

[0047] In block 354, the system determines whether a given corresponding candidate correction pair corresponds to an actual correction pair (for example, as described with respect to the correction identification engine 162 in Figure 1A). In some embodiments, the system may determine whether a given corresponding candidate correction pair corresponds to an actual correction pair based on whether multiple client devices have received that the number of occurrences of a given corresponding candidate correction pair has reached a threshold amount. In additional or alternative embodiments, the system may determine whether a given corresponding candidate correction pair corresponds to an actual correction pair based on query activity associated with multiple client devices. In some variations of these embodiments, query activity on a given client device among multiple client devices may be used in determining whether a given corresponding candidate correction pair corresponds to an actual correction pair, even if the given client device does not provide the system with occurrences of a given corresponding candidate correction pair. In some embodiments, in response to determining that a given corresponding candidate correction pair corresponds to an actual correction pair, the system may use one or more techniques to verify that the given corresponding candidate correction pair actually corresponds to an actual correction pair (e.g., based on the degree of acoustic similarity between a portion of the corresponding predicted text segment and an alternative text segment, based on the Levenshtein edit distance between a portion of the corresponding predicted text segment and an alternative text segment, and / or using other techniques).

[0048] In an iteration of block 354, if the system determines that a given corresponding candidate correction pair does not correspond to an actual correction pair, the system returns to block 352 and may continue to receive corresponding candidate correction pairs from multiple client devices. In an iteration of block 354, if the system determines that a given corresponding candidate correction pair does correspond to an actual correction pair, the system may proceed to block 356. In block 356, the system identifies a subset of multiple client devices that provided a given corresponding candidate correction pair (for example, as described with respect to the client device identification engine 166 in Figure 1).

[0049] In block 358, the system ensures that a global ASR model, which is a global-based counterpart of the corresponding on-device ASR models for multiple client devices, is updated in a distributed manner using a subset of client devices that have provided a given corresponding candidate correction. In other words, for method 200 in Figure 2, an exemplary technique for generating corresponding candidate correction pairs is described. Furthermore, for method 300 in Figure 3, an exemplary technique is described for determining whether a given corresponding candidate correction pair actually corresponds to an actual correction pair that exhibits ASR misrecognition across the entire population of client devices, rather than simply considering a corresponding candidate correction pair generated by a single client device.

[0050] Referring here to Figure 4, a flowchart illustrating an exemplary method 400 for generating a corresponding update of a global automatic speech recognition (ASR) model locally on a client device is shown. For convenience, the operation of method 400 is described with reference to the system that performs the operation. The system of method 400 includes one or more processors and / or other components of a client device (e.g., client device 110 in Figure 1A and / or Figure 1B, client device 610 in Figure 6A and / or Figure 6B, computing device 710 in Figure 7, and / or other client devices). Furthermore, the operation of method 400 is shown in a particular order, but this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0051] In block 452, the system receives instructions from a remote system that, on a given client device of a given user, the global ASR model, which is a global-based counterpart of the corresponding on-device ASR model stored in the given on-device storage of the given client device, should be updated in a distributed manner based on the corresponding candidate correction pairs stored in the given client device's on-device storage. In block 454, the system retrieves at least a portion (e.g., alternative text segments) of the corresponding candidate correction pairs from the given on-device storage of the given client device. In block 456, the system retrieves the corresponding audio data stored in association with the corresponding candidate correction pairs from the given on-device storage of the given client device.

[0052] In block 458, the system processes the corresponding audio data associated with the corresponding candidate correction pair using the corresponding on-device ASR model stored locally in a given on-device storage of a given client device. In block 460, based on processing the corresponding audio data, the system generates a corresponding update of the global ASR model (as described, for example, with respect to the update engine 130 in Figure 1A).

[0053] In block 462, the system determines whether an additional corresponding candidate correction pair is stored in the given on-device storage of a given client device. If, in an iteration of block 462, the system determines that an additional corresponding candidate correction pair is stored in the given on-device storage of a given client device, the system may return to block 454 and generate an additional corresponding update for the additional corresponding candidate correction pair. In other words, the system may generate a corresponding update for each instance of the corresponding correction stored locally in the given on-device storage of a given client device. If, in an iteration of block 462, the system determines that no corresponding additional corresponding candidate correction pair is stored in the given on-device storage of a given client device, the system may proceed to block 464.

[0054] In block 464, the system sends one or more corresponding updates generated locally on a given client device to a remote system. In other words, for method 200 in Figure 2, an exemplary technique for generating corresponding candidate correction pairs is described. Furthermore, for method 300 in Figure 3, an exemplary technique is described for determining whether a given corresponding candidate correction pair actually corresponds to an actual correction pair that indicates ASR misrecognition across a whole group of client devices, rather than simply considering a corresponding candidate correction pair generated on a single client device. Furthermore, for method 400 in Figure 4, an exemplary method is shown in which a given client device that has provided a given corresponding candidate correction pair to the system generates a corresponding update with respect to the given corresponding candidate correction pair.

[0055] Referring here to Figure 5, a flowchart illustrating an exemplary method 500 for remotely updating a global automatic speech recognition (ASR) model in a remote system is shown. For convenience, the operation of method 500 is described with reference to the system in which the operation is performed. The system of method 500 includes one or more processors and / or other components of a remote system (e.g., remote system 160 in Figure 1A, cloud-based automated assistant component(s) 191 in Figure 1B, computing device 710 in Figure 7, one or more high-performance servers, and / or other computing devices). Furthermore, the operation of method 500 is shown in a particular order, but this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0056] In block 552, the system receives corresponding updates from multiple client devices, which are used to update the global ASR model with respect to the actual correction pairs. In block 554, the system updates the global ASR model based on the corresponding updates (for example, as described with respect to the remote training engine 168 in Figure 1A). In some embodiments, the system may wait until the corresponding updates are received from each of the multiple client devices (for example, those identified to be included in a subset in block 456 of method 300 in Figure 3) before updating the global ASR model based on the corresponding updates. In additional or alternative embodiments, once the corresponding updates are received from multiple client devices, the system may update the global ASR model based on those updates.

[0057] In block 556, the system determines whether one or more conditions are met. If, in an iteration of block 556, the system determines that one or more conditions are not met, the system may continue monitoring in block 556 for one or more conditions to be met. In particular, the system may continue updating the global ASR model while monitoring for one or more conditions to be met in block 556. If, in an iteration of block 556, the system determines that one or more conditions are met, the system may proceed to block 558. In block 558, the system sends one or more updated weights and / or the updated global ASR model of the updated global ASR model to multiple client devices.

[0058] In some embodiments, one or more conditions may be specific to the global ASR model and / or the remote system. In these embodiments, one or more conditions may include, for example, whether the global ASR model has been updated based on reaching a corresponding update threshold amount, whether the global ASR model has been updated over a threshold period, whether a threshold period has elapsed since any of the updated global ASR models were last distributed, whether improvements to the updated global ASR model have been measured, and / or other conditions. In additional or alternative embodiments, one or more conditions may be specific to one or more updated weights of the updated global ASR model and / or to each of the multiple client devices that receive the updated global ASR model. In these embodiments, one or more conditions may include, for example, whether a given client device (e.g., one of several client devices) is charging and / or has reached a charging threshold, whether a given client device is being held or used by a given user of a given client device, whether a given time associated with a given location of a given client device (e.g., determined using a given location sensor(s) of a given client device) is within a specific time range, and / or other conditions.

[0059] In other words, for Method 200 in Figure 2, an exemplary technique for generating corresponding candidate correction pairs is described. Furthermore, for Method 300 in Figure 3, an exemplary technique is described for determining whether a given corresponding candidate correction pair actually corresponds to an actual correction pair that exhibits ASR misrecognition across an entire population of client devices, rather than simply considering a corresponding candidate correction pair generated by a single client device. Furthermore, for Method 400 in Figure 4, an exemplary method is shown in which a given client device that has provided a given corresponding candidate correction pair to the system generates a corresponding update with respect to the given corresponding candidate correction pair. Furthermore, for Method 500 in Figure 5, an exemplary method is shown for updating and distributing a global ASR model to prevent and / or mitigate future occurrences of ASR misrecognition characterized by a given corresponding candidate correction pair.

[0060] Referring here to Figures 6A and 6B, various user interfaces illustrating exemplary user interactions with the client device 610 (e.g., instances of client device 110 in Figures 1A and 1B) are shown. The client device 610 in Figures 6A and 6B includes a touch-sensitive display screen 640 that displays and / or streams (i.e., in real time) predictive text segments corresponding to voice utterances provided by the user of the client device 610, according to embodiments disclosed herein. For convenience, actions performed by the client device 610 are described with reference to an automated assistant performing the actions (e.g., automated assistant 195 in Figure 1B).

[0061] The display screen 640 includes text response and / or editing elements 684, which enable the user to provide user input (e.g., touch input or type input) to generate, modify, delete, and / or replace terms(s) via a virtual keyboard. Furthermore, the display screen 640 also includes voice interface elements 685, which, when activated, enable the user to provide user input (e.g., voice input) to confirm actions performed by the client device 610, cancel actions performed by the client device 610, and / or provide voice utterances or additional voice utterances via one or more microphones. In some embodiments, audio data corresponding to voice utterances can be captured via one or more microphones, generating predictive text segments that can be visually rendered on the display screen 640 of the client device 610, and user input to correct the predictive text segments to alternative text segments may be touch input to the predictive text segments(s) included in the text response and / or editing elements 684 of the display screen 640 of the client device 510. In additional and / or alternative embodiments, user input for correcting a predicted text segment to an alternative text segment may be voice input. In some variations of these embodiments, voice input is received in response to touch input to the voice interface element 685, and voice input is received within a threshold period of user input while one or more microphones are activated without touch input to the voice interface element 685 and / or other voice input activation methods. Furthermore, in some embodiments, the display screen 640 also includes system interface elements 681, 682, and 683 which can interact with the user to cause the computing device 610 to perform one or more actions.

[0062] Furthermore, in some embodiments, voice utterances may include actions performed by the automated assistant using the client device 610. Some non-limiting examples of actions may include calling or dialing a phone number, sending a text or SMS message (for example, as shown in Figures 6A and 6B), sending an email, looking up contact information, requesting navigation information, sending a calendar invitation, controlling one or more IoT devices, and / or other actions that can be performed by the automated assistant operating on the client device 610.

[0063] Specifically, referring to Figure 6A, suppose the user of client device 610 provides the voice utterance 652, "I tested positive for COVID, so I will be absent from tomorrow's meeting," and the automated assistant visually renders the predictive text segment 654 (e.g., generated using the on-device ASR engine 122 in Figures 1A and 1B) as "I tested positive for covet, so I will be absent from tomorrow's meeting." In particular, the predictive text segment 654 contains an ASR misrecognition, i.e., "covet" instead of "COVID" as intended by the user. Therefore, suppose the user makes further user interface input to client device 610, such as touch input, and client device 610 corrects part of the predictive text segment 654 (e.g., "covet") to the alternative text segment (e.g., "COVID") as shown by 656A. As a result, the result "I tested positive for COVID, so I will be absent from tomorrow's meeting" is obtained as intended by the user. In this example, the automated assistant may generate a corresponding candidate correction pair [covet,COVID] based on further user interface input that corrects a portion of the predicted text segment 654 (e.g., "covet") to an alternative text segment (e.g., "COVID"), as shown by 656A. Furthermore, the automated assistant may store the candidate correction pair [covet,COVID] in the on-device memory of the client device 610, and optionally in association with audio data capturing the speech utterance 652. In addition, the automated assistant may send the corresponding candidate correction pair to a remote system (e.g., remote system 160 in Figure 1A) without sending the audio data capturing the speech utterance 652.This allows the remote system to determine whether a corresponding candidate correction pair corresponds to an actual correction pair, and if the remote system determines that a corresponding candidate correction pair [covet,COVID] corresponds to an actual correction pair (for example, as described with respect to the correction identification engine 162 in Figure 1A), it causes the client device 610 to generate a corresponding update that can be used when updating the global ASR model.

[0064] In some embodiments, the client device 610 may transmit the corresponding candidate correction pair [covet,COVID] to the remote system only if the number of occurrences of the candidate correction pair [covet,COVID] detected by the client device 610 reaches a threshold amount. In additional or alternative embodiments, the client device 610 may transmit the corresponding candidate correction pair [covet,COVID] to the remote system only if the candidate correction pair [covet,COVID] is included in a list of candidate correction pairs. This list of candidate correction pairs includes candidate correction pairs [covet,COVID] received from the remote system and detected by another client device.

[0065] In some embodiments, a predictive text segment 654, "I tested positive for COVID and will be absent from tomorrow's meeting," may be automatically populated in the text response and / or edit element 684 for user editing. The user can make inputs to the text response and / or edit element 684, as also shown in 656A (e.g., a cursor identifier), and correct, for example, "covet" to "COVID." In additional or alternative embodiments, the automated assistant can visually render a send selectable graphical element 661, an edit selectable graphical element 662, and / or cancel selectable graphical element 663. In some variations of those embodiments, some user inputs are made to the edit selectable graphical element 662 in order to populate the text response and / or edit element 684 with the predictive text segment 654, "I tested positive for COVID and will be absent from tomorrow's meeting," for user editing, thereby populating the text response and / or edit element 684.

[0066] In particular, the correction from "covet" to "COVID" for a portion of the predictive text segment 654 may only be utilized when the remote system generates a corresponding update to the global ASR model in response to determining that the correction is indeed a correction intended for the performance of the on-device ASR model of client device 610. Furthermore, the correction identified in Figure 6A is intended for the performance of the on-device ASR model (e.g., on-device ASR model 122A in Figures 1A and 1B). The remote system may determine that the correction is a correction intended for the performance of the on-device ASR model based on, for example, the number of occurrences of a given corresponding correction pair (e.g., corresponding correction pair [covet,COVID]) received from a population of client devices (e.g., client device 610 and other client devices) reaching a threshold amount, and / or a spike in query activity across the entire population of client devices associated with alternative text segment 656A (e.g., a spike in queries associated with "COVID" in the example in Figure 6A). Furthermore, verifying that the correction is intended to improve the performance of the on-device ASR model can be based, for example, on the similarity between a portion of the predicted text segment 654 (e.g., "covet") and the alternative text segment 656A (e.g., "COVID"). This similarity can be determined based on acoustic similarity, Levenshtein edit distance, and / or other techniques.

[0067] In contrast, referring particularly to Figure 6B, suppose the user instead provides further user interface input to the client device 610, such as touch input, and the client device 610 corrects part of the predicted text segment 654 (e.g., "covet") to an alternative text segment (e.g., "an illness"), as shown by 656B. The result is "I tested positive for illness and will be absent from tomorrow's meeting." In particular, in this example, the user of the client device 610 has simply changed their mind regarding the speech utterance 652 (e.g., by changing "COVID" to "an illness"), despite the ASR misrecognition. Nevertheless, the automated assistant may generate a corresponding candidate correction pair [covet,an illness] based on further user interface input correcting part of the predicted text segment 654 (e.g., "covet") to an alternative text segment (e.g., "an illness"), as shown by 656B. Furthermore, the automated assistant may store candidate correction pairs [covet, an illness] in the on-device memory of the client device 610, and optionally, in association with audio data capturing the speech utterance 652. Additionally, the automated assistant may transmit the corresponding candidate correction pairs to a remote system (e.g., the remote system 160 in Figure 1A) without transmitting the audio data capturing the speech utterance 652. This allows the remote system to determine whether the corresponding candidate correction pair corresponds to an actual correction pair, and if the remote system determines that the corresponding candidate correction pair [covet, an illness] corresponds to an actual correction pair (e.g., as described with respect to the correction identification engine 162 in Figure 1A), it causes the client device 610 to generate a corresponding update that can be used when updating the global ASR model.

[0068] In some embodiments, the client device 610 may transmit the corresponding candidate correction pair [covet, an illness] to the remote system only if the number of occurrences of the candidate correction pair [covet, an illness] detected by the client device 610 reaches a threshold amount. In additional or alternative embodiments, the client device 610 may transmit the corresponding candidate correction pair [covet, an illness] to the remote system only if the candidate correction pair [covet, an illness] is included in a list of candidate correction pairs. This list of candidate correction pairs includes candidate correction pairs [covet, an illness] received from the remote system and detected by another client device.

[0069] However, in this example, it is unlikely that the remote system would determine that candidate correction pair [covet,an illness] corresponds to the actual correction pair, based, for example, on the number of occurrences of a given corresponding correction pair (e.g., corresponding correction pair [covet,an illness]) received from the remote system and a group of client devices (e.g., client device 610 and other client devices) reaching a threshold, and / or on spikes in query activity across the entire group of client devices associated with alternative text segment 656A (e.g., spikes in queries associated with "COVID" in the example in Figure 6A). In other words, it is unlikely that the number of occurrences of candidate correction pair [covet,an illness] received by the remote system will reach a threshold because users often change their minds in various ways. Furthermore, even if the number of occurrences of candidate correction pair [covet,an illness] received by the remote system does reach a threshold, it is unlikely that this correction will be validated as a correction aimed at on-device ASR performance. This is because, for example, there is a lack of acoustic similarity between a portion of the predicted text segment 654 (e.g., "covet") and the alternative text segment 656B (e.g., "an illness").

[0070] Figures 6A and 6B illustrate specific examples, but it should be understood that these examples are provided for illustrative purposes only and are not intended to be limiting. Rather, it should be understood that Figures 6A and 6B are provided to illustrate various user interactions in which the corresponding candidate correction pairs may be generated locally on the client device 610.

[0071] Referring now to Figure 7, a block diagram of an exemplary computing device 710 that may be optionally used to perform one or more embodiments of the technology described herein is shown. In some embodiments, one or more of the client device, cloud-based automated assistant components, and / or other components may include one or more components of the exemplary computing device 710.

[0072] The computing device 710 typically includes at least one processor 714 that communicates with several peripheral devices via a bus subsystem 712. These peripheral devices may include, for example, a storage subsystem 724 including a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices enable user interaction with the computing device 710. The network interface subsystem 716 provides an interface to an external network and is coupled to a corresponding interface device of another computing device.

[0073] The user interface input device 722 may include a keyboard, a pointing device such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen integrated into the display, an audio input device such as a speech recognition system, a microphone, and / or other types of input devices. Generally, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into the computing device 710 or onto a communication network.

[0074] The user interface output device 720 may include a non-visual display such as a display subsystem, printer, fax machine, or audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT) or liquid crystal display (LCD), a projection device, or several other mechanisms for creating visible images. The display subsystem may also provide a non-visual display via an audio output device, etc. In general, the use of the term “output device” is intended to include all possible types of devices and methods for outputting information from the computing device 710 to a user or other machine or computing device.

[0075] The storage subsystem 724 stores programming and data structures that provide some or all of the functionality of the modules described herein. For example, the storage subsystem 724 may include logic for performing selected embodiments of the methods disclosed herein, or logic for implementing the various components shown in Figures 1A and 1B.

[0076] These software modules generally run on processor 714 alone or in combination with other processors. The memory 725 used by the storage subsystem 724 may include several memories, including main random access memory (RAM) 730 for storing instructions and data during program execution, and read-only memory (ROM) 732 for storing fixed instructions. The file storage subsystem 726 can provide persistent storage for program files and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular embodiment may be stored in the storage subsystem 724 or by the file storage subsystem 726, or on another machine accessible by processor 714.

[0077] The bus subsystem 712 provides a mechanism for various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is schematically shown as a single bus, alternative embodiments of the bus subsystem may use multiple buses.

[0078] The computing device 710 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing systems or computing devices. Because computers and networks are constantly changing, the description of the computing device 710 shown in Figure 7 is intended only as a specific example for the purpose of illustrating several embodiments. Many other configurations of the computing device 710 may have more or fewer components than the computing device shown in Figure 7.

[0079] Where the systems described herein may collect or otherwise monitor personal information about a user, or may utilize personal and / or monitoring information, the user may be provided with the opportunity to control whether the program or feature collects user information (e.g., information about the user's social networks, social behavior or activities, occupation, user preferences, or current geographical location), or whether and / or how it receives content that may be more relevant to the user from a content server. Furthermore, certain data may be processed in one or more ways to remove personally identifiable information before being stored or used. For example, a user's identity may be processed in such a way that information that could identify the user personally cannot be determined, or if geographical location information (such as city, zip code, or state level) is obtained, the user's geographical location may be generalized in such a way that the user's specific geographical location cannot be determined. Thus, the user may control how information about them is collected and / or how that information is used.

[0080] In some embodiments, methods performed by one or more processors of a remote system are provided herein and include receiving corresponding candidate correction pairs from a plurality of client devices. Each corresponding candidate correction pair includes a corresponding portion of a corresponding predictive text segment generated using a corresponding on-device automatic speech recognition (ASR) model based on local processing of corresponding audio data in one of the plurality of client devices, and a corresponding alternative text segment generated locally in one of the plurality of client devices based on corresponding modifications to the corresponding portion of the corresponding predictive text segment that resulted in the corresponding alternative text segment. The method further includes determining whether a given corresponding candidate correction pair among the corresponding candidate correction pairs is the corresponding actual correction pair based on whether the number of occurrences of the given corresponding candidate correction pair received from one or more of the multiple client devices has reached a threshold amount; identifying a subset of the multiple client devices that provided the given corresponding candidate correction pair in response to the determination that the given corresponding candidate correction pair is the corresponding actual correction pair; and ensuring that a global ASR model, which is a global-based counterpart of the corresponding on-device ASR model, is updated in a distributed manner using the subset of the multiple client devices.

[0081] These and other embodiments of the Technology may include one or more of the following features:

[0082] In some embodiments, enabling a global ASR model to be updated in a distributed manner using a subset of multiple client devices that have provided a given corresponding candidate correction pair may include sending a corresponding instruction to each client device included in the subset of multiple client devices that have provided a given corresponding candidate correction pair, that the global ASR model should be updated in a distributed manner. Sending a corresponding instruction to a given client device included in the subset of multiple client devices, that the global ASR model should be updated in a distributed manner, may cause the given client device to retrieve the corresponding audio data previously processed to generate the corresponding predictive text segment from the on-device storage of the given client device, generate the corresponding update to the global ASR model based on processing the corresponding audio data and using the corresponding on-device ASR model, and send the corresponding update to a remote system.

[0083] In some variations of these embodiments, generating a corresponding update for a given client device based on processing corresponding audio data and using a corresponding on-device ASR model may cause a given client device to generate a corresponding update based on processing corresponding audio data and comparing at least a portion of the corresponding additional predictive text segments with a corresponding alternative text segment, using a corresponding on-device ASR model stored locally in the client device's on-device storage, in order to generate a corresponding additional predictive text segment.

[0084] In some variations of these embodiments, the corresponding on-device ASR model may include multiple layers, and the corresponding update may be generated for a subset of the multiple layers of the corresponding on-device ASR model.

[0085] In some variations of these embodiments, the method may further include receiving a corresponding update from a given client device, and updating the global ASR model based on the corresponding update received from the given client device and the corresponding additional updates to the global ASR model received from one or more additional client devices among a subset of client devices, which are also included in the subset of client devices.

[0086] In some further variations of these embodiments, the method may further include sending the updated global ASR model to multiple client devices. Sending the updated global ASR model to a given client device may cause the given client device to replace the corresponding on-device ASR model in the given client device's on-device storage with the updated global ASR model.

[0087] Further variations of these embodiments, either additional or alternative, may include transmitting one or more updated global weights of the updated global ASR model to a plurality of client devices. Transmitting one or more updated global weights of the updated global ASR model to a given client device may cause the client device's on-device storage to replace one or more on-device weights of the corresponding on-device ASR model with one or more updated global weights of the updated global ASR model.

[0088] In further variations of these embodiments, either as additions or substitutions, the updated global ASR model may bias subsequent speech processing towards the corresponding alternative text segments.

[0089] Further variations of these embodiments, either adding or substituting, may cause the updated global ASR model to bias subsequent speech processing, excluding some of the corresponding predicted text segments.

[0090] In some embodiments, the method may further include refraining from updating the global ASR model in a distributed manner using one or more of the client devices that provided the given corresponding candidate correction pair, in response to determining that a given corresponding candidate correction pair is not the corresponding actual correction pair.

[0091] In some embodiments, corresponding audio data processed locally on one of several client devices using the corresponding on-device ASR model may not be received by the remote system.

[0092] In some embodiments, corresponding audio data processed locally on one of multiple client devices using the corresponding on-device ASR model may be stored in the on-device storage of the corresponding client device in response to the generation of the corresponding alternative text segment.

[0093] In some embodiments, determining whether a given corresponding candidate correction pair is the corresponding actual correction pair may further be based on corresponding query activity across multiple client devices.

[0094] In some embodiments, methods performed by one or more processors of a client device are provided herein, which include: receiving audio data capturing the speech utterance of a user of the client device via one or more microphones of the client device; processing the audio data using an on-device automatic speech recognition (ASR) model stored locally in the on-device storage of the client device to generate predictive text segments that are expected to correspond to the speech utterances; visually rendering the predictive text segments for presentation to the user on the display of the client device; receiving further user interface input to modify at least a portion of the predictive text segments into alternative text segments in response to the visual rendering of the predictive text segments; and, in response to receiving further user interface input to modify a portion of the predictive text segments into alternative text segments, The process includes: storing a portion of the predicted text segment and the alternative text segment as corresponding candidate correction pairs in the on-device storage of the client device; transmitting the corresponding candidate correction pairs to a remote system; storing audio data capturing the user's speech utterance in the on-device storage of the client device; receiving instructions from the remote system that the global ASR model, which is a global base counterpart of the corresponding on-device ASR model, should be updated in a distributed manner based on the corresponding correction pairs; generating an update to the global ASR model that will be used by the remote system when updating the global ASR model based on the corresponding candidate correction pairs and the audio data, in response to receiving the instructions that the global ASR model should be updated in a distributed manner based on the corresponding correction pairs; and transmitting the update to the remote system so that the update can be used when updating the global ASR model.

[0095] These and other embodiments of the Technology may include one or more of the following features:

[0096] In some embodiments, the method may further include refraining from transmitting audio data capturing the user's speech to a remote system.

[0097] In some embodiments, generating an update used by a remote system when updating a global ASR model based on corresponding candidate correction pairs and audio data may include: processing audio data using an on-device ASR model stored locally in the client device's on-device storage to generate additional predictive text segments that are expected to correspond to speech utterances; and generating an update based on comparing the additional predictive text segments with corresponding alternative text segments.

[0098] In some embodiments, sending an update to a remote system may cause the remote system to update the global ASR model based on the update provided by the client device and a number of additional corresponding updates provided by a number of additional client devices that provided additional instances of the corresponding candidate correction pairs.

[0099] In some variations of these embodiments, the method may further include receiving an updated global ASR model from a remote system and replacing the on-device ASR model with the updated global ASR model in the on-device storage of the client device.

[0100] In some variations of these embodiments, the method may further include receiving one or more updated global weights of an updated global ASR model from a remote system, and replacing one or more on-device weights of an on-device ASR model with one or more updated global weights of an updated global ASR model in the on-device storage of a client device.

[0101] In some embodiments, the instruction to update the global ASR model based on corresponding correction pairs in a distributed manner may be received in response to the remote system determining that the number of occurrences of corresponding correction pairs received from the client device and / or a number of additional client devices has reached a threshold amount.

[0102] In some embodiments, further user interface input may be touch input from the client device to a portion of a predictive text segment that is visually rendered for the user to be presented on the client device's display.

[0103] In some embodiments, further user interface input may be additional utterances from the user in response to a portion of a predictive text segment that is being visually rendered for presentation to the user on the client device's display.

[0104] In some embodiments, the on-device ASR model may include multiple layers, and updates may be generated for subsets of the layers of the on-device ASR model.

[0105] In some embodiments, sending the corresponding candidate correction pair to the remote system may be in response to the client device determining that the number of occurrences of the corresponding candidate correction pair detected has reached a threshold amount.

[0106] In some embodiments, sending a corresponding candidate correction pair to a remote system may be in response to determining that the corresponding candidate correction pair is included in a list of candidate correction pairs received from the remote system.

[0107] In some embodiments, a computing device system is provided, which includes a given client device of a given user, the given client device including at least one client device processor and client device memory for storing instructions, the instructions being, when executed by at least one client device processor, the at least one client device processor receives audio data capturing the speech utterance of a given user via one or more microphones of the given client device, and processes the audio data using an on-device automatic speech recognition (ASR) model stored locally on the given client device to generate predictive text segments that are expected to correspond to the speech utterances, and The system is configured to: make the predicted text segment visually rendered on the display of a given client device for presentation to a given user; receive further user interface input to modify at least a portion of the predicted text segment into an alternative text segment in response to the visual rendering of the predicted text segment; store a portion of the predicted text segment and the alternative text segment locally on the given client device as corresponding candidate correction pairs in response to receiving further user interface input to modify the predicted text segment into an alternative text segment; and store audio data capturing the user's speech locally on the given client device.The computing device system further includes a remote server communicatively coupled to a given client device, the remote server including at least one remote processor and remote memory for storing instructions, the instructions, when executed by at least one remote processor, cause the at least one remote processor to receive a corresponding candidate correction pair from a given client device, determine whether the corresponding candidate correction pair is the corresponding actual correction pair based on whether the number of occurrences of the corresponding candidate correction pair received from the given client device and / or a number of additional client devices in addition to the given client has reached a threshold amount, and in response to the determination that a given corresponding candidate correction pair is the corresponding actual correction pair, cause a global ASR model, which is a global-based counterpart of the corresponding on-device ASR model, to be updated in a distributed manner using the given client device and / or a number of additional client devices.

[0108] Various embodiments may include a non-temporary computer-readable storage medium that stores instructions executable by one or more processors (e.g., a central processing unit (CPU)(or multiple), graphics processing unit (GPU)(or multiple), digital signal processor (DSP)(or multiple), and / or tensor processing unit (TPU)(or multiple)) in order to perform one or more of the methods described herein. Another embodiment may include an automated assistant client device (e.g., a client device that includes at least an automated assistant interface for interfacing with cloud-based automated assistant components(or multiple)) that includes a processor(or multiple) capable of executing the stored instructions in order to perform one or more of the methods described herein. Yet another embodiment may include a system of one or more servers that includes one or more processors capable of executing the stored instructions in order to perform one or more of the methods described herein.

Claims

1. A method implemented by one or more processors of a remote system, Receiving corresponding candidate correction pairs from multiple client devices, wherein each of the corresponding candidate correction pairs is Based on processing the corresponding audio data locally on one of the aforementioned client devices, and using the corresponding on-device automatic speech recognition (ASR) model, the corresponding portion of the corresponding predicted text segment is generated. Receiving includes the corresponding alternative text segment, which is generated locally on one of the multiple client devices and based on the corresponding modification to the corresponding portion of the corresponding predictive text segment that forms the basis of the corresponding alternative text segment; Whether a given corresponding candidate correction pair among the corresponding candidate correction pairs is the corresponding actual correction pair is determined based on whether the number of occurrences of the given corresponding candidate correction pair received from one or more of the plurality of client devices has reached a threshold amount. In response to the determination that the given corresponding candidate correction pair is the corresponding actual correction pair, From among the plurality of client devices, identify a subset of the plurality of client devices that provided the given corresponding candidate correction pair, The global ASR model, which is a global-based counterpart of the corresponding on-device ASR model, is updated in a distributed manner using the corresponding update, wherein the corresponding update is generated by the given client device based on processing the corresponding audio data. Methods that include...

2. The global ASR model is updated using the corresponding update in the distributed manner. This includes transmitting a corresponding instruction to each client device included in the subset of the plurality of client devices that provided the given corresponding candidate correction pair, that the global ASR model should be updated in the distributed manner, and transmitting the corresponding instruction to update the global ASR model in the distributed manner to the given client device included in the subset of the plurality of client devices means that the given client device, From the on-device storage of the given client device, retrieve the corresponding audio data previously processed to generate the corresponding predictive text segment, Based on processing the corresponding audio data and using the corresponding on-device ASR model, the corresponding update of the global ASR model is generated. The remote system transmits the corresponding update, The method according to claim 1, which causes the following to be performed.

3. To cause the given client device to generate the corresponding update based on processing the corresponding audio data and using the corresponding on-device ASR model, the given client device To generate corresponding additional predictive text segments, the corresponding audio data is processed using the corresponding on-device ASR model stored locally in the on-device storage of the client device, The process involves generating the corresponding update based on comparing at least a portion of the corresponding additional predictive text segments with the corresponding alternative text segments, The method according to claim 2, which causes the following to be performed.

4. The method according to claim 2, wherein the corresponding on-device ASR model comprises multiple layers, and the corresponding update is generated for a subset of the multiple layers of the corresponding on-device ASR model.

5. Updating the global ASR model based on the corresponding update received from the given client device and the corresponding additional update of the global ASR model received from one or more additional client devices among the plurality of client devices, which are also included in the subset of the plurality of client devices. The method according to claim 2, further comprising:

6. The process further includes transmitting the updated global ASR model to the plurality of client devices, and transmitting the updated global ASR model to the given client devices means that the given client devices The method according to claim 5, wherein the on-device storage of the given client device replaces the corresponding on-device ASR model with the updated global ASR model.

7. The process further includes transmitting one or more updated global weights of the updated global ASR model to the plurality of client devices, and transmitting the one or more updated global weights of the updated global ASR model to the given client devices means that the given client devices The method according to claim 5, wherein the on-device storage of the given client device replaces one or more on-device weights of the corresponding on-device ASR model with one or more updated global weights of the updated global ASR model.

8. The method according to claim 5, wherein the updated global ASR model biases subsequent speech processing to the corresponding alternative text segment.

9. The method according to claim 5, wherein the updated global ASR model is biased to subsequent speech processing to avoid a portion of the corresponding predicted text segments.

10. In response to the determination that the given corresponding candidate correction pair is not the corresponding actual correction pair, The method according to claim 1, further comprising refraining from updating the global ASR model using one or more of the client devices that provided the given corresponding candidate correction pair in the distributed manner.

11. The method according to claim 1, wherein the corresponding audio data processed locally on one of the plurality of client devices using the corresponding on-device ASR model is not received by the remote system.

12. The method according to claim 1, wherein the corresponding audio data, processed locally in one of the plurality of client devices using the corresponding on-device ASR model, is stored in the on-device storage of the one of the plurality of client devices in response to the generation of the corresponding alternative text segment.

13. The method according to claim 1, further comprising determining whether the given corresponding candidate correction pair is the corresponding actual correction pair based on the corresponding query activity in the plurality of client devices.

14. A non-temporary computer-readable storage medium that, when executed by one or more hardware processors, stores instructions causing the one or more hardware processors to perform an operation according to any one of claims 1 to 13.

Images