Hybrid client-server federated learning of machine learning models

Abstract

Implementations disclosed herein relate to federated learning of a machine learning (“ML”) model based on a gradient(s) generated at a corresponding client device and a remote system. A processor(s) of the corresponding client device can process, using a ML model on the corresponding device(s), client data generated locally at the corresponding client device to generate a corresponding prediction output, generate a corresponding client gradient based on the corresponding prediction output, and transmit the corresponding client gradient to the remote system. A processor(s) of the remote system can process, using a global ML model(s), remote data obtained from a remote database(s) to generate an additional corresponding prediction output, generate a corresponding remote gradient based on the additional corresponding prediction output. Further, the remote system can update the global ML model(s) or weights thereof with the corresponding client gradient and the corresponding remote gradient. The updated global ML model(s) and / or updated weights thereof can be transmitted back to the corresponding client device.

Description

BACKGROUND

[0001] Federated learning of machine learning (ML) model(s) is an increasingly popular ML technique for training ML model(s). In traditional federated learning, a local ML model is stored locally on a user's client device, and a global ML model, which is a cloud-based counterpart of the local ML model, is stored remotely on a remote system (e.g., a server cluster). A client device using the local ML model is able to process a user input detected at the client device to generate a predicted output, and is able to compare the predicted output with a ground truth output to generate a client gradient. Further, the client device is able to transmit the client gradient to the remote system. The remote system is able to update weights of the global ML model with the client gradient and optionally additional client gradients generated in a similar manner at additional client devices. The remote system is able to transmit the global ML model or updated weights of the global ML model to the client device. The client device is then able to replace the local ML model with the global ML model, or replace weights of the local ML model with updated weights of the global ML model, thereby updating the local ML model.

[0002] Notably, these global ML models are typically pre-trained at the remote system without using any client gradients generated locally at the client devices, before federated learning based on multiple remote gradients generated remotely at the remote system. This pre-training is typically based on proxy or biased data, which can not reflect data that will be encountered when the global ML model is deployed at the client devices. After pre-training, the weights of these global ML models are typically updated based only on client gradients generated from data encountered when the global ML model is deployed at the client devices, without using any remote gradients generated remotely at the remote system. However, updating the weights of these global ML models in this manner can result in catastrophic forgetting of information learned during pre-training. Further, client gradients generated based on certain data (e.g., false positives, false negatives, etc.) can be difficult to obtain at the client devices, resulting in poor performance of ML models trained using federated learning. SUMMARY

[0003] Implementations disclosed herein relate to federated learning of machine learning (ML) model(s) based on gradient(s) generated at corresponding client devices and a remote system. A processor(s) of a corresponding client device can generate a corresponding prediction output using a corresponding on-device ML model processing client data generated locally at the corresponding client device, generate a corresponding client gradient based on the corresponding prediction output, and transmit the corresponding client gradient to the remote system. A processor(s) of the remote system can generate an additional corresponding prediction output using a global ML model processing remote data obtained from a remote database(s), generate a corresponding remote gradient based on the additional corresponding prediction output. Further, the remote system can update the global ML model(s) or weights thereof with the corresponding client gradient and the corresponding remote gradient. The updated global ML model(s) and / or updated weights thereof can be transmitted back to the corresponding client device. Notably, the remote gradient described herein is used to update the already trained global model(s).

[0004] For example, assume that audio data is generated at a given client device of a user that captures a spoken utterance of “Assistant, call Ron Smith.” In this example, the given client device can process the audio data using an on-device hotword detection model to generate a prediction output indicating whether a portion of the spoken utterance includes one or more particular words or phrases (e.g., “Assistant,” “Hey Assistant,” and / or any other words or phrases used to invoke or otherwise control an automated assistant) for invoking the automated assistant. In this example, the prediction output can be a prediction metric (e.g., a probability, a binary value, a log-likelihood value, or any other prediction metric). For instance, assume that the prediction output is a probability of 0.80 that satisfies a threshold probability (e.g., 0.75) indicating that the spoken utterance includes one or more particular words or phrases for invoking the automated assistant. Further, the given client device can compare the prediction output (e.g., a probability of 0.80) to a corresponding ground truth output (e.g., assume a probability of 1.0 that the user intended to invoke the automated assistant). The given client device can compare the prediction output and the corresponding ground truth output to generate a client gradient, or more specifically, a hotword detection client gradient associated with the hotword detection model. The hotword detection client gradient can be transmitted to the remote system, and the remote system can utilize the hotword detection client gradient to update weights of a global hotword detection model.

[0005] While the above examples are described with respect to hotword detection client gradients, it should be understood that this is by way of example and not limitation. For example, given a client device can additionally or alternatively process audio data using an automatic speech recognition (ASR) model to generate a predicted output associated with a term corresponding to a spoken utterance. In this example, the predicted output can be a plurality of speech hypotheses (and optionally a predicted measure associated with one or more of the plurality of speech hypotheses) corresponding to the spoken utterance. For example, assume the predicted output for the "Ron" portion of the spoken utterance includes a term hypothesis of "Ron" associated with a probability of 0.55 and a term hypothesis of "Don" associated with a probability of 0.45. Given the client device can compare the predicted output (e.g., the term hypotheses of "Ron" and "Don" and their associated probabilities) to a corresponding ground truth output (e.g., the term "Ron" and a probability of 1.0 that the user intended to call "Ron"). Given the client device can compare the predicted output and the corresponding ground truth output to generate another client gradient, or more specifically, an ASR client gradient associated with the ASR model. Similarly, the ASR client gradient can be transmitted to the remote system, and the remote system can utilize the ASR client gradient to update the weights of the global ASR model. Moreover, while the above examples are described with respect to audio-based ML models, it should also be understood that this is by way of example and not limitation, and the techniques described herein can also be used to generate client gradients for image-based ML models, text-based ML models, and / or other types of ML models. In other words, the type of gradient generated by a given client device can be based on the client data generated or received at the given client device, and can additionally or alternatively be based on the particular ML model used to process the client data.

[0006] The remote system can (and optionally in response to receiving a client gradient from a corresponding client device) obtain remote data from a remote database(s). In some implementations, the remote data can be obtained from, for example, an online video sharing platform, an image sharing platform, an audio sharing platform, and / or a text-based platform that is not subject to access restrictions (i.e., like the client data generated and / or received locally at the corresponding client device). In additional or alternative implementations, and with permission from a corresponding user of the corresponding client device, the remote data can be access-restricted data of the corresponding user that is accessible to the remote system. Remote gradients can be generated at the remote system in the same or similar manner as described above with respect to a given client device, but based on the remote data and using components of the remote system and global ML model(s) accessible to the remote system.

[0007] In some implementations, the remote system is able to analyze the client gradients received from the corresponding client devices and obtain remote data associated with the client data. For example, assume that a given client gradient is generated based on audio data that captures a spoken utterance of “Assistant, call Ron Smith,” but does not invoke the automated assistant based on the predicted output (e.g., a false negative hotword detection client gradient). In this example, the remote system is able to obtain remote audio data, which can result in another false negative (e.g., by filtering the remote audio data to be ambiguous or otherwise affect “Assistant”). Notably, in a federated learning framework, the client gradient can be transmitted to the server without transmitting any underlying client data or predicted output. However, the remote system can know that the gradient is a false negative hotword detection client gradient that was generated based on audio data. In contrast, the remote system is able to obtain remote audio data that can result in a false positive (e.g., the automated assistant was invoked, but it should not have been), which can be more difficult to obtain than a false negative because the intent of the user of the given client device can not be readily ascertainable. Nonetheless, the remote system is able to obtain remote audio data that can result in a false negative to ensure that the gradients used to update the global hotword detection model are sufficiently diverse.

[0008] In some versions of those implementations, the remote system is able to analyze the distribution of the gradients for updating the corresponding global ML model. In analyzing the distribution, the remote system is able to identify the types of client gradients that have been received at the remote system from the corresponding client devices and / or the types of remote gradients that have been generated at the remote system. Further, the remote system is able to determine that certain types of gradients need to be generated based on the distribution lacking a particular type of gradient or a sufficient amount of gradients thereof. In response to determining that those certain types of gradients need to be generated, the remote system is able to generate those certain types of gradients (and optionally a particular amount thereof). For example, assume that a plurality of hotword free invocation client gradients have been generated at the corresponding client device based on gestures provided by the corresponding user of the client device to invoke or otherwise control the automated assistant (e.g., based on hand motions, eye motions, lip motions, etc.). However, it can be difficult to obtain a hotword free invocation client gradient at the client device that does not invoke the automated assistant (e.g., for privacy considerations). In this example, the remote system can determine that a hotword free invocation remote gradient is needed that includes a human making a gesture that does not target invoking the automated assistant. Accordingly, the remote system is able to obtain remote image data of the human making one or more gestures that do not invoke the automated assistant to ensure that there is a sufficient amount and diversity of hotword free invocation gradients in the hotword free invocation gradients used to update the global hotword free invocation model.

[0009] Furthermore, the remote system can update (multiple) corresponding global ML models using client gradients received from the corresponding client devices and remote gradients generated by the remote system. In some implementations, the remote system can select a set of corresponding client gradients to update the weights of (multiple) corresponding global ML models for specific update iterations. In some versions of those implementations, the remote system can select the set of corresponding client gradients based on one or more criteria. One or more criteria can include, for example, the type of client gradients available to the remote system, a threshold number of client gradients available to the remote system, a threshold duration for updating using client gradients, and / or other criteria. For example, the remote system can select a set of 500 audio-based client gradients to update (multiple) audio-based models (e.g., a single audio-based model or multiple different audio-based models), select a set of 200 hot word client gradients to update (multiple) hot word models, and so on. As another example, the system can select a set of 600 image-based client gradients to update (multiple) image-based models (e.g., a single image-based model or multiple different image-based models), select a set of 300 hot word-based client gradients to update a hot word model, and so on. In other implementations, the set of client gradients can include any combination of audio-based client gradients, image-based client gradients, text-based client gradients, and / or any other type of client gradient (or a specific type of client gradient).

[0010] In some versions of those implementations, the remote system can also select a set of corresponding remote gradients to update the weights of the corresponding global ML model. In other versions of those implementations, the remote system can select the set of corresponding remote gradients based on the type of gradients included in the set of corresponding client gradients. For example, if the selected set of corresponding client gradients includes a set of audio-based gradients (or a set of specific types of audio-based gradients), then the selected set of corresponding remote gradients can also include audio-based gradients. In additional or alternative versions of those other implementations, the remote system can select the set of corresponding remote gradients based on the number of client gradients included in the selected set of corresponding client gradients. For example, if the selected set of corresponding client gradients includes a set of 500 audio-based gradients (or a set of specific types of audio-based gradients), then the selected set of corresponding remote gradients can also include 500 audio-based gradients. As another example, if the selected set of corresponding client gradients includes a set of 500 audio-based gradients (or a set of audio-based gradients of a specific type), the selected set of corresponding remote gradients can also include a proportional number of audio-based gradients (e.g., 750 represents 150%, 250 represents 50%, 100 represents 20%, and so on).

[0011] In implementations that update the weights of(multiple) corresponding global ML models using sets of corresponding client gradients and sets of distant gradients, the(multiple) corresponding global ML models can be updated in various ways. For example, the weights of(multiple) corresponding ML models can initially be updated based on a selected set of corresponding client gradients, and subsequently updated based on a selected set of corresponding distant gradients. As another example, the weights of(multiple) corresponding ML models can initially be updated based on a selected set of corresponding distant gradients, and subsequently updated based on a selected set of corresponding client gradients. As yet another example, the first weights of(multiple) first instances of(multiple) corresponding ML models can be updated based on a selected set of corresponding client gradients, and the second weights of(multiple) second instances of(multiple) corresponding ML models can be updated in parallel or serially based on a selected set of corresponding distant gradients. In this example, the average (or some other statistical measure) of the first and second weights can be used as the weights of the updated corresponding ML models. In other implementations, the remote system can use client gradients and remote gradients to update (multiple) corresponding global ML models in a first-in-first-out (FIFO) manner, without assigning gradients to specific iterations that update (multiple) global ML models.

[0012] Various technical advantages can be achieved by using the techniques described herein. As a non-limiting example, when utilizing both client-side gradients and remote gradients generated in the manner described herein, the corresponding global ML model can be updated based on diverse and balanced data, thereby reducing the impact of catastrophic forgetting and producing a more stable global ML model with higher accuracy and / or recall. For example, remote systems can generate remote gradients based on the identification requirements of specific types of gradients for a particular global ML model, which might not be possible within a federated learning framework due to privacy concerns without these techniques. Therefore, when the corresponding global ML model (or its weights) is deployed to client devices, the on-device ML model is also more stable and has higher accuracy and / or recall.

[0013] The above description serves as an overview of some implementations of this disclosure. Further descriptions of those and other implementations are provided below in more detail. Attached Figure Description

[0014] Figure 1A , Figure 1B , Figure 1C and Figure 1D Example process flows illustrating various aspects of this disclosure according to various implementations are described.

[0015] Figure 2A block diagram depicts an example environment in which the implementations disclosed herein can be carried out.

[0016] Figure 3 The flowchart illustrates example methods for generating client-side gradients locally on the client device for updating machine learning models, based on various implementations.

[0017] Figure 4 The flowchart illustrates example methods for remotely generating (multiple) remote gradients at a remote system for updating (multiple) machine learning models, based on various implementations.

[0018] Figure 5 The flowcharts depict various example methods, based on different implementations, for mixing sets of client gradients and sets of remote gradients in updating (multiple) global machine learning models.

[0019] Figure 6 Example architectures of computing devices based on various implementations are described. Detailed Implementation

[0020] Figures 1A-1D An example process flow illustrating various aspects of this disclosure is depicted. Client device 150 in... Figure 1A The diagram is shown in the image, and includes elements covering the client device 150. Figure 1A The components within the box. The on-device machine learning (ML) engine 132A is capable of processing client data 101A using multiple on-device ML models stored in an on-device ML model database 154A to generate multiple prediction outputs 102. The gradient engine 134A is capable of generating client gradients 103 based on the multiple prediction outputs 102. In some implementations, the gradient engine 134A is capable of generating client gradients 103 using supervised learning techniques based on comparing the multiple prediction outputs 102 with multiple ground truth outputs 101B corresponding to client data 101A. In additional or alternative implementations, such as when the multiple ground truth outputs 101B corresponding to client data 101A are unavailable, the gradient engine 134A is capable of using supervised and / or unsupervised learning techniques (e.g., regarding...). Figure 3The client device 150 generates client gradient 103. The client device 150 is then able to transmit the client gradient 103 to the remote system 160 via one or more networks (e.g., any combination of a local area network (LAN), a wide area network (WAN), and / or any other type of network), and optionally does not transmit any client data 101A, (multiple) ground reality outputs 101B, (multiple) prediction outputs 102, and / or any other personally identifiable information. In various implementations, the client device 150 is able to transmit the client gradient 103 to the remote system 160 in response to determining that one or more conditions are met (e.g., regarding...). Figure 3 (Described).

[0021] In some implementations, client gradient 103 (and other gradients described herein) can be derived from a loss function used to train the ML model(s), such that the gradient represents the value of the loss function (or its derivative) obtained from the comparison of the ground reality output(s) 101B with the predicted output(s) 102 (e.g., using supervised learning techniques). For example, when the ground reality output(s) 101B and the predicted output(s) 102 match, gradient engine 134A is able to generate a zero gradient. Furthermore, for example, when the ground reality output(s) 101B and the predicted output(s) 102 do not match, gradient engine 134A is able to generate a non-zero gradient, optionally depending on the degree of mismatch. The degree of mismatch can be based on the degree of mismatch between the deterministic comparisons of the ground reality output(s) 101B and the predicted output(s) 102. In additional or alternative implementations, client gradient 103 (and other gradients described herein) can be derived from the loss function used to train (multiple) ML models, such that the gradient represents the value of the loss function (or its derivative) determined based on (multiple) predicted outputs 102 (e.g., using supervised or semi-supervised learning techniques).

[0022] As described below, client data 101A can be audio data generated by the microphone(s) of client device 150 (e.g., regarding...). Figure 1B (as described), as multiple text fragments provided by user input from client device 150 and / or stored in the device's memory (e.g., about Figure 1C (described), image data generated by (multiple) visual components of the client device (e.g., regarding) Figure 1D(as described), and / or any other data generated locally on the client device 150 and processed by (multiple) ML models. In some implementations, client data 101A can be processed to generate gradient 103 when it is generated or provided to the client device 150 synchronously. In additional or alternative implementations, client data 101A can be stored in a client data database 152A when it is generated or provided to the client device 150, and gradient 103 can then be generated asynchronously using client data 101A. In additional or alternative implementations, on-device ML engine 132A can process client data 101A to generate (multiple) prediction outputs 102, which can be stored in the client data database 152A (optionally associated with client data 101A associated with (multiple) prediction outputs 102), and the (multiple) prediction outputs 102 can then be used to generate gradient 103 asynchronously. The client device database 152A (also referred to herein as on-device memory or on-device storage) can include any data generated or provided to the client device, including, but not limited to, audio data, image data, contact lists, electronic messages (e.g., text messages, emails, social media messages, etc.) and / or any other client data sent or received by the user of client device 150. It is worth noting that client data 101A corresponds to data with restricted access, or data that is not publicly available and / or usable by remote system 160.

[0023] The global ML engine 132B is capable of processing remote data 104A using multiple global ML models stored in a global ML model database 154B to generate multiple predictive outputs 105. Remote data 104A can be obtained from a remote data database 152B. The remote data database 152B can include any data accessible to the remote system 160, including but not limited to public data repositories containing audio data, text data, and / or image data, as well as private data repositories. Furthermore, the remote data database 152B can include data from different types of client devices with different device characteristics or components (e.g., different from client device 150 and / or multiple additional client devices 170 described below). For example, the remote data database 152B can include audio data captured by multiple near-field microphones (e.g., similar to audio data captured by client device 150) and audio data captured by multiple far-field microphones (e.g., audio data captured by other devices). As another example, the remote data database 152B can include image data (or other visual data) captured by different visual components, such as RGB image data, RGB-D image data, CMYK image data, and / or other types of image data captured by various different visual components. Furthermore, the remote system 160 can employ one or more techniques to modify the remote data 104A. These techniques can include filtering audio data to add or remove noise when the remote data 104A is audio data, blurring images when the remote data 104A is image data, and / or other techniques to manipulate the remote data 104A. This allows the remote system 160 to better reflect client data generated by multiple different client devices and / or meet the need for specific types of data (e.g., inducing false positives or false negatives as described herein, ensuring sufficient diversity of audio data as described herein, etc.).

[0024] Gradient engine 134B is capable of generating a remote gradient 106 based on multiple predicted outputs 105. In some implementations, gradient engine 134B can generate the remote gradient 106 using supervised learning techniques based on a comparison between multiple predicted outputs 105 and multiple ground-based outputs 104B corresponding to remote data 104A. In additional or alternative implementations, such as when multiple ground-based outputs 104B corresponding to remote data 104A are unavailable, gradient engine 134B can generate the remote gradient 106 using supervised and / or unsupervised learning techniques. In some implementations, remote system 160 can simulate a client device as a "canary" user, making it appear as if the remote gradient 106 was generated by the actual client device. Remote gradient 106 (and any additional remote gradients) can be stored in gradient databases 185 (e.g., long-term memory and / or short-term memory, such as buffers), and optionally together with client gradient 103 received from client device 150 and corresponding additional gradients 107 received from multiple additional client devices 170 (and optionally may be limited to client devices of the same type as client device 150 (e.g., mobile phones, stand-alone interactive speakers, in-vehicle computing systems, etc.)). Each of the additional gradients 107 received from the multiple additional client devices 170 can be generated based on the same or similar techniques described above with respect to generating gradient 103, but based on client data locally generated or provided at the corresponding location of the multiple additional client devices 170.

[0025] As described above, gradients 103, 106, and / or 107 can be stored in gradient database(s) 185 (or other memory (e.g., buffer)) due to the generation and / or reception of gradients 103, 106, and / or 107. In some implementations, gradients 103, 106, and / or 107 can be indexed by type from multiple different types of gradients, determined based on the ML models on(s) corresponding devices processing client data 101A and / or the corresponding global ML models processing remote data 104A. Multiple different types of gradients can be defined with different degrees of granularity. For example, as per [reference to...] Figures 1B-1C and Figure 2The described gradients can be specifically defined as follows: for example, a hot word gradient generated based on audio data processed using a hot word model(s), an ASR gradient generated based on processed audio data, a VAD gradient generated based on processed audio data processed using a VAD model(s), a continuation dialogue gradient generated based on processed audio data processed using a continuation dialogue model(s), a speech recognition gradient generated based on processed audio data processed using a speech recognition model(s), a face recognition gradient generated based on processed image data processed using a face recognition model(s), a hot word free gradient generated based on processed image data processed using a hot word free model(s), an object detection gradient generated based on processed image data processed using a object detection model(s), a TTS gradient generated based on processed text segments processed using a text-to-speech (TTS) model(s), and / or any other gradient generated based on processed data processed using any other ML model(s). It is noteworthy that a given gradient 103, 106, and / or 107 can belong to one of several different types of gradients. Therefore, as another example, the type of gradient can be more generally defined as, for example, an audio-based gradient generated by processing audio data using one or more audio-based models, an image-based gradient generated by processing image data using one or more image-based models, or a text-based gradient generated by processing text fragments using a text-based model.

[0026] The remote training engine 136 can update one or more weights of the global ML models stored in the global ML model database(s) 154B using at least the client gradient 103, the remote gradient 106, and the additional gradient 107. For example, the remote training engine 136 can identify a specific global ML model stored in the global ML model database(s) 154B to update its weights. In some implementations, the remote training engine 136 can identify a specific global ML model based on the gradient type stored in the gradient database(s) 185. For example, if multiple hot word gradients are stored in the gradient database(s) 185, the remote training engine 136 can identify the global hot word model to update based on the multiple hot word gradients. As another example, if multiple audio-based gradients are stored in the gradient database(s) 185, the remote training engine 136 can identify the global audio-based model to update based on the multiple audio-based gradients. It is worth noting that the remote training engine 136 can identify a single global ML model to be updated at a given time instance or multiple global ML models to be updated in parallel at a given time instance.

[0027] In some implementations, the remote system 160 is able to assign gradients 103, 106, and / or 107 to a specific iteration of updating(multiple) global ML models based on one or more criteria. One or more criteria can include, for example, the type of gradients available to the remote training engine 136, the number of gradient thresholds available to the remote training engine 136, the threshold duration for using gradient updates, and / or other criteria. Specifically, the remote training engine 136 is able to identify multiple sets or subsets of gradients generated by the client device 150 and / or sets generated by the remote system 160. Furthermore, the remote training engine 136 is able to update(multiple) global ML models based on these sets or subsets of gradients. In some other versions of those implementations, the number of gradients in the set of client gradients and the set of remote gradients may be the same or different (e.g., proportional to each other and having more client gradients or more remote gradients). In other versions of those implementations, each subset of client gradients can optionally include client gradients from at least one unique client device that is not included in another subset. In other implementations, the remote system 160 uses client gradients and remote gradients to update (multiple) corresponding global ML models in a first-in-first-out (FIFO) manner, without assigning gradients 103, 106, and / or 107 to specific iterations updating (multiple) global ML models. In this paper (e.g., regarding...), Figure 4 and 5 This describes how to update (multiple) ML models using gradients 103, 106, and / or 107.

[0028] The update distribution engine 138 is capable of transferring updated ML models 108 and / or their weights to one or more of client devices 150 and / or multiple additional client devices 170. In some implementations, the update distribution engine 138 is capable of responding to one or more conditions satisfying client device 150, one or more of the multiple additional client devices 170, and / or remote systems (e.g., regarding...). Figure 3 and 4 The described method transmits (multiple) updated ML models 108 and / or their weights. Upon receiving the updated ML models and / or their weights, the client device is able to replace (multiple) corresponding on-device ML models with (multiple) updated ML models 108, or replace the weights of (multiple) corresponding on-device ML models (e.g., in the (multiple) on-device ML model database 154A) with the weights of (multiple) updated ML models 108. Furthermore, the client device 150 can subsequently use (multiple) updated ML models 108 and / or their weights to make predictions based on other user inputs detected at the client device 150 (e.g., regarding...). Figure 2(Described in more detail). Client device 150 and multiple additional client devices 170 are capable of continuing to generate other client gradients in the manner described herein and transmitting these other client gradients to remote system 160. Furthermore, remote system 160 is capable of continuing to generate other remote gradients in the manner described herein and updating (multiple) global ML models based on these other client gradients and other remote gradients.

[0029] In some implementations, the current approach is to switch. Figure 1B Client data 101A can be audio data 109A generated by the microphone(s) of client device 150. Audio data 109A can capture, for example, spoken words from a user to client device 150, ambient noise in the environment where client device 150 is located, such as ambient noise or ambient dialogue, and / or any other sound that can be captured by the microphone(s) of client device 150. In an implementation where client device 101A is audio data 109A, on-device ML engine 132A can process audio data 109A using one or more on-device audio-based ML models to generate multiple prediction outputs 110, which are used to generate audio-based client gradients. For example, the on-device ML engine 132A can process the audio data 109A using the following methods: an on-device voice activity detection (VAD) model for predicting whether the audio data 109A captures human spoken utterances; an on-device hot word detection model for predicting whether the audio data 109A captures spoken utterances including one or more specific words or phrases that invoke or otherwise cause the automated assistant to control client device 150 or other devices (e.g., other client devices or smart devices); an on-device automatic speech recognition (ASR) model for generating recognized text based on any spoken utterances captured in the audio data 109A; an on-device voice recognition model for processing the audio data 109A to determine whether any spoken utterances captured in the audio data 109A originate from a specific user (e.g., a user of client device 150); an on-device continuation dialogue model for processing the audio data 109A to determine whether the audio data 109A captures subsequent input provided by a user of client device 150; a still-on point model for processing the audio data 109A to predict whether and / or when the user providing the spoken utterance ends the spoken utterance; and / or any other on-device audio-based ML model. It should be understood that the ML models on the aforementioned devices are provided for illustrative purposes only and are not intended as limitations.

[0030] It is worth noting that the multiple prediction outputs 110 can include corresponding prediction outputs generated across one or more of the aforementioned on-device ML models. For example, if audio data 109A is processed using an on-device VAD model and an on-device hot word detection model, the multiple prediction outputs 110 can include corresponding prediction outputs associated with the on-device VAD model and corresponding prediction outputs associated with the on-device hot word detection model. As another example, if audio data 109A is processed using an on-device speech recognition model, an on-device hot word detection model, and an on-device ASR model, the multiple prediction outputs 110 can include corresponding prediction outputs associated with the on-device speech recognition model, corresponding prediction outputs associated with the on-device hot word detection model, and corresponding prediction outputs associated with the on-device ASR model. Therefore, the multiple prediction outputs 110 can include prediction outputs generated across one or more of the on-device ML models. Thus, any resulting client gradient can be a gradient of a specific type based on the multiple on-device ML models used to process audio data 109A.

[0031] In some implementations, the gradient engine 134A is capable of comparing (multiple) predicted outputs 110 with (multiple) corresponding ground reality outputs 109B to use supervised learning techniques (e.g., as per [reference to...]). Figure 3 The description describes the generation of (multiple) client gradients 111. In additional or alternative implementations, such as when (multiple) corresponding ground reality outputs 109B are unavailable, the gradient engine 134A can utilize unsupervised learning techniques to generate (multiple) client gradients 111 (e.g., as per [reference to...]). Figure 3 (As described). In other words, the gradient engine 134A is able to evaluate the predicted outputs 110 relative to some actual outputs that should be generated across the model to identify errors generated by the ML model across the ML model on the ML model (multiple devices). For example, suppose the predicted outputs generated by the hot word detection model across devices are probabilities (e.g., or other prediction metrics, such as log-likelihood, binary values, or other metrics) that indicate whether the audio data 109A includes a specific word or phrase used to invoke the automatic assistant. Further suppose the probability is 0.75 and the threshold probability for invoking the automatic assistant is 0.80. In this example, the automatic assistant may not be invoked. However, further suppose the automatic assistant is invoked within a threshold time period of receiving the audio data 109A. The client device 150 can use this as a supervisory signal to indicate that the automatic assistant should be invoked in response to the received spoken utterance captured in the audio data 109A (e.g., a false negative). In this example, the corresponding ground truth output could be a probability of 1.0 to indicate that the automatic assistant should have been invoked, and the resulting client gradient could be a false negative gradient of the hot word detection model (multiple devices).

[0032] Furthermore, multiple client gradients 111 can be transmitted to a remote system 160 and stored in multiple gradient databases 185. Remote data 104A can be audio data 112A obtained from a remote data database 152B. The remote system 160 can process the audio data 112A in the same or similar manner as described above with respect to the client device 150 to generate multiple prediction outputs 113, but using a global ML engine 132B and multiple corresponding audio-based global ML models stored in a global ML model database 154B. Furthermore, the gradient engine 134B can compare the multiple prediction outputs 113 with multiple corresponding ground truth outputs 112B to generate multiple audio-based remote gradients 114, and these remote gradients 114 can be stored in the multiple gradient databases 185.

[0033] In additional or alternative implementations, and now turning to Figure 1C Client data 101A can be multiple text fragments 115A accessible to client device 150. These multiple text fragments 115A can include any text available to client device 150. For example, multiple text fragments 115A can include one or more terms retrieved from a contact list (e.g., name, physical address, email address, and / or other contact information), messages (e.g., text messages, email messages), and / or any other electronic messages), calendar or task entries, documents, reminders, and / or other text available at client device 150. In an implementation where client device 101A is multiple text fragments, on-device ML engine 132A can use one or more on-device text-based ML models to process the multiple text fragments 115A to generate multiple prediction outputs 117, which are used to generate text-based client gradients. For example, the on-device ML engine 132A can enable the on-device TTS engine 132A1 to use on-device TTS models stored in the on-device ML model database 154A to process (multiple) text segments 115A to generate synthesized speech audio data 116, which includes synthesized speech corresponding to (multiple) text segments 115A. Furthermore, the on-device ML engine 132A enables the on-device ASR engine 132A2 to use on-device ASR models stored in the on-device ML model database 154A to process the synthesized speech audio data 116 to generate (multiple) prediction outputs 117.

[0034] In some implementations, the on-device ASR model can be an end-to-end model for generating predicted text on a character-by-character basis (or another character-by-character basis). A non-limiting example of such an end-to-end model for generating predicted text character-by-character is the Recurrent Neural Network Transformer (RNN-T) model. The RNN-T model is a sequence-to-sequence model that does not employ an attention mechanism. Unlike most sequence-to-sequence models, which typically require processing the entire input sequence (e.g., audio data waveforms or Mel-frequency cepstral coefficients (MFCCs) or other representations) to generate multiple predicted text segments as outputs (multiple) predicted outputs116, the RNN-T model can be used to further process input samples and stream output symbols (e.g., characters in the alphabet). In other implementations, the on-device ASR model is not an end-to-end speech recognition model. In those other implementations, the on-device ASR engine 132A2 can instead generate predicted phonemes (and / or other representations). For example, using this model, the on-device ASR engine 132A2 then uses predicted phonemes (and / or other representations) to determine (multiple) predicted text segments that conform to the phoneme sequence as (multiple) predicted outputs 117. In doing so, the on-device ASR engine 132A2 can optionally employ decoding graphs, dictionaries, and / or (multiple) other resources.

[0035] It is worth noting that, Figure 1CIn the example, the predicted output 117 can include multiple speech hypotheses of multiple text segments 115A generated by the ASR model across devices, and / or synthesized speech audio data 116 (and / or its acoustic features). For example, if the multiple text segments 115A processed by the ML engine 132A on the device correspond to the text segment “Ron Smith” obtained as a name associated with a contact entry on a client device. In this example, the TTS engine 132A1 on the device can process the text segment to generate synthesized speech audio data including synthesized speech corresponding to the contact entry “Ron Smith”. Furthermore, the synthesized speech (or its acoustic features) corresponding to “Ron Smith” can be used as the predicted output 117 and subsequently used to generate text-based gradients to update the TTS model on the device, and multiple speech hypotheses can be used as the predicted output 117 and subsequently used to generate ASR gradients to update the ASR model on the device. For example, gradient engine 134A can compare acoustic features corresponding to the synthesized speech of “Ron Smith” (e.g., Mel filter bank features, audio waveform, prosodic properties, MFCC and / or other acoustic features) with audio data of “Ron Smith”’s spoken utterance provided by the user of client device 150 (e.g., multiple corresponding ground reality outputs 115B) to generate multiple text-based client gradients 118. Furthermore, gradient engine 134A can compare multiple speech hypotheses of “Ron Smith” with actual text segments of “Ron Smith” 110 (e.g., multiple corresponding ground reality outputs 115B) to generate multiple ASR client gradients 118.

[0036] Furthermore, client gradients 118 can be transmitted to remote system 160 and stored in gradient database 185. Remote data 104A can be text fragments 119A obtained from remote data database 152B. Remote system 160 can process text fragments 119A in the same or similar manner as described above with respect to client device 150 to generate prediction outputs 121, but using global ML engine 132B (e.g., global TTS engine 132B1, which generates synthesized speech audio data 120, and global ASR engine 132B2, which generates prediction text fragments as part of prediction outputs 121) and corresponding audio-based global ML models stored in global ML model database 154B. Furthermore, the gradient engine 134B is able to compare (multiple) predicted outputs 121 with (multiple) corresponding ground reality outputs 119B to generate (multiple) remote gradients 122, and the (multiple) remote gradients 122 can be stored in (multiple) gradient databases 185.

[0037] In additional or alternative implementations, and now turning to Figure 1D Client data 101A can be image data 124A generated by the visual components(s) of client device 150. Image data 124A can capture, for example, one or more parts of the environment in which client device 150 is located, including people, objects, and / or anything else in the environment that can be captured in image data 124A. Although referred to as image data, it should be understood that this can include still images and image sequences (e.g., videos). In an implementation where client device 101A is image data 124A, on-device ML engine 132A can process image data 124A using one or more on-device image-based ML models to generate prediction outputs(s) 125, which are used to generate image-based client gradients. For example, the on-device ML engine 132A can process image data 124A using the following methods: an on-device object detection model can process image data 124A to detect objects or features captured in the environment of client device 150; an on-device object classification model can process image data 124A to predict the classification(s) of objects captured in the environment of client device 150; an on-device face recognition model can process image data 124A to determine whether any human captured in the environment corresponds to a specific user (e.g., a user of client device 150); and / or any other on-device image-based ML model. It should be understood that the aforementioned on-device ML models(s) are provided for illustrative purposes only and are not intended to be limiting.

[0038] It is worth noting that the predicted outputs 110 can include corresponding predicted outputs generated across one or more of the aforementioned on-device ML models. For example, if image data 124A is processed using an on-device object detection model and an on-device object classification model, the predicted outputs 125 can include corresponding predicted outputs associated with the on-device object detection model and corresponding predicted outputs associated with the on-device object classification model. Therefore, the predicted outputs 125 can include predicted outputs generated across one or more of the on-device ML models. Thus, any resulting client gradient can be a gradient of a specific type based on the on-device ML models used to process image data 124A.

[0039] In some implementations, the gradient engine 134A is capable of comparing (multiple) predicted outputs 125 with (multiple) corresponding ground reality outputs 124B to use supervised learning techniques (e.g., regarding...). Figure 3The description refers to generating client gradients 126. In additional or alternative implementations, such as when the corresponding ground reality output 124B is unavailable, the gradient engine 134A can utilize unsupervised learning techniques to generate (multiple) client gradients 126 (e.g., as described in...). Figure 3 (As described). In other words, the gradient engine 134A is able to evaluate the predicted outputs 125 relative to some actual outputs that should be generated across the model to identify errors generated by the ML model across the 125 devices. For example, suppose the predicted outputs generated by the hot word recognition model across the devices are probabilities (e.g., or other predictive metrics, such as log-likelihood, binary values, or other metrics) that indicate whether the image data 124A includes a user associated with client device 150. Further assume the probability is 0.65, and the threshold probability for classifying a person as a user of client device 150 is 0.85. In this example, a user of client device 150 may not be identified in this way. However, further assume that the automated assistant is invoked within a threshold time period of receiving the image data 124A via another authentication measure, such as a password from client device 150. Client device 150 can use this as a supervisory signal to indicate that the person captured in the image data 124A is actually a user of client device 150 (e.g., a false negative). In this example, the corresponding ground-based output can be a probability of 1.0 to indicate that image data 124A includes the user of client device 150, and the resulting client gradient can be a false negative gradient of (multiple) face recognition models.

[0040] Furthermore, client-side gradients 126 can be transmitted to remote system 160 and stored in gradient databases 185. Remote data 104A can be image data 127A obtained from remote data database 152B. Remote system 160 can process image data 127A in the same or similar manner as described above with respect to client device 150 to generate prediction output 128, but using global ML engine 132B and corresponding audio-based global ML models stored in global ML model databases 154B. Furthermore, gradient engine 134B can compare the prediction outputs 128 with corresponding ground reality outputs 127B to generate image-based remote gradients 129, and the remote gradients 129 can be stored in gradient databases 185.

[0041] Back Figure 1AIn various implementations, the remote training engine 136 is capable of analyzing gradient distributions. In analyzing the distributions, the remote training engine 136 can identify the types of client gradients already received at the remote system 160 from the client device 150 and / or the types of remote gradients already generated at the remote system 160. Furthermore, the remote training engine 136 can determine the need to generate specific types of gradients based on the absence of specific types of gradients or a sufficient quantity of gradients in the gradient distribution. In response to determining the need to generate these specific types of gradients, the remote training engine 136 can enable the remote system 160 to generate those specific types of gradients (and optionally, a specific number of them). For example, suppose multiple hot-word free-invoking client gradients have already been generated at the client device 150 based on gestures provided by the user of the client device 150 to invoke the automatic assistant (e.g., based on hand movements, eye movements, lip movements, etc.). However, it may be difficult to obtain hot-word free-invoking client gradients at the client device 150 that do not invoke the automatic assistant (e.g., due to privacy considerations). In this example, the remote training engine 136 can determine that gradients are needed that include gestures made by a human that are not intended to invoke the automatic assistant. Therefore, the remote system 160 is able to acquire remote data, which is image data of humans making gestures without invoking an automatic assistant, to ensure that there is sufficient quantity and diversity in the gradients used to update the global hot word free invocation model.

[0042] In other words, the long-range gradients generated at the long-range system 160 can be used as proxies for client data that is difficult to obtain. For example, this data can include false positives and false negatives from (multiple) ML models trained for prediction (e.g., hot word detection models, hot word free-call models, speech recognition models, face recognition models, etc.). Furthermore, the long-range gradients generated at the long-range system 160 can be generated based on specific words or phrases identified by the long-range training engine 136, which typically cause problems and errors in the various ML models described herein. For example, homophones can cause errors in ASR models and hot word detection models. The long-range system 160 can generate long-range gradients based on homophones or phrases that typically cause errors in these models to strengthen (multiple) ML models and avoid catastrophic forgetting.

[0043] Various technical advantages can be achieved by using the techniques described herein. As a non-limiting example, when utilizing both client-side gradients and remote gradients generated in the manner described herein, the corresponding global ML model can be updated based on diverse and balanced data, thereby reducing the impact of catastrophic forgetting and producing a more stable global ML model with higher precision and / or recall. For example, remote systems can generate remote gradients based on the identification requirements of specific types of gradients for a particular global ML model, models that might not be available within a federated learning framework for privacy reasons without these techniques. Therefore, when the corresponding global ML model (or its weights) is deployed to client devices, the on-device ML model is also more stable and has higher precision and / or recall.

[0044] Turn now Figure 2 The client device 250 is illustrated in the implementation, depicting various ML engines included as part of (or communicating with) the automated assistant client 260. The interfaces of the corresponding ML models with the ML engines on various devices are also illustrated. For simplicity, other components of the client device 250 are not shown. Figure 2 The diagram is shown in the image. Figure 2 The illustration shows an example of how the Auto Assistant client 260 can leverage ML engines and their corresponding ML models on various devices when performing various actions.

[0045] Figure 2 The client device 250 is illustrated as having one or more microphones 211, one or more speakers 212, one or more vision components 213, and a display 214 (e.g., a touch-sensitive display). The client device 250 may also include (multiple) pressure sensors, (multiple) proximity sensors, (multiple) accelerometers, (multiple) magnetometers, and / or (multiple) other sensors for generating sensor data other than the audio data captured by the one or more microphones 211. The client device 250 at least selectively executes the automated assistant client 260. Figure 2 In the example, the automated assistant client 260 includes a hot word detection engine 222, a hot word free-calling engine 224, a dialogue continuation engine 226, an ASR engine 228, an object detection engine 230, an object classification engine 232, a voice recognition engine 234, and a face recognition engine 236. The automated assistant client 260 also includes a voice capture engine 216 and a visual capture engine 218. It should be understood that... Figure 2The ML engines and ML models described herein are provided for illustrative purposes only and are not intended to be limiting. For example, the automated assistant client 260 may further include additional and / or alternative engines, such as a TTS engine and a corresponding TTS model, a VAD engine and a corresponding VAD model, an endpoint detector engine and a corresponding endpoint detector model, a lip movement engine, and / or (multiple) other engines and (multiple) associated machine learning models. Furthermore, it should be understood that one or more engines and / or models described herein can be combined such that a single engine and / or model can perform the functionality of multiple engines and / or models described herein.

[0046] One or more cloud-based automated assistant components 270 can optionally be configured via... Figures 1A-1D The described one or more network communications are coupled to one or more computing systems (collectively referred to as "cloud" computing systems) of client device 250, generally indicated by 299. The cloud-based automated assistant component 270 can be implemented, for example, via a high-performance server cluster. In various implementations, an instance of the automated assistant client 260, through its interaction with one or more cloud-based automated assistant components 270, can form a logical instance of an automated assistant from the user's perspective, as indicated by 295, with which the user can perform human-computer interaction (e.g., verbal interaction, gesture-based interaction, and / or touch-based interaction).

[0047] Client device 250 can be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker, a smart device such as a smart TV (or a standard TV equipped with a network dongle with automatic assistant functionality), and / or a wearable device for the user that includes the computing device (e.g., a watch for the user with the computing device, glasses for the user with the computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0048] One or more vision components 213 can take various forms, such as a single-image camera, a stereo camera, a LiDAR component (or (multiple) other laser-based components), a radar component, etc. One or more vision components 213 can be used, for example, via a vision capture engine 218, to capture image data corresponding to visual frames (e.g., image frames, laser-based visual frames) of the environment in which the client device 250 is deployed. In some implementations, such visual frames (multiple) can be used to determine whether a user is near the client device 250 and / or the distance of the user (e.g., the user's face) relative to the client device 250. Such determinations (multiple) can be used, for example, to determine whether activation is required. Figure 2The machine learning engine and / or (multiple) other engines are depicted on various devices. Furthermore, the voice capture engine 218 can be configured to capture (multiple) spoken words of a user and / or other audio data captured via one or more microphones 211.

[0049] As described in this article, this audio data and other non-microphone sensor data (also known as client data) can be obtained by... Figure 2 The various engine processes described above are used to process the above-mentioned... Figures 1A-1D The corresponding ML model (which includes the updated global ML model and / or its updated weights) generated in the manner described is used for prediction on client device 250.

[0050] As some non-limiting examples, the hot word detection engine 222 can use the hot word detection model 222A to predict whether the audio data includes one or more specific words or phrases used to invoke the automatic assistant 295 (e.g., "Ok Google", "Hey Google", "What is the weather"). Google? (How's the weather? Google) or certain functions of the Auto Assistant 295; the Hot Word Free Call Engine 224 can use the Hot Word Free Call Model 224A to predict whether non-microphone sensor data (e.g., image data) includes gestures or signals for calling the Auto Assistant 295 (e.g., based on the user's gaze and optionally further based on the user's mouth movements); the Continue Conversation Engine 226 can use the Continue Conversation Model 226A to predict whether other audio data is for the Auto Assistant 295 (e.g., or for an additional user in the environment of the client device 250); the ASR Engine 228 can use the ASR Model 228A to generate recognized text, or predict (multiple) phonemes and / or (multiple) markers corresponding to the audio data detected at the client device 250, and generate recognized text based on (multiple) phonemes and / or (multiple) markers; the Object Detection Engine 230 can use the Object Detection Model 224A to generate recognized text. 30A is used to predict the location of (multiple) objects in image data including images captured at client device 250; object classification engine 232 is able to use object classification model 232A to predict the object classification of (multiple) objects in image data including images captured at client device 250; voice recognition engine 234 is able to use voice recognition model 234 to predict whether audio data captures spoken words of one or more users of client device 250 (e.g., by generating speaker embeddings or other representations that can be compared with corresponding actual embeddings of one or more users of client device 250); and face recognition engine 236 is able to use face recognition model to predict whether image data captures one or more users in the environment of client device 250 (e.g., by generating image embeddings or other representations that can be compared with corresponding image embeddings of one or more users of client device 250).

[0051] In some implementations, the client device 250 may also include a Natural Language Understanding (NLU) engine 238 and an execution engine 240. The NLU engine 238 may utilize an NLU model 238A to perform on-device natural language understanding on the recognized text, predicted phonemes, and / or predicted markers generated by the ASR engine 228 to generate NLU data. The NLU data may include, for example, multiple slot values ​​corresponding to multiple intentions of spoken utterances and optionally multiple parameters of those intentions. Furthermore, the execution engine 240 may utilize the on-device execution model 240A and generate execution data based on the processing of the NLU data. This execution data may define local and / or remote responses (e.g., answers) to spoken utterances provided by a user of the client device 250, interactions with locally installed applications based on spoken utterances, commands transmitted to multiple Internet of Things (IoT) devices (directly or via multiple corresponding remote systems) based on spoken utterances, and / or other resolving actions performed based on spoken utterances. Then, fulfillment data is provided for local and / or remote execution / enforcement of the defined actions to parse spoken utterances. Execution can include, for example, presenting local and / or remote responses (e.g., visual and / or auditory presentation (optionally utilizing an on-device TTS module)), interacting with locally installed applications, transmitting commands to (multiple) IoT devices, and / or (multiple) other actions. In other implementations, the NLU engine 234 and fulfillment engine 240 can be omitted, and the ASR engine 228 can generate fulfillment data directly based on audio data. For example, suppose the ASR engine 228 uses the ASR model 228A to process the spoken utterance “turn on the light.” In this example, the ASR engine 228 can generate semantic output and then transmit it to the software application associated with the light and / or directly to the light indicating that it should be turned on.

[0052] It is worth noting that (multiple) cloud-based automated assistant components 270 include those mentioned in this article. Figure 2 The described engine and model have a cloud-based counterpart. However, in various implementations, these engines and models may not be invoked, as they can be directly transferred to client device 250 and executed locally on client device 250, as mentioned above. Figures 1A-1DThe description is as follows. Nevertheless, a remote execution module can optionally be included, which performs remote execution based on locally or remotely generated NLU data and / or performance data. Additional and / or alternative remote engines can be included. As described herein, among various implementations, on-device speech processing, on-device image processing, on-device NLU, on-device performance, and / or on-device execution can be preferred, at least due to the reduced latency and / or network usage they provide when parsing spoken utterances (since no client-server round trips are required to parse spoken utterances). However, one or more cloud-based automated assistant components 270 can be selectively utilized at least. For example, such components can be utilized in parallel with on-device components, and output from such components can be utilized when the local components fail. For example, if any on-device engine and / or model fails (e.g., due to relatively limited resources on client device 150), the more stable resources of the cloud can be utilized.

[0053] Turn now Figure 3 The diagram illustrates a flowchart of an example method 300 for generating client-side gradients locally at a client device to update multiple machine learning models. For convenience, the operation of method 300 is described with reference to the system performing the operation. The system of method 300 includes client devices (e.g., client device 150 of Figure 1). Figure 2 250 client devices Figure 6 The computing device 610 and / or other client devices may use one or more processors and / or (multiple) other components. Furthermore, although the operations of method 300 are shown in a specific order, this is not a limitation. One or more operations may be reordered, omitted, or added.

[0054] In box 352, the system identifies client data at the client device. In some implementations, box 352 includes an optional subbox 352A. In optional subbox 352A, the system detects audio data that captures spoken utterances in the client device's environment via one or more microphones of the client device. The audio data may capture at least a portion of the user's spoken utterances on the client device. In other implementations, box 352 includes an optional subbox 352B. In optional subbox 352B, the system detects non-microphone sensor data via multiple non-microphone sensors of the client device. The non-microphone sensor data may include, for example, image data of the client device's environment captured via multiple visual components and / or text fragments of the client device.

[0055] In box 354, the system uses on-device ML models stored locally on the client device to process client data to generate multiple predictive outputs. The multiple predictive outputs generated using on-device ML models can be based on the client data identified in box 352 and the on-device ML models used to process the client data in box 354. For example, in an implementation where the client data corresponds to audio data, one or more audio-based models can be used to process the client data to generate corresponding predictive outputs. For example, a hot word model can be used to process the audio data to generate predictive outputs associated with whether the audio data contains a specific word or phrase used to invoke an automated assistant, and an ASR model can be used to process the audio data to generate predictive outputs associated with predictive text related to the audio data, etc. As another example, in an implementation where the client data corresponds to image data, one or more image-based models can be used to process the client data to generate corresponding predictive outputs. For example, a hot word free invocation model can be used to process image data to generate predictive outputs associated with whether the image data includes a gesture or mouth movement used to invoke an automated assistant, an object recognition model can be used to process image data to generate predictive outputs associated with the classification of one or more objects captured in the image data, and so on.

[0056] In box 356, the system generates gradients based on the predicted output. In some implementations, box 356 includes an optional subbox 356A. In optional subbox 356A, the system generates gradients locally on the client device using supervised learning. In these implementations, the system is able to compare the predicted output with a corresponding ground truth output. The system can utilize one or more supervisory signals as the corresponding ground truth output. For example, suppose the client device generates audio data capturing a specific word or phrase used to invoke the auto-assistant, but the predicted output of the generated hot word model indicates that the audio data does not include the specific word or phrase used to invoke the auto-assistant. Further suppose the user subsequently invokes the auto-assistant (e.g., via additional spoken words or other user input used to invoke the auto-assistant (e.g., squeezing the client device or touch input to an auto-assistant graphical element)). In this example, the system is able to determine that the auto-assistant should be invoked based on the spoken words (e.g., a spurious negative), and the corresponding ground truth output of the predicted output indicates that the auto-assistant should have been invoked.

[0057] As another example, suppose a client device generates audio data capturing a specific word or phrase used to invoke an automated assistant, and the predicted output of the generated hot word model indicates that the audio data includes the specific word or phrase used to invoke the automated assistant. Further suppose the user subsequently cancels the invocation of the automated assistant (e.g., via additional spoken words or other user input used to invoke the automated assistant (e.g., squeezing the client device or touch input to an automated assistant graphical element)). In this example, the system is able to determine that the automated assistant should not be invoked based on spoken words (e.g., false positive), and the corresponding ground truth output of the predicted output indicates that the automated assistant should not be invoked. Although box 356 describes supervision signals for the hot word model, it should be understood that this is merely illustrative and not restrictive, and any supervision signals generated based on user feedback and / or inferred by the system can be utilized. For example, if an ASR model processes audio data and presents a transcription of one or more ASR speech hypotheses to the user, any modifications to the transcription can be used as a supervision signal. Furthermore, for example, if a speech recognition model processes audio data to verify a user's identity, but the user is not identified, and the user subsequently provides a password or other authentication information, that other authentication information can be used as a supervision signal.

[0058] In other implementations, box 356 includes optional sub-boxes 356A and 356B. In optional sub-box 356B, the system generates gradients locally on the client device using unsupervised (or semi-supervised) learning. Notably, in these implementations, no supervisory signal is provided for the system to generate gradients. For example, suppose the client data identified on the client device is image data captured via multiple vision components of the client device, and an image-based ML model on the device attempts to reconstruct the image data based on the encoding of the image data, thereby producing predicted image data. In this example, the system is able to compare the image data with the predicted image data to determine the differences between them, and the system is able to generate gradients based on the determined differences. As another example, suppose the client data detected on the client device is audio data, comprising a first part and a second part following the first part, captured via multiple microphones of the client device, and suppose an ML model on the device seeks to predict the second part of the audio data based on the encoding of the first part of the audio data, thereby producing the predicted second part of the audio data. In this example, the system can compare the second part of the audio data with the predicted second part of the audio data to determine the differences between them, and the system is able to generate gradients based on the determined differences.

[0059] In box 358, the system determines whether the conditions(s) for transmitting the gradients generated in box 356 are met. These conditions(s) can include, for example, that the client device is charging, the client device has at least a threshold charging state, the temperature of the client device (based on temperature sensors on one or more devices) is below a threshold, the client device is not held by a user, multiple time conditions associated with the client device(s) (e.g., every N hours between specific time periods, where N is a positive integer, and / or multiple other time conditions associated with the client device(s)), whether a threshold number of gradients has been generated by the client device, and / or multiple other conditions(s). If, at the iteration in box 358, the system determines that the conditions(s) for transmitting the gradients generated in box 356 are not met, the system can continue to monitor whether the conditions(s) are met in box 358. Notably, while the system monitors the satisfaction of the conditions(s) in box 358, the system can continue to generate multiple additional gradients according to boxes 352-356 of method 300. If, during the iteration in box 358, the system determines that the conditions(s) for transmitting the gradient generated in box 356 are satisfied, then the system is able to proceed to box 360.

[0060] In box 360, the system transmits the generated gradients from the client device to the remote system so that the remote system can use the generated gradients to update the weights of the remote global ML model(s) stored on the remote system. Furthermore, multiple additional client devices are capable of generating additional gradients according to method 300, and can transmit these additional gradients to the remote system when multiple corresponding conditions are met at the additional client devices (e.g., additional client device 170 in Figure 1).

[0061] In box 362, the system receives (multiple) updated ML models and / or their updated weights at the client device and from the remote system. It is noteworthy that boxes 360 and 362 are not connected by an arrow. This indicates that the client device receives (multiple) updated ML models and / or their updated weights when the remote system transmits (multiple) updated ML models and / or their updated weights to the client device based on one or more conditions met by the client device and / or the remote system. The client device is able to replace the corresponding ML model and / or its corresponding weights on the corresponding device with the (multiple) updated ML models and / or their updated weights in its local memory.

[0062] In box 364, the system uses the updated ML model to make at least one prediction based on additional client data identified at the client device. The prediction made at the client device may depend on the updated ML model received from the remote system in box 362.

[0063] Although Figure 3Method 300 is generally described in relation to audio-based model descriptions, but it should be understood that this is for illustrative purposes only and not as a limitation. For example, the techniques described herein can also be used to generate client-side gradients and transmit them to remote systems to update image-based models, text-based models, and / or any other ML models. Furthermore, they can be executed in parallel on the user's and / or other users' client devices and / or attached client devices. Figure 3 Multiple instances of method 300 are used to generate various gradients and / or transmit various gradients to remote systems.

[0064] Turn now Figure 4 This diagram illustrates a flowchart of an example method 400 for remotely generating (multiple) remote gradients at a remote system to update (multiple) machine learning models. For convenience, the operation of method 400 is described with reference to the system performing the operation. The system of method 400 includes remote systems (e.g., remote system 160 in Figure 1, ...). Figure 6 The computing device 610 and / or any other computing device may contain one or more processors and / or (multiple) other components. Furthermore, although the operations of method 400 are shown in a specific order, this is not a limitation. One or more operations may be reordered, omitted, or added.

[0065] In box 452, the system receives multiple client gradients from multiple client devices at a remote system location. These client gradients are generated locally on the multiple client devices based on corresponding client data. This is possible with respect to the description... Figure 3 Method 300 generates each of the multiple client gradients in the same or similar manner on multiple client devices.

[0066] In box 454, the system generates multiple remote gradients. For example, box 454 can perform the operations of sub-boxes 454A, 454B, and 454C to generate remote gradients. In sub-box 454A, the system obtains remote data accessible to the remote system. In some implementations, remote data can be retrieved from online video sharing platforms, image sharing platforms, audio sharing platforms, and / or text-based platforms, which are not as access-restricted as client data generated locally on a corresponding client device and / or received locally on a corresponding client device. In additional or alternative implementations, and with the permission of a corresponding user on one of the multiple client devices, the remote data can be access-restricted data accessible to the remote system. Furthermore, the remote data can correspond to data generated by devices with different characteristics (e.g., different types of microphone(s), different types of visual components(s), etc.). In sub-box 454B, the system processes the remote data using multiple global ML models to generate multiple prediction outputs. In sub-box 454C, the system generates remote gradients based on the multiple prediction outputs for inclusion in the multiple remote gradients. The system can return to subframe 454A to obtain additional long-range data to generate additional gradients, and / or return to subframe 454B to generate additional prediction outputs based on the same long-range data. The system is capable of processing long-range data to correlate with... Figure 3 Method 300 describes the same or similar approach as box 354, but uses (multiple) global ML models instead of (multiple) on-device ML models to generate (multiple) predictive outputs. Furthermore, the system is able to... Figure 3 The same or similar approach described in box 356 of method 300 is based on (multiple) predicted outputs, but generates remote gradients based on (multiple) predicted outputs generated from remote data.

[0067] In some implementations, the system can repeat the process of generating remote gradients until a threshold number of remote gradients are generated. For example, the system can continue generating remote gradients until the number of gradients matches the number of client gradients received from multiple client devices, a threshold number defined by the developer, or some other quantitative threshold. In some versions of those implementations, the threshold number of remote gradients generated by the remote system can be based on the corresponding type of client gradients received from multiple client devices. For example, suppose 1,000 client gradients are received from multiple client devices, and suppose 500 of the 1,000 client gradients are audio-based client gradients, while the other 500 are image-based client gradients. In this example, the system can obtain audio-based remote data, process the audio-based remote data using various audio-based global ML models to generate (multiple) prediction outputs, and process (multiple) prediction outputs to generate audio-based remote gradients. This process can be repeated until there are 500 audio-based remote gradients. Further assuming that of these 500 audio-based client gradients, 300 are client-side hot word gradients, and the remaining 200 are client-side ASR gradients. In this example, the system is able to process remote audio data using (multiple) global hot word models to generate 300 remote hot word gradients, and is able to process remote audio data using (multiple) ASR models to generate 200 remote ASR gradients.

[0068] In box 456, the system selects a set of client gradients from the multiple client gradients received in box 452. In some implementations, the system can select the set of client gradients based on one or more criteria. These criteria can include, for example, the type of client gradients available to the system, a threshold number of client gradients available to the system, a threshold duration for updating using client gradients, and / or other criteria. For example, the system can select a set of 500 audio-based client gradients to update (e.g., a single audio-based model or multiple different audio-based models), select a set of 200 hot word client gradients to update (e.g., multiple hot word models), and so on. As another example, the system can select a set of 600 image-based client gradients to update (e.g., a single image-based model or multiple different image-based models), select a set of 300 hot word client gradients to update (e.g., multiple hot word free-call models), and so on. In other implementations, the set of client gradients can include any combination of audio-based client gradients, image-based client gradients, text-based client gradients, and / or any other type of client gradient (or a specific type of client gradient).

[0069] In box 458, the system selects a set of remote gradients from a plurality of remote gradients. In some implementations, box 458 includes an optional subbox 458A. In optional subbox 458A, the system selects the set of remote gradients based on the type of gradients included in the set of client gradients. For example, if the set of client gradients selected in box 456 includes a set of audio-based gradients (or a set of specific types of audio-based gradients), the set of remote gradients selected in box 458 can also include audio-based gradients. In additional or alternative implementations, box 458 includes an optional subbox 458B. In optional subbox 458B, the system additionally or alternatively selects the set of remote gradients based on the number of client gradients included in the set of client gradients. For example, if the set of client gradients selected in box 456 includes a set of 500 audio-based gradients (or a set of specific types of audio-based gradients), the set of remote gradients selected in box 458 can also include 500 audio-based gradients. As another example, if the set of client gradients selected in box 456 includes a set of 500 audio-based gradients (or a set of specific types of audio-based gradients), the set of remote gradients selected in box 458 can also include a proportional number of audio-based gradients (e.g., 250 represents 50%, 100 represents 20%, etc.). In other implementations, the set of remote gradients can include any combination of audio-based remote gradients, image-based remote gradients, text-based remote gradients, and / or any other type of remote gradient (or a set of specific types of remote gradients).

[0070] In box 460, the system updates the weights of(multiple) global ML models using the set of client gradients and the set of remote gradients. Specifically, the system can update the weights of(multiple) corresponding ML models based on gradients included in the set of client gradients and / or the set of remote gradients. For example, and briefly refer to... Figure 5 , describes in Figure 4 Box 460 shows various example methods for mixing a set of client gradients and a set of remote gradients when updating a global machine learning model (multiple models).

[0071] In some implementations, and such as Figure 5As shown in method 460A, in block 552A, the system is able to update the weights of(multiple) global ML models using a set of client gradients, and then further update the weights of(multiple) global ML models using a set of remote gradients. For example, suppose the set of client gradients includes multiple ASR client gradients and suppose the set of remote gradients includes multiple ASR remote gradients. In this example, the system is able to update the weights of one or more global ASR models based on multiple ASR client gradients, and then subsequently update the weights of one or more global ASR models based on multiple ASR remote gradients.

[0072] In other implementations, and such as Figure 5 As shown in method 460B, in block 552B, the system is able to update the weights of(multiple) global ML models using a set of remote gradients, and then further update the weights of(multiple) global ML models using a set of client gradients. For example, suppose the set of client gradients includes multiple ASR client gradients and suppose the set of remote gradients includes multiple ASR remote gradients. In this example, the system is able to update the weights of one or more global ASR models based on multiple ASR remote gradients, and then subsequently update the weights of one or more global ASR models based on multiple ASR client gradients.

[0073] In other implementations, and such as Figure 5 As shown in method 460C, in block 552C1, the system is able to update the first weights of (multiple) first instances of (multiple) global ML models using a set of client gradients, and in block 552C2, update the second weights of (multiple) second instances of (multiple) global ML models using a set of second gradients in parallel or serial manner. Furthermore, the system is then able to update the weights of the global ML models using the first and second weights. For example, suppose the set of client gradients includes multiple ASR client gradients and suppose the set of remote gradients includes multiple ASR remote gradients. In this example, the system is able to update the first weights of one or more first instances of one or more global ASR models based on multiple ASR client gradients, and update the second weights of one or more second instances of one or more global ASR models in parallel or serial manner based on multiple ASR remote gradients. Furthermore, the system is able to analyze the updated first weights and updated second weights, and update one or more global ASR models using some combination of the updated first weights and updated second weights. For example, the system is able to determine the average of the updated first weights and updated second weights, and use this average weight as the updated weights of one or more global ASR models.

[0074] Although the ASR gradients used to update one or more global ASR models are described...Figure 5 Methods 460A, 460B, and 460C are provided, but it should be understood that they are for illustrative purposes only and not as limitations. For example, the techniques described herein can be used to update various global ML models or their weights based on corresponding gradients associated with various global ML models.

[0075] Back Figure 4 In some alternative implementations, the system can update the weights of multiple global ML models using multiple client gradients received in box 452 and multiple remote gradients 454 generated in box 456 without performing the operations in boxes 456 and 458. In these implementations, the system can process multiple client gradients and multiple remote gradients in a FIFO manner. For example, the system can identify the types of the received client gradients and / or the types of the generated remote gradients, identify the corresponding global ML models associated with the types of client and / or remote gradients, and update the corresponding global ML models when the remote systems receive and / or generate gradients.

[0076] It is worth noting that the system can repeat the operations of boxes 452-460 (or simply 452, 454, and 460) until the update of (multiple) global ML models is complete. The system can determine the completion of the update of (multiple) global ML models based on, for example, a threshold duration of time spent updating (multiple) global ML models, a threshold number of gradients (remote gradients and / or client gradients) since the last update of the weights of the global ML models, a measured improvement on (multiple) global ML models, and / or a threshold duration elapsed since the last training of the global ML models, and / or other criteria. Once the (multiple) global ML models have been updated, the system can proceed to box 458.

[0077] In box 462, the system determines whether multiple conditions for transmitting the global ML model(s) and / or its weights(s) in box 460 are met. These conditions(s) can be based on whether the client devices(s) are ready to receive the global ML model(s) and / or its weights(s) (e.g., in accordance with the above regarding...). Figure 3The conditions described in box 358 are the same as those in the previous section. Other conditions specific to the remote system include, for example, the performance of (multiple) global ML models based on a performance threshold, the number of global ML models updated based on a threshold of gradients, and / or some combination of these conditions. If, in the iteration of box 462, the system determines that the conditions for transferring (multiple) global ML models and / or their weights updated in box 460 are not met, then in box 462, the system is able to continue monitoring whether the conditions are met. Notably, in box 462, while the system monitors the satisfaction of the conditions, the system is able to continue updating the (multiple) global ML models according to boxes 452-460 of method 400. If, in the iteration of box 462, the system determines that the conditions for transferring (multiple) global ML models and / or their weights updated in box 460 are met, then the system is able to proceed to box 464.

[0078] In box 464, the system transmits (multiple) updated global ML models and / or their weights to one or more of a plurality of client devices and from a remote system. The system is capable of transmitting (multiple) updated global ML models and / or their weights to each of the plurality of client devices that transmit client gradients to the remote system, additional client devices besides those that transmit client gradients to the remote system, or a subset of those client devices that transmit client gradients to the remote system.

[0079] In various implementations, it is possible to transfer (multiple) updated global ML models and / or their weights to client devices based on geographic region and / or other attributes (e.g., primary language) of multiple client devices or users of multiple client devices. For example, (multiple) updated global ML models could be for a given language. N One of the available machine learning models (e.g., where) N (It is a positive integer), but it can be updated and provided to client devices that are primarily located in a specific geographic region based on client gradients specific to that geographic region. For example, this allows instances of (multiple) global ASR models to be trained based on region-specific client ASR gradients and to be updated to account for different accents and dialects.

[0080] although Figure 4 Method 400 is generally described in relation to audio-based models and should be understood as illustrative rather than limiting. For example, the techniques described herein can also be used to transmit client gradients to remote systems for updating image-based models, text-based models, and / or any other ML models. Furthermore, the remote system can execute these techniques in parallel. Figure 4Multiple instances of method 400 are used to generate various remote gradients and / or update various (multiple) global ML models.

[0081] Turn now Figure 6 This diagram depicts a block diagram of an example computing device 610 that may be optionally used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a plurality of cloud-based automated assistant components, and / or a plurality of other components may include one or more components of the example computing device 610.

[0082] Computing device 610 typically includes at least one processor 614 that communicates with a plurality of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624, including, for example, a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow users to interact with computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0083] User interface input device 622 may include a keyboard, a pointing device such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. Generally, the term "input device" is used to encompass all possible types of devices and methods of inputting information onto computing device 610 or a communication network.

[0084] User interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual displays, such as via an audio output device. Generally, the term "output device" is used to encompass all possible types of devices and the manner in which information is output from computing device 610 to the user or another machine or computing device.

[0085] Storage subsystem 624 stores the programming and data construction capabilities that provide some or all of the functionality of the modules described herein. For example, storage subsystem 624 may include selected aspects for performing the methods disclosed herein and for implementing... Figures 1A-1D The logic of the various components described in the text.

[0086] These software modules are typically executed by processor 614 alone or in conjunction with other processors. The memory 625 used in storage subsystem 624 can include multiple memories, including main random access memory (RAM) 630 for storing instructions and data during program execution and read-only memory (ROM) 632 for storing fixed instructions. File storage subsystem 626 provides persistent storage for program and data files and may include hard disk drives, floppy disk drives and associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing certain functionalities may be stored by file storage subsystem 626 within storage subsystem 624 or in other machines accessible to processor(s) 614.

[0087] The bus subsystem 612 provides a mechanism for enabling various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0088] The computing device 610 can be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, Figure 6 The description of the computing device 610 depicted herein is intended only as a specific example for illustrating some implementation methods. Many other configurations of the computing device 610 may have... Figure 6 The computing device depicted in the text has more or fewer components.

[0089] In situations where the systems described herein collect or otherwise monitor personal information about users, or where personal and / or monitoring information may be used, users may have the opportunity to control whether programs or functions collect user information (e.g., information about a user's social networks, social behaviors or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from content servers. Furthermore, some data may be processed in one or more ways before storage or use to erase personally identifiable information. For example, a user's identity may be processed to make it impossible to determine the user's personally identifiable information, or, where geographic location information is obtained, the user's geographic location may be generalized (e.g., down to the city, zip code, or state level) to make it impossible to determine the user's specific geographic location. Therefore, users can control how information about themselves is collected and / or used.

[0090] In some implementations, a method is provided executed by one or more processors of a client device, the method comprising receiving a plurality of client gradients from a plurality of corresponding client devices. Each of the plurality of client gradients is locally generated based on processing corresponding audio data captured at least a portion of a corresponding spoken utterance of a corresponding user in a given of the plurality of corresponding client devices. The method further comprises generating a plurality of long-range gradients. Generating each of the plurality of long-range gradients comprises obtaining additional audio data captured at least a portion of additional spoken utterance of an additional user; processing the additional audio data using a global machine learning (ML) model remotely stored in a remote system to generate a predicted output; and generating an additional gradient to be included in the plurality of long-range gradients based on comparing the additional predicted output with a ground truth output corresponding to the additional audio data. The method further comprises selecting a set of client gradients from the plurality of client gradients; selecting a set of additional long-range gradients from the plurality of long-range gradients; and updating the weights of a global ML model using the set of client gradients and the set of additional long-range gradients.

[0091] These and other implementations of this technology can include one or more of the following features.

[0092] In some implementations, updating the weights of the global ML model using the set of client gradients and the set of additional remote gradients can include updating the weights of the global ML model using the set of client gradients; and after updating the weights of the global ML model using the set of additional remote gradients, further updating the weights of the global ML model using the set of additional remote gradients.

[0093] In some implementations, updating the weights of the global ML model using the set of client gradients and the set of additional long gradients can include updating the weights of the global ML model using the set of additional long gradients; and after updating the weights of the global ML model using the set of additional long gradients, further updating the weights of the global ML model using the set of client gradients.

[0094] In some implementations, updating the weights of the global ML model using a set of client gradients and a set of additional long-range gradients can include updating the first weights of a first instance of the global ML model using the set of client gradients; updating the second weights of a second instance of the global ML model in parallel using the set of additional long-range gradients to be updated; and updating the weights of the global ML model using the updated first weights of the first instance of the global ML model and the updated second weights of the second instance of the global ML model. In some versions of those implementations, updating the weights of the global ML model using the updated first weights of the first instance of the global ML model and the updated second weights of the second instance of the global ML model can include determining the updated first weights of the first instance of the global ML model and the updated second weights of the second instance of the global ML model; and updating the weights of the global ML model using the average of the updated first weights and the updated second weights.

[0095] In some implementations, the threshold number of additional gradients can be based on the number of gradients received from multiple corresponding client devices.

[0096] In some implementations, selecting the set of client gradients can include selecting one or more client gradients of a specific type to include in the set of client gradients. In some versions of these implementations, selecting the set of additional longitude gradients can include selecting one or more longitude gradients of a specific type to include in the set of additional longitude gradients.

[0097] In some implementations, selecting the set of client gradients can include selecting one or more client gradients to be included in the set of client gradients based on one or more of the client gradients being of a specific type. In some versions of those implementations, selecting the set of additional longitude gradients can include selecting one or more longitude gradients to be included in the set of additional longitude gradients based on one or more of the longitude gradients being of a specific gradient type.

[0098] In some implementations, the method may also include analyzing the distribution of remote gradients stored in one or more databases accessible to the remote system to identify a specific type of gradient. Multiple generated remote gradients may conform to the specific type of gradient. In some versions of those implementations, identifying the specific type of gradient may include determining a threshold number of remote gradients whose distribution does not include the specific type of gradient, based on the analysis of the remote gradient distribution.

[0099] In some implementations, the method may further include transmitting the updated global ML model or the updated global weights of the global ML model to the client device. Transmitting the global ML model or the updated global weights can occur after the remote system updates the global weights of the global ML model based on the set of client gradients and an additional set of remote gradients. Furthermore, transmitting the updated global ML model or the updated global weights to the client device enables the client device to replace the on-device ML model in its local storage with the updated global ML model or replace the local weights of the on-device ML model with the updated global weights of the global ML model.

[0100] In some implementations, the ground-based output corresponding to the supplementary audio data can be based on supplementary spoken utterances captured in the supplementary audio data. In some versions of those implementations, the supplementary spoken utterances captured in the supplementary audio data can include specific words or phrases, and the ground-based output corresponding to the supplementary audio data can include an indication of whether the supplementary spoken utterances captured specific words or phrases. In additional or alternative versions of those implementations, the supplementary spoken utterances captured in the supplementary audio data can include one or more terms, and the ground-based output corresponding to the supplementary audio data can include transcriptions of one or more terms.

[0101] In some implementations, a method executed by one or more processors of a client device is provided and includes the following steps performed by one or more client processors of the client device: detecting audio data via one or more microphones of the client device, the audio data capturing at least a portion of spoken utterance of a user of the client device; processing the audio data using an on-device machine learning (ML) model locally stored on the client device to generate a predictive output; generating client gradients based on the predictive output; and transmitting the client gradients from the client device to a remote system. The method further includes, via one or more remote processors of the remote system: obtaining additional audio data capturing at least a portion of additional spoken utterance of an additional user; processing the additional audio data using a global ML model remotely stored on the remote system to generate additional predictive output; generating remote gradients based on the additional predictive output; and updating the weights of the global ML model using the client and remote gradients.

[0102] These and other implementations of this technology can include one or more of the following features.

[0103] In some implementations, updating the weights of the global ML model using client and remote gradients can include updating the weights of the global ML model using client gradients and additional client gradients received from the corresponding additional client device; and after updating the weights of the global ML model using client gradients and additional client gradients: further updating the weights of the global ML model using remote gradients and additional remote gradients generated in the remote system.

[0104] In some implementations, updating the weights of the global ML model using client and remote gradients can include updating the weights of the global ML model using remote gradients and additional remote gradients generated at the remote system; and after updating the weights of the global ML model using remote gradients and additional remote gradients, further updating the weights of the global ML model using client gradients and additional client gradients received from the corresponding additional client device.

[0105] In some implementations, updating the weights of the global ML model using client and remote gradients can include updating the first weights of a first instance of the global ML model using client gradients and additional client gradients received from the corresponding additional client device; updating the second weights of a second instance of the global ML model in parallel using remote gradients and additional remote gradients generated at the remote system; and updating the weights of the global ML model using the updated first weights of the first instance of the global ML model and the updated second weights of the second instance of the global ML model.

[0106] In some implementations, the remote gradient generated by one or more processors of the remote system based on processing additional audio data can be the same type of gradient as the client gradient generated by one or more processors of the client device based on processing audio data. In some versions of these implementations, the gradient type can include one of the following: hot word gradient, automatic speech recognition (ASR) gradient, voice activity detection (VAD) gradient, continuation dialogue gradient, or voice recognition gradient.

[0107] In some implementations, the remote gradient generated by one or more processors of the remote system based on processing additional audio data can be a different type of gradient corresponding to the client gradient generated by one or more processors of the client device based on processing audio data.

[0108] In some implementations, generating client gradients based on the predicted output can include comparing the predicted output with ground real-time output corresponding to the audio data, and generating remote gradients based on the additional predicted output can include comparing the additional predicted output with additional ground real-time output corresponding to the additional audio data.

[0109] In some implementations, client gradients can be transmitted over the network, and gradients can be transmitted without transmitting any of the following: audio data and prediction output.

[0110] In some implementations, the method further includes one or more remote processors of the remote system transmitting the updated global ML model or the updated global weights of the global ML model to the client device. The transmission of the global ML model or the updated global weights can occur after the remote system updates the global weights of the global ML model based on the client gradient and the remote gradient. In some versions of those implementations, the method may further include one or more processors of the client device receiving the updated global ML model or the updated global weights of the global ML model from the remote system; and causing the client device to replace the on-device ML model in its local storage with the updated global ML model or replace the local weights of the on-device ML model with the updated global weights of the global ML model.

[0111] In some implementations, a method executed by one or more processors of a client device is provided and includes one or more client processors at the client device: identifying a text segment locally stored on the user's client device; processing the text segment using an on-device text-to-speech (TTS) model locally stored on the client device to generate synthetic speech audio data including synthetic speech corresponding to the text segment; processing the synthetic speech audio data using a local machine learning (ML) model locally stored on the client device to generate a prediction output; generating a gradient based on comparing the prediction output with a ground truth output corresponding to the text segment; and transmitting the generated gradient from the client device to a remote system. The method further includes, by one or more remote processors of a remote system: obtaining additional text segments accessible to the remote system; processing the additional text segments using a global TTS model remotely stored on the remote system to generate additional synthesized speech audio data, the additional synthesized speech audio data including additional synthesized speech corresponding to the additional text segments; processing the additional synthesized speech audio data using a global ML model remotely stored on the remote system to generate additional prediction output; generating additional gradients based on comparing the additional prediction output with additional ground truth output corresponding to the additional text segments; and updating the weights of the global TTS model using at least the gradients received from the client device and the additional gradients generated on the remote system.

[0112] In some implementations, a method is provided executed by one or more processors of a client device, the method comprising receiving a plurality of client gradients from a plurality of corresponding client devices. Each of the plurality of client gradients is locally generated based on processing corresponding image data in a given location on the plurality of corresponding client devices, the corresponding image data capturing at least a portion of a corresponding environment of a corresponding user in a given location on the plurality of corresponding client devices. The method further comprises generating a plurality of remote gradients. Generating each of the plurality of remote gradients comprises obtaining additional image data capturing at least a portion of an additional environment of an additional user; processing the additional image data using a global machine learning (ML) model remotely stored on a remote system to generate a prediction output; and generating an additional gradient to be included in the plurality of remote gradients based on comparing the additional prediction output with a ground truth output corresponding to the additional image data. The method further comprises selecting a set of client gradients from the plurality of client gradients; selecting a set of additional remote gradients from the plurality of remote gradients; and updating the weights of a global ML model using the set of client gradients and the set of additional remote gradients.

[0113] Various implementations can include non-transitory computer-readable storage media whose storage can be executed by one or more processors (e.g., multiple central processing units (CPUs), multiple graphics processing units (GPUs), multiple digital signal processors (DSPs), and / or multiple tensor processing units (TPUs)) to perform methods such as one or more methods described herein. Other implementations can include an automated assistant client device (e.g., a client device including at least an automated assistant interface for connecting to a cloud-based automated assistant component interface) comprising a processor operable to execute stored instructions for performing methods such as one or more methods described herein. Yet another implementation can include a system of one or more servers comprising one or more processors operable to execute stored instructions for performing methods such as one or more methods described herein.

Claims

1. A method for federated learning of machine learning models, implemented by one or more processors, the method comprising: Multiple client gradients are received from multiple corresponding client devices, wherein each of the multiple client gradients is generated locally on a given corresponding client device based on processing corresponding audio data, the corresponding audio data capturing at least a portion of the corresponding spoken words of a corresponding user on the given corresponding client device; Generating a plurality of long-range gradients, wherein generating each of the plurality of long-range gradients includes: Obtain additional audio data, which captures at least a portion of additional spoken utterances from an additional user; The additional audio data is processed using a global ML model stored remotely on a remote system to generate additional predictive output; and Additional gradients are generated to be included in the plurality of remote gradients by comparing the additional predicted output with the ground real-time output corresponding to the additional audio data; Selecting a set of client gradients from the plurality of client gradients, wherein selecting the set of client gradients includes: Based on the fact that one or more of the plurality of client gradients are gradients of a specific type, one or more of the plurality of client gradients are selected to be included in the set of client gradients; Select a set of additional long-range gradients from the plurality of long-range gradients; The weights of the global ML model are updated using the set of client gradients and the set of additional remote gradients, thereby providing an updated global ML model; Determine whether one or more conditions for transmitting the updated global ML model and / or the updated weights of the global ML model are met; In response to the satisfaction of one or more of the conditions, the updated global ML model and / or the updated weights of the global ML model are transmitted to one or more of the plurality of client devices; One or more client processors from the client device: Receive the updated global ML model and / or the updated weights of the global ML model; At least one prediction is made based on additional client data identified at the client device using the updated global ML model and / or the updated weights of the global ML model.

2. The method according to claim 1, wherein, Updating the weights of the global ML model using the set of client gradients and the set of additional long-range gradients includes: The weights of the global ML model are updated using the set of client gradients; and After updating the weights of the global ML model using the set of client gradients: The weights of the global ML model are further updated using the set of additional long-range gradients.

3. The method according to claim 1, wherein, Updating the weights of the global ML model using the set of client gradients and the set of additional long-range gradients includes: The weights of the global ML model are updated using the set of additional long-range gradients; and After updating the weights of the global ML model using the set of additional long-range gradients: The weights of the global ML model are further updated using the set of client gradients.

4. The method according to claim 1, wherein, Updating the weights of the global ML model using the set of client gradients and the set of additional long-range gradients includes: The first weight of the first instance of the global ML model is updated using the set of client gradients; The second weights of the second instance of the global ML model are updated in parallel using the set of additional long-range gradients; and The weights of the global ML model are updated using the updated first weights of the first instance of the global ML model and the updated second weights of the second instance of the global ML model.

5. The method according to claim 4, wherein, Updating the weights of the global ML model using the updated first weights of the first instance of the global ML model and the updated second weights of the second instance of the global ML model includes: Determine the average of the updated first weights of the first instance of the global ML model and the updated second weights of the second instance of the global ML model; and The weights of the global ML model are updated by averaging the updated first weights and the updated second weights.

6. The method according to claim 1, wherein, The threshold number of additional gradients is based on the number of gradients received from the plurality of corresponding client devices.

7. The method according to claim 1, wherein, The set of additional long-range gradients includes: Based on the fact that one or more of the plurality of remote gradients are gradients of the specific type, one or more of the plurality of remote gradients are selected to be included in the set of additional remote gradients.

8. The method of claim 1, further comprising: Analyze the distribution of remote gradients stored in one or more databases accessible to the remote system to identify specific types of gradients, and The generated remote gradients conform to the specific type of gradient.

9. The method according to claim 8, wherein, Identifying the specific type of gradient includes: Based on the analysis of the distribution of the remote gradients, a threshold number of remote gradients that do not include the specific type of gradient is determined.

10. The method according to claim 1, wherein: The transmission of the global ML model and / or the updated global weights occurs after the remote system updates the global weights of the global ML model based on the set of client gradients and the set of additional remote gradients; as well as Transmit the updated global ML model and / or updated global weights to the client device so that the client device replaces the on-device ML model in its local storage with the updated global ML model and / or replaces the local weights of the on-device ML model with the updated global weights of the global ML model.

11. The method according to claim 1, wherein, The ground-based live output corresponding to the additional audio data is based on additional spoken words captured in the additional audio data.

12. The method according to claim 11, wherein, The additional spoken words captured in the additional audio data include specific words or phrases, and the ground live output corresponding to the additional audio data includes an indication of whether the additional spoken words or phrases are captured.

13. The method according to claim 11, wherein, The additional spoken utterances captured in the additional audio data include one or more terms, and wherein the ground-based live output corresponding to the additional audio data includes a transcription of the one or more terms.

14. A method for federated learning for machine learning models, comprising: One or more client processors from the client device: Audio data is detected via one or more microphones of the client device, the audio data capturing at least a portion of the spoken words of the user of the client device; The audio data is processed using an on-device machine learning (ML) model stored locally on the client device to generate a predictive output; Generate client gradients based on the predicted output; as well as Transmit the client gradient from the client device to the remote system; Receive the updated global ML model and / or the updated weights of the global ML model from the remote system; At least one prediction is made based on additional client data identified at the client device using the updated global ML model and / or the updated weights of the global ML model. One or more remote processors of the remote system: Obtain additional audio data, which captures at least a portion of additional spoken utterances from an additional user; The additional audio data is processed using a global ML model remotely stored on the remote system to generate additional predictive output; Analyze the distribution of remote gradients stored in one or more databases accessible by the remote system to identify specific types of gradients; Generate long-range gradients based on the additional prediction output; Wherein, the generated remote gradient conforms to the specific type of gradient; and The weights of the global ML model are updated using the client gradient and the remote gradient, thereby providing an updated global ML model; Determine whether one or more conditions for transmitting the updated global ML model and / or the updated weights of the global ML model are met; In response to the satisfaction of one or more of the conditions, an updated global ML model and / or the updated weights of the global ML model are transmitted to one or more of the plurality of client devices.

15. The method according to claim 14, wherein, Updating the weights of the global ML model using the client gradient and the remote gradient includes: The weights of the global ML model are updated using the client gradients and the additional client gradients received from the corresponding additional client devices; and After updating the weights of the global ML model using the client gradient and the additional client gradient: The weights of the global ML model are further updated using the remote gradients and additional remote gradients generated at the remote system.

16. The method of claim 14, wherein, Updating the weights of the global ML model using the client gradient and the remote gradient includes: The weights of the global ML model are updated using the long-range gradients and additional long-range gradients generated at the long-range system; and After updating the weights of the global ML model using the longitude gradient and the additional longitude gradient: The weights of the global ML model are further updated using the client gradients and the additional client gradients received from the corresponding additional client devices.

17. The method of claim 14, wherein, Updating the weights of the global ML model using the client gradient and the remote gradient includes: The first weight of the first instance of the global ML model is updated using the client gradient and the additional client gradient received from the corresponding additional client device; The second weights of the second instance of the global ML model are updated in parallel using the long-range gradients and additional long-range gradients generated at the long-range system; and The weights of the global ML model are updated using the updated first weights of the first instance of the global ML model and the updated second weights of the second instance of the global ML model.

18. The method according to claim 14, wherein, The remote gradient generated based on the processing of the additional audio data by one or more processors of the remote system has the same type of gradient as the client gradient generated based on the processing of the audio data by one or more processors of the client device.

19. The method according to claim 18, wherein, The gradient type includes one of the following: hot word gradient, automatic speech recognition (ASR) gradient, voice activity detection (VAD) gradient, continuation dialogue gradient, or voice recognition gradient.

20. The method of claim 14, wherein, The remote gradient generated based on the processing of the additional audio data by one or more processors in the remote system has a different type of gradient than the client gradient generated based on the processing of the audio data by one or more processors in the client device.

21. The method according to claim 14, wherein, Generating the client gradient based on the predicted output includes comparing the predicted output with a ground-based live output corresponding to the audio data, and wherein generating the remote gradient based on the additional predicted output includes comparing the additional predicted output with an additional ground-based live output corresponding to the additional audio data.

22. The method according to claim 14, wherein, The client gradient is transmitted over the network, wherein transmitting the gradient includes transmitting the gradient without transmitting either the audio data or the prediction output.

23. The method according to claim 14, wherein: The updated global ML model or updated global weights are transmitted after the remote system updates the global weights of the global ML model based on the client gradient and the remote gradient.

24. The method of claim 23, further comprising: One or more processors from the processors of the client device: The client device replaces the on-device ML model in its local storage with the updated global ML model and / or replaces the local weights of the on-device ML model with the updated global weights of the global ML model.

25. A method for federated learning of machine learning models, comprising: One or more client processors from the client device: Identify text fragments stored locally on the user's client device; The text segment is processed using a text-to-speech (TTS) model stored locally on the client device to generate synthetic speech audio data including synthesized speech corresponding to the text segment; The synthesized speech audio data is processed using a local machine learning (ML) model stored locally on the client device to generate a predictive output; Gradients are generated by comparing the predicted output with the ground-based output corresponding to the text segment; as well as The generated gradient is transmitted from the client device to the remote system; Receive the updated global ML model and / or the updated weights of the global ML model from the remote system; At least one prediction is made based on additional client data identified at the client device using the updated global ML model and / or updated weights. One or more remote processors of the remote system: Obtain additional text fragments that are accessible by the remote system; The additional text segment is processed using a global TTS model remotely stored at the remote system to generate additional synthesized speech audio data, the additional synthesized speech audio data including additional synthesized speech corresponding to the additional text segment; The additional synthesized speech audio data is processed using a global ML model remotely stored at the remote system to generate additional predictive output; Analyze the distribution of remote gradients stored in one or more databases accessible by the remote system to identify specific types of gradients; Additional gradients are generated by comparing the additional predicted output with additional ground reality output corresponding to the additional text fragment; Wherein, the generated remote gradient conforms to the specific type of gradient; and The weights of the global TTS model are updated using at least the gradients received from the client device and additional gradients generated at the remote system, thereby providing an updated global ML model; Determine whether one or more conditions for transmitting the updated global ML model and / or the updated weights of the global ML model are met; In response to the satisfaction of one or more of the conditions, an updated global ML model and / or the updated weights of the global ML model are transmitted to one or more of the plurality of client devices.

26. A method for federated learning of machine learning models, implemented by one or more processors, the method comprising: Multiple client gradients are received from multiple corresponding client devices, wherein each of the multiple client gradients is generated locally on a given corresponding client device based on processing corresponding image data, the corresponding image data capturing at least a portion of the corresponding environment of the corresponding user of the given corresponding client device; Generating a plurality of long-range gradients, wherein generating each of the plurality of long-range gradients includes: Obtain additional image data, which captures at least a portion of the additional environment of the additional user; The additional image data is processed using a globally stored ML model on a remote system to generate additional prediction output; and Additional gradients are generated to be included in the plurality of long-range gradients by comparing the additional prediction output with the ground reality output corresponding to the additional image data; Select a set of client gradients from the plurality of client gradients; The set of client gradients selected includes: Based on the fact that one or more of the plurality of client gradients are gradients of a specific type, one or more of the plurality of client gradients are selected to be included in the set of client gradients; Select a set of additional long-range gradients from the plurality of long-range gradients; and The weights of the global ML model are updated using the set of client gradients and the set of additional remote gradients, thereby providing an updated global ML model; Determine whether one or more conditions for transmitting the updated global ML model and / or the updated weights of the global ML model are met; In response to the satisfaction of one or more of the conditions, the updated global ML model and / or the updated weights of the global ML model are transmitted to one or more of the plurality of client devices; One or more client processors from the client device: Receive the updated global ML model and / or the updated weights of the global ML model; At least one prediction is made based on additional client data identified at the client device using the updated global ML model and / or the updated weights of the global ML model.

27. A system for federated learning of machine learning models, comprising: At least one processor; as well as A memory for storing instructions, which, when executed, cause the at least one processor to perform the method corresponding to any one of claims 1 to 26.

28. A non-transitory computer-readable storage medium storing instructions, which, when executed, cause at least one processor to perform the method corresponding to any one of claims 1 to 26.

Images