On-device personalization of speech synthesis for training of speech models

Abstract

The processor of the client device can: identify a text segment stored locally at the client device; process the text segment using an on-device TTS generator model to generate synthesized speech audio data comprising synthesized speech of the text segment; process the synthesized speech using an on-device ASR model to generate predicted ASR output; and generate a gradient based on comparing the predicted ASR output to ground truth output corresponding to the text segment. The processor of the client device can also: process the synthesized speech audio data using the on-device TTS generator model to make a prediction; and generate a gradient based on the prediction. In these implementations, the generated gradient can be used to update weights of the respective on-device model and / or transmitted to a remote system for use in remote updating of a respective global model. The updated weights and / or updated model can be transmitted to the client device.

Description

Background Technology

[0001] Voice-based user interfaces (APIs) are increasingly used for controlling computers and other electronic devices. They have evolved from early, rudimentary interfaces that could only understand simple, direct commands to more complex interfaces that respond to natural language requests, understand context, and manage back-and-forth dialogues or sessions with the user. Many APIs perform speech recognition on spoken utterances (e.g., using speech-to-text (STT) models) to generate corresponding text, perform semantic analysis of the corresponding text to attempt to determine the meaning of the spoken utterances, take one or more actions based on the determined meaning, track the flow of each session, and annotate spoken utterances and / or corresponding text using user instructions that provided the spoken utterances. Some APIs are also capable of synthesizing speech from text (e.g., using text-to-speech models) to generate corresponding synthesized speech audio data and rendering the synthesized speech audio data audibly at the client device.

[0002] While speech recognition performance has continuously improved, inaccurate speech recognition can still occur in many situations. As a non-limiting example, inaccurate speech recognition can occur with new terms and / or terms that are relatively uncommon (or nonexistent) in the training corpus used to train the speech recognition model. In attempting to effectively recognize new and / or uncommon terms, several techniques have been proposed to generate additional speech recognition hypotheses besides one or more initial hypotheses and to treat these additional hypotheses as candidates for speech recognition. However, such techniques require additional post-processing and still fail to lead to efficient recognition of many terms in many cases, such as when one or more initial hypotheses are too far-fetched and / or when the dictionary used for the additional hypotheses does not include certain terms.

[0003] Furthermore, inaccurate speech recognition can be exacerbated when performed on a device (i.e., on a client device). This can be due to, for example, that on-device speech recognition models are less robust than cloud-based models, that on-device memory and / or processor resources are more limited than cloud-based resources, and / or that additional assumptions about generating dictionaries or language models are more constrained on devices. Summary of the Invention

[0004] Some implementations disclosed herein relate to improving the performance of speech recognition performed locally on the client device by leveraging an on-device automatic speech recognition (ASR) model stored locally on the client device. In some of those implementations, the client device processor: trains an on-device text-to-speech (TTS) generator model based on multiple training instances. This TTS generator model is part of a generative adversarial network (GAN) model stored locally on the client device (e.g., in the client device's RAM and / or ROM). As described in detail herein, the on-device TTS generator model can be trained to include the voice characteristics of the user of the client device. Furthermore, after training the on-device TTS generator model, the client device processor: identifies a text segment stored locally on the client device; processes the text segment using the trained on-device TTS generator model to generate synthetic speech audio data including synthesized speech corresponding to the text segment; processes the synthetic speech audio data using the on-device ASR model to generate a predicted ASR output; and generates gradients based on comparing the predicted ASR output with ground truth outputs corresponding to the text segment. By leveraging a trained on-device TTS generator model to generate synthesized speech audio data used in gradient generation, the on-device ASR model can be updated using gradients generated from synthesized speech audio data based on the voice characteristics of users, including those on client devices, and based on text segments that might be encountered on the client device. This improves the performance of the on-device ASR model. For example, by updating the ASR model in these and other ways, the on-device ASR model can be personalized to recognize text segments that might be encountered on client devices but otherwise unlikely to be encountered on other users' corresponding client devices, making them more difficult to recognize without this personalization of the on-device ASR model.

[0005] For example, the on-device ASR model can be an end-to-end speech recognition model used to generate predicted ASR outputs of predicted text segments, and gradient generation can be based on comparing the predicted text segments with the base fact text segments corresponding to the base fact outputs. Alternatively, the on-device ASR model can be used to generate predicted ASR outputs of predicted phoneme sequences, and gradient generation can be based on comparing the predicted phoneme sequences with the base fact phoneme sequences corresponding to the base fact outputs.

[0006] In some implementations, one or more processors on the client device use generated gradients to update one or more weights of the on-device ASR model. For example, backpropagation and / or other (multiple) techniques can be used to update the local weights of the on-device ASR model based on gradients. This can improve speech recognition performance at the client device using the on-device ASR model for spoken utterances including text segments. Furthermore, this allows the on-device ASR model to be trained on specific text segments without requiring any actual human utterance of those specific text segments (since the TTS generator model is used to generate synthesized speech for those specific text segments). Therefore, it is possible to correctly recognize specific text segments using the on-device ASR model, even though those text segments may not have been included in any previously detected spoken utterances at the client device. Moreover, many implementations repeat this process for a large number of text segments stored locally at the client device, thereby improving on-device speech recognition performance for spoken utterances including any of the text segments. This effectively tailors the on-device ASR model to actual text segments (transient or non-transient) stored on the device and potentially included in spoken utterances involving the client device.

[0007] In some implementations, the on-device ASR model updated based on the generated gradients can be an addition to the on-device ASR model currently being used by the client device to perform speech recognition of spoken utterances detected at the client device. In these implementations, the updated on-device ASR model can be deployed in response to one or more conditions being met to effectively replace the currently used on-device ASR model (and thus become the currently used on-device ASR model). For example, the conditions can include: on-device verification of the updated on-device ASR model; determination on-device that the updated on-device ASR model performs better than the currently used on-device ASR model (e.g., in terms of accuracy and / or recall); and / or the occurrence of at least a threshold amount of training and / or training duration for the updated on-device ASR model. For example, determining that the updated on-device ASR model performs better than the currently used on-device ASR model can be performed based on comparisons using training instances generated according to the techniques described herein but not yet used in training (i.e., alternatively reserved for testing). As another example, determining that the updated on-device ASR model performs better than the currently used on-device ASR model can be based on downloading test utterances on the client device (e.g., from a remote system) and processing the test utterances using the updated on-device ASR model to verify that the updated on-device ASR model is not diverging. Deploying the currently used on-device ASR model to effectively replace the updated on-device ASR model can include updating the weights of the currently used on-device ASR model using the weights of the updated on-device ASR model, or replacing the currently used on-device ASR model with the updated on-device ASR model. Once the updated on-device ASR model has effectively replaced the currently used on-device ASR model and is used for speech recognition, a copy of the updated on-device ASR model can then be used as the new on-device ASR model to be updated.

[0008] In some implementations, the generated gradients are additionally or alternatively transmitted by the client device to the remote system over the network. In these implementations, the remote system uses the generated gradients and additional gradients from other client devices and / or from the remote system to update the global weights of the global ASR model. Additional gradients from other client devices can be similarly generated locally at the corresponding other client devices based on the corresponding locally stored text segments and their locally generated synthesized speech. In various implementations, the client device transmits the generated gradients without transmitting any of the following: text segments, synthesized speech audio data, predicted ASR output, and ground fact output corresponding to the text segments. The remote system can use the generated gradients to update the global ASR model without referencing or using the text segments, synthesized speech audio data, predicted ASR output, and ground fact output corresponding to the text segments. Transmitting only the gradients can, in some instances, utilize fewer network resources than transmitting a combination of larger data sizes, such as text segments, synthesized speech audio data, predicted ASR output, and ground fact output corresponding to the text segments. Furthermore, the transmission of gradients protects the privacy and security of the text segments on the device because the text segments and synthesized speech audio data cannot be derived from the gradients. In some implementations, one or more differential privacy techniques (e.g., adding Gaussian noise) can be used to further ensure that text fragments and / or synthesized speech audio data cannot be derived from gradients.

[0009] In implementations where the remote system updates the global weights of the global ASR model, the remote system can then provide the updated global weights to the client device, enabling the client device to replace the local weights of its on-device ASR model with the updated global weights. In some implementations, the remote system can additionally or alternatively provide the updated global ASR model to the client device, enabling the client device to replace its on-device ASR model with the updated global ASR model. Therefore, by utilizing the updated global weights or the updated global ASR model, on-device speech recognition performance is improved.

[0010] Some additional or alternative implementations disclosed herein relate to improving the performance of speech synthesis performed locally on a client device using an on-device text-to-speech (TTS) generator model stored locally on the client device. In those implementations, the processor of the client device: identifies a text segment stored locally on the client device; processes the text segment using a trained on-device TTS generator model stored locally on the client device to generate synthetic speech audio data including synthesized speech corresponding to the text segment; processes the synthetic speech audio data using a trained on-device TTS discriminator model stored locally on the client device to generate a discriminator output; and generates a gradient based on the discriminator output (e.g., by comparing it with a ground truth output). The discriminator output indicates a prediction regarding whether the synthesized speech corresponds to (i) the synthetic speech audio data generated by the on-device TTS generator model, or alternatively, to (ii) the actual spoken utterance of a user on the client device. For example, the on-device TTS generator model and the on-device TTS discriminator model can be part of a GAN model. The on-device TTS generator model attempts to make the on-device TTS discriminator model predict that the synthesized speech audio data is indeed the spoken utterance of a user on the client device. As a result, it is possible to compare predictions made by the on-device TTS discriminator model with a fact label that indicates the synthesized speech audio data was actually generated by the on-device TTS generator model.

[0011] In some implementations, the generated gradients are used by one or more processors on the client device to update one or more weights of the on-device TTS generator model based on the generated gradients. For example, backpropagation and / or other techniques can be used to update the local weights of the on-device TTS generator model based on gradients. This enables the use of the on-device TTS generator model at the client device to improve speech synthesis performance for spoken utterances including text fragments. Furthermore, this makes it possible to train the on-device TTS generator model based on specific text fragments without requiring any actual human utterance of those specific text fragments. Thus, it is possible to correctly synthesize specific text fragments using the on-device TTS generator model, even though those text fragments may not potentially be included in any previously generated synthesized speech at the client device. Moreover, many implementations repeat this process on a large number of text fragments stored locally at the client device, thereby improving on-device speech synthesis performance for spoken utterances including any text fragments. This effectively customizes the on-device TTS generator model to actual text fragments, which are stored (transiently or non-transiently) on the device and may be included in synthesized speech generated at the client device.

[0012] In some implementations, the on-device TTS generator model updated based on the generated gradients can be an addition to the on-device TTS generator model currently used by the client device when performing speech synthesis at the client device. In those implementations, the updated on-device TTS generator model can be deployed in response to satisfying one or more conditions to effectively replace the currently used on-device TTS generator model (and thus become the currently used on-device TTS generator model). These conditions can be the same as or similar to those described above regarding updating the on-device ASR model. Deploying the currently used on-device TTS generator model to effectively replace the updated on-device TTS generator model can include updating the weights of the currently used on-device TTS generator model using the weights of the updated on-device TTS generator model, or replacing the currently used on-device TTS generator model with the updated on-device TTS generator model. Once the updated on-device TTS generator model has effectively replaced the currently used on-device TTS generator model and is used for speech synthesis, a copy of the updated on-device TTS generator model can be used as the new on-device TTS generator model to be updated.

[0013] In some implementations, the generated gradients are additionally or alternatively transmitted by the client device to the remote system over the network. In those implementations, the remote system uses the generated gradients and additional gradients from the other client device to update the global weights of the global TTS generator model. Similarly, additional gradients from the other client device can be generated locally at the corresponding other client device based on the corresponding locally stored text snippets and their locally generated synthesized speech. In various implementations, the client device transmits the generated gradients without transmitting any of the following: text snippets, synthesized speech audio data, and predictions made at the client device. The remote system can utilize the generated gradients when updating the global TTS generator model without any reference to or use of text snippets, synthesized speech audio data, and predictions made at the client device. Transmitting only gradients utilizes fewer network resources than transmitting a larger combination of text snippets, synthesized speech audio data, and predictions made at the client device. Furthermore, transmitting gradients protects the privacy and security of text snippets on the device because text snippets and synthesized speech audio data cannot be derived from gradients. In some implementations, one or more differential privacy techniques (e.g., adding Gaussian noise) can be used to further ensure that text fragments and / or synthesized speech audio data cannot be derived from gradients.

[0014] In implementations where the remote system updates the global weights of the global TTS generator model, the remote system can then provide the updated global weights to the client device, allowing the client device to replace the local weights of its on-device TTS generator model with the updated global weights. In some implementations, the remote system can additionally or alternatively provide the updated global TTS generator model to the client device, allowing the client device to replace its on-device TTS generator model with the updated global weights. Therefore, on-device speech synthesis performance is improved by utilizing either the updated global weights or the updated global TTS generator model.

[0015] Client devices can utilize various techniques to identify locally stored text segments for use in generating gradients based on the text segments and their corresponding locally generated synthetic speech. For example, text segments can be identified based on whether they are included in a list of contacts, media playlists, aliases of linked smart devices (e.g., smart lights, smart plugs, and / or aliases of other smart devices linked to the client device's account), based on typed input received at the client device, and / or based on spoken utterance received at the client device. As another example, text segments can be identified additionally or alternatively based on determining that the text segment is outside the vocabulary (i.e., a text segment for which the on-device ASR model was not previously trained). As yet another example, text segments can be identified based on determining that previous human utterance detected via the client device's microphone includes the text segment and determining that previous speech recognition of the previous human utterance performed using the on-device ASR model failed to correctly identify the text segment. Determining that previous speech recognition failed to correctly identify the text segment can be based on canceling received user input that was incorrectly predicted based on previous speech recognition. Furthermore, determining prior human utterance includes text fragments that can be based on received user input and / or additional received user input after canceling incorrect predictions based on prior speech recognition.

[0016] To conserve the limited battery, processor, memory, and / or other resources of the client device, in various implementations, the client device performs one or more steps disclosed herein only in response to determining that the current state of the client device satisfies one or more conditions. For example, it is possible to perform actions such as generating synthesized speech audio data, and / or processing the synthesized speech audio data to generate predicted ASR output, and / or generating gradients, and / or updating one or more weights, in response to determining that the current state of the client device satisfies one or more conditions. It is possible to determine whether one or more conditions are met based on sensor data from one or more sensors on the client device. One or more conditions may include, for example: the client device is charging, the client device has at least a threshold charging state, the temperature of the client device (based on temperature sensors on one or more devices) is less than a threshold, the client device is not being held by a user, and / or the client device is connected to an unmetered network (e.g., WiFi).

[0017] In various implementations, the on-device TTS generator model and the on-device TTS discriminator model portion of the GAN model can be trained before gradient generation. In some versions of these implementations, the on-device TTS discriminator model can be trained based on multiple discriminator training instances. Each discriminator training instance can include training instance input and training instance output. The training instance input can include audio data, which includes spoken words from a user on a client device or synthetic speech audio data generated using the on-device TTS generator model (or another TTS generator model), and the training instance output can include a fact label indicating whether the corresponding training instance input corresponds to audio data including spoken words from a user on a client device (e.g., a semantic label for "real" or "human," or its corresponding fact probability, such as "0" or "1"), or synthetic speech audio data generated using the on-device TTS generator model (or another TTS generator model) (e.g., a semantic label for "fake" or "synthetic," or its corresponding fact probability, such as "0" or "1"). Training instances including audio data containing spoken utterances from users of client devices can be considered positive discriminator training instances, and training instances including synthetic speech audio data generated using an on-device TTS generator model (or another TTS generator model) can be considered negative discriminator training instances. Furthermore, when processing training instance inputs for a given discriminator training instance, the on-device TTS discriminator model predicts whether the training instance input corresponds to human speech or synthetic speech. Predictions can include, for example, semantic labels (e.g., “real,” “human,” “fake,” “synthetic,” etc.), binary values ​​(e.g., “0” for “synthetic” or “fake,” and “1” for “real” or “human”), and / or probabilities (e.g., “0.6” for “synthetic” or “fake” and “0.4” for “real” or “human”). Additionally, predictions can be compared to base and base fact outputs to generate a loss, and the on-device TTS discriminator model can be updated based on this loss (e.g., backpropagation across on-device TTS discriminator models to update their weights). This can be repeated for multiple additional discriminator training instances to train the on-device TTS discriminator model.

[0018] In some versions of those implementations, an on-device TTS generator model can be trained based on multiple generator training instances. Each generator training instance can include a training instance input and a training instance output. The training instance input can include a text fragment (e.g., identified in any manner described herein), and the training instance output can include a base fact label (e.g., a semantic label for "fake" or "synthetic," or a corresponding base fact probability, such as "0" or "1") indicating that any resulting audio data processed by the on-device TTS discriminator model corresponds to synthetic speech audio data generated using the on-device TTS generator model. Furthermore, when processing the training instance input of a given generator training instance, the on-device TTS generator model generates synthetic speech audio data including synthetic speech, and when processing the synthetic speech audio data, the on-device TTS discriminator model predicts whether the training instance input corresponds to human speech or synthetic speech. The prediction can include, for example, semantic labels (e.g., "real," "human," "fake," "synthetic," etc.), binary values ​​(e.g., "0" for "synthetic" or "fake," and "1" for "real" or "human"), and / or probabilities (e.g., "0.6"). Furthermore, it is possible to compare predictions with the output of ground truth to generate a loss, and to update the on-device TTS generator model based on this loss (e.g., the loss can be backpropagated across on-device TTS generator models to update their weights). This can be repeated for multiple additional generator training instances to train the on-device TTS generator model. The loss utilized when updating the on-device TTS generator model can be considered an adversarial loss. Notably, when generating synthetic speech audio data, the on-device TTS generator model attempts to deceive the on-device TTS discriminator model into predicting that the synthetic speech audio data corresponds to the human speech of the user on the client device. Therefore, by updating the on-device TTS generator model based on the adversarial loss, the on-device TTS generator model can learn voice characteristics that reflect the voice characteristics of the user on the computing device, since the on-device TTS discriminator model is trained to discriminate between the human speech of the user on the client device and the synthetic speech generated by the on-device TTS generator model (or another TTS generator model).

[0019] In some additional or alternative versions of those implementations, for one or more generator training instances, the training instance output can additionally or alternatively include base-fact audio data corresponding to the text segment input to the corresponding training instance. In some other versions of those implementations, acoustic features of the predicted synthesized speech audio data generated using the on-device TTS generator model can be compared with acoustic features of the base-fact audio data, and an additional loss can be generated based on the comparison of acoustic features. Acoustic features can include, for example, audio waveforms, mel frequency cepstral coefficients (MFCCs), mel filter bank features, values ​​associated with one or more prosodic attributes, neural representations of the audio data (e.g., corresponding embeddings of the base-fact audio data and the predicted synthesized speech audio data), and / or other acoustic features of the synthesized speech audio data and the base-fact audio data that can be compared. The on-device TTS generator model can additionally or alternatively be updated based on the additional loss (e.g., the additional loss can be backpropagated across the on-device TTS generator model to update its weights).

[0020] As described herein, after updating the on-device ASR model and / or the on-device TTS generator model according to the embodiments disclosed herein, it is possible to use the on-device ASR model to process audio data corresponding to spoken words from a user on the corresponding device to generate corresponding predicted ASR output, and / or to use the on-device TTS model to process text segments from a user's client device to generate corresponding synthesized speech audio data including synthesized speech. In embodiments utilizing the on-device ASR model, gradients on the device can be generated based on comparing instances of predicted ASR output with instances of basic fact output. Instances of basic fact text segments on the device can be determined based on one or more actions and / or inactions by the user in response to the content rendered at the device based on instances of predicted ASR output, and / or actions taken at the device based on instances of predicted ASR output. For example, if the user acknowledges a predicted text segment, the predicted text segment can be considered a basic fact text segment. For example, if the spoken word is "Call Francis" and the predicted text segment is "Call Francis," the user can acknowledge the predicted text segment by dialing the phone number of a contact named "Francis" without deleting the generated message. Additionally, for example, if the spoken phrase is "Call Francis" and the predicted text fragment is "Call Francis," then a selectable "Yes" option can be provided for the prompt "Do you want to call Francis?", and the user can confirm the recognized text by selecting the "Yes" option. As another example, if the user modifies the predicted text fragment (e.g., by adding and / or deleting characters; and / or by deleting it and replacing it with alternative text), the modified text can be treated as a basic factual text fragment. For example, if the spoken phrase "Hello" to be included in the text message is provided... If the incorrectly recognized text "Hello Francis, please call me as soon as possible" is incorporated into the text message, then the user can select "Francis" and use... Replace "Francis". Responding to the user selecting "Francis" and using... Replacing "Francis" allows you to utilize the modified text "Hello". Please call me as soon as possible" as a basic factual text fragment. Additionally, for example, if provided is a spoken phrase to be included in the text message, such as "Hello". The message "Please call me as soon as possible" will display the first optional graphic element "Francis" and the second optional graphic element to the user who provided the spoken words. In response to receiving a second optional graphical element from the user The choice of text "Hello" "Please call me as soon as possible" can be used as a basic factual text fragment.

[0021] In the implementation utilizing an on-device TTS generator model, gradients can be generated on the device based on processing predicted synthetic speech audio data using an on-device TTS discriminator model to predict whether the synthesized speech (including in the synthesized speech audio data) is the actual spoken words of a user on the client device or synthesized speech generated by the on-device TTS generator model. The predicted synthetic speech audio data can be generated based on text segments identified as described herein. The predictions made by processing the synthetic speech audio data using the on-device TTS discriminator model can be compared with the underlying fact labels indicating the synthetic speech audio data. For example, if the on-device TTS discriminator model incorrectly predicts that the synthesized speech corresponds to the actual spoken words of a user on the client device, a first gradient can be generated. However, if the on-device TTS discriminator model correctly predicts that the synthesized speech corresponds to synthesized speech generated by the on-device TTS generator model, a second gradient can be generated.

[0022] The gradients generated using these techniques can be transmitted from the client device and over the network to a remote system. In those implementations, the remote system utilizes the generated gradients, along with additional gradients generated from other client devices, in a similar manner (e.g., after local updates of on-device models such as on-device ASR models or on-device TTS generator models), to update the global weights of the global model (e.g., the global ASR model or the global TTS generator model). Note that updating the global weights based on such gradients can occur together with or separately from updating the global weights based on gradients based on locally stored text segments and their locally generated synthesized speech. Also note that transmitting such gradients can occur without transmitting any of the following: audio data, synthesized speech audio data, text segments, predicted ASR output, and / or predictions. The remote system can utilize the generated gradients to update the global model without referencing or using data corresponding to audio data, synthesized speech audio data, text segments, predicted ASR output, and / or predictions. Transmitting only gradients utilizes fewer network resources and protects the privacy and security of data stored locally and / or generated at the client device. In some implementations, one or more differential privacy techniques can be used to further ensure the privacy and security of data stored locally and / or generated on the client device.

[0023] In some implementations, after updating the on-device ASR model and / or the on-device TTS generator model according to the implementations disclosed herein, when the on-device ASR model is processing audio data corresponding to spoken utterances to generate corresponding predicted text segments, and / or when the on-device TTS generator model is processing text segments to generate corresponding predicted synthesized speech audio data, it is also possible to utilize on-device speech recognition and / or on-device speech synthesis bias based on text segments. For example, on-device speech recognition can be biased toward one or more text segments (such as contact aliases, road names, media names, and / or other text segments) stored on the device. The text segments to which on-device speech recognition is biased can be the same and / or can vary between iterations of speech recognition. For example, bias toward a first text segment can occur in a first device context (e.g., based on which applications are running or in the foreground, based on the time of day, based on the day of the week, etc.), and bias toward a different second text segment can occur in a second device context. Biasing on-device speech recognition toward text segments can include, for example, injecting text segments into speech recognition by increasing the probability of the text segments during decoding. After updating the on-device speech recognition model according to the embodiments disclosed herein, biasing the speech recognition on-device based on text segments can be more efficient than biasing without such updates. This is possible because, for example, the on-device speech recognition model has already been trained on a sequence of sounds from the text segments since it has been updated based on synthesized speech (which is based on text segments). As another example, on-device speech synthesis can be biased toward the voice characteristics of the user of the client device. The voice characteristics of the user of the client device can include, for example, prosodic attributes, including prosodic attributes that indicate the user's speech on the client device (e.g., one or more of intonation, pitch, stress, frequency, rhythm, or pauses). It is worth noting that although the synthesized speech includes the user's voice characteristics, the synthesized speech does not need to be an exact match of the user's speech on the client device.

[0024] In various implementations, the global model described herein can be specific to a geographic language and / or a particular language. For example, gradients generated at client devices primarily located in a specific geographic region can be used only to update the global model associated with that specific geographic region. For instance, gradients generated at client devices primarily located in Chicago, Illinois can be used only to update the global model associated with a Midwestern geographic region or a U.S. geographic region. As another example, gradients generated at client devices primarily using Spanish can be used only to update the global model associated with Spanish. In this way, the global model can be one of N available models for a given language and / or specific geographic region.

[0025] Some implementations of the client devices disclosed herein include automated assistant applications that generate on-device speech recognition (or utilize generated on-device speech recognition) and / or generate on-device synthesized speech (or utilize generated on-device synthesized speech). The automated assistant application can be installed "on top" of the client device's operating system and / or can form part (or all) of the client device's operating system. The automated assistant application includes and / or accesses on-device speech recognition and on-device speech synthesis, as well as optional on-device natural language understanding and / or on-device performance. On-device natural language understanding (NLU) can be executed using an on-device NLU module that processes recognized text generated using on-device speech recognition and optional contextual data to generate NLU data. The NLU data can include an intent corresponding to the spoken utterance and optional intent parameters (e.g., slot values). On-device performance can be performed using an on-device performance module that utilizes the NLU data (from the on-device NLU) and optional other local data to determine the action to be taken by parsing the intent of the spoken utterance (and optional intent parameters). This can include determining local and / or remote responses (e.g., answers) to spoken utterances, interactions with locally installed applications based on spoken utterances, transmitting commands to Internet of Things (IoT) devices (directly or via corresponding remote systems) based on spoken utterances, and / or other parsing actions based on spoken utterances. The device then performs local and / or remote execution / running of the determined actions that initiate the parsing of the spoken utterances.

[0026] In various implementations, remote speech processing, remote NLU, remote execution, and / or remote speech synthesis can be utilized at least selectively. For example, recognized text can be selectively transmitted to a remote automation assistant component for remote NLU, remote execution, and / or speech synthesis. For instance, recognized text can be selectively transmitted for remote execution in parallel with on-device execution, or in response to a failure of on-device NLU and / or on-device execution. However, on-device speech processing, on-device NLU, on-device execution, on-device speech synthesis, and / or on-device execution can be at least prioritized due to the reduced latency they provide when parsing spoken utterance (since no client-server round trip is required to parse spoken utterance). Furthermore, in the absence of network connectivity or with limited network connectivity, on-device functionality can be the only available functionality.

[0027] The above description serves as an overview of some embodiments of this disclosure. These embodiments and other embodiments are further described in more detail below.

[0028] Some embodiments disclosed herein include one or more computing devices, which include one or more processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or a tensor processing unit (TPU). One or more of the processors are operable to execute instructions stored in associated memory, and the instructions are configured to cause the execution of any of the methods described herein. The computing device can include, for example, a client assistant device having a microphone, at least one display, and / or other sensor components. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions that can be executed by one or more processors to perform any of the methods described herein. Attached Figure Description

[0029] Figure 1A Example process flows are described, according to various implementations, for updating the ASR model on the device based on gradients generated locally at the client device and / or transmitting gradients to a remote system to update the global ASR model.

[0030] Figure 1B Example process flows are described, according to various implementations, for updating the on-device TTS generator based on gradients generated locally at the client device and / or transmitting gradients to a remote system to update the global TTS generator model.

[0031] Figure 2 It includes those from Figure 1A and Figure 1B The example environment of various components and the block diagram in which the implementations disclosed herein can be implemented.

[0032] Figure 3 A flowchart illustrating example methods for training a TTS generator model locally stored on a client device, according to various implementations, is provided.

[0033] Figure 4 The flowchart illustrates example methods for generating gradients locally at a client device and using the generated gradients to update the weights of an ASR model on the device and / or transmitting the gradients to a remote system, according to various implementations.

[0034] Figure 5 The flowchart illustrates example methods for generating gradients locally at a client device and using the generated gradients to update the weights of a TTS generator model on the device and / or transmitting the gradients to a remote system, according to various implementations.

[0035] Figure 6 The illustrations depict the transmission to a remote system during execution according to various implementation methods. Figure 4 Methods and / or Figure 5 The flowchart shows an example method for generating gradients locally on the client device corresponding to the method.

[0036] Figure 7 The illustrations depict various implementations based on the current execution. Figure 4 Methods and / or Figure 5 The flowchart illustrates an example method that uses the gradients received at the remote client device of the corresponding instance of the method to update the weights of the global model and transmits the updated weights and / or the updated global model to the remote client device.

[0037] Figure 8 The illustration shows example architectures of computing devices according to various implementation methods. Detailed Implementation

[0038] Turn now Figure 1A This describes an example process flow for updating the on-device ASR model based on gradients generated locally at the client device and / or transferring gradients to a remote system to update the global ASR model. Client device 110 in... Figure 1A The diagram shows, and includes, the identifier of client device 110. Figure 1A The component within the frame. The fragment identifier 120 of the client device 110 accesses the on-device storage device 111 to identify the text fragment 101. The on-device storage device 110 may include, for example, read-only memory (ROM) and / or random access memory (RAM). The text fragment 101 identified by the fragment identifier 120 is a text fragment stored transiently or non-transiently in the on-device storage device 111. For example, the text fragment 101 may be: an alias of a contact stored in a contact list; a road name stored as an address in a contact list; the name of a song or other media item included in a playlist of a media application; an alias of a smart device, wherein the alias is stored at the client device and the smart device is associated with the user's account; a text fragment typed via a virtual keyboard at the client device; a text fragment copied to the clipboard at the client device; a text fragment rendered by an application of the computing device (and optionally identified using screenshots and image recognition); a text fragment generated based on spoken words of a user processing the client device 110; or other text fragments. When client device 110 is associated with multiple users (e.g., a shared client device located in the main residence of a family), the storage device 111 on the device is able to include any text fragments of speech corresponding to the user of client device 110 or any other user of client device 110 (e.g., determined using speaker identification or speaker binarization techniques).

[0039] In some implementations, fragment identifier 120 identifies text fragment 101 based on its new encounter or new storage at client device 110. For example, fragment identifier 120 can identify text fragment 101A based on whether it is included in a newly added contact, is an alias for a newly added smart device, is an alias for a newly added song in a playlist, or is generated based on spoken words from a user processing client device 110. In some implementations, fragment identifier 120 identifies text fragment 101A based on determining that it is not in a vocabulary, which can be based on determining that text fragment 101A is not included in a locally stored dictionary. It is noteworthy that in many implementations, any text fragment identified by fragment identifier 120 can be limited to those text fragments derived from spoken words from a user (or multiple users) of client device 110.

[0040] In some implementations, fragment identifier 120 can identify text fragment 101A based on determining that spoken utterance detected via the microphone of client device 110 includes text fragment 101A and determining that previous speech recognition of the previous spoken utterance failed to correctly recognize text fragment 101A. In those implementations, fragment identifier 120 can determine that previous speech recognition failed to correctly recognize the text fragment based on the deletion of user input received at client device 110 that was incorrectly predicted based on previous speech recognition. Furthermore, fragment identifier 120 can determine that the previous human utterance included text fragment 101A based on additional user input received after the deletion of user input that was incorrectly predicted based on previous speech recognition.

[0041] Fragment identifier 120 provides at least text fragment 101A to on-device TTS generator 122A. In some embodiments, fragment identifier 120 provides the identified text fragment as well as additional text fragments to on-device TTS generator 122A. For example, fragment identifier 120 can append additional text fragments before and / or after the text fragment, and provide text fragment 101A with the appended additional text fragments to on-device TTS generator 122A. In some of those embodiments, text fragment generator 120A can process text fragment 101A to generate multiple context-dependent and semantically dissimilar additional text fragments. For example, text fragment generator 120A can determine that additional text fragments are semantically dissimilar based on generating multiple candidate text fragment embeddings on an encoder model. The generated candidate text fragment embeddings can be low-dimensional representations that map candidate text fragments to a low-dimensional candidate text fragment embedding space. The embedding for a given candidate text fragment can be compared with the embedding of the candidate text fragment, and if the comparison indicates that a difference metric is satisfied, the candidate text fragment can be appended or appended to text fragment 101A. For example, a difference metric can satisfy a threshold indicating a sufficient degree of semantic difference between a given text fragment and a selected candidate text fragment. As another example, text fragment generator 120A can determine whether a text fragment is context-dependent based on a defined relation between another text fragment and a specific corpus from which text fragment 101A is identified. For example, if text fragment 101A is identified from a media corpus, the other text fragments “play” and “show me” can have a defined relation to the media corpus, and one of these other text fragments is appended before this text fragment. As another example, if text fragment 101A is identified from a previous spoken utterance of “send a message to Francoisé,” but speech recognition fails to correctly identify “Francoisé,” the other text fragments “arrange a meeting with Francoisé” and “remind me to tell Francoisé the presentation” can be generated. Generating multiple semantically dissimilar and context-dependent other text fragments appended or appended to text fragment 101A allows for the generation of multiple dissimilar training instances based on text fragment 101A.

[0042] On-device TTS generator 122A processes text segment 101 (and any additional text segments appended or added before it) using on-device TTS generator model 152A1 to generate synthesized speech audio data 102, which includes the synthesized speech of the identified text segment 101A. For example, on-device TTS generator 122A can determine the phoneme sequence identified as corresponding to text segment 101 (and any additional text segments appended or added before it) and process the phoneme sequence using on-device TTS generator model 152A1 to generate synthesized speech audio data 102. Synthetic speech audio data 102 can be, for example, in the form of an audio waveform. In determining the phoneme sequence corresponding to text segment 101, on-device TTS generator 122A can access token-to-phoneme mappings, such as optional token-to-phoneme mappings 157, stored locally at client device 110. In some embodiments, the generated synthesized speech audio data 102 can be a mixture of human speech and synthesized speech. For example, if text segment 101A is recognized from a previous spoken utterance of "send a message to Francoisé," but speech recognition fails to correctly identify "Francoisé," additional text segments such as "arrange a meeting with Francoisé" and "remind me to tell Francoisé to attend" can be generated. In these examples, a portion of the synthesized speech audio data 102 for the additional text segments may include audio data capturing the user's voice for "Francoisé," but the remainder of the synthesized speech audio data 102 may correspond to synthesized speech. For example, suppose the synthesized speech audio data 102 corresponds to "arrange a meeting with Francoisé." In this example, the portion corresponding to "arrange a meeting with Francoisé" could be synthesized speech audio data generated using the on-device TTS generator 122A, while the portion corresponding to "Francoisé" could be audio data from the original spoken utterance.

[0043] In some implementations, the on-device TTS generator model 152A is transmitted for storage and use at the client device 110 based on the geographic region of the user of client device 110, the user's primary language, and / or other attributes of client device 100 and / or the user of client device 110 (e.g., by remote system 160 or other components). For example, the on-device TTS generator model 152A1 can be one of N available TTS generator models for a given language, but can be trained based on spoken utterances specific to a particular geographic region and provided to the client device 110 based on the fact that the client device 110 is primarily located in that particular geographic region.

[0044] The on-device TTS generator model 152A1 is trained by the on-device TTS generator training engine 130A to adapt the on-device speech synthesis model 152 to the voice characteristics of the user of the client device 110 based on multiple training instances. Each of the multiple training instances includes a training instance input and a training instance output.

[0045] In some implementations, the on-device TTS generator training engine 130A can identify a basic fact transcription of a user's previous spoken utterances on the client device 110 by generating a transcript (e.g., processing corresponding audio data using an on-device ASR model 154A) and identify that transcript as a basic fact transcript. The basic fact transcript can also include one or more basic fact text fragments for corresponding items included in the user's previous spoken utterances on the client device 110. Identifying a transcript as "basic fact" can optionally depend on a confidence metric of the transcript meeting a threshold and / or on indicating that the transcript represents a correct user action (or inaction) in response to the generation of the transcript. In those implementations, the transcript (or the text fragments included in the transcript) can be used as training instance input for a given training instance, and the corresponding audio data of the captured previous spoken utterances can be used as training instance output for a given training instance. In some versions of these implementations, the previous spoken utterance can be a registered phrase spoken by the user for text-independent or text-related identification, and the registered phrase can be rendered to the user to inform the user what to say during registration. In those implementations, it is possible to utilize the registration phrase rendered to the user as a basic fact transcription (or a basic fact text fragment included in the transcription).

[0046] For example, before generating the synthesized speech audio data 102, the on-device TTS generator training engine 130A is able to identify audio data of previous human speech detected and captured via one or more microphones of the client device 110. Furthermore, the on-device TTS generator training engine 130A is able to identify basic fact transcriptions used for the previous human speech. Additionally, the on-device TTS generator training engine 130A enables the on-device TTS generator 122A to use the on-device TTS generator model 152A1 to process basic fact transformations to generate the previous synthesized speech audio data, and is able to generate a loss based on comparing the previous synthesized speech audio data with previous audio data. The on-device speech TTS generator engine 130A is then able to update the weights of the on-device TTS generator model 152A1 based on the loss (e.g., using backpropagation and / or other training techniques).

[0047] In some alternative or alternative implementations, the on-device TTS generator model 152A1 may be the generator portion of a generative adversarial network (GAN) model. The GAN model may also include an on-device TTS discriminator model 152A2 locally stored at the client device 110. In some versions of these implementations, the on-device TTS discriminator training engine 130B is capable of recognizing multiple training instances (e.g., discriminator training instances) utilized in training the on-device TTS discriminator model 152A2. The on-device TTS discriminator training engine 130B is capable of recognizing multiple positive training instances and multiple negative training instances. For each of the multiple positive training instances, the training instance input may include audio data capturing previous spoken utterances of the user of the client device 110. For each of the multiple positive training instances, the training instance output may include a basic fact label (e.g., binary value, semantic label, probability, etc.) indicating that the training instance input corresponds to the speech of the user of the client device 110. Conversely, for each of the multiple negative training instances, the training instance input can include synthetic audio data that captures synthesized speech generated using the on-device TTS generator model 152A1 (or another speech synthesizer stored locally on the client device or remotely on a remote system). For each of the multiple negative training instances, the training instance output can include a basic fact label (e.g., another binary value, another semantic label, probability, etc.) indicating that the training instance input corresponds to the synthesized speech.

[0048] For example, before generating synthesized speech audio data 102, the on-device TTS discriminator training engine 130B can recognize audio data detected via the microphone of the client device 110 and capturing the user's previous spoken words. Furthermore, the on-device TTS discriminator training engine 130B can recognize synthesized speech audio data capturing synthesized speech generated using the on-device TTS generator model 152A1 (or another speech synthesizer locally stored at the client device or remotely stored at a remote system). Additionally, the on-device TTS discriminator training engine 130B can recognize corresponding basic fact tags for spoken words and synthesized speech. Furthermore, the on-device TTS discriminator training engine 130B enables the on-device TTS discriminator 122B to process audio data (or synthesized speech audio data) using the on-device TTS discriminator model 152A2 to predict whether the audio data (or synthesized speech audio data) corresponds to the spoken words (or portions thereof) of the user on the client device 110 or to synthesized speech generated using the on-device TTS generator model 152A1 (or another speech synthesizer stored locally on the client device or remotely on a remote system), and is able to generate a loss based on the prediction. The on-device TTS discriminator training engine 130B can then update the weights of the on-device TTS discriminator model 152A2 based on the loss (e.g., using backpropagation and / or other training techniques).

[0049] Furthermore, the on-device TTS generator training engine 130A can identify multiple training instances (e.g., generator training instances) used when training the on-device TTS generator model 152A1. Training instances can include a given text segment (and any additional text segments appended or added before it) stored locally on the client device 110. For example, before generating synthetic speech audio data 102 (and optionally after training the on-device TTS discriminator model 152A2), the on-device TTS generator training engine 130A can identify a given text segment stored locally on the client device 110. The on-device TTS generator training engine 130A enables the on-device TTS generator engine 122A to use the on-device TTS generator model 152A1 to process the given text segment (and any additional text segments appended or added before it) to generate synthetic speech audio data including synthetic speech. Furthermore, the on-device TTS discriminator training engine 130B enables the on-device TTS discriminator 122B to process synthesized speech audio data using the on-device TTS discriminator model 152A2 to predict whether the synthesized speech audio data corresponds to the spoken words (or portions thereof) of the user of the client device 110 or to synthesized speech generated using the on-device TTS generator model 152A1, and is able to generate a loss based on the prediction. The on-device TTS generator training engine 130A is then able to update the weights of the on-device TTS generator model 152A1 based on the loss (e.g., using backpropagation and / or other training techniques). This loss can be considered as the adversarial loss utilized in the on-device TTS generator model 152A1 during the training of the GAN model. In other words, during the training of the on-device TTS generator model 152A1, the on-device TTS generator model 152A1 attempts to trick the on-device TTS generator model 152B1 into predicting that the synthesized speech audio data corresponds to audio data capturing the spoken words of the user of the client device 110, rather than synthesized speech generated using the on-device TTS generator model 152A1.

[0050] The trained on-device TTS generator 122A provides synthesized speech audio data 102 to the on-device ASR engine 124. The on-device ASR engine 124 uses the on-device ASR model 154A to process the synthesized speech audio data 102 to generate predicted ASR outputs (e.g., predicted text segments 103A, predicted phoneme sequences 103B, and / or other predicted ASR outputs).

[0051] For example, when the on-device speech ASR model 154A is an end-to-end speech recognition model, the on-device ASR engine 124 can directly use the model to generate predicted text fragments 103A. For example, the on-device speech ASR model 154A can be an end-to-end model for generating predicted text on a character-by-character basis (or on another token-by-token basis). A non-limiting example of such an end-to-end model for generating predicted text on a character-by-character basis is a recurrent neural network transducer (RNN-T) model. The RNN-T model is a form of sequence-to-sequence model that does not employ an attention mechanism. Unlike most sequence-to-sequence models, which typically require processing the entire input sequence (e.g., audio data waveforms or Mel-frequency cepstral coefficients (MFCCs) or other representations) to produce the output (predicted text fragments), the RNN-T model can be used to process input samples and stream output symbols (e.g., characters in the alphabet).

[0052] Additionally, for example, when the on-device speech ASR model 154A is not an end-to-end speech recognition model, the on-device ASR engine 124 can instead generate predicted phonemes 103B (and / or other representations). For example, in the case of such a model, the on-device ASR engine 124 then uses the predicted phonemes 103B (and / or other representations) to determine predicted text segments that match the phoneme sequence. In doing so, the on-device ASR engine 124 can optionally employ decoding graphs, dictionaries, and / or other resources.

[0053] When the ASR engine 124 on the device generates a predicted text segment 103A, it is provided to the gradient engine 126. The gradient engine 126 compares the predicted text segment 103A with the text segment 101A to generate a gradient 105. The gradient can be derived from the loss function used to train the model, such that the gradient represents the value of the loss function (or its derivative) obtained from the comparison between the ground truth output and the predicted output. For example, when the predicted text segment 103A matches the text segment 101A, the gradient engine 126 can generate a zero gradient. Alternatively, for example, when the predicted text segment 103A does not match the text segment 101A, the gradient engine 126 can generate a non-zero gradient, which optionally depends on the degree of mismatch. The degree of mismatch can be based on the degree of mismatch between characters of the text segments, the degree of phoneme mismatch between text segments, and / or based on other deterministic comparisons. As a non-limiting specific example, in each iteration of generating the predicted text fragment 103A, the on-device ASR engine 124 is able to generate a corresponding probability for each of a plurality of characters and select the character with the highest probability as the "next" character. In this example, the gradient engine 126 is able to determine the gradient based on comparing the predicted probability of each iteration with the ground fact probability of each iteration (i.e., where the ground fact character of each iteration is the "next" character in text fragment 101A and is assigned a probability of "1", and all other characters are assigned a probability of "0").

[0054] When the ASR engine 124 on the device generates predicted phonemes 103B, they are provided to the gradient engine 126. The gradient engine 126 compares the predicted phonemes 103B with the fundamental sequence of facts of phonemes 104 determined to correspond to text segment 101A. In some implementations, the phoneme engine 127 can access the token-to-phoneme map 157 to determine the fundamental sequence of facts of phonemes 104 corresponding to text segment 101A. As an example, when the predicted phonemes 103B match phonemes 104, the gradient engine 126 can generate a zero gradient. As another example, when the predicted phonemes 103B do not match phonemes 104, the gradient engine 126 can generate a non-zero gradient, which optionally depends on the degree of mismatch. The degree of mismatch can be based on the number of mismatched phonemes, the number of correctly ordered phonemes, and / or the distance between mismatched phonemes (in phoneme space), and / or based on other deterministic comparisons.

[0055] It should be noted that in embodiments where additional text segments are appended or attached to text segment 101A as described herein, the predicted text segment 103A will also include predictions of the appended and / or attached additional text segments. Predictions of the appended and / or attached additional segments can be ignored when generating gradients (e.g., terms corresponding to the attached additional segments are discarded), or alternatively, the appended and / or attached additional segments can be taken into account when generating gradients (e.g., predictions can be compared with text segments having the attached additional text segments).

[0056] Gradient engine 126 provides gradient 105 to on-device ASR training engine 128 and / or sends gradient 105 to remote system 160. On-device ASR training engine 128 uses gradient 105 to update on-device ASR model 154A upon receiving it. For example, on-device ASR training engine 128 can utilize backpropagation and / or other techniques to update on-device ASR model 154A. Note that in some embodiments, on-device ASR training engine 128 can utilize batch processing techniques to update on-device ASR model 154A based on gradient 105 and additional gradients determined locally at client device 110 based on additional text fragments.

[0057] When remote system 160 receives gradient 105, its remote training engine 162 uses gradient 105 and another gradient 106 from another client device 170 to update the global weights of the global ASR model 154B. This can be achieved based on the gradients described above. Figure 1A The same or similar techniques described for gradient 105 are used to generate additional gradient 106 from another client device 170 (but based on locally recognized text fragments 101A specific to those other client devices 170). Although in Figure 1A Not described herein, additional gradients can be applied at remote system 160 based on the above. Figure 1A It is generated using the same or similar techniques as described in gradient 105 (but based on remotely recognizable text fragment 101A accessible by remote system 160).

[0058] The update allocation engine 164 is capable of providing updated global weights and / or an updated global ASR model itself to client device 110 and / or other client devices in response to one or more conditions being met, as indicated by 107. One or more conditions may include, for example, the threshold duration and / or training amount since the last provision of updated weights and / or an updated speech recognition model. One or more conditions may additionally or alternatively include, for example, the threshold duration elapsed since the last provision of updated weights and / or an improvement in the measurement of the updated speech recognition model. When updated weights are provided to client device 110, client device 110 is capable of replacing the local weights of on-device ASR model 154A with the updated weights. When an updated global ASR model is provided to client device 110, client device 110 is capable of replacing on-device ASR model 154A with the updated global ASR model 154B.

[0059] By transmitting gradient 105 to remote system 160 and updating the global weights of global ASR model 154B based on gradient 105, global ASR model 154B can be updated using on-device ASR model 154A. For example, suppose on-device ASR model 154A knows that the sound "fran-swaz" corresponds to a text segment. This is because it was previously corrected at the client device 110, and the ASR model 154A on the device is updated based on the gradient generated based on that correction. However, the global ASR model 154B may not know that the sound "fran-swaz" corresponds to the text fragment. Nevertheless, by updating the global ASR model 154B based on the gradient generated from this correction, the global ASR model 154B is also able to determine that the sound “fran-swaz” corresponds to a text segment. Instead Or “Francis”. Conversely, if the audio data corresponding to the sound “fran-swaz” is stored at a remote system 160 and annotated by humans, then humans may not know that “fran-swaz” corresponds to… Furthermore, the sound "fran-swaz" was incorrectly annotated as corresponding to... Or “Francis” (e.g., a hard negative). In this example, the global ASR model 154B can be updated to detect the error (e.g., selecting a text segment in response to receiving audio data capturing the sound “fran-swaz”). Or "Francis" instead ).

[0060] Turn now Figure 1BThis describes an example process flow for updating the on-device TTS generator model based on gradients generated locally at the client device and / or transmitting those gradients to a remote system to update the global TTS generator model. Figure 1A The client device 110 can perform additional or alternative operations. Figure 1B An example of the process flow is shown. In some implementations, and with... Figure 1A Instead, the trained on-device TTS generator 122A provides the synthesized speech audio data 102 directly to the gradient engine 126, rather than to the on-device ASR engine 124. The gradient engine 126 compares the synthesized speech audio data 102 with the ground truth audio data 101B. More specifically, the gradient engine 126 is capable of comparing the acoustic features of the synthesized speech audio data 102 and the ground truth audio data 101B. Acoustic features may include, for example, audio waveforms, mel frequency cepstral coefficients (MFCCs), mel filter bank features, values ​​associated with one or more prosodic attributes, neural representations of the audio data (e.g., corresponding embeddings of the ground truth audio data and the predicted synthesized speech audio data), and / or other acoustic features of the synthesized speech audio data and the ground truth audio data that can be compared. The ground truth audio data 101B may be stored in association with text segments 101B processed by the on-device TTS generator 122A (e.g., in on-device storage 111) and may be recognized together with the text segments 101A. The gradient can be derived from the loss function used to train the model, such that the gradient represents the value of that loss function (or its derivative) obtained by comparing the base-fact output with the predicted output. For example, when the acoustic features of the synthesized speech audio data 102 and the base-fact audio data 101B match, the gradient engine 126 can generate a zero gradient. Furthermore, for example, when the acoustic features of the synthesized speech audio data 102 and the base-fact audio data 101B do not match, the gradient engine 126 can generate a non-zero gradient, which optionally depends on the degree of mismatch. The degree of mismatch can be based on the degree of mismatch between characters in the text segment, the degree of phoneme mismatch between text segments, and / or based on other deterministic comparisons.

[0061] In some other or alternative implementations, and with Figure 1AInstead, the trained on-device TTS generator 122A provides the synthesized speech audio data 102 to the on-device TTS discriminator 122B, rather than to the on-device ASR engine 124 (or directly to the gradient engine 126). The on-device TTS discriminator 122B can process the synthesized speech audio data 102 using the on-device TTS discriminator model 152A2 to make a prediction 108. The prediction 108 made by the on-device TTS discriminator model 152A2 can indicate whether the synthesized speech audio data 102 corresponds to the spoken words (or portions thereof) of the user of the client device 110 or to the synthesized speech generated using the on-device TTS generator model 152A1. The on-device TTS discriminator 122B provides the prediction 108 to the gradient engine 126. Prediction 108 can be, for example, a binary value (e.g., where “0” corresponds to synthesized speech and where “1” corresponds to the spoken words of a user of client device 110), a semantic label (e.g., where “synthetic” or “fake” corresponds to synthesized speech and where “human” or “real” corresponds to the spoken words of a user of client device 110), and / or a probability (e.g., “0.65” associated with “synthetic” and “0.35” associated with “real”). Gradient engine 126 can generate gradients based on prediction 108. For example, when prediction 108 includes a binary value or a semantic label and prediction 108 is correct, gradient engine 126 can generate a zero gradient. Furthermore, for example, when prediction 108 includes a probability and prediction 108 is incorrect, gradient engine 126 can generate a non-zero gradient, which optionally depends on the degree of mismatch between the probability and the probability of the underlying fact associated with the correct prediction.

[0062] Note that in embodiments where additional text segments are appended or appended to text segment 101A as described herein, the synthesized speech audio data 102 will also include synthesized speech corresponding to the appended and / or appended additional text segments. When generating gradients, the synthesized speech corresponding to the appended and / or appended additional segments can be ignored (e.g., terms corresponding to the appended additional segments are discarded), or alternatively, the appended and / or appended additional segments can be taken into account when generating gradients (e.g., predictions can be compared with text segments having the appended additional text segments).

[0063] Gradient engine 126 provides gradient 105 to on-device TTS generator training engine 130A and / or transmits gradient 105 to remote system 160. On-device TTS generator training engine 130A uses gradient 105 to update on-device TTS generator model 152A1 when it receives gradient 105. For example, on-device TTS generator training engine 130A can utilize backpropagation and / or other techniques to update on-device TTS generator model 152A1. Note that in some embodiments, on-device TTS generator training engine 130A can utilize batch processing techniques to update on-device TTS generator model 152A1 based on gradient 105 and additional gradients determined locally at client device 110 based on additional text fragments.

[0064] When the remote system 160 receives gradient 105, its remote training engine 162 uses gradient 105 and another gradient 106 from another client device 170 to update the global weights of the global TTS generator model 152B. This can be achieved based on the above... Figure 1B The same or similar techniques described for gradient 105 are used to generate additional gradient 106 from another client device 170 (but based on locally recognized text fragments 101A specific to those other client devices 170). Although in Figure 1B Not described herein, additional gradients can be applied at remote system 160 based on the above. Figure 1B It is generated using the same or similar techniques as described in gradient 105 (but based on remotely recognizable text fragment 101A accessible by remote system 160).

[0065] The update allocation engine 164 is capable of providing updated global weights and / or an updated global TTS generator model itself to client device 110 and / or other client devices in response to one or more conditions being met, as indicated by 107. One or more conditions may include, for example, the threshold duration and / or training amount since the last provision of updated weights and / or an updated speech recognition model. One or more conditions may additionally or alternatively include, for example, the threshold duration elapsed since the last provision of updated weights and / or an improvement in the measurement of the updated speech recognition model. When updated weights are provided to client device 110, client device 110 is capable of replacing the local weights of on-device TTS generator model 152A1 with the updated weights. When an updated global TTS generator model is provided to client device 110, client device 110 is capable of replacing on-device TTS generator model 152A1 with the updated global TTS generator model 152B.

[0066] Now go to Figure 2 In it Figure 1A and Figure 1B In an embodiment where an on-device TTS generator 122 and an on-device ASR engine 124 are included as part of (or communicating with) an automation assistant client 140, a client device 110 is illustrated. An on-device TTS generator model 152A1 that interfaces with the on-device TTS generator 122 is illustrated, and an on-device ASR model 154A that interfaces with the on-device ASR engine 124 is illustrated. Figure 1A Other components are included for simplicity. Figure 2 Not shown in the image. Figure 2 The illustration shows an example of how on-device TTS generator 122 and on-device TTS generator model 152A1 can be used to generate synthesized speech audio data for an automated assistant client 140 to be rendered at client device 110. Furthermore, Figure 2 The illustration shows an example of how the on-device ASR engine 124 and on-device ASR model 154A can be used to generate the recognized text used by the automation assistant client 140 when performing various actions.

[0067] Figure 2 The client device 110 is illustrated to have a microphone 111, a speaker 112, a camera and / or other visual components 113, and a display 114 (e.g., a touch-sensitive display). The client device 110 selectively operates the automation assistant client 140. Figure 1B In the example, the automation assistant client 140 includes an on-device TTS generator 122, an on-device ASR engine 124, an on-device Natural Language Understanding (NLU) engine 144, and an on-device execution engine 145. The automation assistant client 140 further includes a voice capture engine 141 and a visual capture engine 142. The automation assistant client 140 can include additional and / or alternative engines, such as a Voice Activity Detector (VAD), an endpoint detector, a hot word detector, and / or other engines.

[0068] One or more cloud-based automation assistant components 180 can optionally be implemented on one or more computing systems (collectively referred to as “cloud” computing systems), which are communicatively coupled to client device 110 via one or more local and / or wide area networks (e.g., the Internet) generally indicated by 190. The cloud-based automation assistant components 180 can be implemented, for example, via a cluster of high-performance servers.

[0069] In various implementations, an instance of the automation assistant client 140, through its interaction with one or more cloud-based automation assistant components 180, can form a logical instance of the automation assistant 195 from the user's perspective, which the user can use to engage in human-computer interaction (e.g., verbal interaction, gesture-based interaction, and / or touch-based interaction).

[0070] Client device 110 can be, for example: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a user's vehicle computing device (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker, a smart home appliance such as a smart TV (or a standard TV equipped with network software protection with automation assistant capabilities), and / or a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or replacement client devices may be provided.

[0071] The vision component 113 can take various forms, such as a single-lens reflex camera, a stereo camera, a LiDAR component (or other laser-based component), a radar component, etc. The vision component 113 can, for example, be used by the vision capture engine 142 to capture visual frames (e.g., image frames, laser-based visual frames) of the environment in which the client device 110 is deployed. In some embodiments, such visual frames can be used to determine whether a user is near the client device 110 and / or the distance of the user (e.g., the user's face) relative to the client device. For example, this determination can be used to determine whether the ASR engine 124 on the device is activated.

[0072] The speech capture engine 141 can be configured to capture a user's speech and / or other audio data captured via microphone 111. As described herein, such audio data can be utilized by on-device ASR engine 124 (optionally after preprocessing). For example, on-device ASR engine 124 can utilize on-device ASR model 154A to process the audio data of the captured spoken utterance to generate recognized text corresponding to the spoken utterance. On-device NLU engine 140 performs on-device natural language understanding on the recognized text to generate NLU data. NLU engine 144 can optionally utilize one or more on-device NLU models (not shown for simplicity) Figure 1B (See diagram) to generate NLU data. NLU data can include, for example, the intent corresponding to the spoken utterance and optional parameters related to the intent (e.g., slot values). Furthermore, the on-device performance engine 145 uses the NLU data to generate performance data. The on-device performance engine 145 can optionally utilize one or more on-device performance models (not shown in the diagram for simplicity). Figure 2(See diagram) to generate performance data. This performance data can define local and / or remote responses to spoken utterances (e.g., answers), interactions with locally installed applications performed based on spoken utterances, sending commands to Internet of Things (IoT) devices (directly or via corresponding remote systems) based on spoken utterances, and / or other parsed actions performed based on spoken utterances. The performance data is then provided for the local and / or remote execution / running of the determined actions for parsing spoken utterances. The run can include, for example, rendering local and / or remote responses (e.g., visual and / or auditory rendering (via on-device TTS generator 122 using on-device TTS generator model 125A1)), interacting with locally installed applications, sending commands to IoT devices, and / or other actions.

[0073] Display 114 can be used to render the streaming text transcription from the speech recognizer 124 on the device. Display 114 can also be one of the user interface output components through which the visual portion of the response from the automation assistant client 140 is rendered.

[0074] In some implementations, the cloud-based automation assistant component 180 may include: a remote ASR engine 182 for speech recognition using a global ASR model 154B; a remote NLU engine 183 for natural language understanding; a remote performance engine 184 for generating performance data; and / or a remote TTS generator 185 for generating synthesized speech audio data. A remote execution module may also be optionally included, which performs remote execution based on locally or remotely determined performance data. Additional and / or alternative remote engines may be included. As described herein, in various implementations, on-device speech processing, on-device NLU, on-device performance, and / or on-device execution are preferred at least due to the reduced latency and / or network usage they provide when parsing spoken utterance (since no client-server round trip is required to parse spoken utterance). However, one or more cloud-based automation assistant components 180 may be utilized at least selectively. For example, such components may be utilized in parallel with on-device components, and the output from such components may be utilized when a local component fails. For example, on-device execution engine 145 may fail in certain situations (e.g., due to relatively limited resources on client device 160), and remote execution engine 184 can utilize the more robust resources of the cloud to generate execution data in such cases. Remote execution engine 184 can operate in parallel with on-device execution engine 145 and utilize its results when on-device execution fails, or can be invoked in response to determining a failure of on-device execution engine 145.

[0075] In various implementations, the NLU engine (on-device 144 and / or remote 183) is capable of generating output including one or more annotations of the recognized text and annotations of one or more (e.g., all) of the terms in the natural language input. In some implementations, the NLU engine is configured to recognize and annotate various types of syntactic information in the natural language input. For example, the NLU engine may include a lexical module that can segment individual words into lexicals, and / or, for example, annotate lexical categories. The NLU engine may also include a portion of a part-of-speech tagger configured to annotate terms using their syntactic functions. Additionally, for example, in some implementations, the NLU engine may additionally and / or alternatively include a dependency analyzer configured to determine syntactic relations between terms in the natural language input.

[0076] In some implementations, the NLU engine may additionally and / or alternatively include an entity annotator configured to annotate entity references in one or more fragments, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, places (real and imaginary), etc. In some implementations, the NLU engine may additionally and / or alternatively include a coreference resolver (not depicted) configured to group or "cluster" references to the same entity based on one or more contextual cues. In some implementations, one or more components of the NLU engine may depend on annotations from one or more other components of the NLU engine.

[0077] The NLU engine may also include an intent matcher configured to determine the intent of a user interacting with the automation assistant 195. The intent matcher is capable of using various techniques to determine the user's intent. In some implementations, the intent matcher may access one or more local and / or remote data structures that include, for example, multiple mappings between syntaxes and responsive intents. For example, it may be possible to select and / or learn the syntaxes included in the mappings over time, and these syntaxes may represent common user intents. For example, a syntax "Play <artist>" may be mapped to an intent to invoke a responsive action that causes the music of that <artist> to be played on client device 110. Another syntax, "[Weather|Forecast] Today," may match user queries such as "What's the weather like today?" and "What's the forecast for today?". In addition to or in lieu of syntaxes, in some implementations, the intent matcher may employ one or more trained machine learning models, alone or in combination with one or more syntaxes. These trained machine learning models can be trained to recognize intent, for example, by embedding recognized text from spoken utterances into a reduced-dimensional space, and then determining which other embeddings (and therefore, intents) are closest, for example, using techniques such as Euclidean distance, cosine similarity, etc. As seen in the “play <artist>” example grammar above, some grammars have slots (e.g., <artist>) that can be filled with slot values ​​(or “parameters”). Slot values ​​can be determined in various ways. Typically, the user will actively provide the slot value. For example, for the grammar “order me a <toppings> pizza,” the user might say the phrase “order me a sausage pizza,” in which case the slot <toppings> would be automatically filled. Other slot values ​​can be inferred based on, for example, user location, currently rendered content, user preferences, and / or other cues.

[0078] The fulfillment engine (local engine 145 and / or remote engine 184) can be configured to receive the predicted / estimated intent and any associated slot values ​​output by the NLU engine, and fulfill (or “parse”) the intent. In various implementations, the fulfillment (or “parse”) of the user intent can be achieved, for example, by generating / obtaining various fulfillment information (also referred to as fulfillment data) through the fulfillment engine. This can include determining local and / or remote responses to spoken utterances (e.g., answers), interactions with locally installed applications based on spoken utterances, commands sent to Internet of Things (IoT) devices (directly or via corresponding remote systems) based on spoken utterances, and / or other parsing actions based on spoken utterances. The on-device fulfillment then initiates local and / or remote execution / running of the determined actions that parse the spoken utterances.

[0079] Now go to Figure 3This document describes a flowchart illustrating an example method 300 for training a TTS generator model locally stored on a client device. For convenience, the operations of method 300 are described with reference to the system performing the operations. This system includes one or more processors and / or other components of the client device. Furthermore, while the operations of method 300 are shown in a specific order, this is not intended to be restrictive. One or more operations may be reordered, omitted, or added.

[0080] In block 352, the system determines whether one or more conditions are met. Although illustrated prior to block 354, note that block 352 can also be executed before each of blocks 356, 358, 360, 362, 364 (if included), 368 (if included), and / or 368—and / or alternatively, only before a subset of blocks 356, 358, 360, 362, 364 (if included), 368 (if included), and / or 368. In some implementations, block 352 includes determining whether the current state of the client device satisfies one or more conditions. For example, the system can determine the current state of the client device based on sensor data from sensors of the client device, and determine whether the current state of the client device satisfies the conditions. Conditions may include, for example, that the client device is charging, that the client device is at least in a threshold charging state, that the client device is not currently moving and / or has not moved within a threshold time period (e.g., based on sensor data from an accelerometer, magnetometer, and / or other sensors), and / or that the client device is connected to an unmetered network (e.g., WiFi) such that the user of the client device is not charged for data reception and / or transmission. If the system determines that one or more conditions are met, the system may proceed to box 354.

[0081] At box 354, the system identifies a given training instance from multiple training instances locally stored on the user's client device. Each of the multiple training instances can include a training instance input and a training instance output. The training instance input can include, for example, a given text fragment locally stored on the client device. The training instance output can include, for example, basic factual audio data corresponding to the spoken words of the user on the client device corresponding to the given text fragment. Multiple training instances can be generated locally on the user's client device based on the user's spoken words received at the client device. In some implementations, in response to receiving spoken words from the user on the client device, the client device can prompt the user to verify the transcription before using the audio data of the captured spoken words and / or the text fragments transcribed from the spoken words as training instances. In various implementations, training instances can be generated by the device to include semantically dissimilar training instances (e.g., as described above regarding...). Figure 1A and 1B(as described in the text fragment generator 120A).

[0082] In box 356, the system identifies a given text fragment associated with the training instance input of a given training instance.

[0083] At box 358, the system uses an on-device TTS generator model stored locally on the client device to process a given text segment to generate predicted synthetic speech audio data. For example, the system can use the on-device TTS generator model to process a phoneme sequence corresponding to a given text segment to generate synthetic speech audio data. In some implementations, the system generates synthetic speech audio data based on a given text segment along with one or more additional text segments appended before or after the given text segment.

[0084] At box 360, the system uses an on-device TTS discriminator model stored locally on the client device to process the predicted synthetic speech audio data to predict whether the predicted synthetic speech audio data corresponds to: (1) spoken utterances of a user on the client device; or (2) synthetic speech generated by an on-device TTS generator model. The on-device TTS generator model used at box 358 and the on-device TTS discriminator model used at box 360 may be part of a GAN model. The goal of the on-device TTS generator model is to generate synthetic speech audio data, which includes synthetic speech predicted by the on-device TTS discriminator model to correspond to spoken utterances of a user on the client device.

[0085] At box 362, the system generates a loss based on processing the predicted synthesized speech audio data using the on-device TTS discriminator model. This loss can be considered in the adversarial loss generated when training the on-device TTS generator model of the GAN model. If the on-device TTS discriminator model predicts that the synthesized speech audio data corresponds to synthesized speech generated by the on-device TTS generator model, the adversarial loss may be larger if the on-device TTS discriminator model predicts that the synthesized speech audio data corresponds to spoken words of a user on a client device.

[0086] In some implementations... Figure 3Method 300 may include boxes 364 and 362. If boxes 364 and 366 are included, the system compares the predicted synthesized speech audio data with ground-fact audio data associated with the training instance output of a given training example, and generates an additional loss based on the comparison at box 364. The system is capable of comparing acoustic features of the synthesized speech audio data with the audio data of the training instance output to generate an additional loss. Acoustic features may include, for example, audio waveforms, mel frequency cepstral coefficients (MFCCs), mel filter bank features, values ​​associated with one or more prosodic attributes, neural representations of the audio data (e.g., corresponding embeddings of the ground-fact audio data and the predicted synthesized speech audio data), and / or other acoustic features of the synthesized speech audio data and the ground-fact audio data that can be compared.

[0087] In box 368, the system updates the on-device TTS generator model based on a loss and / or other losses. By updating the on-device TTS generator model based on the loss and / or other losses, the on-device TTS generator model is trained to include the voice characteristics of the user on the client device. Although the on-device TTS generator model is described herein as being trained to include the user's voice characteristics, it should be understood that the synthesized speech generated using the trained on-device TTS generator model is not an exact match of the user's speech. In other words, it is possible to train the on-device TTS generator model to include prosodic attributes (e.g., one or more of intonation, pitch, stress, frequency, rhythm, or pauses) that indicate the user's speech on the client device.

[0088] Now go to Figure 4 This document depicts a flowchart of an example method 400 that illustrates the local generation of gradients at a client device and the use of these gradients to update the weights of an ASR model on the device and / or to transmit the gradients to a remote system. For convenience, the operation of method 400 is described with reference to the system performing the operation. This system of method 400 includes one or more processors and / or other components of the client device. Furthermore, although the operations of method 400 are shown in a specific order, this is not intended to be restrictive. One or more operations may be reordered, omitted, or added.

[0089] In box 452, the system determines whether one or more conditions are met. Although illustrated prior to box 454, note that box 452 can also be executed before each of boxes 456, 458, 460, and / or 462—and / or alternatively, can be executed only before a subset of boxes 456, 458, 460, and / or 462. In some implementations, box 352 includes determining whether the current state of the client device satisfies one or more conditions. One or more conditions are described in more detail herein (e.g., regarding...). Figure 3(See box 352). If the system determines that one or more conditions are met, the system can proceed to box 354.

[0090] At box 454, the system identifies a given text fragment stored locally on a user's given client device.

[0091] In box 456, the system uses a trained on-device TTS generator model, locally stored on the client device, to process a given text segment to generate synthesized speech audio data including synthesized speech corresponding to the given text segment. This is based on the execution... Figure 3 Method 300 uses multiple instances to train a trained on-device TTS generator model. For example, the system can use the on-device TTS generator model to process phoneme sequences corresponding to a given text segment to generate synthetic speech audio data. In some implementations, the system generates synthetic speech audio data based on a given text segment along with one or more additional text segments appended before or after the given text segment.

[0092] At box 458, the system uses an on-device ASR model stored locally on the client device to process the synthesized speech audio data to generate a corresponding predicted ASR output. The corresponding predicted ASR output can include, for example, a corresponding predicted text segment, a corresponding predicted phoneme sequence, and / or other predicted ASR outputs. In some embodiments, the on-device ASR model is an end-to-end speech recognition model, and the system generates the predicted output as the corresponding predicted text segment. In some other embodiments, the on-device ASR model is not an end-to-end speech recognition model, and the system generates a predicted phoneme sequence and / or another predicted representation. The corresponding predicted text segment can be determined based on the predicted phonemes and / or another predicted representation.

[0093] In box 460, the system generates gradients by comparing the corresponding predicted ASR output with the basic fact output corresponding to a given text segment. For example, when the system generates a predicted output as the corresponding predicted text segment, it can compare the predicted text segment with the given text segment when generating the gradient. Furthermore, for example, when the system generates a predicted phoneme sequence and determines the predicted text segment based on the predicted phoneme sequence, the predicted phoneme sequence can be compared with the phoneme sequence corresponding to the given text segment when generating the gradient.

[0094] In box 462, the system updates the local weights of the ASR model on the device based on the gradients from box 560 and / or transmits the gradients from box 460 to a remote system (e.g., via the Internet or other wide area networks) without transmitting any of the given text segment, synthesized speech audio data, and the corresponding predicted text segment. When the gradients are transmitted to the remote system, the remote system uses the generated gradients and additional gradients from another client device to update the global weights of the global ASR model. After box 462, the system then returns to box 452.

[0095] Now go to Figure 5 This document depicts a flowchart of an example method 500 that illustrates the local generation of gradients at a client device and the use of these gradients to update the weights of a TTS generator model on the device and / or to transmit the gradients to a remote system. For convenience, the operation of method 500 is described with reference to the system performing the operation. This system of method 500 includes one or more processors and / or other components of the client device. Furthermore, although the operations of method 500 are shown in a specific order, this is not intended to be restrictive. One or more operations may be reordered, omitted, or added.

[0096] In box 552, the system determines whether one or more conditions are met. Although illustrated prior to box 454, note that box 552 can also be executed before each of boxes 556, 558, 560, 562, and / or 564—and / or can alternatively be executed only before a subset of boxes 556, 558, 560, 562, and / or 564. In some implementations, box 552 includes determining whether the current state of the client device satisfies one or more conditions. One or more conditions are described in more detail herein (e.g., regarding...). Figure 3 (See box 352). If the system determines that one or more conditions are met, the system can proceed to box 554.

[0097] In box 554, the system identifies a given text fragment stored locally on a user's given client device.

[0098] In box 556, the system uses a trained on-device TTS generator model, locally stored on the client device, to process a given text segment to generate synthetic speech audio data including synthesized speech corresponding to the given text segment. For example, the system can use the on-device TTS generator model to process a phoneme sequence corresponding to the given text segment to generate synthetic speech audio data. In some implementations, the system generates synthetic speech audio data based on a given text segment along with one or more additional text segments appended before or after the given text segment.

[0099] In box 558, the system uses a trained on-device TTS discriminator model, locally stored on the client device, to process the synthesized speech audio data to predict whether the synthesized speech audio data corresponds to: (1) spoken utterances of a user on the client device; or (2) synthesized speech generated by an on-device TTS generator model. The on-device TTS generator model used in box 556 and the on-device TTS discriminator model used in box 558 may be part of a GAN model. The goal of the on-device TTS generator model is to generate synthesized speech audio data, which includes synthesized speech predicted by the on-device TTS discriminator model to correspond to spoken utterances of a user on the client device.

[0100] In some implementations, method 500 may include optional box 560. If box 560 is included, the system identifies basic fact audio data corresponding to the user's spoken utterances, where the text fragments are basic fact text fragments used for the basic fact audio data. The basic fact audio data can be compared with locally stored data on a given client device (e.g., on...). Figure 1A and 1B A given text fragment is stored in association with the storage device 111 on the device. In some versions of these implementations, box 560 may include an optional sub-box 560A. If sub-box 560A is included, the system filters the basic fact audio data to remove additional spoken words and / or ambient noise. The system is capable of filtering the basic fact audio data to remove additional spoken words by using one or more known techniques. For example, the system is capable of using a sound filtering model that processes the basic fact audio data and the speaker embedding associated with the user of the client device to generate basic fact audio data that includes only the user's spoken words and removes any audio data that does not match the speaker embedding associated with the user of the client device. As another example, the system is capable of using a filtering model or algorithm to remove ambient noise to generate basic fact audio data that isolates the user's spoken words.

[0101] At box 562, the system generates a gradient based on the processing performed by the on-device TTS discriminator model at box 558. The gradient generated at box 562 can be analogous to the adversarial loss of a GAN model (e.g., regarding...). Figure 3(As described in box 362). In some embodiments, when optional box 560 is included, box 562 may include subbox 562A. If subbox 562A is included, the system generates gradients based on comparing synthesized speech audio data with base-fact audio data. The system is capable of comparing acoustic features of the synthesized speech audio data with base-fact audio data to generate gradients. Acoustic features may include, for example, audio waveforms, mel frequency cepstral coefficients (MFCCs), mel filter bank features, values ​​associated with one or more prosodic attributes, and / or other acoustic features of the synthesized speech audio data and base-fact audio data that can be compared.

[0102] In box 564, the system updates the local weights of the on-device TTS generator model based on the gradient of box 562 (and optionally additional gradients of subbox 562A) and / or transmits the gradient of box 562 (and optionally additional gradients of subbox 562A) to a remote system (e.g., via the Internet or other wide area network) (without transmitting any given text segment and synthesized speech audio data). When the gradient is transmitted to the remote system, the remote system uses the generated gradient and additional gradients from another client device to update the global weights of the global TTS generator model. After box 564, the system then returns to box 552.

[0103] Turn now Figure 6 The illustration depicts the transmission of data to a remote system during execution. Figure 4 Method 400 and / or Figure 5 The flowchart below shows an example method 600 for gradients generated locally at the client device, corresponding to an instance of method 500. For convenience, the operations of method 600 are described with reference to the system performing the operations. This system includes one or more processors and / or other components of the client device. Furthermore, although the operations of method 600 are shown in a specific order, this is not intended to be restrictive. One or more operations may be reordered, omitted, or added.

[0104] In box 652, the system determines whether one or more conditions are met. Although illustrated prior to box 654, note that box 652 can also be performed prior to box 656. In some implementations, box 652 includes determining whether the current state of the client device satisfies one or more conditions. One or more conditions are described in more detail herein (e.g., regarding...). Figure 3 (See box 352). If the system determines that one or more conditions are met, the system can proceed to box 654.

[0105] In box 654, the system executes... Figure 4 Method 400 and / or Figure 5The corresponding instance of method 500 determines whether a gradient has been generated on the client device. If the system determines that no gradient has been generated on the client device, then at box 652, the system can continuously monitor whether any gradient has been generated on the client device while simultaneously satisfying the condition. If the system determines that a gradient has been generated on the client device, then the system can proceed to box 656.

[0106] In some implementations, method 600 may optionally include blocks 656 and 658. If block 656 is included, the system may base its implementation on audio data including portions of spoken utterances from additional persons other than the user of the client device and / or ambient noise that meets a noise threshold (e.g., relative to...). Figure 5 The system determines whether any audio data (e.g., basic fact audio data) used in generating any gradients is generated, as described in optional subbox 558a. If the system determines that a gradient is generated based on audio data including portions of additional spoken utterances and / or ambient noise that meets a noise threshold, but the audio data is not filtered, the system can proceed to box 658. In box 658, the system suppresses the propagation of gradients to a remote system. Although the system can suppress the propagation of gradients to a remote system, the system can still update on-device models (e.g., on-device TTS generator models and / or on-device ASR models) based on gradients. However, if the audio data is filtered, the system can proceed to box 660. In implementations that omit boxes 656 and 660, the system can proceed directly from box 654 to box 660.

[0107] In box 660, the system transmits the gradients of box 654 (without transmitting the data on which the gradients were generated) to a remote system (e.g., via the Internet or another wide area network). When the gradients are transmitted to the remote system, the remote system uses the generated gradients, along with additional gradients from another client device, to update the global weights of the global model. For example, if the gradients of box 654 include... Figure 4 The system can update the weights of the global ASR model by using the gradient generated by an instance of method 400. Furthermore, if the gradient of box 654 includes gradients based on... Figure 5 The system can update the weights of the global TTS generator model by using the gradient generated by the instance of method 500. After box 660, the system then returns to box 652.

[0108] Now go to Figure 7 The diagram depicts the process based on execution. Figure 4 Method 400 and / or Figure 5The flowchart illustrates an example method 700, which uses gradients received by a remote client device in a corresponding instance of method 500 to update the weights of the global model and transmits the updated weights and / or the updated global model to the remote client device. For convenience, the operation of method 700 is described with reference to the system performing the operation. This system may include various components of various computer systems, such as one or more server devices. Furthermore, although the operations of method 700 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0109] In box 752, the system receives gradients from a remote client device. For example, the system can receive gradients from a device that is currently executing... Figure 4 Method 400 and / or Figure 5 Method 500 allows multiple remote client devices of the corresponding instance to receive gradients.

[0110] In box 754, the system updates the weights of the global model based on the gradient received at box 752. In some implementations, the global model may be a global ASR model. The system is able to update the weights of the global model based on the gradient received at box 752. Figure 4 The system updates the weights of the global ASR model based on any gradient generated by instances of method 400. In some additional and / or alternative implementations, the global model may be a global TTS generator model. The system is able to update the weights of the global ASR model based on the gradients received at box 752. Figure 6 The method 600 instance generates any gradients to update the weights of the global TTS model. Iterations in boxes 752 and 754 can continue when new gradients are received and / or queued after they have been received.

[0111] At box 756, the system at least periodically determines whether one or more conditions are met, such as one or more conditions described herein (e.g., regarding...). Figure 1A and 1B (Update distribution engine 164). Typically, conditions are used as a proxy to determine whether the global model has been updated to adjust the degree of network resource utilization when transmitting the updated weights of the global model and / or the updated global model itself. In other words, conditions are used as a proxy to determine whether the performance gain of the model adjusts the use of network resources. If so, the system proceeds to box 758 and transmits the currently updated global weights and / or the currently updated global model itself to multiple client devices. In response to a request from a given client device, such as a request during the update process and / or a request sent due to client device idleness and / or billing, the updated weights and / or the updated global model can optionally be transmitted to the given client device.

[0112] Turn now Figure 8This document depicts a block diagram of an example computing device 810 that can be optionally used to perform one or more aspects of the techniques described herein. In some embodiments, one or more of a client device, a cloud-based automation assistant component, and / or other components may include one or more components of the example computing device 810.

[0113] Computing device 810 typically includes at least one processor 814, which communicates with a number of peripheral devices via a bus subsystem 812. These peripheral devices may include a storage subsystem 824 (including, for example, a memory subsystem 825 and a file storage subsystem 826), a user interface output device 820, a user interface input device 822, and a network interface subsystem 816. The input and output devices allow users to interact with computing device 810. The network interface subsystem 816 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0114] User interface input device 822 may include: a keyboard; pointing devices such as a mouse, trackball, touchpad, or graphics tablet; a scanner; a touchscreen integrated into a display; voice input devices such as a voice recognition system or microphone; and / or other types of input devices. Generally, the term "input device" is intended to include all possible types of devices and the manner in which information is input to computing device 810 or a communication network.

[0115] User interface output device 820 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating visual images. The display subsystem may also provide a non-visual display, such as via an audio output device. Generally, the term "output device" is used to encompass all possible types of devices and methods of outputting information from computing device 810 to a user or another machine or computing device.

[0116] Storage subsystem 824 stores the programming and data construction functions of some or all of the modules described herein. For example, storage subsystem 824 may include aspects of selecting and implementing the methods disclosed herein. Figure 1A , Figure 1B and Figure 2 The logic of the various components described in the text.

[0117] These software modules are typically operated by processor 814 alone or in combination with other processors. The memory 825 used in storage subsystem 824 can include multiple memories, including main random access memory (RAM) 830 for storing instructions and data during program execution and read-only memory (ROM) 832 in which fixed instructions are stored. File storage subsystem 826 provides persistent storage for program and data files and may include hard disk drives, floppy disk drives, and associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing the functionality of certain embodiments may be stored by file storage subsystem 826 in storage subsystem 824 or in other machines accessible through processor 814.

[0118] The bus subsystem 812 provides a mechanism for enabling the various components and subsystems of the computing device 810 to communicate with each other as intended. Although the bus subsystem 812 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0119] The computing device 810 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, Figure 8 The description of the computing device 810 depicted herein is intended only as a specific example for illustrating some embodiments. Many other configurations of the computing device 810 may have more... Figure 8 The computing device depicted in the image has more or fewer components.

[0120] In situations where the systems described herein collect or otherwise monitor personal information about users, or may utilize personal and / or monitored information, users may be given the opportunity to control whether programs or features collect user information (e.g., information about a user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from content servers. Additionally, some data may be processed in one or more ways before it is stored or used, resulting in the removal of personally identifiable information. For example, a user's identity may be processed to the point that the user's personally identifiable information cannot be determined, or the user's geographic location may be generalized to a location for which geographic location information (such as city, zip code, or state) has been obtained, making it impossible to determine the user's specific geographic location. Therefore, users may have control over how information about them is collected and / or used.

[0121] In some implementations, a method implemented by one or more processors is provided, and includes training an on-device text-to-speech (TTS) generator model based on multiple training instances. The on-device TTS generator model is part of a generative adversarial network (GAN) model locally stored on the client device, wherein the GAN model also includes an on-device TTS discriminator model. Training the TTS generator model includes training the on-device TTS generator model to generate synthetic speech audio data including the voice characteristics of a user on the client device. The method further includes: after training the on-device TTS generator model, identifying a text segment locally stored on the user's client device; processing the text segment using the trained on-device TTS generator model locally stored on the client device to generate additional synthetic speech audio data including synthetic speech corresponding to the text segment; processing the additional synthetic speech audio data using an on-device automatic speech recognition (ASR) model locally stored on the client device to generate a corresponding predicted ASR output; generating a gradient based on comparing the corresponding predicted ASR output with a ground fact output corresponding to the text segment; and updating the local weights of the on-device ASR model based on the generated gradient.

[0122] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features.

[0123] In some implementations, the multiple training instances can be generator training instances, and each of the multiple training instances includes a training instance input and a training instance output. The training instance input may include a given text segment stored locally on the client device, and the training instance output may include ground fact labels. Training an on-device TTS generator model stored locally on the client device based on a given training instance among the multiple training instances may include: processing the given text segment using the on-device TTS generator model to generate predicted synthetic speech audio data including predicted synthetic speech corresponding to the given text segment; processing the predicted synthetic speech audio data using an on-device TTS discriminator model to predict whether the predicted synthetic speech audio data corresponds to the actual spoken utterance of a user on the client device or the predicted synthetic speech generated by the on-device TTS generator model, and generating a loss based on the ground fact labels and based on the processing using the on-device TTS discriminator model. In some versions of these implementations, the training instance output may also include ground fact audio data, which includes the given spoken utterance of a user corresponding to the given text segment. Training an on-device TTS generator model locally stored on a client device based on a given training instance from a plurality of training instances may further include: comparing predicted synthetic speech audio data, including predicted synthetic speech, with base-fact audio data, including a given spoken utterance from a user; and generating an additional loss based on the comparison between the predicted synthetic speech audio data and the base-fact audio data. In some additional or alternative versions of those implementations, comparing the predicted synthetic speech audio data with the base-fact audio data includes: comparing the acoustic features of the base-fact audio data with the synthetic acoustic features of the predicted synthetic speech audio data. In some additional or alternative versions of those implementations, the method may further include updating the on-device TTS generator model based on one or more of the loss or other losses. Updating the on-device TTS generator model based on one or more of the loss or other losses may include one or more of the cross-device TTS generator model backpropagation loss or other losses.

[0124] In some implementations, the method may further include training an on-device TTS discriminator model based on a plurality of additional training instances prior to training an on-device TTS generator model. In some versions of those implementations, the plurality of additional training instances may be discriminator training instances, and each of the plurality of additional training instances may include an additional training instance input and an additional training instance output. The additional training instance input may include: given audio data comprising a given spoken utterance of a user on the client device, or synthesized speech audio data comprising synthesized speech generated by the on-device TTS generator model, and the training instance output may include a base fact label indicating whether the additional training instance input corresponds to the given audio data or the synthesized speech audio data. Training the on-device TTS discriminator model locally stored at the client device based on a given training instance from the plurality of additional training instances may include processing the given training instance input using the on-device TTS discriminator model to predict whether the given training instance input corresponds to actual spoken utterance of a user on the client device or synthesized speech generated by the on-device TTS generator model. A loss is generated based on the base fact label and based on the processing using the on-device TTS discriminator model, and the on-device TTS discriminator model is updated based on the loss.

[0125] In some implementations, the user's voice characteristics on the client device may include prosodic attributes of the user's voice, wherein the prosodic attributes of the user's voice include one or more of the following: intonation, pitch, stress, frequency, rhythm, and pauses.

[0126] In some implementations, the method may further include transmitting the generated gradients to the remote system via a network, without transmitting any of the following: additional text fragments, additional synthesized speech audio data, and corresponding predicted ASR outputs. The remote system may use the generated gradients and additional gradients from another client device to update the global weights of the global ASR model. In some versions of those implementations, the method may further include receiving the global ASR model or updated global weights at the client device and from the remote system. Receiving the global ASR model or updated global weights may occur after the remote system updates the global weights of the global ASR model based on the gradients and additional gradients. The method may further include, in response to receiving the global ASR model or updated global weights, replacing the on-device ASR model with the global ASR model or replacing the local weights of the on-device ASR model with the updated global weights of the global ASR model in the local storage of the client device.

[0127] In some implementations, a method implemented by one or more processors is provided, and includes: identifying a text segment locally stored on a user's client device; processing the text segment using a trained on-device text-to-speech (TTS) generator model locally stored on the client device to generate synthetic speech audio data including synthesized speech corresponding to the text segment; processing the synthetic speech audio data including synthesized speech corresponding to the text segment using a trained on-device TTS discriminator model locally stored on the client device to determine whether the synthesized speech corresponds to synthetic speech audio data generated by the trained on-device TTS generator model or spoken utterance of the user on the client device; generating a gradient based on the processing by the trained on-device TTS discriminator model; and updating the local weights of the on-device TTS model based on the generated gradient.

[0128] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features.

[0129] In some implementations, the method may further include transmitting the generated gradients to a remote system via a network, without transmitting any of the following: text segments and synthesized speech audio data. The remote system may use the generated gradients and additional gradients from other client devices to update the global weights of the global TTS model.

[0130] In some versions of those implementations, the method may further include identifying base fact audio data corresponding to a given spoken utterance of a user, text fragments being base fact text fragments used for the base fact audio data, and generating additional gradients based on a comparison of synthesized speech audio data and base fact audio data. Updating the local weights of the TTS model on the device may also be based on the generated additional gradients. Transmitting the generated gradients to a remote system via a network may further include transmitting the generated additional gradients without transmitting the base fact audio data. In some other versions of those implementations, the method may further include determining whether to transmit the generated additional gradients to the remote system. Determining whether to transmit the generated additional gradients to the remote system may include determining whether the base fact audio data used in generating the additional gradients captures one or more of the following: additional audio data corresponding to a portion of another user's utterance, or ambient noise that meets a noise threshold. The method may further include: in response to determining that the base fact audio data used in generating the additional gradients captures additional audio data corresponding to a portion of another user's utterance, or ambient noise that meets a noise threshold, suppressing the transmission of the generated additional gradients to the remote system to update the global weights of the global TTS model. In some other or alternative versions of those implementations, the method may further include determining whether the basic fact audio data used in generating the additional gradient captures one or more of the following: additional audio data corresponding to a portion of another user's utterance, or ambient noise that meets a noise threshold; and in response to determining that the basic fact audio data used in generating the additional gradient captures additional audio data corresponding to a portion of another user's utterance or ambient noise that meets a noise threshold, and prior to generating the gradient: filtering the basic fact audio data to remove additional audio data corresponding to a portion of another user's utterance or ambient noise.

[0131] In some additional or alternative versions of those implementations, the method may further include receiving an updated global TTS model or updated global weights at the client device and from a remote system. Receiving the global TTS model or updated global weights follows when the remote system updates the global weights of the global TTS model based on gradients and additional gradients. The method may also include, in response to receiving the global TTS model or updated global weights, replacing the on-device TTS model with the global TTS model or replacing the local weights of the on-device TTS model with the updated global weights in local storage on the client device.

[0132] In some other or alternative versions of those implementations, the global TTS model may be one of several different global TTS models corresponding to several different languages, and the global TTS model among the several different TTS models may correspond to a given language associated with the user of the client device in several different languages.

[0133] In some other or alternative versions of those implementations, the global TTS model may be one of a plurality of distinct global TTS models corresponding to a plurality of different geographic regions, and the global TTS model among the plurality of distinct TTS models may correspond to a given geographic region associated with a user of a client device in a plurality of different geographic regions.

[0134] In some implementations, the method further includes: generating a plurality of semantically distinct alternative text segments based on the text segments; processing a given alternative text segment from the plurality of alternative text segments using an on-device TTS generator model locally stored on the client device to generate alternative synthesized speech audio data including alternative synthesized speech corresponding to the given alternative text segments; identifying additional base fact audio data corresponding to alternative spoken utterances of the user on the client device, wherein the alternative text segments are base fact alternative text segments used for the additional base fact audio data; comparing the synthesized speech audio data with the base fact audio data; generating additional gradients based on the comparison of the alternative synthesized speech audio data with the additional base fact audio data; and further updating one or more local weights of the on-device TTS generator model based on the generated additional gradients.

[0135] In some implementations, a method implemented by one or more processors is provided, and includes: identifying a text segment locally stored on a user's client device; processing the text segment using a trained on-device text-to-speech (TTS) generator model locally stored on the client device to generate synthetic speech audio data including synthesized speech corresponding to the text segment; processing the synthetic speech audio data including synthesized speech corresponding to the text segment using a trained on-device TTS discriminator model locally stored on the client device to determine whether the synthesized speech corresponds to synthetic speech audio data generated by the trained on-device TTS generator model or spoken speech by the user of the client device; generating a gradient based on the processing by the trained on-device TTS discriminator model; and transmitting the generated gradient over a network and to a remote system, wherein the remote system uses the generated gradient and additional gradients from another client device to update the global weights of a global TTS model.

[0136] Furthermore, some implementations include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to cause the execution of any of the foregoing methods. Some implementations also include one or more non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any of the foregoing methods. Some implementations also include a computer program product comprising instructions executable by one or more processors to perform any of the foregoing methods.

[0137] It should be understood that all combinations of the foregoing concepts and other concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein.

Claims

1. A method implemented by one or more processors, the method comprising: A text-to-speech (TTS) generator model on the device is trained based on multiple training instances. This on-device TTS generator model is part of a generative adversarial network (GAN) model stored locally on the client device. The GAN model also includes an on-device TTS discriminator model. Training the TTS generator model involves training it to generate synthesized speech audio data including the voice characteristics of the user on the client device. The training includes jointly training the on-device TTS generator model and the on-device TTS discriminator model using a GAN scheme. After training the TTS generator model on the device: Identify text fragments stored locally on the user's client device; The text segment is processed using a trained on-device TTS generator model stored locally on the client device to generate additional synthetic speech audio data including the synthetic speech corresponding to the text segment; The additional synthesized speech audio data is processed using an on-device automatic speech recognition (ASR) model stored locally on the client device to generate a corresponding predicted ASR output; Gradients are generated by comparing the corresponding predicted ASR output with the basic fact output corresponding to the text segment; and The local weights of the ASR model on the device are updated based on the generated gradients.

2. The method according to claim 1, in, The plurality of training instances are generator training instances; Each of the plurality of training instances includes a training instance input and a training instance output. The training instance input includes a given text fragment stored locally on the client device, and The training instance output includes basic fact labels; and Training a TTS generator model locally stored on the client device based on a given training instance from the plurality of training instances includes: The given text segment is processed using a TTS generator model on the device to generate predicted synthetic speech audio data, including predicted synthetic speech corresponding to the given text segment. The predicted synthesized speech audio data is processed using a TTS discriminator model on the device to predict whether the predicted synthesized speech audio data corresponds to the user's actual spoken words on the client device or the predicted synthesized speech generated by a TTS generator model on the device; and The loss is generated based on the stated fact labels and processing using a TTS discriminator model on the device.

3. The method according to claim 2, in, The training instance output also includes basic factual audio data, which includes the user's given spoken utterances corresponding to the given text segment; and Training the TTS generator model stored locally on the client device based on a given training instance from the plurality of training instances further includes: The predicted synthesized speech audio data, including the predicted synthesized speech, is compared with the factual audio data, including the user's given spoken utterance; and An additional loss is generated by comparing the predicted synthesized speech audio data with the base fact audio data.

4. The method according to claim 3, wherein, Comparing the predicted synthesized speech audio data with the basic fact audio data includes: comparing the acoustic features of the basic fact audio data with the synthetic acoustic features of the predicted synthesized speech audio data.

5. The method according to claim 3, further comprising: Update the TTS generator model on the device based on the loss or one or more of the other losses; as well as Updating the TTS generator model on the device based on the loss or one or more of the other losses includes backpropagating the loss or one or more of the other losses across the TTS generator model on the device.

6. The method according to claim 1, further comprising: Before training the TTS generator model on the device: The TTS discriminator model on the device is trained based on multiple additional training instances.

7. The method according to claim 6, in, The additional training instances are discriminator training instances; Each of the plurality of additional training instances includes an additional training instance input and an additional training instance output. The additional training instance inputs include: given audio data comprising a given spoken utterance of the user on the client device, or synthesized speech audio data comprising synthesized speech generated by a TTS generator model on the device, and The training instance output includes basic fact labels indicating whether the input of the additional training instance corresponds to the given audio data or the synthesized speech audio data; and Training a TTS discriminator model locally stored on the client device based on a given training instance from among the plurality of additional training instances includes: The given training instance input is processed using the TTS discriminator model on the device to predict whether the given training instance input corresponds to the actual spoken speech of the user on the client device or the synthesized speech generated by the TTS generator model on the device. Loss is generated based on the aforementioned basic fact labels and by processing using a TTS discriminator model on the aforementioned device; and The TTS discriminator model on the device is updated based on the loss.

8. The method according to claim 1, wherein, The user's voice characteristics on the client device include the prosodic attributes of the user's voice, wherein the prosodic attributes of the user's voice include one or more of the following: intonation, pitch, stress, frequency, rhythm, and pauses.

9. The method according to claim 1, further comprising: The generated gradient is transmitted to the remote system via a network, without transmitting any of the following: additional text fragments, the additional synthesized speech audio data, and the corresponding predicted ASR output. The remote system uses the generated gradients and additional gradients from other client devices to update the global weights of the global ASR model.

10. The method of claim 9, further comprising: At the client device and from the remote system, the global ASR model or the updated global weights are received, wherein the receipt of the global ASR model or the updated global weights occurs after the remote system updates the global weights of the global ASR model based on the gradient and the additional gradient; and In response to receiving the global ASR model or the updated global weights, the local storage device of the client device replaces the ASR model on the device with the global ASR model or replaces the local weights of the ASR model on the device with the updated global weights of the global ASR model.

11. A method implemented by one or more processors of a client device, the method comprising: Identify text fragments stored locally on the user's client device; The text segment is processed using a trained on-device text-to-speech (TTS) generator model stored locally on the client device to generate synthetic speech audio data that includes synthetic speech corresponding to the text segment. The synthesized speech audio data, which includes synthesized speech corresponding to the text segment, is processed using a trained on-device TTS discriminator model stored locally on the client device to determine whether the synthesized speech corresponds to synthesized speech audio data generated by a trained on-device TTS generator model or the spoken words of the user on the client device. Gradients are generated based on processing by a trained on-device TTS discriminator model; as well as The local weights of the TTS model on the device are updated based on the generated gradients.

12. The method of claim 11, further comprising: The generated gradient is transmitted to the remote system via a network, without transmitting any of the following: the text fragment and the synthesized speech audio data. The remote system uses the generated gradients and additional gradients from other client devices to update the global weights of the global TTS model.

13. The method of claim 12, further comprising: Identify basic fact audio data corresponding to a given spoken utterance of the user, wherein the text segment is a basic fact text segment used for the basic fact audio data; Another gradient is generated by comparing the synthesized speech audio data with the basic fact audio data; The local weights of the TTS model on the device are also updated based on additional gradients generated therefrom; and The generated gradient transmitted to the remote system via the network also includes transmitting additional generated gradients without transmitting the basic fact audio data.

14. The method of claim 13, further comprising: Determining whether to transmit the generated additional gradient to the remote system, wherein determining whether to transmit the generated additional gradient to the remote system includes: Determine whether the basic fact audio data used in generating the additional gradient captures one or more of the following: additional audio data corresponding to portions of another user's utterance, or ambient noise that meets a noise threshold; and In response to determining the additional audio data captured in generating the additional gradient, corresponding to a portion of the additional utterance of the additional user, or the ambient noise satisfying the noise threshold: Suppress the transmission of additional gradients generated to the remote system to update the global weights of the global TTS model.

15. The method of claim 13, further comprising: Determine whether the basic fact audio data used in generating the additional gradient captures one or more of the following: additional audio data corresponding to portions of another user's utterance, or ambient noise that meets a noise threshold; and In response to determining the additional audio data corresponding to the portion of the additional utterance of the additional user or the ambient noise satisfying the noise threshold, which is used when generating the additional gradient, and prior to generating the gradient: The basic fact audio data is filtered to remove additional audio data or ambient noise corresponding to the portion of the additional utterance of the other user.

16. The method of claim 12, further comprising: At the client device and from the remote system, the updated global TTS model or updated global weights are received, wherein receiving the global TTS model or updated global weights occurs after the remote system updates the global weights of the global TTS model based on the gradient and the additional gradient; and In response to receiving the global TTS model or the updated global weights, the local storage device of the client device replaces the TTS model on the device with the global TTS model or replaces the local weights of the TTS model on the device with the updated global weights.

17. The method according to claim 12, wherein, The global TTS model is one of a plurality of distinct global TTS models corresponding to a plurality of different languages, wherein the global TTS model among the plurality of distinct TTS models corresponds to a given language associated with the user of the client device among the plurality of different languages.

18. The method according to claim 12, wherein, The global TTS model is one of a plurality of distinct global TTS models corresponding to a plurality of different geographical regions, wherein the global TTS model among the plurality of distinct TTS models corresponds to a given geographical region associated with the user of the client device among the plurality of different geographical regions.

19. The method of claim 11, further comprising: Generate multiple alternative text fragments that are semantically different from the text fragment based on the text fragment; The TTS generator model stored locally on the client device is used to process a given replacement fragment among the plurality of replacement text fragments to generate replacement synthesized speech audio data including replacement synthesized speech corresponding to the given replacement text fragment; Identify additional basic fact audio data corresponding to the alternative spoken utterance of the user on the client device, wherein the alternative text fragment is a basic fact alternative text fragment used for the additional basic fact audio data; The synthesized speech audio data is compared with the basic fact audio data; Another gradient is generated by comparing the replaced synthetic speech audio data with the additional basic fact audio data; and One or more local weights of the TTS generator model on the device are further updated based on the additional gradients generated.

20. A method implemented by one or more processors of a client device, the method comprising: Identify text fragments stored locally on the user's client device; The text segment is processed using a trained on-device text-to-speech (TTS) generator model stored locally on the client device to generate synthetic speech audio data that includes synthetic speech corresponding to the text segment. The synthesized speech audio data, which includes synthesized speech corresponding to the text segment, is processed using a trained on-device TTS discriminator model stored locally on the client device to determine whether the synthesized speech corresponds to the synthesized speech audio data generated by the trained on-device TTS generator model or the spoken words of the user on the client device. Gradients are generated based on processing by a trained on-device TTS discriminator model; as well as The generated gradients are transmitted over a network to a remote system, which uses the generated gradients and additional gradients from another client device to update the global weights of the global TTS model.

Images