Transient learning and / or federated learning from audio-based machine learning model of audio data stream generated via radio station

Abstract

Implementations disclosed herein relate to utilizing transient learning techniques and / or federated learning techniques to update an audio-based machine learning (ML) model based on processing audio data streams generated via radio stations around the world. This enables an audio-based ML model to learn languages representing and / or understanding around the world, including a tail language without audio data / with little audio data. In various implementations, one or more deduplication techniques may be utilized to ensure that the same audio data stream is not overutilized when updating the audio-based ML model. In various implementations, a given client device may determine whether to employ a transient learning technique or a federated learning technique based on, for example, a state of connection with a remote system. In general, an audio data stream is received at a client device, but transient learning techniques may be implemented at the client device and / or a remote system.

Description

Background Art

[0001] Federated learning of machine learning (ML) models is an increasingly popular ML technique for updating ML models. In traditional federated learning, an on-device ML model is stored locally on a user's client device, and a global ML model, which is a cloud-based counterpart to the on-device ML model, is stored remotely at a remote system (e.g., a server cluster). The client device uses the on-device ML model to process user data detected at the client device to generate a predicted output, and the predicted output can be compared with the ground truth output to generate a client gradient. Further, the client device can transmit the client gradient to the remote system. The remote system can update the weights of the global ML model using the client gradient, and optionally using additional client gradients generated in a similar manner at additional client devices. The remote system can transmit the global ML model or the updated weights of the global ML model to the client device. The client device can then replace the on-device ML model with the global ML model, or replace the weights of the on-device ML model with the updated weights of the global ML model, thereby updating the on-device ML model.

[0002] Transient learning of ML models is another increasingly popular ML technique for updating ML models. In traditional transient learning, the on-device ML model is also stored locally on the user's client device, and the global ML model is also stored remotely at a remote system. However, compared to traditional federated learning, there is a temporal component in traditional transient learning. For example, user data may be transmitted to a remote system to cause certain performance to be performed. When the remote system processes the user data to cause certain performance to be performed, the remote system may also process the user data received from the client device using the global ML model to generate a predicted output, and may generate a remote gradient (e.g., using a self-supervised or unsupervised learning technique) based at least on the predicted output, and discard the user data without storing the user data in a non-transient storage of the remote system. This enables the remote system to update the global ML model based on the remote gradient, while the received user data is temporarily available at the remote system in a privacy-sensitive manner. In addition, for example, the user data may be processed locally at the client device to generate a client gradient in the same or similar manner as described above with respect to federated learning, but the client gradient is transmitted to the remote system so that it is only available for use when updating the global ML model in a transient manner.

[0003] However, scenarios for utilizing these different ML techniques to update ML models are typically limited to updating ML models based on explicit user input provided by users at their respective client devices. Therefore, updating ML models using these ML techniques may take a long time. In addition, user data utilized in these different ML techniques is typically limited to a small subset of spoken utterances and / or commands provided by users for which these ML models are utilized (e.g., "Assistant, set an alarm for 6:00 AM"), and is limited only to languages ​​for which ML models are common or well-defined (e.g., English, Spanish, etc.), rather than tail languages. Therefore, using these ML techniques to update ML models may simply strengthen the ML models to process and / or understand well-known spoken utterances and / or commands using well-known languages. Therefore, there is a need in the art to expand the utility of using these different ML techniques beyond explicit user input and to broaden the diversity of data utilized by these different ML techniques. Summary of the invention

[0004] Implementations described herein involve utilizing various privacy-sensitive machine learning (ML) techniques to update a global ML model based on processing audio data from radio stations around the world. In some implementations, a client device may receive an audio data stream capturing a spoken speech stream of a given language from a given radio station. The client device may process the audio data stream using an on-device ML model stored in an on-device storage of the client device and as an on-device counterpart to the global ML model, and may generate client gradients using unsupervised or semi-supervised learning techniques and based on processing the audio data stream using the on-device ML model. In some versions of those implementations, in accordance with the ephemeral learning ML technique for updating the global ML model, the client device may synchronously transmit the client gradients to a remote system (e.g., a high-performance server or a high-performance server cluster) to cause the remote system to update the global ML model based on the client gradients and with respect to the given language and optionally based on additional client gradients generated in the same or similar manner by additional client devices. In additional or alternative versions of those implementations, in accordance with a federated learning ML technique for updating a global ML model, a client device may asynchronously transmit client gradients to a remote system so that the remote system updates the global ML model based on the client gradients and with respect to a given language and optionally based on additional client gradients generated in the same or similar manner by additional client devices. In additional or alternative implementations, a client device (or another computing device) may transmit an audio data stream directly to a remote system. The remote system may process the audio data stream using a global ML model, and may generate remote gradients using unsupervised or semi-supervised learning techniques and based on processing the audio data stream using the global ML model. In these implementations, the remote system may update the global ML model based on the remote gradients and with respect to a given language and optionally based on other remote gradients generated by the remote system in the same or similar manner.

[0005] Thus, the techniques described herein may utilize a combination of ephemeral learning techniques and federated learning techniques to update a global ML model based on audio data streams from different radio stations broadcasting audio data streams in different languages ​​to extend the usefulness of using these different ML techniques beyond explicit user input and broaden the diversity of data utilized by these different ML techniques.

[0006] For example, it is assumed that a given user of a given client device is located in South Africa, a country located on the African continent that recognizes 11 official languages ​​with many diverse dialects. It is further assumed that, among the 11 official languages ​​of South Africa, English is the only language that the global multilingual ASR ML model is trained to recognize among the 11 official languages ​​of South Africa, although English is not the most commonly used language in South Africa. Therefore, the global multilingual ASR model in this example may not be useful for most South African groups. Although publicly available audio and / or video data repositories may include audio data streams that capture the other 10 official languages ​​of South Africa (as well as other unofficial languages ​​of South Africa), these audio data streams of the other 10 languages ​​(and other unofficial languages) may not be enough to train the global multilingual ASR model, because the number of these audio data streams is very limited, because most of the other 10 languages ​​(and other languages) may be regarded as tail languages, which are not popular in other parts of the world and / or are only used by a very small number of people in the world, such as the languages ​​of South African indigenous tribes that use click consonants. However, further assume that there are hundreds of radio stations in South Africa that play audio data streams in all of these different languages ​​in South Africa. Therefore, by using a combination of the ephemeral learning techniques and the federated learning techniques described herein to update the global multilingual ASR ML model based on the audio data streams in all of these different languages ​​from these hundreds of radio stations, the global multilingual ASR ML model can be trained to recognize all of these languages.

[0007] Although the above example is described with respect to a global ML model as a global multilingual ASR model, it should be understood that this is for example and is not intended to be limiting. Instead, it should be understood that the techniques described herein can be utilized when updating any global ML model that processes audio data (also referred to herein as an "audio-based ML model"), processes features generated based on processing audio data, and / or generates synthesized audio data. For example, the techniques described herein can be utilized to update: a global language representation ML model that extracts features from audio data to generate a rich feature representation of spoken speech captured in the audio data; a global voice activity detection ML model that detects voice activity in audio data; a global language identification ML model that detects a given language spoken in audio data; a global natural language understanding (NLU) ML model that processes features generated based on processing audio data to parse audio data; a global text-to-speech (TTS) ML model that generates synthesized speech audio data in a given language; and / or any other global ML model described herein.

[0008] In various implementations, a client device may determine whether to implement a transient learning technique or a federated learning technique to generate client gradients for updating a global ML model with respect to a given language. In some versions of these implementations, the client device may make such a determination based on a connection state between the client device and a remote system. For example, in response to determining that the connection state indicates that there is a strong or stable connection between the client device and the remote system (e.g., via a Wi-Fi network, a cellular network, etc.), the client device may implement a transient learning technique to generate client gradients locally at the client device and transmit the client gradients to the remote system in a synchronized manner. Additionally or alternatively, the client device may transmit an audio data stream directly to the remote system in a synchronized manner so that the remote system generates a remote gradient. In these instances, the audio data stream may be discarded by the client device after generating the client gradient and / or by the remote system after generating the remote gradient, so that the audio data stream is not stored in a non-transitory memory of the client device or the remote system. In contrast, in response to determining that the connection status indicates that there is a weak or unstable connection between the client device and the remote system, the client device may implement a federated learning technique to generate client gradients locally at the client device and transmit the client gradients to the remote system in an asynchronous manner when the client device subsequently establishes a strong or stable connection with the remote system.

[0009] In additional or alternative versions of those implementations, the client device may make such a determination based on a current location of the client device (e.g., determined using a GPS sensor of the client device). For example, in response to determining that the current location of the client device corresponds to a first location or a first geographic area, the client device may implement a transient learning technique to generate a client gradient locally at the client device and transmit the client gradient to the remote system in a synchronous manner. Additionally or alternatively, the client device may transmit the audio data stream directly to the remote system in a synchronous manner so that the remote system generates the remote gradient. In these instances, the audio data stream may be discarded by the client device after generating the client gradient and / or discarded by the remote system after generating the remote gradient, so that the audio data stream is not stored in a non-transitory memory of the client device or the remote system. In contrast, in response to determining that the current location of the client device corresponds to a second location or a second geographic area, the client device may implement a federated learning technique to generate a client gradient locally at the client device and transmit the client gradient to the remote system in an asynchronous manner when the client device subsequently establishes a strong or stable connection with the remote system.

[0010] In various implementations, the client device and / or the remote system may utilize various language identification techniques to identify a given language of a spoken speech stream captured in an audio data stream. In these implementations, the client device and / or the remote system may implement transient learning techniques and / or federated learning techniques only in response to determining that a given language corresponds to a target language from among a plurality of target languages. Continuing the above example in which a given user of a given client device is located in South Africa, it is further assumed that the audio data stream captures commercials from a given radio station in South Africa. The client device and / or the remote system may process the audio data stream using a language identification ML model to identify a given language spoken during a commercial from a given radio station. In this example, in response to determining that a given language spoken during a commercial from a given radio station corresponds to English, the client device and / or the remote system may avoid implementing transient learning techniques and / or federated learning, assuming that English is not one of the plurality of target languages ​​(i.e., because English is not considered a tail language). In contrast, in response to determining that a given language spoken during a commercial from a given radio station corresponds to Swazi, the client device and / or remote system can implement transient learning techniques and / or federated learning, assuming that Swazi is one of multiple target languages ​​(i.e., because Swazi can be considered a tail language). Accordingly, the client device and / or remote system can utilize various language identification techniques to determine which audio data streams capture spoken speech streams of a given target language that is of interest for generating client gradients and / or remote gradients for updating the global ML model with respect to the given target language.

[0011] In various implementations, the client device and / or the remote system may utilize various deduplication techniques to determine whether the audio data stream has previously been utilized in generating client gradients and / or remote gradients for updating a global ML model for a given language based on the audio fingerprint. In these implementations, the client device and / or the remote system may implement transient learning techniques and / or federated learning techniques only in response to determining that the audio data stream has not previously been utilized in generating client gradients and / or remote gradients for updating a global ML model. For example, the client device and / or the remote system may process the audio data stream using an encoder-decoder ML model or another ML model to generate an embedding (or other lower dimensional representation of the audio data) as an audio fingerprint of the audio data stream. The embedding may be mapped to an embedding space (or another lower dimensional latent space) that enables the embedding to be compared with multiple previously generated embeddings for previously encountered audio data streams. In these instances, if the embedding generated based on the audio data stream matches a given previously generated embedding for a given previously encountered audio data stream, the audio data stream may be discarded without implementing transient learning techniques and / or federated learning techniques. In these instances, if the embedding and a given previously generated embedding are within a threshold distance in the embedding space (e.g., using Euclidean distance, cosine similarity, and / or another distance metric), the audio data stream can be discarded without implementing the ephemeral learning technique and / or the federated learning technique. Otherwise, the client device and / or the remote system can implement the ephemeral learning technique and / or the federated learning technique to generate client gradients and / or remote gradients based on the audio data stream.

[0012] In addition, for example, the client device and / or the remote system can process the audio data stream using local sensitivity hashing to generate an audio hash as an audio fingerprint of the audio data stream. In these instances, if the audio hash generated based on the audio data stream matches a given previously generated audio hash for a given previously encountered audio data stream (e.g., based on determining that the audio hash matches the given previously generated audio hash), the audio data stream can be discarded without implementing the ephemeral learning technique and / or the federated learning technique. Otherwise, the client device and / or the remote system can implement the ephemeral learning technique and / or the federated learning technique to generate a client gradient and / or a remote gradient based on the audio data stream.

[0013] In these implementations, the client device (and other client devices) may transmit the audio fingerprint to a remote system to enable the remote system to generate an audio fingerprint database. The audio fingerprint database may be distributed from the remote system to the client device (and other client devices) to enable the client device (and other client devices) to employ these deduplication techniques. Thus, as the client device population generates these audio fingerprints, the remote system may maintain the database and periodically distribute the audio fingerprint database to ensure that the client device population avoids performing deduplication and / or undesirable processing on a particular audio data stream that is encountered multiple times.

[0014] Continuing with the above example where a given user of a given client device is located in South Africa, assume again that the audio data stream captures commercials from a given radio station in South Africa. Assume further that the commercials from the given radio station in South Africa are played every hour over a period of one week. By using the de-duplication techniques described herein, the client device and / or remote system ensures that client gradients and / or remote gradients are not generated based on an audio data stream that captures commercials played over a period of one week every hour. Thus, these de-duplication techniques not only save computing and / or network resources by avoiding implementing transient learning techniques and / or federated learning techniques in every instance of encountering a commercial, but these de-duplication techniques also ensure that the global ML model is not overfitted to the commercials, thereby producing a more robust global ML model in terms of accuracy and / or precision. Conversely, these de-duplication techniques enable the client device and / or remote system to limit the number of instances of updating the global ML model based on client gradients and / or remote gradients generated based on multiple audio data streams that capture the same spoken speech stream.

[0015] In various implementations, the client device and / or the remote system may utilize unsupervised or self-supervised learning techniques, at least in part due to the lack of a supervisory signal from the audio data stream of the radio station. In some versions of those implementations, a non-limiting example of an unsupervised or self-supervised learning technique that may be utilized to generate client gradients and / or remote gradients corresponds to a teacher-student technique. For example, the client device and / or the remote system may process the audio data stream to generate a predicted output (e.g., using an on-device ML model and / or a global ML, respectively). Further, the predicted output may be compared with a corresponding benchmark output generated based on a benchmark ML model that is also utilized to process the audio data stream. In this instance, the benchmark ML model may be of the same type as the on-device ML model utilized by the client device and the global ML model utilized by the remote system, and the benchmark output may be utilized as a quasi-supervisory signal for generating client gradients and / or remote gradients, respectively. In an instance in which the client device implements a transient learning technique to generate a client gradient based on processing using an on-device ML model, the benchmark ML model may correspond to an on-device ML model, a global ML model, or a separate dedicated benchmark ML model. In instances where the client device implements federated learning techniques to generate client gradients based on processing using an on-device ML model, the baseline ML model may correspond to the on-device ML model or a separate dedicated baseline ML model. In instances where the remote system implements federated learning techniques to generate remote gradients based on processing using a global ML model, the baseline ML model may correspond to the on-device ML model, the global ML model, or a separate dedicated baseline ML model.

[0016] In some further versions of those implementations, the predicted output may be utilized in generating client gradients and / or remote gradients for updating the global ML model only in response to determining that one or more conditions are satisfied. The one or more conditions may include, for example, whether the predicted output satisfies a predicted output threshold, whether the baseline output satisfies a baseline output threshold, and / or other conditions. In other words, the predicted output may be utilized in generating client gradients and / or remote gradients for updating the global ML model only in response to determining that the baseline output provides sufficient quasi-supervisory signal for updating the global ML model.

[0017] In other versions of those implementations, a non-limiting example of an unsupervised or self-supervised learning technique that can be utilized to generate client gradients and / or remote gradients corresponds to a masking technique. For example, a target portion of an audio data stream can be identified. The target portion of the audio data stream can be after the leading portion of the audio data stream, and can be before the trailing portion of the audio data stream. Further, various techniques can be used to mask the target portion of the audio data stream. The target portion of the audio data stream can be selected arbitrarily, or based on one or more criteria, such as a specific segment between n and m seconds of audio data corresponding to the target portion, and / or any other criteria for selecting the target portion of the client data. In this example, the target portion of the audio data stream can correspond to a target audio waveform portion of the audio data stream, the leading portion of the audio data stream can correspond to a leading audio waveform portion received before the target audio waveform portion, and the trailing portion of the audio data stream can correspond to a trailing audio waveform portion received after the target audio waveform portion.

[0018] In those implementations, the on-device ML model and / or the global ML model may be used to process the leading portion of the audio data stream and the trailing portion of the audio data stream to generate a predicted output of a target portion of the audio data stream. For example, in an implementation in which the target portion of the client data corresponds to a target audio waveform portion of the audio data stream, the on-device ML model and / or the global ML model may be used to process the leading audio waveform portion and the trailing audio waveform portion to generate a predicted target audio waveform predicted to correspond to the target audio waveform portion. Further, the predicted target audio waveform may be compared to the masked target audio waveform to generate a client gradient and / or a remote gradient. Thus, the on-device ML model may attempt to reconstruct the target audio waveform portion based on processing the leading audio waveform portion and the trailing audio waveform portion. Notably, this technique may be particularly advantageous when the on-device ML model and the global ML model correspond to a multilingual ASR model because the language may be irrelevant to reconstructing the target audio waveform portion.

[0019] In various implementations implementing the ephemeral learning techniques, the audio data stream may be routed through a buffer (e.g., a buffer of a client device and / or a buffer of a remote system) prior to processing the audio data stream. In some versions of those implementations, the buffer may be utilized to tag the audio data stream with various labels based on a given language of a spoken utterance stream included in the audio data stream (e.g., determined using the language identification techniques described herein), based on whether the audio data stream includes a spoken utterance stream on which a global ML model was previously updated (e.g., determined using the deduplication techniques described herein), based on a name associated with a radio station (e.g., a name of an Internet radio station), based on an amplitude modulation (AM) radio band associated with the audio data stream, based on a frequency modulation (FM) radio band associated with the audio data stream, based on a current location of the client device when the audio data stream is received, and / or based on a geographic region of the client device when the audio data stream is received. In these implementations, if the client device and / or the remote system determines that the audio data stream should not be processed, such as when the audio data stream captures a spoken speech stream in a language that is not of interest, when the audio data stream captures a spoken speech stream that was previously utilized in generating gradients, and / or in other circumstances, the audio data stream can be discarded from the buffer and the client device and / or the remote system can refrain from performing any additional processing on the audio data stream.

[0020] As described herein, various architectures can be utilized to enable a client device and / or a remote system to receive an audio data stream. In some implementations, the audio data stream can be generated by an Internet radio station. In some versions of those implementations, the audio data stream can be received at the client device through one or more networks (e.g., the Internet), and can be processed locally at the client device using transient learning techniques and / or federated learning techniques to generate a client gradient that is transmitted to the remote system. In additional or alternative versions of those implementations, the audio data stream can be received at the client device through one or more networks (e.g., the Internet) and / or at another computing device that communicates with a remote system (e.g., one or more other servers), and can be transmitted to the remote system so that the remote system generates a remote gradient. In additional or alternative implementations, the audio data stream can be generated by an AM or FM radio station. In some versions of those implementations, the audio data stream can be received at the client device via an internal transceiver of the client device and / or via an external transceiver of the client device (e.g., an external transceiver that can be connected to the client device via a mini-jack of the client device or via one or more networks (e.g., Bluetooth)). In these implementations, the client device can sweep the AM radio band and / or the FM radio band to receive the audio data stream, or be tuned to a specific AM radio band and / or the FM radio band. The client device and / or the remote system can implement the transient learning technique and / or the federated learning technique in the same or similar manner as described above to generate the client gradient and / or the remote gradient.

[0021] In various implementations, after updating the global ML model based at least on the client gradients and / or the remote gradients, the remote system may cause the updated global ML model (or updated global weights of the global ML model) to be distributed to the client device and / or additional client devices. In response to receiving the updated global ML model (or updated global weights of the global ML model), the client device and / or additional client devices may replace the on-device ML model with the updated global ML model (or replace the on-device weights of the on-device ML model with the updated global weights of the global ML model) in the on-device storage. In some versions of those implementations, the remote system may cause the updated global ML model (or updated global weights of the global ML model) to be distributed to the client device and / or additional client devices only in response to determining that one or more conditions are satisfied. The one or more conditions may include, for example, whether the current time at the current location of the client device corresponds to a specific time of day, whether the current day of the week at the current location of the client device corresponds to a specific day of the week, whether the global ML model has been updated based on a threshold number of gradients, whether the performance of the updated global ML model satisfies a performance threshold, and / or other criteria.

[0022] By using the techniques described herein, one or more technical advantages may be achieved. As a non-limiting example, the techniques described herein enable updating of a global ML model based on processing audio data streams available via radio stations that broadcast audio data streams in different languages. By enabling updating of a global ML model based on processing audio data streams available via radio stations, the global ML model can be updated based on diverse audio data streams in diverse languages ​​that are not otherwise available. Therefore, the global ML model may be more robust (e.g., in terms of precision and / or recall) and may be provided to a more diverse group of users. For example, the language identification techniques described herein enable updating of a global ML model with respect to specific tail languages ​​that are not well defined for most users or that are not well known to most users. In addition, for example, the de-duplication techniques described herein enable updating of a global ML model with respect to diverse audio data streams of these specific tail languages ​​that are not well defined for most users or that are not well known to most users.

[0023] The above description is provided as an overview of only some of the implementations disclosed herein. Those implementations and other implementations are described in more detail herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1A and Figure 1B Depicted are example process flows demonstrating various aspects of the present disclosure that may be implemented by client devices and remote systems and in which implementations disclosed herein may be implemented.

[0025] Figure 2 A block diagram depicts an example environment in which implementations disclosed herein may be implemented.

[0026] Figure 3 Depicted is a flow chart illustrating an example method of ephemeral learning implemented by a client device based on an audio stream generated by a given radio station, according to various implementations.

[0027] Figure 4 Depicted is a flow chart illustrating an example method of ephemeral learning implemented by a remote system based on an audio stream generated by a given radio station, according to various implementations.

[0028] Figure 5 Depicted is a flow chart illustrating an example method of deduplication techniques utilized in ephemeral learning implemented by a client device according to various implementations.

[0029] Figure 6 Depicted is a flow chart illustrating an example method of deduplication techniques utilized in ephemeral learning implemented by a remote system, according to various implementations.

[0030] Figure 7 Depicted is a flow chart illustrating an example method of determining whether a client device should implement an ephemeral learning technique or a federated learning technique according to various implementations.

[0031] Figure 8 Depicted are example architectures for computing devices in accordance with various implementations. DETAILED DESCRIPTION

[0032] Now turn to Figure 1A and Figure 1B , depicts an example process flow demonstrating various aspects of the present disclosure that may be implemented by client device 120 and remote system 160 . Figure 1A and Figure 1B The client device 120 shown may include at least Figure 1A and Figure 1B ' represents the components within the box of the client device 120. Similarly, Figure 1A and Figure 1B The remote system 150 shown may include at least Figure 1A and Figure 1B ' represents components within the box of remote system 150. As described herein, Figure 1A and Figure 1B The process flow can be utilized to implement ephemeral learning and / or federated learning of audio-based machine learning (ML) models based on audio data streams generated via radio stations around the world.

[0033] Specific reference Figure 1A , assume that an audio data stream 124 is received at a client device 120. The audio data stream 124 may be received via an internal transceiver 122A of the client device 120 (e.g., integral to the client device 120) or an external transceiver 122B coupled to the client device (e.g., via an audio jack of the client device 120 or by other means). Further, the audio data stream 124 may be generated by a variety of sources, such as a radio station transmitting the audio data stream at a different frequency (e.g., an amplitude modulation (AM) frequency, a frequency modulation (FM) frequency, etc.) via a radio tower 110A, an Internet radio source 110B, and / or other radio sources. In some implementations, the client device 120 may cause the audio data stream 124 to be buffered (e.g., in one or more buffers 132) while the client device 120 initially processes the audio data stream 124 to determine whether to implement ephemeral learning techniques and / or federated learning techniques on the audio data stream 124, and / or where to implement the ephemeral learning techniques and / or federated learning techniques (e.g., locally at the client device 120, such as Figure 1A As shown in the process flow of , or implemented at the remote system 160, as Figure 1B shown).

[0034] like Figure 1A As shown, the client device 120 may include various engines to perform various operations. For example, the client device 120 may include an on-device ML model engine 134, a learning engine 136, a gradient engine 138, a connection state engine 140, a language recognition engine 142, and a deduplication engine 144. Further, the remote system may include a remote update engine 180 and an update distribution engine 182. Although in Figure 1A The client device 120 and the remote system 160 are depicted as including specific engines that execute process flows in a specific manner, but it should be understood that this is to illustrate the various techniques described herein and is not intended to be limiting. For example, one may combine Figure 1A The various engines depicted in the examples may include other engines that may be added or omitted, and may be modified with respect to Figure 1A Describe the process flow (e.g., Figure 1B The process flow described in the example above is reordered.

[0035] In general, the on-device ML model engine 134 may process the audio data stream 124 using an audio-based on-device ML model that is stored in an on-device memory or storage of the client device 120 (e.g., stored in an on-device ML model database 120A) and is an on-device counterpart of an audio-based global ML model to be updated (e.g., stored in a global ML model database 160A). The one or more predicted outputs 134A generated by the on-device ML model engine 134 may be based on the type of the audio-based on-device ML model utilized when processing the audio data stream 124 (and the type of the audio-based global ML model to be updated based on the processing of the audio data stream 124). For example, assume that the on-device audio-based ML model and the global audio-based ML model are corresponding language representation ML models. In this example, the one or more predicted outputs 134A may include, for example, a rich feature representation of the audio data stream 124, such as a description of one or more sounds captured in the audio data stream 124, relationships between sounds captured in the audio data stream 124, and / or other features of the audio data stream 124. In contrast, assume that the audio-based on-device ML model and the audio-based global ML model are corresponding multilingual automatic speech recognition (ASR) models. In this example, the one or more predicted outputs 134A may include, for example, recognized text in a given language, such as one or more terms corresponding to a spoken utterance stream captured in the audio data stream 124. Although the above examples are described with respect to specific audio-based ML models, it should be understood that this is for example only and is not intended to be limiting, and with respect to Figure 2 Other non-limiting examples of audio-based ML models are described.

[0036] Further, the gradient engine 136 may generate a gradient 136A based on processing of the audio data stream 124 by the on-device ML model engine 134 and / or one or more predicted outputs 134A generated by the on-device ML model engine 134. The gradient 136A may be transmitted to the remote system 160 for use in updating an audio-based global ML model that is a global counterpart to the audio-based on-device ML model used in processing the audio data stream 124. In generating the gradient 136A, the gradient engine 136 may utilize the learning engine 138. In some implementations, the learning engine 138 may employ one or more supervised learning techniques, while in other implementations, the learning engine 138 may employ one or more unsupervised or semi-supervised learning techniques. However, in various implementations, since the audio data stream 124 is generated by a given radio station, an explicit supervisory signal for employing one or more of the supervised learning techniques may not be available. Thus, the techniques described herein are generally described with respect to one or more of unsupervised or semi-supervised learning techniques, although the learning engine 138 may additionally or alternatively utilize one or more of supervised learning techniques.

[0037] In some versions of those implementations, one or more of the unsupervised or semi-supervised learning techniques may correspond to a teacher-student technique. When implementing the teacher-student technique, the learning engine 138 may process the audio data stream 124 using one or more corresponding audio-based benchmark ML models to generate one or more benchmark outputs. In these implementations, the one or more corresponding audio-based benchmark ML models may be the same ML model as the audio-based on-device ML model and / or the audio-based global ML model, or another audio-based ML model of the same type as the audio-based on-device ML model and the audio-based global ML model (e.g., stored in the on-device ML model database 120A and / or the global ML model database 160A). Further, one or more benchmark outputs may be utilized as quasi-supervisory signals to be utilized when generating the gradient 136A. For example, the gradient engine 136 may compare one or more predicted outputs 134A with one or more benchmark outputs to generate the gradient 136A.

[0038] In some further versions of those implementations, one or more reference outputs may be utilized as quasi-supervisory signals only in response to determining that one or more conditions are satisfied. The one or more conditions may include, for example, whether one or more of the predicted outputs 134 satisfy a predicted output threshold, whether one or more of the reference outputs satisfy a reference output threshold, and / or other conditions. In other words, one or more reference outputs may be utilized as quasi-supervisory signals only in response to determining that one or more reference outputs provide a viable supervisory signal for one or more predicted outputs 134A.

[0039] For example, again assuming that the audio-based on-device ML model and the audio-based global ML model are corresponding multilingual automatic speech recognition (ASR) models. In this example, the on-device ML engine 134 can process the audio data stream 124 using the on-device multilingual ASR model to generate recognized text in a given language as one or more predicted outputs 134A. Further, the learning engine 138 can process the audio data stream 124 using the baseline multilingual ASR model to generate recognized baseline text as one or more baseline outputs. In this example, assuming that one or more conditions are met, the gradient engine 136 can compare the recognized text with the recognized baseline text to generate a gradient 136A.

[0040] In additional or alternative versions of those implementations, one or more of the unsupervised or semi-supervised learning techniques may correspond to a masking technique. When implementing the masking technique, the learning engine 138 may identify a target portion of the audio data stream 124 that is less than all of the audio data included in the audio data stream 124. Thus, the target portion of the audio data stream 124 may be proximate to a pre-portion of the audio data stream 124 that precedes the target portion of the audio data stream 124 and / or a post-portion of the audio data stream 124 that follows the target portion of the audio data stream 124. The target portion of the audio data stream 124 may be selected arbitrarily or based on one or more criteria, such as a specific segment between n and m seconds of the audio data stream 124 being identified as the target portion, and / or any other criteria for selecting the target portion of the audio data stream 124. Further, the learning engine 138 may mask the target portion of the audio data stream 124. Thus, the output 134A of the stream one or more predictions may include one or more predictions about the masked target portion of the audio data stream 124.

[0041] For example, again assuming that the audio-based on-device ML model and the audio-based global ML model are corresponding language representation ML models. In this example, the target portion of the audio data stream 124 may correspond to a target audio waveform portion of the audio data stream 124, the front portion of the audio data stream 124 may correspond to a front audio waveform portion before the target audio waveform portion, and the rear portion of the audio data stream 124 may correspond to a rear audio waveform portion after the target audio waveform portion. Further, the on-device ML engine 134 may process the front audio waveform portion before the target audio waveform portion and / or the rear audio waveform portion after the target audio waveform portion using the on-device language representation ML model to generate one or more predicted outputs 134A, such as a prediction of the target audio waveform portion. It is noteworthy that the prediction of the target audio waveform portion may include, for example, a predicted audio waveform of the target audio waveform, one or more predicted features of the target audio waveform (e.g., predicted amplitude, predicted wavelength, predicted phase, predicted period, and / or other features), one or more predicted features of the audio data stream 124 (e.g., predicted MFCC, predicted melbank features, and / or other features), and / or other predicted representations of the target portion of the audio data stream 124. In other words, the on-device ML model engine 134 may attempt to reconstruct the target audio waveform portion based on processing the leading audio waveform portion and / or the trailing audio waveform portion. In this example, the gradient engine 136 may compare the prediction made about the target portion of the audio data stream 124 with the actual features of the audio data stream 124 to generate a gradient 136A.

[0042] Although specific unsupervised or semi-supervised learning techniques are described herein, it should be understood that those techniques are provided for example and are not intended to be limiting. Instead, it should be understood that any unsupervised or semi-supervised learning technique that can be utilized when generating gradients based on processing the audio data stream 124 can be utilized and is contemplated herein.

[0043] In some implementations, the gradient 136A (and optionally one or more gradients 190A generated in the same or similar manner by one or more corresponding additional client devices 190, each of which includes the same or similar components and / or engines described with respect to the client device 120) can be generated in an ephemeral manner such that the audio data stream 124 is not stored in a transient memory or storage of the client device 120. In some versions of these implementations, the audio data stream 124 can be discarded after the gradient 136A is generated. Further, the gradient 136A can be transmitted to the remote system 160 in a synchronous manner (e.g., in response to generating the gradient 136A). In other words, the client device 120 can cause the audio data stream 124 (e.g., via one or more buffers 132) to be processed to generate the gradient 136A while it is temporarily available at the client device 120, and the gradient 136A is synchronously transmitted to the remote system 160 to reduce memory consumption at the client device 120 (hence referred to as "ephemeral learning" due to the ephemeral nature of this learning technique). In additional or alternative implementations, the client device 120 may update the on-device audio-based ML model utilized in generating the gradient 136A, and transmit one or more updated on-device weights of the updated on-device audio-based ML model to the remote system 160 in place of the gradient 136A.

[0044] In additional or alternative implementations, the gradient 136A (and optionally one or more gradients 190A generated in the same or similar manner by one or more corresponding additional client devices 190) may be generated in a federated manner such that the audio data stream 124 may not be stored in a temporary memory or storage of the client device 120. In some versions of these implementations, the audio data stream 124 may be discarded after the gradient 136A is generated. However, the gradient 136A may be stored in the memory or storage of the client device 120 and transmitted to the remote system 160 in an asynchronous manner (e.g., at a different time in time after the gradient 136A is generated). In other words, the client device 120 may have the audio data stream 124 processed to generate the gradient 136A, but wait for the gradient 136A to be transmitted to the remote system 160.

[0045] In these implementations, the client device 120 may determine to implement ephemeral learning or federated learning based on, for example, the connection state of the connection between the client device 120 and the remote system 160 (and optionally based on the connection strength of the connection). For example, the connection state engine 140 may determine whether the client device 120 is connected to the remote system 160 (e.g., via one or more networks, such as one or more local area networks, one or more wide area networks, and / or one or more other networks) and / or the strength of the connection between the client device 120 and the remote system 160 (e.g., via one or more of the networks). The connection state engine 140 may make such a determination when the audio data stream 124 is temporarily stored at the client device 120 (e.g., in one or more buffers 132). For example, in response to the connection state engine 140 determining that the client device 120 and the remote system 160 have a strong and / or stable connection on one or more of the networks, the client device 120 may implement ephemeral learning to generate and transmit the gradient 136A to the remote system 160 because the connection state enables the gradient 136A to be synchronously transmitted to the remote system 160. In contrast, in response to connection state engine 140 determining that client device 120 and remote system 160 have a weak and / or unstable connection on one or more of the networks, client device 120 may implement federated learning to generate gradient 136A and transmit it to remote system 160. Figure 7 ) describes these techniques in more detail.

[0046] In some implementations, the client device 120 may process the audio data stream 124 to generate the gradient 136A only in response to determining that a given language of the spoken utterance stream captured in the audio data stream 124 corresponds to a target language. For example, the language identification engine 142 may initially process the audio data stream 124 using one or more language identification models (e.g., stored in the on-device ML model database 120A) to identify the given language. Further, the language identification engine 142 may determine whether the given language corresponds to the target language (e.g., stored in the target language database 142A). It is worth noting that the target language can be one of a plurality of target languages ​​and can be defined, for example, by a developer to enable the audio-based global ML model to be updated with respect to the target language. In other words, some languages ​​(e.g., English, Spanish, French, German) may not be of interest because there is a large amount of data available to update the audio-based global ML model with respect to these languages. Thus, these techniques enable developers to target specific target languages ​​of interest (e.g., Swazi in South Africa) because there is little or no data available to update the audio-based global ML model with respect to these languages. This document (e.g., regarding Figure 5 ) describes these techniques in more detail.

[0047] In some implementations, the client device 120 may process the audio data stream 124 to generate the gradient 136A only in response to determining that the audio data stream 124 has not been previously utilized in generating gradients for updating the audio-based global ML model or has not been previously utilized in generating gradients for updating the audio-based global ML model for more than a threshold number of instances. For example, the deduplication engine 144 may initially process the audio data stream 124 to generate an audio fingerprint of the audio data stream 124. Further, the deduplication engine 144 may determine whether the audio fingerprint matches a previously generated audio fingerprint (e.g., stored in an audio fingerprint database 144A). Notably, the audio fingerprint may correspond to an embedding, an audio hash, and / or any other representation of the audio data stream 124 that enables the audio data stream 124 to be compared with other audio data streams. In other words, the deduplication engine 144 can be utilized to ensure that the audio-based global ML model is not constantly updated based on the same audio data stream (e.g., commercials on a given radio station), to prevent the audio-based global ML model from overfitting to the same audio data stream, and to ensure diversity in the underlying audio data streams utilized when generating gradients. Figure 5 ) describes these techniques in more detail.

[0048] In various implementations, assuming that the gradient 136A is transmitted to the remote system 160 (e.g., in a synchronous or asynchronous manner), and assuming that the one or more gradients 190A are generated in the same or similar manner by one or more corresponding additional client devices 190, the remote system 160 may use the remote update engine 180 to update the audio-based global ML model, and then use the update distribution engine 182 to distribute the updated audio-based ML model 182A to the client device 120 (and optionally, one or more of the additional client devices 190). In some versions of those implementations, the remote system 160 may update the audio-based global ML model in a streaming manner (e.g., when the gradient 136A and the one or more gradients 190A are received from the corresponding client device). In additional or alternative versions of those implementations, the remote system 160 may store the gradient 136A and the one or more gradients 190A in one or more databases (e.g., the gradient database 176B), and may update the audio-based global ML model in response to determining that one or more conditions for updating the audio-based global ML model are satisfied. The one or more conditions for updating the audio-based global ML model may include, for example, a specific time of day, a specific day of the week, whether a threshold number of gradients are available to update the audio-based global ML model, and / or other conditions.

[0049] When updating the audio-based global ML model, the remote system 160 may update one or more weights of the audio-based global ML model based on the gradient 136A and the one or more gradients 190A received from the corresponding client device. In some implementations, the remote update engine 180 may utilize a gradient descent algorithm to update one or more of the global weights. In some versions of those implementations, the remote update engine 180 may average the gradient 136A and the one or more gradients 190A received from the corresponding client device before updating one or more of the global weights using the gradient descent algorithm. In additional or alternative versions of those implementations, the remote update engine 180 may utilize each or a subset of the gradient 136A and the one or more gradients 190A received from the corresponding client device to update one or more of the global weights using a gradient descent algorithm.

[0050] When distributing the updated audio-based ML model 182A to the client device 120 (and optionally, one or more of the additional client devices 190), the update distribution engine 182 may determine whether one or more conditions for the updated audio-based ML model 182A (or one or more of its global weights) are satisfied. The one or more conditions may be based on whether the client device 120 is ready to receive the client device 120, such as whether the client device 120 is charging, whether the client device 120 has at least a threshold state of charge, whether the temperature of the client device 120 (based on one or more corresponding on-device temperature sensors) is below a threshold, whether the client device 120 is held by a user, a time condition associated with the client device 120 (e.g., between a specific time period, every N hours, where N is a positive integer, and / or other time conditions), and / or other conditions. Further, one or more conditions may additionally or alternatively be based on other conditions specific to the remote system 160, such as whether the performance of the updated audio-based ML model 182A satisfies a performance threshold, whether the updated audio-based ML model 182A has been updated based on a threshold number of gradients, and / or other conditions.

[0051] Thus, when the client device 120 (and optionally, one or more of the additional client devices 190) receives the updated audio-based ML model 182A (or one or more of its global weights), the on-device audio-based ML model (or one or more of its on-device weights) may be replaced with the updated audio-based ML model 182A (or one or more of its global weights). This process may be repeated to continuously update the audio-based global ML model. In some implementations, this process may be repeated to continuously update the audio-based global ML model until it is determined that the audio-based global ML model has converged.

[0052] Specific reference Figure 1B , again assuming that the audio data stream 124 is received at the client device 120. However, Figure 1B In and with Figure 1B In contrast, transient learning is performed by remote system 160 and is not performed locally at one or more of client device 120 and / or additional client device 190. Figure 1B In and with Figure 1B In contrast, client device 120 and / or one or more of additional client devices 190 are utilized as a proxy for the remote system to obtain audio data stream 124 from client device 120 and one or more audio data streams 190B from a corresponding one of additional client devices 190 .

[0053] It is worth noting that in Figure 1B In the example, the remote system 160 includes information about Figure 1A Many of the components and engines described have remote-based counterparts. For example, Figure 1BThe one or more buffers 172 of the remote system in the client device 120 can be utilized in the same or similar manner as described above with respect to the one or more buffers 132 of the client device 120; the global ML model engine 174 of the remote system 160 can be utilized to generate one or more predicted outputs 174A in the same or similar manner as described with respect to the on-device ML model engine 134 but using the audio-based global ML model stored in the global ML model database 160A; the gradient engine 176 of the remote system 160 can be utilized to generate gradients 176A at the remote system 160 in the same or similar manner as described with respect to the gradient engine 136 of the client device; the learning engine 178 of the remote system 160 can enable one or more supervised, unsupervised, or semi-supervised learning techniques to be utilized in accordance with The gradient 176A is generated in the same or similar manner as described with respect to the learning engine 138 of the client device 120; the language identification engine 184 of the remote system 160 can be utilized in the same or similar manner as described with respect to the language identification engine 142 of the client device 120, but utilizing remote memory or storage (e.g., target language database 184A) to determine whether a given language of the spoken utterance stream captured in the audio data stream 124 corresponds to the target language; and the deduplication engine 186 of the remote system 160 can be utilized in the same or similar manner as described with respect to the deduplication engine 144 of the client device, but utilizing remote memory or storage (e.g., audio fingerprint database 186A) to determine whether the audio data stream 124 has been previously utilized in updating the audio-based global ML model. However, it should be noted that the remote system 160 does not include a remote counterpart to the connection state engine 140, as this determination may be made exclusively at the client device 120. Therefore, it should be understood that the ephemeral learning described herein may be performed locally at the client device 120, remotely from the client device 120 (e.g., performed at the remote system 160), or both.

[0054] Now turn to Figure 2 , depicts a client device 250 shown in an implementation in which various on-device ML engines are included as part of (or in communication with) an automated assistant client 240. The corresponding ML models are also shown as interfacing with the various on-device ML engines. For simplicity, other components of the client device 250 are not shown. Figure 2 Shown in. Figure 2 One example of how automated assistant client 240 can utilize various on-device ML engines and their corresponding ML models to perform various actions is shown.

[0055] Figure 2The client device 250 in FIG. 1 is shown as having one or more microphones 211, one or more speakers 212, one or more visual components 213, and a display 214 (e.g., a touch-sensitive display). The client device 250 may further include pressure sensors, proximity sensors, accelerometers, magnetometers, and / or other sensors for generating other sensor data in addition to the audio data captured by the one or more microphones 211. The client device 250 selectively executes at least the automated assistant client 240. Figure 2 In the example of , the automated assistant client 240 includes a language representation engine 222, a voice activity detection engine 224, a hot word detection engine 226, an ASR engine 228, a multilingual ASR engine 230, a continuous conversation engine 232, a language recognition engine 234, and a speech recognition engine 236. The automated assistant client 240 further includes a voice capture engine 216 and a visual capture engine 218. It should be understood that Figure 2 The ML engines and ML models depicted in are provided for example purposes and are not intended to be limiting. For example, automated assistant client 240 may further include additional and / or alternative engines, such as a text-to-speech (TTS) engine and corresponding TTS models, an endpoint detector engine and corresponding endpoint detector models, and / or other engines along with associated ML models. Furthermore, it should be understood that one or more engines and / or models described herein may be combined such that a single engine and / or model may perform the functionality of multiple engines and / or models described herein.

[0056] One or more cloud-based automated assistant components 270 may optionally be implemented on one or more computing systems (collectively, “cloud” computing systems) that are communicatively coupled to client device 250 via one or more networks, generally indicated by 299. Cloud-based automated assistant components 270 may be implemented, for example, via a cluster of high-performance servers. In various implementations, an instance of automated assistant client 240, through its interaction with one or more cloud-based automated assistant components 270, may form what appears to be, from the user's perspective, a logical instance of an automated assistant, generally indicated by 295, with which the user may engage in human-computer interaction (e.g., verbal interaction, gesture-based interaction, and / or touch-based interaction).

[0057] For example, client device 250 may be: a desktop computing device, a desktop computing device, a laptop computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker, a smart home appliance such as a smart TV (or a standard TV equipped with a network-connected dongle with automated assistant capabilities), and / or a user wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0058] The one or more visual components 213 can take various forms, such as a single-lens camera, a stereo camera, a LIDAR component (or other laser-based component), a radar component, etc. The one or more visual components 213 can be used, for example, by the visual capture engine 218 to capture image data corresponding to a visual frame (e.g., an image frame, a laser-based visual frame) of the environment in which the client device 250 is deployed. In some implementations, such visual frames can be utilized to determine whether a user is present in the vicinity of the client device 250 and / or the distance of a given user of the client device 250 (e.g., the user's face) relative to the client device 250. Such determinations can be utilized, for example, to determine whether to activate a user's facial recognition function. Figure 2 Further, the speech capture engine 218 may be configured to capture the user's spoken words and / or other audio data captured via one or more of the microphones 211.

[0059] As described in this article, the audio data stream can be Figure 2 to process the various engines described in the foregoing to use the corresponding ML model at the client device 250 and / or using the foregoing (e.g., regarding Figure 1A , Figure 1B and Figures 3 to 7 ) makes predictions at one or more of the cloud-based automated assistant components 270.

[0060] As some non-limiting examples, the respective language representation engines 222, 272 may utilize the respective language representation models 222A, 272A to generate rich feature representations of the audio data stream and / or the spoken speech stream captured in the audio data stream; the respective voice activity detection engines 224, 274 may utilize the respective voice activity detection models 224A, 274A to predict whether the audio data stream includes voice activity of the user of the client device 250 and / or other users; the respective hot word detection engines 226, 276 may utilize the respective language representation models 226A, 276A to predict whether the audio data stream includes one or more specific words or phrases for invoking the automated assistant 295 (e.g., “Ok Assistant”, “Hey Assistant”, “What is the weather Assistant?” ... (How's the weather, assistant? ", etc.) or one or more specific words or phrases for invoking certain functions of the automated assistant 295; the corresponding ASR engines 228, 278 can use the corresponding ASR models 228A, 278A to generate recognized text in a given language based on processing the audio data stream, or predict phonemes and / or word units corresponding to the audio data stream detected at the client device 250, and generate recognized text in a given language based on the phonemes and / or word units; the corresponding multilingual ASR engines 230, 280 can use the corresponding ASR models 230A, 280A to generate recognized text in multiple languages ​​based on processing the audio data stream, or predict phonemes and / or word units corresponding to the audio data stream detected at the client device 250, and generate recognized text in multiple languages ​​based on the phonemes and / or word units; the corresponding continuous dialogue engines 232, 282 can use the corresponding continuous dialogue models 232A, 282A to predict whether further audio data streams are directed to the automated assistant 295 The respective language identification models 234, 284 may utilize the respective language identification models 234A, 284A to predict a given language of the spoken speech stream captured in the audio data stream; and the respective speech recognition engines 236, 286 may utilize the respective speech recognition models 236A, 286A to predict whether the audio data stream captures the spoken speech stream of one or more users of the client device 250 (e.g., by generating speaker embeddings or other representations that can be compared with corresponding actual embeddings of one or more of the users of the client device 250).

[0061] In some implementations, the client device 250 and one or more of the cloud-based automated assistant component 270 may further include a natural language understanding (NLU) engine 238, 294 and a fulfillment engine 240, 296, respectively. The NLU engine 238, 294 may utilize a corresponding NLU model 238A, 294A to perform natural language understanding on the recognized text, predicted phonemes, and / or predicted word units generated by, for example, the ASR engine 228, 278 and / or the multilingual ASR engine 230, 280 to generate NLU data. The NLU data may include, for example, an intent corresponding to a spoken utterance and, optionally, slot values ​​for parameters of the intent. Further, the client device 250 and one or more of the cloud-based automated assistant component 270 may further include a fulfillment engine 240, 296, respectively. The fulfillment engine 240, 296 may utilize a corresponding fulfillment model or rule 240A, 296A and generate fulfillment data based on processing the NLU data. The fulfillment data may define a specific fulfillment in response to user input (e.g., spoken words, typed input, touch input, gesture input, and / or any other type of user input) provided by a user of the client device 250. The specific fulfillment may include an interaction to be performed with a locally installed application based on the user input, a command to be transmitted to an Internet of Things (IoT) device based on the user input (directly or via a corresponding remote system), and / or other resolution actions to be performed based on the user input. The fulfillment data is then provided for local and / or remote performance / execution of the determined action so that the specific fulfillment of the user input is performed. The execution may include, for example, rendering a local and / or remote response (e.g., visually and / or audibly (optionally using an on-device TTS module)), interacting with a locally installed application, transmitting a command to an IoT device, and / or other actions. In other implementations, the NLU engine 238, 294 and the fulfillment engine 240, 296 may be omitted, and the ASR engine 228, 278 and / or the multilingual ASR engine 230, 280 may generate the fulfillment data directly based on the user input. For example, assume that the ASR engine 228, 278 and / or the multilingual ASR engine 230, 280 processes an audio data stream capturing the spoken utterance "turn on the lights" using a corresponding model. In this example, the ASR engine 228, 278 and / or the multilingual ASR engine 230, 280 may generate a semantic output that is then transmitted to a software application associated with the lights and / or directly to the lights, indicating that the lights should be turned on.

[0062] It is worth noting that the cloud-based automated assistant component 270 includes the Figure 2The cloud-based counterparts of the engines and models described herein. However, in some implementations, these engines and models may not be utilized because these engines and models can be directly transmitted to the client device 250 and executed locally at the client device 250, while in other implementations, these engines and models can be utilized specifically when the client device 250 detects any user input and transmits the user input to the cloud-based automated assistant component 270. In various implementations, these engines and models executed at the client device 250 and the cloud-based automated assistant component 270 can be utilized in combination with each other in a distributed manner. Nevertheless, a remote execution module can be optionally included, which performs remote execution based on NLU data and / or fulfillment data generated locally or remotely. Additional and / or alternative remote engines can be included. As described herein, in various implementations, on-device speech processing, on-device image processing, on-device NLU, on-device fulfillment and / or on-device execution can be prioritized at least due to the delay and / or network usage reduction they provide when resolving spoken utterances (because resolving spoken utterances does not require client-server round trips). However, one or more cloud-based automated assistant components 270 may be at least selectively utilized. For example, such components may be utilized in parallel with on-device components, and outputs from such components may be utilized when local components fail. For example, if any on-device engine and / or model fails (e.g., due to relatively limited resources of client device 250), then the more robust resources of the cloud may be utilized.

[0063] Now turn to Figure 3 , depicts a flow chart illustrating an example method 300 of transient learning implemented by a client device based on an audio stream generated by a given radio station. For convenience, the operations of the method 300 are described with reference to a system performing the operations. The system of the method 300 includes a client device (e.g., Figure 1A and Figure 1B The client device 120, Figure 2 The client device 250, Figure 8 The method 300 may be implemented by one or more processors and / or other components of the computing device 810 and / or other client device. In addition, although the operations of the method 300 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0064] At box 352, the system receives an audio data stream that captures a spoken speech stream in a given language from a given radio station that is being actively consumed by a user of a client device. The spoken speech stream may capture, for example, spoken speech including commercials or advertising content from a given radio station, spoken speech including discjockey content from a given radio station, spoken speech including podcast content from a given radio station, spoken speech including music content from a given radio station, and / or other content sources from a given radio station. However, in some implementations, the system may discard any audio data stream that captures spoken speech including music content from a given radio station. In some implementations, the audio data stream may have a fixed length (e.g., 5 seconds, 10 seconds, 15 seconds, etc.). In additional or alternative implementations, the audio data stream may have a dynamic length (e.g., the length of commercials or advertising content, the length of a discjockey, the length of a podcast content, etc.).

[0065] At block 354, the system causes the generation of gradients for updating the global machine learning (ML) model with respect to the given language. The system may return to blocks 352 to 352 to continue receiving additional audio data streams that capture additional spoken utterance streams from a given radio station that the user of the client device is actively consuming. This enables the system to perform concurrent operations while receiving audio data streams from a given radio station that the user of the client device is actively consuming. Figure 3 It is worth noting that at block 354, the gradient has not yet been generated. Instead, at block 354, the system may perform various operations (e.g., Figure 5 The audio fingerprint operation, such as Figure 5 The speech recognition operation described herein and / or other operations described herein) may be used to determine whether to generate a gradient based on an audio data stream.

[0066] At box 356, the system may determine whether to generate the gradient locally at the client device or to generate the gradient away from the client device. In some implementations, the system may determine by default that the gradient is generated locally at the client device. In other implementations, the system may determine by default that the gradient is generated away from the client device. In yet other implementations, the system may determine to generate the gradient locally at the client device in some instances, but to generate the gradient away from the client device in other instances. For example, the system may prioritize generating the gradient locally at the client device to reduce the amount of network resources consumed when streaming audio data to a remote system, but in response to determining that a threshold amount of computing resources has been consumed at the client device, the audio data stream is started to be streamed to the remote system. In addition, for example, the system may periodically switch between generating the gradient locally at the client device and generating the gradient away from the client device to more evenly distribute the computing resources consumed by the client device and the remote system. It should be understood that these examples are not intended to be limiting, and any other criteria for determining whether to generate the gradient locally at the client device or to generate the gradient away from the client device are contemplated herein.

[0067] If at the iteration of block 356, the system determines that the gradient is generated locally at the client device, the system may proceed to block 358. At block 358, the system processes the audio data stream using the on-device ML model that is the on-device counterpart of the global ML model and is stored in an on-device memory or storage of the client device. At block 360, the system generates the gradient using an unsupervised or self-supervised learning technique and based on processing the audio data stream using the on-device ML model. At block 362, the system discards the audio data stream. At block 364, the system transmits the gradient to the remote system to cause the remote system to update the global ML model based on the gradient. In other words, the system may cause the execution of the gradient locally at the client device with respect to the global ML model. Figure 1A Describes the operations of a process flow.

[0068] If at an iteration of block 356, the system determines that the gradients are generated locally away from the client device, the system may proceed to block 366. At block 366, the system transmits the audio data stream to the remote system to cause the remote system to update the global ML model based on processing the audio data stream. In other words, rather than the system causing the gradients to be generated locally at the client device based on processing the audio data stream locally at the client device, the system may cause the audio data stream to be transmitted to the remote system to cause the remote system to process the audio data stream and generate the gradients based on processing the audio data stream (e.g., as described with respect to FIG. 1 ). Figure 1BIn doing so, the system may cause the remote system to perform operations. For example, at box 366A, the system causes the remote system to process the audio data stream using the global ML model. Further, at box 366B, the system causes the remote system to generate gradients based on processing the audio data stream using unsupervised or self-supervised learning techniques. Moreover, at box 366C, the system causes the remote system to discard the audio data stream. Additionally, at box 366D, the system causes the remote system to update the global ML model based on the gradient.

[0069] Thus, the system may initially receive an audio data stream at a client device, and may determine whether to generate gradients to be utilized in updating a global ML model locally at the client device or remotely from the client device. In making this determination, the system may take into account computing resources that will be consumed at the client device when generating the gradients and / or network resources that will be consumed when transmitting the audio data stream to a remote system to reduce consumption of computing and / or network resources while still enabling the system to update a global ML model based on a diverse audio data stream generated by radio stations around the world. Thus, a global ML model updated in this manner is more robust in processing and / or understanding more languages ​​of users around the world.

[0070] Now turn to Figure 4 , depicts a flow chart illustrating an example method 400 of transient learning implemented by a remote system based on an audio stream generated by a given radio station. For convenience, the operations of method 400 are described with reference to a system performing the operations. The system of method 400 includes a remote system (e.g., Figure 1A and Figure 1B Remote system 160, Figure 2 Cloud-based automated assistant component 270, Figure 8 The method 400 may be performed by one or more processors and / or other components of the computing device 810 and / or other remote system. In addition, although the operations of the method 400 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0071] At box 452, the system receives an audio data stream capturing a spoken speech stream in a given language from a given client device, the audio data stream being initially received at the given client device from a given radio station that a user of the given client device is actively consuming. At box 454, the system processes the audio data stream using a global ML model. The system may return to box 452 to continue receiving additional audio data streams. At box 456, the system generates gradients based on processing the audio data stream using unsupervised or self-supervised learning techniques. At box 458, the system discards the audio data stream. At box 460, the system updates the global ML model for the given language based on the gradients.

[0072] In other words, according to Figure 4 In an implementation of method 400, the system can utilize a given client device as a proxy for obtaining an audio data stream. Further, the system can process the audio data stream to generate gradients (e.g., such as those with respect to Figure 1B This enables ephemeral learning to be performed not only locally at the client device, but also at the remote system. In an additional or alternative implementation, the system may obtain the audio data stream directly from a given radio station to further reduce consumption of network resources.

[0073] although Figure 4 The method 400 of FIG. 4 is described in a specific manner, but it should be understood that this is for the purpose of example and is not intended to be limiting. For example, the system may perform concurrent execution with respect to different audio data streams received from a given client device and / or additional client devices. Figure 4 As another example, the system may initially process the audio data stream to determine whether to generate gradients (e.g., as described with respect to Figure 6 described above).

[0074] Now turn to Figure 5 , depicts a flow chart illustrating an example method 500 of utilizing deduplication techniques in ephemeral learning implemented by a client device. For convenience, the operations of the method 500 are described with reference to a system performing the operations. The system of the method 500 includes a client device (e.g., Figure 1A and Figure 1B The client device 120, Figure 2 The client device 250, Figure 8 The method 500 may be implemented by one or more processors and / or other components of the computing device 810 and / or other client device. In addition, although the operations of the method 500 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0075] At box 552, the system receives an audio data stream that captures a spoken speech stream in a given language from a given radio station. The spoken speech stream may capture, for example, spoken speech including commercials or advertising content from a given radio station, spoken speech including disc jockey content from a given radio station, spoken speech including podcast content from a given radio station, spoken speech including music content from a given radio station, and / or other content sources from a given radio station. However, in some implementations, the system may discard any audio data stream that captures spoken speech including music content from a given radio station. In some implementations, the audio data stream may have a fixed length (e.g., 5 seconds, 10 seconds, 15 seconds, etc.). In additional or alternative implementations, the audio data stream may have a dynamic length (e.g., the length of commercials or advertising content, the length of a program host, the length of a podcast content, etc.). In some implementations, the audio data stream may correspond to an audio data stream actively consumed by a user of a client device when the user actively listens to a given radio station. In additional or alternative implementations, the audio data stream may correspond to an audio data stream that is accessible at the client device via a given radio station but is not actively consumed by a user of the client device.

[0076] At block 554, the system generates an audio fingerprint of the audio data stream based on processing the audio data stream. In some implementations, the audio fingerprint of the audio data stream may correspond to the embedding. In some versions of these implementations, the system may use (e.g., stored in Figure 1A and Figure 1B The audio data stream is processed by an encoder-decoder ML model (or another ML model in the on-device ML model database 120A) to generate an embedding (or other lower dimensional representation of the audio data). Further, the embedding can be mapped to an embedding space (or another lower dimensional latent space) that enables the embedding to be compared with previously encountered audio data streams (e.g., stored in Figure 1A The system may compare the audio fingerprint of the audio data stream to a plurality of previously generated embeddings (e.g., as described with respect to block 556) in the audio fingerprint database 144A. In additional or alternative implementations, the audio fingerprint of the audio data stream may correspond to an audio hash. In some versions of these implementations, the system may use (e.g., stored in Figure 1A and Figure 1B The local sensitivity hash of the ML model database 120A on the device is processed to generate an audio hash (e.g., a vector representation of the audio data stream). Further, this enables the audio hash to be compared with the previously encountered audio data stream (e.g., stored in Figure 1AThe audio fingerprint database 144A is used to compare the audio hashes to multiple previously generated audio hashes (e.g., as described with respect to box 556).

[0077] At block 556, the system determines whether the audio data stream was previously utilized in generating gradients for updating the global ML model. For example, in implementations where the audio fingerprint corresponds to an embedding, if the embedding generated based on the audio data stream matches a given previously generated embedding for a given previously encountered audio data stream, the system can determine that the audio data stream was previously utilized in generating gradients for updating the global ML model. For example, if the embedding generated based on the audio data stream is within a threshold distance in the embedding space (e.g., using Euclidean distance, cosine similarity, and / or another distance metric) from the given previously generated embedding, the system can determine that the embedding matches the given previously generated embedding. Otherwise, the system can determine that the embedding generated based on the audio data stream does not match any given previously generated embedding.

[0078] As another example, in an implementation where the audio fingerprint corresponds to an audio hash, if the audio hash generated based on the audio data stream matches a given previously generated audio hash for a given previously encountered audio data stream, the system can determine that the audio data stream was previously utilized in generating gradients for updating the global ML model. For example, if the corresponding vector representing the audio data stream satisfies a similarity threshold, the system can determine that the audio hash generated based on the audio data stream matches the given previously generated audio hash. Otherwise, the system can determine that the embedding generated based on the audio data stream does not match any given previously generated embedding.

[0079] Notably, in these implementations, a remote system communicatively coupled to a population of client devices (eg, including the client device) may maintain and periodically distribute an audio fingerprint database (eg, Figure 1B For example, each of the client devices in the population can generate an audio fingerprint generated locally at the client device and transmit it to the remote system, and the remote system can store the audio fingerprint in the audio fingerprint database along with any audio fingerprints generated at the remote system. Further, the remote system can distribute the audio fingerprint database (or updates thereto) to each of the client devices in the population to ensure that the corresponding audio fingerprint database (e.g., Figure 1A The audio fingerprint database 144A) is up to date. This ensures that the client device population does not continuously generate gradients based on the same audio data stream, and ensures that the audio data streams have sufficient diversity when generating gradients that are utilized when updating the global ML model.

[0080] If at the iteration of block 556, the system determines that the audio data stream was previously utilized in generating gradients for updating the global ML model, the system can proceed to block 558. At block 558, the system avoids any further processing of the audio data stream. At block 560, the system discards the audio data stream. The system can return to block 552 to perform additional iterations of method 500.

[0081] If at the iteration of box 556, the system determines that the audio data stream was previously utilized in generating gradients for updating the global ML model, the system can proceed to box 562. At box 562, the system determines whether the given language corresponds to a target language. For example, the system can use a language recognition model (e.g., stored in the on-device ML model database 120A) to process the audio data stream (or recognized text generated based on processing the audio data stream (e.g., generated using a multilingual ASR model)) to identify the given language associated with the spoken utterance stream captured in the audio data stream. Further, the system can compare the given language with one or more target languages ​​(e.g., stored in the target language database 142A of Figure 1) to determine whether to generate gradients to be utilized when updating the global ML with respect to the given language. In other words, the system can discard audio data streams that capture spoken utterances in some languages ​​while continuing to process audio data streams that capture spoken utterances in other languages.

[0082] If at the iteration of block 562, the system determines that the given language does not correspond to the target language, the system can proceed to block 558. At block 558, the system avoids any further processing of the audio data stream. At block 560, the system discards the audio data stream. The system can return to block 552 to perform additional iterations of method 500. If at the iteration of block 562, the system determines that the given language corresponds to the target language, the system can proceed to Figure 3 The system may further include a block 356 of method 300 to determine whether to perform ephemeral learning locally at the client device or remotely from the client device. In other words, the system may cause the client device to perform these deduplication techniques to reduce the amount of computational and / or network resources that would otherwise be consumed in processing the previously processed audio data stream. Further, assuming the system determines that the gradients were not previously utilized in generating the gradients utilized in updating the global ML model, the system may implement ephemeral learning locally at the client device as well as implement ephemeral learning remotely from the client device.

[0083] although Figure 5The method 500 of is described in a specific manner, but it should be understood that this is for example and is not intended to be limiting. For example, the deduplication operations of box 556 and box 562 can be switched so that the language identification deduplication technique is performed before the audio fingerprint deduplication technique. In addition, for example, one of the deduplication operations of box 556 and box 562 can be omitted so that only the language identification deduplication technique is performed or only the audio fingerprint deduplication technique is performed.

[0084] Now turn to Figure 6 , depicts a flow chart illustrating an example method 600 of utilizing deduplication techniques in ephemeral learning implemented by a remote system. For convenience, the operations of method 600 are described with reference to a system performing the operations. The system of method 600 includes a remote system (e.g., Figure 1A and Figure 1B Remote system 160, Figure 2 Cloud-based automated assistant component 270, Figure 8 The method 600 may be performed by one or more processors and / or other components of the computing device 810 and / or other remote system. In addition, although the operations of the method 600 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0085] At block 652, the system receives from a given client device an audio data stream capturing a spoken speech stream in a given language, the audio data stream initially received at the given client device from a given radio station. At block 654, the system generates an audio fingerprint of the audio data stream based on processing the audio data stream.

[0086] At block 656, the system determines whether the audio data stream was previously utilized in generating gradients for updating the global ML model. If, at an iteration of block 656, the system determines that the audio data stream was previously utilized in generating gradients for updating the global ML model, the system may proceed to block 658. At block 658, the system refrains from any further processing of the audio data stream. At block 660, the system discards the audio data stream. The system may return to block 652 to perform additional iterations of method 600. If, at an iteration of block 656, the system determines that the audio data stream was previously utilized in generating gradients for updating the global ML model, the system may proceed to block 662.

[0087] At block 662, the system determines whether the given language corresponds to the target language. If at an iteration of block 662, the system determines that the given language does not correspond to the target language, the system can proceed to block 658. At block 658, the system refrains from any further processing of the audio data stream. At block 660, the system discards the audio data stream. The system can return to block 652 to perform additional iterations of method 600. If at an iteration of block 662, the system determines that the given language corresponds to the target language, the system can proceed to Figure 4 , and utilizing transient learning at the remote system to generate gradients for updating the global ML model.

[0088] in other words, Figure 6 Method 600 with Figure 5 The method 500 is the same or similar, but the system is a remote system according to Figure 6 The method 600 is implemented by the client device according to Figure 5 It is worth noting that in these implementations, the audio data stream may be initially received by a given client device, but Figure 6 The operations of method 600 may be performed by components specific to the remote system. Figure 6 The method 600 is described with respect to a single client device and a single audio data stream, but it should be understood that this is for example and is not intended to be limiting. Rather, it should be understood that the system can be configured to perform multiple audio data streams in parallel based on different audio data streams received from different client devices. Figure 6 Multiple iterations of method 600.

[0089] Now turn to Figure 7 , depicts a flow chart illustrating an example method 700 for determining whether a client device should implement an ephemeral learning technique or a federated learning technique. For convenience, the operations of method 700 are described with reference to a system performing the operations. The system of method 700 includes a client device (e.g., Figure 1A and Figure 1B The client device 120, Figure 2 The client device 250, Figure 8 The method 700 may be implemented by one or more processors and / or other components of the computing device 810 and / or other client device. In addition, although the operations of the method 700 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0090] At block 752, the system receives an audio data stream (e.g., in the form of audio data about a given language) from a given radio station that captures a spoken speech stream in a given language. Figure 5 552 of method 500).

[0091] At block 754, the system determines whether to implement federated learning or ephemeral learning when generating gradients for updating the global ML model. For example, the system may determine whether to implement federated learning or ephemeral learning based on the connection state between the client device receiving the audio data stream and the remote system and / or the connection strength between the client device and the remote system (e.g., as described in relation to Figure 1A Additionally or alternatively, the system may determine whether to implement federated learning or transient learning based on the location of the client device. For example, if the location of the client device is within a specific geographic area, the system may determine to implement federated learning instead of transient learning.

[0092] If at the iteration of block 754, the system determines that federated learning is implemented, the system can proceed to block 756. At block 756, the system processes the audio data stream using the on-device ML model as the on-device counterpart of the global ML model. At block 758, the system generates gradients based on processing the audio data stream using the on-device ML model using an unsupervised or self-supervised learning technique. At block 760, the system asynchronously transmits the gradients to the remote system so that the remote system updates the global ML model based on the gradients. The operations of blocks 756 and 758 can be described in detail with respect to Figure 3 The operations of block 358 and block 360 of method 300 may be performed in the same or similar manner as described above.

[0093] However, with Figure 3 Compared to the operation of box 364 of method 300 of , the transmission of the gradient to the remote system at box 760 is asynchronous. In these implementations, the system may have determined to implement federated learning to generate gradients based on a weak and / or unstable connection between the client device and the remote system. Therefore, due to the weak and / or unstable connection between the client device and the remote system, the system may store the gradients in a temporary memory or storage of the client device instead of attempting to transmit the gradients to the remote system in response to the system causing the gradients to be generated locally at the client device. Nevertheless, when a strong and / or stable connection is subsequently established between the client device and the remote system (e.g., via a Wi-Fi network, a cellular network, etc.), the system may still cause the gradients to be transmitted to the remote system so that the remote system updates the global ML model based on the gradients. In some versions of these implementations, the audio data stream may be discarded in response to the gradients being generated, while in other versions of these implementations, the system may store the audio data stream in a temporary memory or storage of the client device until the gradients are transmitted to the remote system in an asynchronous manner, and then discard the audio data stream.

[0094] If at the iteration of block 754, the system determines that transient learning is achieved, the system may proceed to Figure 3 In other words, the system may cause the client device to determine whether to implement ephemeral learning or federated learning when generating gradients. Further, assuming that the system determines to implement ephemeral learning based on a strong and / or stable connection between the client device and the remote system, the system may proceed to Figure 3 In an implementation where the system determines at an iteration of block 356 that the ephemeral learning is to be implemented locally at the client device or remotely from the client device, the system may transmit the gradient to the remote system in a synchronized manner at an iteration of block 364 (e.g., in response to the gradient being generated) without storing the gradient in a transient memory or storage of the client device. In an implementation where the system determines at an iteration of block 356 that the ephemeral learning is to be implemented remotely from the client device, the system may transmit the audio data stream to the remote system in a synchronized manner at an iteration of block 366.

[0095] although Figure 7 Method 700 and Figure 5 The method 500 is described as a single method, but it should be understood that this is for the purpose of illustrating the techniques described herein and is not intended to be limiting. Figure 7 Method 700 and Figure 5 The techniques described in method 500 (or aspects thereof) may be combined into a single method.

[0096] Now turn to Figure 8 , depicts a block diagram of an example computing device 810 that can optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, cloud-based automated assistant components, and / or other components can include one or more components of the example computing device 810.

[0097] The computing device 810 typically includes at least one processor 814 that communicates with a number of peripheral devices via a bus subsystem 812. These peripheral devices may include a storage subsystem 824 (including, for example, a memory subsystem 825 and a file storage subsystem 826), a user interface output device 820, a user interface input device 822, and a network interface subsystem 816. The input and output devices allow interaction with a user of the computing device 810. The network interface subsystem 816 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0098] The user interface input device 822 may include a keyboard, a pointing device such as a mouse, a trackball, a touch pad or a graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 810 or onto a communication network.

[0099] The user interface output device 820 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanisms for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Generally speaking, the use of the term "output device" is intended to include devices and methods of all possible types of information output to a user or to another machine or computing device from the computing device 810.

[0100] The storage subsystem 824 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 824 may include selected aspects of the methods disclosed herein and the storage subsystem 824 may include a plurality of storage devices and a plurality of storage devices. Figure 1A , Figure 1B and Figure 2 The logic of the various components depicted.

[0101] These software modules are typically executed by the processor 814 alone or in combination with other processors. The memory 825 used in the storage subsystem 824 may include multiple memories, including a main random access memory (RAM) 830 for storing instructions and data during program execution and a read-only memory (ROM) 832 in which fixed instructions are stored. The file storage subsystem 826 can provide persistent storage for program and data files, and can include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations can be stored in the storage subsystem 824 by the file storage subsystem 826, or in other machines that can be accessed by the processor 814.

[0102] The bus subsystem 812 provides a mechanism for the various components and subsystems of the computing device 810 to communicate with each other as intended. Although the bus subsystem 812 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0103] The computing device 810 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Figure 8 The description of computing device 810 depicted in FIG. 8 is intended only as a specific example for illustrating some implementations. Many other configurations of computing device 810 may have more Figure 8 The computing devices depicted may have greater or fewer components.

[0104] In cases where the systems described herein collect or otherwise monitor personal information about a user or can make use of personal information and / or monitored information, the user may be provided with an opportunity to control whether a program or function collects user information (e.g., information about the user's social network, social behavior or activity, occupation, the user's preferences, or the user's current geographic location) or to control whether and / or how to receive content from a content server that may be more relevant to the user. Moreover, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or, in the case of obtaining geographic location information, the user's geographic location may be generalized (such as to a city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, the user may control how information about the user is collected and / or used.

[0105] In some implementations, a method performed by one or more processors of a client device is provided, the method comprising: receiving an audio data stream capturing a spoken utterance stream in a given language from a given radio station; generating an audio fingerprint of the audio data stream based on processing the audio data stream; determining whether the audio data stream has been previously utilized in generating gradients for updating a global machine learning (ML) model with respect to the given language based on comparing the audio fingerprint of the audio data stream to an audio fingerprint database; and in response to determining that the audio data stream has not been previously utilized in generating gradients for updating the global ML model with respect to the given language: processing the audio data stream using an on-device ML model that is stored in an on-device storage of the client device and is an on-device counterpart of the global ML model; generating gradients based on processing the audio data stream using the on-device ML model using an unsupervised or self-supervised learning technique; and transmitting the gradients to a remote system to be utilized in updating the global ML model with respect to the given language.

[0106] These and other implementations of the technology may include one or more of the following features.

[0107] In some implementations, the method may further include, in response to determining that the audio data stream has been previously utilized in generating gradients for updating the global ML model with respect to the given language: discarding the audio data stream.

[0108] In some implementations, the method may further include: receiving an audio fingerprint database from a remote system; and storing the audio fingerprint database in an on-device storage of the client device. In some versions of those implementations, the remote system may have previously generated the audio fingerprint database based on a plurality of corresponding audio data streams received from the client device and a plurality of additional client devices and capturing corresponding spoken utterance streams in a plurality of different languages ​​including a given language from a plurality of different radio stations.

[0109] In some implementations, generating an audio fingerprint of the audio data stream based on processing the audio data stream may include: processing the audio data stream using a local sensitivity hash to generate an audio hash as the audio fingerprint. In some versions of those implementations, determining whether the audio data stream has been previously utilized in generating gradients for updating a global ML model for a given language based on comparing the audio fingerprint of the audio data stream to an audio fingerprint database may include: comparing the audio hash generated based on processing the audio data stream with a plurality of previously generated audio hashes, and determining whether the audio data stream has been previously utilized in generating gradients for updating a global ML model for a given language based on the comparison. The plurality of previously generated audio hashes may have been previously generated based on processing the corresponding audio data streams, and the plurality of previously generated audio hashes may be stored in the audio fingerprint database.

[0110] In some implementations, generating an audio fingerprint for the audio data stream based on processing the audio data stream may include: processing the audio data stream using an encoder portion of an encoder-decoder ML model to generate an embedding as the audio fingerprint. In some versions of those implementations, determining whether the audio data stream has been previously utilized in generating gradients for updating a global ML model for a given language based on comparing the audio fingerprint of the audio data stream to an audio fingerprint database may include: comparing the embedding generated based on processing the audio data stream with a plurality of previously generated embeddings, and determining whether the audio data stream has been previously utilized in generating gradients for updating a global ML model for a given language based on the comparison. The plurality of previously generated embeddings may have been previously generated based on processing the corresponding audio data stream, and the plurality of previously generated embeddings may be stored in the audio fingerprint database.

[0111] In some implementations, the method may further include: processing the audio data stream using an on-device language identification model stored in an on-device storage of the client device to identify a given language, and determining whether the given language is one of a plurality of target languages. In some versions of those implementations, generating an audio fingerprint of the audio data stream may be responsive to determining that the given language is one of a plurality of target languages. In some versions of those implementations, the method may further include: in response to determining that the given language is not one of a plurality of target languages: avoiding any further processing of the audio data stream; and discarding the audio data stream. In some versions of those implementations, a developer associated with the global ML model may provide an indication of the plurality of target languages.

[0112] In some implementations, the remote system may update one or more global weights of the global ML model using the gradient to generate an updated global ML model. In some versions of those implementations, the method may further include: receiving one or more global weights of the updated global ML model from the remote system; or receiving the updated global ML model from the remote system. In some further versions of those implementations, the method may further include: replacing one or more on-device weights of the on-device ML model with one or more global weights of the updated global ML model in the on-device storage of the client device; or replacing the on-device ML model with the updated global ML model in the on-device storage of the client device.

[0113] In some implementations, unsupervised or self-supervised learning techniques may include one or more of the following: a teacher-student technique, or a mask technique.

[0114] In some implementations, the global ML model can be a global feature extractor model that is updated for a given language to extract features from an audio data stream.

[0115] In some implementations, the global ML model can be a multilingual automatic speech recognition (ASR) model that is updated for a given language to recognize text from an audio data stream.

[0116] In some implementations, a method performed by one or more processors of a client device is provided, the method comprising: receiving an audio data stream capturing a spoken speech stream in a given language from a given radio station; generating an audio fingerprint of the audio data stream based on processing the audio data stream; determining whether the audio data stream has been previously utilized in generating gradients for updating a global machine learning (ML) model with respect to the given language based on comparing the audio fingerprint of the audio data stream to an audio fingerprint database; and in response to determining that the audio data stream has not been previously utilized in generating gradients for updating the global ML model with respect to the given language: transmitting the audio data stream to a remote system. Transmitting the audio data stream to the remote system causes the remote system to: process the audio data stream using the global ML model; generate gradients based on processing the audio data stream using the global ML model using an unsupervised or self-supervised learning technique; and update the global ML model based on the gradients.

[0117] These and other implementations of the technology may include one or more of the following features.

[0118] In some implementations, transmitting the audio data stream to the remote system can further cause the remote system to discard the audio data stream after generating the gradient.

[0119] In some implementations, a method performed by one or more processors of a remote system is provided, the method comprising: receiving an audio data stream capturing a spoken speech stream in a given language from a given client device, the audio data stream initially received at the given client device from a given radio station; generating an audio fingerprint of the audio data stream based on processing the audio data stream; determining whether the audio data stream has been previously utilized in generating gradients for updating a global machine learning (ML) model with respect to the given language based on comparing the audio fingerprint of the audio data stream to an audio fingerprint database; and in response to determining that the audio data stream has not been previously utilized in generating gradients for updating the global ML model with respect to the given language: processing the audio data stream using the global ML model; generating gradients based on processing the audio data stream using the global ML model using an unsupervised or self-supervised learning technique; and updating the global ML model with respect to the given language based on the gradients.

[0120] In some implementations, a method performed by one or more processors of a client device is provided, the method comprising: receiving an audio data stream capturing a spoken speech stream in a given language from a given radio station; determining whether to implement federated learning or ephemeral learning to generate gradients for updating a global machine learning (ML) model with respect to the given language based on a connection state between the client device and a remote system; in response to determining to implement federated learning to generate gradients to be utilized in updating the global ML model with respect to the given language: processing the audio data stream using an on-device ML model that is stored in an on-device storage of the client device and is fully an on-device counterpart of a local ML model; generating gradients based on processing the audio data stream using the on-device ML model using an unsupervised or self-supervised learning technique; and asynchronously transmitting the gradients to a remote system to be utilized in updating the global ML model for a given language; and generating gradients to be utilized in updating the global ML model for a given language in response to determining that ephemeral learning is achieved: processing the audio data stream using the on-device ML model; generating gradients based on processing the audio data stream using the on-device ML model using an unsupervised or self-supervised learning technique; and synchronously transmitting the gradients to a remote system to be utilized in updating the global ML model for a given language.

[0121] These and other implementations of the technology may include one or more of the following features.

[0122] In some implementations, the method may further include: determining to implement federated learning to generate gradients for use in updating the global ML model for the given language based on a connection status between the client device and the remote system indicating that the client device cannot connect to the remote system. In some versions of those implementations, asynchronously transmitting the gradients to the remote system to be used in updating the global ML model for the given language may include: after generating the gradients: determining that a connection has been established between the client device and the remote system; and in response to determining that a connection has been established between the client device and the remote system: transmitting the gradients to the remote system to be used in updating the global ML model for the given language.

[0123] In some implementations, the method may further include: determining to implement ephemeral learning to generate gradients utilized when updating the global ML model for the given language based on a connection state between the client device and the remote system indicating that the client device is connected to the remote system. In some versions of those implementations, synchronously transmitting the gradients to the remote system to be utilized when updating the global ML model for the given language may include: transmitting the gradients to the remote system to be utilized when updating the global ML model for the given language without subsequently establishing any connection between the client device and the remote system.

[0124] In some implementations, determining whether to implement federated learning or ephemeral learning to generate gradients for updating the global ML model for a given language can be further based on the location of the client device.

[0125] In some implementations, a method performed by one or more processors of a client device is provided, the method comprising: receiving an audio data stream capturing a spoken utterance stream in a given language from a given radio station; determining whether to implement federated learning or ephemeral learning to generate gradients for updating a global machine learning (ML) model with respect to the given language based on a connection state between the client device and a remote system; in response to determining to implement federated learning to generate gradients to be utilized in updating the global ML model with respect to the given language: processing the audio data stream using an on-device ML model that is stored in an on-device storage of the client device and is an on-device counterpart of the global ML model; generating gradients based on processing the audio data stream using the on-device ML model using an unsupervised or self-supervised learning technique; and asynchronously transmitting the gradients to the remote system to be utilized in updating the global ML model with respect to the given language; and in response to determining to implement ephemeral learning to generate gradients to be utilized in updating the global ML model with respect to the given language: synchronously transmitting the audio data stream to the remote system. Synchronously transmitting the audio data stream to the remote system enables the remote system to: process the audio data stream using the global ML model; generate gradients using an unsupervised or self-supervised learning technique and based on processing the audio data stream using the global ML model; and update the global ML model based on the gradients.

[0126] These and other implementations of the technology may include one or more of the following features.

[0127] In some implementations, synchronously transmitting the audio data stream to the remote system can further cause the remote system to: discard the audio data stream after generating the gradient.

[0128] In some implementations, a method performed by one or more processors of a client device is provided, the method comprising: receiving an audio data stream capturing a spoken speech stream in a given language from a given radio station being actively consumed by a user of the client device; and causing generation of gradients for updating a global machine learning (ML) model at a remote system with respect to the given language. In some versions of those implementations, causing generation of gradients comprises: processing the audio data stream using an on-device ML model that is stored in an on-device storage of the client device and is an on-device counterpart of a global ML model; using unsupervised or self-supervised learning techniques and generating gradients based on processing the audio data stream using the on-device ML model; discarding the audio data stream; and transmitting the gradients to the remote system to be utilized in updating the global ML model with respect to the given language. In other implementations, causing generation of gradients comprises transmitting the audio data stream to the remote system. Transmitting the audio data stream to the remote system causes the remote system to: process the audio data stream using the global ML model; using unsupervised or self-supervised learning techniques and generating gradients based on processing the audio data stream using the global ML model; discarding the audio data stream; and updating the global ML model based on the gradients.

[0129] In some implementations, a method performed by one or more processors of a remote system is provided, the method comprising: receiving an audio data stream capturing a spoken speech stream in a given language from a given client device, the audio data stream initially received at the given client device from a given radio station being actively consumed by a user of the given client device; processing the audio data stream using a global machine learning (ML) model; generating gradients using unsupervised or self-supervised learning techniques and based on processing the audio data stream using the global ML model; discarding the audio data stream; and updating the global ML model with respect to the given language based on the gradients.

[0130] Various implementations may include a non-transitory computer-readable storage medium storing instructions that are executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described herein. Other implementations may include an automated assistant client device (e.g., a client device that includes at least an automated assistant interface for interfacing with a cloud-based automated assistant component) that includes a processor that is operable to execute the stored instructions to perform a method such as one or more of the methods described herein. Still other implementations may include a system of one or more servers that include one or more processors that are operable to execute the stored instructions to perform a method such as one or more of the methods described herein.

Claims

1. A method implemented by one or more processors of a client device, the method comprising: receiving an audio data stream capturing a stream of spoken speech in a given language from a given radio station; generating an audio fingerprint of the audio data stream based on processing the audio data stream; determining whether the audio data stream has been previously utilized in generating gradients for updating a global machine learning (ML) model with respect to the given language based on comparing the audio fingerprint of the audio data stream to an audio fingerprint database; and In response to determining that the audio data stream has not been previously utilized in generating gradients for updating the global ML model with respect to the given language: processing the audio data stream using an on-device ML model, the on-device ML model being stored in an on-device storage of the client device and being an on-device counterpart of the global ML model; generating the gradient based on processing the audio data stream using the on-device ML model using an unsupervised or self-supervised learning technique; as well as The gradients are transmitted to the remote system to be utilized in updating the global ML model with respect to the given language.

2. The method of claim 1, further comprising: In response to determining that the audio data stream has been previously utilized in generating gradients for updating the global ML model with respect to the given language: The audio data stream is discarded.

3. The method of claim 1 or claim 2, further comprising: receiving the audio fingerprint database from the remote system; as well as The audio fingerprint database is stored in the on-device storage of the client device.

4. The method of claim 3, wherein the remote system previously generated the audio fingerprint database based on a plurality of corresponding audio data streams received from the client device and a plurality of additional client devices and capturing corresponding spoken speech streams in a plurality of different languages ​​including the given language from a plurality of different radio stations.

5. The method of claim 1 or claim 2, wherein generating the audio fingerprint of the audio data stream based on processing the audio data stream comprises: The audio data stream is processed using a local sensitivity hash to generate an audio hash as the audio fingerprint.

6. The method of claim 5, wherein determining whether the audio data stream has been previously utilized in generating gradients for updating a global ML model with respect to the given language based on comparing the audio fingerprint of the audio data stream to an audio fingerprint database comprises: comparing the audio hash generated based on processing the audio data stream to a plurality of previously generated audio hashes, wherein the plurality of previously generated audio hashes were previously generated based on processing the corresponding audio data streams, and wherein the plurality of previously generated audio hashes are stored in the audio fingerprint database; as well as A determination is made based on the comparison whether the audio data stream has been previously utilized in generating gradients for updating a global ML model with respect to the given language.

7. The method of claim 1 or claim 2, wherein generating the audio fingerprint of the audio data stream based on processing the audio data stream comprises: The audio data stream is processed using an encoder portion of an encoder-decoder ML model to generate an embedding as the audio fingerprint.

8. The method of claim 7, wherein determining whether the audio data stream has been previously utilized in generating gradients for updating a global ML model with respect to the given language based on comparing the audio fingerprint of the audio data stream to an audio fingerprint database comprises: comparing the embedding generated based on processing the audio data stream to a plurality of previously generated embeddings, wherein the plurality of previously generated embeddings were previously generated based on processing the corresponding audio data streams, and wherein the plurality of previously generated embeddings are stored in the audio fingerprint database; as well as A determination is made based on the comparison whether the audio data stream has been previously utilized in generating gradients for updating a global ML model with respect to the given language.

9. The method of any preceding claim, further comprising: processing the audio data stream to identify the given language using an on-device language identification model stored in the on-device storage of the client device; as well as A determination is made as to whether the given language is one of a plurality of target languages, wherein generating the audio fingerprint of the audio data stream is responsive to determining that the given language is one of the plurality of target languages.

10. The method of claim 9, further comprising: In response to determining that the given language is not one of the plurality of target languages: refraining from any further processing of the audio data stream; as well as The audio data stream is discarded.

11. The method of claim 9, wherein a developer associated with the global ML model provides an indication of the plurality of target languages.

12. The method of any preceding claim, wherein the remote system utilizes the gradient to update one or more global weights of the global ML model to generate an updated global ML model.

13. The method of claim 12, further comprising: receiving the one or more global weights of the updated global ML model from the remote system, or The updated global ML model is received from the remote system.

14. The method of claim 13, further comprising: replacing one or more on-device weights of the on-device ML model with the one or more global weights of the updated global ML model in the on-device storage of the client device, or The on-device ML model is replaced with the updated global ML model in the on-device storage of the client device.

15. A method as claimed in any preceding claim, wherein the unsupervised or self-supervised learning technique comprises one or more of: a teacher-student technique, or a mask technique.

16. A method as claimed in any preceding claim, wherein the global ML model is a global feature extractor model that is updated with respect to the given language to extract features from the audio data stream.

17. A method as claimed in any preceding claim, wherein the global ML model is a multilingual Automatic Speech Recognition (ASR) model, the ASR model being updated for the given language to recognize text from the audio data stream.

18. A method implemented by one or more processors of a client device, the method comprising: receiving an audio data stream capturing a stream of spoken speech in a given language from a given radio station; generating an audio fingerprint of the audio data stream based on processing the audio data stream; determining whether the audio data stream has been previously utilized in generating gradients for updating a global machine learning (ML) model with respect to the given language based on comparing the audio fingerprint of the audio data stream to an audio fingerprint database; and In response to determining that the audio data stream has not been previously utilized in generating gradients for updating the global ML model with respect to the given language: transmitting the audio data stream to the remote system, wherein transmitting the audio data stream to the remote system causes the remote system to: Processing the audio data stream using the global ML model; generating the gradient using the unsupervised or self-supervised learning technique and based on processing the audio data stream using the global ML model; and The global ML model is updated based on the gradients.

19. The method of claim 18, wherein transmitting the audio data stream to the remote system further causes the remote system to: After generating the gradient: The audio data stream is discarded.

20. A method implemented by one or more processors of a remote system, the method comprising: receiving, from a given client device, an audio data stream capturing a spoken speech stream in a given language, the audio data stream initially received at the given client device from a given radio station; generating an audio fingerprint of the audio data stream based on processing the audio data stream; determining whether the audio data stream has been previously utilized in generating gradients for updating a global machine learning (ML) model with respect to the given language based on comparing the audio fingerprint of the audio data stream to an audio fingerprint database; and In response to determining that the audio data stream has not been previously utilized in generating gradients for updating the global ML model with respect to the given language: Processing the audio data stream using the global ML model; generating the gradient using an unsupervised or self-supervised learning technique and based on processing the audio data stream using the global ML model; and The global ML model is updated with respect to the given language based on the gradient.

21. A method implemented by one or more processors of a client device, the method comprising: receiving an audio data stream capturing a stream of spoken speech in a given language from a given radio station; determining whether to implement federated learning or ephemeral learning to generate a gradient for updating a global machine learning (ML) model for the given language based on a connection state between the client device and a remote system; In response to determining to implement federated learning to generate the gradient utilized in updating the global ML model with respect to the given language: processing the audio data stream using an on-device ML model, the on-device ML model being stored in an on-device storage of the client device and being an on-device counterpart of the global ML model; generating the gradient based on processing the audio data stream using the on-device ML model using an unsupervised or self-supervised learning technique; as well as asynchronously transmitting the gradients to the remote system to be utilized in updating the global ML model with respect to the given language; and In response to determining that ephemeral learning is achieved, generating the gradient utilized in updating the global ML model with respect to the given language: processing the audio data stream using the on-device ML model; generating the gradient using the unsupervised or self-supervised learning technique and based on processing the audio data stream using the on-device ML model; as well as The gradients are synchronously transmitted to the remote system to be utilized in updating the global ML model with respect to the given language.

22. The method of claim 21, further comprising: Determining to implement federated learning to generate the gradient utilized in updating the global ML model with respect to the given language is based on the connection status between the client device and the remote system indicating that the client device cannot connect to the remote system.

23. The method of claim 22, wherein asynchronously transmitting the gradient to the remote system to be utilized in updating the global ML model with respect to the given language comprises: After generating the gradient: determining that a connection has been established between the client device and the remote system; as well as In response to determining that the connection has been established between the client device and the remote system: The gradients are transmitted to the remote system to be utilized in updating the global ML model with respect to the given language.

24. The method of any one of claims 21 to 23, further comprising: Determining to implement ephemeral learning to generate the gradient utilized in updating the global ML model with respect to the given language is based on the connection status between the client device and the remote system indicating that the client device is connected to the remote system.

25. The method of claim 24, wherein synchronously transmitting the gradient to the remote system to be utilized in updating the global ML model with respect to the given language comprises: The gradients are transmitted to the remote system to be utilized in updating the global ML model with respect to the given language without subsequently establishing any connection between the client device and the remote system.

26. The method of any one of claims 21 to 25, wherein determining whether to implement federated learning or ephemeral learning to generate gradients for updating the global ML model with respect to the given language is further based on a location of the client device.

27. A method implemented by one or more processors of a client device, the method comprising: receiving an audio data stream capturing a stream of spoken speech in a given language from a given radio station; determining whether to implement federated learning or ephemeral learning to generate a gradient for updating a global machine learning (ML) model for the given language based on a connection state between the client device and a remote system; In response to determining to implement federated learning to generate the gradient utilized in updating the global ML model with respect to the given language: processing the audio data stream using an on-device ML model, the on-device ML model being stored in an on-device storage of the client device and being an on-device counterpart of the global ML model; generating the gradient based on processing the audio data stream using the on-device ML model using an unsupervised or self-supervised learning technique; as well as asynchronously transmitting the gradients to the remote system to be utilized in updating the global ML model with respect to the given language; as well as In response to determining that ephemeral learning is achieved, generating the gradient utilized in updating the global ML model with respect to the given language: and synchronously transmitting the audio data stream to the remote system, wherein synchronously transmitting the audio data stream to the remote system causes the remote system to: Processing the audio data stream using the global ML model; generating the gradient using the unsupervised or self-supervised learning technique and based on processing the audio data stream using the global ML model; and The global ML model is updated based on the gradients.

28. The method of claim 27, wherein synchronously transmitting the audio data stream to the remote system further causes the remote system to: The audio data stream is discarded after generating the gradient.

29. A method implemented by one or more processors of a client device, the method comprising: receiving an audio data stream capturing a spoken speech stream in a given language from a given radio station being actively consumed by a user of the client device; as well as causing gradients to be generated for updating a global machine learning (ML) model at a remote system with respect to the given language, Wherein causing the gradient to be generated comprises: processing the audio data stream using an on-device ML model, the on-device ML model being stored in an on-device storage of the client device and being an on-device counterpart of the global ML model; generating the gradient based on processing the audio data stream using the on-device ML model using an unsupervised or self-supervised learning technique; discarding the audio data stream; and transmitting the gradients to the remote system to be utilized in updating the global ML model with respect to the given language; or Wherein causing the gradient to be generated comprises: transmitting the audio data stream to the remote system, wherein transmitting the audio data stream to the remote system causes the remote system to: Processing the audio data stream using the global ML model; generating the gradient using the unsupervised or self-supervised learning technique and based on processing the audio data stream using the global ML model; discarding the audio data stream; and The global ML model is updated based on the gradients.

30. A method implemented by one or more processors of a remote system, the method comprising: receiving, from a given client device, an audio data stream capturing a stream of spoken speech in a given language, the audio data stream initially received at the given client device from a given radio station being actively consumed by a user of the given client device; Processing the audio data stream using a global machine learning (ML) model; generating gradients based on processing the audio data stream using the global ML model using an unsupervised or self-supervised learning technique; discarding the audio data stream; as well as The global ML model is updated with respect to the given language based on the gradient.

Images