Method and system for training a speech model

Abstract

A method (300) of training a speech model (146) includes receiving a fixed set of training utterances (144) at a voice-enabled device (110), where each training utterance in the fixed set of training utterances includes a transcription (144t) paired with a speech representation (144r) of the corresponding training utterance. The method further includes sampling noisy audio data (202) from an environment of the voice-enabled device. For each training utterance in the fixed set of training utterances, the method further includes augmenting the speech representation of the corresponding training utterance using the noisy audio data sampled from the environment of the voice-enabled device to generate a noisy audio sample (204), and pairing each noisy audio sample with the corresponding transcription of the corresponding training utterance. The method further includes training the speech model on the noisy audio samples generated for each speech representation in the fixed set of training utterances.

Description

Technical Field

[0001] This disclosure relates to speech personalization and joint training using real-world noise. Background Technology

[0002] Automatic speech recognition (ASR) is an important technology used in mobile devices and other devices. Generally, ASR attempts to provide an accurate transcription of what a person has said. Devices implementing ASR can also use hot words or wake words, which help determine when a given utterance is directed at the system, rather than at another person present in the environment.

[0003] A major challenge for ASR systems is that a single model needs to be able to process speech input from all users in a single setting, where there are significant differences in accents, vocabulary, and background noise types among different users. Another challenge for personalized speech recognition models is the need for accurate transcription to label the audio data, which makes it impractical to train the entire speech recognition model from scratch on the user's device. Summary of the Invention

[0004] One aspect of this disclosure provides a method for training a speech model. The method includes receiving a fixed set of training utterances at data processing hardware of a voice-enabled device, wherein each training utterance in the fixed set of training utterances includes a corresponding transcription paired with a corresponding speech representation of the corresponding training utterance. The method also includes sampling noisy audio data from the environment of the voice-enabled device by the data processing hardware. For each training utterance in the fixed set of training utterances, the method includes: augmenting the corresponding speech representation of the corresponding training utterance using the noisy audio data sampled from the environment of the voice-enabled device to generate one or more corresponding noisy audio samples; and pairing each of the one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance by the data processing hardware. The method also includes training the speech model on the one or more corresponding noisy audio samples generated for each speech representation in the fixed set of training utterances by the data processing hardware.

[0005] In some examples, the method further includes denoising the corresponding speech representation by data processing hardware to remove any pre-existing noise before enhancing the corresponding speech representation of the corresponding training utterance. In some implementations, the method further includes, after pairing each of one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance, storing the pairing of each of the one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance by data processing hardware on memory hardware in communication with the data processing hardware.

[0006] In some configurations, when the speech model includes a speech recognition model, the method further includes, for each speech representation in a fixed set of training utterances and for each noisy audio sample of one or more noisy audio samples generated for the corresponding speech representation, the data processing hardware determines a corresponding probability distribution on possible speech recognition hypotheses of the corresponding speech representation or the corresponding noisy audio sample for output by the speech model; and the data processing hardware generates a loss term based on the corresponding probability distribution on possible speech recognition hypotheses of the corresponding speech representation or the corresponding noisy audio sample. In these configurations, training the speech model may include updating the parameters of the speech recognition model using the loss term generated for each speech representation in the fixed set of training utterances and for each noisy audio sample of one or more noisy audio samples generated for each corresponding speech representation in the fixed set of training utterances. Training the speech model may additionally or alternatively include transmitting the loss term generated for each speech representation in the fixed set of training utterances and for each noisy audio sample of one or more noisy audio samples generated for each corresponding speech representation in the fixed set of training utterances to a central server. The central server can be configured to use joint learning to update the parameters of the server-side speech recognition model based on: loss terms received from the data processing hardware of the voice-enabled device; and other loss terms received from other voice-enabled devices, wherein the other loss terms are based on different noisy audio data sampled by the corresponding other voice-enabled device and received from each other voice-enabled device.

[0007] In some examples, for at least one training utterance in a fixed set of training utterances, the method further includes: acquiring, by data processing hardware, a corresponding spoken utterance sampled from the environment of a voice-enabled device, the corresponding spoken utterance being phonetically similar to the corresponding speech representation of the corresponding training utterance and paired with a corresponding transcription, the corresponding transcription being different from the corresponding transcription paired with the corresponding speech representation of the at least one training utterance. In these examples, training a speech model on a fixed set of training utterances and one or more corresponding noisy audio samples is further based on the corresponding spoken utterance acquired for at least one training utterance in the fixed set of training utterances. Acquiring the corresponding spoken utterance for at least one training utterance in the fixed set of training utterances may include: sampling the corresponding spoken utterance from the environment of a voice-enabled device; determining, based on a comparison of a corresponding embedding generated for the corresponding spoken utterance and a corresponding embedding generated for the corresponding speech representation of the at least one training utterance, that the corresponding spoken utterance sampled from the environment is phonetically similar to the corresponding speech representation of the at least one corresponding training utterance; acquiring the corresponding transcription of the corresponding spoken utterance sampled from the environment of a voice-enabled device; and determining that the corresponding transcription of the corresponding spoken utterance is different from the corresponding transcription paired with the corresponding speech representation of the at least one training utterance. A portion of the embedding model or speech model can generate a corresponding embedding for each of the corresponding spoken utterance and the corresponding speech representation of at least one training utterance. In these examples, the corresponding speech representation of at least one training utterance represents the spoken representation of a specific fixed term. The speech model may include a hot word detection model trained to detect specific fixed terms. For the hot word detection model, training the hot word detection model to detect specific fixed terms includes using the corresponding spoken utterance obtained for at least one training utterance as negative training samples.

[0008] Another aspect of this disclosure provides a system for training a speech model. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. These operations include receiving a fixed set of training utterances at a voice-enabled device, wherein each training utterance in the fixed set of training utterances includes a corresponding transcription paired with a corresponding speech representation of the corresponding training utterance. These operations also include sampling noisy audio data from the environment of the voice-enabled device. For each training utterance in the fixed set of training utterances, the operation includes: augmenting the corresponding speech representation of the corresponding training utterance using the noisy audio data sampled from the environment of the voice-enabled device to generate one or more corresponding noisy audio samples; and pairing each of the one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance. The operation also includes training a speech model on the one or more corresponding noisy audio samples generated for each speech representation in the fixed set of training utterances. This aspect may include one or more of the following optional features.

[0009] In some examples, the operation further includes denoising the corresponding speech representation to remove any pre-existing noise before enhancing the corresponding speech representation of the corresponding training utterance. In some implementations, the operation further includes storing, on memory hardware in communication with data processing hardware, the pairing of each of the one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance after pairing each of the one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance.

[0010] In some configurations, when the speech model includes a speech recognition model, the operation further includes, for each speech representation in a fixed set of training utterances and for each noisy audio sample of one or more noisy audio samples generated for the corresponding speech representation, determining a corresponding probability distribution on possible speech recognition hypotheses of the corresponding speech representation or the corresponding noisy audio sample to be output by the speech model; and generating a loss term based on the corresponding probability distribution on possible speech recognition hypotheses of the corresponding speech representation or the corresponding noisy audio sample. In these configurations, training the speech model may include updating the parameters of the speech recognition model using the loss term generated for each speech representation in the fixed set of training utterances and for each noisy audio sample of one or more noisy audio samples generated for each corresponding speech representation in the fixed set of training utterances. Training the speech model may additionally or alternatively include transmitting the loss term generated for each speech representation in the fixed set of training utterances and for each noisy audio sample of one or more noisy audio samples generated for each corresponding speech representation in the fixed set of training utterances to a central server. The central server can be configured to use joint learning to update the parameters of the server-side speech recognition model based on: loss terms received from the data processing hardware of the voice-enabled device; and other loss terms received from other voice-enabled devices, wherein the other loss terms are based on different noisy audio data sampled by the corresponding other voice-enabled device and received from each other voice-enabled device.

[0011] In some examples, for at least one training utterance in a fixed set of training utterances, the operation further includes: acquiring a corresponding spoken utterance sampled from the environment of a voice-enabled device, the corresponding spoken utterance being phonetically similar to the corresponding speech representation of the corresponding training utterance and paired with a corresponding transcription, the corresponding transcription being different from the corresponding transcription paired with the corresponding speech representation of the at least one training utterance. In these examples, training the speech model on the fixed set of training utterances and one or more corresponding noisy audio samples is further based on the corresponding spoken utterance acquired for at least one training utterance in the fixed set of training utterances. Acquiring the corresponding spoken utterance for at least one training utterance in the fixed set of training utterances may include: sampling the corresponding spoken utterance from the environment of a voice-enabled device; determining, based on a comparison of the corresponding embedding generated for the corresponding spoken utterance and the corresponding embedding generated for the corresponding speech representation of the at least one training utterance, that the corresponding spoken utterance sampled from the environment is phonetically similar to the corresponding speech representation of the at least one corresponding training utterance; acquiring the corresponding transcription of the corresponding spoken utterance sampled from the environment of a voice-enabled device; and determining that the corresponding transcription of the corresponding spoken utterance is different from the corresponding transcription paired with the corresponding speech representation of the at least one training utterance. A portion of the embedding model or speech model can generate a corresponding embedding for each of the corresponding spoken utterance and the corresponding speech representation of at least one training utterance. In these examples, the corresponding speech representation of at least one training utterance represents the spoken representation of a specific fixed term. The speech model may include a hot word detection model trained to detect specific fixed terms. For the hot word detection model, training the hot word detection model to detect specific fixed terms includes using the corresponding spoken utterance obtained for at least one training utterance as negative training samples.

[0012] Implementations of the system or method may include one or more of the following optional features. In some implementations, sampling noisy audio data from the environment of a voice-enabled device includes: randomly sampling noise from the environment of the voice-enabled device immediately before, during, or after a voice interaction between the voice-enabled device and a user associated with the voice-enabled device. In other implementations, sampling noisy audio data from the environment of a voice-enabled device includes: acquiring contexts and / or time windows in which the user of the voice-enabled device frequently interacts with the voice-enabled device; and sampling noisy audio data from the environment of the voice-enabled device during the acquired contexts and / or time windows in which the user of the voice-enabled device frequently interacts with the voice-enabled device. In still other implementations, sampling noisy audio data from the environment of a voice-enabled device includes: randomly sampling noisy audio data from the environment of the voice-enabled device throughout the day; and applying weights to any noisy audio data sampled from the environment during contexts and / or time windows in which the user of the voice-enabled device interacts with the voice-enabled device more frequently. A digital signal processor (DSP) of data processing hardware can sample noisy audio data from the environment of a voice-enabled device. In some examples, the corresponding speech representation of at least one training utterance includes the original audio waveform of the corresponding training utterance. In other examples, the corresponding speech representation of at least one training utterance includes an audio feature representation of the corresponding training utterance.

[0013] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description, the drawings, and the claims. Attached Figure Description

[0014] Figure 1A This is a schematic diagram of an example speech environment used to train a speech model using noisy audio data.

[0015] Figure 1B This is a schematic diagram of an example speech environment that uses joint learning to train a speech model using noisy audio data.

[0016] Figure 2A and 2B This is a schematic diagram of an example enhancement system for a speech environment.

[0017] Figure 3 This is a flowchart illustrating an example of the operational setup for training a speech model.

[0018] Figure 4 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described in this paper.

[0019] The same reference numerals in the various figures indicate the same elements. Detailed Implementation

[0020] As computing devices become ubiquitous in everyday tasks, technologies that enable or facilitate interaction between users and the devices themselves have become commonplace. A more prevalent method is through user interaction with the computing device using their voice. In this sense, various computing devices (also known as voice-enabled devices) are configured to process and perform actions or tasks based on voice-based input. For example, a user can interact with a voice-enabled device to expect the device to generate synthesized speech in response or to perform a task based on a spoken request (also known as a query or command). To illustrate, a user could submit a query to a voice-enabled device by saying "play my morning jazzplaylist" and expect the device to play songs from the user's morning jazz playlist. Since speech is the primary form of human communication, voice-based input is a natural and convenient way to communicate with devices, which can be performed in virtually any setting (e.g., completely hands-free).

[0021] To process speech-based input, computing devices are equipped with speech recognition capabilities. One challenge these speech recognition systems face is that, to accurately recognize speech that could contribute to a good user experience, they need to be able to handle speech input that comes from dictation variations and often contains a degree of background noise. In other words, a robust speech recognition system can accurately extract the context of a user's speech, even if the speech from that particular user includes, for example, dialectal differences (e.g., accent, word choice, word stress, lexical differences, etc.) compared to another user using the same speech recognition technology.

[0022] Furthermore, voice-enabled devices can include some degree of voice-based personalization. To illustrate, a single voice-enabled device can be used by multiple users in the same environment. For example, the voice-enabled device belongs to a family of three, including a father, mother, and teenage son. Here, the teenage son might be a bit playful and try to modify the default hotword associated with the device, which activates the device from sleep mode to perform voice processing (i.e., wakes the device to listen to voice commands or queries). For example, the son might change the default hotword "Hey Google" to the phrase "Hey Mike," thinking it would be funny if his father felt frustrated when the device didn't listen to him. What the son doesn't know is that the device might be configured so that each user or registered user of the device has a speaker profile associated with it. In this example, the device might have a voice model specific to the son (e.g., a speech recognition model or a hotword model). In this case, the son might not realize that the device recognizes his voice when he changes the hotword, so the device actually changes the hotword configuration of the son's profile, rather than changing the hotword associated with the father's profile. Therefore, the son might feel confused when his father uses the default hot phrase "Hey Google" to interact with the device in a non-frustrated way. The joke could backfire completely when the son forgets the hot phrase "Hey Mike" and gets frustrated when the device ignores him when he repeats "Hey Google."

[0023] Unfortunately, to develop a speech model that understands the speech of a specific user among other users, the speech model needs to be trained on audio training data corresponding to the user's speech (e.g., recorded speech data). Furthermore, the robustness of a user-specific model is typically based on the amount of audio training data that includes the user's speech. In other words, a speech model that learns the characteristics and / or features of user speech requires user speech samples to identify those characteristics and ensure that they can be repeatedly and / or reliably identified when the user speaks to a speech-enabled device.

[0024] In one approach, a standard speech model typically learns to recognize speakers using a large corpus of training samples corresponding to human speech. After training the standard speech model on speech samples, an auxiliary process can further refine the user-specific standard speech model by training it on user-specific audio samples. This auxiliary process is sometimes called fine-tuning because it takes a coarsely trained model (i.e., the standard speech model) and adjusts this coarsely trained model to perform speech processing more accurately for user-specific speech.

[0025] The fine-tuning process can occur locally (meaning at the device itself where voice is enabled) or remotely (meaning at a location far from the voice-enabled device, such as a remote server). Users can utilize the local fine-tuning process when user-specific audio samples reside locally on the device (i.e., on the voice-enabled device) rather than in remote storage. On-device fine-tuning is effective when user-specific audio samples reside locally because they do not need to be transmitted to a remote system to perform the fine-tuning. This avoids the local resource consumption required to transmit user-specific audio samples to a remote system. Furthermore, users may prefer to retain ownership of their user-specific audio samples (e.g., based on privacy concerns) rather than handing them over to a remote service for fine-tuning; mitigating potential privacy issues or potential misuse of user data.

[0026] An example of a local approach for fine-tuning is federated learning. Federated learning refers to a process where a central source (e.g., a remote server or some coordinating device) exists, but each user-specific device retains its own learning autonomy, thus preserving user data autonomy. With federated learning, voice-enabled devices can collaboratively learn a shared predictive model while storing all their own training data on their devices. In other words, the machine learning process using fine-tuning is decoupled from the need for remotely stored data. In some examples, the federated learning process requires the voice-enabled device to download the current speech model (e.g., a coarsely trained standard speech model) and improve it by learning from audio data on the device. Here, the audio data on the device is user-specific audio samples. In some examples, once the fine-tuning process of federated learning occurs on the device, the device communicates any updates made to the current speech model (e.g., a coarsely trained standard speech model) to the central source. For example, the device sends model gradient updates to the central source via privacy-preserving optimization methods (e.g., via encrypted communication). As each device deploys a speech model, each device can contribute updates to the current speech model from the fine-tuning process to a central source. This allows the central source to aggregate all updates across all devices to improve the current model, while keeping the updated model as a shared model for all devices. The federated learning process can also maintain the privacy of updates sent to the central server by immediately combining updates from one device with updates from another (e.g., averaging the updates together) to hide potentially unique updates from any single device.

[0027] While fine-tuning provides another layer of voice personalization, the process inherently requires a large amount of voice data for each user to ensure accuracy. For example, some neural network-based models require tens of hours of recorded speech to generate a high-fidelity model specific to a single user. However, there are unavoidable situations where large amounts of recorded speech data are neither available nor publicly accessible for training voice models. This makes it difficult to fine-tune the model for each user. Another complicating factor is that some users may rarely interact with voice-enabled devices. Due to infrequent interaction, even on-device learning techniques may not be able to collect enough data to train an accurate model for a particular user. Furthermore, even when a user's voice-enabled device collects user-specific audio samples, these samples may include noisy data, where background noise (e.g., speech from other users) exists in addition to the user's voice. As a strategy to isolate a particular user's voice from the audio data at the cost of additional processing resources, some methods may attempt to eliminate or filter background noise.

[0028] To overcome the limitations of a limited number of speech samples containing user voice for personalizing speech models, the fine-tuning process can leverage noisy data available on voice-enabled devices to better personalize the speech model to the user's environment. By utilizing noisy data available on voice-enabled devices, the speech model can be trained with some form of noisy data to make it more robust to the types of noise typically encountered by the user's voice-enabled device. Furthermore, instead of attempting to remove noisy data contained in the audio samples, the training process can use this noise as an additional means of recognizing the user and / or their environment. In other words, this noisy data training approach can improve speech recognition accuracy without any labeled data or frequent user interaction. Another advantage of this approach is that training using noisy data can be easily incorporated into the fine-tuning process, such as on-device speech personalization and / or joint learning.

[0029] refer to Figure 1A and 1BIn some implementations, the voice system 100 includes a user 10 transmitting spoken utterance 12 to a voice-enabled device 110 (also referred to as device 110 or user device 110). The user 10 (i.e., the speaker of utterance 12) can speak utterance 12 as a query or command to solicit a response from device 110 or to have device 110 perform a task based on the content of utterance 12. Device 110 is configured to capture audible sounds from one or more users 10 within the voice system 100. The audible sounds may refer to the spoken utterance 12 of user 10, which serves as an audible query, a command to device 110, or audible communication captured by device 110. The audible sounds may also include ambient noise 14 captured at device 110 based on the audio capture sensitivity of device 110. A voice-enabled interface (e.g., a digital assistant) can process queries or commands by answering them and / or causing the execution of commands.

[0030] Here, device 110 is configured to capture audio data corresponding to speech 12 and / or noise 14 in the environment surrounding device 110. The audio data may characterize spoken speech 12, ambient noise 14, or some combination of both. Device 110 may correspond to any computing device associated with user 10 and capable of receiving audio signals. Some examples of user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-book readers, etc.), computers, wearable devices (e.g., smartwatches), music players, casting devices, smart appliances (e.g., smart TVs), and Internet of Things (IoT) devices, remote controls, smart speakers, etc. Device 110 includes data processing hardware 112 and memory hardware 114 that communicates with and stores instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations related to speech processing.

[0031] Device 110 also includes an audio subsystem with an audio capture device (e.g., an array of one or more microphones) 116 for capturing audio data 12, 14 within the speech environment and converting it into electrical signals. While device 110 implements the audio capture device 116 in the illustrated example, the audio capture device 116 may not physically reside on device 110 but may communicate with the audio subsystem (e.g., a peripheral device of device 110). For example, device 110 may correspond to a vehicle infotainment system utilizing a microphone array distributed throughout the vehicle's location.

[0032] Furthermore, device 110 is configured to communicate with remote system 130 via network 120. Remote system 130 may include remote resources 132, such as remote data processing hardware 134 (e.g., a remote server or CPU) and / or remote memory hardware 136 (e.g., a remote database or other storage hardware). Device 110 can utilize remote resources 132 to perform various functions related to speech processing. For example, device 110 is configured to perform a training process 200 to train a speech model 146 for user personalization tailored to device 110. These systems 140, 200 may reside on device 110 (referred to as on-device systems), reside remotely (e.g., reside on remote system 130) but communicate with device 110, or some combination thereof. For example, Figure 1B The illustrated remote system 130 serves as a central server for implementing the speech system 140, enabling the sharing of the speech model 146 associated with the speech system 140 among multiple devices 110, 110a-n (e.g., shown as devices 110, 110a-c). In some embodiments, the speech system 140 resides on the remote system 130 such that the speech system 140 can initially train the speech model 146 on a large corpus of training data (e.g., training speech utterances) accessible to the remote system 130 (e.g., stored in remote memory hardware 136). This allows the initial training of the model 146 to utilize the more powerful computing resources available at the remote server, which may not be available at the voice-enabled device 110. However, the speech system 140 can also be executed locally on device 102 to initially train the speech model 146 on device 102 without requiring the remote system 130.

[0033] In contrast, the training process 200 is executed locally on each user device 110 and configured to generate user-specific audio training samples for each corresponding user 10. To generate user-specific noisy audio samples for the corresponding user 10, the training process 200 uses audio data 12, 14 captured in the environment of device 110 to augment a fixed set of training utterances 144, 144a-n received from the speech system 140. By executing the training process 200 locally to generate user-specific noisy audio samples, the audio data 12, 14 captured from the environment of the user 10 is stored on the voice-enabled device 110 and is not shared with the remote system 130.

[0034] The initial fixed set of training utterances 144, 144a-n refers to transcribed speech representations sampled from multiple different speakers (i.e., various speakers). Speech representations can be in their original audio form, in a form corresponding to a feature representation (e.g., Mel filter bank energy or micro-features), or some type of intermediate feature of the corresponding speech model. In some examples, training utterances 144 are drawn from publicly available datasets. Each training utterance 144 includes a corresponding speech representation 144r (… Figure 2A and 2B Transcription of the words mentioned in the paired discourse 144t (144t) Figure 2A and 2B The speech representation 144r of utterance 144 can refer to any way of representing the speech features or characteristics contained in utterance 144. For example, speech representation 144r can include embeddings or vector strings, where each dimension of the embedding represents one or more speech features. In some examples, the transcription 144t of each training utterance 144 is manually generated (e.g., handwritten utterances) to help ensure the accuracy level of speech model 146.

[0035] Speech system 140 can be configured to generate various types of speech models 146 based on speech processing to be performed at user device 110. Some examples of speech models 146 include hot word models, speech recognition models, and / or endpoint models. A hot word model is a model trained to detect the presence of a specific word or phrase in the audio data representing spoken utterance 12 without performing speech recognition on the audio data. When a specific word or phrase is present, the hot word model is configured to activate various speech processing systems. In other words, a "hot word" refers to a specific word or phrase that, when detected by the hot word model, triggers the activation of one or more speech processing systems to process the hot word and / or one or more other words following the hot word in spoken utterance 12. When speech system 140 trains hot word model 146, training utterance 144 includes labels indicating whether a hot word is present in a particular training utterance 144 (e.g., positive training utterance 144) or not present in a particular training utterance 144 (e.g., negative training utterance 144). Here, each positive training utterance 144 includes a corresponding phonetic representation 144r representing a hot word, while each negative training utterance 144 includes a corresponding phonetic representation 144r that does not represent a hot word. The phonetic representation 144r may or may not be paired with the corresponding transcription 144r.

[0036] On the other hand, the speech recognition model refers to speech model 146, which is trained to recognize terms representing spoken utterance 12 in input audio data to generate a transcription of utterance 12, which can be displayed and / or interpreted to recognize the command or inquiry expressed by the spoken utterance. For example, when the voice-enabled device 110 captures the utterance 12 “what is the weather like today?” spoken by user 10, the speech recognition model is configured to process the audio data representing utterance 112 to interpret the terms in order to determine that user 10 is requesting a report on the current weather.

[0037] In some examples, training a speech recognition model involves using a gradient descent optimization algorithm with a loss function or loss term during iterative processes of updating model parameters (e.g., model weights) to minimize the loss function / term. More specifically, training process 200 may backpropagate an estimate of the loss function gradient to adjust the current parameters of the model's neural network. As the model parameters decrease the loss function, the model can approach an optimal state, where model 146 receives a speech representation 144r and predicts a speech recognition result that matches the corresponding transcription 144t of a given training utterance 144. In some examples, training process 200 generates a loss term based on the predicted speech recognition result compared to the corresponding transcription 144t of the training utterance 144 (e.g., a probability distribution over possible speech recognition assumptions).

[0038] Continue to refer to Figure 1A and 1BUser device 110 receives an untrained or pre-trained speech model 146 from speech system 140. Speech system 140 may push the pre-trained or untrained speech model 146 to device 110, or the pre-trained or untrained speech model 146 may be pre-loaded onto user device 110. Speech system 140 also transmits a fixed set of training utterances 144, 144a-n to device 110. The number of training utterances 144 in the fixed set of training utterances 144 may vary, but as an example, the fixed set of training utterances 144 may include approximately one thousand utterances. Here, each training utterance 144 in the fixed set of training utterances 144 includes a corresponding transcription 144t paired with a corresponding speech representation 144r. Device 110 may further perform a training process 200 to enhance the fixed set of training utterances 144 by using noisy audio data 202 sampled from the environment of voice-enabled device 110 to fine-tune model 146 to personalize model 146 for a specific user 10 of device 110. In some examples, training process 200 filters or uses a subset of a fixed set of training utterances 144. Here, the subset of the fixed set of training utterances 144 may correspond to a certain number of fixed sets of training utterances 144 that have a specific level of relevance to user 10 of device 110 (e.g., the subset corresponds to utterances 144 from speakers having similar characteristics to user 10, such as speech embeddings similar to user 10 of device 110). Training process 200 samples noisy audio data 202 from the environment of voice-enabled device 102, and for each training utterance 144 in the fixed set of training utterances, training process 200 uses the noisy audio data 202 to augment the corresponding speech representation 144r to generate one or more corresponding noisy audio samples 204. Here, each of the one or more corresponding noisy audio samples 204 generated for the corresponding training utterance 144 may be paired with the same corresponding transcription 144t of the corresponding training utterance 144. Therefore, the training process 200 can generate multiple noisy audio samples 202 for each speech representation 144r in a fixed set of training utterances 144, such that each of the noisy audio samples 204 contains noisy audio data 202 unique to the environment of the voice-enabled device 110. Finally, in order to personalize the speech model 146, the training process 200 trains an untrained or pre-trained speech model 146 on one or more corresponding noisy audio samples 204 generated from each speech representation 144r in the fixed set of training utterances 144.

[0039] Training process 200 operates under the following premise: Although none of the training utterances 144 in the fixed set of training utterances 144 have been spoken by user 10 and the number of training utterances 144 is insufficient for robust model training, the number of these training utterances 144 can be doubled by augmenting each speech representation 144r once or multiple times using noisy audio data 202 collected from the environment of user device 110, to provide a sufficient number of personalized training utterances 144 for robustly training the speech model 146. Training process 200 can collect noisy audio data 202 by sampling any audible sound detectable by audio capture device 116 at device 110. Here, noisy audio data 202 may not include user 10's speech, but noisy audio data 202 is still considered user-specific because noisy audio data 202 corresponds to audible noise that may be unique to user 10's environment, and is therefore associated with noise that user device 102 would typically encounter when user 10 speaks. Figure 1A As shown, user 10 may have a dog that produces unique sounds as background noise 14 when user 10 interacts with device 110. For example, when user 10 gives a voice command to device 110, the dog may howl, bark, or whine. By collecting the background noise 14, including the noise emitted by the user's dog, training process 200 can form user-specific noisy audio samples 204. Furthermore, user 10 may also live near a railway crossing, allowing training process 200 to collect noisy audio data 202 associated with sounds produced by train horns, railway warnings, and trains traveling along the tracks.

[0040] Using the collection of noisy audio data 202, the training process 200 can augment the original training utterances 144 with the noisy audio data 202 to generate one or more noisy audio samples 204. These samples are user-specific because they contain noisy audio data 202 specific to the user 10's environment. By augmenting multiple training utterances 144 with noisy audio data 202, the training process 200 can generate a sufficient number of noisy audio samples 204 for training the speech model 146. Based on this augmentation process, the training process 200 is able to generate a large number of noisy audio samples 204 from a finite, fixed number of original training utterances 144, such that the noisy audio samples 204 are variations of one or more samples of noisy audio data 202 combined with one or more training utterances 144 from a fixed set of training utterances 144. The training process 200 can be deployed as a fine-tuning process to generate a personalized speech model 146 at the user device 110 (e.g., as shown in the image). Figure 1A (as shown) or facilitates a shared voice model 146 available to multiple devices 110 at a central location (e.g., as shown) Figure 1B (As shown).

[0041] For details, please refer to the following: Figure 1B When a joint learning approach is used during a training process 200 executed on a cluster of user devices 110, device 110 receives a pre-trained model 146, and a fine-tuning process uses the training process 200 to generate an update 208 to the model 146. This update 208 is then transmitted back to the central location to form shared models 146s. Here, the update 208 is generated in response to the training process 200 that trains model 146 on user-specific noisy audio samples 204. Therefore, each device 110 contributes a different user-specific update 208 based on the corresponding user-specific noisy audio sample 204, so that the remote system 130 can use joint learning to merge all user-specific updates 208 into the shared model 146s. In an example where the shared speech model 146s includes a speech recognition model capable of recognizing speech across a wide range of noisy environments, such as... Figure 1B The illustration shows three devices 110a-c, each belonging to a corresponding user 10, 10a-c. Each device 110 can independently execute a training process 200 to generate a corresponding device-specific update 208 by training an untrained or pre-trained model 146 on user-specific noisy audio samples 204. In the example shown, the training process 200 executed on each device 110a, 110b, 110c generates corresponding device-specific updates 208a, 208b, 208c to be transmitted back to the speech system 140 and applied through joint learning to generate a shared speech model 146s. In some examples, the update 208 includes some representation of the model parameters (e.g., model weights) and / or loss term after training the speech model 146 on user-specific noisy audio samples 204. Based on all updates 208 received by the speech system 140, the speech system 140 generates a shared model 146s that incorporates the updates 208 to some extent.

[0042] Although not shown, training process 200 can train a single model 146 shared among multiple devices 110 owned or accessible to user 10. For example, user 10 owns a smartwatch, a smartphone, and a smart speaker for a total of three devices 110 associated with user 10. Here, training process 200 can collect samples of noisy audio data 202 from each device 110 associated with user 10. Using these samples of noisy audio data 202 from each device 110, training process 200 can train a model 146 shared among these devices 110 to understand user 10's speech. Specifically, training process 200 uses one or more samples of noisy audio data 202 to augment one or more training utterances 144 from a fixed set of training utterances 144 to form noisy audio samples 204 to train the model 146 to be shared among user 10's devices 110. In other words, this approach is similar to a joint learning approach occurring for devices 110 associated with user 10.

[0043] refer to Figure 2A and 2B The training process 200 includes a collection phase 210, an enhancement phase 220, and a training phase 230. The collection phase 210 is configured to collect ambient noise 14 to generate samples of noisy audio data 202. In some examples, the digital signal processor of the data processing hardware 112 of device 110 is used as the collection phase 210 of the training process 200. The collection phase 210 can collect any audible noise in the environment within the detectable range of the audio capture device 116. The collection phase 210 can be configured to collect ambient noise 14 in a variety of ways. In one approach, the collection phase 210 collects ambient noise 14 based on the type of model 146 being fine-tuned using the training process 200. To illustrate, when the training process 200 fine-tunes a pre-trained hot word model to better personalize the hot word model for the environment of the device 110 implementing the hot word model, the collection phase 210 can be configured to collect ambient noise 14 by sampling the noise 14 during the time (e.g., a time window) when the user 10 interacts with the device 110. For example, the collection phase 210 collects samples of noisy audio data 202 within a period of time immediately before, during, or immediately after the voice interaction between user 10 and device 110. Specifically, for a hot word model, the time immediately before or after user 10 utters a hot word in utterance 12 can be a favorable time to sample background noise 14 to generate noisy audio data 202. Figure 2AThe illustration shows that collection phase 210 collects background noise 14, 14a immediately before the spoken utterance 12 to generate a first sample of noisy audio data 202, 202a, and collects background noise 14, 14b immediately after the spoken utterance 12 to generate a second sample of noisy audio data 202, 202b. Conversely, when model 146 is a speech recognition model, collection phase 210 may attempt to sample background noise 14 during the spoken interaction between user 10 and device 110.

[0044] In some implementations, the collection phase 210 is further configured to collect audio samples similar to a specific training utterance 144 from a fixed set of training utterances 144 received from the speech system 140. To collect audio samples similar to a specific training utterance 144, the collection phase 210 may sample spoken utterance 12 at device 110 and determine whether spoken utterance 12 has phonetic similarity to the specific training utterance 144. To make this determination, the collection phase 210 may compare the embedding of spoken utterance 12 with the embedding of training utterance 144 (e.g., speech representation 144r) (e.g., by using an embedding model). In some examples, the collection phase 210 works in conjunction with an embedding model associated with speech model 146 (e.g., an acoustic model (AM) associated with speech model 146) to perform this embedding comparison for phonetic similarity. In some configurations, the collection phase 210 includes identifying a minimum distance between the training utterance 144 and the collected audio samples to be considered a phonetic similarity threshold (e.g., an adjustable phonetic similarity threshold). By incorporating a pronunciation similarity threshold, collection phase 210 avoids potential confusion where negative training samples are actually designated as difficult-to-identify positive samples due to pronunciation similarity. For example, a hot word model trained to detect the hot word "Hey Google" might have a negative training example, "Hey Computer," which could be incorrectly detected by the model. In this example, collection phase 210 could identify some degree of pronunciation similarity between "Hey Google" and "Hey Computer," but the distance between these phrases does not meet the pronunciation similarity threshold. In this example, failing to meet the pronunciation similarity threshold means that the two phrases are not similar enough that "Hey Computer" should be identified as a difficult-to-identify positive example of "Hey Google."

[0045] In some configurations, when collection phase 210 determines a context and / or time window in which user 10 of device 110 frequently interacts with device 110, collection phase 210 collects one or more samples of background noise 14 to form noisy audio data 202. For example, device 110 is located in the home of a first user 10. During a weekday week (e.g., Monday to Friday), user 10 tends to interact with device 110 frequently before the start of the workday (e.g., between 6 a.m. and 8 a.m.) and then after the end of the workday (e.g., between 6 p.m. and 10 p.m.). Here, on Monday to Friday, collection phase 210 may collect a greater number of background noise samples in the time windows of 6 a.m. to 8 a.m. and 6 p.m. to 10 p.m. than in the time window of 8 a.m. to 6 p.m. As a different example, a second user 10 may be a traveling salesperson who tends to use an assistant app on his or her device 110 to navigate to different clients or make sales calls. This means that between 9 a.m. and 5 p.m. on weekdays, user 10 most frequently interacts with device 110 using his or her voice. When collection phase 210 determines these time periods (e.g., time windows) and / or context patterns of interaction behavior, collection phase 210 can be configured to collect samples of background noise 14 during these frequent interaction periods.

[0046] Additionally or alternatively, the collection phase 210 can be configured to randomly sample background noise 14 within the environment of the device 110. In some cases, when the collection phase 210 randomly samples noisy audio data 202 in the environment, the collection phase 210 may apply weights 212 to one or more samples of the noisy audio data 202 when the samples are consistent with the interaction between the user 10 and the device 110. For example, Figure 2A and 2B The illustration shows that first weights 212, 212a can be associated with a first sample 202a of noisy audio data, while second weights 212, 212b can be associated with a second sample 202b of noisy audio data. As an example of when the collection phase 210 applies weights 212, the collection phase 210 applies weights to samples of noisy audio data 202 when samples occur during times when user 10 frequently interacts with device 110. For example, when the collection phase 210 collects random samples of noisy audio data 202 for the first user 10 between 6:00 AM and 8:00 AM as described above, the collection phase 210 applies weights 212 to random samples of noisy audio data 202 in order to identify samples that are more likely to be relevant when training model 146.

[0047] Enhancement phase 220 is configured to generate one or more noisy audio samples 204 for each speech representation 144r in a fixed set of training utterances 144. Enhancement phase 220 generates each noisy audio sample 204 by combining noisy audio data 202 sampled by collection phase 210 with the speech representation 144r of the corresponding training utterance 144. Here, noisy audio samples 204 can be formed from a single sample or multiple samples of noisy audio data 202 collected in collection phase 210, such that a single noisy audio sample 204 can be constructed from any number of samples of noisy audio data 202 mixed together at once. For example, a first sample of noisy audio data 202 corresponding to a dog barking can be mixed with a second sample of noisy audio data 202 corresponding to a baby crying, and then combined with the training utterances 144 from the fixed set of training utterances 144 to form a single noisy audio sample 204. The enhancement phase 220 may randomly mix noisy audio data 202 with training utterances 144 or systematically mix noisy audio data 202 with training utterances 144. As briefly discussed above, the enhancement phase 220 may filter or modify the training utterances 144 from a fixed set when forming noisy audio samples 204. This may include splitting or trimming the training utterances 144 when combining the training utterances 144 with one or more samples of noisy audio data 202.

[0048] Once enhancement phase 220 generates noisy audio sample 204, it can pair noisy audio sample 204 with the corresponding transcription 144t of training utterance 144, which in turn pairs with the speech representation 144r enhanced by noisy audio data 202. In other words, the newly formed noisy audio sample 204 can have the label of the underlying training utterance 144. To illustrate, if training utterance 144 is labeled as a positive training example for training a hot word model, then when the corresponding speech representation 144r of that training utterance 144 is enhanced by noisy audio data 202 to form noisy audio sample 204, noisy audio sample 204 will also be a positive training example additionally combined with background noise 14 of device 110 specific to user 10. Since the substance of the positive training example of the hot word remains unchanged, the label of training utterance 144 is preserved for noisy audio sample 204. The same concept applies similarly to the transcription 144t corresponding to training utterance 144. For example, each of one or more noisy audio samples 204 generated for a speech representation 144r may have a corresponding transcription 144t paired with that speech representation 144r as a ground truth label for training a speech recognition model.

[0049] Training phase 230 receives noisy audio samples 204, 204a-n from enhancement phase 220 for generating a personalized speech model 146P. In some examples, training phase 230 locally caches the noisy audio samples 204 and the untrained or pre-trained speech model 146 on device 110 (e.g., in the memory hardware 114 of device 110). Training phase 230 performs fine-tuning of the speech model 146 to form the personalized speech model 146P. Training phase 230 can use the noisy audio samples 204 to fine-tune the speech model 146 using techniques such as gradient descent, wherein training phase 230 iteratively generates a loss term and updates parameters to provide the personalized speech model 146P.

[0050] In such Figure 2B In some configurations, the training process 200 also includes a denoising stage 240, which is configured to denoise the training utterances 144 from a fixed set of training utterances 144 from the speech system 140. Denoising refers to the process of removing existing noise. By removing existing noise, the denoising stage 240 helps ensure that the background noise 14 used by the enhancement stage 220 to generate noisy audio samples 204 is not confused with the noise associated with one or more training utterances 144. In other words, if the denoising stage 240 does not remove existing noise within one or more training utterances 144, the existing noise may propagate errors during the training stage 230, which could potentially impair the training effectiveness of the noisy audio samples 204 for generating personalized speech models 146P. Furthermore, since the training utterances 144 are unlikely to be utterances 12 spoken by the user 10 of the device 110, the existing noise may be misinterpreted or weighted as user-specific. When removing existing noise, this removal in denoising stage 240 can occur before enhancement stage 220, allowing generator 220 to denoise training utterance 144 before enhancement stage 220 combines training utterance 114 with noisy audio data 202 to form noisy audio sample 204. Otherwise, generator 230 may have difficulty separating noise from training utterance 144 when background noise 14 is also introduced using noisy audio data 202.

[0051] In some implementations, the training process 200 further implements a machine learning model that learns how to best generate noise for a specific training utterance 144 based on the noisy audio data 202 collected by the collection phase 210. In these implementations, the machine learning model is combined with the enhancement phase 220 as a noise generator operation to generate noisy audio samples 204, which are optimized for use as training examples for generating personalized speech models 146P. This ability to learn to generate noisy audio samples depends on the fact that a particular type of background noise 14 may be more unique to the user's environment than other types.

[0052] Figure 3 This is a flowchart of an exemplary operational arrangement of a method 300 for training a speech model 146. In operation 302, method 300 receives a fixed set of training utterances 144, 144a-n at a speech-enabled device 110, wherein each training utterance 144 in the fixed set of training utterances 144, 144a-n includes a corresponding transcription 144t paired with a corresponding speech representation 144r of the corresponding training utterance 144. In operation 304, method 300 samples noisy audio data 202 from the environment of the speech-enabled device 110. For each training utterance 144 in the fixed set of training utterances 144, 144a-n, method 300 performs operations 306, 306a-b. In operation 306a, method 300 uses the noisy audio data 202 sampled from the environment of the speech-enabled device 110 to augment the corresponding speech representation 144r of the corresponding training utterance 144 to generate one or more corresponding noisy audio samples 204. In operation 306b, method 300 pairs each of one or more corresponding noisy audio samples 204 with a corresponding transcription 144t of the corresponding training utterance 144. In operation 308, method 300 trains speech model 146 on one or more corresponding noisy audio samples 204 generated for each speech representation 144r in a fixed set for training utterances 144, 144a-n.

[0053] Figure 4 This is a schematic diagram of an example computing device 400 that can be used to implement the systems (e.g., voice system 140) and methods (e.g., training process 200 and / or method 300) described in this document. The computing device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the ways in which the inventions described and / or claimed in this document can be implemented.

[0054] Computing device 400 includes a processor 410, a memory 420, a storage device 430, a high-speed interface / controller 440 connected to the memory 420 and a high-speed expansion port 450, and a low-speed interface / controller 460 connected to a low-speed bus 470 and the storage device 430. Each of components 410, 420, 430, 440, 450, and 460 is interconnected using various buses and can be mounted on a common motherboard or otherwise suitably mounted. Processor 410 can process instructions for execution within computing device 400, including instructions stored in memory 420 or on storage device 430 for displaying graphical information for a graphical user interface (GUI) on an external input / output device such as a display 480 coupled to the high-speed interface 440. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and various types of memory, can be suitably used. Furthermore, multiple computing devices 400 can be connected, each providing a portion of the necessary operation (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0055] Memory 420 stores information non-transitory within computing device 400. Memory 420 may be a computer-readable medium, a volatile storage unit, or a non-volatile storage unit. Non-transitory memory 420 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0056] Storage device 430 provides mass storage for computing device 400. In some embodiments, storage device 430 is a computer-readable medium. In various embodiments, storage device 430 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory, or other similar solid-state storage device or device array (including devices in a storage area network or other configuration). In other embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 420, storage device 430, or memory on processor 410.

[0057] High-speed controller 440 manages bandwidth-intensive operations of computing device 400, while low-speed controller 460 manages less bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 440 is coupled to memory 420, display 480 (e.g., via a graphics processor or accelerator), and high-speed expansion port 450 (which can accept various expansion cards (not shown)). In some embodiments, low-speed controller 460 is coupled to storage device 430 and low-speed expansion port 490. Low-speed expansion port 490, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices (e.g., switches or routers), for example, via a network adapter.

[0058] The computing device 400 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 400a or multiple times as a laptop computer 400b in a set of such servers 400a, or as part of a rack server system 400c.

[0059] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, which includes at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0060] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0061] The processes and logical flows described in this specification can be executed by one or more programmable processors (also known as data processing hardware) that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logical flows can also be executed by special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data or operatively coupled to it to receive data or transfer data to it, or both, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0062] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen) for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending files to and receiving files from the device used by the user, for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's client device.

[0063] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A method for training a speech model, comprising: At the data processing hardware of a voice-enabled device, a trained speech model and a fixed set of training utterances are received from a remote system. The fixed set of training utterances is used by the remote system to train the speech model for use by multiple voice-enabled devices. Each training utterance in the fixed set of training utterances includes a corresponding transcription paired with a corresponding speech representation of that corresponding training utterance. The data processing hardware samples noisy audio data from the environment of the voice-enabled device, wherein sampling the noisy audio data from the environment of the voice-enabled device includes: Randomly sample the noisy audio data from the environment of the voice-enabled device throughout the day; and Weights are applied to any randomly sampled noisy audio data from the environment during a context and / or time window when the user of the voice-enabled device interacts with the voice-enabled device more frequently, in order to identify that the randomly sampled noisy audio data is more likely to be relevant when training the speech model. For each training utterance in the fixed set of training utterances: The data processing hardware uses the noisy audio data sampled from the environment of the voice-enabled device to augment the corresponding speech representation of the training utterance to generate one or more corresponding noisy audio samples; and The data processing hardware pairs each of the one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance; and The speech model for the voice-enabled device is personalized by the data processing hardware based on one or more corresponding noisy audio samples generated for each speech representation in a fixed set for the training utterances.

2. The method according to claim 1, wherein, Sampling the noisy audio data from the environment of the voice-enabled device includes at least one of the following: immediately before, during, or immediately after a voice interaction between the voice-enabled device and the user associated with the voice-enabled device; and randomly sampling noise from the environment of the voice-enabled device.

3. The method according to claim 1, wherein, Samples of the noisy audio data from the environment of the voice-enabled device include: Identify the context and / or time window when the user of the voice-enabled device frequently interacts with the voice-enabled device; Determine one of the contexts and / or time windows in which the user of the voice-enabled device frequently interacts with the voice-enabled device; and In response to determining that the scene and / or one of the time windows is occurring, the noisy audio data is sampled from the environment of the voice-enabled device.

4. The method according to claim 1, wherein, The digital signal processor (DSP) of the data processing hardware samples the noisy audio data from the environment of the voice-enabled device.

5. The method according to claim 1, further comprising: Before enhancing the corresponding speech representation of the corresponding training utterance, the data processing hardware denoises the corresponding speech representation to remove any previously present noise.

6. The method according to claim 1, further comprising: When the speech model includes a speech recognition model, for each speech representation in the fixed set of training utterances and each noisy audio sample of the one or more noisy audio samples generated for the corresponding speech representation: The data processing hardware determines the corresponding probability distribution on the possible speech recognition hypotheses of the corresponding speech representation or the corresponding noisy audio sample, and outputs it by the speech model. as well as The data processing hardware generates a loss term based on the corresponding probability distribution on the possible speech recognition assumptions of the corresponding speech representation or the corresponding noisy audio sample.

7. The method according to claim 6, wherein, Personalizing the speech model includes updating the parameters of the speech recognition model using a loss term generated for each speech representation in a fixed set of training utterances and for each of the one or more noisy audio samples generated for each corresponding speech representation in the fixed set of training utterances.

8. The method according to claim 6, wherein, The personalized speech model includes: The central server transmits the loss term generated for each speech representation in the fixed set of training utterances and for each of the one or more noisy audio samples generated for each corresponding speech representation in the fixed set of training utterances. The central server is configured to use joint learning to update the parameters of the server-side speech recognition model based on the following: The loss term received from the data processing hardware of the voice-enabled device; and Additional loss terms received from other voice-enabled devices, the additional loss terms being based on different noisy audio data sampled by the other voice-enabled devices and received from each other voice-enabled device.

9. The method according to claim 1, wherein, The corresponding speech representation of at least one training utterance includes the original audio waveform of the corresponding training utterance.

10. The method according to claim 1, wherein, The corresponding speech representation of at least one training utterance includes the audio feature representation of the corresponding training utterance.

11. The method of claim 1, further comprising, for at least one training utterance in a fixed set of training utterances: The data processing hardware acquires corresponding spoken utterances sampled from the environment of the voice-enabled device, the corresponding spoken utterances being: The corresponding phonetic representation that is phonetically similar to the corresponding training utterance; and Paired with a corresponding transcription, which is different from the corresponding transcription paired with the corresponding speech representation of the at least one training utterance. in, Personalizing the speech model on a fixed set of training utterances and one or more corresponding noisy audio samples is further based on the corresponding spoken utterances obtained for at least one training utterance in the fixed set of training utterances.

12. The method according to claim 11, wherein, Obtaining the corresponding spoken utterance for at least one training utterance from the fixed set of training utterances includes: The corresponding spoken words are sampled from the environment of the voice-enabled device; Based on a comparison of the corresponding embedding generated for the corresponding spoken utterance and the corresponding embedding generated for the corresponding speech representation of the at least one training utterance, it is determined that the corresponding spoken utterance sampled from the environment is phonetically similar to the corresponding speech representation of the at least one corresponding training utterance; Acquire the corresponding transcription of the spoken utterance sampled from the environment of the voice-enabled device; and The corresponding transcription of the corresponding spoken utterance is determined to be different from the corresponding transcription paired with the corresponding speech representation of the at least one training utterance.

13. The method according to claim 12, wherein, The embedding model, or a portion thereof, generates the corresponding embedding for each of the corresponding spoken utterance and the corresponding speech representation of the at least one training utterance.

14. The method of claim 11, wherein: The corresponding phonetic representation of the at least one training utterance represents the spoken representation of a specific fixed term; The speech model includes a hot word detection model trained to detect specific fixed terms; and The personalized hot word detection model for detecting the specific fixed term includes using the corresponding spoken utterance obtained for the at least one training utterance as a negative training sample.

15. The method according to any one of claims 1-14, further comprising: After each of the one or more corresponding noisy audio samples is paired with the corresponding transcription of the corresponding training utterance, the pairing of each of the one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance is stored by the data processing hardware on memory hardware in communication with the data processing hardware.

16. A system for training a speech model, comprising: Data processing hardware; as well as Memory hardware communicating with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: At a voice-enabled device, a trained speech model and a fixed set of training utterances are received from a remote system. The fixed set of training utterances is used by the remote system to train the speech model for use by multiple voice-enabled devices. Each training utterance in the fixed set of training utterances includes a corresponding transcription paired with a corresponding speech representation of that corresponding training utterance. Sample noisy audio data from the environment of the voice-enabled device, wherein sampling the noisy audio data from the environment of the voice-enabled device includes: Randomly sample the noisy audio data from the environment of the voice-enabled device throughout the day; and Weights are applied to any randomly sampled noisy audio data from the environment during a context and / or time window when the user of the voice-enabled device interacts with the voice-enabled device more frequently, in order to identify that the randomly sampled noisy audio data is more likely to be relevant when training the speech model. For each training utterance in the fixed set of training utterances: The data processing hardware uses the noisy audio data sampled from the environment of the voice-enabled device to augment the corresponding speech representation of the training utterance to generate one or more corresponding noisy audio samples; and The data processing hardware pairs each of the one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance; and The speech model is personalized for the voice-enabled device based on one or more corresponding noisy audio samples generated for each speech representation in a fixed set of training utterances.

17. The system according to claim 16, wherein, Sampling the noisy audio data from the environment of the voice-enabled device includes randomly sampling noise from the environment of the voice-enabled device at least one before, during, or after a voice interaction between the voice-enabled device and the user associated with the voice-enabled device.

18. The system according to claim 16, wherein, Samples of the noisy audio data from the environment of the voice-enabled device include: Identify the context and / or time window when the user of the voice-enabled device frequently interacts with the voice-enabled device; Determine one of the contexts and / or time windows in which the user of the voice-enabled device frequently interacts with the voice-enabled device; and In response to determining that the scene and / or one of the time windows is occurring, the noisy audio data is sampled from the environment of the voice-enabled device.

19. The system according to claim 16, wherein, The digital signal processor (DSP) of the data processing hardware samples the noisy audio data from the environment of the voice-enabled device.

20. The system according to claim 16, wherein, The operation further includes: denoising the corresponding speech representation to remove any previously existing noise before enhancing the corresponding speech representation of the corresponding training utterance.

21. The system according to claim 16, wherein, The operation further includes: when the speech model includes a speech recognition model, for each speech representation in a fixed set of training utterances and each noisy audio sample in one or more noisy audio samples generated for the corresponding speech representation: Determine the corresponding probability distribution on possible speech recognition hypotheses for the corresponding speech representation or the corresponding noisy audio sample, to be output by the speech model; and The loss term is generated based on the corresponding probability distribution on the possible speech recognition assumptions of the corresponding speech representation or the corresponding noisy audio sample.

22. The system according to claim 21, wherein, Personalizing the speech model includes updating the parameters of the speech recognition model using a loss term generated for each speech representation in a fixed set of training utterances and for each of the one or more noisy audio samples generated for each corresponding speech representation in the fixed set of training utterances.

23. The system according to claim 21, wherein, The personalized speech model includes: The central server transmits the loss term generated for each speech representation in the fixed set of training utterances and for each of the one or more noisy audio samples generated for each corresponding speech representation in the fixed set of training utterances. The central server is configured to use joint learning to update the parameters of the server-side speech recognition model based on the following: The loss term received from the data processing hardware of the voice-enabled device; and Additional loss terms received from other voice-enabled devices, the additional loss terms being based on different noisy audio data sampled by the other voice-enabled devices and received from each other voice-enabled device.

24. The system according to claim 16, wherein, The corresponding speech representation of at least one training utterance includes the original audio waveform of the corresponding training utterance.

25. The system according to claim 16, wherein, The corresponding speech representation of at least one training utterance includes an audio feature representation of the corresponding training utterance.

26. The system according to claim 16, wherein, The operation further includes: for at least one training utterance in the fixed set of training utterances, Obtain the corresponding spoken utterance sampled from the environment of the voice-enabled device, the corresponding spoken utterance being: The pronunciation is similar to the corresponding speech representation of the corresponding training utterance; and Paired with a corresponding transcription, which is different from the corresponding transcription paired with the corresponding speech representation of the at least one training utterance. Personalizing the speech model on the fixed set of training utterances and the one or more corresponding noisy audio samples is further based on the corresponding spoken utterance obtained for at least one training utterance in the fixed set of training utterances.

27. The system according to claim 26, wherein, Obtaining the corresponding spoken utterance for at least one training utterance from the fixed set of training utterances includes: The corresponding spoken words are sampled from the environment of the voice-enabled device; Based on a comparison of the corresponding embedding generated for the corresponding spoken utterance and the corresponding embedding generated for the corresponding speech representation of the at least one training utterance, it is determined that the corresponding spoken utterance sampled from the environment is phonetically similar to the corresponding speech representation of the at least one corresponding training utterance; Acquire the corresponding transcription of the spoken utterance sampled from the environment of the voice-enabled device; and The corresponding transcription of the spoken utterance is determined to be different from the corresponding transcription paired with the corresponding speech representation of the at least one training utterance.

28. The system according to claim 27, wherein, The embedding model, or a portion thereof, generates the corresponding embedding for each of the corresponding spoken utterance and the corresponding speech representation of the at least one training utterance.

29. The system according to claim 26, wherein: The corresponding phonetic representation of the at least one training utterance represents the spoken representation of a specific fixed term; The speech model includes a hot word detection model trained to detect specific fixed terms; and The personalized hot word detection model for detecting the specific fixed term includes using the corresponding spoken utterance obtained for the at least one training utterance as a negative training sample.

30. The system according to any one of claims 16-29, wherein, The operation further includes: after pairing each of the one or more corresponding noisy audio samples with the corresponding transcription of the corresponding training utterance, the data processing hardware stores the pairing of each of the one or more corresponding noisy samples with the corresponding transcription of the corresponding training utterance on memory hardware in communication with the data processing hardware.

Images