Knowledge Distillation with Domain Mismatch for Speech Recognition

Abstract

The method (300) includes receiving distilled data (220) including a plurality of out-of-domain training utterances (222). For each particular out-of-domain training utterance in the distilled data, the method includes generating a corresponding augmented out-of-domain training utterance (232) and generating a pseudo-label (240) corresponding to the corresponding augmented out-of-domain training utterance using a teacher ASR model (170) trained with training data corresponding to the target domain. The method also includes distilling a student ASR model (210) from the teacher ASR model by training the student ASR model using the corresponding augmented out-of-domain training utterance paired with the corresponding pseudo-label generated by the teacher ASR model.

Description

[Technical Field]

[0001] The present disclosure relates to training speech recognition models. [Background technology]

[0002] Speech recognition systems are increasingly being used to transcribe speech into text in many everyday applications. These speech recognition systems may be embedded in user devices such as smart home devices or smartphones, or may be used in cloud-related services. Summary of the Invention

[0003] One aspect of the present disclosure provides a computer-implemented method for training a speech recognition model. The method, when executed by data processing hardware, causes the data processing hardware to perform operations. The operations include receiving distilled data including a plurality of out-of-domain training utterances. For each particular out-of-domain training utterance in the distilled data, the operations include generating a corresponding augmented out-of-domain training utterance and generating a pseudo-label corresponding to the corresponding augmented out-of-domain training utterance using a teacher automatic speech recognition (ASR) model trained with training data corresponding to a target domain. The operations also include distilling a student ASR model from the teacher ASR model by training the student ASR model using the corresponding augmented out-of-domain training utterance paired with the corresponding pseudo-label generated by the teacher ASR model.

[0004] Implementations of computer-implemented methods or systems of the present disclosure may include one or more of the following optional features. In some embodiments, distilling a student ASR model from a teacher ASR model includes training the student ASR model to recognize utterances corresponding to the target domain. In some examples, before receiving the distilled data, the teacher ASR model is trained with training data including a plurality of target-domain training utterances, and after receiving the distilled data, the training data is unavailable to the teacher ASR model and the student ASR model. In some embodiments, the training data includes a plurality of target-domain training utterances, and one or more of the target-domain training utterances of the plurality of target-domain training utterances are not included in the distilled data. In some examples, augmenting each out-of-domain training utterance includes at least one of adding noise, adding reverberation, or manipulating timing.

[0005] In some embodiments, the student ASR model executes in a cloud computing environment, and the teacher ASR model executes on one or more user devices communicating with the cloud computing environment. The training data may include a plurality of target domain training utterances, each of which is received at each of the one or more user devices. In some examples, generating a pseudo-label corresponding to the corresponding augmented out-of-domain training utterance includes generating a probability distribution over possible speech recognition hypotheses, wherein the pseudo-label includes the N-best speech recognition hypotheses having the highest probability. In some embodiments, distilling the student ASR model from the teacher ASR model includes, for each particular augmented out-of-domain training utterance, generating a transcription corresponding to the particular augmented out-of-domain training utterance using the student ASR model; generating a loss based on the pseudo-label corresponding to the particular augmented out-of-domain training utterance and the transcription corresponding to the particular augmented out-of-domain training utterance; and updating parameters of the student ASR model using the loss. The loss may be at least one of a cross-entropy loss, a Kullback-Leibler (KL) divergence loss, or an L2 loss.

[0006] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving distilled data including a plurality of out-of-domain training utterances. For each particular out-of-domain training utterance in the distilled data, the operations include generating a corresponding augmented out-of-domain training utterance and generating a pseudo-label corresponding to the corresponding augmented out-of-domain training utterance using a teacher automatic speech recognition (ASR) model trained with training data corresponding to a target domain. The operations also include distilling a student ASR model from the teacher ASR model by training the student ASR model using the corresponding augmented out-of-domain training utterance paired with the corresponding pseudo-label generated by the teacher ASR model.

[0007] Implementations of computer-implemented methods or systems of the present disclosure may include one or more of the following optional features. In some embodiments, distilling a student ASR model from a teacher ASR model includes training the student ASR model to recognize utterances corresponding to the target domain. In some examples, before receiving the distilled data, the teacher ASR model is trained with training data including a plurality of target-domain training utterances, and after receiving the distilled data, the training data is unavailable to the teacher ASR model and the student ASR model. In some embodiments, the training data includes a plurality of target-domain training utterances, and one or more of the target-domain training utterances of the plurality of target-domain training utterances are not included in the distilled data. In some examples, augmenting each out-of-domain training utterance includes at least one of adding noise, adding reverberation, or manipulating timing.

[0008] In some embodiments, the student ASR model executes in a cloud computing environment, and the teacher ASR model executes on one or more user devices communicating with the cloud computing environment. The training data may include a plurality of target domain training utterances, each of which is received at each of the one or more user devices. In some examples, generating a pseudo-label corresponding to the corresponding augmented out-of-domain training utterance includes generating a probability distribution over possible speech recognition hypotheses, wherein the pseudo-label includes the N-best speech recognition hypotheses having the highest probability. In some embodiments, distilling the student ASR model from the teacher ASR model includes, for each particular augmented out-of-domain training utterance, generating a transcription corresponding to the particular augmented out-of-domain training utterance using the student ASR model; generating a loss based on the pseudo-label corresponding to the particular augmented out-of-domain training utterance and the transcription corresponding to the particular augmented out-of-domain training utterance; and updating parameters of the student ASR model using the loss. The loss may be at least one of a cross-entropy loss, a Kullback-Leibler (KL) divergence loss, or an L2 loss.

[0009] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of an exemplary audio environment using a speech recognition model for transcribing speech. [Figure 2A] FIG. 1 is a schematic diagram of an example training process for training a speech recognition model using knowledge distillation with domain mismatch. [Figure 2B]FIG. 10 is a schematic diagram of another exemplary training process for training a speech recognition model using knowledge distillation with domain mismatch. [Figure 3] 1 is a flowchart of an exemplary operational arrangement for a method for training a speech recognition model using knowledge distillation with domain mismatch. [Figure 4] FIG. 1 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0011] Like reference symbols in the various drawings refer to like elements.

[0012] Knowledge distillation can be used to transfer knowledge from one model (i.e., a teacher model) to another model (i.e., a student model). Traditional knowledge distillation assumes that data in the target domain used to train the teacher model is available to train the student model. Alternatively, complex and advanced methods are required to generate appropriate data for training the student model. However, in many cases, training data matching the target domain is not available to train the student model. For example, for automatic speech recognition (ASR) of long-tail languages ​​(e.g., Sub-Saharan African languages), the amount of training data available to train the server-side model may be insufficient. While federated training can be used to train the server-side model, on-device data used to train an on-device speech recognition model typically should not leave the user device for privacy reasons. Therefore, the server-side model (i.e., a student model) cannot be trained using traditional knowledge distillation based on the on-device model (i.e., a teacher model) because there is no available training data matching the target domain in which the on-device model was trained. Therefore, there is a need for a method and system for performing knowledge distillation to train a student model when the domain of the training data available for training the student model does not match the domain of the training data used to train the teacher model. Although the examples disclosed herein relate to training a student ASR model using knowledge distillation with domain mismatch, the disclosed examples may be used to train other types of models using knowledge distillation with domain mismatch. That is, when the training data for training the teacher model is from a domain that does not match (i.e., is different from) the domain of the training data available for training the student model.

[0013] 1 , in some embodiments, a voice environment 100 includes a user 104 interacting with a voice-enabled device 10 (also referred to as a user device 10) using spoken utterances 106. Here, the spoken utterances 106 correspond, for example, to dictation for transcription or to queries that request a response from the user device 10 or cause the user device 10 to perform a task specified by the query. In this sense, the user 104 may have a conversation-like interaction with the user device 10 to perform a computing activity or find an answer to a question. In the illustrated example, the system 102 includes an ASR model 170 of a speech recognition system 150 for generating a transcription 172 of the utterance 106. The transcription 172 can then be processed by the digital assistant 20 to generate a response to the utterance 106 or perform a task specified by the utterance 106. In some implementations, the digital assistant 20 includes a natural language processing / understanding (NLP / NLU) module executing on the user device 10 or on the remote computing system 70 to process the transcription 172 to understand the utterance 106. The digital assistant 20 can provide a response as text in the user interface 22 on the display 16c of the user device 10 or as an audio signal 108 output by the speaker 16b of the user device 10. In some examples, the digital assistant 20 generates text representing the response, and a text-to-speech (TTS) system converts the text into the audio signal 108 as synthesized speech. In the illustrated example, the user 104 speaks the utterance 106 asking, "Who educated Alexander the Great?" and the digital assistant 20 responds with audio data 108 representing the response, "Aristotle."

[0014] The user device 10 is associated with a user 104 and may correspond to any computing device capable of capturing audio data 162 and providing textual or audible output. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches, smart glasses, smart goggles, AR headsets, VR headsets, etc.), smart appliances, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 in communication with the data processing hardware 12, storing instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes one or more input / output devices 16, 16a-c, such as an audio capture device 16, 16a (e.g., a microphone) for capturing and converting spoken speech 106 into electrical signals, an audio output device 16, 16b (e.g., a speaker) for communicating audible audio signals (e.g., as output audio data from the user device 10), and a display 16, 16c for displaying visual content. Of course, any number and / or type(s) of other input / output devices 16 may be used. The input / output devices 16 may be present on or in communication with the user device 10.

[0015] The speech recognition system 150 executes on the user device 10 of the user 104 and / or on a remote computing system 70 (e.g., one or more remote servers of a distributed system executed in a cloud computing environment) that communicates with the user device 10 via the network 40. The speech recognition system 150 includes an input subsystem 160 configured to receive utterances 106 spoken by the user 104 and captured by the audio capture device 16a and convert each utterance 106 into a corresponding digital format associated with input acoustic frames 162 (also commonly referred to as audio data 162) that can be processed by the speech recognition system 150. An ASR model 170 of the speech recognition system 150 then receives as input the audio data 162 corresponding to the current utterance 106 and generates / predicts as output a corresponding transcription 172 (e.g., a recognition result / hypothesis) of the utterance 106.

[0016] The remote computing system 70 includes data processing hardware 72 and memory hardware 74 in communication with the data processing hardware 72. The memory hardware 74 stores instructions that, when executed on the data processing hardware 72, cause the data processing hardware 72 to perform one or more operations, such as those disclosed herein.

[0017] The ASR model 170 may have different types of neural network architectures. In some examples, the ASR model 170 includes a conformer-based encoder and a listen-attend-spell (LAS) decoder. In some implementations, the conformer-based encoder may include 600 million trainable parameters, and the LAS decoder may include 200 million trainable parameters. In other examples, the ASR model 170 includes an RNN-T model with an encoder-decoder architecture, where the decoder architecture of the RNN-T model includes a combination / prediction network. The audio encoder of the ASR model 170 may include a cascade encoder architecture.

[0018] 2A illustrates an exemplary training process 200a for training a student ASR model 210 from an ASR model 170 (i.e., teacher ASR model 170) using knowledge distillation with domain mismatch. That is, domain mismatch occurs when the training data used to train the teacher ASR model 170 is from a target domain that does not match (i.e., is different from) the domain of the training data available to train the student ASR model 210. The training process 200a includes a knowledge distillation training process in which the teacher ASR model 170 distills its knowledge into the student ASR model 210 by training the student ASR model 210 with distilled data 220 that includes multiple out-of-domain training utterances 222, 222a-n. Notably, the out-of-domain training utterances 222 originate from a domain different from the target domain (i.e., the target-domain training utterances) of the training data used to train the teacher ASR model 170 prior to knowledge distillation into the student ASR model 210. Here, the student ASR model 210 learns to predict transcriptions 212, 212a-n that match the pseudo-labels 240, 240a-n generated by the teacher ASR model 170 in order to train the student ASR model 210 to recognize utterances corresponding to the target domain. In an example of federated learning, the student ASR model 210 executes in a cloud computing environment (e.g., a server-side model executing on a remote computing system 70), and the teacher ASR model 170 executes on a user device 10 that communicates with the cloud computing environment. Here, the target-domain training utterances used to train the teacher ASR model 170 are received at the user device 10 from a user 104 of the user device 10 and are not available to train the student ASR model 210.Thus, the target domain training utterances are processed only by the teacher ASR model 170 for the purpose of training the teacher ASR model 170, and thereby neither the target domain training utterances nor any sensitive information derived from the target domain training utterances are transmitted / communicated to the cloud computing environment.

[0019] 2A shows a student ASR model 210 being trained using a single teacher ASR model 170 executing on a user device 10. However, the student ASR model 210 may be trained using one or more teacher ASR models 170 executing on one or more user devices 10, where the target domain training utterances used to train each teacher ASR model 170 are received at each of the one or more user devices 10 from a corresponding user. Notably, each teacher ASR model 170 may be associated with its own target domain training utterances, and the utterances are not made available for training other teacher ASR models 170 or student ASR models 210. The architecture of the teacher ASR model 170 may differ from the architecture of the student ASR model 210. Additionally or alternatively, the teacher ASR model 170 and the student ASR model 210 may have different sizes. In particular, for ASR for long-tail languages, we found that the teacher ASR model 170 was 32 times smaller than the student ASR model 210 and still achieved good knowledge distillation.

[0020] During the training process 200a, the extender 230 receives the out-of-domain training utterances 222 and generates, for each particular out-of-domain training utterance 222, a corresponding extended out-of-domain training utterance 232, 232a-n. The extender 230 generates the extended out-of-domain training utterance 232, for example, by adding noise, adding reverberation, or manipulating the timing of the noise, reverberation, or the timing of the reverberation, to the corresponding out-of-domain training utterance 222. In some implementations, the extender 230 generates the extended out-of-domain training utterance 232 using Gaussian noise injection (GNI). For example, for an out-of-domain training utterance 222 represented by Log-Mel input features, the extender 230 may extend the out-of-domain training utterance 222 by randomly varying the values ​​of one or more input features of the out-of-domain training utterance 222 by adding a corresponding random value obtained from a Gaussian distribution to each value of the one or more different input features. Here, the extender 230 may not vary all of the input features. The mean and variance of the Gaussian distribution can be calculated from the corresponding frequency channels of the input features. Alternatively, one or more input features can be replaced with corresponding random values ​​obtained from the Gaussian distribution. Here, the GNI can be tuned using a single hyperparameter representing a probability ranging from zero to 100 percent indicating whether a particular input feature is changed / replaced, so that the GNI-based augmentation can be easily adjusted. Notably, the target domain training utterances used to train the teacher ASR model 170 are not available after receiving the distilled data 220, and the distilled data 220 need not include any such target domain training utterances. In the illustrated example, the augmenter 230 generates a single augmented training utterance 232 for each training utterance 222. However, the augmenter 230 may generate multiple augmented training utterances 232 for each training utterance 222.

[0021] For each particular augmented out-of-domain training utterance 232, the teacher ASR model 170 generates a pseudo-label 240, 240a-n, corresponding to the particular augmented out-of-domain training utterance 232. In some implementations, the teacher ASR model 170 generates the corresponding pseudo-label 240 by generating a probability distribution over possible speech recognition hypotheses for the particular augmented out-of-domain training utterance 232, where the pseudo-label 240 is the speech recognition hypothesis with the highest probability. Notably, the teacher ASR model 170 may generate a pseudo-label 240 for a particular augmented out-of-domain training utterance 232 that differs from the pseudo-label generated by the teacher ASR model 170 for the corresponding out-of-domain training utterance 222. Thus, the augmenter 230 effectively and dynamically creates a large unlabeled dataset at near-zero cost to increase the abundance and coverage of the distilled data 220, thereby providing more opportunities for the teacher ASR model 170 to distill knowledge in the target domain into the student ASR model 210.

[0022] For each particular augmented out-of-domain training utterance 232, the student ASR model 210 generates a corresponding predicted transcription 212 that corresponds to the particular augmented out-of-domain training utterance 232. In some implementations, the student ASR model 210 generates the corresponding predicted transcription 212 by generating a probability distribution over possible speech recognition hypotheses for the particular augmented out-of-domain training utterance 232, where the corresponding predicted transcription 212 is the speech recognition hypothesis with the highest probability.

[0023]

number

[0024]

number

[0025] 2B illustrates another exemplary training process 200b for training a student ASR model 210 from an ASR model 170 (i.e., a teacher ASR model 170) using knowledge distillation with domain mismatch. Compared to the training process 200a of FIG. 2A, the training process 200b of FIG. 2B includes an additional augmenter 260 for modifying augmented out-of-domain training utterances 232 to form student training utterances 262, 262a-n for input to the student ASR model 210. Here, the teacher ASR model 170 is still processing the augmented out-of-domain training utterances 232, and therefore, the pseudo labels 240 are unaffected by the augmenter 260. In some examples, the augmenter 260 adds noise (e.g., using GNI) or performs some other type of spectral augmentation. Thus, because the student ASR model 210 learns from a noisy student learning framework, the student ASR model 210 learns that slight variations in the input utterance can result in the same pseudo label 240, thus providing an additional degree of robustness to the training process 200b. Here, the extender 260 may create multiple student training utterances 262 for each extended out-of-domain training utterance 232.

[0026] 3 is a flowchart of an exemplary operational arrangement for a computer-implemented method 300 of training an ASR model using knowledge distillation with domain mismatch. The operations may be performed by data processing hardware 410 (FIG. 4) (e.g., data processing hardware 12 of user device 10 or data processing hardware 72 of remote computing system 70) based on executing instructions stored in memory hardware 420 (e.g., memory hardware 14 of user device 10 or memory hardware 74 of remote computing system 70).

[0027] At operation 302, the method 300 includes receiving distilled data 220 including a plurality of out-of-domain training utterances 222. For each particular out-of-domain training utterance 222 in the distilled data 220, the method 300 includes generating a corresponding augmented out-of-domain training utterance 232 at operation 304 and generating a pseudo label 240 corresponding to the corresponding augmented out-of-domain training utterance 232 using a teacher ASR model 170 trained with training data corresponding to the target domain at operation 306. At operation 308, the method 300 includes distilling the student ASR model 210 from the teacher ASR model 170 by training the student ASR model 210 using the corresponding augmented out-of-domain training utterance 232 paired with the corresponding pseudo label 240 generated by the teacher ASR model 170.

[0028] 4 is a schematic diagram of an exemplary computing device 400 that can be used to implement the systems and methods described herein. Computing device 400 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the scope of the present invention as described and / or claimed herein.

[0029] The computing device 400 includes a processor 410 (i.e., data processing hardware) that can be used to implement the data processing hardware 12 and / or 72, a memory 420 (i.e., memory hardware) that can be used to implement the memory hardware 14 and / or 74, a storage device 430 (i.e., memory hardware) that can be used to implement the memory hardware 14 and / or 74, a high-speed interface / controller 440 that connects to the memory 420 and a high-speed expansion port 450, and a low-speed bus 470 and a low-speed interface / controller 460 that connects to the storage device 430 that can be used to store the distilled data 220. Each of the components 410, 420, 430, 440, 450, and 460 is interconnected using various buses and may be mounted on a common motherboard or may exist in other ways as needed. Processor 410 may process instructions for execution within computing device 400, including instructions stored in memory 420 or storage device 430, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 480 connected to high-speed interface 440. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as desired. Also, multiple computing devices 400 may be connected (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system), with each device providing multiple portions of multiple required operations.

[0030] The memory 420 stores information non-transiently within the computing device 400. The memory 420 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 420 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0031] Storage device 430 can provide mass storage for computing device 400. In some embodiments, storage device 430 is a computer-readable medium. In various different implementations, storage device 430 may be a device array including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional embodiments, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 420, storage device 430, or memory on processor 410.

[0032] The high-speed controller 440 manages bandwidth-intensive operations of the computing device 400, while the low-speed controller 460 manages less bandwidth-intensive operations. This role assignment is merely exemplary. In some implementations, the high-speed controller 440 is coupled to memory 420, a display 480 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 450 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 460 is coupled to a storage device 430 and a low-speed expansion port 490. The low-speed expansion port 490 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and can connect to one or more input / output devices such as a keyboard, pointing device, scanner, or, for example, via a network adapter, a network device such as a switch or router.

[0033] The computing device 400 may be implemented in many different forms, as shown in the figure. For example, it may be implemented as a standard server 400a, or multiple times in a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

[0034] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor. Such programmable processor may be special-purpose or general-purpose and may be coupled to receive and transmit data and instructions from a storage system, at least one input device, and at least one output device.

[0035] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0036] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0037] The processes and logic flows described herein may be implemented by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose processors, as well as any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operably connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0038] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer that has a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touch screen for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, such as acoustic input, speech input, or tactile input. Additionally, a computer can interact with a user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0039] Unless expressly stated to the contrary, the phrase "at least one of A, B, or C" is intended to indicate any combination or subset of A, B, and C, such as (1) at least one A alone, (2) at least one B alone, (3) at least one C alone, (4) at least one A and at least one B, (5) at least one A and at least one C, (6) at least one B and at least one C, and (7) at least one A, at least one B, and at least one C. Furthermore, unless expressly stated to the contrary, the phrase "at least one of A, B, and C" is intended to indicate any combination or subset of A, B, and C, such as (1) at least one A alone, (2) at least one B alone, (3) at least one C alone, (4) at least one A and at least one B, (5) at least one A and at least one C, (6) at least one B and at least one C, and (7) at least one A, at least one B, and at least one C. Furthermore, unless expressly stated to the contrary, "A or B" is intended to indicate any combination of A and B, such as (1) A alone, (2) B alone, and (3) A and B.

[0040] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (300) executed by data processing hardware (510) to cause the data processing hardware (510) to perform an operation, the operation comprising: receiving distilled data (220) including a plurality of out-of-domain training utterances (222); For each particular out-of-domain training utterance (222) in the distilled data (220), generating corresponding augmented out-of-domain training utterances (232); generating pseudo-labels (240) corresponding to the corresponding augmented out-of-domain training utterances (232) using a supervised automatic speech recognition (ASR) model (170) trained with training data corresponding to the target domain; and distilling the student ASR model (210) from the teacher ASR model (170) by training the student ASR model (210) using the corresponding augmented out-of-domain training utterances (232) paired with the corresponding pseudo-labels (240) generated by the teacher ASR model (170).

2. 2. The computer-implemented method of claim 1, wherein distilling the student ASR model from the teacher ASR model includes training the student ASR model to recognize utterances corresponding to the target domain.

3. Prior to receiving the distilled data, the teacher ASR model is trained with the training data, which includes a plurality of target-domain training utterances; 3. The computer-implemented method of claim 1, wherein after receiving the distilled data, the training data is unavailable to the teacher ASR model and the student ASR model.

4. the training data includes a plurality of target domain training utterances; 4. The computer-implemented method (300) of claim 1, wherein one or more of the target domain training utterances of the plurality of target domain training utterances are not included in the distillation data (220).

5. 5. The computer-implemented method of claim 1, wherein augmenting each out-of-domain training utterance comprises at least one of adding noise, adding reverberation, or manipulating timing.

6. The student ASR model (210) is executed in a cloud computing environment (70); The computer-implemented method (300) of any of claims 1 to 5, wherein the teacher ASR model (170) runs on one or more user devices (10) that communicate with the cloud computing environment (70).

7. 7. The computer-implemented method of claim 6, wherein the training data includes a plurality of target domain training utterances, each target domain training utterance received at each of the one or more user devices.

8. generating the pseudo-labels (240) corresponding to the corresponding augmented out-of-domain training utterances (232) includes generating a probability distribution over possible speech recognition hypotheses; The computer-implemented method (300) of any of claims 1 to 7, wherein the pseudo labels (240) comprise the N-best speech recognition hypotheses with the highest probabilities.

9. Distilling the student ASR model (210) from the teacher ASR model (170) includes, for each particular augmented out-of-domain training utterance (232), generating transcriptions (212) corresponding to the specific augmented out-of-domain training utterances (232) using the student ASR model (210); generating a loss (252) based on the pseudo-labels (240) corresponding to the particular out-of-domain training utterances (232) and the transcriptions (212) corresponding to the particular out-of-domain training utterances (232); and updating parameters of the student ASR model (210) using the loss (252).

10. 10. The computer-implemented method of claim 9, wherein the loss comprises at least one of a cross-entropy loss, a Kullback-Leibler (KL) divergence loss, or an L2 loss.

11. A system (400) comprising: data processing hardware (410); and memory hardware (420) in communication with the data processing hardware (410), the memory hardware (420) storing instructions that, when executed by the data processing hardware (410), cause the data processing hardware (410) to: receiving distilled data (220) including a plurality of out-of-domain training utterances (222); For each particular out-of-domain training utterance (222) in the distilled data (220), generating corresponding augmented out-of-domain training utterances (232); generating pseudo-labels (240) corresponding to the corresponding augmented out-of-domain training utterances (232) using a supervised automatic speech recognition (ASR) model (170) trained with training data corresponding to the target domain; and distilling the student ASR model (210) from the teacher ASR model (170) by training the student ASR model (210) using the corresponding augmented out-of-domain training utterances (232) paired with the corresponding pseudo-labels (240) generated by the teacher ASR model (170).

12. 12. The system of claim 11, wherein distilling the student ASR model from the teacher ASR model includes training the student ASR model to recognize utterances corresponding to the target domain.

13. Prior to receiving the distilled data, the teacher ASR model is trained with the training data, which includes a plurality of target-domain training utterances; 13. The system (400) of claim 11 or 12, wherein after receiving the distilled data (220), the training data is unavailable to the teacher ASR model (170) and the student ASR model (210).

14. the training data includes a plurality of target domain training utterances; The system (400) of any of claims 11 to 13, wherein one or more of the target domain training utterances of the plurality of target domain training utterances are not included in the distilled data (220).

15. The system (400) of any of claims 11-14, wherein enhancing the respective out-of-domain training utterances (222) comprises at least one of adding noise, adding reverberation, or manipulating timing.

16. The student ASR model (210) is executed in a cloud computing environment (70); The system (400) of any of claims 11 to 15, wherein the teacher ASR model (170) runs on one or more user devices (10) that communicate with the cloud computing environment (70).

17. 17. The system (400) of claim 16, wherein the training data includes a plurality of target domain training utterances, each target domain training utterance received at each of the one or more user devices (10).

18. generating the pseudo-labels (240) corresponding to the corresponding augmented out-of-domain training utterances (232) includes generating a probability distribution over possible speech recognition hypotheses; The system (400) of any of claims 11 to 17, wherein the pseudo labels (240) comprise the N-best speech recognition hypotheses with the highest probabilities.

19. Distilling the student ASR model (210) from the teacher ASR model (170) includes, for each particular augmented out-of-domain training utterance (232), generating transcriptions (212) corresponding to the specific augmented out-of-domain training utterances (232) using the student ASR model (210); generating a loss (252) based on the pseudo-labels (240) corresponding to the particular out-of-domain training utterances (232) and the transcriptions (212) corresponding to the particular out-of-domain training utterances (232); and updating parameters of the student ASR model (210) using the loss (252).

20. 20. The system of claim 19, wherein the loss comprises at least one of a cross-entropy loss, a Kullback-Leibler (KL) divergence loss, or an L2 loss.

Images