Adaptive distillation

Abstract

A method (400) for distilling one or more trained teacher automatic speech recognition (ASR) models (210) into a multilingual student model (200), the method (400) comprising receiving a plurality of teacher training examples (152) and a plurality of student training examples (154). The method also comprises training one or more teacher ASR models using the plurality of teacher training examples. Each teacher ASR model is configured to output a respective text representation of a respective audio input. The method further comprises training a multilingual student ASR model using the plurality of student training examples to generate a multilingual student ASR model (222), and distilling the trained one or more teacher ASR models to the multilingual student ASR model using an adjustable distillation loss weight. The student ASR model is configured to receive an audio input (14) and output a corresponding text representation (142) of the received audio input.

Description

Technical Field

[0001] This disclosure relates to adaptive distillation. Background Technology

[0002] With the increasing popularity of Automatic Speech Recognition (ASR) in recent years, ASR is being applied more widely to languages ​​around the world. Unfortunately, some of these languages ​​have limitations that affect the quality or robustness of ASR models. For example, language resources can vary greatly, referring to the resources that ASR models utilize to train and improve accuracy and robustness. Due to these resource differences, ASR models may experience varying degrees of performance degradation, which inevitably impacts the user experience of applications or programs using ASR models. Summary of the Invention

[0003] This disclosure provides a computer-implemented method for distilling one or more trained teacher-based automatic speech recognition (ASR) models into multilingual student models. When executed on data processing hardware, the computer-implemented method causes the data processing hardware to perform operations of receiving multiple teacher training examples and multiple student training examples. The operations also include training one or more teacher ASR models using the multiple teacher training examples. Each teacher ASR model is configured to output a corresponding text representation of a given audio input. The operations also include generating a multilingual student ASR model by training a multilingual student ASR model using multiple student training examples and distilling the trained one or more teacher ASR models into a multilingual student ASR model using adjustable distillation loss weights. Each student ASR model is configured to receive audio input and output a corresponding text representation of the received audio input.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, one or more teacher ASR models are configured to collectively identify fewer languages ​​than the multilingual student ASR models. Adjustable distillation loss weights may include constant values. In some other implementations, training the multilingual student model spans n training steps, and the adjustable distillation loss weights include a decreasing function that decreases based on the n training steps.

[0005] In some examples, each of the one or more teacher ASR models and the multilingual student ASR model includes a recurrent neural network sensor (RNN-T) architecture. In these implementations, the adjustable distillation loss weights may include a decreasing function based on the RNN-T loss corresponding to the one or more teacher ASR models. Alternatively, the adjustable distillation loss weights in these examples may include a decreasing function based on a first RNN-T loss corresponding to the one or more teacher ASR models and a second RNN-T loss corresponding to the multilingual student ASR model. Here, the first RNN-T loss corresponding to the one or more teacher ASR models is decreased over a period of time, and the second RNN-T loss corresponding to the multilingual student ASR model is increased over a period of time.

[0006] Each of the one or more teacher ASR models can correspond to a monolingual ASR model. Alternatively, one or more teacher ASR models can correspond to a single multilingual ASR model.

[0007] Another aspect of this disclosure provides a system for distilling one or more trained teacher-based automatic speech recognition (ASR) models into multilingual student models. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving a plurality of teacher training examples and a plurality of student training examples. The operations also include training one or more teacher ASR models using the plurality of teacher training examples. Each teacher ASR model is configured to output a corresponding text representation of a given audio input. The operations further include training a multilingual student ASR model using the plurality of student training examples and distilling the trained one or more teacher ASR models into a multilingual student ASR model using adjustable distillation loss weights to generate a multilingual student ASR model. Each student ASR model is configured to receive audio input and output a corresponding text representation of the received audio input.

[0008] This aspect may include one or more of the following optional features. In some implementations, one or more teacher ASR models are configured to collectively identify fewer languages ​​than the multilingual student ASR models. The adjustable distillation loss weights may include constant values. In some other implementations, training the multilingual student model spans n training steps, and the adjustable distillation loss weights include a decreasing function that decreases based on the n training steps.

[0009] In some examples, each of the one or more teacher ASR models and the multilingual student ASR model includes a recurrent neural network sensor (RNN-T) architecture. In these implementations, the adjustable distillation loss weights may include a decreasing function based on the RNN-T loss corresponding to the one or more teacher ASR models. Alternatively, the adjustable distillation loss weights in these examples may include a decreasing function based on a first RNN-T loss corresponding to the one or more teacher ASR models and a second RNN-T loss corresponding to the multilingual student ASR model. Here, the first RNN-T loss corresponding to the one or more teacher ASR models is decreased over a period of time, and the second RNN-T loss corresponding to the multilingual student ASR model is increased over a period of time.

[0010] Each of the one or more teacher ASR models can correspond to a monolingual ASR model. Alternatively, one or more teacher ASR models can correspond to a single multilingual ASR model.

[0011] Details of one or more implementations of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the specification, the drawings, and the claims. Attached Figure Description

[0012] Figure 1A and 1B This is a schematic diagram of an example speech environment using an adaptive automatic speech recognition model.

[0013] Figure 2A This is a schematic diagram of an example adaptive model formed by more than one monolingual teacher model.

[0014] Figure 2B This is a schematic diagram of an example adaptive model formed by a single multilingual teacher model.

[0015] Figure 3 yes Figure 1A-2B A schematic diagram of an example model architecture for an adaptive model.

[0016] Figure 4 This is a flowchart illustrating an example layout of operations used to generate adaptive models.

[0017] Figure 5 This is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein.

[0018] In the various figures, the same reference numerals denote the same elements.

[0019] Specific implementation method

[0020] refer to Figure 1A and1B In some implementations, system 100 includes a user 10 who transmits spoken utterance 12 to a voice-enabled device 110 (also referred to as device 110 or user device 110). User 10 (i.e., the speaker of utterance 12) can speak utterance 12 as a query or command to request a response from device 110 or to cause device 110 to perform a task specified by the query. Device 110 is configured to capture the voice of one or more users 10 within the voice environment of user device 110. Here, audio sound can refer to the spoken utterance 12 of user 10, which serves as an audible query, a command from device 110, or audible communication captured by device 110. The voice-enabled system of device 110 or associated with device 110 (e.g., a digital assistant interface) can respond to the query and / or cause the command to be executed.

[0021] Here, device 110 captures audio data 14 corresponding to the speech 12 of user 10. Device 110 can correspond to any computing device associated with user 10 and capable of receiving audio data 14. Some examples of user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-book readers, etc.), computers, wearable devices (e.g., smartwatches), music players, broadcasting equipment, smart appliances (e.g., smart TVs) and Internet of Things (IoT) devices, remote controls, smart speakers, etc. Device 110 includes data processing hardware 112 and memory hardware 114 that communicates with and stores instructions to the data processing hardware 112, which, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations related to speech and / or text processing. In some examples, device 110 includes one or more applications (i.e., software applications), each of which can utilize one or more speech processing systems 140 and one or more speech processing systems 200 associated with device 110 to perform various functions within the application.

[0022] Device 110 also includes an audio subsystem having an audio capture device (e.g., a microphone) 116 and a voice output device (e.g., a speaker) 118. The audio capture device 116 is used to capture audio data 14 within a speech environment and convert it into an electrical signal, and the voice output device 118 is used to transmit an audible audio signal (e.g., a synthesized playback signal 154 from device 110). Although device 110 implements a single audio capture device 116 in the example shown, without departing from the scope of this disclosure, device 110 may implement an array of audio capture devices 116, whereby one or more audio capture devices 116 in the array may not physically reside on device 110 but communicate with the audio subsystem (e.g., peripherals of device 110). For example, device 110 may correspond to a vehicle infotainment system that utilizes a microphone array placed throughout the vehicle.

[0023] Furthermore, device 110 is configured to communicate with remote system 130 via network 120. Remote system 130 may include remote resources 132, such as remote data processing hardware 134 (e.g., a remote server or CPU) and / or remote memory hardware 136 (e.g., a remote database or other storage hardware). Device 110 may utilize remote resources 132 to perform various functions related to speech processing. For example, device 110 is configured to perform speech recognition using speech recognition system 140. These systems 140 and 200 may reside on device 110 (referred to as on-device systems) or reside remotely (e.g., on remote system 130) but communicate with device 110. In some examples, some of these systems 140 and 200 reside locally or on-device, while others reside remotely. In other words, any of these systems 140 and 200 can be local or remote in any combination. For example, when the size or processing requirements of systems 140 and 200 are considerable, systems 140 and 200 may reside in a remote system 130. However, when device 110 can support the size or processing requirements of one or more systems 140 and 200, one or more systems 140 and 200 may reside on device 110 using data processing hardware 112 and / or memory hardware 114. Optionally, one or more of systems 140 and 200 may reside locally / on the device and remotely. For example, one or more of systems 140 and 200 may by default execute on the remote system 130 when a connection to network 120 between device 110 and remote system 130 is available, but when the connection is lost or network 120 is unavailable, systems 140 and 200 may instead execute locally on device 110.

[0024] Speech recognition system 140 receives audio data 14 as input and uses adaptive automatic speech recognition (ASR) model 200 (also called adaptive model 200) to transcribe the audio signal into transcript 142 as output. Generally, by converting audio data 14 into transcript 142, speech recognition system 140 allows device 110 to recognize when spoken words 12 from user 10 correspond to a query, command, or some other form of audio communication. Transcription 142 refers to the text sequence that device 110 can then use to generate a response to a query or command. For example, if user 10 asks device 110 the question “What will the weather be like today?”, device 110 transmits audio data 14 corresponding to the question “What will the weather be like today?” to speech recognition system 140. Speech recognition system 140 converts audio data 14 into a copy including the text “What will the weather be like today?” Device 110 can then use the text or a portion of the text to determine a response to the query. For example, to determine the weather for a given day (i.e., today), device 110 might transmit text (e.g., “what will the weather be like today?”) or identify portions of text (e.g., “weather” and “today”) to a search engine. The search engine can then return one or more search results interpreted by device 110 to generate a response for user 10.

[0025] Reference Figure 1B The adaptive model 200 of the speech recognition system 140 can be a multilingual speech recognition model. A multilingual speech recognition model is a model capable of generating transcriptions 142 in more than one language (i.e., multiple languages). For example, Figure 1B The diagram illustrates a speech recognition system 140 receiving audio data 14 and an adaptive model 200 in a multilingual version. The adaptive model 200 converts the audio data 14 into the utterance “what will the weather be like today?”. This is followed by three different transcriptions 142, 142a-c. Here, the first transcription 142a is a Spanish (named SP) translation of “How is the weather today?” or “…” "el clima hoy?" The second transcription 142b is the Swedish (designated SW) translation of "what will the weather be like today?" or "hur blir". The third transcription 142c is the German (referred to as DE) translation of "what will the weather be like today?" or "Wie wird dasWetter heute sein?". Multilingual speech recognition models are advantageous for multilingual users who can speak different languages, or can improve the performance of speech recognition models for low-resource languages ​​by learning shared representations from data available in other languages ​​(i.e., high-resource languages). For example, there may be a large amount of training data for languages ​​like American English, but a small amount of training data may be available for languages ​​like Zulu. Here, when the adaptive model 200 is a multilingual model, the adaptive model 200 can utilize a large number of American English training examples to compensate for the lack of Zulu training data.

[0026] Figure 2A and Figure 2B This is an example of the process of generating an adaptive model 200 (also called a student model 200). The adaptive model 200 can be formed from one or more teacher models 210, such that the adaptive model 200 can be referred to as a student model. That is, the teacher model 210 has a neural network that is distilled into a student model (e.g., the adaptive model 200) to form a neural network of the student model or to influence it in some way. Distillation generally refers to the process of training a neural network using a pre-trained network. Using distillation, neurons in the pre-trained network that are less critical to the desired output (e.g., similar to self-weights) can be reduced to form a more streamlined neural network (i.e., a distilled neural network). Distillation can make the distilled neural network more accurate and / or more compact compared to the pre-trained network. In other words, when a pre-trained network is formed, the pre-trained network may have already formed neurons that ultimately have less influence on the desired output when the training of the pre-trained network is complete; therefore, the pre-trained network includes neurons that can be removed or modified to reduce any harmful influence from these neurons or to remove unnecessary neurons. For ASR models, distillation may be advantageous in low-resource situations, where student models can learn behavior from teacher models that have learned from high-resource situations.

[0027] However, distillation is not without its challenges when transferring knowledge to the student model 200. For example, a difficulty in performing knowledge distillation for the student model 200 is how to balance the learning processes 220. That is, the student model 200 can be taught through both distillation process 220, distillation process 220a, and its own training processes 220 and 220b. Because multiple learning processes 220 are involved in generating the student model 200, the performance of the trained student model 200 can vary based on the balance between these processes 220. During the learning process 220, one or more teacher models 210 are first trained to build a neural network for distillation process 220a. During the training process of one or more teacher models 210, the teacher model 210 receives multiple teacher training samples 152, multiple teacher training samples 152a-n (e.g., from the training sample database 150), and is trained using the teacher training samples 152 to teach each teacher model 210 to predict the texture representation of the corresponding audio input as output. In this respect, training samples (e.g., teacher training sample 152 or student training sample 154) allow the model to learn fundamental truths, since training samples 152 and 154 include audio samples and their corresponding transcriptions (i.e., text representations). Once one or more teacher models 210 are trained, the one or more trained teacher models 210 can distill their knowledge into student models 200.

[0028] In addition to distillation process 220a from one or more teacher models 210, student model 200 also learns from training processes 220 and 220b. In training process 220b, much like the teacher training process, student model 200 learns from student training samples 154 to predict text representations of audio inputs. In both distillation process 220a and training process 220b, student model 200 is configured to balance how much knowledge it has acquired from these processes 220a and 220b using weights 222, 222a-b. That is, each process 220a and 220b is a sequence of training steps. At each training step, the loss for each process 220 is calculated and used to influence the next training step. For example, in general, student model 200 aims to minimize the loss of a given process 220 to approximate a neural network capable of accurately predicting text representations of a given input audio. Because each process 220 has an associated loss, the entire learning process can be represented by a total loss as a combination of the distillation loss of distillation process 220a and the training loss (e.g., RNN-T loss) of training process 220b. Therefore, to instruct the student model 200 how to balance these processes 220a and 220b, the student model 200 uses adjustable weights 222 applied to the loss of any process. In some examples, the adjustable weights 222 are applied to the distillation loss and are therefore referred to as adjustable distillation weights 222a.

[0029] In some configurations, the tunable distillation weights 222a are configured as constant values. However, in other configurations, the tunable distillation weights 222a can be a decreasing function that decreases with the number of training steps. That is, the student model 200 becomes less concerned with the distillation process loss over time. When one or more teacher models 210 and student models 200 have an RNN-T model architecture (e.g., in an end-to-end streaming application), the tunable distillation weights 222a can be a decreasing function based on the RNN-T loss corresponding to one or more teacher models 210. Alternatively, for the RNN-T structures of both models 200 and 210, the tunable distillation weights 222a can also take into account the RNN-T loss of the student model 200. Here, the tunable distillation weights 222a can be used to address the RNN-T loss of the student model 200 as a function that decreases based on a first RNN-T loss from one or more teacher models 210 and increases based on a second RNN-T loss from the student model 200.

[0030] For details, please refer to the following: Figure 2A One or more teacher models 210 may correspond to multiple monolingual teacher models 210, which distill each of their knowledge into student models 200 to form a multilingual student model 200. Conversely, Figure 2B One or more teacher models 210 can be described as a single multilingual teacher model 210 that decomposes its knowledge into multilingual student models 200. In any of these cases, one or more teacher models 210 may collectively recognize fewer languages ​​than the multilingual student models 200. For example, in the case where student models 200 have their own training process 220a, student models 200 are able to expand their language corpus to include more languages ​​than those distilled into teacher models 210.

[0031] Reference Figure 3 The adaptive distillation process can be applied to different types of speech recognition models. Generally, when the adaptive model 200 is a specific type of speech recognition model, the teacher model 210 and the student model (i.e., the adaptive model 200) are both models of the same type used for distillation purposes. An increasingly popular model is a sequence-to-sequence model called a recurrent neural network sensor (RNN-T). RNN-T does not employ attention mechanisms, and unlike other sequence-to-sequence models that typically require processing the entire sequence (e.g., an audio waveform) to produce an output (e.g., a sentence), RNN-T processes input samples and streams output symbols continuously, a feature particularly attractive for real-time communication. For example, speech recognition with RNN-T can output characters one by one as spoken words. Here, RNN-T uses feedback loops to feed back the symbols predicted by the model to itself to predict the next symbol. Because decoding an RNN-T involves beam searching through a single neural network rather than a large decoder graph, RNN-T can be scaled to a fraction of the size of server-based speech recognition models. As the size decreases, RNN-T can be fully deployed on the device and can operate offline (i.e., without a network connection); thus, the unreliability of communication networks is avoided.

[0032] When the adaptive model 200 is an RNN-T model, it is a neural network model corresponding to an encoder-decoder architecture that can be trained end-to-end to map an input sequence (e.g., an input audio signal) to a target sequence (e.g., words or characters spoken in the audio signal). In other words, given an input sequence (e.g., a sequence of real-valued vectors), the RNN-T model attempts to predict the labeled target sequence. Here, the input sequence can be a raw feature vector, such as log-mel filter bank energy features or other neural network encoded features.

[0033] Continue to refer to Figure 3 The adaptive model 200 includes an encoder network 302 and a decoder network 304. Here, the encoder network 302 includes an encoder 310, which reads the sequence x = (x1, x2, ..., x...) of d-dimensional feature vectors. T ),in Furthermore, a high-order feature representation, also known as the encoder embedding e, is generated at each time step. The decoder network 304 receives the high-order feature representation e and decodes it using a joint layer 320 and a prediction network 330. The joint layer 320, which incorporates the prediction network 330, can be considered a feedforward neural network, where the joint layer 320 computes the logits of the input to the prediction network 330. In other words, the joint layer 320 compares the high-order feature representation e1 output by the encoder network 302 with the previously predicted y. r–1 The embeddings are combined to produce the decoder output y. r The decoder output can be a probability distribution, P(y i |y i-1 , ..., y0, x), in the current sub-word unit, y i Given the preceding unit, {y i-1 , ..., y0}, input, x. Although not shown, the decoder network 304 may include processing the output y from the decoder 304. r The softmax layer is then used. The output of the softmax layer is then used to select orthographic elements during beam search. Depending on the configuration of model 200, the softmax layer can be integrated with or separate from the decoder network 304.

[0034] Figure 4 This is a flowchart illustrating an example arrangement of a method 400 for dividing one or more trained teacher-trained automatic speech recognition (ASR) models 210 into multilingual student models 200. In operation 402, method 400 includes receiving multiple teacher training examples and multiple student training examples.

[0035] In operation 404, method 400 includes training one or more teacher ASR models 210 using multiple teacher training examples. Each teacher ASR model 210 is configured to output a corresponding text representation of the corresponding audio input.

[0036] In operation 406, the method includes generating a multilingual student ASR model 200 by performing sub-operations 406a and 406b. Sub-operation 406a includes training the multilingual student ASR model 200 using multiple student training examples. The student ASR model 200 is configured to receive audio input and output a corresponding text representation of the received audio input. Sub-operation 406b includes distilling one or more trained teacher ASR models 210 into the multilingual student ASR model 200 using adjustable distillation loss weights.

[0037] Figure 5This is a schematic diagram of an exemplary computing device 500 that can be used to implement the systems and methods described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0038] Computing device 500 includes a processor 510 (e.g., data processing hardware 134), a memory 520 (e.g., memory hardware 136), a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and a storage device 530. Each of components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and may be suitably mounted on a common motherboard or otherwise mounted. Processor 510 can process instructions for execution within computing device 500, including instructions stored in memory 520 or on storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device (e.g., a display 580 coupled to the high-speed interface 540). In other implementations, multiple processors and / or multiple buses, as well as multiple memories and memory types, may be suitably used. In addition, multiple computing devices 500 can be connected, each of which provides a portion of the necessary operation (e.g., as a server library, a set of blade servers, or a multiprocessor system).

[0039] Memory 520 stores information non-transitorily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-transitory memory 520 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) used by computing device 500. Examples of non-volatile memory include (but are not limited to) flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0040] Storage device 530 provides mass storage for computing device 500. In some embodiments, storage device 530 is a computer-readable medium. In various implementations, storage device 530 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array, including devices in a storage area network or other configurations. In other implementations, a computer program product is tangibly contained in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0041] High-speed controller 540 manages bandwidth-intensive operations of computing device 500, while low-speed controller 560 manages less bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 540 is coupled to memory 520, display 580 (e.g., via a graphics processor or accelerator), and high-speed expansion port 550, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 560 is coupled to storage device 530 and low-speed expansion port 590. Low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices, such as switches or routers, for example, via a network adapter.

[0042] The computing device 500 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 500a or multiple times as a laptop computer 500b in a set of such servers 500a, or as part of a rack server system 500c.

[0043] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0044] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product used to provide machine instructions and / or data to a programmable processor, non-transitory computer-readable media, devices and / or apparatuses (e.g., disks, optical disks, memory, programmable logic devices (PLDs)), including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0045] The processes and logic flows described in this specification can be executed by one or more programmable processors that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for receiving or transferring data to one or more mass storage devices used to store data, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; CD-ROMs and DVD-ROMs. Processors and memory can be supplemented by or integrated into dedicated logic circuits.

[0046] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer with a display device, such as a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touchscreen for displaying information to the user, and optionally also includes a keyboard and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0047] Various implementations have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other implementations are also within the scope of the appended claims.

Claims

1. A computer-implemented method (400) for distilling one or more trained teacher-based automatic speech recognition (ASR) models into multilingual student models, characterized in that, When executed by the data processing hardware (134), the data processing hardware (134) is caused to operate, the operation including: Receive multiple teacher training examples (152) and multiple student training examples (154); One or more teacher-trained automatic speech recognition (ASR) models (210) are trained using the plurality of teacher training examples (152), each teacher ASR model (210) including a recurrent neural network sensor (RNN-T) architecture and configured to output a corresponding text representation of the corresponding audio input (14); and The following generates a multilingual student ASR model (200) including an RNN-T architecture: The multilingual student ASR model (200) is trained using the plurality of student training examples (154), the student ASR model (200) being configured to receive audio input (14) and output a corresponding text representation (142) of the received audio input (14); and One or more trained teacher ASR models (210) are distilled into the multilingual student ASR model (200) by applying adjustable weights to the distillation loss of the multilingual student ASR model to generate adjustable distillation loss weights (222). The adjustable distillation loss weights (222) include a decreasing function based on a first RNN-T loss corresponding to the one or more teacher ASR models (210) and a second RNN-T loss corresponding to the multilingual student ASR model (200).

2. The method (400) according to claim 1, characterized in that, The one or more teacher ASR models (210) are configured to collectively recognize fewer languages ​​than the multilingual student ASR models (200).

3. The method (400) according to claim 1, characterized in that: Training the multilingual student ASR model (200) involves n training steps; and The adjustable distillation loss weight (222) includes a decreasing function that decreases based on the n training steps.

4. The method (400) according to claim 1, characterized in that, The decreasing function: Reduce the first RNN-T loss corresponding to the one or more teacher ASR models (210) over a period of time; as well as Increase the second RNN-T loss corresponding to the multilingual student ASR model (200) over a period of time.

5. The method (400) according to claim 1, characterized in that, Each of the one or more teacher ASR models (210) corresponds to a monolingual teacher ASR model (210).

6. The method (400) according to claim 1, characterized in that, The one or more teacher ASR models (210) correspond to a single multilingual ASR model.

7. A system (100) for distilling one or more trained teacher automatic speech recognition (ASR) models into multilingual student models, characterized in that, include: Data processing hardware (134); as well as A memory hardware (136) communicating with the data processing hardware (134), the memory hardware (136) storing instructions that, when executed on the data processing hardware (134), cause the data processing hardware (134) to perform operations, the operations including: Receive multiple teacher training examples (152) and multiple student training examples (154); One or more teacher-trained automatic speech recognition (ASR) models (210) are trained using the plurality of teacher training examples (152), each teacher ASR model (210) including a recurrent neural network sensor (RNN-T) architecture and configured to output a corresponding text representation of the corresponding audio input (14); and The following generates a multilingual student ASR model (200) including an RNN-T architecture: The multilingual student ASR model (200) is trained using the plurality of student training examples (154), the student ASR model (200) being configured to receive audio input (14) and output a corresponding text representation (142) of the received audio input (14); and One or more trained teacher ASR models (210) are distilled into the multilingual student ASR model (200) by applying adjustable weights to the distillation loss of the multilingual student ASR model to generate adjustable distillation loss weights (222). The adjustable distillation loss weights (222) include a decreasing function based on a first RNN-T loss corresponding to the one or more teacher ASR models (210) and a second RNN-T loss corresponding to the multilingual student ASR model (200).

8. The system (100) according to claim 7, characterized in that, The one or more teacher ASR models (210) are configured to collectively identify fewer languages ​​than the multilingual student ASR models (200).

9. The system (100) according to claim 7, characterized in that, Training the multilingual student ASR model (200) involves n training steps; and The adjustable distillation loss weight (222) includes a decreasing function that decreases based on the n training steps.

10. The system (100) according to claim 7, characterized in that, The decreasing function: Reduce the first RNN-T loss corresponding to the one or more teacher ASR models (210) over a period of time; as well as Increase the second RNN-T loss corresponding to the multilingual student ASR model (200) over a period of time.

11. The system (100) according to claim 7, characterized in that, Each of the one or more teacher ASR models (210) corresponds to a monolingual teacher ASR model (210).

12. The system (100) according to claim 7, characterized in that, The one or more teacher ASR models (210) correspond to a single multilingual ASR model.

Images