Self-supervised learning for audio processing

Abstract

A method, computer system, and apparatus including a computer program encoded on a computer storage medium for training an audio processing neural network including at least (1) a first encoder network having a first set of encoder network parameters and (2) a decoder network having a set of decoder network parameters. The system obtains a set of unlabeled audio data segments and generates a set of encoder training examples from the set of unlabeled audio data segments. The system trains a second encoder neural network including at least the first encoder neural network on the generated set of encoder training examples. The system also obtains one or more labeled training examples and trains the audio processing neural network on the labeled training examples.

Description

[Background technology]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 304,455, filed January 28, 2022.

[0002] This specification relates to training neural networks for performing audio processing.

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. Summary of the Invention

[0004] Described herein is a system, implemented as a computer program on one or more computers at one or more locations, for training an audio processing neural network, i.e., for determining network parameters of the audio processing neural network.

[0005] Generally, an audio processing neural network is configured to perform an audio processing task and includes at least (1) a first encoder network having a first set of encoder network parameters and (2) a decoder network having a set of decoder network parameters.

[0006] In one innovative aspect, this specification describes a method for training an audio processing neural network, the method being implemented by a system including one or more computers.

[0007] The system obtains a set of unlabeled audio data segments, each of which includes a plurality of frames of audio data corresponding to a plurality of time points.

[0008] The system further generates a set of encoder training examples from the set of unlabeled audio data segments. Each encoder training example includes a respective encoder training input and a respective encoder target label. To generate each encoder training example, the system generates a respective encoder training input by processing a respective unlabeled audio data segment in the set of unlabeled audio data segments, generates a respective set of projection vectors by applying a projection matrix to each of the set of audio data frames in each unlabeled audio data segment, and processes the respective set of projection vectors to generate a respective encoder target label for the encoder training example. The elements of the projection matrix are values ​​randomly selected according to a first distribution. The first distribution may be a uniform distribution. In some implementations of the provided method, the values ​​of the elements of the projection matrix remain unchanged while generating the set of encoder training examples.

[0009] The system then trains a second encoder neural network, which includes at least the first encoder neural network, on the generated set of encoder training examples to update at least a first set of encoder network parameters of the first encoder neural network to a set of pre-trained values.

[0010] The system also obtains one or more labeled training examples, each of which includes an audio data training input and a corresponding audio processing label.

[0011] The system further performs training of the audio processing neural network with the labeled training examples to update at least (1) a first set of network parameters of the encoder network and (2) a second set of network parameters of the decoder network, the first set of network parameters being initialized with a set of pre-trained values.

[0012] In some implementations of the provided methods, the system may further receive an audio input from an input device, process the audio input using at least the audio processing neural network after the audio processing neural network has been trained, and generate an audio processed output corresponding to the received audio input. The system may further output the audio processed output via an output device.

[0013] In some implementations of the provided methods, the audio processing task is a speech recognition task and the audio processing neural network is configured to process the audio data input to generate an output that characterizes linguistic content represented by the audio data input. For example, the output may include a predicted text transcript of the audio data input.

[0014] In some implementations of the provided method, each audio data frame includes a plurality of audio signal filter bank coefficients. For example, each audio data frame may include a vector of filter bank coefficients, such as a vector of log-mel filter bank coefficients. In one example, the audio data frame is an 80-dimensional log-mel filter bank coefficient.

[0015] In some implementations of the provided method, the system further performs normalization of each audio data frame in the set of unlabeled audio data segments to a predetermined mean and a predetermined standard deviation before generating the set of encoder training examples from the set of unlabeled audio data segments.

[0016] In some implementations of the provided method, to generate the respective encoder training inputs, the system generates respective masked audio data segments from the respective unlabeled audio data segments. In particular, the system selects one or more audio data frames in the respective unlabeled audio data segments and replaces data in the one or more selected audio data frames with noise data values. The noise data values ​​may be sampled from a second distribution, such as a normal distribution with a mean of zero.

[0017] The system may train a second encoder neural network with the encoder training task of predicting labels for one or more selected audio data frames in each unlabeled audio data segment from one or more input audio data frames in each masked audio data segment. The set of audio data frames in each unlabeled audio data segment may include the one or more selected audio data frames.

[0018] To select one or more audio data frames in each unlabeled audio data segment, the system may determine, for each given audio data frame in each unlabeled audio data segment, whether to select the given audio data frame with a predetermined probability, and further, in response to a determination to select the given audio data frame, select the given audio data frame and a predetermined number of consecutive audio data frames after the given audio data frame in the respective unlabeled audio data segment.

[0019] In some implementations of the provided method, the values ​​of the elements of the projection matrix remain unchanged while generating the set of encoder training examples.

[0020] In some embodiments of the provided method, processing the respective sets of projection vectors to generate the respective encoder target labels includes, for each given projection vector in the respective sets of projection vectors, the system identifies a codebook vector from the set of codebook vectors that has the shortest distance from the given projection vector. The system then determines an element value of the encoder target label as an index of the identified codebook vector in the set of codebook vectors. In those embodiments, the encoder training task may be a task of predicting an index of the identified codebook vector for one or more selected audio frames from each masked audio data segment. The set of codebook vectors may be randomly generated according to a third distribution, such as a uniform distribution. The codebook vectors may be left unchanged during the generation of the set of encoder training examples.

[0021] In some embodiments of the provided methods, the second encoder neural network further comprises a softmax layer after the first encoder network.

[0022] In some implementations of the provided methods, the audio processing neural network is a non-streaming model configured to predict an audio processing output at a given time point based on audio data input including data frames corresponding to time points before and after the given time point.

[0023] In some implementations of the provided methods, the audio processing neural network is a streaming model configured to predict an audio processing output at a given time point based on audio data input including data frames corresponding to time points before or after the given time point.

[0024] In some embodiments of the provided methods, the decoder network includes one or more recurrent neural networks (RNNs). For example, the decoder network may include one or more LSTM networks. In some embodiments, the decoder network may include an RNN transducer.

[0025] In some implementations of the provided method, the audio processing neural network further includes one or more additional layers including a projection layer between the first encoder network and the decoder network.

[0026] The present specification also provides a system including one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the training method described above.

[0027] The present specification also provides one or more computer storage media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the training method described above.

[0028] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages.

[0029] This specification describes techniques for performing self-supervised learning that combines pre-training of audio processing neural networks using unsupervised data and fine-tuning using labeled training examples. Compared to fully supervised learning techniques, self-supervised learning provides a significant improvement in the quality of audio processing models, as this approach allows models to learn from large unsupervised datasets in addition to supervised learning. The ability to learn from unsupervised data is particularly beneficial when labeled data is limited, opening up new possibilities for low-resource languages ​​and domains.

[0030] Furthermore, the techniques described herein provide several advantages over conventional self-supervised learning techniques. Conventional self-supervised learning techniques typically involve learning feature representations of input audio data and then mapping the feature representations to discrete tokens (e.g., discrete text tokens). This can lead to challenges that can hinder the applicability and effectiveness of audio processing models. For example, the integration of representation learning and self-supervised learning often requires a model to serve as a provider of audio representations while still being effective for downstream tasks. However, an effective representation learning model is not always effective for downstream tasks. For example, a good representation learning model may need access to the future context of an utterance, while a downstream task may require a low-latency model that prohibits access to future context. Furthermore, the objectives of representation learning and self-supervised learning are not always aligned, and the combination of these two aspects often results in complex and computationally intensive algorithms.

[0031] The techniques described herein overcome certain challenges associated with conventional self-supervised learning techniques because they do not require learning feature representations of unlabeled audio data during pre-training. Instead of performing feature representation learning, in some implementations of the described techniques, the system trains an encoder network of an audio processing model to predict masked sections of an audio data input based on unmasked audio data processed with a random projection quantizer. The random projection quantizer does not require representation learning and is independent of the audio processing model. As a result, the described techniques can be used in a wide range of audio processing network architectures and can achieve state-of-the-art prediction accuracy and speed for both streaming and non-streaming tasks while remaining computationally efficient.

[0032] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. [Brief description of the drawings]

[0033] [Figure 1] FIG. 1 illustrates an exemplary training system for training an audio processing neural network. [Diagram 2] FIG. 1 illustrates an example process for generating self-supervised training examples for training an encoder network. [Diagram 3] 1 is a flow diagram illustrating an example process for training an audio processing neural network. [Figure 4] FIG. 1 illustrates a performance comparison between an audio processing neural network trained using random projection quantization pre-training and an audio processing neural network trained using a baseline method. [Diagram 5]FIG. 13 illustrates another performance comparison between an audio processing neural network trained using random projection quantization pre-training and an audio processing neural network trained using a baseline method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Like reference numbers and designations in the various drawings indicate like elements.

[0035] 1 illustrates an example of a training system 100 for training an audio processing neural network 160. System 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, in which the systems, components, and techniques described herein below may be implemented.

[0036] Typically, after the training process, system 100 or another system can use the trained audio processing neural network 160 to process audio data and perform audio processing tasks (e.g., speech recognition tasks, language identification tasks, voice recognition tasks, music transcription tasks, audio classification tasks, etc.).

[0037] In a particular example of a speech recognition task, a trained neural network is used to process audio data (e.g., data received from an audio sensing device such as a microphone) and generate a speech recognition output (e.g., an output indicative of the spoken content in the audio data, such as a text transcript of the spoken content).

[0038] As another example of a hot word detection task, if the input to a neural network is a sequence of audio data representing spoken speech, the output generated by the neural network can indicate whether a particular word or phrase (a "hot word") was spoken in the utterance.

[0039] As an example of a language identification task, if the input to a neural network is a sequence of audio data representing spoken speech, the output produced by the neural network can identify the natural language that is spoken.

[0040] As an example of a voice recognition task, the input to the neural network may be a sequence of audio data representing spoken audio from a particular person, and the output generated by the neural network may identify the identity of the particular person.

[0041] As an example of a music transcription task, the input to the neural network may be an audio data sequence representing a musical melody or musical performance, and the output generated by the neural network may identify a transcription of the musical melody or performance to automatically generate musical notation.

[0042] As an example of an audio classification task, the input to the neural network may be a sequence of audio data representing sounds recorded by an audio sensor, and the output generated by the neural network may identify the type of sound indicative of characteristics or attributes of the sound (e.g., the identity of the object that produced the sound (e.g., a particular type of device, a certain type of animal), the state of the object that produced the sound (e.g., under normal or abnormal conditions)), or may identify the type or state of the sensor that recorded the sound.

[0043] In general, the input audio data includes multiple data frames corresponding to multiple time points. In some cases, the audio processing neural network 160 may be a non-streaming model. That is, the audio processing neural network 160 is configured to make a prediction for a given time point t based on both past and future data frames of the input audio data (i.e., data frames at corresponding time points both before and after the given time point). In some other cases, the audio processing neural network 160 may be a streaming model, where the network is configured to predict the audio processing output for a given time point based on past data frames of the input audio data (i.e., data frames at corresponding time points up to the given time point), but not based on future data frames of the input audio data.

[0044] The audio processing neural network 160 typically includes an encoder network 162 configured to map audio data input to coded data that encodes contextual information of the audio data input, and a decoder network 164 configured to map inputs derived from the coded data to audio processing outputs.

[0045] The encoder network 162 and the decoder network 164 may have any suitable architecture. A particular example of an encoder architecture is described in Zhang, et al., “Pushing the limits of semisupervised learning for automatic speech recognition,” arXiv preprint arXiv:2010.10504, 2020. Another example of an encoder architecture is described in Yu, et al., “FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization,” in Proc. ICASSP, 2021. As a particular example, the decoder architecture may include an RNN transducer with an additional layer of unidirectional LSTM (RNN-T).

[0046] The system 100 trains the audio processing neural network 160 using a self-supervised learning process.

[0047] The system 100 includes a pre-training engine 120 that performs a pre-training process, and a fine-tuning engine 150 that performs a fine-tuning training process.

[0048] During pre-training, the pre-training engine 120 uses the unlabeled audio data 110 to update the network parameters of the encoder network 122 to a set of pre-trained values. As used herein, unlabeled audio data refers to audio data for which training labels are not available or, even when some of the training labels are available, no training labels are used during training.

[0049] During fine-tuning, the fine-tuning engine 150 uses the labeled training data 140 to train the audio processing neural network 160 end-to-end and update both the network parameters of the encoder network 122 (the parameters are initialized to pre-trained values) and the network parameters of the decoder network 124. The labeled training data 140 typically includes a set of labeled training examples. Each labeled training example includes an audio data training input and a corresponding audio processing label. The audio processing label is the target output of the audio processing neural network 160 for an audio processing task when the corresponding audio data training input is processed by the audio processing neural network 160.

[0050] Typically, the pre-training process trains the encoder network 162 to learn the contextual information in the unlabeled audio data 110.

[0051] To pre-train the encoder network 162 using the unlabeled audio data 110, the pre-training engine 120 of the system 100 defines one or more self-supervised training tasks from the unlabeled audio data 110 and trains the encoder network 162 on the one or more self-supervised training tasks.

[0052] For example, a self-supervised training task is one that requires the encoder network to process a partially masked, unlabeled audio data segment (i.e., the data in the masked portion has been replaced by other data (e.g., data sampled from a noise signal)) to predict the encoded data of the masked portion of the data segment.

[0053] In particular, the system may generate a set of encoder training examples 130 for a self-supervised training task from a set of unlabeled audio data segments, where each encoder training example includes a respective encoder training input and a respective encoder target label.

[0054] The pre-training engine 120 may generate each encoder training input for each encoder training example 130 by processing each unlabeled audio data segment. In some embodiments, the audio segment masking engine 122 may generate the encoder training input by masking one or more sections of each unlabeled audio data segment. That is, the audio segment masking engine 122 may replace data in one or more sections of consecutive frames of the audio data segment with other data that is unrelated to the content of the audio data segment (e.g., data values ​​sampled from a noise signal, or data values ​​set to a predetermined fixed value, e.g., zero).

[0055] The pre-training engine 120 may generate a target label for the encoder in each encoder training example 130 by performing a projection onto each unlabeled audio data segment followed by quantization. To perform the projection, the random projection engine 124 may apply a projection matrix to each of a set of audio data frames in each unlabeled audio data segment to generate projection data including, for example, a set of projection vectors. For example, to generate a projection vector for an audio data frame including a vector of filter bank coefficients, the random projection engine may multiply the projection matrix by the vector of filter bank coefficients.

[0056] In some cases, the projection data is generated from a set of audio data frames selected to be masked by the audio segment masking engine 122 during generation of the respective encoder training inputs. In some other cases, the projection data may be generated from all data frames in each unlabeled audio data segment. The elements of the projection matrix may be values ​​randomly selected according to a predefined distribution, such as a uniform distribution.

[0057] To perform quantization, the quantization engine 126 identifies a template vector (or codebook vector) from the set of template vectors that has the shortest distance from a given projection vector, and assigns the element value of the encoder's target label as the index of the identified codebook vector in the set of codebook vectors. The set of codebook vectors may also be generated randomly according to a predefined distribution. In some implementations, the element values ​​of the projection matrix and / or the codebook vectors remain fixed during the pre-training process. In that case, the self-supervised training task for the encoder may be the task of predicting the index of the identified codebook vector for a selected audio frame from each masked audio data segment.

[0058] Typically, a random projection performs a dimensionality reduction of the audio signal, while a random codebook provides an approximated discrete representation of the audio data distribution.

[0059] In some embodiments, during pre-training, the pre-training engine 120 may include a pre-trained neural network head 162b (e.g., including one or more network layers) 162b as an output layer for the encoder network 162 to output a predicted label. That is, the pre-training engine 120 trains a neural network 162a including the encoder network 162 and the pre-trained neural network head 162b. The pre-trained neural network head 162b is not included during the fine-tuning process when the encoder network 162 and the decoder network 164 are trained end-to-end. The pre-training engine 120 may use the pre-trained neural network head 162b to accommodate different architectures of the encoder network 162 and / or different types of tasks, so that the same training framework may be used for various situations. In particular, the pre-trained neural network head 162b may be used to select the coding embeddings output by the previous layer and generate a predicted label. In some embodiments, the system may reuse the same pre-trained encoder network 162 for multiple different downstream tasks, or may train a multi-task decoder that performs multiple different tasks in parallel.

[0060] In a particular example, the pre-trained neural network head 162b may be a softmax layer that maps the selected embedding(s) to their respective logits (e.g., via one or more learned linear transformations) and then maps the logits to probability distributions over labels (e.g., over codebook vector indices).

[0061] For example, to perform training for a non-streaming task, the system 100 may use the softmax layer 162b to select the encoding embeddings corresponding to both time points before and after the masked frame to generate a predicted label for the masked frame. Alternatively, to perform training for a streaming task, the system 100 may use the softmax layer 162b to select only the encoding embeddings corresponding to the time points of the masked frame, and optionally only the encoding embeddings corresponding to time points before the masked frame to generate a predicted label for the masked frame. Optionally, for each masked set of one or more frames, the softmax layer 162b processes the encoding embeddings for the time points corresponding to the masked set to generate a prediction for the masked set of frames.

[0062] Once the encoder training examples 130 are generated, the pre-training engine 120 may update the network parameters of the encoder network 162 to pre-trained values ​​by minimizing a self-supervised loss calculated based on the generated encoder training examples 130. The pre-training engine 120 may update the parameters via any suitable back-propagation based machine learning method, for example using an Adam or AdaGrad optimizer.

[0063] Similarly, during fine-tuning, the fine-tuning training engine 150 updates the network parameters of the encoder network 162 and the decoder network 164 by minimizing training losses, including supervised training losses, which are calculated based at least on the labeled training examples 140.

[0064] In some implementations, the audio processing neural network 160 further includes one or more additional layers, including a projection layer, between the encoder network 162 and the decoder network 164. The additional layers can be used to help the audio processing neural network 160 adapt to downstream audio processing tasks. For example, the projection layer is configured to multiply the output of the previous layer with a matrix, allowing the output of the encoder to be projected into dimensions that match the input dimensions of the decoder.

[0065] Although the above description describes a configuration in which self-supervised pre-training (using unlabeled data 110) is performed before supervised fine-tuning (using labeled data 140), in some cases, pre-training and fine-tuning may be performed together. For example, the self-supervised pre-training loss may be used as an auxiliary loss during the fine-tuning process. That is, the system 100 may train the network on batches including labeled data and unlabeled data, or may train the network alternately on labeled data and unlabeled data. For unlabeled data, the system 100 may use the self-supervised pre-training loss to update the encoder 162. For labeled data, the system 100 may either (i) use the supervised loss to update the networks of the encoder 162 and the decoder 164, or (ii) use both the supervised loss and the self-supervised loss to update the networks of the encoder and the decoder.

[0066] After the audio processing neural network 160 is trained, the system 100 or another system can use the trained audio processing neural network 160 to perform an audio processing task. For example, the system 100 can receive an audio input from an input device and process the audio input using at least the trained audio processing neural network 160 to generate an audio processing output 180 corresponding to the received audio input. The system 100 can further output the audio processing output 180 via an output device. For example, the audio processing task can be a speech recognition task, and the audio processing neural network 160 is configured to process an audio data input to generate an output that characterizes the linguistic content represented by the audio data input. In this case, the output 180 can include a predicted text transcript of the audio data input and can be presented to a user using an output device such as a computer display.

[0067] 2 illustrates an exemplary process 200 for generating self-supervised training examples for training an encoder network. For convenience, the process 200 is described as being performed by one or more computer systems located at one or more locations. For example, the training system 100 of FIG. 1 may be suitably programmed in accordance with this specification to perform the process 200.

[0068] As shown in Figure 2, the system receives an unlabeled audio segment 210. The unlabeled audio segment 210 includes a set of frames of audio data corresponding to a number of time points (e.g., points along the horizontal axis of 210). Each frame of audio data characterizes an audio signal at a corresponding time point. In one example, the frames of audio data specify a number of audio signal filter bank coefficients.

[0069] The system generates encoder training inputs 232 by masking the unlabeled audio segments 210. In particular, the system can select one or more sections of consecutive audio data frames in the unlabeled audio data segments 210 and replace data in the selected audio data frames with noise data values. The noise data values ​​can be sampled from a predefined distribution (e.g., a normal distribution with a mean of 0 and a standard deviation of 0.1). In some implementations, the masking method can sample every frame with a certain probability whether to apply a mask to a section starting from the frame. Each mask section extends from a starting frame of a fixed length.

[0070] The system applies a projection matrix 224 to the frames of audio data in the unlabeled audio data segment 210 to generate a set of projection vectors 225. The elements of the projection matrix 224 are randomly selected values ​​according to a predefined distribution (e.g., a uniform distribution or any other suitable distribution). The projection matrix 224 can be generated using any suitable method. For example, the Xavier initialization process can be used to generate the projection matrix 224. For details of the Xavier initialization process, see Glorot, et al., “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-256. PMLR, 13-15 May 2010. In some implementations, the elements of the projection matrix 224 are fixed during the generation of the encoder training examples.

[0071] In some implementations, before applying the projection matrix 224, the system may perform normalization of each audio data frame in the unlabeled audio data segment 210 to a predefined mean and a predefined standard deviation (e.g., mean 0 and standard deviation 1). The normalization may prevent random projections from collapsing into a small subset of vectors.

[0072] After the projection vectors 225 are generated, the system can map the projection vectors 225 to a set of encoder target labels 234 using a codebook lookup process based on a codebook that includes a set of indexed codebook vectors 226. In particular, for each given projection vector, the system identifies a codebook vector from the codebook that has the shortest distance from the given projection vector, and determines the element value of the encoder target label as the index of the identified codebook vector in the codebook. The codebook vectors can be vectors that are randomly generated according to a suitable distribution (e.g., uniform distribution). In some implementations, the codebook vectors remain unchanged while generating the encoder training examples.

[0073] 3 is a flow diagram illustrating an example process 300 for training an audio processing neural network. The audio processing neural network is configured to perform an audio processing task and includes at least (1) a first encoder network having a first set of encoder network parameters and (2) a decoder network having a set of decoder network parameters.

[0074] For convenience, process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a training system (such as training system 100 described with reference to FIG. 1) may be suitably programmed in accordance with this specification to perform process 300.

[0075] At 310, the system obtains a set of unlabeled audio data segments, each of which includes a plurality of frames of audio data corresponding to a plurality of time points.

[0076] At 320, the system generates a set of encoder training examples from the set of unlabeled audio data segments, each of which includes a respective encoder training input and a respective encoder target label.

[0077] To generate each encoder training example, the system generates a respective encoder training input by processing each unlabeled audio data segment in the set of unlabeled audio data segments, generates a respective set of projection vectors by applying a projection matrix to each of the set of audio data frames in each unlabeled audio data segment, and processes the respective set of projection vectors to generate a respective encoder target label for the encoder training example. The elements of the projection matrix are values ​​randomly selected according to a first distribution. The first distribution may be a uniform distribution. In some embodiments, the values ​​of the elements of the projection matrix remain unchanged while generating the set of encoder training examples.

[0078] At 330, the system trains a second encoder neural network, which includes at least the first encoder neural network, on the generated set of encoder training examples to update at least a first set of encoder network parameters of the first encoder neural network to a set of pre-trained values.

[0079] At 340, the system obtains one or more labeled training examples, each of which includes an audio data training input and a corresponding audio processing label that is a target output of the audio processing neural network 160 for an audio processing task when the corresponding audio data training input is processed by the audio processing neural network.

[0080] At 350, the system further performs training of the audio processing neural network with the labeled training examples to update at least (1) a first set of network parameters of the encoder network and (2) a second set of network parameters of the decoder network, where the first set of network parameters are initialized with a set of pre-trained values.

[0081] FIG. 4 illustrates a performance comparison 400 between an audio processing neural network trained using the random-projection quantization pre-training ("BEST-RQ") described with reference to FIGS. 1, 2, and 3 and an audio processing neural network trained using state-of-the-art baseline methods. State-of-the-art baseline methods include "wav2vec 2.0" described in Baevski, et al., "wav2vec 2.0: A framework for self-supervised learning of speech representations," arXiv preprint arXiv:2006.11477, 2020b, and "w2v-BERT" described in Chung, et al., "W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training," arXiv preprint arXiv:2108.06209, 2021.

[0082] An audio processing neural network was trained to perform a speech recognition task. By using the model trained on the LibriLight dataset, we calculated the word error rate (WER) and relative latency. As shown in Figure 4, the training method described herein typically outperforms baseline methods in both WER and latency for both streaming and non-streaming pre-training.

[0083] 5 shows a performance (specifically, WER) comparison 500 of audio processing neural networks for a speech recognition task for multiple languages. As shown in FIG. 5, the training method described herein ("BEST-RQ") generally outperforms the baseline method for most of the languages ​​tested.

[0084] The term "configured" is used herein for systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the operating system to perform the operation or action. A system of one or more computer programs configured to perform a particular operation or action means that the one or more programs contain instructions that, when executed by a data processing apparatus, cause the apparatus to perform the operation or action. Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, embodied computer software or firmware, computer hardware, including the structures disclosed herein and structural equivalents thereof, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or for controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random access memory device, or a serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded into an artificially generated propagated signal (e.g., a machine-generated electrical, optical or electromagnetic signal that is generated to encode information for transmission to a suitable receiving device for execution by a data processing device).

[0085] The term "data processing apparatus" refers to data processing hardware and encompasses any kind of apparatus, device, and machine for data processing, including, by way of example, a programmable processor, a computer, or multiple processors or multiple computers. An apparatus may be or include special purpose logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit)). An apparatus optionally includes, in addition to hardware, code that creates an execution environment for computer programs (e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof).

[0086] A computer program (also referred to or described as a program, software, software application, app, module, software module, script, or code) can be written in any form of programming language, such as compiled or interpreted, or declarative or procedural, and it can be deployed in any form, including as a stand-alone program or module, as a component, subroutine, or other unit suitable for use in a computing environment. A program may correspond to a file in a file system, but this is not necessarily the case. A program can be stored in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), or in a single file dedicated to the program, or in multiple cooperating files (e.g., a file that stores one or more modules, subprograms, or portions of code). A computer program can be deployed to run on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0087] The term "database" is used broadly herein to refer to any collection of data. The data need not be structured in any particular way, or even at all, and may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0088] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine may be implemented as one or more software modules or components and may be installed on one or more computers in one or more locations. In some cases, one or more computers may be dedicated to a particular engine, and in other cases, multiple engines may be installed and executed on the same computer or computers.

[0089] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by performing operations on input data to generate output. The processes and logic flows may also be performed by special purpose logic circuitry (e.g., FPGAs or ASICs), or a combination of special purpose logic circuitry and one or more programmed computers.

[0090] A computer suitable for executing a computer program may be based on a general-purpose or dedicated microprocessor, or both, or on other types of central processing units. Typically, the central processing unit receives instructions and data from a read-only memory, a random access memory, or both. The basic components of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special purpose logic circuitry. Typically, the computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or the computer is operatively connected to receive data from, transmit data to, or both of, one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks for storing data. However, a computer need not have such devices. Additionally, computers may be incorporated into other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive, etc.), to name just a few.

[0091] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0092] To interact with a user, embodiments of the subject matter described herein may be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and a pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to interact with the user. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, the computer may interact with the user by sending documents to a device used by the user and receiving documents from a device used by the user (e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser). The computer may also interact with the user by sending a text message or other form of message to a personal device (e.g., a smart phone running a messaging application) and then receiving a response message from the user.

[0093] A data processing device for implementing machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and numerically-computing parts of machine learning training or production (i.e., inference, workloads).

[0094] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework or the Jax framework.

[0095] An embodiment of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., as a data server, or a middleware component, e.g., an application server, or a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an application that allows a user to interact with an embodiment of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), e.g., the Internet.

[0096] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to a user device, for example for the purpose of displaying the data to and receiving user input from a user interacting with the device functioning as a client. Data generated at the user device (e.g., results of user interaction) can be received at the server from the device.

[0097] Although the present specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claimable subject matter, but rather as descriptions of features that may be unique to certain embodiments of a particular invention. Certain features described herein as separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described as a single embodiment may also be implemented in multiple embodiments, either separately or in any suitable subcombination. Furthermore, even if multiple features are described above as functioning in a particular combination and are initially recited as such in the claims, one or more features may, in some cases, be deleted from such combinations recited in the claims, and further, the combinations recited in the claims may be directed to subcombinations or variations of subcombinations.

[0098] Similarly, although operations may be illustrated in the figures and / or described in the claims in a particular order, this should not be construed as requiring that such operations be performed in the particular order or sequence shown, nor should it be construed as requiring that all of the operations shown be performed in order to achieve a desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and the program components and systems described may typically be integrated into a single software product or packaged into multiple software products.

[0099] Specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still produce desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A computer-implemented method for training an audio processing neural network configured to perform an audio processing task, the audio processing neural network including at least (1) a first encoder network having a first set of encoder network parameters, and (2) a decoder network having a set of decoder network parameters, the method comprising: obtaining a set of unlabeled audio data segments, each unlabeled audio data segment including a respective plurality of frames of audio data corresponding to a plurality of time points; generating a set of encoder training examples from the set of unlabeled audio data segments, each encoder training example including a respective encoder training input and a respective encoder target label, said generating including, for each encoder training example: generating respective encoder training inputs by processing each unlabeled audio data segment in the set of unlabeled audio data segments; generating a respective set of projection vectors by applying a projection matrix to each of a set of audio data frames in the respective unlabeled audio data segment, the elements of the projection matrix being values ​​randomly selected according to a first distribution; processing each set of projection vectors to generate a respective encoder target label for each of the encoder training examples; generating a set of encoder training examples, training a second encoder neural network that includes at least the first encoder neural network on the set of generated encoder training examples to update at least a first set of encoder network parameters of the first encoder neural network to a set of pre-trained values; obtaining one or more labeled training examples, each labeled training example including an audio data training input and a corresponding audio processing label; performing training of the audio processing neural network with the labeled training examples to update at least (1) a first set of network parameters of the encoder network and (2) a second set of network parameters of the decoder network, the first set of network parameters being initialized with the set of pre-trained values; 4. A computer-implemented method comprising:

2. receiving an audio input from an input device; after the audio processing neural network has been trained, processing the audio input using at least the audio processing neural network to generate the audio processed output corresponding to the received audio input; The method of claim 1 further comprising:

3. The method of claim 2 , further comprising outputting the audio processing output via an output device.

4. 4. The method of claim 1, wherein the audio processing task is a speech recognition task and the audio processing neural network is configured to process an audio data input to generate an output that characterizes linguistic content represented by the audio data input.

5. The method of claim 4 , wherein the output comprises a predicted text transcript of the audio data input.

6. The method of claim 1 , wherein each frame of audio data comprises a plurality of audio signal filterbank coefficients.

7. 7. The method of claim 1, further comprising: performing a normalization of each audio data frame in the set of unlabeled audio data segments to a predetermined mean and a predetermined standard deviation before generating the set of encoder training examples from the set of unlabeled audio data segments.

8. Generating the respective encoder training inputs includes generating respective masked audio data segments from the respective unlabeled audio data segments, and generating the respective masked audio data segments includes: selecting one or more frames of audio data in each unlabeled segment of audio data; and replacing data in the one or more selected frames of audio data with noise data values.

9. Performing training of the second encoder neural network includes:

9. The method of claim 8, comprising training the second encoder neural network with an encoder training task to predict labels for the one or more selected audio data frames in the respective unlabeled audio data segments from one or more input audio data frames in the respective masked audio data segments.

10. The method of claim 9 , wherein the set of audio data frames in the respective unlabeled audio data segment includes the one or more selected audio data frames.

11. A method according to any one of claims 8 to 10, wherein the noise data values ​​are sampled from a second distribution.

12. The method of claim 11 , wherein the second distribution is a normal distribution with a mean of zero.

13. Selecting the one or more frames of audio data in each unlabeled audio data segment may include:

13. The method of claim 8, comprising: for each given audio data frame of the respective unlabelled audio data segment, determining with a predetermined probability whether to select the given audio data frame; and in response to determining to select the given audio data frame, selecting the given audio data frame and a predetermined number of consecutive audio data frames after the given audio data frame in the respective unlabelled audio data segment.

14. The method of claim 1 , wherein the values ​​of the elements of the projection matrix remain unchanged while generating the set of encoder training examples.

15. The method of claim 1 , wherein the first distribution is a uniform distribution.

16. Processing the respective sets of projection vectors to generate target labels for the respective encoders includes, for a given projection vector in the respective sets of projection vectors, identifying a codebook vector from a set of codebook vectors that has a smallest distance from the given projection vector; determining an element value of the encoder target label as an index of the identified codebook vector in the set of codebook vectors; 16. The method of any one of claims 1 to 15, comprising:

17. 17. The method of claim 16 and dependent on claim 10, wherein the training task is to predict, from the respective masked audio data segments, an index of the codebook vector identified for the one or more selected audio frames.

18. The method of claim 16 or 17, wherein the set of codebook vectors is randomly generated according to a third distribution.

19. The method of claim 18 , wherein the third distribution is a uniform distribution.

20. 20. The method of claim 18 or 19, wherein the set of codebook vectors remains unchanged while generating the set of encoder training examples.

21. 21. The method of claim 1, wherein the second encoder neural network further comprises a softmax layer after the first encoder network.

22. 22. The method of claim 1, wherein the audio processing neural network is a non-streaming model configured to predict an audio processing output at a given time point based on an audio data input comprising data frames corresponding to time points before and after the given time point.

23. 22. The method of claim 1, wherein the audio processing neural network is a streaming model configured to predict an audio processing output at a given time point based on an audio data input comprising a data frame prior to or corresponding to the given time point.

24. 24. The method of claim 1, wherein the decoder network comprises one or more recurrent neural networks (RNNs).

25. 25. The method of claim 24, wherein the one or more RNNs include one or more LSTM networks.

26. The method of claim 25 , wherein the decoder network includes an RNN transducer.

27. 27. The method of claim 1, wherein the audio processing neural network further comprises one or more additional layers including a projection layer between the first encoder network and the decoder network.

28. 1. A computer-implemented method comprising: receiving new audio data input; and processing the new audio data input using an audio processing neural network trained using the method of any of claims 1 to 27 to generate an audio processing output for an audio processing task.

29. 29. A system comprising one or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform a method according to any one of claims 1 to 28.

30. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 28.

Images