Self-supervised learning for audio processing
By combining self-supervised learning and pre-training and fine-tuning methods of unsupervised data in audio processing neural networks, the problem of poor audio processing models in low-resource languages and fields is solved, and the problem of poor audio processing models in low-resource languages and fields is realized Efficient and accurate audio processing tasks.
Patent Information
- Application Number
- CN202380027524.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-01-28
- Filing Date
- 2023-01-30
- Publication Date
- 2024-11-05
AI Technical Summary
The existing audio processing neural network training methods are not effective in low-resource languages and fields, and the self-supervised learning technology has problems such as inconsistent learning and downstream task goals and high computing requirements.
Using self-supervised learning combined with unsupervised data pre-training and fine-tuning methods, unmasked audio data is processed through a random projection quantizer to predict the masked segment, avoiding the necessity of feature representation learning, and is applicable to a wide range of applications. audio processing network architecture.
Improve the quality of the audio processing model, overcome the challenges of conventional self-supervised learning, and achieve high accuracy and high efficiency for streaming and non-streaming tasks, suitable for low resource conditions.
Smart Images

Figure CN118901098A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. Provisional Application No. 63 / 304,455, filed on January 28, 2022. Background Art
[0003] This specification relates to training neural networks for performing audio processing.
[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict the output of a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input according to the current values of the corresponding set of parameters. Summary of the invention
[0005] This specification describes a system implemented as a computer program on one or more computers at one or more locations that trains an audio processing neural network, ie, determines network parameters of the audio processing neural network.
[0006] Generally speaking, an audio processing neural network is configured to perform an audio processing task and includes at least (1) a first encoder network having a first encoder network parameter set and (2) a decoder network having a decoder network parameter set.
[0007] In one innovative aspect, the specification describes a method for training an audio processing neural network. The method is implemented by a system including one or more computers.
[0008] The system obtains a set of unlabeled audio data segments. Each unlabeled audio data segment includes a corresponding plurality of audio data frames corresponding to a plurality of time points.
[0009] The system also generates an encoder training sample set from an unlabeled audio data segment set. Each encoder training sample includes a corresponding encoder training input and a corresponding encoder target label. In order to generate each encoder training sample, the system generates a corresponding encoder training input by processing a corresponding unlabeled audio data segment in the unlabeled audio data segment set, generates a corresponding projected vector set by applying a projection matrix to each audio data frame in the audio data frame set in the corresponding unlabeled audio data segment, and processes the corresponding projected vector set to generate the corresponding encoder target label of the encoder training sample. The elements of the projection matrix are values randomly selected according to a first distribution. The first distribution can be a uniform distribution. In some implementations of the provided method, the values of the elements of the projection matrix remain unchanged during the generation of the encoder training sample set.
[0010] The system then performs training of a second encoder neural network including at least the first encoder neural network on the generated set of encoder training examples to update at least a first encoder network parameter set of the first encoder neural network to a pre-trained set of values.
[0011] The system also obtains one or more labeled training examples. Each labeled training example includes an audio data training input and a corresponding audio processing label.
[0012] The system also performs training of the audio processing neural network on the labeled training examples to update at least (1) a first set of network parameters of the encoder network and (2) a second set of network parameters of the decoder network, wherein the first set of network parameters is initialized with a pre-trained set of values.
[0013] In some implementations of the provided method, the system also receives an audio input from an input device, and after the audio processing neural network has been trained, processes the audio input using at least the audio processing neural network to generate an audio processing output corresponding to the received audio input. The system can also output the audio processing output through an output device.
[0014] In some implementations of the provided methods, the audio processing task is a speech recognition task, and the audio processing neural network is configured to process the audio data input to generate an output representing the language content represented by the audio data input. For example, the output may include a predicted text transcription of the audio data input.
[0015] In some implementations of the provided method, each audio data frame includes a plurality of audio signal filter bank coefficients. For example, each audio data frame may include a vector of filter bank coefficients, such as a vector of log-mel filter bank coefficients. In one example, the audio data frame is an 80-dimensional log-mel filter bank coefficient.
[0016] In some implementations of the provided method, the system further performs normalization of each audio data frame in the set of unlabeled audio data segments to a predefined mean and a predefined standard deviation before generating the encoder training example set from the set of unlabeled audio data segments.
[0017] In some implementations of the provided method, to generate corresponding encoder training inputs, the system generates corresponding masked audio data segments from corresponding unlabeled audio data segments. In particular, the system selects one or more audio data frames in the corresponding unlabeled audio data segments and replaces the data in the one or more selected audio data frames with noise data values. The noise data values can be sampled from a second distribution (such as a normal distribution with a mean of 0).
[0018] The system can train a second encoder neural network on the encoder training task to predict labels of one or more selected audio data frames in a corresponding unlabeled audio data segment from one or more input audio data frames in a corresponding masked audio data segment. The set of audio data frames in the corresponding unlabeled audio data segment can include the one or more selected audio data frames.
[0019] In order to select one or more audio data frames in a corresponding unlabeled audio data segment, the system can determine, for each given audio data frame in the corresponding unlabeled audio data segment, whether to select a given audio data frame with a predefined probability, and in response to the given audio data frame having been determined to be selected, select the given audio data frame and a predefined number of consecutive audio data frames after the given audio data frame in the corresponding unlabeled audio data segment.
[0020] In some implementations of the provided methods, the values of the elements of the projection matrix remain unchanged during the generation of the encoder training example set.
[0021] In some implementations of the provided method, processing the corresponding projected vector set to generate the corresponding encoder target tag includes, for each given projected vector in the corresponding projected vector set, the system identifying the codebook vector from the codebook vector set with the shortest distance to the given projected vector. The system then determines the element value of the encoder target tag as the index of the identified codebook vector in the codebook vector set. In these implementations, the training task of the encoder can be the task of predicting the index of the codebook vector identified for one or more selected audio frames from the corresponding masked audio data segment. The codebook vector set can be randomly generated according to a third distribution (such as a uniform distribution). The codebook vector set can remain unchanged during the generation of the encoder training sample set.
[0022] In some implementations of the provided methods, the second encoder neural network also includes a softmax layer after the first encoder network.
[0023] In some implementations of the provided methods, the audio processing neural network is a non-streaming model configured to predict an audio processing output for a given time point based on an audio data input that includes data frames corresponding to time points before and after the given time point.
[0024] In some implementations of the provided methods, the audio processing neural network is a streaming model configured to predict an audio processing output for a given time point based on an audio data input that includes data frames corresponding to time points before or at the given time point.
[0025] In some implementations of the provided methods, the decoder network includes one or more recurrent neural networks (RNNs). For example, the decoder network may include one or more LSTM networks. In some implementations, the decoder network may include an RNN transducer.
[0026] In some implementations of the provided methods, the audio processing neural network further includes one or more additional layers including a projection layer between the first encoder network and the decoder network.
[0027] The present specification also provides a system, which includes one or more computers and one or more storage devices storing instructions, and when the instructions are executed by the one or more computers, the one or more computers perform the above-mentioned training method.
[0028] The present specification also provides one or more computer storage media storing instructions, which, when executed by one or more computers, cause the one or more computers to perform the above training method.
[0029] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following advantages.
[0030] This specification describes techniques for performing self-supervised learning that combines pre-training of an audio processing neural network using unsupervised data with fine-tuning using labeled training examples. Self-supervised learning provides a significant improvement in the quality of audio processing models compared to fully supervised learning techniques because this approach enables the model to learn from large unsupervised datasets in addition to supervised learning. The ability to learn from unsupervised data is particularly beneficial when labeled data is limited and opens up new opportunities for low-resource languages and domains.
[0031] In addition, the techniques described in this specification provide several advantages over conventional self-supervised learning techniques. Conventional self-supervised learning techniques typically involve learning feature representations before mapping feature representations of input audio data to discrete word units (e.g., discrete text word units). This can lead to challenges that may hinder the applicability and effectiveness of audio processing models. For example, the integration of representation learning and self-supervised learning typically requires the model to serve the purpose of providing audio representations that are still valid for downstream tasks. However, an effective representation learning model may not always be effective for downstream tasks. For example, a good representation learning model may require access to the future context of an utterance, while a downstream task may require a low-latency model that prohibits access to future contexts. In addition, the goals of representation learning and self-supervised learning are not always consistent, and the combination of the two aspects typically results in complex and computationally demanding algorithms.
[0032] The techniques described in this specification do not require learning feature representations for unlabeled audio data during pre-training, and therefore can overcome certain challenges associated with conventional self-supervised learning techniques. In some implementations of the described techniques, instead of performing feature representation learning, the system trains an encoder network of an audio processing model to predict masked segments of an audio data input based on unmasked audio data that has been processed with a random projection quantizer. The random projection quantizer does not require representation learning and is independent of the audio processing model. Therefore, the described techniques can be used in a wide range of audio processing network architectures and can achieve state-of-the-art prediction accuracy and speed for both streaming and non-streaming tasks while maintaining computational efficiency.
[0033] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 An example training system for training an audio processing neural network is shown.
[0035] Figure 2 An example process for generating self-supervised training examples for training an encoder network is shown.
[0036] Figure 3 is a flow chart illustrating an example process for training an audio processing neural network.
[0037] Figure 4 Shown is a performance comparison between an audio processing neural network trained using random projection quantization pre-training and an audio processing neural network trained using a baseline method.
[0038] Figure 5Another performance comparison between an audio processing neural network trained using random projection quantization pre-training and an audio processing neural network trained using a baseline method is shown.
[0039] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION
[0040] Figure 1 An example of a training system 100 is shown for training an audio processing neural network 160. System 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.
[0041] In general, after the training process, the system 100 or another system can use the trained audio processing neural network 160 to process audio data to perform audio processing tasks, such as speech recognition tasks, language recognition tasks, voice recognition tasks, music transcription tasks, audio classification tasks, etc.
[0042] In the specific example of a speech recognition task, a trained neural network can be used to process audio data (e.g., data received from an audio sensing device such as a microphone) to generate speech recognition output (e.g., output indicative of spoken content in the audio data, such as a text transcription of the spoken content).
[0043] As another example of a hot word detection task, if the input to a neural network is a sequence of audio data representing a spoken utterance, the output generated by the neural network may indicate whether a particular word or phrase (a "hot word") was spoken in the utterance.
[0044] As an example of a language identification task, if the input to a neural network is a sequence of audio data representing a spoken utterance, the output generated by the neural network may identify the natural language in which the utterance was spoken.
[0045] As an example of a speech recognition task, the input to a neural network may be a sequence of audio data representing spoken utterances from a particular person, and the output generated by the neural network may identify the identity of the particular person.
[0046] As an example of a music transcription task, the input to the neural network may be a sequence of audio data representing a musical melody or a musical performance, and the output generated by the neural network may identify a transcription of the musical melody or performance to automatically generate a musical score.
[0047] As an example of an audio classification task, the input to the neural network may be a sequence of audio data representing sounds recorded by an audio sensor, and the output generated by the neural network may identify a classification of the sound that indicates a nature or attribute of the sound, such as the identity of the subject making the sound (e.g., a particular type of device, type of animal), the condition of the subject making the sound (e.g., under normal or abnormal conditions), or identifying the type or condition of the sensor that recorded the sound.
[0048] In general, the input audio data includes multiple data frames corresponding to multiple time points. In some cases, the audio processing neural network 160 can be a non-streaming model. That is, the audio processing neural network 160 is configured to predict for a given time point t based on both past data frames and future data frames of the input audio data (i.e., data frames with corresponding time points both before and after the given time point). In some other cases, the audio processing neural network 160 can be a streaming model, in which the network is configured to predict the audio processing output for a given time point based on past data frames of the input audio data (i.e., data frames with corresponding time points up to the given time point) rather than based on future data frames.
[0049] The audio processing neural network 160 generally includes an encoder network 162 configured to map audio data input to encoded data that encodes context information in the audio data input, and a decoder network 164 configured to map input derived from the encoded data to audio processing outputs.
[0050] The encoder network 162 and the decoder network 164 can have any suitable architecture. A specific example of an encoder architecture is described in Zhang et al., “Pushing the limits of semisupervised learning for automatic speech recognition”, arXiv preprint, arXiv:2010.10504, 2020. Another example of an encoder architecture is described in Yu et al., “FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization”, Proc. ICASSP, 2021. As a specific example, the decoder architecture can include an RNN Transducer (RNN-T) with an additional layer of unidirectional LSTM.
[0051] The system 100 trains the audio processing neural network 160 using a self-supervised learning process.
[0052] The system 100 includes a pre-training engine 120 that performs a pre-training process and a fine-tuning engine 150 that performs a fine-tuning training process.
[0053] During pre-training, the pre-training engine 120 uses the unlabeled audio data 110 to update the network parameters of the encoder network 122 to a pre-trained set of values. In this specification, unlabeled audio data refers to audio data for which training labels are not available or even if some of the training labels are available, they are not used during training.
[0054] During fine-tuning, the fine-tuning engine 150 uses the labeled training data 140 to train the audio processing neural network 160 end-to-end to update the network parameters of the encoder network 122 (where the parameters are initialized to pre-trained values) and the network parameters of the decoder network 124. The labeled training data 140 generally includes a set of labeled training examples. Each labeled training example includes an audio data training input and a corresponding audio processing tag. The audio processing tag is the target output of the audio processing neural network 160 for the audio processing task when the corresponding audio data training input is processed by the audio processing neural network 160.
[0055] Generally speaking, the pre-training process trains the encoder network 162 to learn context information in the unlabeled audio data 110 .
[0056] To pre-train encoder network 162 using unlabeled audio data 110 , pre-training engine 120 of system 100 defines one or more self-supervised training tasks from unlabeled audio data 110 , and trains encoder network 162 on the one or more self-supervised training tasks.
[0057] For example, a self-supervised training task can be one that requires an encoder network to process an unlabeled audio data segment, where certain portions of the data segment have been masked (i.e., the data in the masked portions have been replaced by other data, such as data sampled from a noise signal), to predict encoded data for the masked portions of the data segment.
[0058] In particular, the system can generate a set of encoder training examples 130 for a self-supervised training task from a set of unlabeled audio data segments. Each encoder training example includes a corresponding encoder training input and a corresponding encoder target label.
[0059] The pre-training engine 120 may generate a corresponding encoder training input for each encoder training example 130 by processing the corresponding unlabeled audio data segment. In some implementations, the audio segment masking engine 122 may generate the encoder training input by masking one or more segments of the corresponding unlabeled audio data segment. That is, the audio segment masking engine 122 may replace data in one or more segments of consecutive frames of the audio data segment with other data that is independent of the content of the audio data segment, such as data values sampled from a noise signal or data values set to a predetermined fixed value (e.g., zero).
[0060] The pre-training engine 120 can generate encoder target labels in each encoder training example 130 by performing projection on the corresponding unlabeled audio data segment and then performing quantization. To perform the projection, the random projection engine 124 can apply a projection matrix to each audio data frame in the set of audio data frames in the corresponding unlabeled audio data segment to generate projected data including, for example, a set of projected vectors. For example, to generate a projected vector of an audio data frame including a filter bank coefficient vector, the random projection engine can multiply the projection matrix by the filter bank coefficient vector.
[0061] In some cases, the projected data is generated from a set of audio data frames that have been selected to be masked by the audio segment masking engine 122 during the generation of the corresponding encoder training input. In some other cases, the projected data may be generated from all data frames in the corresponding unlabeled audio data segment. The elements of the projection matrix may be values randomly selected according to a predefined distribution, such as a uniform distribution.
[0062] In order to perform quantization, the quantization engine 126 can identify the template vector with the shortest distance from the given projected vector from the template vector (or codebook vector) set, and assign the element value of the encoder target mark as the index of the identified codebook vector in the codebook vector set. The codebook vector set can also be randomly generated according to a predefined distribution. In some implementations, the element values of the projection matrix and / or the codebook vector remain fixed during the pre-training process. In this case, the self-supervised training task of the encoder can be a task of predicting the index of the codebook vector identified for the selected audio frame from the corresponding masked audio data segment.
[0063] In general, random projections perform dimensionality reduction on the audio signal, while random codebooks provide an approximate discrete representation of the distribution of the audio data.
[0064] In some implementations, during pre-training, the pre-training engine 120 may include a pre-trained neural network head 162b (e.g., including one or more network layers) 162b as an output layer of the encoder network 162 to output predicted tags. That is, the pre-training engine 120 trains a neural network 162a including an encoder network 162 and a pre-trained neural network head 162b. When the encoder network 162 and the decoder network 164 are trained end-to-end, the pre-trained neural network head 162b will not be included during the fine-tuning process. The pre-training engine 120 can use the pre-trained neural network head 162b to adapt to different architectures and / or different types of tasks of the encoder network 162, so that the same training framework can be used for various scenarios. In particular, the pre-trained neural network head 162b can be used to select the encoded embedding output by the previous layer to generate the predicted tags. In some implementations, the system can reuse the same pre-trained encoder network 162 for multiple different downstream tasks, or can train a multi-task decoder that performs multiple different tasks in parallel.
[0065] In a particular example, the pre-trained neural network head 162b can be a softmax layer that maps the selected embeddings to corresponding logits (e.g., through one or more learned linear transformations) and then maps the logits to a probability distribution over labels (e.g., over codebook vector indices).
[0066] For example, to perform training for a non-streaming task, the system 100 may use a softmax layer 162b to select encoded embeddings corresponding to time points both before and after the masked frame to generate predicted labels for the masked frame. Alternatively, to perform training for a streaming task, the system 100 may use a softmax layer 162b to select only those encoded embeddings corresponding to time points of the masked frame and, optionally, time points before the masked frame to generate predicted labels for the masked frame. In some cases, for each masked set of one or more frames, the softmax layer 162b processes the encoded embeddings of the time points corresponding to the masked set to generate predictions for the masked set of frames.
[0067] Once the encoder training examples 130 have been generated, the pre-training engine 120 may update the network parameters of the encoder network 162 to the pre-trained values by minimizing the self-supervised loss calculated based on the generated encoder training examples 130. The pre-training engine 120 may update the parameters by any suitable back-propagation based machine learning technique (e.g., using an Adam or AdaGrad optimizer).
[0068] Similarly, during fine-tuning, the fine-tuning training engine 150 updates the network parameters of the encoder network 162 and the decoder network 164 by minimizing a training loss, which includes a supervised training loss calculated based on at least the labeled training examples 140 .
[0069] In some implementations, the audio processing neural network 160 also includes one or more additional layers, including a projection layer between the encoder network 162 and the decoder network 164. The additional layers can be used to help the audio processing neural network 160 adapt to downstream audio processing tasks. For example, the projection layer is configured to multiply the output of the previous layer with a matrix and allow the output of the encoder to be projected to a dimension that matches the input dimension of the decoder.
[0070] Although the above description describes a configuration in which self-supervised pre-training (using unlabeled data 110) is performed before supervised fine-tuning (using labeled data 140), in some cases, pre-training and fine-tuning can be performed jointly. For example, a self-supervised pre-training loss can be used as an auxiliary loss during the fine-tuning process. That is, the system 100 can train a network on a batch including labeled data and unlabeled data, or can train a network alternately on labeled data and unlabeled data. For unlabeled data, the system 100 can use the self-supervised pre-training loss to update the encoder 162. For labeled data, the system 100 can (i) use supervised losses to update the encoder network 162 and the decoder network 164, or (ii) use both supervised losses and self-supervised losses to update the encoder and decoder networks.
[0071] After the audio processing neural network 160 has been trained, the system 100 or another system can use the trained audio processing neural network 160 to perform an audio processing task. For example, the system 100 can receive an audio input from an input device and process the audio input using at least the trained audio processing neural network 160 to generate an audio processing output 180 corresponding to the received audio input. The system 100 can also output the audio processing output 180 through an output device. For example, the audio processing task can be a speech recognition task, and the audio processing neural network 160 is configured to process the audio data input to generate an output that represents the language content represented by the audio data input. In this case, the output 180 can include a predicted text transcription of the audio data input and can be presented to a user using an output device (such as a computer display).
[0072] Figure 2An example process 200 for generating self-supervised training examples for training an encoder network is shown. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a computer appropriately programmed according to the present specification Figure 1 The training system 100 can perform process 200.
[0073] like Figure 2 As shown, the system receives an unlabeled audio segment 210. The unlabeled audio segment 210 includes a set of audio data frames corresponding to a plurality of time points (e.g., points along the horizontal axis of 210). Each audio data frame represents an audio signal at a corresponding time point. In one example, the audio data frame specifies a plurality of audio signal filter bank coefficients.
[0074] The system generates encoder training input 232 by masking unlabeled audio segments 210. In particular, the system can select one or more segments of consecutive audio data frames in the unlabeled audio data segments 210 and replace the data in the selected audio data frames with noise data values. The noise data values can be sampled from a predefined distribution, such as a normal distribution with a mean of 0 and a standard deviation of 0.1. In some implementations, the masking strategy can sample at each frame whether to apply a mask to a segment starting with the frame with a fixed probability. Each mask segment spans a fixed length from the starting frame.
[0075] The system applies projection matrix 224 to the audio data frames in unlabeled audio data segment 210 to generate a set of projected vectors 225. The elements of projection matrix 224 are randomly selected values according to a predefined distribution (such as a uniform distribution or any other appropriate distribution). Any appropriate method can be used to generate projection matrix 224. For example, the Xavier initialization process can be used to generate projection matrix 224. Details of the Xavier initialization process can be found in Glorot et al. "Understanding the difficulty of training deep feedforward neural networks", in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 9, pp. 249-256. PMLR, May 13-15, 2010. In some implementations, the elements of projection matrix 224 are fixed during the generation of encoder training examples.
[0076] In some implementations, the system can perform normalization of each audio data frame in the unlabeled audio data segment 210 to a predefined mean and a predefined standard deviation (e.g., 0 mean and standard deviation 1) before applying the projection matrix 224. Normalization can prevent random projections from collapsing into a small subset of vectors.
[0077] After the projection vector 225 has been generated, the system can use a codebook lookup process based on a codebook containing a set of indexed codebook vectors 226 to map the projected vector 225 to a set of encoder target tags 234. In particular, for each given projected vector, the system identifies the codebook vector from the codebook that has the shortest distance to the given projected vector and determines the element value of the encoder target tag as the index of the identified codebook vector in the codebook. The codebook vector can be a vector randomly generated according to an appropriate distribution (e.g., a uniform distribution). In some implementations, the codebook vector remains unchanged during the generation of encoder training examples.
[0078] Figure 33 is a flow chart illustrating an example process 300 for training an audio processing neural network. The audio processing neural network is configured to perform an audio processing task and includes at least (1) a first encoder network having a first encoder network parameter set and (2) a decoder network having a decoder network parameter set.
[0079] For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system (e.g., reference 100) appropriately programmed according to the present specification may be used. Figure 1 The training system 100 described above can perform process 300.
[0080] At 310, the system obtains a set of unlabeled audio data segments. Each unlabeled audio data segment includes a corresponding plurality of audio data frames corresponding to a plurality of time points.
[0081] At 320, the system generates a set of encoder training examples from the set of unlabeled audio data segments. Each encoder training example includes a corresponding encoder training input and a corresponding encoder target label.
[0082] To generate each encoder training example, the system generates a corresponding encoder training input by processing a corresponding unlabeled audio data segment in a set of unlabeled audio data segments, generates a corresponding projected vector set by applying a projection matrix to each audio data frame in a set of audio data frames in the corresponding unlabeled audio data segment, and processes the corresponding projected vector set to generate a corresponding encoder target label for the encoder training example. The elements of the projection matrix are values randomly selected according to a first distribution. The first distribution can be a uniform distribution. In some implementations, the values of the elements of the projection matrix remain unchanged during the generation of the encoder training example set.
[0083] At 330, the system performs training of a second encoder neural network including at least the first encoder neural network on the generated set of encoder training examples to update at least a first set of encoder network parameters of the first encoder neural network to a pre-trained set of values.
[0084] At 340, the system obtains one or more labeled training examples. Each labeled training example includes an audio data training input and a corresponding audio processing label. The audio processing label is the target output of the audio processing neural network 160 for the audio processing task when the corresponding audio data training input is processed by the audio processing neural network.
[0085] At 350, the system also performs training of the audio processing neural network on the labeled training examples to update at least (1) a first set of network parameters of the encoder network and (2) a second set of network parameters of the decoder network, wherein the first set of network parameters is initialized with a pretrained set of values.
[0086] Figure 4 An audio processing neural network (as shown in reference) trained using random projection quantization pre-training ("BEST-RQ") is shown. Figure 1 , Figure 2 and Figure 3 A performance comparison 400 between an audio processing neural network trained using a baseline state-of-the-art approach, including “wav2vec2.0” as described in Baevski et al., “wav2vec 2.0: A framework for self-supervised learning of speech representations,” arXiv preprint, arXiv:2006.11477, 2020b, and “w2v-BERT” as described in Chung et al., “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” arXiv preprint, arXiv:2108.06209, 2021.
[0087] The audio processing neural network is trained to perform speech recognition tasks. The word error rate (WER) and relative latency are calculated by using the trained model on the LibriLight dataset. Figure 4 As shown, for both streaming and non-streaming pre-training, the training method described in this specification generally outperforms the baseline method in terms of both WER and latency.
[0088] Figure 5 A comparison 500 of the performance (specifically, WER) of audio processing neural networks for speech recognition tasks in multiple languages is shown. Figure 5 As shown, for most of the tested languages, the training method described in this specification ("BEST-RQ") generally outperforms the baseline method.
[0089] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that enables the system to perform these operations or actions in operation. For one or more computer programs configured to perform a specific operation or action, it means that the one or more programs include instructions that enable the device to perform these operations or actions when executed by a data processing device. The embodiments of the subject matter and functional operations described in this specification can be implemented in a digital electronic circuit system, in a tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible non-temporary storage medium for data processing equipment to execute or control the operation of the data processing equipment. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to appropriate receiver devices for execution by the data processing apparatus.
[0090] The term "data processing equipment" refers to data processing hardware and includes all kinds of equipment, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The equipment may also be or also include special-purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0091] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store portions of one or more modules, subroutines, or code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.
[0092] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and may be stored on a storage device in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed differently.
[0093] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.
[0094] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmed computers.
[0095] A computer suitable for executing a computer program can be based on a general or special microprocessor or both, or any other kind of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for fulfilling or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit system or incorporated into the special logic circuit system. Generally, the computer will also include one or more large-capacity storage devices for storing data, such as a magnetic disk, a magneto-optical disk or an optical disk, or be operably coupled to receive data from one or more large-capacity storage devices or transfer data to one or more large-capacity storage devices or both. However, the computer does not have to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.
[0096] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks.
[0097] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device (e.g., a smart phone running a messaging application), and receiving a response message from the user in return.
[0098] A data processing device used to implement a machine learning model may also include, for example, dedicated hardware accelerator units for processing general-purpose and computationally intensive parts of machine learning training or production (i.e., inference, workloads).
[0099] Machine learning models can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework or the Jax framework).
[0100] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.
[0101] A computing system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from it. Data generated at the user device, for example, the result of a user interaction, can be received at the server from the device.
[0102] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that may be peculiar to a particular embodiment of a particular invention. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may involve a sub-combination or a variant of a sub-combination.
[0103] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0104] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.
Claims
1. A computer-implemented method for training an audio processing neural network, the audio processing neural network being configured to perform an audio processing task, wherein the audio processing neural network comprises at least (1) a first encoder network having a first encoder network parameter set and (2) a decoder network having a decoder network parameter set, the method include: Obtaining a set of unlabeled audio data segments, each unlabeled audio data segment comprising a corresponding plurality of audio data frames corresponding to a plurality of time points; Generating a set of encoder training examples from the set of unlabeled audio data segments, wherein each encoder training example comprises a corresponding encoder training input and a corresponding encoder target label, the generating comprising for each encoder training example: generating the corresponding encoder training input by processing a corresponding unlabeled audio data segment from the set of unlabeled audio data segments, generating a corresponding set of projected vectors by applying a projection matrix to each of the set of audio data frames in the corresponding unlabeled audio data segment, wherein elements of the projection matrix are values randomly selected according to a first distribution, and processing the corresponding set of projected vectors to generate corresponding encoder target labels for the encoder training examples; performing training of a second encoder neural network including at least the first encoder neural network on the generated set of encoder training examples to update at least the first encoder network parameter set of the first encoder neural network to a pre-trained set of values; Obtain one or more labeled training examples, each labeled training example comprising an audio data training input and a corresponding audio processing label; as well as Training of the audio processing neural network is performed on the labeled training examples to update at least (1) the first set of network parameters of the encoder network and (2) the second set of network parameters of the decoder network, wherein the first set of network parameters is initialized with the pretrained set of values.
2. The method according to claim 1, further comprising: include: receiving audio input from an input device; as well as After the audio processing neural network has been trained, the audio input is processed using at least the audio processing neural network to generate an audio processing output corresponding to the received audio input.
3. The method according to claim 2, further comprising: include: The audio processed output is output via an output device.
4. A method as claimed in any one of the preceding claims, wherein the audio processing task is a speech recognition task and the audio processing neural network is configured to process an audio data input to generate an output representing a language content represented by the audio data input.
5. The method of claim 4, wherein the output comprises a predicted text transcription of the audio data input.
6. A method as claimed in any preceding claim, wherein each frame of audio data comprises a plurality of audio signal filter bank coefficients.
7. The method according to any one of the preceding claims, further comprising: include: Before generating the encoder training example set from the set of unlabeled audio data segments, normalization of each audio data frame in the set of unlabeled audio data segments to a predefined mean and a predefined standard deviation is performed.
8. The method of any one of the preceding claims, wherein generating the corresponding encoder training input comprises generating corresponding masked audio data segments from the corresponding unlabeled audio data segments, include: selecting one or more audio data frames in the corresponding unlabeled audio data segment; as well as Data in the one or more selected frames of audio data are replaced with noise data values.
9. The method of claim 8, wherein training of the second encoder neural network is performed include: The second encoder neural network is trained on an encoder training task to predict labels of the one or more selected audio data frames in the corresponding unlabeled audio data segment from one or more input audio data frames in the corresponding masked audio data segment.
10. The method of claim 9, wherein the set of audio data frames in the corresponding unlabeled audio data segment includes the one or more selected audio data frames.
11. A method as claimed in any one of claims 8 to 10, wherein the noise data values are sampled from a second distribution. The method of claim 11 , wherein the second distribution is a normal distribution with a mean of 0.
13. The method of any one of claims 8 to 12, wherein the one or more audio data frames in the corresponding unlabeled audio data segment are selected include: For each given audio data frame in the corresponding unlabeled audio data segment, determine whether the given audio data frame is selected with a predefined probability, and in response to the given audio data frame having been determined to be selected, select the given audio data frame and a predefined number of consecutive audio data frames after the given audio data frame in the corresponding unlabeled audio data segment.
14. A method as claimed in any preceding claim, wherein the values of the elements of the projection matrix remain unchanged during generation of the encoder training example set.
15. The method of any one of the preceding claims, wherein the first distribution is a uniform distribution.
16. The method of any one of the preceding claims, wherein processing the corresponding set of projected vectors to generate the corresponding encoder target signature comprises, for each given projected vector in the corresponding set of projected vectors: identifying, from a set of codebook vectors, a codebook vector having a shortest distance to the given projected vector; and The value of an element of the encoder target flag is determined as an index of the identified codebook vector in the set of codebook vectors.
17. A method as claimed in claim 16 and also dependent on claim 10, wherein the training task is the task of predicting the index of the codebook vector identified for the one or more selected audio frames from the corresponding masked audio data segment.
18. The method of claim 16 or 17, wherein the set of codebook vectors is randomly generated according to a third distribution. The method of claim 18 , wherein the third distribution is a uniform distribution.
20. The method of claim 18 or 19, wherein the set of codebook vectors remains unchanged during generation of the encoder training sample set.
21. The method of any of the preceding claims, wherein the second encoder neural network further comprises a softmax layer following the first encoder network.
22. A method as described in any of the preceding claims, wherein the audio processing neural network is a non-streaming model configured to predict an audio processing output at a given time point based on an audio data input, wherein the audio data input includes data frames corresponding to time points before and after the given time point.
23. The method of any one of claims 1 to 21, wherein the audio processing neural network is a streaming model configured to predict an audio processing output at a given point in time based on an audio data input, the audio data input comprising data frames corresponding to time points before or at the given point in time.
24. A method as described in any of the preceding claims, wherein the decoder network comprises one or more recurrent neural networks (RNNs).
25. The method of claim 24, wherein the one or more RNNs comprise one or more LSTM networks.
26. The method of claim 25, wherein the decoder network comprises an RNN transformer.
27. A method as described in any of the preceding claims, wherein the audio processing neural network further includes one or more additional layers, the one or more additional layers comprising a projection layer between the first encoder network and the decoder network.
28. A computer-implemented method, include: Receive new audio data input; as well as The new audio data input is processed using an audio processing neural network that has been trained using a method as claimed in any preceding claim to generate an audio processing output of an audio processing task.
29. A system comprising one or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 28.
30. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 28.
