Unsupervised data selection via discrete speech representations for automatic speech recognition

Abstract

A method (500) includes obtaining an untagged training data corpus (358) including a plurality of spoken utterances (360), each corresponding spoken utterance of the plurality of spoken utterances including audio data (362) characterizing the corresponding spoken utterance. The method also includes receiving a target domain (324). The method further includes using a contrast data selection model (310) to select a subset of utterances corresponding to the target domain from the untagged corpus of training data. The method includes training an automatic speech recognition (ASR) model on a subset of utterances (200).

Description

Technical Field

[0001] The present disclosure relates to unsupervised data selection via discrete speech representation for automatic speech recognition. Background Art

[0002] Automatic speech recognition (ASR) is a computer process that transcribes audio input of spoken language (i.e., speech) into text. Generally speaking, ASR models are trained on a large pool of audio data to learn how to accurately transcribe speech. Based on the continuous development of deep neural networks, modern ASR models continue to improve in terms of accuracy (e.g., low word error rate (WER)) and latency (e.g., the delay between the user speaking and the transcription). However, one challenge in developing ASR models based on deep learning is that the parameters of the ASR model tend to overfit the training data, which makes it difficult for the ASR model to generalize unseen data when the training data is not extensive enough. Therefore, training the ASR model on a larger training data set improves the accuracy of the ASR model. Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations, the operations comprising: obtaining an unlabeled training data corpus comprising a first set of spoken utterances, and receiving a target domain. Here, each corresponding spoken utterance in the first set of spoken utterances comprises audio data representing the corresponding spoken utterance. The operations also include: using a contrastive data selection model to select a subset of spoken utterances corresponding to the target domain from the unlabeled training data corpus, and training an automatic speech recognition (ASR) model on the subset of spoken utterances corresponding to the target domain selected from the unlabeled training data corpus.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, training the ASR model includes training the ASR model on a subset of spoken utterances using a self-supervised training technique. In some examples, the selected subset of spoken utterances includes a smaller number of spoken utterances than the number of spoken utterances in the unlabeled training data corpus. In some implementations, the operation also includes refining the ASR model by training the ASR model on labeled training data, the labeled training data including a second set of spoken utterances and a corresponding true value transcription paired with each spoken utterance in the second set of spoken utterances.

[0005] In some implementations, selecting a subset of spoken utterances includes, for each corresponding spoken utterance in an unlabeled training data corpus: quantizing audio data representing the spoken utterance to produce a corresponding discrete word-gram sequence; generating a domain relevance score based on the corresponding discrete word-gram sequence; and determining to select the corresponding spoken utterance or omit selection of the corresponding spoken utterance based on the domain relevance score. Here, generating the domain relevance score may include: processing the corresponding discrete word-gram sequence of the corresponding spoken utterance using a first language model to generate a first output; processing the corresponding discrete word-gram sequence of the utterance using a second language model to generate a second output; and comparing the first output with the second output to determine the domain relevance score. The first language model is trained on target training data corresponding to a target domain, and the second language model is trained on general training data corresponding to multiple domains. Optionally, the domain relevance score may be based on a difference between the first output and the second output divided by the number of words in the corresponding discrete word-gram sequence of the corresponding spoken utterance. In some examples, the first language model and the second language model include N-gram language models. In addition, quantizing the spoken utterance may include mapping one or more continuous features of the audio data representing the spoken utterance into a discrete word-gram sequence. The comparison data selection model may include a self-supervised learning model.

[0006] Another aspect of the present disclosure includes data processing hardware and memory hardware that communicates with the data processing hardware and stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations comprising: obtaining an unlabeled training data corpus comprising a first set of spoken utterances, and receiving a target domain. Here, each corresponding spoken utterance in the first set of spoken utterances comprises audio data characterizing the corresponding spoken utterance. The operations further comprise: using a contrastive data selection model to select a subset of spoken utterances corresponding to the target domain from the unlabeled training data corpus, and training an automatic speech recognition (ASR) model on the subset of spoken utterances corresponding to the target domain selected from the unlabeled training data corpus.

[0007] This aspect of the disclosure may include one or more of the following optional features. In some implementations, training the ASR model includes training the ASR model on a subset of spoken utterances using a self-supervised training technique. In some examples, the selected subset of spoken utterances includes a smaller number of spoken utterances than the number of spoken utterances in the unlabeled training data corpus. In some implementations, the operation also includes refining the ASR model by training the ASR model on labeled training data, the labeled training data including a second set of spoken utterances and a corresponding true value transcription paired with each spoken utterance in the second set of spoken utterances.

[0008] In some implementations, selecting a subset of spoken utterances includes, for each corresponding spoken utterance in an unlabeled training data corpus: quantizing audio data representing the spoken utterance to produce a corresponding discrete word-gram sequence; generating a domain relevance score based on the corresponding discrete word-gram sequence; and determining to select the corresponding spoken utterance or omit selection of the corresponding spoken utterance based on the domain relevance score. Here, generating the domain relevance score may include: processing the corresponding discrete word-gram sequence of the corresponding spoken utterance using a first language model to generate a first output; processing the corresponding discrete word-gram sequence of the utterance using a second language model to generate a second output; and comparing the first output with the second output to determine the domain relevance score. The first language model is trained on target training data corresponding to a target domain, and the second language model is trained on general training data corresponding to multiple domains. Optionally, the domain relevance score may be based on a difference between the first output and the second output divided by the number of words in the corresponding discrete word-gram sequence of the corresponding spoken utterance. In some examples, the first language model and the second language model include N-gram language models. In addition, quantizing the spoken utterance may include mapping one or more continuous features of the audio data representing the spoken utterance into a discrete word-gram sequence. The comparison data selection model may include a self-supervised learning model.

[0009] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of an example automatic speech recognition training and implementation system.

[0011] Figure 2 It is a schematic diagram of the Recurrent Neural Network-Transducer (RNN-T) model architecture.

[0012] Figure 3A and Figure 3B is a schematic diagram of an example process for selecting data from an unlabeled training data set for pre-training an automatic speech recognition model.

[0013] Figure 4 is a schematic diagram of an example process for training an automatic speech recognition model using selected unlabeled training data and labeled training data.

[0014] Figure 5 is a flow diagram of an example arrangement of operations of a method of pre-training an automatic speech recognition model using selected training data.

[0015] Figure 6is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0016] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0017] Automatic speech recognition (ASR) technology has achieved tremendous improvements in accuracy and latency with the implementation of machine learning methods for training. ASR models are typically trained on a large training data set that includes audio data of spoken language (i.e., utterances) paired with true value transcriptions (i.e., labels) of the audio data. During training, the ASR model is fed with utterances to produce output transcriptions. The output transcriptions are compared to the corresponding true value transcriptions to determine a supervision loss, where one or more parameters of the ASR model are trained based on the supervision loss, for example using stochastic gradient descent. This process continues for the entire training data set to further refine the model.

[0018] The process of training a model with labeled training data is called "supervised" learning. One disadvantage of supervised learning is that it may be difficult to obtain enough labeled training data to train an ASR model for low-resource domains and / or low-resource languages. For example, a particular language may be underrepresented in a labeled training data set, and any ASR model trained using that labeled training data set will produce less accurate results for users who speak in that particular language. One way to enhance training data for underrepresented domains is to pre-train the ASR model using untranscribed audio data (i.e., unlabeled training data) that is more easily available in large quantities, called "self-supervised" training. Self-supervised pre-training of speech representations has achieved impressive results in improving ASR word error rates (WER).

[0019] The implementation of this article further improves the self-supervised training of ASR models by providing a simple and flexible framework for selecting unlabeled data for pre-training. More specifically, the implementation relates to a contrast data selection model that processes unlabeled training data into discrete speech representations and applies contrast data selection to generate an unlabeled training data subset for pre-training. Since the unlabeled training data set may be quite large, selecting a smaller data subset for pre-training allows the benefits of using more easily available unlabeled data in large quantities without having to train on the entire unlabeled training data set. Furthermore, the ASR model is pre-trained on a smaller and more concentrated unlabeled training data set, resulting in higher efficiency than known ASR training methods (i.e., greatly reducing computations in pre-training). In addition, the selected data is used to produce an ASR that performs better for underrepresented data in the labeled training data set.

[0020] Figure 1 A system 100 is shown that includes an ASR system 105 that implements an ASR model 200 resident on a user device 102 of a user 104 and / or on a remote system 150 (e.g., one or more servers of a distributed system executing in a cloud computing environment) that communicates with the user device 102. Although the user device 102 is depicted as a mobile computing device (e.g., a smart phone), the user device 102 may correspond to any type of computing device, such as, but not limited to, a tablet device, a laptop / desktop computer, a wearable device, a digital assistant device, a smart speaker / display, a smart appliance, an automotive infotainment system, or an Internet of Things (IoT) device, and is equipped with data processing hardware 111 and memory hardware 113.

[0021] In some implementations, the user device 102 communicates with a remote system 150 (also referred to herein as a cloud computing environment) via a network 140. The remote system 150 may include scalable / elastic resources 151, which include computing resources 154 (e.g., data processing hardware) and / or storage resources 156 (e.g., memory hardware). A data repository 158 (i.e., a remote storage device) may be overlaid on the storage resources 146 to allow one or more user devices 102 or computing resources 154 to scalably use the storage resources 146. The ASR system 105 may be executed locally on the user device 102 (e.g., on the data processing hardware 111) or remotely (e.g., at the remote system 150).

[0022] The user device 102 includes an audio subsystem 108 configured to receive streaming audio 107 of an utterance 106 spoken by a user 104 (e.g., the user device 102 may include one or more microphones for recording the spoken utterance 106) and convert the utterance 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 105. In the example shown, the user speaks a corresponding utterance 106 of the phrase "What is the weather in New York City?" in natural language English, and the audio subsystem 108 converts the streaming audio 107 of the utterance 106 into a corresponding acoustic frame 110 for input to the ASR system 105. Thereafter, the ASR model 200 receives the acoustic frame 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (e.g., a recognition result / hypothesis) of the utterance 106 as output. In the example shown, the user device 102 and / or the remote system 150 also executes a user interface generator 107 that is configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 102. In some configurations, the transcription 120 output from the ASR system 105 is processed, for example, by a natural language understanding (NLU) module executed on the user device 102 or the remote system 150 to execute the user command. Additionally or alternatively, a text-to-speech system (e.g., executed on any combination of the user device 102 or the remote system 150) can convert the transcription into synthesized speech for audible output by another device. For example, the original utterance 106 can correspond to a message that the user 104 is sending to a friend, where the transcription 120 is converted into synthesized speech to audibly output to the friend to listen to the message conveyed in the original utterance 106.

[0023] As described in more detail below, the ASR model 200 may be trained using a two-part training process 400 ( Figure 4) is trained, and the two-part training process includes a pre-training phase using unlabeled training data selected via a training data selection process 300. The training data selection process 300 utilizes a contrastive data selection model 310, a quantizer 320, and a word-based data selector 330 to select unlabeled training data for pre-training the ASR model 200. The training data selection process 300 uses components 310, 320, and 330 to process an unlabeled training data corpus 358 and select an unlabeled training data subset 359 (i.e., a utterance subset 359) that is relevant to the target domain. For example, the unlabeled training data corpus 358 can be stored in a data repository 158. Although the quantizer 320 and the word-based data selector 330 are shown as being deployed in the contrastive data selection model 310, the components 310, 320, and 330 can be arranged in any suitable configuration to select an unlabeled training data subset 359 that is relevant to the target domain from a large utterance set 358 corresponding to multiple domains. Referring below Figure 4 Described in more detail, the training process 400 pre-trains the ASR model 200 using an unlabeled training data subset 359 selected from the unlabeled training data corpus 358. In some implementations, the comparative data selection model 310 implements a self-supervised (or unsupervised) machine learning technique to process the unlabeled training data corpus 358 and select the unlabeled training data subset 359.

[0024] Figure 1 The system 100 is presented for illustration purposes only and is not intended to be limiting. For example, although only a single example of each component is shown, any number of components 102, 150, 151, and 158 may be communicatively coupled to the system 100. In addition, although some components are shown as being located in the cloud computing environment 150, in some implementations, those components may be hosted locally on the user device 102. Alternatively, although some components are shown as being hosted on the user device 102, in some implementations, those components may be hosted in the cloud computing environment 150. In addition, in various implementations, some or all of the components 105, 108, and 107 are hosted locally on the user device 102, remotely (such as in the cloud computing environment 150), or some combination thereof.

[0025] refer to Figure 2, an example frame-aligned based transducer model 200 includes a recurrent neural network-transducer (RNN-T) model architecture that complies with latency constraints associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the frame-aligned based transducer model 200 may include other architectures, such as transformer-transducer and conductor-transducer model architectures, etc. The RNN-T model 200 provides a small computational footprint and uses less memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device 102 (e.g., without the need to communicate with a remote server). The RNN-T model 200 includes an encoder network 210, a prediction network 220, and a joint network 230. The encoder network 210, which is roughly similar to an acoustic model (AM) in a conventional ASR system, includes a stack of multi-head attention layers (e.g., Conformer layers or Transformer layers) or a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder reads a d-dimensional feature vector sequence (e.g., acoustic frame 110 ( Figure 1 ))x =(x1, x2, ··· , x T ),in , and a higher-order feature representation is generated at each output step. The higher-order feature representation is symbolically represented as .

[0026] Similarly, the prediction network 220 may include an LSTM network that, like a language model (LM), trains the sequence y0, . . . , y of non-whitespace symbols output so far by the final Softmax layer 240 ui−1 Processing as dense representation . Optionally, the prediction network 220 may include an embedding lookup table. Finally, using the RNN-T model architecture, the representations produced by the encoder network 210 and the prediction / decoder network 220 are combined by the joint network 230. The prediction network 220 may be replaced by an embedding lookup table to improve latency by outputting the sparse embeddings looked up instead of processing dense representations. The joint network then predicts , which is the distribution about the next output symbol. In other words, the joint network 230 generates a probability distribution over possible speech recognition hypotheses at each output step (e.g., time step). Here, "possible speech recognition hypotheses" correspond to a set of output labels that each represent a symbol / character using a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, for example, one label for each of the 26 letters in the English alphabet, and one label specifies a space. Therefore, the joint network 230 can output a set of values ​​that indicates the probability of occurrence of each output label in a predetermined set of output labels. This set of values ​​can be a vector and can indicate a probability distribution over a set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possible punctuation and other symbols), but the set of output labels is not limited to this. For example, in addition to graphemes or in place of graphemes, the set of output labels may include word fragments and / or entire words. The output distribution of the joint network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output y of the joint network 230 is i 100 different probability values ​​may be included, one for each output label. The probability distribution may then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, fragments, and / or words) during a beam search (e.g., by a Softmax layer 240) to determine the transcription 120.

[0027] The Softmax layer 240 may employ any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 at the corresponding output step. In this manner, the RNN-T model 200 does not make conditional independence assumptions, but instead the prediction of each symbol is conditioned not only on the acoustics, but also on the sequence of labels output so far. The RNN-T model 200 does assume that the output symbol is independent of future acoustic frames 110, which allows the RNN-T model to be employed in a streaming manner.

[0028] In some examples, the encoder network (i.e., audio encoder) 210 of the RNN-T model 200 includes a stack of multi-head attention (e.g., self-attention) layers / blocks such as a conformer block or a transformer block. Here, each conformer block (or other type of multi-head attention block) includes a series of multi-head self-attention layers, depth-wise convolutional layers, and feed-forward layers. The prediction network 220 can have two 2,048-dimensional LSTM layers, each of which is also followed by a 640-dimensional projection layer. Alternatively, the prediction network 220 can include a stack of transformer blocks or conformer blocks, or an embedding lookup table instead of an LSTM layer. Finally, the joint network 230 can also have 640 hidden units. The softmax layer 240 can be composed of a unified set of word fragments or graphemes that are generated using all unique word fragments or graphemes in multiple training data sets.

[0029] Figure 3A A schematic diagram of an example training data selection process 300 for selecting data from an unlabeled training data corpus 358 for pre-training an automatic speech recognition model 200 is shown. Here, an unlabeled training data corpus 358 (stored in a data repository 158) may be fed to a comparative data selector model 310 along with a domain identifier 313 indicating a target domain for which the ASR model 200 is being trained. The comparative data selector model 310 may then select an unlabeled training data subset 359 corresponding to a target domain 324 from the unlabeled training data corpus 358. For example, the unlabeled training data corpus 358 may include a plurality of spoken utterances 360, 360a-n across a plurality of domains, wherein each spoken utterance 360 ​​includes corresponding unpaired audio data 362 that characterizes the spoken utterance 360. As used herein, unpaired audio data 362 refers to audio data that characterizes a spoken utterance 360 ​​that is not paired with a corresponding transcription of the spoken utterance 360. In other words, the unpaired audio data 362 can be interchangeably referred to as untranscribed audio data. The comparative data selector model 310 can then process the corresponding unpaired audio data 362 for each spoken utterance 360 ​​from the unlabeled training data corpus 358 (i.e., utterance collection 358) to select spoken utterances 360 corresponding to the target domain to include in the unlabeled training data subset 359 (i.e., spoken utterance subset 359).

[0030] The comparative data selection process 300 can select an unlabeled training data subset 359 from the unlabeled training data corpus 358 for pre-training the ASR model 200, whereby the utterances 360 selected to be included in the unlabeled training data subset 359 are most similar to the target domain 324 that the ASR model 200 is being pre-trained to learn. That is, the process 300 is able to identify in-domain and domain-near utterances 360 from the unlabeled training data corpus 358 to include in the unlabeled training data subset 359 for pre-training the ASR model 200. Notably, the spoken utterances 360 included in the unlabeled training data subset 358 selected by the process 300 enable the on-the-fly synthesis of different utterances during batch construction, such that new speaker embeddings z and latent variables Z can be sampled for each spoken utterance 360 ​​in the batch.

[0031] The unlabeled training data corpus 358 includes unpaired audio data 362 representing numerous spoken utterances 360 from across a wide range of domains, and includes much greater linguistic diversity than the target domain 324 in which the ASR model 200 is being trained to learn. As previously discussed, labeled training data may not provide coverage of all domains because labeled training data for underrepresented domains may be difficult to obtain. By using the unlabeled training data 358, and more specifically, by using a subset of unlabeled training data 359 corresponding to the target domain 324 selected from the unlabeled training data corpus 358, the ASR model 200 can be more fully trained to learn underrepresented domains. In addition, by selecting a subset of unlabeled training data 359 corresponding to the target domain from the large unlabeled training data corpus 358, the ASR model 200 can be trained more efficiently than training on the entire unlabeled training data corpus 358, which includes data across multiple domains that are different from the target domain that the ASR model 200 is being trained to learn. Unlabeled training data corpus 358 may change dynamically to incorporate new training data 358 from which training data selection process 300 may select.

[0032] The training data selection process 300 aims to improve the match between the audio-only utterances represented by the selected unlabeled training data subset 359 and the target domain 324, which in turn reduces the computational resources required to utilize large amounts of non-domain-specific data. Therefore, the training data selection process 300 reduces computational and memory costs by selecting an unlabeled training data subset 359 that includes utterances that best match the target domain 324 that the ASR model 200 is being trained to learn.

[0033] refer to Figure 3BIn some examples, the training data selection process 300 selects an unlabeled training data subset 359 that best matches the target domain 324 from an available unlabeled training data corpus 358 by providing a domain identifier 313 associated with the target domain 324 as an input to a general language model (LM) 335 previously trained on a general utterance corpus. As previously described, the unlabeled training data corpus 358 spans a wide variety of different domains. In these examples, the general LM 335 may include a maximum entropy (MaxEnt LM) that can optionally accept the domain identifier 313 as input, as described in U.S. Patent No. 9,842,592, filed on February 12, 2014, the contents of which are incorporated herein by reference in their entirety. Here, the domain identifier 313 associated with the target domain 324 may allow the MaxEnt LM to output an unlabeled training data subset 359 selected from the unlabeled training data corpus 358, which may include those utterances 360 having words and / or phrases related to the target domain 324. In some configurations, rather than evaluating the likelihood of words, the statistical language model operates in reverse mode to randomly generate utterances that match the statistical distribution of words related to the target domain 324.

[0034] Continue to refer Figure 3B , the training data selection process 300 implements a comparative data selection model 310 as well as a quantizer 320 and a word-based data selector 330 to analyze the audio data 362 representing each utterance 360 ​​from the unlabeled training data set 358 to determine whether the utterance 360 ​​corresponds to the target domain 324, whereby when the utterance 360 ​​corresponds to the target domain 324, the process then adds the utterance 360 ​​to the unlabeled training data subset 359. The training data selection process 350 begins by feeding the audio data 362 representing the spoken utterance 360 ​​to the quantizer 320. The quantizer 320 can map the continuous features of the utterance (i.e., the audio data 362) into a discrete word-gram sequence 312. The quantizer 320 can be placed at different depths of the ASR system 105. In some implementations, the quantizer 320 is deployed immediately after the feature encoder and before any transformer / conformer representation learning layer. In these implementations, the discrete word-gram sequence 312 is of a low semantic level and therefore contains information such as pitch, background noise, and other cofounding details conveyed by the audio data 362. In other implementations, the quantizer 320 is implemented on top of all representation learning layers. In general, any suitable quantizer 320 and related configurations may be used to quantize the audio data 362 representing each utterance 360 ​​into a corresponding discrete word-gram sequence 312.

[0035] The domain-specific LM 325 and the general LM 335 can each receive a corresponding discrete word sequence 312 for each spoken utterance 360 ​​output from the quantizer 320 to enable comparative selection of spoken utterances 360 for inclusion in the unlabeled training data subset 359. Here, the domain-specific LM 325 is trained on a corpus of the target domain 324. In other words, the domain-specific LM 325 is considered to be trained on data belonging to the target domain 324, and the ASR model 200 is being trained for learning. On the other hand, the general LM 335 is trained on a corpus of a general domain (i.e., multiple domains). Multiple domains span many different domains. In some examples, the language models 325, 335 are trained using an n-gram language model. In other examples, the language models 325, 335 are trained using a neural network language model training.

[0036] The process evaluates each discrete word-gram sequence 312 using two LMs 325, 335 by determining a first probability 326 associated with the discrete word-gram sequence 312 occurring in the target domain 324. (determined by the domain-specific LM 325), and determining a second probability 336 associated with the discrete word-gram sequence 312 occurring in the general domain Thereafter, for each utterance 360 ​​in the unlabeled training data set 358, the process 300 determines a domain relevance score (DRS) 346 at the scorer 345 based on the first probability 326, the second probability 336, and the corresponding vector quantized representation (i.e., discrete word sequence 312) of the utterance 360. For example, the DRS 346 for each utterance 360 ​​may be calculated as follows.

[0037] After determining the domain relevance scores 346, the process 300 selects the utterances 360 with the N best scores 346 to include in the unlabeled training data subset 359. In some implementations, the scorer 345 compares the domain relevance score 346 of each utterance 360 ​​to a threshold score, and selects the utterance 360 ​​when the DRS 346 of the utterance 360 ​​satisfies the threshold score. In other implementations, the threshold score is selected based on the distribution of the domain relevance scores 346. The unlabeled training data corpus 358 may include millions of utterances 360, while the unlabeled training data subset 359 may be much smaller. However, the unlabeled training data subset 359 related to the target domain 324 may include a number of utterances 360 that is greater than the number of utterances present in the available labeled training data for the target domain 324. By using unlabeled training data 358, specifically an unlabeled training data subset 359 corresponding to the target domain 324, to pre-train the ASR model 200, the model 200 will provide better results for speech spoken in the target domain 324, even when the target domain 324 is underrepresented in the labeled training data conventionally used to train the ASR model 200. Notably, the unpaired audio data 362 representing the spoken utterances 360 selected by the training data selection process 300 to be included in the unlabeled training data subset 359 can be used to pre-train the ASR model 200, while the labeled training data can be used to fine-tune the ASR model 200 via supervised learning. The pre-trained ASR model 200 can include an encoder (e.g., Figure 2 As used herein, the pre-trained ASR model 200 may include an encoder network 210 of the pre-trained ASR model 200.

[0038] Figure 4 A training process 400 for training an ASR model 200 is shown. In some implementations, the process 400 employs a two-step training technique including pre-training and training. Pre-training a model is a technique for initializing a model, which can then be further fine-tuned based on labeled training data 410. For an ASR model 200, pre-training can include starting the ASR model 200 with selected unlabeled training data 359, the selected unlabeled training data including unpaired audio data 362 representing a plurality of spoken utterances 360 of one or more speakers. As described above ( Figure 3A), unlabeled training data 359 may include a plurality of utterances corresponding to the target domain 324, wherein the unlabeled training data 359 is a subset 360 of utterances selected from a larger corpus of utterances across a plurality of different domains. The unlabeled training data 359 used for pre-training may be speech synthesized from a reference transcription using a predetermined voice and / or non-synthesized speech samples spoken by a real human. In some implementations, pre-training the model 200 may include implementing one or more self-supervised training techniques for learning from unlabeled training data.

[0039] After pre-training is complete, the process 400 can fine-tune the parameters of the pre-trained ASR model 200. The training process 400 includes training, for example, the encoder and / or decoder of the ASR model 200, either individually or jointly in any suitable combination. Here, the encoder 210 can be pre-trained on an unlabeled training data subset 359, and then the pre-trained encoder 210 is fine-tuned on a labeled training input together with the decoder (e.g., the prediction and joint network 230). The process 400 includes feeding the labeled training input 410 to the ASR model 200. In some implementations, the labeled training input 410 includes a plurality of utterances 412 corresponding to audio of speech samples spoken by a variety of different speakers. Here, the plurality of utterances 412 can be different from the utterances 360 of the unlabeled training data 358 used to pre-train the model 200. In addition, each labeled training input 410 can be paired with a corresponding label 414, which indicates a target output associated with the corresponding utterance 412. For example, the label 414 can be a true value transcription corresponding to the corresponding utterance 412. Upon receiving the labeled training input 410, the ASR model 200 can generate an output 415 (e.g., the transcription 120). The ASR model 200 can process the labeled training input 410 for speech recognition in any suitable manner.

[0040] In some implementations, the loss function 430 compares the output 415 to the label 414 to generate a supervised loss term 440, which indicates the difference between the label 414 (i.e., the target output) and the output 415 (i.e., the predicted output). The loss function 430 can implement any suitable technique to determine the loss, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross entropy, hinge loss, multi-class loss, etc. The training process can apply the supervised loss term 440 to the ASR model 200 via back propagation to update the parameters of the ASR model 200. Here, a technique such as stochastic gradient descent back propagates the supervised loss term 440 to adjust one or more parameters of the ASR model 200. In some implementations, when the ASR model 200 is properly trained, the model is frozen. In other words, the parameters remain unchanged for a period of time until it is considered that the ASR model 200 needs to be retrained (e.g., when enough new labeled training data 410 is obtained) or replaced.

[0041] Figure 5 359 is a flowchart of an exemplary arrangement of operations of a method 500 for pre-training an automatic speech recognition model 200 using selected unlabeled training data 359. The method 500 may use data stored in memory hardware 620 ( Figure 6 ) on the data processing hardware 610 ( Figure 6 ) is executed on the . Data processing hardware 610 and memory hardware 620 may reside on Figure 1 and computing device 600 ( Figure 6 ) on the corresponding remote system 150.

[0042] At operation 502, the method 500 includes obtaining an unlabeled training data corpus 358 including a plurality of spoken utterances 360, each of which includes corresponding audio data 362 representing the utterance 360. At operation 504, the method 500 also includes receiving a target domain 324. At operation 506, the method 500 includes using the comparative data selection model 310 to select a subset of spoken utterances 360 corresponding to the target domain 324 from the unlabeled training data corpus 358. At operation 508, the method 500 includes training an automatic speech recognition (ASR) model 200 on the subset of utterances 359.

[0043] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0044] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0045] Figure 6 6 is a schematic diagram of an example computing device 600 that can be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0046] The computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to the memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and the storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 610 can process instructions for execution within the computing device 600, including instructions stored in the memory 620 or on the storage device 630, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 680 coupled to the high-speed interface 640). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. In addition, multiple computing devices 600 can be connected, each of which provides part of the necessary operations (for example, as a server library, a blade server group, or a multi-processor system).

[0047] The memory 620 stores information non-temporarily within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 620 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0048] The storage device 630 is capable of providing mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configurations). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0049] The high-speed controller 640 manages bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 640 is coupled to a memory 620, a display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.

[0050] As shown, computing device 600 may be implemented in a variety of different forms. For example, it may be implemented as a standard server 600a or multiple times as a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0051] Various implementations of the systems and techniques described herein may be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that may be executed and / or interpreted on a programmable system that includes at least one programmable processor that may be special purpose or general purpose and that may be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0052] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0053] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, a processor suitable for executing a computer program includes both a general-purpose microprocessor and a special-purpose microprocessor, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or to transfer data to one or more mass storage devices or both. However, a computer does not necessarily have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0054] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0055] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (500) which, when executed by data processing hardware (610), causes the data processing hardware (610) to perform operations comprising: Obtaining an unlabeled training data corpus (358) comprising a first set of spoken utterances (360), each corresponding spoken utterance (360) in the first set of spoken utterances (360) comprising audio data (362) representing the corresponding spoken utterance (360); receive_target_field(324); using a contrastive data selection model (310) to select a subset (360) of spoken utterances from the unlabeled training data corpus (358) that corresponds to the target domain (324); as well as An automatic speech recognition (ASR) model (200) is trained on the subset of spoken utterances (360) selected from the unlabeled training data corpus (358) corresponding to the target domain (324).

2. The computer-implemented method (500) of claim 1, wherein training the ASR model (200) comprises training the ASR model (200) on the spoken utterance subset (360) using a self-supervised training technique.

3. The computer-implemented method (500) of claim 1 or 2, wherein the selected subset of spoken utterances (360) includes a smaller number of spoken utterances (360) than the number of spoken utterances (360) in the unlabeled training data corpus (358).

4. The computer-implemented method (500) of any one of claims 1 to 3, wherein the operations further comprise refining the ASR model (200) by training the ASR model (200) on labeled training data (410), the labeled training data (410) comprising: second oral utterance set (360); as well as A corresponding ground-truth transcription (120) is paired with each spoken utterance (360) in the second set of spoken utterances (360).

5. The computer-implemented method (500) of any one of claims 1 to 4, wherein selecting the subset of spoken utterances (360) comprises, for each corresponding spoken utterance (360) in the unlabeled training data corpus (358): quantizing the audio data (362) representing the spoken utterance (360) to produce a corresponding discrete word-gram sequence (312); generating a domain relevance score (346) based on the corresponding discrete word-gram sequence (312); and A determination is made based on the domain relevance score (346) to select the corresponding spoken utterance (360) or to omit selection of the corresponding spoken utterance (360).

6. The computer-implemented method (500) of claim 5, wherein generating the domain relevance score (346) further comprises: processing the corresponding discrete word-gram sequence (312) of the corresponding spoken utterance (360) using a first language model (325) trained on target training data corresponding to the target domain (324) to generate a first output (326); processing the corresponding discrete word-gram sequence (312) of the utterance using a second language model (335) trained on generic training data corresponding to a plurality of domains to generate a second output (336); as well as The first output (326) is compared to the second output (336) to determine the domain relevance score (346).

7. The computer-implemented method (500) of claim 6, wherein the domain relevance score (346) is based on a difference between the first output (326) and the second output (336) divided by the number of word-grams in the corresponding discrete word-gram sequence (312) of the corresponding spoken utterance (360).

8. The computer-implemented method (500) of claim 6 or 7, wherein the first language model (325) and the second language model (335) comprise N-gram language models.

9. The computer-implemented method (500) of any one of claims 5 to 8, wherein quantizing the spoken utterance (360) comprises mapping one or more continuous features of the audio data (362) characterizing the spoken utterance (360) into the discrete word-gram sequence (312).

10. The computer-implemented method (500) of any one of claims 1 to 9, wherein the comparative data selection model (310) comprises a self-supervised learning model.

11. A system (100), comprising: Data processing hardware (610); as well as Memory hardware (620) in communication with the data processing hardware (610), the memory hardware (620) storing instructions that, when executed on the data processing hardware (610), cause the data processing hardware (610) to perform operations, the operations comprising: Obtaining an unlabeled training data corpus (358) comprising a first set of spoken utterances (360), each corresponding spoken utterance (360) in the first set of spoken utterances (360) comprising audio data (362) representing the corresponding spoken utterance (360); receive_target_field(324); using a contrastive data selection model (310) to select a subset (360) of spoken utterances from the unlabeled training data corpus (358) that corresponds to the target domain (324); and An automatic speech recognition (ASR) model is trained on the subset (360) of spoken utterances corresponding to the target domain (324) selected from the unlabeled training data corpus (358).

12. The system (100) of claim 11, wherein training the ASR model (200) comprises using a self-supervised training technique to train the ASR model (200) on the subset of spoken utterances (360).

13. The system (100) of claim 11 or 12, wherein the selected subset of spoken utterances (360) includes a smaller number of spoken utterances (360) than the number of spoken utterances (360) in the unlabeled training data corpus (358).

14. The system (100) of any one of claims 11 to 13, wherein the operations further comprise refining the ASR model (200) by training the ASR model (200) on labeled training data (410), the labeled training data (410) comprising: second oral utterance set (360); as well as A corresponding ground-truth transcription (120) is paired with each spoken utterance (360) in the second set of spoken utterances (360).

15. The system (100) of any one of claims 11 to 14, wherein selecting the subset of spoken utterances (360) comprises, for each corresponding spoken utterance (360) in the unlabeled training data corpus (358): quantizing the audio data (362) representing the spoken utterance (360) to produce a corresponding discrete word-gram sequence (312); generating a domain relevance score (346) based on the corresponding discrete word-gram sequence (312); and A determination is made based on the domain relevance score (346) to select the corresponding spoken utterance (360) or to omit selection of the corresponding spoken utterance (360).

16. The system (100) of claim 15, wherein generating the domain relevance score (346) further comprises: processing the corresponding discrete word-gram sequence (312) of the corresponding spoken utterance (360) using a first language model trained on target training data corresponding to the target domain (324) to generate a first output (326); processing a corresponding discrete word-gram sequence (312) of the utterance using a second language model trained on generic training data corresponding to a plurality of domains to generate a second output (336); as well as The first output (326) is compared to the second output (336) to determine the domain relevance score (346).

17. The system (100) of claim 16, wherein the domain relevance score (346) is based on a difference between the first output (326) and the second output (336) divided by the number of tokens in the corresponding discrete token sequence (312) of the corresponding spoken utterance (360).

18. The system (100) of claim 16 or 17, wherein the first language model and the second language model comprise N-gram language models.

19. The system (100) of any one of claims 15 to 18, wherein quantizing the spoken utterance (360) comprises mapping one or more continuous features of the audio data (362) characterizing the spoken utterance (360) into the discrete word-gram sequence (312).

20. The system (100) according to any one of claims 11 to 19, wherein the comparative data selection model (310) comprises a self-supervised learning model.

Images