Induced data selection for masked speech modeling

Abstract

A method of guided data selection for masked speech modeling includes obtaining a sequence of coded representations corresponding to an utterance. For each respective coded representation, the method includes processing the respective coded representation to generate a corresponding probability distribution for possible speech recognition hypotheses and assigning to each coded representation a confidence score as the highest probability from the corresponding probability distribution for possible speech recognition hypotheses. The method also includes selecting a set of unmasked coded representations to mask based on the confidence scores assigned to the sequence of coded representations. The method also includes generating a set of masked coded representations by masking the selected set of unmasked coded representations, where each masked coded representation corresponds to a respective one of the unmasked coded representations in the selected set of unmasked coded representations.

Description

[Technical Field]

[0001] This disclosure relates to the selection of induced data for masked speech modeling. [Background technology]

[0002] Automatic speech recognition (ASR), the process of receiving audio input and transcribing it into text, is a highly important technology used in mobile and other devices. Generally, automatic speech recognition attempts to provide an accurate transcription of what a person says by receiving audio input (e.g., speech utterances) and transcribing the audio input into text. Modern ASR models continue to improve in both accuracy (e.g., low word error rate (WER)) and latency (e.g., delay between user utterance and transcription), based on the ongoing development of deep neural networks. However, one challenge in developing deep learning-based ASR models is that the parameters of the ASR model tend to overfit to the training data, thereby causing the ASR model to have the problem of generalizing to unknown data when the training data is not sufficiently large. As a result, training an ASR model with a relatively large training dataset improves the accuracy of the ASR model. To increase the amount of training data used to train an ASR model, synthesized speech and / or data-augmented speech can be incorporated. [Overview of the project] [Means for solving the problem]

[0003] One aspect of the present disclosure provides a computer implementation method, when executed on data processing hardware, that causes data processing hardware to perform an operation for induced data selection for masked speech modeling. The operation includes obtaining a sequence of coded representations corresponding to utterances. For each coded representation in the sequence of coded representations, the operation includes processing the coded representation using a scoring model to generate a corresponding probability distribution for possible speech recognition assumptions for the coded representation, and assigning a confidence score to the coded representation as the highest probability from the corresponding probability distribution for possible speech recognition assumptions for the coded representation. The operation also includes selecting a set of unmasked coded representations to be masked from the sequence of coded representations based on the confidence scores assigned to the sequence of coded representations. The operation also includes generating a set of masked coded representations by masking the selected set of unmasked coded representations. Each masked coded representation in the set of masked coded representations corresponds to each of the unmasked coded representations in the selected set of unmasked coded representations.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, selecting a set of unmasked coded representations to be masked includes selecting the top K coded representations with the highest confidence scores from a sequence of coded representations. In these implementations, K may be based on a predetermined proportion of coded representations in the sequence of coded representations to be masked. Optionally, the predetermined proportion may be equal to 40 percent.

[0005] In some examples, the operation further includes using a quantizer to generate a corresponding target context vector for each unmasked encoding in a selected set of unmasked encodings to be masked. In these examples, the operation also includes generating a corresponding contrast context vector for each masked encoding in a set of masked encodings, generating a contrast loss based on the corresponding contrast context vector and corresponding target context vector generated for each unmasked encoding corresponding to each masked encoding, and pre-training an audio encoder using the contrast losses generated for the set of masked encodings. In some implementations, the operation further includes: generating a corresponding K-means cluster for each unmasked coding representation in a selected set of unmasked coding representations to be masked, using a cluster module; generating a cross-entropy loss for each masked coding representation in a set of masked coding representations, based on the corresponding contrast context vector and corresponding K-means cluster generated for each unmasked coding representation corresponding to each masked coding representation; and pre-training an audio encoder using the cross-entropy loss generated for the set of masked coding representations.

[0006] The operation may further include determining the final training goal based on the contrast loss and cross-entropy loss for each masked encoding in the set of masked encodings, and pre-training the audio encoder with the final training goal generated for the set of masked encodings. In some implementations, the operation further includes determining the speech level confidence score by averaging the confidence scores in the set of masked encodings, weighting the final training goal based on the speech level confidence score, and pre-training the audio encoder with the weighted final training goal. Optionally, the operation further includes extracting bottleneck features from the contrast context vector, where the operation further includes refining each corresponding K-mean cluster using the extracted bottleneck features.

[0007] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware that stores instructions, when executed on the data processing hardware, causing the data processing hardware to perform an action. The action includes obtaining a sequence of coded representations corresponding to a speech. For each coded representation in the sequence of coded representations, the action includes processing the coded representation using a scorer model to generate a corresponding probability distribution for possible speech recognition assumptions for the coded representation, and assigning a confidence score to the coded representation as the highest probability from the corresponding probability distribution for possible speech recognition assumptions for the coded representation. The action also includes selecting a set of unmasked coded representations to be masked from the sequence of coded representations based on the confidence score assigned to the sequence of coded representations. The action also includes generating a set of masked coded representations by masking the selected set of unmasked coded representations. Each masked coded representation in the set of masked coded representations corresponds to each of the unmasked coded representations in the selected set of unmasked coded representations.

[0008] Implementations of this disclosure may include one or more of the following features. In some implementations, selecting a set of unmasked coded representations to be masked includes selecting the top K coded representations with the highest confidence scores from a sequence of coded representations. In these implementations, K may be based on a predetermined proportion of coded representations in the sequence of coded representations to be masked. Optionally, the predetermined proportion may be equal to 40 percent.

[0009] In some examples, the operation further includes using a quantizer to generate a corresponding target context vector for each unmasked encoding in a selected set of unmasked encodings to be masked. In these examples, the operation also includes generating a corresponding contrast context vector for each masked encoding in a set of masked encodings, generating a contrast loss based on the corresponding contrast context vector and corresponding target context vector generated for each unmasked encoding corresponding to each masked encoding, and pre-training an audio encoder using the contrast losses generated for the set of masked encodings. In some implementations, the operation further includes: generating a corresponding K-means cluster for each unmasked coding representation in a selected set of unmasked coding representations to be masked, using a cluster module; generating a cross-entropy loss for each masked coding representation in a set of masked coding representations, based on the corresponding contrast context vector and corresponding K-means cluster generated for each unmasked coding representation corresponding to each masked coding representation; and pre-training an audio encoder using the cross-entropy loss generated for the set of masked coding representations.

[0010] The operation may further include determining the final training goal based on the contrast loss and cross-entropy loss for each masked encoding in the set of masked encodings, and pre-training the audio encoder with the final training goal generated for the set of masked encodings. In some implementations, the operation further includes determining the speech level confidence score by averaging the confidence scores in the set of masked encodings, weighting the final training goal based on the speech level confidence score, and pre-training the audio encoder with the weighted final training goal. Optionally, the operation further includes extracting bottleneck features from the contrast context vector, where the operation further includes refining each corresponding K-mean cluster using the extracted bottleneck features.

[0011] Details of one or more implementations of this disclosure are described in the accompanying drawings and the following description. Other embodiments, features, and advantages will become apparent from the description and drawings, as well as from the claims. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic diagram of an exemplary speech recognition system. [Figure 2] This is a schematic diagram of an exemplary speech recognition model. [Figure 3A] This is a schematic diagram of the example masking request (ask-to-mask) training process. [Figure 3B] This is a schematic diagram of the exemplary masking request training process. [Figure 3C] This is a schematic diagram of the exemplary masking request training process. [Figure 4] This is a schematic diagram of an exemplary masking module. [Figure 5]An exemplary sequence flowchart of operations for a computer implementation method of induced data selection for mask voice modeling. [Figure 6] A schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein. **DETAILED DESCRIPTION**

[0013] Like reference numerals in the various drawings indicate like elements.

[0014] Automated speech recognition (ASR) has made tremendous progress with the introduction of the sequence-to-sequence (Seq2Seq) model that maps from audio to a character sequence. At the same time, text-to-speech (TTS) or speech synthesis systems have successfully applied the Seq2Seq model to obtain state-of-the-art natural, realistic-sounding synthesized speech that may be indistinguishable from human speech to the human ear.

[0015] One challenge in developing a deep learning-based ASR model is that the parameters of the ASR model tend to overfit the training data, which causes the ASR model to have the problem of generalizing unknown data when the training data is not large enough. Therefore, training the ASR model with a relatively large training data set improves the accuracy of the ASR model. For example, the ASR model can be trained with a training data set containing transcribed speech for about 10,000 hours using machine learning or other statistical methods. However, when the domain associated with the training data is different from the domain in which the ASR model is deployed during inference, the performance of the ASR model degrades. For example, training the ASR model with speech in a domain associated with video conferencing is not relatively effective when recognizing speech related to voice search queries, and vice versa.

[0016] The performance of ASR models has been shown to improve by pre-training them on a large amount of unlabeled speech or text data, and then fine-tuning the pre-trained ASR model by training it on a relatively small set of labeled speech or text data. In particular, masked speech modeling (MSM) is a pre-training technique that receives an unlabeled sequence of input speech frames corresponding to training utterances and masks a portion of the input speech frames for each training utterance. The MSM pre-training method then learns speech representations from the masked input speech frames. However, the input speech frames selected for masking are arbitrarily chosen, even if not all input speech frames contain relevant information for learning meaningful representations. In other words, selecting speech frames that contain relevant information for masking may be more beneficial for the ASR model to learn meaningful representations during pre-training.

[0017] Therefore, the implementations described herein concern methods and systems for induced data selection for MSM. More specifically, the MSM pre-training process may perform a masking module that acquires a sequence of encoded representations corresponding to utterances. The masking module processes each encoded representation to generate a corresponding probability distribution for possible speech recognition assumptions and assigns a confidence score as the highest probability from the corresponding probability distribution. Based on the confidence scores assigned to the sequence of encoded representations, the masking module masks a selected set of encoded representations. Advantageously, the masking module selects encoded representations for masking that contain relevant information, thereby increasing the number of meaningful representations that the ASR model learns during pre-training. Furthermore, as will become apparent, the masking module may determine an utterance-level confidence score for the entire utterance and weight the final training goal used to train the ASR model based on the utterance-level confidence score.

[0018] Figure 1 shows an ASR system 100 that implements an automated speech recognition (ASR) model 200 residing on user device 102 of user 104 and / or on a remote computing device 201 communicating with user device 102 (for example, one or more servers in a distributed system running in a cloud computing environment). User device 102 is shown as a mobile computing device (for example, a smartphone), but user device 102 may be any type of computing device, including, but not limited to, a tablet device, laptop / desktop computer, wearable device, digital assistant device, smart speaker / display, smart gadget, in-car infotainment system, or Internet of Things (IoT) device, and comprises data processing hardware 111 and memory hardware 113.

[0019] The user device 102 includes an audio subsystem 108 configured to receive a speech 106 uttered by the user 104 (for example, the user device 102 may include one or more microphones for recording the uttered speech 106) and convert the speech 106 into a corresponding digital format associated with an input acoustic frame 110 that can be processed by the ASR system 100. In the illustrated example, the user utters each speech 106 in natural English for the phrase "What is the weather in New York City?", and the audio subsystem 108 converts the speech 106 into a corresponding acoustic frame 110 for input to the ASR system 100. The ASR model 200 then receives the acoustic frame (i.e., sequence of input speech frames) 110 corresponding to the speech 106 as input and generates / predicts a corresponding transcription 120 (e.g., recognition result / assumption) of the speech 106 as output. In the illustrated example, user device 102 and / or remote computing device 201 also run a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to user 104 of user device 102. In some configurations, the transcription 120 output from ASR system 100 is processed by a natural language understanding (NLU) module running on user device 102 or remote computing device 201, for example, to execute user commands. Additionally or alternatively, a text-to-speech system (for example, running on any combination of user device 102 or remote computing device 201) may convert the transcription into synthesized speech for audible output by another device. For example, the original utterance 106 may correspond to a message that user 104 is sending to a friend, where the transcription 120 is converted into synthesized speech for audible output to the friend who is to hear the message conveyed in the original utterance 106.

[0020] Referring to Figure 2, the exemplary ASR model 200 includes a Recurrent Neural Network-Transducer (RNN-T) model architecture that adheres to latency constraints associated with the interactive application. The use of the RNN-T model architecture is exemplary, and the ASR model 200 may include other architectures, in particular, transformer transducer and conformer transducer model architectures. The RNN-T model architecture of the ASR model (i.e., the RNN-T model) 200 enables a small computational footprint, utilizes fewer memory requirements than conventional ASR architectures, and makes the RNN-T model architecture suitable for performing speech recognition entirely on the user device 102 (e.g., without requiring communication with a remote server). The RNN-T model 200 includes an encoder network 210, a prediction network 220, and a collaborative network 230. The encoder network 210 is generally similar to the acoustic model (AM) of conventional ASR systems and includes a recurrent network of a stack of self-attention layers (such as conformer layers or transformer layers) or stacked Long Short-Term Memory (LSTM) layers. For example, the encoder is a sequence of d-dimensional feature vectors (e.g., acoustic frame 110 (Figure 1)) x = (x1, x2, ..., x T ) read, and here,

[0021]

number

[0022] And, at each output step, a higher-order feature representation is generated. This higher-order feature representation is

[0023]

number

[0024] It is written as follows.

[0025] Similarly, the prediction network 220 is also an LSTM network, which, like a language model (LM), uses the sequence of non-blank symbols output so far by the final softmax layer 240, namely y0, ..., y ui-1 Process and create a dense expression

[0026]

number

[0027] The representations produced by the encoder and predictor / decoder networks 210 and 220 are finally combined by the collaborative network 230 using an RNN-T model architecture. The predictor network 220 may be replaced with an embedding lookup table to improve latency by outputting sparsely looked-up embeddings instead of processing dense representations. The collaborative network then

[0028]

number

[0029] The network predicts the distribution of the next output symbols. In other words, at each output step (e.g., a time step), the collaborative network 230 generates a probability distribution of possible speech recognition assumptions, where “possible speech recognition assumptions” correspond to a set of output labels, each representing a symbol / character in a given natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet and one label for a space. Thus, the collaborative network 230 may output a set of values ​​indicating the likelihood of each occurrence of a given set of output labels. This set of values ​​may be a vector and can represent a probability distribution of the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), but the set of output labels is not limited to that. For example, the set of output labels may include word pieces and / or entire words in addition to graphemes. The output distribution of the collaborative network 230 may include the posterior probability value for each of the different output labels. Therefore, if there are 100 different output labels representing different graphemes or other symbols, the output y of the collaborative network 230 will be... i This can include 100 different probability values, one for each output label. The probability distribution can then be used to select scores and assign them to candidate orthographic elements (e.g., graphemes, wordpieces, and / or words) in a beam search process (e.g., by a softmax layer 240) to determine the transcription 120.

[0030] The softmax layer 240 may use any technique to select the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model 200 in the corresponding output step. Thus, the RNN-T model 200 does not make conditional independence assumptions; rather, the prediction of each symbol is conditional not only on acoustics but also on the sequence of labels output up to that point. The RNN-T model 200 assumes that the output symbols are independent of future acoustic frames 110, thereby enabling the RNN-T model 200 to be used in a streaming manner.

[0031] In some examples, the encoder network (i.e., audio encoder) 210 of the RNN-T model 200 includes a stack of self-attention layers / blocks, such as conformer blocks, where each conformer block includes a series of multi-head self-attention, depth-direction convolutional layers, and feedforward layers. The prediction network 220 may have two 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer. Alternatively, the prediction network 220 may include a stack of transformers or conformer blocks, or an embedded lookup table, instead of LSTM layers. Finally, the collaborative network 230 may also have 640 hidden units. The softmax layer 240 may consist of a set of unified wordpieces or graphemes generated using all unique wordpieces or graphemes across multiple training datasets.

[0032] Figures 3A–3C illustrate an exemplary ask-to-mask (ATM) training process 300 using different MSM architectures for pre-training the ASR model 200 (Figure 2). The ATM training process 300 (also simply called the “training process 300”) can pre-train the ASR model 200 using available training data, which includes a set of non-speaking text utterances 320, a set of transcribed non-synthesized speech utterances 304, and / or untranscribed non-synthesized speech utterances 306. Each non-speaking text utterance 320 contains text-only data (i.e., unpaired data) such that each non-speaking text utterance 320 is not paired with any corresponding spoken audio representation (i.e., speech) of the utterance. A non-speaking text utterance 320 may contain any sequence of text chunks, including words, wordpieces, phonemes, and / or graphemes. Each untranscribed non-synthesized speech utterance 306 (also simply called "untranscribed speech utterance 306") contains audio-only data (i.e., unpaired data) such that the untranscribed speech utterance is not paired with any corresponding transcription. On the other hand, each transcribed non-synthesized speech utterance 304 (also simply called "transcribed speech utterance 304") contains a corresponding transcription (not shown) which is paired with the corresponding non-synthesized speech representation of the corresponding transcribed speech utterance.

[0033] The training data may also include synthesized speech representations (e.g., synthesized speech) 332 for each of the multiple non-speaking training text utterances 320. That is, each non-speaking training text utterance 320 includes non-speaking text that is text-only data, i.e., unpaired data, such that each non-speaking training text utterance is not paired with any synthesized or unsynthesized speech. Thus, the text-to-speech (TTS) system 330 can generate a corresponding synthesized speech representation 332 for each of the non-speaking training text utterances 320. In particular, the synthesized speech representation may include Mel-frequency spectrogram frames for training the ASR model 200, thereby eliminating the need for the training process 300 to include a vocoder and / or synthesizer to synthesize the Mel-frequency spectrogram frames into synthesized speech.

[0034] The TTS system 330 can apply speaker embeddings z when converting non-spoken text utterances 320 to generate a synthesized speech expression 332 having a specific speaking style and prosody associated with the speaker embeddings. The TTS system 330 can apply a number of different speaker embeddings z, each associated with different speaker characteristics of the resulting synthesized speech expression 332. Similarly, the TTS system 330 can alter the prosody and other production qualities of the synthesized utterances. In some examples, the training process 300 applies data augmentation to at least one of the sample utterances of the synthesized speech expression 332. Data augmentation may include, but is not limited to, adding noise to the corresponding synthesized speech expression, manipulating timing (e.g., stretching), or adding reverberation. Data augmentation may add different synthesis recording conditions to the synthesized speech expression 332.

[0035] For simplicity, the training process 300 includes a control self-supervised loss unit 300a (Figure 3A), a cross-entropy self-supervised loss unit 300b (Figure 3B), and a final training target self-supervised loss unit 300c (Figure 3C). The training process 300 can pre-train the audio encoder 210 using any combination of losses derived from the control self-supervised loss unit 300a (Figure 3A), the cross-entropy self-supervised loss unit 300b (Figure 3B), and the final training target self-supervised loss unit 300c (Figure 3C).

[0036] Continuing to refer to Figures 3A to 3C, in the illustrated example, the audio encoder 210 of the ASR model 200 (Figure 2) includes a stack of self-attention layers, each containing a multi-head (e.g., 8-head) self-attention mechanism. For example, the stack of self-attention layers may include a stack of conformer layers or transformer layers. In the illustrated example, the audio encoder 210 includes a conformer encoder containing a stack of conformer blocks, each conformer block containing a series of multi-head self-attention layers, depth-direction convolutional layers, and feedforward layers. The conformer encoder 210 can be divided into a feature encoder containing a convolutional subsampling block 212 and a context network containing a linear layer 214 and a stack of conformer blocks 216. In some implementations, the convolutional subsampling block 212 has two 2D convolutional layers, both with a stride of (2,2), resulting in a quarter reduction in feature sequence length.

[0037] The convolutional subsampling block 212 receives as input a sequence of input speech frames (e.g., Mel frequency spectrograms such as acoustic frame 110 in Figure 1) associated with each transcribed speech utterance 304, each untranscribed unsynthesized speech utterance 306, and each synthesized speech representation 332, and generates as output a sequence of encoded representations 211, 211a~n, corresponding to each of the transcribed speech utterances 304, the untranscribed speech utterances 306, or the synthesized speech representations 332. Each encoded representation 211 in the sequence of encoded representations 211 may represent a grapheme, phoneme, wordpiece, or word. Furthermore, each sequence of encoded representations 211 corresponds to each of the utterances in the training data. The sequence of encoded representations 211 output from the convolutional subsampling block 212 is (E=[e1,e2,...,e T ]) may be fed to a masking module 400, which masks a selected set of encoded unmasked representations 211.

[0038] Referring here to Figure 4, in some implementations, the masking module 400 includes a scorer model 410 and a masker 420. The scorer model 410 (also simply called "scorer 410") takes a sequence of encoded representations 211 and processes each respective encoded representation 211 to generate a corresponding probability distribution 414 for each possible speech recognition assumption 412 for each encoded representation 211. The sequence of encoded representations 211 received by the scorer 410 is not masked. Furthermore, the scorer 410 generates a probability distribution at the frame level (for example, for each encoded representation 211). The probability distribution 414 includes the probability associated with each possible speech recognition assumption (i.e., label). In some cases, the scorer 410 uses the probability distribution 414, P=p(e t =l|E); l∈L (1) It can be determined by that.

[0039] In Equation 1, P represents the probability distribution 414 for each coded representation 211, and l represents each of the latent speech recognition assumptions (i.e., labels) from a set of multiple latent speech recognition assumptions L (i.e., labels) in the codebook. For example, as shown in Figure 4, the scorer 410 receives three coded representations 211a to c, each representing a word corresponding to the utterance "Show Google Adsense". In this example, the scorer 410 processes a first coded representation 211a representing the word "Show" to generate a corresponding first probability distribution 414 for possible speech recognition assumptions 412 for the first coded representation 211a; processes a second coded representation 211b representing the word "Google" to generate a corresponding second probability distribution 414 for possible speech recognition assumptions 412 for the second coded representation 211b; and processes a third coded representation 211c representing the word "Adsense" to generate a corresponding third probability distribution 414 for possible speech recognition assumptions 412 for the third coded representation 211c. Here, the first probability distribution 414 includes probabilities of 0.7 and 0.3 for possible speech recognition assumptions 412 "show" and "snow," respectively; the second probability distribution 414 includes probabilities of 0.9 and 0.1 for possible speech recognition assumptions 412 "Google" and "Doodle," respectively; and the third probability distribution 414 includes probabilities of 0.6 and 0.4 for possible speech recognition assumptions 412 "Adsense" and "Accents," respectively.

[0040] In some implementations, the scorer 410 is an external ASR model trained on training data similar to the target domain (i.e., in-domain data). The scorer 410 may be a frame-synchronous ASR model that utilizes a connectionist temporal classification (CTC) objective such that the ASR model generates a frame-level (e.g., for each encoded representation 211) probability distribution. The scorer 410 may generate a probability distribution 414 based on the likelihood that each speech recognition assumption 412 for each encoded representation 211 is an accurate transcription of the corresponding utterance. In other implementations, the scorer 410 generates a probability distribution 414 based on the similarity between each speech recognition assumption 412 and the target domain. For example, the target domain for the ASR model 200 may be associated with video conferencing, thereby causing the scorer 410 to generate higher probabilities for speech recognition assumptions 412 that are likely to belong to the target domain (e.g., video conferencing). Optionally, the scorer 410 may use some combination of transcription accuracy and similarity to the target domain to generate the probability distribution 414. Thus, a speech recognition assumption 412 with high probability indicates that the speech recognition assumption 412 is likely to contain meaningful information for the ASR model 200 (Figure 2) to learn meaningful representations during pre-training.

[0041] Furthermore, the scorer 410 assigns a reliability score 416 as the highest probability from the corresponding probability distribution 414 relating to the speech recognition assumption 412 for each encoded representation 211. The scorer 410 assigns a reliability score 416 as the highest probability from the corresponding probability distribution 414 relating to the speech recognition assumption 412.

[0042]

number

[0043] It can be assigned by.

[0044] In equation 2, st This represents the confidence score 416 for each encoded representation 211. The scorer 410 has K masking start indices {i1,...,i k}of,

[0045]

number

[0046] Sampling is performed using the following probability.

[0047] One point to note is that in Equation 3,

[0048]

number

[0049] This ensures that the scorer 410 samples without substitution. Thus, the scorer 410 samples the initial encoded representation 211 in the sequence of encoded representations 211 with a probability proportional to the probability for each possible speech recognition assumption 412. Continuing the above example, the scorer 410 assigns 0.7 as the confidence score 416 for the first encoded representation 211a, 0.9 as the confidence score 416 for the second encoded representation 211b, and 0.6 as the confidence score 416 for the third encoded representation 211c, because these probabilities are the highest probabilities in their respective probability distributions 414. As shown in Figure 4, the confidence scores 416 assigned to each encoded representation 211 are indicated by dashed boxes.

[0050] In some examples, the scorer 410 assigns the confidence score 416 as a low probability from the corresponding probability distribution 414 for the speech recognition assumption 412 for each encoded representation 211. The scorer 410 may also assign the confidence score as a mixture of high and low probabilities. In particular, the scorer 410 assigns the confidence score 416 as

[0051]

number

[0052] It may be assigned to a low probability.

[0053] The masker 420 receives the confidence score 416 assigned to each coded expression 211 in the sequence of coded expressions 211 and selects a set of unmasked coded expressions 211 to be masked. In other words, the masker 420 only needs to select the top K coded expressions 211 from the sequence of unmasked coded expressions 211 from which the ASR model 200 learns meaningful expressions during pre-training. Here, K represents the number of coded expressions 211 that the masker 420 selects. In some cases, K is based on a predetermined ratio of coded expressions 211 in the sequence of coded expressions 211 to be masked. For example, K may represent a predetermined ratio of 40 percent, but it will be understood that the predetermined ratio can be any ratio. Thus, in this example, the masker 420 may set K to 4 for utterances containing 10 coded expressions 211 in the sequence of coded expressions 211.

[0054] The masker 420 generates a set of masked coded representations 211, 211m by masking a selected set of unmasked coded representations 211. Continuing with the example shown in Figure 4, the predetermined ratio may be 33 percent, thereby causing the masker 420 to select two coded representations 211 out of a total of three coded representations 211 that have the highest reliability score 416. Here, the masker 420 selects the first coded representation 211a and the second coded representation 211b as the set of unmasked coded representations 211 for masking because these coded representations 211 have the highest assigned reliability score 416. The masker 420 then generates a set of masked coded representations 211ma, 211mb by masking the first coded representation 211a and the second coded representation 211b, which are indicated by black boxes. Therefore, each masked coded representation 211m in the set of masked coded representations 211m corresponds to each of the unmasked coded representations 211 in the selected set of unmasked coded representations 211. As shown in Figure 4, the first coded representation 211a corresponds to the first masked coded representation 211ma, and the second coded representation 211b corresponds to the second masked coded representation 211mb. As will become clear, the training process 300 (Figure 3) simply pre-trains the ASR model 200 (Figure 2) using the set of masked coded representations 211m.

[0055] Referring to Figure 3A, the contrast self-supervised loss unit 300a of the training process 300 generates the contrast loss term 342 and the contrast training objective 345. Specifically, the linear layer 214 and conformer block 216 of the context network receive a set of masked encoded representations 211m and generate the corresponding contrast context vector 215(c) for each corresponding masked encoded representation 211m in the set of masked encoded representations 211m. jOutputs ∈ C). That is, the audio encoder 210 generates a contrast context vector 215 for all masked time instances j ∈ J. Moreover, the quantizer 218 receives as input a selected set of unmasked encoded features 211 and generates as output a corresponding quantization vector 219 (Q = [q1, q2,..., q T ) for each respective unmasked encoded feature 211. Each target context vector (i.e., the target context vector) 219 includes L dimensions that denote the number of codes in the target or codebook. The quantizer 218 may be a Gumbel softmax quantizer.

[0056] Subsequently, the contrast loss module 340 generates a corresponding contrast loss term 342 (L ctr ) based on the corresponding contrast context vector 215 and the corresponding target context vector 219 generated for each respective unmasked encoded representation 211 for each masked encoded representation 211m. In some examples, the training process 300 pre-trains the audio encoder 210 by directly using the contrast loss term 342. In other examples, the contrast loss module 340 determines a diversity loss (L wv ) for deriving a contrast training target 345 (L div ) used to pre-train the audio encoder 210. In these examples, the contrast loss module 340 derives the contrast training target 345 as L wv = L ctr + 0.1·L div (5) .

[0057] In Equation 5,

[0058]

Number

[0059] and L divThis represents the diversity loss. The contrast loss term 342 represents both positive and negative examples depending on the codebook, and the diversity loss promotes increased use of quantized codebook representations. In particular, the diversity loss promotes equal use of V entries in each of the G codebooks by maximizing the entropy of the averaged softmax distribution across the codebook entries for each codebook. The contrast loss module 340 represents the diversity loss (L div )of,

[0060]

number

[0061] This can be derived by [method].

[0062] The training process 300 can train the ASR model 200 (Figure 2) by updating the parameters of the audio encoder 210 based on the control training objective 345 and / or the control loss term 342.

[0063] Referring here to Figure 3B, the cross-entropy loss section 300b of the training process 300 generates a cross-entropy loss 355 for pre-training the ASR model 200 (Figure 2). In particular, the linear layer 214 and conformer block 216 of the context network receive a set of masked coded representations 211m and the corresponding contrast context vector 215(c j The audio encoder 210 outputs a contrast context vector 215 for each corresponding masked encoded representation 211m in the set of masked encoded representations 211m. In other words, the audio encoder 210 generates a contrast context vector 215 for all masked time instances j ∈ J. Furthermore, the cluster module 222 receives a selected set of unmasked encoded features 211 as input and outputs the corresponding K-mean cluster 223 (Y=[y1,y2,...,y TThe output is generated as follows. Optionally, the cluster module 222 may receive the bottleneck features 217 extracted from the control context vector 215 and use the extracted bottleneck features 217 to further refine each K-mean cluster 223. It should be noted that each K-mean cluster 223 represents a target for each of the control context vectors 215.

[0064] Subsequently, the cross-entropy loss module 350 calculates the cross-entropy loss 355(L) based on the corresponding contrast context vector 215 and corresponding K-mean cluster 223 generated for each unmasked encoded representation 211 corresponding to each masked encoded representation 211m. ce The result is derived. The training process 300 pre-trains the audio encoder 210 by updating the parameters of the audio encoder 210 based on the cross-entropy loss 355.

[0065] Referring here to Figure 3C, the final training goal of the training process 300, the self-supervised loss unit 300c, generates the final training goal 365 for pre-training the ASR model 200 (Figure 2). Specifically, the linear layer 214 and conformer block 216 of the context network receive a set of masked encoded representations 211m and generate the corresponding contrast context vector 215(c) for each corresponding masked encoded representation 211m in the set of masked encoded representations 211m. j It outputs ∈C). In other words, the audio encoder 210 generates a contrast context vector 215 for all masked time instances j ∈ J. Furthermore, the audio encoder 210 may further refine the contrast context vector 215 to generate a refined contrast context vector 213. The audio encoder 210 outputs the refined contrast context vector 213,

[0066]

number

[0067] It is generated by [this method].

[0068] In Equation 7, y j represents the refined contrast context vector 213. Here, the contrast context vector 215 is the target for the target context vector 219, and the refined contrast context vector is the target for the K-mean cluster 223. The quantization module 224 may receive a selected set of unmasked coded features 211 as input. In some cases, the quantization module 224 includes a quantizer 218 (Figure 3A) and a cluster module 222 (Figure 3B). Thus, the quantization module 224 uses the quantizer 218 (Figure 3A) to generate the corresponding quantization vector 219 (Q=[q1,q2,...,q T ]) is generated as output, and the cluster module 222 (Figure 3B) is used to generate the corresponding K-mean cluster 223 (Y=[y1,y2,...,y T This generates ]) as output.

[0069] In some examples, the target loss module 360 ​​determines the final training target 365(L) based on the control training target 345 and the cross-entropy loss 355. wb In particular, the target loss module 360 ​​derives the contrast loss term 342 and the contrast loss target 345 based on the corresponding contrast context vector 215 and the corresponding target context vector 219 generated for each unmasked encoded representation 211 corresponding to each masked encoded representation 211m. Furthermore, the target loss module 360 ​​derives the cross-entropy loss 355(L) based on the corresponding refined contrast context vector 213 and the corresponding K-means cluster 223 generated for each unmasked encoded representation 211 corresponding to each masked encoded representation 211m.ce ) is derived. For each masked coded representation 211m, the target loss module 360 ​​determines the final training target based on the contrast loss term 342 (or contrast loss target 345) and the cross-entropy loss 355. L wb =L ce +L wv (8) It will be determined by that.

[0070] In Equation 8, L wb represents the final training goal 365. The training process 300 can train the audio encoder 210 by updating the parameters of the audio encoder 210 based on the final training goal 365.

[0071] As described above, the training process 300 determines the loss for each encoded representation 211 (e.g., at the frame level) in the sequence of encoded representations 211 corresponding to utterances. In some cases, the training process 300 may benefit from selecting training samples at a coarser utterance level rather than the frame level. Thus, the training process 300 calculates the confidence score 416 for all masked encoded representations 211m in the set of masked encoded representations 211.

[0072]

number

[0073] By averaging these values, a vocal level reliability score can be determined.

[0074] In equation 9, s uThis represents the vocal level reliability score. Furthermore, the training process 300 may weight the final training objective 365 based on the vocal level reliability score. For example, the training process 300 assigns a higher weight to the final training objective 365 associated with a higher vocal level reliability score. Therefore, a weighted final training objective 365 with a higher weight will have a greater impact on the pre-training of the audio encoder 210 than a weighted final training objective 365 with a lower weight.

[0075] Figure 5 is a flowchart illustrating an exemplary sequence of operations for method 500 of induced data selection for masked speech modeling. Method 500 can be executed on data processing hardware 610 (Figure 6) using instructions stored in memory hardware 620 (Figure 6). The data processing hardware 610 and memory hardware 620 may reside on user device 102 and / or remote computing device 201 in Figure 1, corresponding to computing device 600 (Figure 6).

[0076] In operation 502, method 500 includes the step of obtaining a sequence of encoded representations 211, 211a~n corresponding to the utterance 106. For each encoded representation 211 in the sequence of encoded representations 211, method 500 performs operations 504 and 506. In operation 504, method 500 includes the step of processing each encoded representation 211 using a scorer model 410 to generate a corresponding probability distribution 414 for possible speech recognition assumptions 412 for each encoded representation 211. In operation 506, method 500 includes the step of assigning a confidence score 416 to each encoded representation 211 as the highest probability from the corresponding probability distribution 414 for speech recognition assumptions 412 for each encoded representation 211. In operation 508, method 500 includes the step of selecting a set of unmasked encoded representations 211 to be masked from the sequence of encoded representations 211, based on the confidence scores 416 assigned to the sequence of encoded representations 211. In operation 510, method 500 includes the step of generating a set of masked coded representations 211, 211m by masking a selected set of unmasked coded representations 211, where each masked coded representation 211m in the set of masked coded representations 211m corresponds to each of the unmasked coded representations 211 in the selected set of unmasked coded representations 211.

[0077] Figure 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described herein. The computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are illustrative and do not limit the implementations of the invention described and / or claimed herein.

[0078] The computing device 600 includes a processor 610, memory 620, storage device 630, a high-speed interface / controller 640 connected to memory 620 and high-speed expansion port 650, and a low-speed bus 670 and a low-speed interface / controller 660 connected to storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 may be interconnected using various buses and mounted on a common motherboard or in other ways as needed. The processor 610 can process instructions for execution within the computing device 600, including instructions stored in memory 620 or on storage device 630 for displaying graphical information about a graphical user interface (GUI) on an external input / output device such as a display 680 coupled to the high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used with multiple memories and multiple types of memory as needed. Furthermore, multiple computing devices 600 may be connected, each providing a necessary part of the operation (for example, as a server bank, a group of blade servers, or a multiprocessor system).

[0079] Memory 620 stores information non-temporarily within the computing device 600. Memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporarily stored memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (for example, typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disks or tapes.

[0080] The storage device 630 can provide large-capacity storage to the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product, when executed, includes instructions that perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as memory 620, the storage device 630, or memory on the processor 610.

[0081] The high-speed controller 640 manages bandwidth-intensive operations for the computing device 600, while the low-speed controller 660 manages more bandwidth-intensive operations. Such a division of roles is illustrative only. In some implementations, the high-speed controller 640 is coupled to memory 620, display 680 (e.g., through a graphics processor or accelerator), and high-speed expansion port 650, which can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to storage device 630 and low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), but may also be coupled to one or more input / output devices such as a keyboard, pointing device, or scanner, or to network devices such as a switch or router, for example, through a network adapter.

[0082] The computing device 600 may be implemented in several different forms, as shown in the figure. For example, the computing device 600 may be implemented as a standard server 600a, or multiple times in a group of such servers 600a, or as a laptop computer 600b, or as part of a rack server system 600c.

[0083] Various implementations of the systems and techniques described herein can be realized in digital electronic and / or optical circuit configurations, integrated circuit configurations, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, the programmable processor may be dedicated or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0084] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for programmable processors and may be implemented in highly procedural and / or object-oriented programming languages, as well as in assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-temporary computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor that includes a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0085] The processes and logical flows described herein can be implemented by one or more programmable processors, also called data processing hardware, which execute one or more computer programs to perform functions by acting on input data and producing outputs. Processes and logical flows can also be implemented by special-purpose logic circuit configurations, such as FPGAs (field programmable gate arrays) and ASICs (application-specific integrated circuits). Processors suitable for executing computer programs include, for example, both general-purpose and dedicated microprocessors, as well as any one or more processors in any type of digital computer. Generally, processors will receive instructions and data from read-only memory or random-access memory or both. Essential elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer may also include one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or may be operably coupled to mass storage devices to receive data from or transfer data, or both. However, a computer is not required to have such devices. Computer-readable media suitable for storing computer program instructions and data include, for example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory can be complemented by or incorporated into special-purpose logic circuit configurations.

[0086] To enable interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or touchscreen, and optionally a keyboard and pointing device for providing input to the computer, such as a mouse or trackball. Other types of devices may also be used to enable interaction with a user, for example, the feedback given to the user may be any form of sensory feedback, such as visual feedback, auditory feedback or haptic feedback, and input from the user may be received in any form, including acoustic, voice or haptic input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser in response to a request received from a web browser on the user's client device.

[0087] Several implementations have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other implementations are within the scope of the following claims. [Explanation of Symbols]

[0088] 100 Automated Speech Recognition (ASR) Systems 102 User Devices 107 Interface Generator 108 Audio Subsystem 111 Data Processing Hardware 113 Memory Hardware 200 ASR model, RNN-T model 201 Remote Computing Devices 210 encoder network, Audio encoder 212 Convolutional Subsampling Blocks 214 Linear Layer 216 Conformer Block 218 Quantizer 220 Predictive Networks 222 Cluster Modules 224 Quantization Module 230 Joint Networks 240 Softmax Layer 330 Text-to-Speech (TTS) Systems 340 Controlled Loss Module 350 Cross-Entropy Loss Module 360 Target Loss Module 400 Masking Modules 410 Scorer Model, Scorer 420 Masking Tape 600 Computing Devices 600a Standard Server, Server 600b Laptop Computer 600c Rack Server System 610 Data processing hardware, processors, and components 620 Memory Hardware Components 630 Storage devices, components 640 High-Speed ​​Interfaces / Controllers, Components 650 high-speed expansion ports, components 660 Low-Speed ​​Interface / Controller, Components 670 Slow Bus 680 displays 690 Low-Speed ​​Expansion Ports

Claims

1. A computer implementation method (500) for inducible data selection for masked speech modeling, wherein the computer implementation method (500), when executed on data processing hardware (610), causes the data processing hardware (610) to perform an operation, the operation being Obtaining a sequence of encoded representations (211) corresponding to the utterance (106), For each of the encoded representations (211) in the sequence of encoded representations (211), Using the scorer model (410), each of the encoded representations (211) is processed to generate a corresponding probability distribution (414) for possible speech recognition assumptions (412) for each of the encoded representations (211). A reliability score (416) is assigned to each of the above-mentioned encoded representations (211) as the highest probability from the corresponding probability distribution (414) relating to the possible speech recognition assumptions (412) for each of the above-mentioned encoded representations (211), Based on the reliability score (416) assigned to the sequence of the coded representation (211), a set of unmasked coded representations (211) to be masked is selected from the sequence of the coded representation (211), A computer implementation method (500) comprising generating a set of masked coded representations (211m) by masking the selected set of unmasked coded representations (211), wherein each masked coded representation (211m) in the set of masked coded representations (211m) corresponds to each of the unmasked coded representations (211) in the selected set of unmasked coded representations (211).

2. The computer implementation method (500) according to claim 1, wherein selecting the set of unmasked coded representations (211) to be masked comprises selecting the top K coded representations (211) having the highest confidence score (416) from the sequence of coded representations (211).

3. The computer implementation method (500) according to claim 2, wherein K is based on a predetermined ratio of encoded representations (211) in the sequence of encoded representations (211) to be masked.

4. The computer implementation method (500) according to claim 3, wherein the predetermined ratio is equal to 40 percent.

5. The aforementioned operation is, For each of the unmasked coded representations (211) in the selected set of unmasked coded representations (211) to be masked, a quantizer (218) is used to generate a corresponding target context vector (219) for each of the unmasked coded representations (211), For each masked encoded representation (211m) in the set of masked encoded representations (211m), A corresponding contrast context vector (215) is generated for each of the masked encoded representations (211m), A contrast loss (342) is generated based on the corresponding target context vector (219) and the corresponding contrast context vector (215) generated for the respective unmasked encoded representation (211) corresponding to each masked encoded representation (211m), A computer implementation method (500) according to any one of claims 1 to 4, further comprising pre-training an audio encoder (210) using the contrast loss (342) generated for the set of masked encoded representations (211m).

6. The aforementioned operation is, For each of the unmasked coded representations (211) in the selected set of unmasked coded representations (211) to be masked, the cluster module (222) is used to generate a corresponding K-means cluster (223) for each of the unmasked coded representations (211), For each masked coded representation (211m) in the set of masked coded representations (211m), a cross-entropy loss (355) is generated based on the corresponding K-mean cluster (223) generated for the corresponding unmasked coded representation (211m) corresponding to the respective masked coded representation (211m), and the corresponding contrast context vector (215). The computer implementation method (500) according to claim 5, further comprising pre-training the audio encoder (210) using the cross-entropy loss (355) generated for the set of masked encoded representations (211m).

7. The aforementioned operation is, For each masked encoded representation (211m) in the set of masked encoded representations (211m), the final training target (365) is determined based on the contrast loss (342) and the cross-entropy loss (355). The computer implementation method (500) according to claim 6, further comprising pre-training the audio encoder (210) using the final training target (365) generated for the set of masked encoded representations (211m).

8. The aforementioned operation is, The speech level reliability score (416) is determined by averaging the reliability scores (416) in the set of masked encoded representations (211m), The final training objective (365) is weighted based on the aforementioned vocalization level reliability score (416), The computer implementation method (500) according to claim 7, further comprising pre-training the audio encoder (210) using the weighted final training goal (365).

9. The computer implementation method (500) according to claim 6, further comprising extracting a bottleneck feature (217) from the contrast context vector (215).

10. The computer implementation method (500) according to claim 9, further comprising using the extracted bottleneck features (217) to refine each corresponding K-mean cluster (223).

11. Data processing hardware (610), A system (100) comprising a data processing hardware (610) and a memory (620) hardware that communicates with the data processing hardware (610), wherein the memory (620) hardware, when executed on the data processing hardware (610), stores instructions that cause the data processing hardware (610) to perform an operation, and the operation is Obtaining a sequence of encoded representations (211) corresponding to the utterance (106), For each of the encoded representations (211) in the sequence of encoded representations (211), Using the scorer model (410), each of the encoded representations (211) is processed to generate a corresponding probability distribution (414) for possible speech recognition assumptions (412) for each of the encoded representations (211). A reliability score (416) is assigned to each of the above-mentioned encoded representations (211) as the highest probability from the corresponding probability distribution (414) relating to the possible speech recognition assumptions (412) for each of the above-mentioned encoded representations (211), Based on the reliability score (416) assigned to the sequence of the coded representation (211), a set of unmasked coded representations (211) to be masked is selected from the sequence of the coded representation (211), A system (100) comprising generating a set of masked coded representations (211m) by masking the selected set of unmasked coded representations (211), wherein each masked coded representation (211m) in the set of masked coded representations (211m) corresponds to each of the unmasked coded representations (211) in the selected set of unmasked coded representations (211).

12. The system (100) according to claim 11, wherein selecting the set of unmasked coded representations (211) to be masked comprises selecting the top K coded representations (211) having the highest confidence score (416) from the sequence of coded representations (211).

13. The system (100) according to claim 12, wherein K is based on a predetermined ratio of encoded representations (211) in the sequence of encoded representations (211) to be masked.

14. The system (100) according to claim 13, wherein the predetermined ratio is equal to 40 percent.

15. The aforementioned operation is, For each of the unmasked coded representations (211) in the selected set of unmasked coded representations (211) to be masked, a quantizer (218) is used to generate a corresponding target context vector (219) for each of the unmasked coded representations (211), For each masked encoded representation (211m) in the set of masked encoded representations (211m), A corresponding contrast context vector (215) is generated for each of the masked encoded representations (211m), A contrast loss (342) is generated based on the corresponding target context vector (219) and the corresponding contrast context vector (215) generated for the respective unmasked encoded representation (211) corresponding to each masked encoded representation (211m), The system (100) according to any one of claims 11 to 14, further comprising pre-training an audio encoder (210) using the contrast loss (342) generated for the set of masked encoded representations (211m).

16. The aforementioned operation is, For each of the unmasked coded representations (211) in the selected set of unmasked coded representations (211) to be masked, the cluster module (222) is used to generate a corresponding K-means cluster (223) for each of the unmasked coded representations (211), For each masked coded representation (211m) in the set of masked coded representations (211m), a cross-entropy loss (355) is generated based on the corresponding K-mean cluster (223) generated for the corresponding unmasked coded representation (211m) corresponding to the respective masked coded representation (211m), and the corresponding contrast context vector (215). The system (100) according to claim 15, further comprising pre-training the audio encoder (210) using the cross-entropy loss (355) generated for the set of masked encoded representations (211m).

17. The aforementioned operation is, For each masked encoded representation (211m) in the set of masked encoded representations (211m), the final training target (365) is determined based on the contrast loss (342) and the cross-entropy loss (355). The system (100) according to claim 16, further comprising pre-training the audio encoder (210) using the final training target (365) generated for the set of masked encoded representations (211m).

18. The aforementioned operation is, The speech level reliability score (416) is determined by averaging the reliability scores (416) in the set of masked encoded representations (211m), The final training objective (365) is weighted based on the aforementioned vocalization level reliability score (416), The system (100) according to claim 17, further comprising pre-training the audio encoder (210) using the weighted final training goal (365).

19. The system (100) according to claim 16, further comprising extracting a bottleneck feature (217) from the contrast context vector (215).

20. The system (100) according to claim 19, further comprising using the extracted bottleneck features (217) to refine each corresponding K-mean cluster (223).

Images