Contrastive Siamese Networks for Semi-Supervised Speech Recognition

The contrastive Siamese network leverages both labeled and unlabeled data for semi-supervised speech recognition, enhancing training efficiency and accuracy by utilizing unlabeled data to improve ASR systems.

JP2024541132AActive Publication Date: 2024-11-07GOOGLE LLC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024519720
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-09-30
Filing Date
2021-12-14
Publication Date
2024-11-07
Estimated Expiration
2041-12-14

AI Technical Summary

Technical Problem

Collecting large amounts of labeled training data for automatic speech recognition (ASR) systems is difficult due to time, cost, and privacy concerns, making supervised learning inefficient.

Method used

A contrastive Siamese network is employed for semi-supervised speech recognition, utilizing both labeled and unlabeled audio data. The network includes an unsupervised subnetwork that trains on unlabeled audio samples to generate higher-order feature representations and an unsupervised loss term, while a supervised subnetwork uses labeled data for supervised learning.

Benefits of technology

This approach allows for effective training of ASR systems using unlabeled data, reducing the reliance on labeled data and addressing the challenges of data collection, while maintaining accuracy and improving speech recognition performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541132000001_ABST
    Figure 2024541132000001_ABST
Patent Text Reader

Abstract

The method (400) includes receiving a plurality of unlabeled audio samples (303) corresponding to a spoken utterance (106) that are not paired with a corresponding transcription (308). In a target branch (310) of the control Siamese network (300), the method also includes generating a sequence of encoder outputs (312) for the plurality of unlabeled audio samples, and modifying a time characteristic of the encoder outputs to generate a sequence of target branch outputs (316). In an augmented branch (320) of the control Siamese network, the method also includes performing augmentation on the unlabeled audio samples, generating a sequence of augmented encoder outputs (326) for the augmented unlabeled audio samples, and generating a prediction (329) of the sequence of target branch outputs generated in the target branch. The method also includes determining an unsupervised loss term (330) and updating parameters of the audio encoder based on the unsupervised loss term.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to symmetric Siamese networks for semi-supervised speech recognition. [Background technology]

[0002] Automatic speech recognition (ASR) systems receive audio input and attempt to provide an accurate transcription of what a person said by transcribing the audio input into text. In many cases, supervised learning is used to train an ASR system with large amounts of labeled training data that includes audio data and corresponding transcriptions. However, obtaining the large amounts of labeled training data required to train an ASR system is often difficult due to the amount of time required, cost, and / or privacy issues associated with collecting large labeled training datasets. Training an ASR system with unlabeled training data that includes only audio data can alleviate some of the challenges of collecting large amounts of labeled training data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent Application Serial No. 17 / 210,465 Summary of the Invention [Means for solving the problem]

[0004] One aspect of the present disclosure provides a control Siamese network for training a speech recognition model. The control Siamese network includes an unsupervised sub-network trained with a plurality of unlabeled audio samples corresponding to spoken utterances that are not paired with any corresponding transcription. The unsupervised sub-network includes a target branch configured to receive a sequence of acoustic frames extracted from the unlabeled audio samples as an input to an audio encoder of the speech recognition model, and generate, at each of a plurality of time steps, a target branch output for a corresponding acoustic frame in the sequence of acoustic frames input to the audio encoder at the corresponding time step. The unsupervised sub-network also includes an augmented branch configured to perform augmentation on the sequence of acoustic frames extracted from the unlabeled audio samples to generate a sequence of augmented acoustic frames, generate, at each of the plurality of time steps, a high-order feature representation for the corresponding augmented acoustic frame in the sequence of augmented acoustic frames as an output from the audio encoder, and generate, at each of the plurality of time steps, a prediction of the target branch output generated by the target branch at the corresponding time step using the high-order feature representation output from the audio encoder at the corresponding time step. The unsupervised sub-network is configured to determine, at each of the plurality of time steps, an unsupervised loss term based on a target branch output generated by the target branch at the corresponding time step and a prediction of the target branch output generated by the expanded branch at the corresponding time step, where the unsupervised sub-network is also configured to update parameters of the audio encoder based on the unsupervised loss term determined at each of the plurality of time steps.

[0005] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the unsupervised loss term includes a contrastive loss term; The augmentation performed on the sequence of audio frames may include temporal modification and masking; In some examples, the target branch is further configured to generate, at each of the plurality of time steps, a high-order feature representation for a corresponding audio frame in the sequence of audio frames input to the audio encoder at the corresponding time step as an output from the audio encoder; wherein the target branch is configured to generate a target branch output for the corresponding audio frame by modifying temporal characteristics of the high-order feature representation; In these examples, modifying temporal characteristics of the high-order feature representation may include modifying, at each of the plurality of time steps, temporal characteristics of the high-order feature representation generated as an output from the audio encoder for the corresponding audio frame to match temporal characteristics associated with the high-order feature representation generated as an output from the audio encoder for the corresponding augmented audio frame at the corresponding time step.

[0006] In some implementations, the expanded branch includes a predictive network of a transformer layer configured to receive as an input, at each of a plurality of time steps, a high-level feature representation output from an audio encoder at a corresponding time step, and generate as an output, a prediction of a target branch output generated by a target branch at the corresponding time step. In some examples, the control Siamese network includes a supervised sub-network trained with a plurality of labeled audio samples corresponding to spoken utterances paired with corresponding transcriptions. In these examples, at each of the plurality of output steps for each labeled audio sample, the supervised sub-network is configured to generate a corresponding speech recognition result for the labeled audio sample using a speech recognition model, and determine a supervised loss term based on the corresponding speech recognition result for the labeled audio sample and the corresponding transcription of the labeled audio sample. Here, the supervised sub-network updates parameters of the speech recognition model based on the supervised loss term determined at each of the plurality of output steps for each labeled audio sample in the plurality of labeled audio samples.

[0007] The corresponding speech recognition results generated for the labeled audio samples using the speech recognition model may include a probability distribution for possible speech recognition hypotheses for the labeled audio samples in the corresponding output step. In some examples, the supervised sub-network is configured to update parameters of the speech recognition model based on the supervised loss term without relying on the unsupervised network updating parameters of the audio encoder of the speech recognition model. In other examples, the supervised sub-network is further configured to apply data augmentation to at least one of the labeled audio samples in the plurality of labeled audio samples input to the speech recognition model. In these other examples, the applied data augmentation includes at least one of adding noise, adding reverberation, or manipulating timing.

[0008] In some implementations, the trained speech recognition model includes a transformer transducer (TT) model including an audio encoder configured to receive as input a sequence of acoustic frames extracted from audio data characterizing a spoken utterance and generate, at each of a plurality of time steps, a high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. In these implementations, the TT model also includes a label encoder configured to receive as input a sequence of non-blank symbols output by a final softmax layer and generate, at each of a plurality of time steps, a dense representation, and a collaborative network configured to receive as input the high-order feature representation generated by the audio encoder at each of a plurality of time steps and the dense representation generated by the label encoder at each of a plurality of time steps and generate, at each of a plurality of time steps, a probability distribution over possible speech recognition hypotheses at a corresponding time step. Here, the audio encoder includes a neural network having a stack of strided convolutional layers and transformer layers.

[0009] Another aspect of the disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for training a speech recognition model using a control Siamese network. The operations include receiving a plurality of unlabeled audio samples corresponding to a spoken utterance that is not paired with a corresponding transcription. In a target branch of the control Siamese network, the operations include using an audio encoder of the speech recognition model to generate a sequence of encoder outputs for the plurality of unlabeled audio samples, and modifying temporal characteristics of the encoder outputs to generate a sequence of target branch outputs. In an augmented branch of the control Siamese network, the operations include performing augmentation on the unlabeled audio samples, using an audio encoder of the speech recognition model to generate a sequence of augmented encoder outputs for the augmented unlabeled audio samples, and using a predictive network configured to receive the sequence of augmented encoder outputs to generate a prediction of the sequence of target branch outputs generated in the target branch. The operations also include determining an unsupervised loss term based on the target branch outputs generated in the target branch and a prediction of the target sequence of branch outputs generated in the augmented branch. The operations also include updating parameters of the audio encoder based on the unsupervised loss.

[0010] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the unsupervised loss term includes a contrast loss term. Performing the augmentation on the unlabeled audio samples may include performing time correction and masking on the unlabeled audio samples. In some examples, the operations further include generating a high-order feature representation for the plurality of unlabeled audio samples as an output from the audio encoder. In these examples, modifying temporal characteristics of the encoder output to generate a sequence of target branch outputs includes modifying temporal characteristics of the encoder output generated as an output from the audio encoder to match temporal characteristics associated with the sequence of augmented encoder outputs from the audio encoder. In some implementations, the operations further include receiving the sequence of augmented encoder outputs as an input to a prediction network of a transformer layer of the augmented branch, and generating a prediction of the sequence of target branch outputs generated in the target branch as an output from the prediction network of the transformer layer of the augmented branch.

[0011] In some examples, the operations further include receiving a plurality of labeled audio samples corresponding to a spoken utterance paired with a corresponding transcription, generating speech recognition results for the labeled audio samples using a speech recognition model, determining a supervised loss term based on the speech recognition results for the labeled audio samples and the corresponding transcription of the labeled audio samples, and updating parameters of the speech recognition model based on the supervised loss term. In these examples, the operations may further include updating parameters of the speech recognition model based on the supervised loss term independent of updating parameters of the audio encoder based on the unsupervised loss term. Optionally, the operations further include applying data augmentation to at least one of the labeled audio samples, where applying data augmentation may include at least one of adding noise, adding reverberation, or manipulating timing.

[0012] In some implementations, the speech recognition model includes a transformer transducer (TT) model, and the operations further include receiving a plurality of unlabeled audio samples corresponding to the spoken utterance, not paired with a corresponding transcription, as input to an audio encoder of the TT model, generating, at each of the plurality of time steps, a sequence of acoustic frames extracted from the audio data characterizing the spoken utterance, by the audio encoder, receiving, at each of the plurality of time steps, a sequence of non-blank symbols output by the final softmax layer as input to a label encoder of the TT model, and generating, at each of the plurality of time steps, a dense representation, by the label encoder. In these implementations, the operations also include receiving, as input to a collaborative network of the TT model, the high-order feature representation generated by the audio encoder at each of the plurality of time steps and the dense representation generated by the label encoder at each of the plurality of time steps, and generating, at each of the plurality of time steps, a probability distribution over possible speech recognition hypotheses at the corresponding time step, by the collaborative network. Here, the audio encoder includes a neural network having a stack of strided convolutional layers and transformer layers.

[0013] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram of an audio environment in which an exemplary speech recognition model is implemented. [Diagram 2] FIG. 2 is a schematic diagram of the exemplary speech recognition model of FIG. 1. [Figure 3A] FIG. 1 is a schematic diagram of the supervised portion of a control Siamese network that performs a semi-supervised training process for a speech recognition model. [Figure 3B]FIG. 1 is a schematic diagram of the unsupervised portion of a control Siamese network that performs a semi-supervised training process for a speech recognition model. [Figure 4] 1 is a flowchart of an example sequence of operations for a method of training a speech recognition model using a control Siamese network. [Diagram 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Like reference numbers in the various drawings indicate like elements.

[0016] Automatic speech recognition (ASR) systems are often trained using supervised training techniques that leverage labeled training data. The labeled training data includes audio data of the speech and a corresponding transcription of the speech. Collecting large amounts of labeled training data is often difficult due to the associated cost, time required to collect the training data, and user privacy concerns. In some cases, the ASR system is trained using unlabeled training data that includes only audio data of the speech without any corresponding transcription. In these cases, the ASR may use only the unlabeled training data to train the speech recognition system (i.e., self-supervised training), or the unlabeled training data may be used to train the speech recognition system in addition to the labeled training data (i.e., semi-supervised training).

[0017] Implementations herein are directed to a contrast Siamese network that uses a semi-supervised training technique for training a speech recognition model. The Siamese network includes a supervised sub-network that trains the speech recognition model with labeled audio samples that include utterances and corresponding transcriptions. That is, the supervised sub-network receives acoustic frames extracted from the labeled audio samples and predicts speech recognition results. The supervised sub-network then determines a loss by comparing the predicted speech recognition results with the corresponding transcriptions, and updates the speech recognition model based on the loss.

[0018] The Siamese network also includes an unsupervised sub-network that trains the speech recognition model with unlabeled audio samples. The unsupervised sub-network may train the speech recognition model in addition to or as an alternative to the supervised sub-network. The unsupervised sub-network includes a target branch that receives acoustic frames from the unlabeled audio samples and generates a target branch output for each acoustic frame. The unsupervised sub-network also includes an augmented branch that performs augmentation on the acoustic frames from the unlabeled audio samples and generates a high-order feature representation (i.e., "encoder output") using the augmented acoustic frames. Thus, the augmented branch uses the high-order feature representation to predict the target branch output generated by the target branch. Using the target branch output from the target branch and the prediction of the target branch output by the augmented branch, the unsupervised sub-network determines an unsupervised loss term and updates parameters of an audio encoder of the speech recognition model based on the unsupervised loss term.

[0019] 1 is an example of an audio environment 100. In the audio environment 100, a user 104 may interact with a computing device, such as a user device 10, through voice input. The user device 10 is configured to capture sounds (e.g., streaming audio data) from one or more users 104 in the audio environment 100. Here, the streaming audio data may refer to spoken utterances 106 by the user 104 that function as audible inquiries, commands intended for the user device 10, or audible communications captured by the user device 10. A voice-enabled system of the user device 10 may field the inquiries or commands by responding to the inquiries and / or implementing / satisfying the commands through one or more downstream applications.

[0020] The user device 10 may correspond to any computing device associated with a user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a computer, a wearable device (e.g., a smart watch), a smart appliance, an Internet of Things (IoT) device, a vehicle infotainment system, a smart display, a smart speaker, etc. The user device 10 includes a data processing hardware 12 and a memory hardware 14 in communication with the data processing hardware 12, which stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 with audio capture devices (e.g., microphones) 16, 16a for capturing and converting spoken utterances 106 in the audio environment 100 into electrical signals, and audio output devices (e.g., speakers) 16, 16b for conveying audible audio signals (e.g., as output audio data from the user device 10). Although the user device 10 implements a single audio capture device 16a in the illustrated example, the user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, whereby one or more capture devices 16a of the array may not be physically on the user device 10 but may be in communication with the audio system 16.

[0021] In the speech environment 100, an automated speech recognition (ASR) system 118 implementing the speech recognition model 200 resides on the user device 10 of the user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executing in a cloud computing environment) that communicates with the user device 10 via the network 40. The user device 10 and / or the remote computing device (i.e., remote server) 60 also include an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 and captured by the audio capture device 16a and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 118. In the illustrated example, the user speaks each utterance 106, and the audio subsystem 108 converts the utterances 106 into corresponding audio data (e.g., acoustic frames) 110 for input to the ASR system 118. The speech recognition model 200 then receives as input audio data 110 corresponding to the utterance 106 and generates / predicts as output a corresponding transcription 120 (e.g., speech recognition result / hypothesis) of the utterance 106. As described in more detail below, the speech recognition model 200 may include a transformer transducer (TT) model 200 trained with a variable look-ahead audio context that allows the TT model 200 to set different durations of the look-ahead audio context during inference when performing speech recognition depending on how sensitive the query specified by the utterance 106 is to latency and / or how much tolerance the user 104 has for latency. For example, a digital assistant application 50 running on the user device 10 may require the speech recognition to be streaming so that words, word pieces, and / or individual characters appear on the screen as soon as they are spoken.Furthermore, it is likely that the user 104 of the user device 10 has a low tolerance for latency when issuing a query for the digital assistant application 50 to perform. In such a scenario, when minimizing speech recognition latency is preferred, the speech recognition model 200 may apply zero or minimal look-ahead audio context (also referred to as "correct context") to provide streaming transcription capabilities in real time as the user 104 is speaking the utterance 106. On the other hand, when the user has a higher tolerance for speech recognition latency and / or the utterance 106 to be recognized is associated with a long speech, the same speech recognition model 200 may apply a duration of look-ahead audio context sufficient to provide accurate transcription 120, but may incur increased latency based on the duration of the look-ahead audio context. Thus, the ASR system 118 need only implement a single speech recognition model 200 for multiple different speech recognition tasks to provide both streaming and non-streaming transcription capabilities without having to utilize a separate ASR model for each task.

[0022] In some implementations, the speech recognition model 200 performs both streaming and non-streaming speech recognition in parallel on the audio data 110. For example, in the illustrated example, the speech recognition model 200 performs streaming speech recognition on the audio data 110 to produce the partial speech recognition result 120, 120a, and non-streaming speech recognition on the same audio data 110 to produce the final speech recognition result 120, 120b. Notably, the speech recognition model 200 may use a first look-ahead audio context, which may be set to zero (or about 240 milliseconds) to produce the partial speech recognition result 120a, and a second look-ahead audio context of longer duration than the first look-ahead audio context to produce the final speech recognition result 120b. Thus, the final speech recognition result 120b for the input utterance 106 may lag from the partial speech recognition result 120a for the input utterance by a duration that is based on the difference between the second look-ahead audio context and the first look-ahead audio context.

[0023] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of the transcription 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 may display partial speech recognition results 120a in a streaming manner during time 1, followed by final speech recognition results 120b during time 2. In some configurations, the transcription 120 output from the ASR system 118 is processed by a natural language understanding (NLU) module executing on the user device 10 or the remote computing device 60, for example, to execute the user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device 10 or the remote computing device 60) may convert the transcription into synthetic speech for audible output by the user device 10 and / or another device.

[0024] In the illustrated example, a user 104 interacts with a program or application 50 (e.g., a digital assistant application 50) of the user device 10 that uses an ASR system 118. For example, FIG. 1 shows a user 104 communicating with the digital assistant application 50, which displays a digital assistant interface 18 on the screen of the user device 10 to show a conversation between the user 104 and the digital assistant application 50. In this example, the user 104 asks the digital assistant application 50, "What time is the concert tonight?" This question from the user 104 is a spoken utterance 106 that is captured by the audio capture device 16a and processed by the audio system 16 of the user device 10. In this example, the audio system 16 receives the spoken utterance 106 and converts it into acoustic frames 110 for input to the ASR system 118.

[0025] Continuing with this example, while receiving an acoustic frame (i.e., audio data) 110 corresponding to an utterance 106 as the user 104 speaks, the speech recognition model 200 uses a first look-ahead audio context to encode the acoustic frame 110, and then decodes the encoded acoustic frame 110 into a partial speech recognition result 120a using the first look-ahead audio context. During time 1, the user interface generator 107 presents a representation of the partial speech recognition result 120a of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18 in a streaming manner, such that words, word pieces, and / or individual characters appear on the screen as soon as they are spoken. In some examples, the first look-ahead audio context is equal to zero.

[0026] In parallel, and after all of the acoustic frames 110 corresponding to the utterance 106 have been received, the speech recognition model 200 encodes all of the acoustic frames 110 corresponding to the utterance 106 using a second look-ahead audio context, and then decodes the acoustic frames 110 into a final speech recognition result 120b using the second look-ahead audio context. The duration of the second look-ahead audio context may be 1.2 seconds, 2.4 seconds, or any other duration. In some examples, an indication such as an endpoint indicating that the user 104 has finished speaking the utterance 106 triggers the speech recognition model 200 to encode all of the acoustic frames 110 using the second look-ahead audio context. During time 2, the user interface generator 107 presents a representation of the final speech recognition result 120b of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18. In some implementations, the user interface generator 107 replaces the representation of the partial speech recognition result 120a with a representation of the final speech recognition result 120b. For example, if the final speech recognition result 120b is estimated to be more accurate than the partial speech recognition result 120a that was generated without utilizing the look-ahead audio context, the final speech recognition result 120b that is ultimately displayed as the transcription 120 may determine any phrases that may have been misrecognized in the partial speech recognition result 120a. In this example, the streaming partial speech recognition result 120a output by the speech recognition model 200 and displayed on the screen of the user device 10 at time 1 is associated with low latency, giving the user 104 a sense of responsiveness that the user's query is being processed, while the final speech recognition result 120b output by the speech recognition model 200 and displayed on the screen at time 2 utilizes the look-ahead audio context to improve the speech recognition quality in terms of accuracy, but at the expense of increased latency. However, because the partial speech recognition results 120a are displayed as the user speaks the utterance 106, the higher latency associated with producing and ultimately displaying the final recognition results is not noticeable to the user 104.

[0027] In the example shown in FIG. 1, the digital assistant application 50 may use natural language processing to respond to a question posed by the user 104. Natural language processing generally refers to the process of interpreting written words (e.g., partial speech recognition results 120a and / or final speech recognition results 120b) and determining whether the written words prompt any action. In this example, the digital assistant application 50 uses natural language processing to recognize that the question from the user 104 is about the user's schedule, and more specifically, about a concert in the user's schedule. By using natural language processing to recognize these details, the automated assistant returns a response 19 to the user's query, which states, "Doors open at 6:30 p.m. and the show starts at 8 p.m." In some configurations, the natural language processing occurs on a remote server 60 in communication with the data processing hardware 12 of the user device 10.

[0028] Referring to FIG. 2, the speech recognition model 200 may provide end-to-end (E2E) speech recognition by integrating acoustic, pronunciation, and language models into a single neural network, without requiring a lexicon or a separate text normalization component. Various structures and optimization mechanisms can increase accuracy and reduce model training time. In some implementations, the speech recognition model 200 includes a transformer transducer (TT) model architecture, which adheres to latency constraints associated with interactive applications. The TT model 200 may include the TT model 200 described in U.S. Patent Application No. 17 / 210,465, filed March 23, 2021, the entire contents of which are incorporated herein by reference. The TT model 200 enables a small computational footprint and uses less memory requirements than conventional ASR architectures, making the TT model architecture suitable for performing speech recognition entirely on the user device 10 (e.g., without requiring communication with a remote server 60). The TT model 200 includes an audio encoder 210, a label encoder 220, and a collaborative network 230. The audio encoder 210 is generally similar to an acoustic model (AM) in a conventional ASR system and includes a neural network with a stack of strided convolutional layers 212 (FIG. 3) and transformer layers 214 (FIG. 3). For example, the audio encoder 210 receives a sequence of d-dimensional feature vectors (e.g., the acoustic frame 110 (FIG. 1)) x=(x1, x2, ..., x T ), where:

[0029]

number

[0030] At each time step, we obtain a high-level feature representation (also called the "encoder output"), denoted by ah1, ..., ah TEach transformer layer 214 of the audio encoder 210 may include a normalization layer, a masked multi-head attention layer with relative position encoding, a residual connection, a stacking / unstacking layer, and a feedforward layer. Similarly, the label encoder 220 may also include a neural network or lookup table embedding model of the transformer layer, which is modeled as a language model (LM) on the sequence of non-blank symbols output so far by the final softmax layer 240, i.e., y0,...,y ui-1 to produce a dense representation Ih u In implementations when the label encoder 220 includes a neural network of transformer layers, each transformer layer may include a normalization layer, a masked multi-head attention layer with relative position encoding, a residual connection, a feedforward layer, and a dropout layer. In these implementations, the label encoder 220 may include two transformer layers. In implementations when the label encoder 220 includes a lookup table embedding model with bigram label contexts, the embedding model is configured to learn a d-dimensional weight vector for each possible bigram label context, where d is the dimension of the output of the audio and label encoders 210, 220. In some examples, the total number of parameters in the embedding model is N 2 ×d, where N is the vocabulary size for labels. Here, the learned weight vector is then used as an embedding of the bigram label context in the TT model 200 to yield the fast label encoder 220 runtime.

[0031] Finally, using the TT model architecture, the representations generated by the audio and label encoders 210, 220 are encoded in a dense layer J u,t The collaboration network 230 then combines P(z u,t|x,t,y1,…,y u-1 ), which is a distribution over the next output symbol. In other words, the collaborative network 230 generates a probability distribution 242 over possible speech recognition hypotheses at each output step (e.g., time step), where a "possible speech recognition hypothesis" corresponds to a set of output labels (also called "phonetic units"), each of which represents a grapheme (e.g., a symbol / character) or word piece in a specified natural language. For example, when the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters of the English alphabet, and one label that specifies a space. Thus, the collaborative network 230 may output a set of values ​​that indicate the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector (e.g., a one-hot vector) and may indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), although the set of output labels is not so limited. For example, the set of output labels may include wordpieces and / or entire words in addition to or instead of graphemes. The output distribution of the collaborative network 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output z of the collaborative network 230 may be u,t may contain 100 different probability values, one for each output label. The probability distributions can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and / or words) in the beam search process to determine the transcription 120 (e.g., by the softmax layer 240).

[0032] The softmax layer 240 may utilize any technique for selecting the output label / symbol with the highest probability in the distribution as the next output symbol predicted by the TT model 200 in the corresponding output step. In this way, the TT model 200 does not make a conditional independence assumption, but rather the prediction of each symbol is conditioned not only on the acoustic effects but also on the sequence of labels output so far. Although the speech recognition model 200 is described as having a TT model architecture, the speech recognition model 200 may also include other types of transducer-based architectures, such as a conformal transducer (CT) model architecture or a recurrent neural network transducer (RNN-T) model architecture.

[0033] 3A and 3B show schematic diagrams of a control Siamese network 300 that performs a semi-supervised training process for training a speech recognition model 200 (FIG. 2). The control Siamese network includes a supervised sub-network training process 301 (FIG. 3A) and an unsupervised sub-network training process 302 (FIG. 3B). The supervised sub-network training process (i.e., supervised sub-network) 301 trains the speech recognition model 200 using a plurality of labeled audio samples 305 including sequences of acoustic frames 306 extracted from a spoken utterance 106 paired with corresponding transcriptions (i.e., labels) 308. The unsupervised sub-network training process (i.e., unsupervised sub-network) 302 trains the speech recognition model 200 using a plurality of unlabeled audio samples 303 including sequences of acoustic frames 304 extracted from a spoken utterance 106 without any paired transcriptions.

[0034] In some examples, the acoustic frames 306 used by the supervised sub-network (i.e., the supervised portion) 301 are the same as the acoustic frames 304 used by the unsupervised sub-network (i.e., the unsupervised portion) 302. That is, the supervised portion 301 and the unsupervised portion 302 may simultaneously use the same acoustic frames 304, 306 to train the speech recognition model 200. In other examples, the acoustic frames 306 used to train the supervised portion 301 are different from the acoustic frames 304 used to train the unsupervised portion 302. This scenario is particularly beneficial because unlabeled audio samples 303, which do not have any corresponding transcriptions, are easy to obtain and can be utilized to train the speech recognition model 200. Thus, the speech recognition model 200 may be trained with any combination of labeled audio samples 305 and / or unlabeled audio samples 303. In some examples, the sequence of acoustic frames 304, 306 extracted from the unlabeled audio samples 303 and the labeled audio samples 305 includes log-mel filter bank energies. A greater number of acoustic frames 304 may be used to train the unsupervised portion 302 than the number of acoustic frames 306 used to train the supervised portion 301. Optionally, a greater number of acoustic frames 306 may be used to train the supervised portion 301 than the number of acoustic frames 304 used to train the unsupervised portion 302. In some examples, the number of acoustic frames 306 used to train the supervised portion 301 and the number of acoustic frames 304 used to train the unsupervised portion 302 are the same.

[0035] The supervised and unsupervised portions 301 and 302 share the same audio encoder 210, which includes a stack of strided convolutional layers 212 and transformer layers 214 that are trained together using the same Adam optimizer and the same learning rate. When the speech recognition model 200 corresponds to a conformer transducer model architecture, the audio encoder 210 may include a conformer layer instead of a transformer layer.

[0036] 3A, the supervised portion 301 of the control Siamese network 300 trains the speech recognition model 200 using a plurality of labeled audio samples 305. The plurality of labeled audio samples 305 includes a sequence of acoustic frames 306 extracted from the labeled audio samples 305 and corresponding labels / transcriptions 308. The supervised portion 301 shares the same audio encoder 210 from the speech recognition model 200 as the unsupervised portion 302, in addition to the label encoder 220 and collaborative network 230 (not shown in FIG. 3A).

[0037] In some implementations, the supervised portion 301 includes a data augmentation module 365 that applies data augmentation to at least one acoustic frame 306 extracted from the labeled audio samples 305 to generate a sequence of augmented acoustic frames 306, 306A. The data augmentation module 365 of the supervised portion 301 may be the same (or a different) data augmentation module 360 ​​(FIG. 3B) of the unsupervised portion 302. In some examples, the data augmentation module 365 of the supervised portion 301 applies a different data augmentation technique than the data augmentation module of the unsupervised portion. Applying data augmentation to the acoustic frames 306 promotes acoustic diversity of the audio frames used to train the speech recognition model 200. In some examples, the data augmentation module 360 ​​includes a time correction component that manipulates the timing of the sequence of acoustic frames 306. The data augmentation module 360 ​​may additionally or alternatively include a time masking component that masks portions of the acoustic frames 306. Other techniques applied by the data augmentation module 360 ​​may include adding / injecting noise and / or adding reverberation to the labeled audio samples 305. One data augmentation technique includes using multi-style training (MTR) to inject various environmental noises into the labeled audio samples 305. Another data augmentation technique that the data augmentation module 360 ​​may apply in addition to or instead of MTR includes using Spectral Augmentation (SpecAugment) to make the acoustics of the labeled audio samples 305 more similar to the anti-acoustic effects of other labeled audio samples 305. Combined, MTR and SpecAugment may inject noise into the labeled audio samples 305, insert a random external noise source in front of the representation in time, tile it over and over, and filter the noise-injected labeled audio samples prior to training the speech recognition model 200.

[0038] The audio encoder 210 of the supervised portion 301 receives the extended sequence of acoustic frames 306A and generates an encoder output 341 for each extended acoustic frame 306A. The encoder output 341 may include a probability distribution of possible speech recognition hypotheses. In particular, the strided convolutional layer 212 receives the extended acoustic frames 306A and generates a corresponding output 211. The transformer layer 214 receives the output 211 from the strided convolutional layer 212 and generates the encoder output 341.

[0039] The label encoder 220 is a streaming transformer that does not pay attention to future labels 308. Thus, the label encoder 220 receives the labels 308 corresponding to the augmented acoustic frames 306A received by the audio encoder 210 and encodes the language embeddings 344 (i.e., the dense representation Ih u (FIG. 2)). The supervised part 301 includes a dense layer 346 that processes the language embeddings 344 from the label encoder 220 and the encoder output 341 (i.e., acoustic embeddings) from the audio encoder 210 to produce a corresponding speech recognition result 342 for each acoustic frame 306 (e.g., the augmented acoustic frame 306A) input to the speech recognition model 200 at the corresponding time step. The dense layer 346 includes a trainable bias vector 347 that performs a linear operation on the encoder output 341 and the language embeddings 344 to generate the speech recognition result 342. The loss module 351 of the supervised part 301 determines a supervised loss term 350 based on the output from the dense layer 346 for the obtained speech recognition result 342. That is, the loss module 351 compares the speech recognition result 342 with the label (e.g., the ground truth transcription) 308 to generate the supervised loss 350. The supervised loss term (e.g., RNN-T loss) 350 may be represented by: r t =linear(tanh(linear(a t )+linear(l t ))) (2)

[0040] In formula 2, r t represents a logit vector that specifies the probability that a grapheme contains a blank symbol, and a t represents the encoder output 341 from the audio encoder 210, and l t represents the language embeddings 344 from the label encoder 220, and linear represents a traditional dense layer 346 with a trainable bias vector 347.

[0041] The supervised portion 301 updates parameters of the speech recognition model 200 based on supervised loss terms 350 determined in each of a plurality of output steps for each labeled audio sample 305 in the plurality of labeled audio samples 305. In some implementations, the supervised portion 301 is configured to update parameters of the speech recognition model 200 based on the supervised loss terms 350 without relying on the unsupervised portion 302 having updated parameters of the audio encoder 210 of the speech recognition model 200.

[0042] 3B, the unsupervised portion 302 trains the speech recognition model 200 using a plurality of unlabeled audio samples 303 including a sequence of acoustic frames 304 extracted from a spoken utterance 106 that are not paired with any transcription. As shown in these examples, the unsupervised portion 302 of the control Siamese network 300 includes a target branch 310 and an augmented branch 320 that share the same audio encoder 210 of the speech recognition model 200 (FIG. 2) including a stack of strided convolutional layers 212 and transformer layers 214. The unsupervised portion 302 is configured to extract linguistic information by matching the output sequences of the transformer audio encoder 210 (i.e., the sequence of encoder outputs 312, 326) from the target branch 310 and the augmented branch 320.

[0043] The target branch 310 is configured to generate a target branch output 316 based on a sequence of acoustic frames 304 extracted from the unlabeled audio samples 303. The audio encoder 210 of the target branch 310 receives the sequence of acoustic frames 304 and generates, at each time step, a high-order feature representation 312. In particular, the strided convolutional layer 212 receives an acoustic frame 304 from the sequence of acoustic frames 304 and generates an output 213. The transformer layer 214 receives the output 213 from the strided convolutional layer 212 and generates a high-order feature representation (i.e., encoder output) 312 for the corresponding acoustic frame 304.

[0044] The transformer audio encoder 210 included in the target branch 310 and the extended branch 320 benefit from the positional embedding to capture the temporal dynamics of the sequence of audio frames 304. Therefore, to avoid the audio encoder 210 in the unsupervised part 302 generating encoder outputs 312, 326 with low contrastive loss based only on the positional embedding, it is necessary to modify the temporal characteristics of the high-order feature representation 312 output by the audio encoder 210 in the target branch 310. In other words, without modifying the temporal characteristics of the high-order feature representation 312 output by the audio encoder 210 in the target branch 310, all outputs of the unsupervised part 302 of the contrastive Siamese network 300 would "collapse" to a constant value. Therefore, the target branch 310 applies a gradient stopping operation 314 that modifies the temporal characteristics of the high-order feature representation 312 to generate the target branch output 316 for the corresponding audio frame 304. In some implementations, the gradient stopping operation 314 modifies temporal characteristics of the high-order feature representation 312 output by the audio encoder 210 by modifying the temporal characteristics of the high-order feature representation 312 to match temporal characteristics associated with the corresponding augmented acoustic frame 304, 304A input to the audio encoder 210 in the augmented branch 320 to generate a corresponding high-order feature representation 326. As will become apparent, the high-order feature representation 326 generated by the audio encoder 210 in the augmented branch 320 corresponds to an augmented high-order feature representation 326 (or an augmented encoder output) having modified temporal characteristics that is based on the augmented acoustic frame 304A input to the audio encoder 210.

[0045] The augmentation branch 320 of the unsupervised portion 302 includes a data augmentation module 360 ​​that applies data augmentation to each acoustic frame 304 extracted from the unlabeled audio samples 303. The augmentation module 360 ​​receives a sequence of acoustic frames 304 and generates a sequence of augmented acoustic frames 304A. Applying data augmentation to the acoustic frames 304 promotes acoustic diversity of the audio frames used to train the speech recognition model 200. In some examples, the data augmentation module 360 ​​includes a time correction component 322 that manipulates the timing of the sequence of acoustic frames 304. The data augmentation module 360 ​​may additionally or alternatively include a time masking component 324 that masks portions of the acoustic frames 304. Other techniques applied by the data augmentation module 360 ​​may include adding / injecting noise and / or adding reverberation of the labeled audio samples. One data augmentation technique includes using multi-style training (MTR) to inject diverse environmental noises into the unlabeled audio samples 303. Another data augmentation technique that the data augmentation module 360 ​​may apply in addition to or instead of MTR includes using spectral augmentation (SpecAugment) to make the acoustic effects of the augmented acoustic frames 304 more similar to the anti-acoustic effects of other unlabeled audio samples 303. In combination, MTR and SpecAugment may inject noise into the labeled audio samples 303, insert a random external noise source in front of the representation in time, tile it over and over, and filter the noise-injected unlabeled audio samples 303 prior to training the speech recognition model 200.

[0046] The audio encoder 210 of the augmented branch 320 receives the augmented sequence of acoustic frames 304A from the data augmentation module 360 ​​and generates a high-order feature representation 326 for the corresponding augmented acoustic frame 304A. In particular, the strided convolutional layer 212 receives the augmented acoustic frame from the sequence of augmented acoustic frames 304A and generates an output 215. The transformer layer 214 receives the output 215 from the strided convolutional layer 212 and generates a high-order feature representation (i.e., an augmented encoder output) 326 for the corresponding augmented acoustic frame 304A. Subsequently, the predictive transformer network 328 receives the high-order feature representation 326 and generates a prediction 329 of the target branch output 316 generated by the target branch 310 at the corresponding time step. That is, the predictive transformer 328 may use the contrastive loss 330 to learn to generate a prediction 329 as an output from the augmented branch 320 that matches the target branch output 316 at each corresponding time step.

[0047] The unsupervised portion 302 determines an unsupervised loss term 330 based on the target branch output 316 produced by the target branch 310 and a prediction 329 of the target branch output 316 produced by the expanded branch 320. In some examples, the unsupervised loss term 330 includes a contrast loss term represented by:

[0048]

number

[0049] In Equation 1, M includes a set of masked frame indexes, K includes a set of interference indexes, and h t is the encoder output, and c tis the convolutional neural network output. In other examples, the supervised loss term includes a reconstruction loss term L1 or a cosine distance loss term. The unsupervised part 302 updates parameters of the audio encoder 210 based on the unsupervised loss term 330 determined at each of the multiple time steps. Notably, using the audio frame 304, the target branch 310 generates an expected representation (i.e., the target branch output 316) based on the current state of the audio encoder 210, and the extended branch 320 aims to match the expected representation using the extended audio frame 304A.

[0050] 4 is a flowchart of an exemplary sequence of operations for a computer-implemented method 400 of training a speech recognition model using a control Siamese network. In operation 402, the method 400 includes receiving a plurality of unlabeled audio samples 303 corresponding to a spoken utterance 106 that is not paired with any corresponding transcription (i.e., label 308). The target branch 310 of the control Siamese network 300 performs operations 404 and 406. In operation 404, the method 400 includes using an audio encoder 210 of the speech recognition model 200 to generate a sequence of encoder outputs 312 for the plurality of unlabeled audio samples 303. In operation 406, the method 400 includes modifying a time characteristic of the encoder output 312 to generate a sequence of target branch outputs 316 using a gradient stopping operation 314.

[0051] The augmentation branch 320 of the control Siamese network 300 performs operations 408-412. In operation 408, the method 400 includes performing augmentation on the unlabeled audio samples 303 using the augmentation module 360. In operation 410, the method 400 includes generating a sequence of augmented encoder outputs 326 for the augmented unlabeled audio samples 304A using the audio encoder 210 of the speech recognition model 200. In operation 412, the method includes generating a prediction 329 of the sequence of target branch outputs 316 generated in the target branch 310.

[0052] At operation 414, the method includes determining an unsupervised loss term 330 based on the target branch outputs 316 generated in the target branch 310 and a prediction 329 of the sequence of the target branch outputs 316 generated in the extension branch 320. At operation 416, the method 400 includes updating parameters of the audio encoder 210 based on the unsupervised loss 330.

[0053] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0054] Non-transitory memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transitory memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0055] 5 is a schematic diagram of an exemplary computing device 500 that can be used to implement the systems and methods described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the invention described and / or claimed herein.

[0056] Computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 that connects to memory 520 and high-speed expansion port 550, and a low-speed interface / controller 560 that connects to low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 may be interconnected using various buses and mounted on a common motherboard or otherwise as appropriate. Processor 510 can process instructions for execution within computing device 500, including instructions stored in memory 520 or on storage device 530 for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memories, as appropriate. Also, multiple computing devices 500 may be connected, each providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0057] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0058] The storage device 530 is capable of providing mass storage to the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0059] The high-speed controller 540 manages bandwidth-intensive operations for the computing device 500, and the low-speed controller 560 manages less bandwidth-intensive operations. Such an allocation of roles is merely exemplary. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and to a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), but may also be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or a networking device, such as a switch or router, for example, through a network adapter.

[0060] The computing device 500 may be implemented in a number of different forms, as shown in the figure, such as as a standard server 500a or in a group of multiple such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0061] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or translatable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from and to transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0062] These computer programs (also known as programs, software, software applications or codes) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0063] The processes and logic flows described herein can be implemented by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special purpose logic circuitry, such as FPGAs (field programmable gate arrays) and ASICs (application specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to a mass storage device to receive data from, or transfer data to, the mass storage device, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0064] To enable interaction with a user, one or more aspects of the present disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touch screen, for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to enable interaction with the user, e.g., feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0065] Although several implementations have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0066] 10. User Devices 12 Data Processing Hardware 14 Memory Hardware 16 Audio capture devices, audio output devices, audio systems 16a Audio Capture Device 16b Audio Output Device 18 Digital Assistant Interface 19 responses 40 Network 50 Digital Assistant Applications, Applications 60 Remote Computing Devices, Remote Servers 100 Audio Environment 104 users 106 Spoken utterances, utterances, input utterances 107 User Interface Generator 108 Audio Subsystem 110 Input audio frame, audio frame, audio data 118 Automated Speech Recognition (ASR) Systems 120 Transcription, partial speech recognition results, final speech recognition results 120a Partial speech recognition result 120b Final voice recognition result 200 Speech recognition model, Transformer Transducer (TT) model 210 Audio Encoder, Transformer Audio Encoder 211 Corresponding output, output 212 strided convolutional layer 214 Transformer Layer 220 Label Encoder 230 Collaborative Network 240 Softmax Layers 242 Probability Distribution 300 Control Siamese Network 301 Supervised Sub-network, Supervised Part 302 Unsupervised Subnetwork, Unsupervised Part 303 Unlabeled Audio Samples 304 Acoustic Frame, Corresponding Acoustic Frame, Extended Acoustic Frame, Audio Frame 304A Extended Acoustic Frame, Acoustic Frame, Extended Audio Frame, Extended Unlabeled Audio Samples 305 Labeled Audio Samples 306 Acoustic Frame, Extended Acoustic Frame 306A Extended Acoustic Frame, Acoustic Frame 308 Corresponding transcription, corresponding label / transcription, future label, corresponding label, label 310 Target Branch 312 Encoder output, high-level feature representation 314 Gradient stop action 316 Target Branch Output 320 Extended Branch, Extended Branch 322 Time Correction Components 324 Temporal Masking Components 326 Encoder output, high-level feature representation 328 Predictive Transformer Network, Predictive Transformer 329 Predictions 330 Contrast loss, Unsupervised loss term, Unsupervised loss 341 Encoder Output 342 Voice Recognition Results 344 Language Embedding 346 Dense Layer 347 Trainable Bias Vectors 350 Supervised loss terms, Supervised loss 351 Loss Module 360 Data Expansion Module, Expansion Module 365 Data Expansion Module 400 Computer-implemented method, method 500 computing devices 500a Standard Server 500b Laptop Computer 500c Rack Server System 510 Processor, components 520 Memory, components, non-transient memory 530 Storage devices, components 540 High Speed ​​Interface / Controller, Components 550 High Speed ​​Expansion Port, Components 560 Low-speed interface / controller, components 570 Slow Bus 580 Display 590 Low Speed ​​Expansion Port L1 reconstruction loss term

Claims

1. A control Siamese network (300) for training a speech recognition model (200), the control Siamese network comprising an unsupervised sub-network (302) trained with a plurality of unlabeled audio samples (303) corresponding to a spoken utterance (106) that has not been paired with a corresponding transcription (308); The unsupervised sub-network (302) comprises: receiving, as an input to an audio encoder (210) of the speech recognition model (200), a sequence of acoustic frames (304) extracted from the unlabeled audio samples (303); generating, at each of a plurality of time steps, a target branch output (316) for a corresponding audio frame (304) in the sequence of audio frames (304) input to the audio encoder (210) at the corresponding time step; A target branch (310) configured to: performing an extension on the sequence of acoustic frames (304) extracted from the unlabeled audio samples (303) to generate a sequence of extended acoustic frames (304A); generating, at each of the plurality of time steps, as output from the audio encoder (210), a high-level feature representation (326) for a corresponding enhanced audio frame (304A) in the sequence of enhanced audio frames (304A); generating, at each of the plurality of time steps, a prediction (329) of the target branch output (316) generated by the target branch (310) at the corresponding time step using the high-level feature representation (326) output from the audio encoder (210) at the corresponding time step; an expanded branch (320) configured to Equipped with The unsupervised sub-network (302) comprises: determining, at each of the plurality of time steps, an unsupervised loss term (330) based on the target branch output (316) produced by the target branch (310) at the corresponding time step and the prediction (329) of the target branch (310) output produced by the expanded branch (320) at the corresponding time step; updating parameters of the audio encoder (210) based on the unsupervised loss term (330) determined at each of the plurality of time steps; A control Siamese network (300) configured to perform:

2. The contrastive Siamese network of claim 1 , wherein the unsupervised loss term comprises a contrastive loss term.

3. 3. The control Siamese network of claim 1 or 2, wherein the augmentation performed on the sequence of acoustic frames comprises time modification and masking.

4. The target branch (310) is and further configured to generate, at each of a plurality of time steps, as output from the audio encoder (210), a high-level feature representation (312) for the corresponding audio frame (304) in the sequence of audio frames (304) input to the audio encoder (210) at the corresponding time step; The target branch (310) is 4. The contrast Siamese network (300) of claim 1, configured to generate the target branch output (316) for the corresponding acoustic frame (304) by modifying temporal characteristics of the high-level feature representation (312).

5. 5. The contrast Siamese network of claim 4, wherein modifying the temporal characteristics of the high-level feature representation (312) comprises modifying, at each of the plurality of time steps, the temporal characteristics of the high-level feature representation (312) generated as output from the audio encoder (210) for the corresponding acoustic frame (304) to match temporal characteristics associated with the high-level feature representation (312) generated as output from the audio encoder (210) for the corresponding extended acoustic frame (304A) at the corresponding time step.

6. The expanded branch (320) performs, at each of the plurality of time steps: receiving as input the high-level feature representation (326) output from the audio encoder (210) at the corresponding time step; generating, as an output, the prediction (329) of the target branch output (316) produced by the target branch (310) at the corresponding time step; 6. The control Siamese network (300) of claim 1, comprising a transformer layer prediction network (328) configured to:

7. a supervised sub-network (301) trained on a plurality of labeled audio samples (305) corresponding to a spoken utterance (106) paired with a corresponding transcription (308); The supervised sub-network (301) comprises: In each of a plurality of output steps for each labeled audio sample (305), using the speech recognition model (200) to generate corresponding speech recognition results (342) for the labeled audio samples (305); determining a supervised loss term (350) based on the corresponding speech recognition result (342) for the labeled audio sample (305) and the corresponding transcription (308) of the labeled audio sample (305); updating parameters of the speech recognition model (200) based on the supervised loss terms (350) determined in each of the plurality of output steps for each labeled audio sample (305) in the plurality of labeled audio samples (305); The control Siamese network (300) of any one of claims 1 to 6, configured to:

8. 8. The control Siamese network of claim 7, wherein the corresponding speech recognition result (342) generated for the labeled audio sample (305) using the speech recognition model (200) comprises a probability distribution over possible speech recognition hypotheses for the labeled audio sample (305) at a corresponding output step.

9. 9. The control Siamese network (300) of claim 7 or 8, wherein the supervised sub-network (301) is configured to update the parameters of the speech recognition model (200) based on the supervised loss terms (350) without relying on the unsupervised network (302) having updated the parameters of the audio encoder (210) of the speech recognition model (200).

10. 10. The control Siamese network of claim 7, wherein the supervised sub-network is further configured to apply data augmentation to at least one of the labeled audio samples among the plurality of labeled audio samples input to the speech recognition model.

11. 11. The contrast Siamese network of claim 10, wherein the applied data augmentation comprises at least one of adding noise, adding reverberation, or manipulating timing.

12. The trained speech recognition model (200) includes a Transformer Transducer (TT) model (200), The TT model (200) is receiving as input a sequence of acoustic frames (110) extracted from audio data characterizing a spoken utterance (106); At each of a plurality of time steps, a high-order feature representation (ah) for a corresponding acoustic frame (110) in the sequence of acoustic frames (110) is calculated. t ) and The audio encoder (210) configured to: The non-blank symbols (y ui-1 ) as input; At each of the plurality of time steps, a dense representation (ih u ) and A label encoder (220) configured to: The high-level feature representation (ah) generated by the audio encoder (210) at each of the plurality of time steps t ), and the dense representation (ih u ) as input; generating, at each of the plurality of time steps, a probability distribution 242 over possible speech recognition hypotheses at the corresponding time step; A collaborative network (230) configured to Equipped with 12. The contrast Siamese network of claim 1, wherein the audio encoder comprises a neural network having a stack of strided convolutional layers and transformer layers.

13. A computer-implemented method (400) that, when executed on data processing hardware (12), causes the data processing hardware (12) to perform operations, the operations including: receiving a plurality of unlabeled audio samples (303) corresponding to a spoken utterance (306) that is not paired with a corresponding transcription (308); In the target branch (310) of the control Siamese network (300), generating a sequence of encoder outputs (312) for the plurality of unlabeled audio samples (303) using an audio encoder (210) of the speech recognition model (200); and modifying a time characteristic of the encoder output (312) to generate a sequence of target branch outputs (316); In the expansion branch (320) of the control Siamese network (300), performing an augmentation on the unlabeled audio samples (303); generating a sequence of augmented encoder outputs (326) for the augmented unlabeled audio samples (304A) using the audio encoder (210) of the speech recognition model (200); and generating a prediction (329) of the sequence of target branch outputs (316) generated at the target branch (310) using a prediction network (328) configured to receive the sequence of the extended encoder outputs (326); determining an unsupervised loss term (330) based on the target branch output (316) generated at the target branch (310) and the prediction (329) of a sequence of the target branch outputs (316) generated at the extension branch (320); updating parameters of the audio encoder (210) based on the unsupervised loss term (330); A computer-implemented method (400).

14. 14. The computer-implemented method of claim 13, wherein the unsupervised loss terms include a contrastive loss term.

15. 15. The computer-implemented method (400) of claim 13 or 14, wherein performing an augmentation on the unlabeled audio samples (303) comprises performing time correction and masking on the unlabeled audio samples (303).

16. 16. The computer-implemented method (400) of any one of claims 13 to 15, wherein the operations further comprise generating a high-order feature representation for the plurality of unlabeled audio samples (303) as output from the audio encoder (210).

17. 17. The computer-implemented method (400) of claim 16, wherein modifying the temporal characteristics of the encoder output (312) to generate the sequence of target branch outputs (316) comprises modifying the temporal characteristics of the encoder output (312) generated as output from the audio encoder (210) to match temporal characteristics associated with the sequence of extended encoder outputs (326) from the audio encoder (210).

18. The operation includes: receiving the sequence of the augmented encoder outputs (326) as input to a prediction network (328) of a transformer layer of the augmented branch (320); generating, as an output from the prediction network (328) of a transformer layer of the expanded branch (320), the prediction (329) of the sequence of the target branch outputs (316) generated at the target branch (310); 18. The computer-implemented method (400) of any one of claims 13 to 17, further comprising:

19. The operation includes: receiving a plurality of labeled audio samples (305) corresponding to a spoken utterance (306) paired with a corresponding transcription (308); generating speech recognition results (342) for the labeled audio samples (305) using the speech recognition model (200); determining a supervised loss term (350) based on the speech recognition results (342) for the labeled audio samples (305) and the corresponding transcriptions (308) of the labeled audio samples (305); updating parameters of the speech recognition model (200) based on the supervised loss term (350); 19. The computer-implemented method (400) of any one of claims 13 to 18, further comprising:

20. 20. The computer-implemented method of claim 19, wherein the speech recognition results (342) generated for the labeled audio samples (305) using the speech recognition model (200) include a probability distribution over possible speech recognition hypotheses for the labeled audio samples (305) at a corresponding output step.

21. 21. The computer-implemented method (400) of claim 19 or 20, wherein the operations further include updating parameters of the speech recognition model (200) based on the supervised loss (350) terms independent of updating parameters of the audio encoder (210) based on the unsupervised loss terms (330).

22. 22. The computer-implemented method (400) of any one of claims 19 to 21, wherein the operations further comprise applying data augmentation to at least one of the labeled audio samples (305).

23. 23. The computer-implemented method (400) of claim 22, wherein applying data augmentation comprises at least one of adding noise, adding reverberation, or manipulating timing.

24. The speech recognition model (200) includes a transformer transducer (TT) model (200); The operation includes: receiving, as input to the audio encoder (210) of the TT model (200), a number of unlabeled audio samples (303) corresponding to a spoken utterance (306) that is not paired with a corresponding transcription (308); The audio encoder (210) extracts an acoustic frame (ah) from the audio data characterizing a spoken utterance (106) at each of a plurality of time steps. t ) and The non-blank symbols (y ui-1 ) and The label encoder (220) generates a dense representation (ih u ), and The high-level feature representations (ah) generated by the audio encoder (210) at each of the plurality of time steps and the dense representations (ah) generated by the label encoder (220) at each of the plurality of time steps are used as inputs to the collaborative network (230) of the TT model (200). t ) and generating, by the collaborative network (230), at each of the plurality of time steps, a probability distribution (242) over possible speech recognition hypotheses at the corresponding time step; Further comprising:

24. The computer-implemented method (400) of any one of claims 13 to 23, wherein the audio encoder (210) comprises a neural network having a stack of strided convolutional layers and transformer layers.

Citation Information

Patent Citations

  • Consistency prediction on streaming sequence models

    WO2021178108A1

  • Transformer transducer: one model unifying streaming and non-streaming speech recognition

    US11741947B2