Enhancing training of sequence transduction models using lexical level loss

Abstract

A method (500) includes, for each training sample (410) of a plurality of training samples: processing a corresponding training input feature (415) using a sequence transduction model (200) to obtain one or more output lexical sequence hypotheses (432), the one or more output lexical sequence hypotheses each comprising one or more predicted generic lexical elements (204); and for each hypothesis, determining a lexical-level loss (462) based on: a number of special lexical inserts each associated with a corresponding predicted special lexical that appears in the hypothesis but does not appear in a corresponding sequence of true-value output lexical, and determining the lexical-level loss (462) based on the number of special lexical inserts each associated with a corresponding predicted special lexical that appears in the hypothesis but does not appear in the corresponding sequence of true-value output lexical; and a number of special lemma deletions, each associated with a corresponding true value special lemma of the set of true value special lemma that does not appear in the hypothesis. The method further includes training the sequence transduction model to minimize an additive error rate based on lexical-level losses determined for the plurality of training samples.

Description

Technical Field

[0001] This disclosure relates to the training of sequence transduction models. Background Art

[0002] Sequence transduction models are constructed and trained to transform an input sequence into an output sequence. Example sequence transduction models include, but are not limited to: speech recognition models for transforming an input sequence of audio features into a transcription comprising a sequence of words or sub-word units, character recognition models for transforming a sequence of handwritten characters into a sequence of word or sub-word text segments, and machine translation models for transforming a first sequence of words in a first language into a second sequence of words in a second language. Summary of the Invention

[0003] One aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for enhancing the training of a sequence transduction model using a token-level loss. The operations include receiving a plurality of training samples, each of the plurality of training samples comprising a corresponding training input feature sequence paired with a corresponding ground-truth output token sequence, the ground-truth output token sequence comprising a set of ground-truth general tokens and a set of ground-truth special tokens. For each training sample of the plurality of training samples, the operations include: processing the corresponding training input feature sequence using the sequence transduction model to obtain one or more output token sequence hypotheses, each output token sequence hypothesis comprising one or more predicted general tokens; and, for each corresponding output token sequence hypothesis obtained for the training sample, determining a per-sample token-level loss based on: the number of special token insertions, each of the special token insertions being associated with a corresponding predicted special token that appears in the corresponding output token sequence hypothesis but not in the corresponding ground-truth output token sequence; and the number of special token deletions, each of the special token deletions being associated with a corresponding ground-truth special token from the set of ground-truth special tokens that does not appear in the corresponding output token sequence hypothesis. The operations further include training the sequence transduction model to minimize an additive error rate based on the per-sample token-level losses determined for the plurality of training samples.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation further includes: determining that the per-sample token-level loss is further based on the total number of ground-truth output tokens in the corresponding ground-truth output token sequence and the corresponding number of predicted general token errors relative to the set of ground-truth general tokens in the corresponding ground-truth token sequence; and when determining the per-sample token-level loss, the number of special token insertions and the number of special token deletions are each weighted higher than the corresponding number of predicted general token errors to force the sequence transduction model to reduce the special token insertion and deletion rates during training. In some examples, the operation further includes, for each of a plurality of training samples: using the sequence transduction model to process the corresponding training input feature sequence to predict a probability distribution over possible output tokens; and training the sequence transduction model based on the negative log of the probability distribution of the corresponding ground-truth output token sequence conditioned on the corresponding training input feature sequence. Here, training the sequence transduction model based on the negative log of the probability distribution may include: initially training the sequence transduction model based on the negative log of the probability distribution to initialize the sequence transduction model; and training the sequence transduction model to minimize the additive error rate based on the per-sample token-level loss may include: fine-tuning the initialized sequence transduction model according to minimizing the additive error rate based on the per-sample token-level loss.

[0005] In some examples, processing a corresponding training input feature sequence to obtain one or more output token sequence hypotheses includes: processing the corresponding training input feature sequence to obtain an N-best list of output token sequence hypotheses, where each corresponding output token sequence hypothesis in the N-best list has a corresponding probability score assigned by a sequence transduction model; and determining a per-sample token-level loss further based on the corresponding probability scores of the corresponding output token sequence hypotheses. In some implementations, the training input feature sequence includes a sequence of input audio frames representing an utterance including a specific key phrase; the set of ground-truth general tokens in each corresponding ground-truth output token sequence includes a set of word or sub-word unit tokens forming the ground-truth transcription of the utterance represented by the sequence of input audio frames; the set of ground-truth special tokens in each corresponding ground-truth output token sequence includes at least one ground-truth keyword token that indicates the corresponding position in the ground-truth transcription immediately following where the specific key phrase appears in the ground-truth transcription; the one or more predicted general tokens of each corresponding output token sequence hypothesis in the one or more output token sequence hypotheses include a predicted sequence of word or sub-word tokens forming a corresponding candidate transcription of the utterance; and each corresponding predicted special token that appears in a corresponding output token sequence hypothesis but does not appear in the corresponding ground-truth output token sequence includes a predicted key phrase token that indicates the corresponding position in the corresponding candidate transcription immediately following where the sequence transduction model predicts detection of the specific key phrase.

[0006] In some implementations, the training input feature sequence includes a sequence of input audio frames representing multiple utterances spoken by at least two different speakers; the set of ground-truth general tokens in each corresponding ground-truth output token sequence includes a set of word or sub-word unit tokens forming the ground-truth transcription of the multiple spoken utterances represented by the sequence of input audio frames; the set of ground-truth special tokens in each corresponding ground-truth output token sequence includes a set of one or more ground-truth speaker change tokens, where each of the one or more ground-truth speaker change tokens indicates the corresponding position in the ground-truth transcription of the multiple utterances where a speaker change occurs; the one or more predicted general tokens of each corresponding output token sequence hypothesis in the one or more output token sequence hypotheses include a predicted sequence of word or sub-word unit tokens forming a corresponding candidate transcription of the multiple utterances; and each corresponding predicted special token that appears in a corresponding output token sequence hypothesis but does not appear in the corresponding ground-truth output token sequence includes a predicted speaker change token that indicates the corresponding position in the corresponding candidate transcription where the sequence transduction model detects the corresponding speaker change event.

[0007] In some examples, the operation further includes, for each of a plurality of training samples: determining a customized Levenshtein distance between each corresponding output token sequence hypothesis obtained for a corresponding training input feature sequence and the corresponding ground truth output token sequence; and based on the customized Levenshtein distance: identifying the number of special token insertions for each corresponding output token sequence hypothesis; and identifying the number of special token deletions for each corresponding output token sequence hypothesis. Here, the customized Levenshtein distance determined between each corresponding output token sequence hypothesis and the corresponding ground truth output token sequence can prevent the sequence transduction model from allowing substitutions between special tokens and general tokens during training of the sequence transduction model.

[0008] In some implementations, the sequence transduction model includes a recurrent neural network transducer (RNN-T) model architecture. Additionally, the sequence transduction model can include at least one of a speech recognition model, a character recognition model, an end-pointing model, a speaker turn detection model, or a machine translation model.

[0009] Another aspect of the present disclosure provides a system that includes data processing hardware and memory hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a plurality of training samples, each of the plurality of training samples including a corresponding training input feature sequence paired with a corresponding ground truth output token sequence, the ground truth output token sequence including a set of ground truth general tokens and a set of ground truth special tokens. For each of the plurality of training samples, the operations include: using a sequence transduction model to process the corresponding training input feature sequence to obtain one or more output token sequence hypotheses, each output token sequence hypothesis including one or more predicted general tokens; and, for each corresponding output token sequence hypothesis obtained for a training sample, determining a per-sample token-level loss based on: the number of special token insertions, each of the special token insertions being associated with a corresponding predicted special token that appears in the corresponding output token sequence hypothesis but not in the corresponding ground truth output token sequence; and the number of special token deletions, each of the special token deletions being associated with a corresponding ground truth special token from the set of ground truth special tokens that does not appear in the corresponding output token sequence hypothesis. The operations further include training the sequence transduction model to minimize an additive error rate based on the per-sample token-level loss determined for one or more output token sequence hypotheses obtained for each of the plurality of training samples.

[0010] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation further includes: determining that the per-sample token-level loss is further based on the total number of ground-truth output tokens in the corresponding ground-truth output token sequence and the corresponding number of predicted generic token errors relative to the set of ground-truth generic tokens in the corresponding ground-truth token sequence; and when determining the per-sample token-level loss, the number of special token insertions and the number of special token deletions are each weighted higher than the corresponding number of predicted generic token errors to force the sequence transduction model to reduce the special token insertion and deletion rates during training. In some examples, the operation further includes for each of a plurality of training samples: using the sequence transduction model to process the corresponding training input feature sequence to predict a probability distribution over possible output tokens; and training the sequence transduction model based on the negative logarithm of the probability distribution of the corresponding ground-truth output token sequence conditioned on the corresponding training input feature sequence. Here, training the sequence transduction model based on the negative logarithm of the probability distribution may include: initially training the sequence transduction model based on the negative logarithm of the probability distribution to initialize the sequence transduction model; and training the sequence transduction model to minimize the additive error rate based on the per-sample token-level loss may include: fine-tuning the initialized sequence transduction model according to minimizing the additive error rate based on the per-sample token-level loss.

[0011] In some examples, processing a corresponding training input feature sequence to obtain one or more output token sequence hypotheses includes: processing the corresponding training input feature sequence to obtain an N-best list of output token sequence hypotheses, where each corresponding output token sequence hypothesis in the N-best list has a corresponding probability score assigned by a sequence transduction model; and determining a per-sample token-level loss for each corresponding output token sequence hypothesis in the N-best list further based on the corresponding probability score of the corresponding output token sequence hypothesis. In some implementations, the training input feature sequence includes a sequence of input audio frames representing an utterance including a specific key phrase; the set of ground-truth general tokens in each corresponding ground-truth output token sequence includes a set of word or sub-word unit tokens forming the ground-truth transcription of the utterance represented by the sequence of input audio frames; the set of ground-truth special tokens in each corresponding ground-truth output token sequence includes at least one ground-truth keyword token, the at least one ground-truth keyword token indicating a corresponding position in the ground-truth transcription immediately following the occurrence of the specific key phrase in the ground-truth transcription; one or more predicted general tokens in each corresponding output token sequence hypothesis of the one or more output token sequence hypotheses include a predicted sequence of word or sub-word tokens forming a corresponding candidate transcription of the utterance; and each corresponding predicted special token that appears in a corresponding output token sequence hypothesis but does not appear in the corresponding ground-truth output token sequence includes a predicted key phrase token, the predicted key phrase token indicating a corresponding position in the corresponding candidate transcription immediately following the detection of the specific key phrase by the sequence transduction model prediction.

[0012] In some implementations, the training input feature sequence includes a sequence of input audio frames representing multiple utterances spoken by at least two different speakers; the set of ground-truth general tokens in each corresponding ground-truth output token sequence includes a set of word or sub-word unit tokens forming the ground-truth transcription of the multiple spoken utterances represented by the sequence of input audio frames; the set of ground-truth special tokens in each corresponding ground-truth output token sequence includes a set of one or more ground-truth speaker change tokens, the one or more ground-truth speaker change tokens each indicating a corresponding position in the ground-truth transcription of the multiple utterances where a speaker change occurs; one or more predicted general tokens in each corresponding output token sequence hypothesis of the one or more output token sequence hypotheses include a predicted sequence of word or sub-word unit tokens forming a corresponding candidate transcription of the multiple utterances; and each corresponding predicted special token that appears in a corresponding output token sequence hypothesis but does not appear in the corresponding ground-truth output token sequence includes a predicted speaker change token, the predicted speaker change token indicating a corresponding position in the corresponding candidate transcription where the sequence transduction model detects a corresponding speaker change event.

[0013] In some examples, the operation further includes, for each of a plurality of training samples: determining a customized Levenshtein distance between each corresponding output token sequence hypothesis obtained for a corresponding training input feature sequence and the corresponding ground-truth output token sequence; and based on the customized Levenshtein distance: identifying the number of special token insertions for each corresponding output token sequence hypothesis; and identifying the number of special token deletions for each corresponding output token sequence hypothesis. Here, the customized Levenshtein distance determined between each corresponding output token sequence hypothesis and the corresponding ground-truth output token sequence can prevent a sequence transduction model from allowing substitutions between special tokens and general tokens during training of the sequence transduction model.

[0014] In some implementations, the sequence transduction model includes a recurrent neural network transducer (RNN-T) model architecture. Additionally, the sequence transduction model can include at least one of a speech recognition model, a character recognition model, an end-pointing model, a speaker turn detection model, or a machine translation model.

[0015] Details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of an example system including a sequence transduction model for performing speech recognition.

[0017] Figure 2 is Figure 1 a schematic diagram of an example recurrent neural network transducer (RNN-T) model of the sequence transduction model.

[0018] Figure 3 is Figure 3 a schematic diagram of an example bundled and simplified prediction network of the RNN-T model.

[0019] Figure 4 is a schematic diagram of an exemplary two-stage process for enhancing training of a sequence transduction model using a token-level loss.

[0020] Figure 5 is a flowchart of an example arrangement of operations of a computer-implemented method for enhancing training of a sequence transduction model using a token-level loss.

[0021] Figure 6 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0022] In the various figures, the same reference symbols indicate the same elements. Detailed Description

[0023] In addition to transforming between an input sequence and an output sequence, sequence transduction models have also been constructed to detect special input conditions and generate special outputs (e.g., special output tokens or other types of indicators) when the special input conditions are detected. That is, a sequence transduction model can be constructed and trained to process an input data sequence to generate a predicted output sequence that includes a special output (e.g., a special output token or other type of indicator) in addition to other normal / general predicted outputs when the sequence transduction model detects a corresponding special input condition in the input data. Here, the predicted general output is associated with an input that is not associated with the special input condition. For example, an automatic speech recognition (ASR) model can be constructed and trained to detect special input conditions in an input audio feature sequence (i.e., the captured input audio data) and include or embed a special output token representing the detected special input condition in the output transcription (i.e., the output sequence or words or sub-word units). Here, the special input conditions include, but are not limited to, speaker changes (i.e., changes in the person speaking), verbal key phrases (e.g., verbal hot words such as "Hey Assistant", or verbal warm words such as "Volume Up"). In an example, the ASR model generates a predicted transcription "Hey Assistant" from the captured input audio data representing the verbal utterance "Hey Assistant, what time is it". <hw>,what time is it(Hey, assistant <hw>, What time is it)”. Here, the transcription includes the special output token that immediately follows "Hey Assistant" to indicate to the ASR model that "Hey Assistant" was recognized as an oral hotword. <hw>", as well as general tokens representing other spoken words. Other example sequence transduction models include, but are not limited to, character recognition models and machine translation models. Here, example special input conditions include, but are not limited to, potentially offensive words, special characters, equations, and proper names.

[0024] However, traditional methods of training sequence transduction models for detecting special input conditions have problems, at least because special input conditions in the training data are sparse relative to general inputs. For example, the occurrence rate of speaker changes, key phrases, etc. in spoken discourse training samples may be several orders of magnitude lower than that of non-special spoken words. Additionally, in some examples, the only possible special tokens (e.g., only " <st>"and" <hw”)比可能的通用词元(例如,英语语言中的所有词)少得多。传统上,序列转导模型已使用针对整个输出序列(即包括通用输出和不频繁的特殊输出)计算的负对数似然损失进行训练,并且训练数据中特殊输入条件的稀疏性使在训练期间不强调特殊输出,这劣化了对特殊输入条件的检测准确性。

[0025] The implementation of this article aims to use word-level loss to enhance the training of sequence transduction models to detect special input conditions. In a specific implementation disclosed in this article, the training process initially trains the sequence transduction model in a first training pass using training data including a set of training samples / multiple training samples, each of which includes an input data sequence paired with a corresponding true value output sequence. Here, the true value output includes, in addition to the general output, special true value outputs for corresponding special input conditions in the input data. The example training sample includes captured input audio data representing the spoken utterance "Hey Assistant, what is today's weather" and the corresponding true value transcription "Hey Assistant, what is today's weather?". <hw>,what is today’s weather(Hey, assistant <hw>, "What's the weather like today)". Here, the ground truth transcription includes the special output token " that immediately follows "Hey Assistant" to indicate that the speech recognition model recognized "Hey Assistant" as an oral hotword / keyword phrase <hw>", and a general token for other oral words or sub-word units representing the utterance. Initial training in the first training pass can be performed using a conventional loss function (e.g., minimizing negative log-likelihood) based on the difference between the predicted output of the sequence transduction model and the corresponding ground truth output, both the predicted output and the corresponding ground truth output including special outputs for special input conditions. It is noted that during the first training pass, the loss is calculated for the entire predicted output sequence for each specific training sample (i.e., including the general output and the infrequent special outputs).

[0026] Then, the training process retrains or fine-tunes the training of the sequence transduction model in a second training pass using the same training data. For each training sample during the second training pass, the training process performs beam search to identify the N best predicted output sequences (i.e., output token sequence hypotheses) based on the input data of the training sample. The training process performs retraining in the second training pass using a loss function that is a weighted sum of a token-level loss function and a conventional loss function. The conventional loss function can be the same as the loss function used during the initial training in the first training pass. In some implementations, the token-level loss function includes a minimum additive error rate, such as minimum word error rate (MWER) or minimum Bayesian risk based on word-level editing (EMBR), and the loss associated with special tokens is given a greater loss value than the loss associated with conventional tokens.

[0027] Reference Figure 1 , the example system 100 includes a user device 110 configured to capture input audio data 122 (i.e., input audio feature sequence 122) and communicate with a cloud computing environment 140 via a network 130. The input audio data represents one or more utterances 120, 120a - 120n spoken by one or more speakers (e.g., a user or a person) 10, 10a - 10n. In some implementations, the user device 110 and / or the cloud computing environment 140 execute a sequence transduction model 200 configured to receive input data / features and generate a predicted output sequence (e.g., an output token sequence). In the illustrated example, the sequence transduction model 200 includes an automatic speech recognition (ASR) model 200 configured to generate one or more predicted transcripts 202, 202a - 202n for the spoken utterances 120. Each transcript 202 includes an output token sequence that includes general tokens 204, 204a - 204n corresponding to general spoken words in the spoken utterance 120. More specifically, the sequence of general tokens 204 includes words or sub - word tokens that form the transcript 202 of the utterance 120. The ASR model 200 can include any transducer - based architecture, including but not limited to transformer - transducer (T - T), recurrent neural network transducer (RNN - T), and / or conformer - transducer (C - T). The ASR model 200 is also constructed and trained to detect special input conditions in the spoken utterance 120 and include or embed special tokens 206, 206a - 206n in the predicted output token sequence (i.e., the transcript 202), where the special tokens 206 represent the detected special input conditions. Example special input conditions include but are not limited to speaker changes (i.e., a change in the person speaking), spoken key phrases (e.g., spoken hot words such as "Hey Alexa", or spoken warm words such as "Volume Up"). In the example, for the spoken utterance 120 "Hey Assistant, what time is it" by a single user 10, the ASR model 200 generates the predicted transcript 202 "HeyAssistant <hw>", what time is it”. Here, the transcription 202 includes a special token 206 for indicating that the ASR model 200 recognized "Hey Alexa" as an oral hotword <hw>”and the general word tokens 204 representing other common words in the transcription 202 (e.g., "what", "time", "is", and "it"). In another example, for the multi-speaker oral utterance 120 including "word1 word2" spoken by the first user 10a, "word3 word4" spoken by the second user 10b, and "word5 word6" spoken by the first user 10a, the ASR model 200 generates the predicted transcription 202 "word1 word2 <st>word3 word4 <st>"word5 word6". Here, the transcription 202 includes a special token 206 for indicating that the ASR model has recognized a speaker change. <st>", and generic tokens 204 representing the identified words in the transcription 202 (e.g., "word1", "word2", "word3", "word4", "word5", and "word6"), where the identified words are spoken in the multi-speaker oral discourse 120. For clarity, the system 100 and the sequence transduction model 200 are described with reference to an ASR model 200 used to transcribe the oral discourse 120 and detect oral special input conditions such as speaker changes and special words (e.g., hot words and warm words). Additionally, for clarity, an example of using the ASR model 200 is used to describe the disclosed example method for enhanced training of a sequence transduction model using a token-level loss. However, other types of sequence transduction models can be used for predicting other types of output sequences from other types of input sequences and for detecting other types of special input conditions. For example, the sequence transduction model can include a character recognition model for transcribing a sequence of written characters into a text sequence, where the special input conditions include, but are not limited to, special characters, equations, offensive words, and proper names. Another example sequence transduction model includes a machine translation model for translating a sequence of input words in a first language into a sequence of output words in a second language different from the first language, where the special input conditions include, but are not limited to, untranslated proper names. Additionally, those of ordinary skill in the art will recognize that the disclosed example method for enhanced training of a sequence transduction model using a token-level loss can be used to enhance the training of other types of sequence transduction models.

[0028] In the example shown, the ASR model 200 includes a recurrent neural network transducer (RNN-T) model (see Figure 2 ), and is located on the user device 110 and / or in the cloud computing environment 140. The user device 110 and / or the cloud computing environment 140 also include an audio subsystem 150 configured to receive the discourse 120 spoken by the user 10 and captured by the audio capture device 116a, and convert the captured discourse 120 into a corresponding digital format associated with input audio data 122 (i.e., acoustic frames) that can be processed by the ASR model 200. Thereafter, the ASR model 200 receives the audio data 122 corresponding to the discourse 120 as input and generates / predicts a corresponding predicted transcription 202 (e.g., recognition result / hypothesis) of the discourse 120 as output.

[0029] The user device 110 may correspond to any computing device associated with the user 10 and capable of capturing audio data. Some examples of the user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smartwatches), smart appliances, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, etc. The user device 110 includes data processing hardware 112 and memory hardware 114 that communicates with the data processing hardware 112 and stores instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations. The user device 110 further includes an audio system 116 having: an audio capture device (e.g., a microphone) 116, 116a for capturing the spoken utterance 120 and converting the spoken utterance into an electrical signal and a voice output device (e.g., a speaker) 116, 116b for communicating an audible audio signal (e.g., as output audio data from the device 110). Although in the example shown, the user device 110 implements a single audio capture device 116a, the user device 110 may implement an array of audio capture devices 116a without departing from the scope of the present disclosure, where one or more of the capture devices 116a in the array may not be physically located on the user device 110 but communicate with the audio system 116.

[0030] The output 160 may receive the transcription 202 output from the sequence transduction model 200, which includes a sequence of general tokens 204 associated with words or sub-word units (i.e., graphemes, phonemes, word fragments) representing the words identified in the spoken utterance 120 and any special tokens 206 that the model 200 is trained to detect and embed at corresponding positions in the transcription. The output 160 may include a program (e.g., the digital assistant 50) or other components / software that the special token 206 is configured to trigger to perform an operation. For example, when the special token 206 includes a hotword <hw>When it does, the output can include a natural language understanding / processing (NLU / NLP) module executed on the user device 110 and / or the cloud computing environment 140, which performs query interpretation on the general tokens 204 in the transcription 202 to identify the user command / query, and then instructs downstream components / applications to perform the action / operation specified by the command.

[0031] The output 160 can also include a user interface generator executed on the user device 110 and / or the cloud computing environment 140, which is configured to present a representation of the transcription 202 to the user 10 of the user device 110. In some examples, the special token 206 can indicate a speaker change token, which indicates the corresponding position in the transcription 202 where the sequence transduction model 200 detects a corresponding speaker change event. In these examples, the output 160 corresponding to the user interface generator can annotate the transcription 202 with speaker labels based on the speaker change tokens. The user interface generator can display the initial speech recognition result 202a in a streaming manner and then display the final speech recognition result 202b.

[0032] In the example shown, the user 104 can interact with a program or application 50 (e.g., a digital assistant application 50) of the user device 110 that uses the ASR model 200. For example, the digital assistant application 50 can display a digital assistant interface on the screen of the user device 110 for depicting the interaction between the user 10 and the digital assistant application 50. For example, the digital assistant application 50 can use NLP / NLU to respond to questions posed by the user 10 to determine whether the written language prompts any action.

[0033] The cloud computing environment 140 can be a distributed or virtualized system with scalable / elastic resources 142. The resources 142 include computing resources 144 (e.g., data processing hardware) and / or storage resources 146 (e.g., memory hardware) that communicate with the computing resources 144 and store instructions that, when executed by the computing resources 144, cause the computing resources 144 to perform one or more operations. Alternatively, the operations of the cloud computing environment 140 can be implemented using a central or remote server with data processing hardware and memory hardware.

[0034] Figure 2 FIG. 0 is a schematic diagram of an exemplary recurrent neural network transducer model 200 (i.e., RNN-T model 200) that is trained using long-form training utterances to improve speech recognition of long-form utterances during inference. The RNN-T model 200 provides a small computational footprint and utilizes fewer memory requirements than conventional ASR architectures, making the RNN-T model 200 suitable for performing speech recognition entirely on the user device 110 (e.g., without the need to communicate with a remote server).

[0035] The RNN-T model 200 includes an encoder network 210, a prediction network 300, a joint network 220, and a final softmax layer 230. The prediction network 300 and the joint network 220 can together provide an RNN-T decoder. The encoder network 210, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a recurrent network of stacked long short-term memory (LSTM) layers. For example, the encoder reads a sequence of d-dimensional feature vectors (e.g., acoustic frames 122( Figure 1 )) x = (x1, x2, …, x T ), where and produces a high-order feature representation 212 at each time step. The high-order feature representation 212 is represented as

[0036] Similarly, the prediction network 300 is also an LSTM network that processes a sequence y0, ..., y ui-1 of non-blank symbols 232 output so far by the final softmax layer 230, like a language model (LM), into a dense or hidden representation 350. As described in more detail below, the representation 350 includes a single embedding vector. Notably, the sequence of non-blank symbols 232 received at the prediction network 300 captures the language dependencies between the non-blank symbols 232 predicted during previous time steps so far to assist the joint network 220 in predicting the probability of the next output symbol or blank symbol during the current time step. As described in more detail below, to facilitate techniques for reducing the size of the prediction network 300 without sacrificing the accuracy / performance of the RNN-T model 200, the prediction network 300 can receive a limited history sequence y ui-n 、..., y ui-1 of non-blank symbols 232 that is limited to the N previous non-blank symbols 232 output by the final softmax layer 230.

[0037] The joint network 300 combines the high-order feature representation 212 produced by the encoder network 210 and the representation produced by the prediction network 300 are combined with the single embedding vector 350. The joint network 220 predicts a distribution over the next output symbol 222. In other words, the joint network 220 generates a probability distribution 222 over the possible speech recognition hypotheses at each time step. Here, "possible speech recognition hypotheses" correspond to a set of output labels each representing a symbol / character in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, e.g., one label for each of the 26 letters in the English alphabet, and one label for the designated space. Thus, the joint network 220 can output a set of values indicating the likelihood of occurrence of each output label in a predetermined set of output labels. The set of values can be a vector and can indicate the probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation marks and other symbols), but the set of output labels is not limited to this. For example, as a supplement or alternative to graphemes, the set of output labels may include word fragments and / or whole words. The output distribution of the joint network 220 may include posterior probability values for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output Z i 232 of the joint network 220 may include 100 different probability values, one for each output label. Then, the probability distribution can be used (e.g., by the softmax layer 230) to select candidate orthographic elements (e.g., graphemes, word fragments, and / or words) during the beam search process and assign scores to them for determining the transcription 202.

[0038] The final softmax layer 230 receives the probability distribution Z i 232 of the final speech recognition result 220b and selects the output label / symbol with the highest probability to produce the transcription. The final softmax layer 230 can employ any technique to select the output label / symbol with the highest probability in the distribution Z i 232. In this way, the RNN-T model 200 does not make a conditional independence assumption. Instead, the prediction of each symbol 232 is conditioned not only on the acoustics but also on the sequence y ui-n 、......、y ui-1 of the labels 232 output so far. y u The RNN-T model 200 does assume that the output symbols 232 are independent of future acoustic frames 122, which allows the RNN-T model to be employed in a streaming fashion.

[0039] In some examples, the encoder network 210 of the RNN-T model 200 includes eight 2,048-dimensional LSTM layers, each followed by a 740-dimensional projection layer. In other implementations, the encoder network 210 includes multiple multi-head attention layers. For example, the multiple multi-head attention layers can include a network of conformer or transformer layers. The prediction network 220 can have two 2,048-dimensional LSTM layers, each of which is also followed by a 740-dimensional projection layer and an embedding layer with 128 units. Finally, the joint network 220 can also have 740 hidden units. The softmax layer 230 can be composed of a unified word segment or grapheme group, which is generated using all unique word segments or graphemes in the training data. When the output symbols / labels include word segments, the set of output symbols / labels can include 4,096 different word segments. When the output symbols / labels include graphemes, the set of output symbols / labels may include fewer than 100 different graphemes.

[0040] Figure 3 is a schematic diagram of an example prediction network 300 for the RNN-T model 200. The prediction network 300 receives a sequence y of non-blank symbols 232a - 232n ui-n 、......、y ui-1 as input, and the sequence of non-blank symbols is limited to the N previous non-blank symbols 232a - 232n output by the final softmax layer 230. In some examples, N is equal to two. In other examples, N is equal to five. However, the present disclosure is non-limiting and N can be equal to any integer. The sequence of non-blank symbols 232a - 232n indicates the initial speech recognition result 202( Figure 1 ). In some implementations, the prediction network 300 includes a multi-head attention mechanism 302 that shares a shared embedding matrix 304 across each head 302A - 302H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, the multi-head attention mechanism 302 can have any number of heads. Notably, the multi-head attention mechanism significantly improves performance with a minimal increase in model size. As described in more detail below, each head 302A - 302H includes its own row of position vectors 308, and instead of concatenating the outputs 318A - 318H from all heads to cause an increase in model size, the outputs 318A - 318H are averaged by the head averaging module 322.

[0041] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses the shared embedding matrix 304 for the sequence y of non-blank symbols 232a - 232n received as input at corresponding time steps over multiple time steps ui-n ,..., y ui-1 Each non - whitespace symbol in generates corresponding embeddings 306, 306a - 306n (e.g., ). It should be noted that, because the shared embedding matrix 304 is shared across all heads of the multi - head attention mechanism 302, all other heads 302B - 302H generate the same corresponding embeddings 306 for each non - whitespace symbol. Head 302A also assigns the corresponding position vectors PV Aa - PV An 308, 308Aa - 308An (e.g., ) to the sequence y of non - whitespace symbols 232a - 232n ui-n ,..., y ui-1 for each corresponding non - whitespace symbol. The corresponding position vector PV 308 assigned to each non - whitespace symbol indicates the position in the history of the sequence of non - whitespace symbols (e.g., the N previous non - whitespace symbols 232a - 232n output by the final softmax layer 230). For example, the first position vector PV Aa is assigned to the most recent position in the history, while the last position vector PV An is assigned to the last position in the history of the N previous non - whitespace symbols output by the final softmax layer 230. It should be noted that each embedding in the embeddings 306 can include the same dimension (i.e., dimension size) as each position vector in the position vectors PV 308.

[0042] Although the corresponding embeddings generated by the shared embedding matrix 304 for each non - whitespace symbol in the sequence y of non - whitespace symbols 232a - 232n ui-n ,..., y ui-1 are the same at all heads 302A - 302H of the multi - head attention mechanism 302, each head 302A - 302H defines a different set / row of position vectors 308. For example, the first head 302A defines a row of position vectors PV Aa - PV An 308Aa - 308An, the second head 302B defines a different row of position vectors PV Ba - PV Bn 308 Ba - 308 Bn ,..., and the H - th head 302H defines another different row of position vectors PV Ha - PV Hn 308 Ha - 308 Hn .

[0043] For each non - blank symbol in the received sequence of non - blank symbols 232a - 232n, the first head 302A also weights the corresponding embedding 306 proportionally to the similarity between the corresponding embedding and the respective position vector PV 308 assigned to it via the weight layer 310. In some examples, the similarity can include cosine similarity (e.g., cosine distance). In the illustrated example, the weight layer 310 outputs a weighted embedding sequence 312, 312Aa - 312An, where each of the weighted embeddings is associated with the corresponding embedding 306 and weighted proportionally to the respective position vector PV 308 assigned to it. In other words, the weighted embedding 312 output by the weight layer 310 for each embedding 306 can correspond to the dot product between the embedding 306 and the corresponding position vector PV 308. The weighted embeddings 312 can be interpreted as focusing on the embeddings in proportion to the degree of similarity with which they are positioned relative to their respective position vectors PV308. To increase the computational speed, the prediction network 300 includes non - recurrent layers, and thus, the weighted embedding sequence 312Aa - 312An is not concatenated but is averaged by the weighted average module 316 to generate a weighted average 318A of the weighted embeddings 312Aa - 312An as the output of the first head 302A, expressed as:

[0044]

[0045] In equation (1), h represents the index of the head 302, n represents the position in the context, and e represents the embedding dimension. Additionally, in equation (1), H, N, and d e include the corresponding dimension sizes. The position vector PV 308 does not have to be trainable and can include random values. Notably, even though the weighted embeddings 312 are averaged, the position vector PV 308 can potentially preserve position history information, thereby alleviating the need to provide recurrent connections at each layer of the prediction network 300.

[0046] The operations described above for the first head 302A are similarly performed by each of the other heads 302B - 302H of the multi - head attention mechanism 302. Due to the different sets of positioning vectors PV 308 defined by each head 302, the weight layer 310 outputs weighted embedding sequences 312Ba - 312Bn, 312Ha - 312Hn at each of the other heads 302B - 302H, which are different from the weighted embedding sequence 312Aa - 312Aa at the first head 302A. Thereafter, the weighted average module 316 generates corresponding weighted averages 318B - 318H of the weighted embeddings 312 of the non - blank symbol sequence as the outputs from each of the other corresponding heads 302B - 302H.

[0047] In the example shown, the prediction network 300 includes a head averaging module 322 that averages the weighted averages 318A - 318H output from the corresponding heads 302A - 302H. A projection layer 326 with SWISH can receive as input the output 324 corresponding to the average of the weighted averages 318A - 318H from the head averaging module 322 and generate a projection output 328 as the output. A final layer normalization 330 can normalize the projection output 328 to provide a single embedding vector at the corresponding time step among multiple time steps. 350. The prediction network 300 generates only a single embedding vector at each time step among multiple time steps after the initial time step. 350.

[0048] In some configurations, the prediction network 300 does not implement the multi - head attention mechanism 302 and only performs the operations described above with respect to the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa - 312An is simply passed through the projection layer 326 and the layer normalization 330 to provide a single embedding vector. 350.

[0049] In some implementations, to further reduce the size of the RNN - T decoder (i.e., the prediction network 300 and the joint network 220), parameter tying between the prediction network 300 and the joint network 220 is applied. Specifically, for a vocabulary size |V| and an embedding dimension d e , the shared embedding matrix 304 at the prediction network is Meanwhile, at the joint network 220, the last hidden layer has dimension size d h , and the feed - forward projection weights from the hidden layer to the output logits will be where there is an additional blank token in the vocabulary. Thus, the feed - forward layer corresponding to the last layer of the joint network 220 includes a weight matrix [d h , |V|]. By having the prediction network 300 bind the size of the embedding dimension d e to the dimension d h of the last hidden layer of the joint network 220, the feed - forward projection weights of the joint network 220 and the shared embedding matrix 304 of the prediction network 300 can share their weights for all non - blank symbols via a simple transpose transformation. Since the two matrices share all their values, the RNN - T decoder only needs to store the values in memory once instead of storing two separate matrices. By setting the size of the embedding dimension d e to be equal to the size of the hidden layer dimension d h , the RNN - T decoder reduces by an amount equal to the embedding dimension d e The number of parameters that is the product of the vocabulary size |V|. This weight tying corresponds to a regularization technique.

[0050] Return reference Figure 1 , system 100 includes a two-stage training process 400 that is located on user device 110 and / or cloud computing environment 140. In some examples, the sequence transduction model 200 is located on user device 110, while the training process 400 is located on cloud computing environment 140. However, both the sequence transduction model 200 and the training process 400 can be located on user device 110 and / or cloud computing environment 140.

[0051] The two-stage training process 400 (see Figure 4 ) trains the sequence transduction model 200 (e.g., the ASR model 200) on training data 405 that includes a plurality of training samples 410, 410a - 410n. Each training sample 410 includes an oral training utterance 415 (i.e., an input audio feature sequence) and a corresponding ground truth output 420 (e.g., a ground truth output token sequence representing a transcription 420 of the utterance 415). Here, the ground truth output 420 includes special tokens for corresponding special input conditions in the input data in addition to the general tokens of the oral words. That is, the ground truth output 420 is a transcription augmented with special tokens. An example training sample 410 includes the oral utterance 415 "Hey Assistant, what is today’s weather” and the corresponding ground truth transcription 420 "Hey Assistant <hw>, "what is today’s weather”. Here, the ground truth transcription 420 includes a special token " immediately following "Hey Assistant" for indicating that the speech recognition model 200 recognized "Hey Assistant" as an oral hot word <hw>", and a generic token for other verbal words representing the utterance (e.g., "what", "is", "today’s", and "weather"). It is noted that, in this example, the generic token can include word or sub-word unit tokens that form the transcription of the utterance 415.

[0052] The training process 400 trains the sequence transduction model 200 (e.g., the ASR model 200) on a sequence transduction task (e.g., a speech recognition performance task). The first stage of the training process 400 initially trains the sequence transduction model 200 to reduce a conventional loss function (e.g., maximizing negative log-likelihood), which is based on the difference between the ground truth output tokens 420 and the output tokens 204, 206 generated by the sequence transduction model 200 based on the input verbal training utterance 415. Here, the negative log-likelihood function is determined based on all output tokens (i.e., both generic tokens and special tokens) in the ground truth output tokens 420 and the output tokens 204, 206 predicted / generated by the sequence transduction model 200 based on the input verbal training utterance 415. It is noted that the sparsity of the special input conditions in the training data 405 causes the training process 400 not to emphasize special tokens during the first stage of the training process, which may degrade the detection accuracy of the special input conditions.

[0053] To prevent the learning of special tokens from being de-emphasized due to the relative sparsity of special tokens compared to generic tokens, the second stage of the training process 400 retrains or fine-tunes the sequence transduction model 200 using the same training data 405 in a second training pass. For each training sample 410 during the second training pass, the training process 400 performs beam search to identify the N best predicted output sequences (i.e., the N best output token sequence hypotheses) based on the input verbal training utterance 415. The second stage of the training process 400 performs retraining using a loss function that is a weighted sum of a token-level loss function and a conventional loss function. The conventional loss function can be the same as the loss function used during the initial training in the first training pass (e.g., maximizing negative log-likelihood). In some implementations, the token-level loss function includes a minimum additive error rate, such as MWER or word-level EMBR, and a greater loss value is assigned / appointed to the difference associated with special tokens than to the difference associated with generic tokens. Thus, during the second training pass, the training process 400 prioritizes the difference associated with special tokens over the difference associated with generic tokens. In this way, the training process 400 prioritizes the learning associated with predicting special tokens, which thus compensates for the sparsity of the special input conditions in the training data 405.

[0054] Figure 4 It is a schematic diagram of an example two-pass training process 400 for enhancing the training of a sequence transduction model 200 using a token-level loss. The sequence transduction model 200 may include Figure 2 an RNN-T model 200, which includes an encoder 210 and a decoder 430, where the decoder 430 collectively includes a prediction network 300 and a joint network 220. The training process 400 may be executed on a cloud computing environment 140 (i.e., on computing resources 144) and / or on a user device 110 (i.e., on data processing hardware 112).

[0055] For each training sample 410 in the set 405 of training samples, in the first stage of the training process 400, the RNN-T model 200 is used to process the corresponding spoken training utterance 415 to determine the probability Z i 222 of the most likely speech recognition hypothesis of the training utterance 415 being correct (i.e., the likelihood Z i 222 that y 232 is equal to the corresponding ground truth 420). Thereafter, the log-likelihood loss function module 440 determines the negative log-likelihood loss term i 442 based on the probability Z 222. The likelihood loss term 442 can be expressed as:

[0056]

[0057] where p(y * |x) = Z i . The first stage of the two-stage training process 400 applies an update 444 to the speech recognition model 200 based on the negative log-likelihood loss term 442 of each training sample 410 to initialize the speech recognition model 200 in the first training pass.

[0058] For each training sample 410 in the set of training samples 410, during the retraining of the RNN-T model 200 in the second training pass, the second stage of the training process 400 uses the RNN-T model 200 to process the corresponding spoken training utterance 415 to determine the probability Z i 222 (as described above), and obtain one or more speech recognition hypotheses 432, 432a - 432n (i.e., output token sequence hypotheses 432) for the training utterance 415. Thereafter, for each training sample 410 and each speech recognition hypothesis 432 output by the RNN - T model 200 for the corresponding training utterance 415, the sequence alignment module 450 determines the corresponding token - level cost for aligning the speech recognition hypothesis 432 for the training utterance 415 and the corresponding ground - truth transcription 420. In some implementations, a customized Levenshtein distance is used to determine the token - level cost for aligning sequences. Here, let

[0059] ·H ij be the j - th speech recognition hypothesis 432 among the N - best speech recognition hypotheses 432 for the i - th training sample 410, where i ∈ [1, M], j ∈ [1, N], and M is the number of training samples 410;

[0060] ·P ij be the probability Z ij for H i 222; and

[0061] ·G i be the ground - truth transcription 420 (i.e., ground - truth output token sequence) for the i - th training sample 410,

[0062] The following cost expression can be used to determine the example customized Levenshtein distance example between the sequences H ij and G i :

[0063]

[0064] where A represents the token of H ij and B represents the token of G i and <st>Denotes a special token. It is worth noting that equation (2) does not allow substitutions between special tokens and general tokens. It is worth noting that K is greater than 1 (e.g., slightly greater than 1, such as 1.1), such that during retraining, special token insertions and deletions have a greater impact compared to general word errors. Here, the customized Levenshtein distance only allows substitutions between general tokens, and special tokens can only be corrected, deleted, or inserted. The sequence alignment module 450 uses the costs of equations (3)-(5) to determine the optimal alignment of H ij and G i for each training sample i 410, which minimizes the customized Levenshtein distance.

[0065] The second stage of the training process 400 continues with the token-level loss function module 460, which determines the token-level cost for each speech recognition hypothesis H ij 432, such as MWER or word-level EMBR. That is, the second stage of the training process 400 determines the number of special token insertions FA ij for each speech recognition hypothesis H ij 432, the number of special token deletions FR ij and the number of general token changes W ij required to match each aligned speech recognition hypothesis H i 432 with the ground truth transcription G ij . For each speech recognition hypothesis H ij 432, the token-level loss function module 460 then determines the corresponding token-level loss 462, 462a-462n. The token-level loss 462 represents an additive error rate, such as MWER or word-level EMBR. In some examples, such as for speaker change detection, the token-level loss 462 is:

[0066]

[0067] where the values of the parameters a, β, and γ control the relative contributions of each sub-component, and Qi is the total number of output tokens in the ground truth sequence G i 420. In some implementations, the values of the parameters β and γ are set to be much greater than the value of the parameter a to force a reduction in the special token insertion and deletion rates. In alternative examples, for example, such as for hotword or wakeword detection, the token-level loss 462 is:

[0068]

[0069] wherein

[0070] FA ij = max(0, H ij - G i ), (8)

[0071] and

[0072] FR ij = max(0, G i - H ij ). (9)

[0073] In some implementations, the token-level loss of equation (6) 462 is first used for speaker change detection, and then the token-level loss of equation (7) 462 is used for special keyword detection (e.g., hotword or wakeword detection). In other implementations, a combination of equation (6) and equation (7) is used to determine the token-level loss 462.

[0074] Thereafter, the loss combination module 470 of the second stage of the training process 400 determines the overall token-level loss for the set of training data 405. The overall token-level loss can be expressed as

[0075]

[0076] and determines the combined loss 472, 472a - 472n, which can be expressed as

[0077]

[0078] where the value of the parameter λ controls the relative contributions of the token-level loss and the negative log-likelihood loss term 442 determined by the log-likelihood loss function module 440 (i.e., see equation (2)). The second stage of the two-stage training process 400 applies the update 474 to the speech recognition model 200 based on the combined loss 472 of each training sample 410 to retrain the speech recognition model 200 in the second training pass.

[0079] Figure 5 It is a flowchart of an exemplary arrangement of operations of a computer-implemented method 500 for enhancing the training of a sequence transduction model using a token-level loss. At operation 502, method 500 includes: receiving a plurality of training samples 410, each of the plurality of training samples including a corresponding training input feature sequence 415 paired with a corresponding ground-truth output token sequence 420, the ground-truth output token sequence 420 including a set of ground-truth general tokens and a set of ground-truth special tokens.

[0080] For each training sample 410 among the plurality of training samples 410, method 500 includes: at operation 504, using the sequence transduction model 200 to process the corresponding training input feature sequence 415 to obtain one or more output token sequence hypotheses 432, each output token sequence hypothesis 432 including one or more predicted general tokens 204.

[0081] For each training sample 410 among the plurality of training samples 410, method 500 further includes: at operation 506, for each corresponding output token sequence hypothesis 432 obtained for the training sample 410, determining a per-sample token-level loss 462 based on: the number of special token insertions FA ij , each of the special token insertions being associated with a corresponding predicted special token that appears in the corresponding output token sequence hypothesis 432 but not in the corresponding ground-truth output token sequence 420; and the number of special token deletions FR ij , each of the special token deletions being associated with a corresponding ground-truth special token in the set of ground-truth special tokens 420 that does not appear in the corresponding output token sequence hypothesis 432.

[0082] At operation 508, method 500 includes training the sequence transduction model 200 to minimize an additive error rate based on the per-sample token-level losses 462 determined for the plurality of training samples 410.

[0083] Figure 6 It is a schematic diagram of an example computing device 600 that can be used to implement the systems and methods described in this document. The computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functions are only intended to be exemplary and are not intended to limit the implementations of the invention described and / or claimed in this document.

[0084] The computing device 600 includes a processor 610 (i.e., data processing hardware) that can be used to implement data processing hardware 112 and / or 144, a memory 620 (i.e., memory hardware) that can be used to implement memory hardware 114 and / or 146, a storage device 630 (i.e., memory hardware) that can be used to implement memory hardware 114 and / or 146, a high-speed interface / controller 640 connected to the memory 620 and the high-speed expansion port 650, and a low-speed interface / controller 660 connected to the low-speed bus 670 and the storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and can be mounted on a common motherboard or otherwise as appropriate. The processor 610 can process instructions for execution within the computing device 600, including instructions stored in the memory 620 or on the storage device 630, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 680 coupled to the high-speed interface 640. In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. Additionally, multiple computing devices 600 can be connected, where each device provides a portion of the necessary operations (e.g., as a server group, blade server cluster, or multi-processor system).

[0085] The memory 620 stores information non-transitorily within the computing device 600. The memory 620 can be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transitory memory 620 can be a physical device for temporarily or permanently storing programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware such as a bootstrap program). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and magnetic disk or tape.

[0086] The storage device 630 can provide large-capacity storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory, or other similar solid-state memory devices, or an array of devices (including devices in a storage area network or other configurations). In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 620, the storage device 630, or the memory on the processor 610.

[0087] The high-speed controller 640 manages the bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages the lower bandwidth-intensive operations. Such a division of responsibilities is merely exemplary. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, via a network adapter.

[0088] The computing device 600 can be implemented in many different forms, as shown in the figure. For example, it can be implemented as a standard server 600a or multiple times as a group of such servers 600a, implemented as a laptop computer 600b, or implemented as part of a rack server system 600c.

[0089] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuitry, integrated circuit systems, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor, which can be dedicated or general-purpose and can be coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0090] A software application (i.e., software resource) can be computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application", "app", or "program". Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0091] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level programming and / or object-oriented programming languages and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., a disk, optical disk, memory, programmable logic device (PLD)) that provides machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal that provides machine instructions and / or data to a programmable processor.

[0092] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware) that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, for example, both general and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special-purpose logic circuitry.

[0093] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touch screen) for displaying information to the user and possibly a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including voice, speech, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0094] Unless there is an express contrary statement, the phrase "at least one of A, B, or C" is intended to refer to any combination or sub-group of A, B, C, such as: (1) only at least one A; (2) only at least one B; (3) only at least one C; (4) at least one A and at least one B; (5) at least one A and at least one C; (6) at least one B and at least one C; and (7) at least one A, at least one B, and at least one C. Further, unless there is an express contrary statement, the phrase "at least one of A, B, and C" is intended to refer to any combination or sub-group of A, B, C, such as: (1) only at least one A; (2) only at least one B; (3) only at least one C; (4) at least one A and at least one B; (5) at least one A and at least one C; (6) at least one B and at least one C; and (7) at least one A, at least one B, and at least one C.

[0095] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims.< / st> < / hw> < / hw> < / hw> < / st> < / st> < / st> < / hw> < / hw> < / hw> < / hw> < / hw> < / st> < / hw> < / hw> < / hw>

Claims

1. A computer-implemented method (500), which, when executed on data processing hardware (610), causes the data processing hardware (610) to perform operations, characterized in that, The operations include: Receiving a plurality of training samples (410), each training sample including a corresponding training input feature sequence (415) paired with a corresponding ground-truth output token sequence (420), the ground-truth output token sequence (420) including a set of ground-truth general tokens and a set of ground-truth special tokens; For each training sample (410) among the plurality of training samples (410): Using a sequence transduction model (200) to process the corresponding training input feature sequence (415) to obtain one or more output token sequence hypotheses (432), each output token sequence hypothesis (432) including one or more predicted general tokens (204); and For each corresponding output token sequence hypothesis (432) obtained for the training sample (410), determining a per-sample token-level loss (462) based on: The number of special token insertions, each special token insertion associated with a corresponding predicted special token (206) that appears in the corresponding output token sequence hypothesis (432) but not in the corresponding ground-truth output token sequence (420); and The number of special token deletions, each special token deletion associated with a corresponding ground-truth special token in the set of ground-truth special tokens that does not appear in the corresponding output token sequence hypothesis (432); and Training the sequence transduction model (200) to minimize an additive error rate based on the per-sample token-level loss (462) determined for the plurality of training samples (410).

2. The computer-implemented method (500) according to claim 1, characterized in that: Determining the per-sample token-level loss (462) is further based on the total number of ground-truth output tokens (420) in the corresponding ground-truth output token sequence (420), and the corresponding number of predicted general token errors relative to the set of ground-truth general tokens in the corresponding ground-truth token sequence (420); and When determining the per-sample token-level loss (462), the number of special token insertions and the number of special token deletions are each weighted higher than the corresponding number of predicted general token errors to force the sequence transduction model (200) to reduce special token insertion and deletion rates during training.

3. The computer-implemented method (500) according to claim 1 or 2, characterized in that, The operations further include, for each training sample (410) among the plurality of training samples (410): Using the sequence transduction model (200) to process the corresponding training input feature sequence (415) to predict a probability distribution (222) over possible output tokens; And Training the sequence transduction model (200) based on the negative logarithm (442) of the probability distribution of the corresponding ground-truth output token sequence (420) conditioned on the corresponding training input feature sequence (415).

4. The computer-implemented method (500) according to claim 3, characterized in that: Training the sequence transduction model (200) based on the negative logarithm (442) of the probability distribution includes: initially training the sequence transduction model (200) based on the negative logarithm (442) of the probability distribution to initialize the sequence transduction model (200); and Training the sequence transduction model (200) to minimize the additive error rate based on the per-sample token-level loss (462) includes: fine-tuning the initialized sequence transduction model (200) according to minimizing the additive error rate based on the per-sample token-level loss (462).

5. The computer-implemented method (500) according to any one of claims 1 to 4, characterized in that: Processing the corresponding training input feature sequence (415) to obtain one or more output token sequence hypotheses (432) includes: processing the corresponding training input feature sequence (415) to obtain an N-best list of output token sequence hypotheses (432), where each corresponding output token sequence hypothesis (432) in the N-best list has a corresponding probability score assigned by the sequence transduction model (200); and Determining the per-sample token-level loss (462) is further based on the corresponding probability scores of the corresponding output token sequence hypotheses (432).

6. The computer-implemented method (500) according to any one of claims 1 to 5, characterized in that: The training input feature sequence (415) includes a sequence of input audio frames representing an utterance (120) including a specific key phrase; The set of ground-truth general tokens in each corresponding ground-truth output token sequence (420) includes a set of word or sub-word unit tokens that form the ground-truth transcription of the utterance (120) represented by the sequence of input audio frames; The set of ground-truth special tokens in each corresponding ground-truth output token sequence (420) includes at least one ground-truth keyword token that indicates the corresponding position in the ground-truth transcription immediately after the specific key phrase appears in the ground-truth transcription; The one or more predicted general tokens (204) of each corresponding output token sequence hypothesis (432) in the one or more output token sequence hypotheses (432) include a sequence of predicted word or sub-word tokens that form a corresponding candidate transcription of the utterance; and Each corresponding predicted special token that appears in the corresponding output token sequence hypothesis (432) but does not appear in the corresponding ground-truth output token sequence (420) includes a predicted key phrase token that indicates the corresponding position in the corresponding candidate transcription immediately after the sequence transduction model (200) predicts the detection of the specific key phrase.

7. The computer-implemented method (500) according to any one of claims 1 to 6, characterized in that: The training input feature sequence (415) includes a sequence of input audio frames representing multiple utterances spoken by at least two different speakers; The set of ground-truth general tokens in each corresponding ground-truth output token sequence (420) includes a set of word or sub-word unit tokens that form the ground-truth transcription of the multiple spoken utterances (120) represented by the sequence of input audio frames; The set of ground truth special tokens in each corresponding ground truth output token sequence (420) includes a set of one or more ground truth speaker change tokens, each ground truth speaker change token indicating a corresponding position in the ground truth transcription of the plurality of utterances where a speaker change occurs; Each corresponding output token sequence hypothesis (432) of the one or more output token sequence hypotheses (432) includes a predicted sequence of words or sub-word unit tokens that form a corresponding candidate transcription of the plurality of utterances; and Each corresponding predicted special token that appears in the corresponding output token sequence hypothesis (432) but does not appear in the corresponding ground truth output token sequence (420) includes a predicted speaker change token, the predicted speaker change token indicating a corresponding position in the corresponding candidate transcription where the sequence transduction model (200) detects a corresponding speaker change event.

8. The computer-implemented method (500) according to any one of claims 1 to 7, characterized in that, The operation further includes, for each training sample (410) among a plurality of training samples (410): Determining a customized Levenshtein distance between each corresponding output token sequence hypothesis (432) of the one or more output token sequence hypotheses (432) obtained for the corresponding training input feature sequence (415) and the corresponding ground truth output token sequence (420); and Based on the customized Levenshtein distance: Identifying the number of special token insertions for each corresponding output token sequence hypothesis (432); and Identifying the number of special token deletions for each corresponding output token sequence hypothesis (432).

9. The computer-implemented method (500) according to claim 8, characterized in that, The customized Levenshtein distance determined between each corresponding output token sequence hypothesis (432) and the corresponding ground truth output token sequence (420) prevents the sequence transduction model (200) from allowing substitutions between special tokens and general tokens during training of the sequence transduction model (200).

10. The computer-implemented method (500) according to any one of claims 1 to 9, characterized in that, The sequence transduction model (200) includes a recurrent neural network transducer RNN-T model architecture.

11. The computer-implemented method (500) according to any one of claims 1 to 10, characterized in that, The sequence transduction model (200) includes at least one of a character recognition model, a speech recognition model, an end-pointing model, a speaker turn detection model, or a machine translation model.

12. A system (100), characterized in that, Comprising: Data processing hardware (610); Memory hardware (620), the memory hardware (620) communicating with the data processing hardware (610) and storing instructions that, when executed by the data processing hardware (610), cause the data processing hardware (610) to perform operations, the operations including: Receiving a plurality of training samples (410), each training sample including a corresponding training input feature sequence (415) paired with a corresponding ground truth output token sequence (420), the ground truth output token sequence (420) including a set of ground truth general tokens and a set of ground truth special tokens; For each of the plurality of training samples (410): Use a sequence transduction model (200) to process the corresponding training input feature sequence (415) to obtain one or more output token sequence hypotheses (432), each output token sequence hypothesis (432) including one or more predicted general tokens (204); and For each corresponding output token sequence hypothesis (432) obtained for the training sample (410), determine a per-sample token-level loss (462) based on: The number of special token insertions, each special token insertion associated with a corresponding predicted special token that appears in the corresponding output token sequence hypothesis (432) but not in the corresponding ground-truth output token sequence (420); and The number of special token deletions, each special token deletion associated with a corresponding ground-truth special token in the set of ground-truth special tokens that does not appear in the corresponding output token sequence hypothesis (432); and Train the sequence transduction model (200) to minimize an additive error rate based on the per-sample token-level loss (462) determined for the plurality of training samples (410).

13. The system (100) according to claim 12, characterized in that: Determining the per-sample token-level loss (462) is further based on the total number of ground-truth output tokens (420) in the corresponding ground-truth output token sequence (420), and the corresponding number of predicted general token errors relative to the set of ground-truth general tokens in the corresponding ground-truth token sequence (420); and When determining the per-sample token-level loss (462), the number of special token insertions and the number of special token deletions are each weighted higher than the corresponding number of predicted general token errors to force the sequence transduction model (200) to reduce special token insertion and deletion rates during training.

14. The system (100) according to claim 12 or 13, characterized in that, The operations further include, for each of the plurality of training samples (410): Use the sequence transduction model (200) to process the corresponding training input feature sequence (415) to predict a probability distribution (222) over possible output tokens; And Train the sequence transduction model (200) based on the negative log (442) of the probability distribution of the corresponding ground-truth output token sequence (420) conditioned on the corresponding training input feature sequence (415).

15. The system (100) according to claim 14, characterized in that: Training the sequence transduction model (200) based on the negative log (442) of the probability distribution includes: initially training the sequence transduction model (200) based on the negative log (442) of the probability distribution to initialize the sequence transduction model (200); and Training the sequence transduction model (200) to minimize the additive error rate based on the per-sample token-level loss (462) includes: fine-tuning the initialized sequence transduction model (200) according to minimizing the additive error rate based on the per-sample token-level loss (462).

16. The system (100) according to any one of claims 12 to 15, characterized in that: Processing the corresponding training input feature sequence (415) to obtain one or more output token sequence hypotheses (432) includes: processing the corresponding training input feature sequence (415) to obtain an N-best list of output token sequence hypotheses (432), each corresponding output token sequence hypothesis (432) in the N-best list having a corresponding probability score assigned by the sequence transduction model (200); and Determining the per-sample token-level loss (462) further based on the corresponding probability scores of the corresponding output token sequence hypotheses (432).

17. The system (100) according to any one of claims 12 to 16, characterized in that: The training input feature sequence (415) includes a sequence of input audio frames representing an utterance (120) including a particular key phrase; The set of ground-truth general tokens in each corresponding ground-truth output token sequence (420) includes a set of word or sub-word unit tokens that form a ground-truth transcription of the utterance (120) represented by the sequence of input audio frames; The set of ground-truth special tokens in each corresponding ground-truth output token sequence (420) includes at least one ground-truth keyword token that indicates a corresponding position in the ground-truth transcription immediately after the particular key phrase appears in the ground-truth transcription; The one or more predicted general tokens (204) of each corresponding output token sequence hypothesis (432) in the one or more output token sequence hypotheses (432) include a sequence of predicted word or sub-word tokens that form a corresponding candidate transcription of the utterance; and Each corresponding predicted special token that appears in the corresponding output token sequence hypothesis (432) but does not appear in the corresponding ground-truth output token sequence (420) includes a predicted key phrase token that indicates a corresponding position in the corresponding candidate transcription immediately after the sequence transduction model (200) predicts detection of the particular key phrase; 18. The system (100) according to any one of claims 12 to 17, characterized in that: The training input feature sequence (415) includes a sequence of input audio frames representing multiple utterances spoken by at least two different speakers; The set of ground-truth general tokens in each corresponding ground-truth output token sequence (420) includes a set of word or sub-word unit tokens that form a ground-truth transcription of the multiple spoken utterances (120) represented by the sequence of input audio frames; The set of ground-truth special tokens in each corresponding ground-truth output token sequence (420) includes a set of one or more ground-truth speaker change tokens, each ground-truth speaker change token indicating a corresponding position in the ground-truth transcription of the multiple utterances where a speaker change occurs; The one or more predicted general tokens (204) of each corresponding output token sequence hypothesis (432) in the one or more output token sequence hypotheses (432) include a sequence of predicted word or sub-word unit tokens that form a corresponding candidate transcription of the multiple utterances; and Each corresponding predicted special token that appears in the corresponding output token sequence hypothesis (432) but does not appear in the corresponding ground truth output token sequence (420) includes a predicted speaker change token that indicates the corresponding position in the corresponding candidate transcription where the sequence transduction model (200) detected a corresponding speaker change event.

19. The system (100) according to any one of claims 12 to 18, characterized in that, The operation further includes, for each training sample (410) among a plurality of training samples (410): determining a customized Levenshtein distance between each corresponding output token sequence hypothesis (432) obtained for the corresponding training input feature sequence (415) and the corresponding ground truth output token sequence (420); and based on the customized Levenshtein distance: identifying the number of special token insertions for each corresponding output token sequence hypothesis (432); and identifying the number of special token deletions for each corresponding output token sequence hypothesis (432).

20. The system (100) according to claim 19, characterized in that, The customized Levenshtein distance determined between each corresponding output token sequence hypothesis (432) and the corresponding ground truth output token sequence (420) prevents the sequence transduction model (200) from allowing substitutions between special tokens and general tokens during the training of the sequence transduction model (200).

21. The system (100) according to any one of claims 12 to 20, characterized in that, The sequence transduction model (200) includes a recurrent neural network transducer RNN-T model architecture.

22. The system (100) according to any one of claims 12 to 21, characterized in that, The sequence transduction model (200) includes at least one of a speech recognition model, a character recognition model, an endpointing model, a speaker turn detection model, or a machine translation model.

Images