Multitask learning for end-to-end automatic speech recognition confidence and deletion estimation

Abstract

The method (500) includes receiving a speech recognition result (232) corresponding to an utterance (204) of the utterance (12). For each sub-word unit in the sequence of hypothesis sub-word units of the speech recognition result, the method also includes obtaining a respective confidence embedding (242) associated with the corresponding output step when the corresponding sub-word unit is output from the first speech recognizer (200), generating a confidence feature vector (312), generating an acoustic context vector (322), and generating a respective confidence output score (302) for the corresponding sub-word unit based on the confidence feature vector and the acoustic context vector received as input by an output layer of the confidence prediction module. The method also includes determining an utterance-level confidence score (350) for the utterance of the utterance based on the respective confidence output scores generated for each sub-word unit in the sequence of hypothesis sub-word units.

Description

[Technical Field]

[0001] This disclosure relates to multi-task training for end-to-end automatic speech recognition confidence and deletion estimation. [Background technology]

[0002] Modern automatic speech recognition (ASR) systems focus not only on providing quality / accuracy (e.g., low word error rate (WER)), but also on providing low latency (e.g., short delay between user speech and appearance of the utterance). Furthermore, today's ASR systems require the ASR system to decode utterances in a streaming manner, corresponding to displaying the utterance's utterance in real time, or even faster than real time, as the user speaks. By way of example, when an ASR system is deployed on a mobile phone that encounters direct user interactivity, an application on the mobile phone using the ASR system may require that speech recognition be streamed, such that words, word pieces, and / or individual characters appear on the screen immediately as they are spoken. Here, mobile phone users may have a low tolerance for latency. Because of this low tolerance, speech recognition is intended to operate on mobile devices in a manner that minimizes the impact of latency and inaccuracies that can adversely affect the user's experience. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Application Serial No. 17 / 182,592 Summary of the Invention [Means for solving the problem]

[0004] One aspect of the present disclosure provides a computer-implemented method for multi-task training for end-to-end automatic speech recognition confidence and deletion estimation. When executed on data processing hardware, the computer-implemented method causes the data processing hardware to perform operations including receiving, from a first speech recognizer, a speech recognition result corresponding to an utterance spoken by a user. The speech recognition result includes a sequence of hypothesis subword units forming one or more words of the utterance, each subword unit being output from the first speech recognizer at a corresponding output step. The operations also include using a confidence prediction module to obtain, for each subword unit in the sequence of hypothesis subword units, a respective confidence embedding associated with the corresponding output step when the corresponding subword unit is output from the first speech recognizer. The operations further use the confidence prediction module to generate a confidence feature vector using a first attention mechanism that is self-associated with the corresponding subword unit and each confidence embedding obtained for any other subword unit in the sequence of hypothesis subword units that follows the corresponding subword unit. The operations continue to use the confidence prediction module to generate an acoustic context vector using a second attention mechanism that is cross-associated with the sequence of encodings each associated with the corresponding acoustic frame segmented from the acoustic data corresponding to the utterance. Furthermore, the operations then use the confidence prediction module to generate a respective confidence output score for the corresponding subword unit based on the confidence feature vector and the acoustic feature vector received as input by the output layer of the confidence prediction module as output from the output layer of the confidence prediction module. The operations also include determining an utterance-level confidence score for the utterance's expression based on the respective confidence output scores generated for each subword unit in the sequence of hypothesis subword units.

[0005] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, determining the utterance-level confidence score for the utterance form includes determining, for each word of the one or more words of the utterance form, a respective word-level confidence score equal to a respective confidence output score generated for the last sub-word unit in the corresponding word. These implementations further include aggregating the respective word-level confidence scores determined for each of the one or more words of the utterance form to determine the utterance-level confidence score for the utterance form.

[0006] In some additional implementations, the operations further include using a hierarchical attention mechanism to estimate a probability that the speech recognition result for the utterance will be correctly recognized; determining an utterance-level loss for the speech recognition result based on the estimated probability that the speech recognition result for the utterance will be correctly recognized and a ground truth label indicating whether the utterance was correctly recognized; and training a confidence prediction model for the utterance-level loss.

[0007] In some examples, the confidence prediction model and the first speech recognizer are jointly trained. The operations can include estimating the number of deletions at each position in a sequence of words of an utterance, determining a deletion loss for the speech recognition results and the estimated number of deletions and a ground truth number of deletions at each position in the sequence of words, and training a confidence prediction model on the deletion loss.

[0008] In some implementations, the operations further include rescoring candidate speech recognition results recognized by the first speech recognizer after training the confidence prediction model. A sub-word unit can include a word piece or a grapheme.

[0009] In some examples, the first speech recognizer includes a transformer decoder model configured to generate a plurality of candidate hypotheses, each corresponding to a candidate representation for the utterance and represented by a respective sequence of hypothesis subword units. In these implementations, the transformer decoder model may include a Recurrent Neural Network-Transducer (RNN-T) model architecture.

[0010] In some implementations, the operations further include determining whether the utterance-level confidence score for the utterance form satisfies a confidence threshold. In these implementations, if the utterance-level confidence score for the utterance form does not satisfy the confidence threshold, the operations further include rejecting the utterance form output from the first speech recognizer and instructing a second speech recognizer to process acoustic data corresponding to the utterance spoken by the user to retranscribe the utterance spoken by the user. In these implementations, the first speech recognizer can reside on a user device associated with the user, and the second speech recognizer can execute on a remote server in communication with the user device, and the user device can transmit the acoustic data to the remote server if the utterance-level confidence score for the utterance form does not satisfy the confidence threshold. Also, in these implementations, the second speech recognizer can be more computationally intensive than the first speech recognizer.

[0011] Another aspect of the present disclosure provides a system for multi-task training for end-to-end automatic speech recognition confidence and deletion estimation. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving, from a first speech recognizer, a speech recognition result corresponding to an expression of an utterance spoken by a user. The speech recognition result includes a sequence of hypothesis subword units forming one or more words of the expression of the utterance, each subword unit being output from the first speech recognizer at a corresponding output step. The operations also include using a confidence prediction module to obtain, for each subword unit in the sequence of hypothesis subword units, a respective confidence embedding associated with the corresponding output step when the corresponding subword unit is output from the first speech recognizer. The operations also use a confidence prediction module to generate a confidence feature vector using a first attention mechanism that is self-associated with the respective confidence embeddings for the corresponding subword unit and confidence embeddings obtained for any other subword units in the sequence of hypothesis subword units that follows the corresponding subword unit. The operations continue to use the confidence prediction module to generate an acoustic context vector using a second attention mechanism that is cross-associated with the sequence of encodings each associated with a corresponding acoustic frame segmented from the acoustic data corresponding to the utterance. The operations further use the confidence prediction module to generate a respective confidence output score for the corresponding subword unit based on the confidence feature vector and the acoustic feature vector received as input by the output layer of the confidence prediction module as output from the output layer of the confidence prediction module.The operations also include determining an utterance-level confidence score for the utterance form based on the respective confidence output scores generated for each subword unit in the sequence of hypothesis subword units.

[0012] This aspect may include one or more of the following optional features: In some implementations, determining the utterance-level confidence score for the utterance form includes determining, for each word of the one or more words of the utterance form, a respective word-level confidence score equal to a respective confidence output score generated for a last sub-word unit in the corresponding word. These implementations further include aggregating the respective word-level confidence scores determined for each of the one or more words of the utterance form to determine the utterance-level confidence score for the utterance form.

[0013] In some additional implementations, the operations further include using a hierarchical attention mechanism to estimate a probability that the speech recognition result for the utterance will be correctly recognized; determining an utterance-level loss for the speech recognition result based on the estimated probability that the speech recognition result for the utterance will be correctly recognized and a ground truth label indicating whether the utterance was correctly recognized; and training a confidence prediction model for the utterance-level loss.

[0014] In some examples, the confidence prediction model and the first speech recognizer are jointly trained. The operations can include estimating the number of deletions at each position in a sequence of words of an utterance, determining a deletion loss for the speech recognition results and the estimated number of deletions and a ground truth number of deletions at each position in the sequence of words, and training a confidence prediction model on the deletion loss.

[0015] In some implementations, the operations further include rescoring candidate speech recognition results recognized by the first speech recognizer after training the confidence prediction model. A sub-word unit can include a word piece or a grapheme.

[0016] In some implementations, the first speech recognizer includes a transformer decoder model configured to generate a plurality of candidate hypotheses, each corresponding to a candidate representation for the utterance and represented by a respective sequence of hypothesis subword units. In these implementations, the transformer decoder model may include a Recurrent Neural Network-Transducer (RNN-T) model architecture.

[0017] In some examples, the operations further include determining whether the utterance-level confidence score for the utterance form satisfies a confidence threshold. In these examples, if the utterance-level confidence score for the utterance form does not satisfy the confidence threshold, the operations further include rejecting the utterance form output from the first speech recognizer and instructing a second speech recognizer to process acoustic data corresponding to the utterance spoken by the user to retranscribe the utterance spoken by the user. In these examples, the first speech recognizer can reside on a user device associated with the user, and the second speech recognizer can execute on a remote server in communication with the user device, and the user device can transmit the acoustic data to the remote server if the utterance-level confidence score for the utterance form does not satisfy the confidence threshold. Also, in these examples, the second speech recognizer can be more computationally intensive than the first speech recognizer.

[0018] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a diagram of an exemplary speech environment. [Figure 2] 1 is a schematic diagram of an exemplary speech recognizer and a confidence prediction model deployed on the speech recognizer. [Figure 3] 3 is a schematic diagram of training the confidence prediction module of FIG. 2 using multi-task learning. [Figure 4] 1 is a diagram of an exemplary confidence-based routine for selecting an appropriate speech recognizer based on utterance level confidence. [Figure 5] 1 is a flowchart of an exemplary arrangement of operations for a method of determining an utterance-level confidence score of a phrasing output by a speech recognizer. [Figure 6] 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0020] Like reference symbols in the various drawings indicate like elements.

[0021] Automatic speech recognition (ASR) systems focus not only on providing quality / accuracy (e.g., low word error rate (WER)) but also on providing low latency (e.g., short delay between user speech and expression appearance). Recently, end-to-end (E2E) ASR models, such as recurrent neural network-transducers (RNN-T), transformers or conformer transformers, and attention-based encoder-decoder models, have gained popularity to achieve state-of-the-art performance in accuracy and latency. In contrast to traditional hybrid ASR systems that include separate acoustic, pronunciation, and language models, E2E models apply a sequence-to-sequence approach to jointly learn acoustic and language modeling in a single neural network that is trained end-to-end from training data, e.g., utterance-expression pairs.

[0022] Examples of sequence-to-sequence models include "attention-based" models and "listen-attend-spell" (LAS) models. LAS models transcribe speech utterances into characters using a listener component, an attendant component, and a speller component. Here, the listener is a recurrent neural network (RNN) encoder that receives acoustic input (e.g., a time-frequency representation of the speech input) and maps the acoustic input to a higher-level feature representation. The attendant learns alignments between input features and predicted subword units (e.g., graphemes or word pieces) in conjunction with the higher-level features. The speller is an attention-based RNN decoder that generates strings of characters from the input by generating probability distributions over a set of hypothesis words. Due to their unified structure, all components of the model can be trained jointly as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose entire architecture is built on a neural network. The complete neural network functions without external and / or manually designed components (e.g., finite-state transducers, lexicons, or sentence normalization modules). Furthermore, when training E2E models, these models typically do not require bootstrapping from decision trees or time alignment from separate systems.

[0023] Furthermore, today's ASR systems require that they decode utterances in a streaming manner, corresponding to displaying a representation of the utterance in real time, or even faster than real time, as the user speaks. For example, when an ASR system is deployed on a mobile phone that encounters direct user interactivity, applications on the mobile phone using the ASR system may require that speech recognition be streamed, such that words, word pieces, and / or individual characters appear on the screen immediately as they are spoken. Here, mobile phone users may have a low tolerance for latency. Because of this low tolerance, speech recognition strives to operate on mobile devices in a manner that minimizes the impact of latency and inaccuracies that can adversely affect the user experience. However, sequence-to-sequence models, such as the LAS model, which function by reviewing the entire acoustic input sequence before generating an output sentence, do not allow streaming output to be received as input. This deficiency can pose problems for deploying LAS models for speech applications that are latency-sensitive and / or require a real-time voice representation. This makes the LAS model, by itself, a less than ideal model for mobile technologies (eg, mobile telephony) that often rely on real-time applications (eg, real-time communication applications).

[0024] Another form of sequence-to-sequence model, known as recurrent neural network transformer (RNN-T), does not use an attention mechanism. Unlike other sequence-to-sequence models, which typically require processing an entire sequence (e.g., an acoustic waveform) to generate an output (e.g., a sentence), RNN-T processes input samples continuously and streams output symbols—features that are particularly attractive for real-time communication. For example, speech recognition using RNN-T can output a character for each spoken word. Here, RNN-T uses a feedback loop in which the symbol predicted by the model is fed back to itself to predict the next symbol. Because RNN-T decoding involves beam search through a single neural network instead of a large decoder graph, RNN-T can scale to a fraction of the size of server-based speech recognition models. This reduced size allows RNN-T to be deployed entirely on a device and can operate offline (i.e., without a network connection), thereby avoiding unreliability issues associated with communication networks.

[0025] However, RNN-T models alone still lag behind traditional large-scale state-of-the-art models (e.g., server-based models using separate AM, PM, and LM) in terms of quality (e.g., speech recognition accuracy). Non-streaming E2E, LAS models still have speech recognition quality comparable to traditional large-scale state-of-the-art models. To take advantage of the quality of non-streaming E2E LAS models, two-pass speech recognition systems (e.g., as shown in Figure 2A) have been developed that include a first-pass component of an RNN-T network followed by a second-pass component of an LAS network. With this design, the two-pass model takes advantage of the streaming nature of the low-latency RNN-T model while improving the accuracy of the RNN-T model via the second pass incorporating the LAS network. While the LAS network increases latency compared to the RNN-T model alone, the latency increase is moderately small and meets the latency constraints for on-device operation. In terms of accuracy, the two-pass model achieves a 17-22% reduction in WER compared to RNN-T alone, and has a similar WER compared to conventional large-scale models.

[0026] Confidence scores are an important feature of ASR systems that support many downstream applications and reduce speech recognition errors. For example, unlabeled utterances with recognition results output from an ASR model that achieves high confidence can be used to perform semi-supervised training of an ASR model, reducing the cost of training using only transcribed utterances. On the other hand, in applications such as voice dialogue systems in which a user interacts with a digital assistant running on a computing device, utterances with recognition results that achieve low word-level confidence can prompt the user to correct any transcription errors. Furthermore, recognition results with low confidence can result in the acoustics for the corresponding utterance being passed to a different, more computationally extensive ASR model (e.g., on the server side) to improve recognition of the utterance.

[0027] While conventional hybrid ASR systems can easily infer word-level confidence scores from word posterior probabilities computed from lattice networks or confusion networks and then aggregate them to provide utterance-level confidence, the deep neural networks used by E2E ASR models tend to exhibit overconfidence when predicting words. Because many E2E ASR models are configured to output recognition results at the subword level, simply learning a confidence score for each subword recognized by the ASR model using the corresponding fixed subword tokenization for the word as a reference sequence can result in inaccurate ground truth labels used to train the confidence prediction model because the recognition results may contain multiple valid tokenizations. For example, the reference fixed subword sequence for the utterance "Good morning" may be "go, od, morn, ing," while the hypothetical subword sequence recognized by the ASR model may be "go, od, mor, ning, mom." Here, even though the word "morning" is correctly recognized by the ASR model, the sub-word labels for the corresponding hypothetical sub-words "mor" and "ning" recognized by the ASR model will be assigned inappropriate labels because these labels do not match the corresponding reference fixed sub-words "morn" and "ing" for the word "morning."

[0028] To mitigate drawbacks due to inconsistencies between reference-fixed subword sequences associated with predicting subword confidence scores for hypothetical subword sequences recognized by an ASR model, embodiments herein are directed to a confidence prediction module that applies self-attention to predict word-level confidence for each recognized word using only the confidence of the last hypothetical subword unit recognized by the ASR model that constitutes the corresponding word. The contents of U.S. Application No. 17 / 182,592, filed February 23, 2021, are incorporated by reference in its entirety. Learning only the word-level confidence of each hypothetical word recognized by an ASR model provides an estimate of the proportion of correct words relative to the total number of words in the hypothesis. However, the training labels do not provide a signal as to whether the entire utterance was correctly recognized or whether there are deletions in the hypotheses recognized by the ASR model. Therefore, word-level confidence estimation does not model deletions, and the confidence of the entire utterance, based on the aggregation of the word-level confidences of the words in the utterance, discards many useful word-level training signals. Embodiments herein are further directed to applying multi-task learning to jointly train confidence models using goals related to word training, utterance training, and deletion training.

[0029] 1 illustrates an exemplary speech environment 100 in which users 10 interact with user devices 110 via voice input. User devices 110 (also commonly referred to as devices 110) include computing devices configured to capture sounds (e.g., streaming acoustic data) from one or more users 10 in the speech-enabled environment 100. Here, streaming acoustic data 202 can refer to utterances spoken by users 10 that function as audible queries, commands to devices 110, or audible communications captured by devices 110. The speech-enabled system of device 110 can process queries or commands by replying to queries and / or cause the commands to be implemented.

[0030] The user device 110 may correspond to any computing device capable of receiving acoustic data 202. Some examples of the user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, Internet of Things (IoT) devices, smart speakers / displays, vehicle infotainment systems, etc. The user device 110 includes data processing hardware 112 and memory hardware 114 in communication with the data processing hardware 112, the memory hardware 114 storing instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations. The user device 110 further includes an acoustic subsystem 116 having sound capture devices (e.g., microphones) 116, 116a for capturing and converting spoken utterances 12 into electrical signals within the speech-enabled system 100, and speech output devices (e.g., speakers) 116, 116b for communicating audible acoustic signals (e.g., as output acoustic data from the device 110). While the user device 110 implements a single sound capture device 116a in the illustrated example, the user device 110 could implement an array of sound capture devices 116a without departing from the scope of this disclosure, whereby one or more capture devices 116a in the array may not be physically present on the user device 110 and can be in communication with the acoustic subsystem 116. In the illustrated example, the user device 110 (e.g., using hardware 112, 114) implements a speech recognizer 200 configured to perform speech recognition on acoustic data 202 corresponding to the utterances 12 spoken by the user 10. Here, the sound capture device 116 a is configured to capture acoustic sounds representing the utterance 12 and convert the acoustic sounds into associated acoustic data 202 in a digital format compatible with the speech recognition device 200 .The digital format associated with the acoustic data 202 may correspond to an acoustic frame, such as a mel frame (e.g., a parameterized acoustic frame), for example, a parameterized acoustic frame corresponds to a log-mel filter bank energy.

[0031] While FIG. 1 shows user device 100 implementing speech recognizer 200 for performing speech recognition on the device, other embodiments include a remote server 410 (FIG. 4) implementing speech recognizer 200 by processing acoustic data 202 transmitted by user device 110 over a network and providing a representation 204 of acoustic data 202 back to user device 110. In some additional embodiments, user device 110 utilizes both a local speech recognizer 200 resident on user device 110 and a server-side speech recognizer 402 (FIG. 4) executing on remote server 410. Here, user device 110 can use local speech recognizer 200 when a network connection is unavailable or for speech applications that are latency-sensitive and / or require a streaming representation, while utilizing server-side speech recognizer 402 when additional resources are needed to improve speech recognition accuracy, as described in more detail below with reference to FIG. 4.

[0032] In some examples, a user 10 interacts with a program or application 118 executing on a user device 110 using a speech recognizer 200. For example, FIG. 1 depicts a user 10 communicating with an automated assistant application 118. In this example, the user (e.g., Bill) 10 greets the automated assistant application by speaking an utterance 12. "Good morning" is captured by an audio capture device 116a and converted into corresponding acoustic data 202 (e.g., acoustic frames) for processing by the speech recognizer 200. In this example, the speech recognizer 200 transcribes the acoustic data 202 representing the utterance 12 into a representation 204 (e.g., a written representation of "Good morning"). The automated assistant application 118 can then apply natural language processing to the representation 204 to generate a response 119 for output to the user 10, conveying the message, "Good morning Bill, the first meeting on your calendar today is at 9:00 AM." Natural language processing typically refers to the translation of written language (e.g., idiom 204) and the process of determining whether the written language prompts some action. In this example, assistant application 118 uses natural language processing to recognize that utterance 12 spoken by user 10 is intended to cause assistant application 118 to access user 10's calendar application and provide response 119 indicating the time of user 10's first meeting today. That is, by using natural language processing to recognize these details, assistant application 118 returns response 119 to user 10 as a synthesized speech representation for audible output via acoustic output device 116a and / or as text for display on a screen in communication with user device 110. In some examples, user device 110 displays idiom 204 of utterance 12 spoken by user 10 and the corresponding response 119 from assistant application 118 as on-screen dialogue. In some configurations, natural language processing may occur on a remote system in communication with data processing hardware 112 of user device 110.

[0033] In some examples, a speech recognizer 200 processes incoming acoustic data 202 in real time to provide a streaming representation 204. Here, the speech recognizer 200 is configured to generate a sequence of hypothesis sub-word units making up words of an utterance 12 spoken by a user 10. The hypothesis sub-word units may include word pieces or individual characters (e.g., graphemes). In the example shown, the sequence of hypothesis sub-word units recognized by the speech recognizer includes "SOS_go od_mor ning," where "SOS" indicates the start of the speech tag and individual word boundary indicators ("_") indicate the beginning / start sub-word units for each word.

[0034] 2, in some implementations, a speech recognizer 200 includes a Recurrent Neural Network-Transducer (RNN-T) decoder 220 for predicting a speech recognition result / hypothesis 232 and implements a confidence prediction module (CEM) 300 for estimating a confidence 302 of the speech recognition result / hypothesis 232. The CEM 300 can use the utterance-level confidence 302 to rescore the speech recognition result / hypothesis 232 predicted by the speech recognizer 200. Here, the speech recognition result / hypothesis 232 corresponds to a sequence of sub-word units, such as word pieces or graphemes, which, when aggregated, together form an expression 204 for the utterance. Generally speaking, the speech recognizer 200 includes an encoder 210 and an RNN-T decoder 220, which includes a prediction network and a collaboration network. The RNN-T decoder 220 may generate multiple candidate hypotheses H as output 222, and the CEM 300 may rescore / rerank the top K candidate hypotheses H to identify the highest-scoring candidate hypothesis as the final recognition result corresponding to the representation format 204 (FIG. 1). Furthermore, although the examples herein show the decoder 220 having an RNN-T model architecture, the decoder 220 may similarly include other types of transformer model architectures without departing from the scope of this disclosure. For example, the decoder 220 may include one of a Transformer-Transducer, a Convolutional Neural Network-Transducer (ConvNet-Transducer), or a Conformer-Transducer model architecture instead of the RNN-T model architecture.

[0035] At least one shared encoder 210 is configured to receive as input acoustic data 202 corresponding to utterance 12 as a sequence of acoustic frames. The acoustic frames may be pre-processed by acoustic subsystem 116 into parameterized acoustic frames (e.g., mel-frames and / or spectral frames). In some implementations, the parameterized acoustic frames correspond to log-mel filterbank energies having log-mel characteristics. For example, a parameterized input acoustic frame representing acoustic data 202 input into encoder 210 may be x=(x,...,x) T ) and

[0036]

number

[0037] is the log-mel filter bank energy, T represents the number of frames in x, and d represents the number of log-mel features. In some examples, each parameterized acoustic frame contains 128-dimensional log-mel features computed within a short shifting window (e.g., 32 ms and shifted every 10 ms). Individual features can be stacked using previous frames (e.g., three previous frames) to form a higher dimensional vector (e.g., a 512-dimensional vector using three previous frames). The features forming the vector can then be downsampled (e.g., to a 30 ms frame rate). The encoder 210 processes an acoustic frame x of the acoustic data 202 input to the encoder 210. 1:T For each 1:T Each frame number in x, denoted by T, corresponds to a respective time step.

[0038] Although the structure of encoder 210 can be implemented in different ways, in some implementations, encoder 210 includes a long-short-term memory (LSTM) neural network. For example, the LSTM neural network can include eight LSTM layers, where each layer can have 2,048 hidden units followed by a 640-dimensional projection layer. In some examples, a time compaction layer with a compaction factor of N=2 is inserted after the second LSTM layer of encoder 210.

[0039] In some implementations, the RNN-T decoder 220 includes a prediction network and a collaborative network. Here, the prediction network can have two LSTM layers of 2,048 hidden units, 640-dimensional projections per layer, and a 128-unit embedding layer. The outputs 212 of the shared encoder 210 and the prediction network can be fed into a collaborative network that includes a softmax prediction layer. In some examples, the collaborative network of the RNN-T decoder 220 includes 640 hidden units followed by a softmax layer that predicts 4,096 mixed-example word pieces.

[0040] The RNN-T decoder 220 receives as input the encoding e produced as output 212 from the encoder 210 for each acoustic frame x, and as output 222 the hypothesized subword units y1, y2, y3, ..., y M , y1, y2, y3, ..., y4, .... In the example where user 10 utters "good morning," one candidate hypothesis H may include a first sequence of hypothesis sub-word units [_go, od, _mor, ning], and another candidate hypothesis H may include a second sequence of hypothesis sub-word units [_go, od, _morn, ing]. There may also be many other candidate hypotheses H. Here, the individual candidate hypotheses H are represented by hypothesis sub-word units y1, y2, y3, ..., y5, .... M Each sequence of y corresponds to a candidate representation for the utterance 12. The hypothesized subword units y, y, y, ..., y MEach sub-word unit y in each respective sequence of i represents a probability distribution over all possible sub-units. The sub-word unit with the highest probability in the probability distribution may be selected as the hypothesized sub-word in each sequence of hypothesized sub-word units.

[0041] Continuing with reference to FIG. 2, the CEM 300 receives as input the encoded signals e, e output from the encoder 210. 1:T 212 and the top K candidate hypotheses H generated as output 222 from RNN-T decoder 220. In one example, K is equal to 4, so the top four candidate hypotheses H are provided as input to CEM 300 for rescoring by determining an utterance-level confidence 350 for each hypothesis H. The candidate hypothesis H associated with the highest confidence 350 may be output as representation 204.

[0042] To reduce the size of the softmax layer, the subword unit vocabulary of possible subword units is typically smaller compared to the word vocabulary. The subword unit vocabulary may include graphemes or word pieces (WPs). An exemplary WP vocabulary may include 4,096 WPs. While the examples of this disclosure use WPs as subword units generated as output from a speech recognizer, graphemes may similarly be utilized as subword units output from a speech recognizer without departing from the scope of this disclosure. Thus, to compute a word error rate (WER) for a candidate hypothesis H, each sequence of hypothesis subword units (e.g., WPs) is combined with its corresponding word sequence w1, w2, ..., w L Since the first subword unit of each word (e.g., WP) begins with a word boundary indicator ("_"), this procedure for converting a subword sequence to a word sequence is uniquely determined. Similarly, y j , where q represents the qth WP of the jth word. jWords containing WP j A simple technique for computing word confidence is c word (w j )=agg(y j , 1), …, c(y j , Q j ) (1) where agg can be an arithmetic mean, minimum, product or neural network. However, the individual words w j Since can be split into multiple different valid WP combinations due to the discrepancy between WP accuracy and word accuracy, when we use Equation 1 to predict word-level confidence, we find the word w with the fewest WP edits. j This would result in an undesirable computational burden during training, as it would require searching all possible reference tokenizations for . As used herein, a WP edit includes a correct (cor) label if the hypothesis WP matches the reference WP, a substitution (sub) label if the valid hypothesis WP does not match the reference WP, and an insertion (ins) label if the hypothesis WP is misrecognized. Table 1 below shows an example where the word "morning" is correctly transcribed, but results in two substitutions in the WP edit distance output.

[0043] [Table 1]

[0044] To remedy the inherent discrepancy between WP accuracy and word accuracy introduced by a speech recognizer 200 outputting at the WP levels shown in Table 1, embodiments herein are directed to a transformer-based CEM 300 that utilizes the confidence output in the final WP of every word as the word-level confidence, while ignoring the confidence of all other WPs preceding every word. During training, the speech recognizer 200 is frozen and ground truth WP labels of corrections, insertions, and substitutions are used.

[0045] Figure 3 shows an example of a transformer-based CEM 300 deployed on a speech recognizer 200. For clarity, Figure 3 shows the hypotheses WP y1, y2, y3, ..., y output by the RNN-T decoder 220. M The confidence output c(y i ) 302. In particular, Figure 3 shows CEM 300 predicting the confidence output c("ning") for the "ning" WP in each of the hypothetical sequences of WPs [_go, od, _mor, ning] that translate to the corresponding word sequence "good morning."

[0046] The RNN-T decoder 220 outputs as output 222 the hypothesized subword units y1, y2, y3, ..., y MHere, the RNN-T decoder 220 may generate four candidate hypotheses as output 222. In the illustrated example for the utterance 12 "Good morning," one candidate hypothesis H generated as output 222 from the RNN-T decoder 220 may include a first sequence of hypothesized sub-word units [_go, od, _mor, ning], and another candidate hypothesis H may include a second sequence of hypothesized sub-word units [_go, od, _morn, ing]. Simultaneously, the encoder 210 generates an encoding e where T conveys the acoustic context corresponding to multiple acoustic frames x segmented from the utterance 12. 1:T Generate a sequence of 212.

[0047] A sub-word unit (y i ) for each confidence output c(y i ) 302, a confidence embedding b(y i ) 242 is provided as input to the CEM 300. In the example shown, the i-th subword unit corresponds to the WP “ning”. Here, the confidence embedding b(y i ) 242 is the input subword + position embedding Emb(y i ) features (e.g., Emb(ning)), log-posterior log(p(y i )) feature and one or more of the top K(i) features. The log-posterior log(p(y1)) feature conveys the log-posterior log(p(y1)) feature for the subword unit y icontains the WP "ning", and the top K(i) features indicate the K largest log-probabilities at decoder index (e.g., time step) i. In other words, the top K(i) features provide a probability value for each candidate hypothesis H among the top K at decoder index (e.g., time step) i. Because both speech recognizer 200 and CEM 300 are configured to generate output at the sub-word (e.g., WP) level for each time step, implementing CEM 300 as a transformer allows for: (1) generating a sub-word (e.g., WP) output for every word c word (w j ) confidence output c(y j , Q j ) as a dedicated word-level confidence 302, it is possible to (1) use the word edit distance output as ground truth training labels (i.e., label modifications, substitutions, and insertions), and (2) incorporate information / features from all WPs that construct the word. In the example shown, the self-attention mechanism 310 of the transformer-based CEM 300 calculates the confidence embedding b(y i ) 242, and confidence embeddings for earlier subword units in the same word, we apply self-attention to the confidence feature vector b. The confidence feature vector b can be expressed as follows: b = {b(y1), ..., b(y i )} (2)

[0048] Furthermore, the acoustic mutual attention mechanism 320 of the transformer-based CEM 300 calculates the confidence output c(y i To improve the accuracy of predicting the ensemble of the speech components e, e ) 302, we use the acoustic mutual attention (CA(e)) to estimate the speech components e, e ) 302 output from the encoder 210. 1:T212 sequence to generate an acoustic context vector 322. As used herein, the terms "acoustic context vector" and "acoustic feature vector" can be used interchangeably. Finally, a softmax output layer 340 uses the self-attention confidence feature vector SA(b) and the cross-attention CA(e) acoustic context vector 322 to generate the [f(y1),…,f(y M )]=Transformer(CA(e),SA(b)) (3) [c(y i ),i(y i ),s(y i )]=Softmax(f(y1)) (4) The dedicated confidence output c(y i ) 302, and the softmax output layer 340 is trained with the ground truth WP labels of the modifications, substitutions, and insertions associated with the WP edit distance. The CEM 300 generates the confidence output c(y j , Q j ) to create the word c word (w j ) and determine the confidence 302 for each word, and calculate the word accuracy rate as follows:

[0049]

number

[0050] can be estimated. c word (w j )=c(y j , Q j ) (5)

[0051]

number

[0052] The word c in the example shownword The confidence 302 for (morning) corresponds to the confidence output c(ning) of the final WP building the word.

[0053]

number

[0054] contains one type of speech level confidence score.

[0055] The CEM 300 can be trained jointly with the speech recognizer 200, or the CEM 300 and the speech recognizer can be trained separately from each other. In some examples, the CEM 300 is trained using a binary cross-entropy word-level loss as follows:

[0056]

number

[0057] Table 1 shows that when the hypothesis word matches the reference word, the word w j If the Levenshtein word-level edit distance for d(w j ) is equal to 1, and if the hypothesis word does not match the reference word, the word w j If the Levenshtein word-level edit distance for d(w j ) is equal to zero. Notably, because the speech recognizer 200 and CEM 300 output at the subword level (e.g., output all WPs), Table 1 also shows that the CEM 300 applies the end of word mask loss m to focus only on the final WPs that make up a word, and ignores WP losses associated with earlier WPs that make up the same word.

[0058] The CEM can be further trained on the utterance level loss using the following ground truth:

[0059]

number

[0060] Here, if there is a deletion, the value of e becomes zero and the signal is back-propagated to the internal features of the CEM 300. To extract the utterance features and make predictions, the hierarchical attention mechanism 380 performs the following u i =tanh(W1f(y i )+b) (9)

[0061]

number

[0062] Implemented and s utt The parameters W1, b, and w2 that generate the binary cross-entropy loss L=-[e log s utt + (1 - e) log(1 - s utt )], where s utt provides an estimate of the probability that the entire utterance will be recognized with zero WER, which is useful for ranking utterances. utt does not provide a raw WER estimate. Intuitively, the utterance feature s utt is the token feature y1~y M The MLP of the hierarchical attention mechanism 380 includes two layers, which may include hidden and output dimensions of 320 and 1, respectively.

[0063] CEM can further estimate the raw word error rate (WER) of the hypothesis H based on the estimated number of deletions, where the tertiary task is to estimate the ground truth e as the number of deletions between the j-1 and j words. jIn general, the number of deletions at any position can include any non-negative integer. To extract deletion features and make predictions, r w (w j )=MLP(f(w j )) (12)

[0064]

number

[0065] We can train an MLP using the Poisson regression loss as follows: (f(w j )) is generated using self-attention and contains linguistic information up to the jth word. Therefore, the number of deletions e just before the jth word j is suitable for predicting

[0066] Therefore, the WER estimate can be calculated as follows:

[0067]

number

[0068] 4 illustrates a schematic diagram 400 of an exemplary confidence-based routine for selecting an appropriate speech recognizer to transcribe an utterance 12. In the illustrated example, a first speech recognizer 200 is operating as a default speech recognizer for generating a transcription 204 by processing incoming acoustic data 202 corresponding to an utterance 12 spoken by a user 10. The first speech recognizer 200 may correspond to a local speech recognizer executing on a user device 110 associated with the user 10. The first speech recognizer 200 also implements a CEM 300 for determining an utterance-level confidence score 350 for a speech recognition result 222 corresponding to the transcription 204 output by the first speech recognizer 200.

[0069] In some implementations, the confidence-based routine determines whether the utterance-level confidence score 350 for the utterance 12 transcribed by the first speech recognizer 200 satisfies a confidence threshold. In the example shown, an utterance-level confidence score 350 greater than the confidence threshold satisfies the confidence threshold, while an utterance-level confidence score 350 less than or equal to the confidence threshold does not satisfy the confidence threshold. If the utterance-level confidence score 350 satisfies (e.g., is greater than) the confidence threshold (e.g., decision block 450 is “yes”), the rendering 204 produced by the first speech recognizer 200 is accepted, achieving on-device gains in quality, latency, and reliability. Here, the accepted representation 204 can be displayed on the user device 110, or can continue to be displayed on the user device 110, and / or can be passed to a downstream natural language understanding (NLU) module for translation and, if necessary, for performing associated actions / operations.

[0070] If the utterance-level confidence score 350 does not satisfy (e.g., is less than) the confidence threshold (e.g., decision block 450 returns "no"), the confidence-based routine rejects the transcription 204 generated by the first speech recognizer 200 and passes the acoustic data 202 to the second speech recognizer 402 for processing to retranscribe the utterance 12. The transcription 204 generated by the second speech recognizer 402 can be returned to the user device 110 and / or to a downstream NLU module for translation. In an example where the first speech recognizer 200 is local and runs on the device 110, and the second speech recognizer 402 is server-side and runs on a remote server 410, the confidence-based routine causes the user device 110 to send the acoustic data 202 over a network (not shown) to the remote server 410 so that the second speech recognizer 402 running on the remote server 410 can transcribe the utterance 12. The second speech recognizer 402 may utilize a large language model trained on large language model training data to make the second speech recognizer 402 better suited to recognizing proper nouns or to recognize fewer common words that were not present in the training data used to train the first speech recognizer 200.

[0071] In some examples, the first speech recognizer 200 will typically be more accurate (e.g., achieve a lower word error rate) at recognizing short-form utterances than the second speech recognizer 402, which typically implements a larger language model and lexicon, and the first speech recognizer 200 may ultimately be less accurate at recognizing long-tail utterances than the second speech recognizer 402. Thus, a confidence-based routine may generate transcriptions 204 and transcribe most utterances on the device 110 using the first speech recognizer 200, and send all utterances to the second speech recognizer 402 with a confidence score 350 below the confidence threshold for quality, latency, and reliability.

[0072] While the second speech recognizer 402 is shown as server-side, the second speech recognizer 402 may also execute on a device. For example, the second speech recognizer 402 may be associated with a more computationally intensive speech recognizer that can generate more accurate speech recognition results for a particular utterance than the first speech recognizer 200, but at the expense of reduced latency and increased power consumption. Thus, a confidence-based routine may utilize the second speech recognizer 402 to transcribe the utterance 12 if the utterance-level confidence score associated with the recognition result generated by the first speech recognizer 200 is below a confidence threshold.

[0073] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0074] Non-transitory memory may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. Non-transitory memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0075] 5 is a flowchart of an exemplary arrangement of operations for a method 500 of predicting word-level confidence for a word recognized by a speech recognizer using only the confidence of the final hypothesis subword unit for that word. Data processing hardware 112 (FIG. 1) can execute instructions stored on memory hardware 114 (FIG. 1) to perform the exemplary arrangement of operations for method 500. Method 500 includes, at operation 502, receiving a speech recognition result 232 from a first speech recognizer 200 corresponding to a representation 204 of an utterance 12 spoken by user 10. First speech recognizer 200 can be configured with the two-pass decoding architecture discussed above with reference to FIGS. 2A and 2B. Here, speech recognition result 232 includes a sequence of hypothesis subword units forming one or more words of utterance 12, which are the highest-scoring candidate hypotheses rescored by a rescoring decoder of speech recognizer 200, and each individual subword unit is output from speech recognizer 200 at a corresponding output step.

[0076] The method 500 performs operations 504, 506, 508, and 510 for each sub-word unit in the sequence of hypothesis sub-word units using the confidence prediction module (CEM) 300. At operation 504, the method 500 includes obtaining a respective confidence embedding 242 representing a set of confidence features associated with the corresponding output step when the corresponding sub-word unit is output from the first speech recognizer 200. At operation 506, the method 500 calculates the corresponding sub-word unit and the confidence embedding b(y)-b(y) obtained for any other sub-word units in the sequence of hypothesis sub-word units that follow the corresponding sub-word unit. 1-i ) for each confidence embedding b(y i ) 242. At operation 508, the method 500 generates a confidence feature vector SA(b) using a first attention mechanism 310 that is self-attached to the corresponding acoustic frame x segmented from the acoustic data 202 corresponding to the utterance 12. T and the associated acoustic codings e, e 1:T 252 sequences using a second attention mechanism 320 that is cross-attached to generate an acoustic context vector CA(e) 322. At operation 510, the method 500 includes generating, as output from the output layer 340 of the CEM 300, respective confidence output scores 302 for the corresponding subword units based on the confidence feature vector SA(b) and the acoustic feature vector CA(e) 322 received as input by the output layer of the CEM 300.

[0077] The method 500 includes, at operation 512, determining an utterance-level confidence score 350 for the utterance form based on the respective confidence output scores generated for each subword unit in the sequence of hypothesis subword units. If the utterance-level confidence score 350 is less than a confidence threshold, the operation may further include rejecting the utterance form 204 output by the first speech recognizer 200 and instructing the second speech recognizer 402 to process acoustic data 14 corresponding to the utterance to retranscribe the utterance. The second speech recognizer 402 may be more computationally intensive than the first speech recognizer 200 and therefore may be more accurate when performing speech recognition on the acoustic data 14. For example, the first speech recognizer 200 may run entirely on-device, providing near-zero latency streaming representation capabilities, while the second speech recognizer 402 may run on a remote server utilizing potentially limitless computing and memory resources to perform accurate speech recognition with reduced latency. Thus, a low speech level confidence score 350 in the representation 204 output by the on-device first speech recognizer 200 may serve as an indicator to invoke a more computationally intensive second speech recognizer 200. Without departing from the scope of this disclosure, the second speech recognizer 402 may also run on-device but may be more computationally intensive to improve speech recognition accuracy, at the expense of reduced latency and increased power consumption and computing.

[0078] 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are merely exemplary and do not limit the scope of the present invention as described and / or claimed herein.

[0079] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connecting to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connecting to a low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or mounted in any other suitable manner. Processor 610 is capable of processing instructions for execution within computing device 600, including instructions stored in memory 620 or on storage device 630 for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 coupled to high-speed interface 640. Other implementations may use multiple processors and / or multiple buses, along with multiple memories and types of memories, where appropriate. It is also possible to connect multiple computing devices 600 (eg, as a server bank, a group of blade servers, or a multi-processor system) where each device provides a portion of the required operations.

[0080] The memory 620 stores information non-transiently within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 620 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0081] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device or tape device, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 620, the storage device 630, or memory on the processor 610.

[0082] High-speed controller 640 manages bandwidth-intensive operations for computing device 600, while low-speed controller 660 manages less bandwidth-intensive operations. This allocation of roles is merely exemplary. In some implementations, high-speed controller 640 is coupled to memory 620, display 680 (e.g., via a graphics processor or accelerator), and high-speed expansion port 650, which can accept various expansion cards (not shown). In some implementations, low-speed controller 660 is coupled to storage device 630 and low-speed expansion port 690. Low-speed expansion port 690, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, positioning device, scanner, etc., or can be coupled to a networked device, such as a switch or router, for example, via a network adapter.

[0083] Computing device 600, as shown, can be implemented in many different forms. For example, computing device 600 can be implemented as a standard server 600a or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0084] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuitry, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs that can be executed and / or translated on a programmable system that includes at least one programmable processor, which may be a special-purpose or general-purpose programmable processor, coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0085] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0086] The processes and logic flows described herein may be implemented by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or be operatively coupled to receive data from or transfer data to them, or both. However, a computer need not necessarily have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0087] To provide for user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from devices used by the user, for example, by sending web pages to a web browser on the user's client device in response to a request received from the web browser.

[0088] Although a number of embodiments have been described above, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims. [Explanation of symbols]

[0089] 10 users 12 Spoken Utterances 14 Acoustic data 100 Speech environment, speech-enabled environment, system 110 User Devices 112 Data Processing Hardware 114 Memory Hardware 116 Acoustic subsystem, sound capture devices (e.g., microphones), speech output devices (e.g., speakers) 116a Sound capturing devices (e.g., microphones) 116b Speech output device (e.g., speaker) 118 Programs or Applications, Automated Assistant Applications 119 Responses 200 Speech recognition device, first speech recognition device 202 Streaming Acoustic Data 204 Streaming Representation 210 shared encoder, encoder 212 Output 220 RNN-T decoder 222 output 232 Speech recognition results / hypotheses 242 Confidence Embedding 252 Acoustic encoding e, e 1:T 300 Confidence Forecasting Module (CEM) 302 Confidence, confidence output c(y i ), confidence output score 310 Self-Attention Mechanism, First Attention Mechanism 320 Acoustic Mutual Attention Mechanism 322 Acoustic context vector, acoustic feature vector 340 Softmax Output Layer 350 Vocal Level Confidence Score 380 Hierarchical Attention Mechanism 400 Schematic of an exemplary confidence-based routine 402 Server-side speech recognition device, second speech recognition device 410 Remote Server 450 Decision Block A method for predicting word-level confidence for a word recognized by a speech recognizer using only the confidences of the final hypothesized subword units for the 500 words. 502 operation 600 Exemplary Computing Device 600a Standard Server 600b laptop computer 600c Rack Server System 610 processor 620 Memory, Non-transient Memory 630 Storage Devices 640 High-Speed ​​Interface / Controller 650 High-Speed ​​Expansion Port 660 Low-speed interface / controller 670 Slow Bus 680 display 690 Low-Speed ​​Expansion Port CA(e) Acoustic mutual attention, acoustic context vector, acoustic feature vector e acoustic encoding e 1:T acoustic encoding H candidate hypothesis SA(b) Confidence feature vector y1, y2, y3, …, y M Hypothesis subword unit

Claims

1. When executed on data processing hardware (112), the data processing hardware (112) receiving, from a first speech recognizer (200), a speech recognition result (232) corresponding to a representation (204) of an utterance (12) spoken by a user (10), the speech recognition result (232) including a sequence of hypothesis sub-word units forming one or more words of the representation (204) of the utterance (12), each sub-word unit being output from the first speech recognizer (200) in a corresponding output step; A confidence prediction module (300) is used to predict, for each sub-word unit in the sequence of hypothesized sub-word units: obtaining respective confidence embeddings (242) associated with the corresponding output sub-word units when the corresponding sub-word units are output from the first speech recognizer (200); generating a confidence feature vector (312) using the confidence embeddings (242) obtained for any other sub-word units in the sequence of hypothesis sub-word units that follow the corresponding sub-word unit and a first attention mechanism (310) that is self-associated with the respective confidence embeddings (242) for the corresponding sub-word units; generating acoustic context vectors (322) using a second attention mechanism (320) that correlates with a sequence of encodings (212) associated with corresponding acoustic frames segmented from acoustic data (202) corresponding to the utterance (12); generating, as output from an output layer (340) of the confidence prediction module (300), respective confidence output scores (302) for the corresponding sub-word units based on the confidence feature vectors (312) and the acoustic context vectors (322) received as input by the output layer (340) of the confidence prediction module (300); determining an utterance-level confidence score (350) for the idiom (204) of the utterance (12) based on the respective confidence output scores (302) generated for each sub-word unit in the sequence of hypothesis sub-word units; and performing an operation including The operation is using a hierarchical attention mechanism (380) to estimate the probability that the speech recognition result (232) for the utterance (204) will be correctly recognized; determining an utterance-level loss for the speech recognition result (232) based on an estimated probability that the speech recognition result (232) for the utterance (12) will be correctly recognized and a ground truth label indicating whether the utterance (12) was correctly recognized; training a confidence prediction model for the utterance level loss; The computer-implemented method (500) further comprises:

2. When executed on data processing hardware (112), the data processing hardware (112) receiving, from a first speech recognizer (200), a speech recognition result (232) corresponding to a representation (204) of an utterance (12) spoken by a user (10), the speech recognition result (232) including a sequence of hypothesis sub-word units forming one or more words of the representation (204) of the utterance (12), each sub-word unit being output from the first speech recognizer (200) in a corresponding output step; A confidence prediction module (300) is used to predict, for each sub-word unit in the sequence of hypothesized sub-word units: obtaining respective confidence embeddings (242) associated with the corresponding output sub-word units when the corresponding sub-word units are output from the first speech recognizer (200); generating a confidence feature vector (312) using the confidence embeddings (242) obtained for any other sub-word units in the sequence of hypothesis sub-word units that follow the corresponding sub-word unit and a first attention mechanism (310) that is self-associated with the respective confidence embeddings (242) for the corresponding sub-word units; generating acoustic context vectors (322) using a second attention mechanism (320) that correlates with a sequence of encodings (212) associated with corresponding acoustic frames segmented from acoustic data (202) corresponding to the utterance (12); generating, as output from an output layer (340) of the confidence prediction module (300), respective confidence output scores (302) for the corresponding sub-word units based on the confidence feature vectors (312) and the acoustic context vectors (322) received as input by the output layer (340) of the confidence prediction module (300); determining an utterance-level confidence score (350) for the idiom (204) of the utterance (12) based on the respective confidence output scores (302) generated for each sub-word unit in the sequence of hypothesis sub-word units; and performing an operation including The operation is estimating the number of deletions at each position in a sequence of words of said utterance (12); determining a deletion loss based on the estimated number of deletions and a ground truth number of deletions at each position in the sequence of words; training a confidence prediction model for the deletion loss; The computer-implemented method (500) further comprises:

3. When executed on data processing hardware (112), the data processing hardware (112) receiving, from a first speech recognizer (200), a speech recognition result (232) corresponding to a representation (204) of an utterance (12) spoken by a user (10), the speech recognition result (232) including a sequence of hypothesis sub-word units forming one or more words of the representation (204) of the utterance (12), each sub-word unit being output from the first speech recognizer (200) in a corresponding output step; A confidence prediction module (300) is used to predict, for each sub-word unit in the sequence of hypothesized sub-word units: obtaining respective confidence embeddings (242) associated with the corresponding output sub-word units when the corresponding sub-word units are output from the first speech recognizer (200); generating a confidence feature vector (312) using the confidence embeddings (242) obtained for any other sub-word units in the sequence of hypothesis sub-word units that follow the corresponding sub-word unit and a first attention mechanism (310) that is self-associated with the respective confidence embeddings (242) for the corresponding sub-word units; generating acoustic context vectors (322) using a second attention mechanism (320) that correlates with a sequence of encodings (212) associated with corresponding acoustic frames segmented from acoustic data (202) corresponding to the utterance (12); generating, as output from an output layer (340) of the confidence prediction module (300), respective confidence output scores (302) for the corresponding sub-word units based on the confidence feature vectors (312) and the acoustic context vectors (322) received as input by the output layer (340) of the confidence prediction module (300); determining an utterance-level confidence score (350) for the idiom (204) of the utterance (12) based on the respective confidence output scores (302) generated for each sub-word unit in the sequence of hypothesis sub-word units; and performing an operation including The operation is using a hierarchical attention mechanism (380) to estimate the probability that the speech recognition result (232) for the utterance (204) will be correctly recognized; determining an utterance-level loss for the speech recognition result (232) based on an estimated probability that the speech recognition result (232) for the utterance (12) will be correctly recognized and a ground truth label indicating whether the utterance (12) was correctly recognized; training a confidence prediction model for the utterance level loss; estimating the number of deletions at each position in a sequence of words of said utterance (12); determining a deletion loss based on the estimated number of deletions and a ground truth number of deletions at each position in the sequence of words; training a confidence prediction model for the deletion loss; The computer-implemented method (500) further comprises:

4. determining the utterance level confidence score (350) for the expression (204) of the utterance (12), determining, for each word of the one or more words of the representation (204) of the utterance (12), a respective word level confidence score (302) equal to the respective confidence output score (302) generated for the last sub-word unit in the corresponding word; aggregating the respective word-level confidence scores (302) determined for each of the one or more words of the expression (204) to determine the utterance-level confidence score (350) for the expression (204) of the utterance (12); 4. The computer-implemented method of claim 1, comprising:

5. 5. The computer-implemented method of claim 1, wherein the confidence prediction model and the first speech recognizer (200) are jointly trained.

6. 6. The computer-implemented method of claim 1, wherein the operations further comprise rescoring candidate speech recognition results (232) recognized by the first speech recognizer (200) after training a confidence prediction model.

7. The computer-implemented method of claim 1 , wherein the sub-word units comprise word pieces.

8. The computer-implemented method of claim 1 , wherein the sub-word units comprise graphemes.

9. 9. The computer-implemented method of claim 1, wherein the first speech recognizer (200) includes a transformer-decoder model configured to generate a plurality of candidate hypotheses, each candidate hypothesis corresponding to a candidate representation for the utterance (12) and represented by a respective sequence of hypothesis subword units.

10. 10. The computer-implemented method of claim 9, wherein the transformer decoder model comprises a Recurrent Neural Network-Transducer (RNN-T) model architecture.

11. The operation is determining whether the utterance level confidence score (350) for the expression (204) of the utterance (12) satisfies a confidence threshold; the utterance level confidence score (350) for the expression (204) of the utterance (12) does not satisfy the confidence threshold; rejecting the idiom output from the first speech recognizer (200); instructing a second speech recognizer (402) to process acoustic data (202) corresponding to the utterance (12) spoken by the user (10) to retranscribe the utterance (12) spoken by the user; 11. The computer-implemented method of claim 1, further comprising:

12. the first speech recognition device (200) resides on a user device (110) associated with the user (10); the second speech recognizer (402) running on a remote server (410) in communication with the user device (110); 12. The computer-implemented method of claim 11, wherein the user device transmits the acoustic data to the remote server if the utterance level confidence score for the expression of the utterance does not satisfy the confidence threshold.

13. 13. The computer-implemented method of claim 11 or 12, wherein the second speech recognizer (402) is more computationally intensive than the first speech recognizer (200).

14. data processing hardware (112); memory hardware (114) in communication with the data processing hardware (112) and for storing instructions; the instructions, when executed on the data processing hardware (112), cause the data processing hardware (112) to: receiving, from a first speech recognizer (200), a speech recognition result (232) corresponding to a representation (204) of an utterance (12) spoken by a user (10), the speech recognition result (232) including a sequence of hypothesis sub-word units forming one or more words of the representation (204) of the utterance (12), each sub-word unit being output from the first speech recognizer (200) in a corresponding output step; A confidence prediction module (300) is used to predict, for each sub-word unit in the sequence of hypothesized sub-word units: obtaining respective confidence embeddings (242) associated with the corresponding output sub-word units when the corresponding sub-word units are output from the first speech recognizer (200); generating a confidence feature vector (312) using the confidence embeddings (242) obtained for any other sub-word units in the sequence of hypothesis sub-word units that follow the corresponding sub-word unit and a first attention mechanism (310) that is self-associated with the respective confidence embeddings (242) for the corresponding sub-word units; generating acoustic context vectors (322) using a second attention mechanism (320) that correlates with a sequence of encodings (212) associated with corresponding acoustic frames segmented from acoustic data (202) corresponding to the utterance (12); generating, as output from an output layer (340) of the confidence prediction module (300), respective confidence output scores (302) for the corresponding sub-word units based on the confidence feature vectors (312) and the acoustic context vectors (322) received as input by the output layer (340) of the confidence prediction module (300); determining an utterance-level confidence score (350) for the idiom (204) of the utterance (12) based on the respective confidence output scores (302) generated for each sub-word unit in the sequence of hypothesis sub-word units; and performing an operation including The operation is using a hierarchical attention mechanism (380) to estimate the probability that the speech recognition result (232) for the utterance (204) will be correctly recognized; determining an utterance-level loss for the speech recognition result (232) based on an estimated probability that the speech recognition result (232) for the utterance (12) will be correctly recognized and a ground truth label indicating whether the utterance (12) was correctly recognized; training a confidence prediction model for the utterance level loss; The system (100) further includes:

15. data processing hardware (112); memory hardware (114) in communication with the data processing hardware (112) and for storing instructions; the instructions, when executed on the data processing hardware (112), cause the data processing hardware (112) to: receiving, from a first speech recognizer (200), a speech recognition result (232) corresponding to a representation (204) of an utterance (12) spoken by a user (10), the speech recognition result (232) including a sequence of hypothesis sub-word units forming one or more words of the representation (204) of the utterance (12), each sub-word unit being output from the first speech recognizer (200) in a corresponding output step; A confidence prediction module (300) is used to predict, for each sub-word unit in the sequence of hypothesized sub-word units: obtaining respective confidence embeddings (242) associated with the corresponding output sub-word units when the corresponding sub-word units are output from the first speech recognizer (200); generating a confidence feature vector (312) using the confidence embeddings (242) obtained for any other sub-word units in the sequence of hypothesis sub-word units that follow the corresponding sub-word unit and a first attention mechanism (310) that is self-associated with the respective confidence embeddings (242) for the corresponding sub-word units; generating acoustic context vectors (322) using a second attention mechanism (320) that correlates with a sequence of encodings (212) associated with corresponding acoustic frames segmented from acoustic data (202) corresponding to the utterance (12); generating, as output from an output layer (340) of the confidence prediction module (300), respective confidence output scores (302) for the corresponding sub-word units based on the confidence feature vectors (312) and the acoustic context vectors (322) received as input by the output layer (340) of the confidence prediction module (300); determining an utterance-level confidence score (350) for the idiom (204) of the utterance (12) based on the respective confidence output scores (302) generated for each sub-word unit in the sequence of hypothesis sub-word units; and performing an operation including The operation is estimating the number of deletions at each position in a sequence of words of said utterance (12); determining a deletion loss based on the estimated number of deletions and a ground truth number of deletions at each position in the sequence of words; training a confidence prediction model for the deletion loss; The system (100) further includes:

16. data processing hardware (112); memory hardware (114) in communication with the data processing hardware (112) and for storing instructions; the instructions, when executed on the data processing hardware (112), cause the data processing hardware (112) to: receiving, from a first speech recognizer (200), a speech recognition result (232) corresponding to a representation (204) of an utterance (12) spoken by a user (10), the speech recognition result (232) including a sequence of hypothesis sub-word units forming one or more words of the representation (204) of the utterance (12), each sub-word unit being output from the first speech recognizer (200) in a corresponding output step; A confidence prediction module (300) is used to predict, for each sub-word unit in the sequence of hypothesized sub-word units: obtaining respective confidence embeddings (242) associated with the corresponding output sub-word units when the corresponding sub-word units are output from the first speech recognizer (200); generating a confidence feature vector (312) using the confidence embeddings (242) obtained for any other sub-word units in the sequence of hypothesis sub-word units that follow the corresponding sub-word unit and a first attention mechanism (310) that is self-associated with the respective confidence embeddings (242) for the corresponding sub-word units; generating acoustic context vectors (322) using a second attention mechanism (320) that correlates with a sequence of encodings (212) associated with corresponding acoustic frames segmented from acoustic data (202) corresponding to the utterance (12); generating, as output from an output layer (340) of the confidence prediction module (300), respective confidence output scores (302) for the corresponding sub-word units based on the confidence feature vectors (312) and the acoustic context vectors (322) received as input by the output layer (340) of the confidence prediction module (300); determining an utterance-level confidence score (350) for the idiom (204) of the utterance (12) based on the respective confidence output scores (302) generated for each sub-word unit in the sequence of hypothesis sub-word units; and performing an operation including The operation is using a hierarchical attention mechanism (380) to estimate the probability that the speech recognition result (232) for the utterance (204) will be correctly recognized; determining an utterance-level loss for the speech recognition result (232) based on an estimated probability that the speech recognition result (232) for the utterance (12) will be correctly recognized and a ground truth label indicating whether the utterance (12) was correctly recognized; training a confidence prediction model for the utterance level loss; estimating the number of deletions at each position in a sequence of words of said utterance (12); determining a deletion loss based on the estimated number of deletions and a ground truth number of deletions at each position in the sequence of words; training a confidence prediction model for the deletion loss; The system (100) further includes:

17. determining the utterance level confidence score (350) for the expression (204) of the utterance (12), determining, for each word of the one or more words of the representation (204) of the utterance (12), a respective word level confidence score (302) equal to the respective confidence output score (302) generated for the last sub-word unit in the corresponding word; aggregating the respective word-level confidence scores (302) determined for each of the one or more words of the expression (204) to determine the utterance-level confidence score (350) for the expression (204) of the utterance (12); 17. The system of any one of claims 14 to 16, comprising:

18. 18. The system of claim 14, wherein the confidence prediction model and the first speech recognizer (200) are jointly trained.

19. 19. The system of claim 14, wherein the operations further comprise rescoring candidate speech recognition results (232) recognized by the first speech recognizer (200) after training a confidence prediction model.

20. 20. The system of claim 14, wherein the sub-word units comprise word pieces.

21. 21. The system of claim 14, wherein the sub-word units comprise graphemes.

22. 22. The system of claim 14, wherein the first speech recognizer (200) comprises a transformer-decoder model configured to generate a plurality of candidate hypotheses, each candidate hypothesis corresponding to a candidate representation form for the utterance (12) and represented by a respective sequence of hypothesis subword units.

23. 23. The system of claim 22, wherein the transformer decoder model comprises a Recurrent Neural Network-Transducer (RNN-T) model architecture.

24. The operation is determining whether the utterance level confidence score (350) for the expression (204) of the utterance (12) satisfies a confidence threshold; the utterance level confidence score (350) for the expression (204) of the utterance (12) does not satisfy the confidence threshold; rejecting the idiom output from the first speech recognizer (200); instructing a second speech recognition device (402) to process acoustic data (202) corresponding to the utterance (12) spoken by the user (10) to retranscribe the utterance (12) spoken by the user (10); 24. The system of any one of claims 14 to 23, further comprising:

25. the first speech recognition device (200) resides on a user device (110) associated with the user (10); the second speech recognizer (402) running on a remote server (410) in communication with the user device (110); 25. The system of claim 24, wherein the user device transmits the acoustic data to the remote server if the utterance level confidence score for the expression of the utterance does not satisfy the confidence threshold.

26. 26. The system of claim 24 or 25, wherein the second speech recognizer (402) is more computationally intensive than the first speech recognizer (200).

Images