Context aware neural confidence estimation for rare word speech recognition

Abstract

An automatic speech recognition (ASR) system includes an ASR model (200), a neural associative memory (NAM) bias model (240), and a confidence estimation model (CEM) (300). The ASR model comprises: an audio encoder (205) configured to encode a sequence of audio frames (202) characterizing a spoken utterance into a sequence of higher order feature representations (252); and a decoder (230) configured to receive the sequence of high-order feature representations and output a final speech recognition result (232). The NAM bias model is configured to receive bias contextual information (242) and modify the sequence of higher order feature representations based on the bias contextual information to generate a bias scene vector (262) as output. The CEM is configured to calculate a confidence of the final speech recognition result output by the decoder (302). The CEM is connected to the bias context vector generated by the NAM bias model.

Description

Technical Field

[0001] The present disclosure relates to context-aware neural confidence estimation for rare word speech recognition. Background Art

[0002] Modern automatic speech recognition (ASR) systems focus not only on providing quality / accuracy (e.g., low word error rate (WER)), but also on providing low latency (e.g., a short delay between a user speaking and the transcription appearing). In addition, when ASR systems are used today, the ASR systems are required to decode the utterances as the user speaks in a streaming manner corresponding to displaying a transcription of the utterances in real time or even faster than real time. For example, when the ASR system is deployed on a mobile phone that experiences direct user interaction, an application on the mobile phone using the ASR system may require that the speech recognition is streamed so that the words, word fragments and / or single characters appear on the screen as soon as they are spoken. Here, the user of the mobile phone may also have a low tolerance for latency. Due to this low tolerance, speech recognition strives to operate on the mobile device in a manner that minimizes the impact of latency and inaccuracy that may adversely affect the user experience. Summary of the invention

[0003] One aspect of the present disclosure provides an automatic speech recognition (ASR) system, which includes an ASR model, a neural associative memory (NAM) bias model and a confidence estimation model (CEM). The ASR model includes: an audio encoder, which is configured to encode a sequence of audio frames representing spoken speech into a sequence of high-order feature representations; and a decoder, which is configured to receive a sequence of first high-order feature representations and output a final speech recognition result. The NAM bias model is configured to receive bias context information, and based on the bias context information, modify the sequence of high-order feature representations to generate a bias context vector as an output to encourage the decoder to recognize a bias phrase indicated by the bias context information. The CEM is configured to calculate the confidence of the final speech recognition result output by the decoder. The CEM is connected to the bias context vector generated by the NAM bias model.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the NAM neural bias model and the CEM are trained separately. For example, the CEM may be trained for training bias context vectors corresponding to high-order feature representations encoded by an audio encoder from input audio frames that have been modified by the NAM bias model based on bias context information extracted from training utterances for input to the NAM bias model. The bias phrases may include rare words and out-of-vocabulary (OOV) terms that are not used to train the audio encoder and decoder.

[0005] In some examples, the NAM bias model is initially trained on the basis of the ASR model, and the CEM is trained on the training data based on the initially trained NAM neural bias model. In these examples, the CEM can randomly sample context information from the training data.

[0006] The audio encoder may include a cascaded encoder architecture with a causal encoder and a non-causal encoder. The audio encoder may include a stack of multi-head attention layers, such as a stack of conformer layers or transformer layers. Additionally or alternatively, the CEM may include a multi-head attention block with self-attention and acoustic cross-attention.

[0007] Another aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations, the operations comprising receiving a sequence of acoustic frames representing an utterance spoken by a user, receiving bias context information, and performing speech recognition on the sequence of acoustic frames using an automatic speech recognition (ASR) model in the following manner: generating a sequence of high-order feature representations by an audio encoder of the ASR model, each high-order feature representation being associated with a corresponding acoustic frame in the sequence of acoustic frames; modifying the sequence of high-order feature representations by a neural associative memory (NAM) bias model using the bias context information to generate a corresponding sequence of bias context vectors; and generating, by a decoder of the ASR model, based on the bias context vectors, a sequence of hypothesized subword units corresponding to a final speech recognition result of the utterance, the sequence of hypothesized subword units forming one or more words of the utterance. Here, each hypothesized subword unit is output from the decoder at a corresponding output step. Based on the bias context vector, the operations further include generating a corresponding confidence output score for each corresponding sub-word unit in the sequence of hypothesized sub-word units using a confidence estimation model (CEM).

[0008] This aspect may include one or more of the following optional features. In some implementations, the bias context vector encourages the decoder to recognize any bias phrases indicated by the bias context information, and the bias phrases include rare words and out-of-vocabulary (OOV) terms that are not used to train the audio encoder and decoder. In some examples, the NAM neural bias model and the CEM are trained separately. For example, the CEM can be trained for training bias context vectors corresponding to high-order feature representations encoded by the audio encoder from input audio frames that have been modified by the NAM bias model based on bias context information extracted from training utterances for input to the NAM bias model.

[0009] In some examples, the NAM bias model is initially trained on the basis of the ASR model, and the CEM is trained on the training data based on the initially trained NAM neural bias model. In these examples, the CEM can randomly sample context information from the training data.

[0010] The audio encoder may include a cascaded encoder architecture having a causal encoder and a non-causal encoder. The audio encoder may include a stack of multi-head attention layers, such as a stack of conformer layers or transformer layers.

[0011] In some implementations, the operations further include: for each word in one or more words formed by the sequence of hypothesized subword units, determining a corresponding word-level confidence score for the word, the corresponding word-level confidence score being equal to a corresponding confidence output score generated for the last subword unit in the word; and determining an utterance-level confidence score for the speech recognition result by aggregating the corresponding word-level confidence scores determined for the one or more words of the utterance. The subword units may include word fragments.

[0012] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic diagram of an example speech environment.

[0014] Figure 2 is a schematic diagram of an example speech recognizer utilizing a context-aware confidence estimation model and a neural associative memory (NAM) bias model.

[0015] Figure 3 is a schematic diagram of an example context-aware estimation model and NAM bias model superimposed on a speech recognizer.

[0016] Figure 4 is a schematic diagram of an example confidence-based routine for selecting an appropriate speech recognizer based on utterance-level confidence.

[0017] Figure 5 is a flow diagram of an example operational arrangement of a method for generating confidence output scores for corresponding sub-word units in a sequence of hypothesized sub-word units.

[0018] Figure 6 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0019] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0020] Automatic speech recognition (ASR) systems focus not only on providing quality / accuracy (e.g., low word error rate (WER)), but also on providing low latency (e.g., a short delay between the user speaking and the transcription appearing). Recently, end-to-end (E2E) ASR models, such as recurrent neural network transducers (RNN-T), transformer or conformer transformers, and attention-based encoder-decoder models have become increasingly popular in achieving state-of-the-art performance in terms of accuracy and latency. Compared to conventional hybrid ASR systems that include separate acoustic, pronunciation, and language models, E2E models adopt a sequence-to-sequence approach to jointly learn acoustic and language modeling in a single neural network that is trained end-to-end from training data (e.g., utterance-transcription pairs).

[0021] Examples of sequence-to-sequence models include "attention-based" models and "listen-attend-spell" (LAS) models. The LAS model uses a listener component, an attender component, and a speller component to transcribe speech utterances into characters. Here, the listener is a recurrent neural network (RNN) encoder that receives an audio input (e.g., a time-frequency representation of a speech input) and maps the audio input to a higher-level feature representation. The attender focuses on higher-level features to learn the alignment between input features and predicted sub-word units (e.g., graphemes or word fragments). The speller is an attention-based RNN decoder that generates a sequence of characters from the input by generating a probability distribution over a set of hypothesized words. Using an integrated structure, all components of the model can be jointly trained as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is entirely built by a neural network. The full neural network runs without external and / or manually designed components (e.g., finite state converters, dictionaries, or text normalization modules). Furthermore, when training E2E models, these models typically do not require bootstrapping from decision trees or temporal alignment from a separate system.

[0022] In addition, when using ASR systems today, the ASR system is required to decode the speech in a streaming mode corresponding to the transcription of the speech displayed in real time or even faster than real time when the user speaks. For example, when the ASR system is deployed on a mobile phone that experiences direct user interaction, the application on the mobile phone using the ASR system may require that speech recognition is streamed so that as soon as a word, word fragment and / or single character is spoken, the word, word fragment and / or single character appears on the screen. Here, the user of the mobile phone may also have a low tolerance for delay. Due to this low tolerance, speech recognition strives to run on the mobile device in a way that minimizes the impact of delays and inaccuracies that may adversely affect the user experience. However, sequence-to-sequence models such as LAS models work by checking the entire audio input sequence before generating the output text, but do not allow streaming output when receiving the input. Due to this defect, deploying LAS models for speech applications that are sensitive to delays and / or require real-time voice transcription may cause problems. This makes the LAS model alone not an ideal model for mobile technologies (e.g., mobile phones) that usually rely on real-time applications (e.g., real-time communication applications).

[0023] Another form of sequence-to-sequence model called recurrent neural network transducer (RNN-T) does not use an attention mechanism, and unlike other sequence-to-sequence models that usually need to process the entire sequence (e.g., audio waveform) to generate output (e.g., sentence), RNN-T continuously processes input samples and streams output symbols, a feature that is particularly attractive for real-time communication. For example, speech recognition using RNN-T can output the characters spoken one by one. Here, RNN-T uses a feedback loop that feeds back the symbols predicted by the model to itself to predict the next symbol. Because decoding RNN-T includes beam searching through a single neural network rather than a large decoder graph, RNN-T can be scaled to a portion of the size of a server-based speech recognition model. As the size is reduced, RNN-T can be fully deployed on the device and can be run offline (i.e., without a network connection), thus avoiding the problem of unreliable communication networks.

[0024] However, the RNN-T model alone still lags behind large state-of-the-art traditional models (e.g., server-based models with separate AM, PM, and LM) in terms of quality (e.g., speech recognition accuracy). However, the non-streaming E2E LAS model has speech recognition quality comparable to large state-of-the-art conventional models. To leverage the quality of the non-streaming E2E LAS model, a two-pass speech recognition system (e.g., Figure 2A), the two-pass speech recognition system includes a first-pass component of the RNN-T network, followed by a second-pass component of the LAS network. With this design, the two-pass model benefits from the streaming nature of the low-latency RNN-T model, while improving the accuracy of the RNN-T model by combining the second pass of the LAS network. Although the LAS network increases latency compared to the RNN-T model alone, the increase in latency is quite slight and meets the latency constraints of device operation. In terms of accuracy, the two-pass model achieves a 17% to 22% WER reduction when compared to the RNN-T alone, and has a similar WER compared to large traditional models.

[0025] The accuracy of an ASR system can be affected by various factors, such as the speaker's accent, background noise, recording channel, spoken content, etc. Confidence scores can be used to measure the reliability of ASR output and predict recognition errors. Confidence scores are an important feature of an ASR system that supports many downstream applications to mitigate speech recognition errors. For example, unlabeled speech with recognition results that achieve high confidence output from an ASR model can be used for semi-supervised training of an ASR model, which can reduce the cost of training using only transcribed speech. On the other hand, in applications such as oral dialogue systems in which a user interacts with a digital assistant executed on a computing device, speech with recognition results that achieve low word-level confidence can prompt the user to correct any incorrectly transcribed words. In addition, recognition results with low confidence can result in the audio of the corresponding speech being passed to a different ASR model (e.g., server-side) to improve recognition of the speech.

[0026] While traditional hybrid ASR systems can easily estimate word-level confidence scores based on the word posterior probabilities computed from lattices or confusion networks and then aggregate them to provide utterance-level confidence, the deep neural networks employed by E2E ASR models tend to exhibit overconfidence when predicting words. Since many E2E ASR models are configured to output recognition results at the subword level, simply using the corresponding fixed subword tokenization of the word as the reference sequence to learn the confidence scores of each subword recognized by the ASR model may result in incorrect ground truth labels for training the confidence estimation model because the recognition results may contain multiple valid tokenizations. For example, the reference fixed subword sequence for the utterance "Good morning (早上好)" could be "go, od, morn, ing", while the hypothesized subword sequence recognized by the ASR model could be "go, od, mor,ning, mom". Here, although the word "morning (早上)" is correctly recognized by the ASR model, the subword labels for the corresponding hypothesized subwords "mor" and "ning" recognized by the ASR model would be marked as incorrect because they do not match the corresponding reference fixed subwords "morn" and "ing" of the word "morning (早上)".

[0027] To mitigate the drawbacks associated with estimating the subword confidence scores of the hypothesized subword sequence recognized by the ASR model due to mismatches between reference fixed subword sequences, the ASR model can utilize a confidence estimation model (CEM) that applies self-attention in order to estimate the word-level confidence of each recognized word using only the confidence of the final hypothesized subword units that make up the corresponding word recognized by the ASR model. The CEM disclosed herein is additionally capable of utilizing cross-attention to attend to the acoustic context of the recognized utterance as well as the linguistic context of multiple hypotheses (e.g., the N-best list) recognized by the ASR model during the first pass. It is evident that leveraging both acoustic and linguistic cross-attention can improve the accuracy of the confidence scores estimated by the CEM.

[0028] Notably, traditional CEMs do not depend on the decoder features of the ASR model's decoder. To produce reliable confidence scores consistent with the output produced by the decoder, the CEM can be trained on the same training data used to train the E2E ASR model, but additional augmentation is required. However, one drawback of training the CEM independently of the decoder features is the inability to utilize the context information that is typically incorporated into the decoder to improve the ASR output quality. For example, techniques such as context bias and language model fusion have been incorporated into ASR modeling by inputting external context information to improve rare word recognition performance. Unfortunately, the CEM is unaware of the context information and is therefore unable to accurately predict the confidence scores of words that have been corrected by the context information.

[0029] To further improve the confidence accuracy of CEM, the implementation herein involves a context-aware CEM that is capable of considering context information when predicting confidence scores. Specifically, the implementation herein involves utilizing CEM and a neural associative memory (NAM) context bias model to tightly integrate context bias and confidence estimation into a single E2E ASR model. The NAM context bias model can incorporate context information into the encoder of the E2E model using a multi-head attention mechanism. By utilizing the NAM context bias model to provide context information for biasing as input to the encoder, the CEM can be trained based on biased encoder features output from the encoder and biased by the NAM context bias model. It is apparent that the resulting model learns to pay attention to context information to improve the confidence score output by the CEM for context-corrected words.

[0030] Figure 1 2 is an example speech environment 100 in which users 10 interact with user devices 110 through voice input. User devices 110 (also generally referred to as devices 110) include a computing device that is configured to capture sounds (e.g., streaming audio data) from one or more users 10 within the speech-enabled environment 100. Here, streaming audio data 202 may refer to spoken utterances of users 10 that serve as audible queries, commands to devices 110, or audible communications captured by devices 110. The speech-enabled system of device 110 may process the query or command by answering the query and / or causing the command to be executed.

[0031] The user device 110 may correspond to any computing device capable of receiving the audio data 202. Some examples of the user device 110 include, but are not limited to, a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a computer, a wearable device (e.g., a smart watch), a smart home appliance, an Internet of Things (IoT) device, a smart speaker / display, an in-vehicle infotainment system, etc. The user device 110 includes data processing hardware 112 and memory hardware 114 that communicates with the data processing hardware 112 and stores instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations. The user device 110 further includes an audio subsystem 116 having an audio capture device (e.g., a microphone) 116, 116a for capturing spoken utterances 12 within the speech-enabled environment 100 and converting the spoken utterances into electrical signals; and a speech output device (e.g., a speaker) 116, 116b for transmitting audible audio signals (e.g., as output audio data from the device 110). Although in the example shown, the user device 110 implements a single audio capture device 116a, the user device 110 may implement an array of audio capture devices 116a without departing from the scope of the present disclosure, wherein one or more capture devices 116a in the array may not physically reside on the user device 110, but may communicate with the audio subsystem 116. In the example shown, the user device 110 (e.g., using hardware 112, 114) implements a speech recognizer 200 that is configured to perform speech recognition on audio data 202 corresponding to an utterance 12 spoken by the user 10. Here, the audio capture device 116a is configured to capture sounds representing the utterance 12 and convert the sounds into audio data 202 associated with a digital format compatible with the speech recognizer 200. The digital format associated with the audio data 202 may correspond to an acoustic frame such as a Mel frame (e.g., a parameterized acoustic frame). For example, the parameterized acoustic frame corresponds to a log-mel filter bank energy.

[0032] Although Figure 1 The user device 100 is shown implementing the speech recognizer 200 to perform speech recognition on the device, but other implementations include a remote server 410 ( Figure 4 ), which implements the speech recognizer 200 by processing the audio data 202 sent by the user device 110 via the network and providing a transcription 204 of the audio data 202 back to the user device 110. In some additional implementations, the user device 110 utilizes the local speech recognizer 200 resident on the user device 110 and the server-side speech recognizer 402 executed on the remote server 410 ( Figure 4Here, the user device 110 may use the local speech recognizer 200 when a network connection is unavailable or for voice applications that are latency sensitive and / or require streaming transcription, and may utilize the server-side speech recognizer 402 when additional resources are needed to improve speech recognition accuracy, as described below with reference to Figure 4 Describe in more detail.

[0033] In some examples, user 10 interacts with a program or application 118 executing on user device 110 that uses speech recognizer 200. For example, Figure 1 A user 10 is depicted communicating with an automated assistant application 118. In this example, the user (e.g., Bill) 10 directs a query toward the automated assistant application 118 by speaking an utterance 12, “Call Maikel,” which is captured by the audio capture device 116a and converted into corresponding audio data 202 (e.g., as acoustic frames) for processing by a speech recognizer 200. In this example, the speech recognizer 200 transcribes the audio data 202 representing the utterance 12 into a transcription 204 (e.g., a textual representation of “Call Maikel”). Here, the automated assistant application 118 may apply natural language processing to the transcription 204 to generate a response 119 for output to the user 10 that conveys the message “Calling Maikel on his cell phone now.” Natural language processing generally refers to the process of interpreting written language (e.g., transcription 204) and determining whether the written language prompts any action. In this example, the assistant application 118 uses natural language processing to recognize that the utterance 12 spoken by the user 10 is intended to invoke the assistant application 118 to access the user 10's contact list and initiate a phone call to a contact named "Maikel", whereby the application 118 provides a response 119 indicating that a phone call has been initiated to Maikel's phone. That is, by recognizing these details using natural language processing, the assistant application 118 returns the response 119 to the user 12 as a synthesized speech representation for audible output through the audio output device 116a and / or as text for display on a screen in communication with the user device 110. In some examples, the user device 110 displays a transcription 204 of the utterance 12 spoken by the user 10 and the corresponding response 119 from the assistant application 118 on the screen as a dialog. In some configurations, the natural language processing can occur on a remote system in communication with the data processing hardware 112 of the user device 110.

[0034] In some examples, the speech recognizer 200 processes the incoming audio data 202 in real time to provide a streaming transcription 204. Here, the speech recognizer 200 is configured to generate a sequence of hypothetical sub-word units that constitute the words of the utterance 12 spoken by the user 10. The hypothetical sub-word units may include word fragments or single characters (e.g., graphemes). In the example shown, the sequence of hypothetical sub-word units recognized by the speech recognizer includes "SOS_call_mai kel", where 'SOS' indicates the start of a speech token, and each word boundary indicator ('_') indicates the start / starting sub-word unit of each word.

[0035] refer to Figure 2 In some implementations, the speech recognizer 200 is configured in a two-pass decoding architecture and implements both a neural associative memory (NAM) bias model 240 and a confidence estimation module (CEM) 300. The NAM bias model 240 is configured to receive bias context information 242 to encourage the speech recognizer 200 to recognize bias phrases indicated by the bias context information 242. The CEM 300 is configured to estimate the confidence 302 of the final speech recognition result 232 predicted by the speech recognizer 200. Notably, the speech recognizer 200 can utilize a two-pass architecture, wherein the final speech recognition result 232 corresponds to a series of sub-word units, such as word fragments or graphemes, which when aggregated together form a transcription 204 of an utterance. In general, the two-pass architecture of the speech recognizer 200 includes an encoder 205 and a decoder 230. More specifically, the encoder 205 includes a cascaded encoder 205 having a model structure, wherein the encoding path includes two encoders 210, 250, which are cascaded so that the output of one encoder 210 feeds the input of another decoder 230 before decoding. Here, the encoders 210, 250 can be cascaded regardless of the underlying architecture of each encoder. The first or shared encoder 210 can correspond to a causal encoder 210, and the second or acoustic encoder 250 can correspond to a non-causal encoder 250 cascaded with the causal encoder 210. The shared encoder 210 can include multiple unidirectional long short-term memory (LSTM) layers or multiple multi-head attention layers, such as a conformer layer or a transformer layer. Similarly, the acoustic encoder 250 can be overlaid on the causal encoder 210 and include multiple unidirectional LSTM layers or multiple multi-head attention layers, such as a conformer layer or a transformer layer.

[0036] In some implementations, the encoders 210, 250 include a stack of 512-dimensional conformer layers. Causal convolutions and left-context attention layers can be used for each conformer layer to strictly restrict the model from using future inputs. A multi-head (e.g., 8-head) attention mechanism can be used in the self-attention layer. The cascaded encoders 210, 260 may include 17 conformer layers. Here, the first encoder 210 may include 15 conformer layers, while the second encoder 250 may include two conformer layers that accept an additional right context (e.g., 5.04 seconds). Optionally, other types of layers such as transformer layers combined with self-attention mechanisms can be used instead of conformer layers.

[0037] In other implementations, one encoder is constructed with an LSTM structure, while the other encoder is constructed using a bidirectional LSTM layer or a conformer layer (e.g., a conformer-transducer). In other words, the encoders 210, 250 may have different architectures or similar architectures. For example, the cascade encoder 205 may be roughly similar to an acoustic model (AM) in a traditional ASR system, and may include a recurrent network of stacked long short-term memory (LSTM) layers. Here, the first encoder 210 is a streaming encoder including a unidirectional long short-term memory (LSTM) layer, while the second encoder 250 is a non-streaming encoder including a bidirectional LSTM layer or a conformer layer. In the cascade encoder 205 where both encoders 210, 250 include LSTM layers, the second encoder 250 receiving the output of the first encoder 210 may utilize the LSTM layer of the first encoder 210, so that the second encoder 250 includes fewer LSTM layers than the first encoder 210 (and fewer LSTM layers than a completely non-streaming model). By having fewer LSTM layers, the cascaded encoder 205 can reduce the number of more computationally expensive bidirectional layers, making the model 200 leaner than simply combining a traditional streaming model with a traditional non-streaming model.

[0038] At least one shared encoder 210 is configured to receive as input audio data 202 corresponding to an utterance 12 as a sequence of acoustic frames. The acoustic frames may have been previously processed by the audio subsystem 116 into parameterized acoustic frames (e.g., Mel frames and / or spectral frames). In some implementations, the parameterized acoustic frames correspond to log-Mel filter bank energies having log-Mel features. For example, a parameterized input acoustic frame representing the audio data 202 input to the encoder 210 may be represented as x=(x1, ..., x T ),in is the logarithmic Mel filter bank energy, T represents the number of frames in x, and d represents the number of logarithmic Mel features. In some examples, each parameterized acoustic frame includes 128-dimensional logarithmic Mel features calculated within a short shift window (e.g., 32 milliseconds and shifted every 10 milliseconds). Each feature can be stacked with a previous frame (e.g., the previous three frames) to form a higher dimensional vector (e.g., a 512-dimensional vector using three previous frames). The features forming the vector can then be downsampled (e.g., to a frame rate of 30 milliseconds).

[0039] For each acoustic frame x of the audio data 202 input to the encoder 210 1:T , the encoder 210 is configured to generate a first high-order feature representation as an output 212. The first high-order feature representation 212 output at each corresponding time step can be represented as e s Similarly, the second encoder 250 is cascaded to the first encoder 210 and is trained to receive the first high-order feature representation 212 e s as input, and produces a second high-level feature representation as output 252. This second high-level feature representation is denoted as e a Each frame number denoted by T in x corresponds to a corresponding time step. It is worth noting that the second encoder 250 is configured to be adapted to generate a first high-order feature representation e for each acoustic frame x as output 212 from the shared encoder 210 s , suitable for predicting the final speech recognition result 232 and the confidence 302 estimate of the CEM 300. Although the examples herein depict the first encoder 210 and the second encoder 250 being directly connected to and shared by the same decoder 230, other configurations of the speech recognizer 200 may include the first encoder 210 connected to the decoder 230, while the second encoder 250 is connected to a different decoder. Thus, in the example shown, the decoder 230 receives the first high-order feature representation e s and the second high-order feature representation e a Both as input.

[0040] The decoder 230 may include a recurrent neural network-transducer (RNN-T) architecture having a union layer 231 and a prediction network 233. The decoder 230 uses the union layer 231 to combine (i.e., when the model 200 operates in a non-streaming mode) the first high-order feature representation output by the cascade encoder 205. s and the second high-order feature representation e a , and the previous prediction y from the prediction network 233 r-1) to produce the decoder output. When the model 200 operates in streaming mode, the joint layer 231 receives the output of the prediction network 233 and only receives the first high-order feature representation e output from the first encoder 210. s The output of decoder 230 may be the current subword unit y i (Given a sequence of N previous non-whitespace symbols ) and the probability distribution on the input x Although not shown, the model 200 may also include a softmax layer that receives the output of the decoder 230. In some implementations, the softmax layer is separate from the decoder 230 and processes the output y from the decoder 230. r The output of the Softmax layer is then used in a beam search process to select orthogonal elements. In some implementations, the Softmax layer is integrated with the decoder 230 so that at each time step, the output 232 of the decoder 204 is r Represents the output of the Softmax layer.

[0041] Decoder 230 is configured to generate a probability distribution of possible speech recognition hypotheses at each time step. In other words, joint network 231 generates a probability distribution of possible speech recognition hypotheses at each output step (e.g., time step). Here, "possible speech recognition hypotheses" correspond to a set of output labels / symbols (also referred to as "speech units"), each of which represents a grapheme (e.g., symbol / character) or a word fragment in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, for example, one label for each of the 26 letters in the English alphabet, and one label specifies a space. Therefore, joint network 231 can output a set of values, which indicates the possibility of each of the predetermined set of output labels appearing. The set of values ​​can be a vector (e.g., a unique hot vector) and can indicate a probability distribution of a set of output labels. In some cases, the output label is a grapheme (e.g., a separate character, and possible punctuation and other symbols), but the set of output labels is not limited to this. For example, the set of output labels may include word fragments and / or entire words in addition to or in lieu of graphemes. Output labels may also be other types of speech units, such as phonemes or subphonemes. The output distribution of the joint network 231 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the joint network 231 may include 100 different probability values, one for each output label. The probability distribution may then be used (e.g., by a Softmax layer) to select candidate orthographic elements (e.g., graphemes, word fragments, and / or words) and assign scores thereto in a beam search process for use in determining the transcription 204.

[0042] Within the decoder 230, the prediction network 233 may have two 2,048-dimensional LSTM layers, each followed by a 640-dimensional projection layer, such that the LSTM-based prediction network may have approximately 23.4 million parameters. In other configurations, the prediction network 233 may alternatively include a conformer layer or a transformer layer in place of the LSTM layer. In yet other configurations, the prediction network 233 includes a V2 embedding lookup table that includes an embedding prediction network. At each time step, the V2 embedding lookup table may receive as input the first two predictions (e.g., one-hot vectors) output by the joint network 231, calculate a corresponding embedding d1, d2 for each of the first two predictions, and provide the concatenated output [d1, d2] to the joint layer 231. In contrast, the V2 embedding lookup table may have only approximately two (2) million parameters, while the LSTM-based prediction network may include approximately 23.4 million parameters. Finally, the joint network 231 may also be a single-layer neural network with 640 hidden units. The Softmax layer may be composed of a uniform set of word fragments or graphemes generated using all unique word fragments or graphemes in multiple training datasets. In some implementations, to limit the number of future contexts seen by the cascaded encoder model 200, the second encoder 250 uses a certain number of multi-head attention layers (e.g., two conformer layers) and a certain amount of correct context (e.g., five seconds of correct context).

[0043] During the streaming speech recognition mode, the decoder 230 receives the first high-level feature representation e generated by the first encoder 210 for each acoustic frame x as output 212 s as input, and generates one or more candidate hypotheses H as output in a streaming manner, each candidate hypothesis consisting of hypothesis subword units y1, y2, y3, ..., y M For example, in the example of user 10 saying "Good morning", one candidate hypothesis H may include a first sequence of hypothesized subword units [_go, od, _mor, ning], while another candidate hypothesis H may include a second sequence of hypothesized subword units [_go,od, _morn, ing]. There may be a large number of other candidate hypotheses H. Here, the hypothesized subword units y1, y2, y3, ..., y4, y5, y6, y7, y8, y9, y10, y11, y12, y13, y14, y15, y16, y17, y18, y19, y20, y21, y22, y23, y24, y25, y26, y27, y30, y31, y32, y33, y34, y35, y36, y37, y38, y39, y40, y41, y42, y43, y44, y45, y46, y47, y48, y49, y50, y51, y52, y53, y54, y55, y56, y57, y58, y59, y60, y61, y62, y63, y64, y7 M The corresponding sequence of y1, y2, y3, ..., y2 corresponds to the candidate transcription of utterance 12. Assume that the subword units y1, y2, y3, ..., y2 M Each subword unit y in each corresponding sequence of irepresents a probability distribution of possible sub-units. The sub-word unit with the highest probability in the probability distribution can be selected as the hypothetical sub-word in the corresponding sequence of hypothetical sub-word units.

[0044] In some implementations, speech recognizer 200 operates in a non-streaming mode. Non-streaming mode may occur, for example, in non-delay intensive applications, such as when user 10 is viewing a transcription of a voicemail left on his / her cell phone. As described above, this type of application benefits from using future context to improve performance (i.e., improve speech recognition accuracy) in exchange for increased processing time. Here, speech recognizer 200 first generates a first high-order representation e at each time step using first encoder 210. s to be input to the second encoder 250, but the decoder 230 does not represent the first high-level feature e s Therefore, by using the second high-order feature representation e a sequence, the decoder 230 can generate a more accurate final speech recognition result 232 at the expense of reduced latency.

[0045] As described above, the NAM bias model 240 receives bias context information 242 to encourage the speech recognizer 200 to recognize bias phrases indicated by the bias context information 242. Here, the bias context information 242 may include bias phrases, such as rare words or out-of-vocabulary (OOV) terms that are not present or scarce in the training data used to train the speech recognizer 200. The NAM bias model 240 is intended to adaptively (dynamically) edit recognition results based on the context information 242 received as text to improve recognition of rare words and OOV terms. The context bias information 242 and the bias phrases included therein may include names / addresses in a user's contact list, city / street names in a neighboring geographic area, and rare words / OOV terms previously used by the user, to name just a few non-limiting examples. It is worth noting that the NAM bias model 240 is able to store the token transformation chain 244 in a key-value memory and apply the attention mechanism 260 to traverse the transformation chain 244 and modify / bias the second high-order feature representation generated by the second encoder 250. a, the bias context information 242 is incorporated into the speech recognizer 200. More specifically, the NAM bias model 240 includes a context encoder that is configured to generate a bidirectional context embedding for each bias phrase indicated by the bias context information 242. Each bias phrase can be represented by a corresponding word fragment identifier that identifies the word fragments that form each bias phrase, so that the context encoder calculates a corresponding context embedding for each bias phrase. Thereafter, the neural associative memory is updated by storing associative transitions between word fragment subsequences of the same bias phrase to learn conditional transitions for new bias phrases. In the associative memory, each word fragment token in each phrase is used as a key to map to the next word fragment token (left shift), so that the key-value pair (k l , v l ) is constructed from two consecutive context embeddings. Thus, the associative memory creates a transformation chain 244 to allow the current word to be easily restored from the memory by traversing the bias phrase in a fine-grained manner using its previous element. The attention mechanism includes a multi-head attention mechanism that uses multiple heads in a single pass to represent the word by using the second high-order feature representation e a Instead of partially decoded text of the context audio representation to compute the relevant bias context, the attention mechanism 260 is enabled to learn across multimodal context.

[0046] The entire speech recognizer 200, including the cascaded encoder 205 and decoder 230, can be jointly trained with the NAM bias model 240 to ensure that the second high-order feature representation e a The biased context information 242 is modified / biased so as to encourage the decoder 230 to recognize the biased phrase (when present) indicated by the biased context information 242. In contrast, an external bias module that is typically added on top of a speech recognizer, such as an external language model utilizing a traditional finite state transducer (FST), while effectively biasing the output speech recognition results toward the biased phrase, is not easily integrated with the CEM 300 because the FST performs post-prediction editing. For example, the external bias module can utilize the context bias information to detect a rare word (such as Maikel) in the speech input “My name is Maikel” that the speech recognizer cannot recognize, but the CEM 300 will still assign a low confidence score because the CEM 300 will not be aware of the context bias information. Therefore, external bias modules are not suitable candidates for integration with CEM 300, as CEM 300 tends to underestimate the confidence of speech recognition results containing rare / OOV terms corrected by contextual bias information.

[0047] Based on the second high-order feature representation e generated by the second encoder 250 aand a transformation chain 244 associated with the bias context information 242 output from the NAM bias model 240, the attention mechanism 260 generates a bias context vector e via a projection layer (not shown) a biased 262. Biased context vector e a biased 262 is provided as input to the decoder 230 and the CEM 300. Here, the decoder 230 may bias the context vector e a biased 262 to decode to predict the final speech recognition result 232, which includes any bias phrases indicated by the bias context information 242 when present in the input audio data 202 corresponding to the utterance 12 spoken by the user 10. At the same time, the CEM 300 can use the bias context vector e in estimating the confidence 302 of the final speech recognition result 232 predicted by the decoder 230 of the speech recognizer 200. a biased 262.

[0048] In order to reduce the size of the softmax layer that performs beam search on the output of the decoder, the sub-word unit vocabulary of possible sub-word units is typically smaller than the word vocabulary. The sub-word unit vocabulary may include graphemes or word fragments (WP). An example WP vocabulary may include 4,096 WPs. Although the examples of the present disclosure use WP as the sub-word unit generated from the output of a speech recognizer, graphemes may be similarly utilized as the sub-word unit output from a speech recognizer without departing from the scope of the present disclosure. Therefore, in order to calculate the word error rate (WER) of a candidate hypothesis H, it is necessary to convert the corresponding sequence of hypothesized sub-word units (e.g., WP) into their corresponding word sequences w1, w2, ..., w L Since the first subword unit of each word (e.g., WP) begins with a word boundary indicator ('_'), this process for converting a subword sequence into a word sequence is uniquely determined. Similarly, for the word sequence including Q j WP word w j , where y j, q represents the qth WP of the jth word, and a simple technique for calculating word confidence can be expressed as follows.

[0049] (1)

[0050] where agg can be the arithmetic mean, minimum, product, or neural network. However, due to the mismatch between WP correctness and word correctness, each word w jcan be divided into multiple different valid WP combinations, so using Formula 1 to estimate word-level confidence will generate unnecessary computational burden during training because the tokenization with the least WP edits needs to be searched among all possible reference tokenizations. As used herein, WP edits include correct (cor) labels when the hypothesized WP matches the reference WP, replacement (sub) labels when the valid hypothesized WP does not match the reference WP, and insertions (ins) when the hypothesized WP is misidentified. Table 1 below shows an example where the word "morning" is correctly transcribed but results in two substitutions in the WP edit distance output.

[0051]

[0052] Table 1.

[0053] To address the inherent mismatch between WP correctness and word correctness produced by speech recognizer 200 output at the WP level as depicted in Table 1, implementations herein utilize CEM 300 to provide confidence output at the final WP of each word as word-level confidence while ignoring the confidences of all other previous WPs for each word. Figure 3 The CEM 300 uses the bias context vector e output from the NAM bias model 240. a biased 262 to incorporate an example of knowledge of any bias phrases that may be present in the final speech recognition results 232 output from the decoder 230 for which the CEM 300 is performing word-level confidence estimation. For clarity, Figure 3 Only when operating in non-streaming mode, the CEM 300 is depicted in the hypothetical WPs y1, y2, y3, ..., y2 representing the final recognition result 232 output from the decoder 230. M The confidence output c(y) of the predicted i-th WP in the corresponding sequence i ) 302 action. Specifically, Figure 3 Depicts the confidence output c("ning") of the CEM 300 prediction of the "ning" WP in the corresponding sequence [_go, od, _mor, ning] of the hypothesized WP converted to the corresponding word sequence "good morning".

[0054] The second encoder 250 generates a second high-level feature representation of the sequence e a 1:T252, where T corresponds to the number of acoustic frames x segmented from the utterance 12. At the same time, the NAM bias model 240 receives contextual bias information 242 including bias phrases, wherein each bias phrase can be represented by a corresponding word fragment identifier, which identifies the word fragments that form the bias phrase, so that the NAM bias model 240 calculates the corresponding context embedding of each bias phrase. Thereafter, the neural associative memory is updated by storing associative transformations between word fragment subsequences of the same bias phrase to learn conditional transformations for new bias phrases. In the associative memory, each word fragment token in each phrase is used as a key to map to the next word fragment token (left shift), so that the key-value pair (k l , v l ) is constructed from two consecutive context embeddings. Thus, the associative memory creates a transition chain 244 to allow the current word to be easily recovered from the memory by traversing the bias phrase in a fine-grained manner using its previous element.

[0055] The multi-head attention (MHA) mechanism 260 receives the transformation chain 244 and the second high-order feature representation e a 1:T 252 to generate the bias context vector e a biased 262. Biased context vector e a biased 262 is provided as input to the decoder 230 and the acoustic cross attention mechanism 320 of the CEM 300. Based on the bias context vector e combined with the context bias information 242 a biased 262, the decoder 230 is configured to determine a probability distribution of possible sub-word units at each time step in the plurality of time steps to provide hypothesized sub-word units p(y1), p(y2), p(y3), ..., p(y M ), whereby linear + softmax 235 can output the highest second-pass log probability The associated candidate hypothesis H is used as the final recognition result 232. Here, the final recognition result 232 includes a hypothesis subword (eg, WP) unit sequence [sos_go, od, _mor, ningeos], which has speech start (sos) and speech end (eos) markers.

[0056] For each subword unit (y ) in the hypothetical subword (eg, WP) unit sequence [sos_go, od, _mor,ning eos] representing the final recognition result 232, i ), embedding the confidence of the set of confidence features obtained from the speech recognizer 200 into b(y i) 301 is provided as input to CEM 300 to determine the corresponding confidence output c(y i ) 302. In the example shown, the i-th subword unit corresponds to WP “ning”. Here, the confidence embedding b(y i ) 301 conveys the softmax posterior feature of the decoder 230 , input subword embedding Emb(y i ) features (e.g., Emb(ning)), log-posterior log(p(y i )) features and one or more of the previous K(i) features. The softmax posterior feature indicates the internal features of WP “ning”, the bias context vector e a biased,1:T The internal features of 262 and the penultimate layer activation of decoder 230 The log-posteriori log(p(y1)) feature indicates the relationship between the subword unit y i The probability value associated with the probability / likelihood of WP "ning" is included, and the top K(i) features indicate the K largest log-probabilities at decoder index (e.g., time step) i. In other words, the top K(i) features provide a probability value for each candidate hypothesis H in the top K at decoder index (e.g., time step) i. Since the speech recognizer 200 and CEM 300 are configured to generate outputs for each time step at the sub-word (e.g., WP) level, implementing CEM 300 as a transformer allows: (1) by utilizing each word c word (w j ) is the confidence output c(y j , Q j ) as a dedicated word-level confidence 302, using the word edit distance output as the true training label; and (2) incorporating information / features from each WP that constitutes the word. In the example shown, the self-attention mechanism 310 of the transformer-based CEM 300 embeds b(y) based on the confidence of the i-th sub-word unit corresponding to the WP “ning”. i ) 301 and the confidence embedding of the earlier subword unit in the same word to apply self-attention to the confidence feature vector b. The confidence feature vector b can be expressed as follows.

[0057] (2)

[0058] b= (3)

[0059] In addition, the acoustic cross-attention mechanism 320 of the transformer-based CEM 300 applies acoustic cross-attention (CA(e)) to the bias context vector e a biased,1:T 262 to generate an acoustic context vector 322 to improve the corresponding confidence output c(y) of the estimated i-th subword unit corresponding to WP “ning” i ) 302. Finally, the linear+sigmoid block 340 uses the self-attention confidence feature vector SA(b) and the cross-attention CA(e) acoustic context vector 322 to allow the transformer-based CEM 300 to use the confidence output c(y at the final WP j , Q j ) for each word c word (w j ) generates a dedicated confidence 302 as shown below.

[0060] (4)

[0061] (5)

[0062] Among them, the word c in the example shown word The confidence 302 of (morning) corresponds to the confidence output c(ning) of the final WP constituting the word.

[0063] The implementation involves jointly training a speech recognizer 200 (i.e., a cascaded encoder 205 and a decoder 230) and a NAM bias model 240. Here, the training data for jointly training the speech recognizer 200 and the NAM bias model 240 may include multi-domain training utterances, each of which includes audio data that characterizes the training utterance paired with a corresponding transcription of the training utterance. In some examples, the CEM 300 is trained separately from the jointly trained speech recognizer 200 and the NAM bias model 240. In these examples, although the separately trained CEM is unaware of the training examples, more specifically, the second high-order feature representation 252 biased / modified by the NAM bias model 240 provides improved confidence accuracy of the results that have been corrected by the context information 242. It is noteworthy that the separately trained CEM can be trained on the basis of the same speech recognizer 200.

[0064] In other examples, the NAM bias model 240 is jointly trained with the pre-trained speech recognizer 200, and the CEM 300 is subsequently trained based on the jointly trained speech recognizer 200 and the NAM bias model 240. In these other examples, the CEM 300 is trained based on the jointly trained speech recognizer 200 and the NAM bias model 240 using a binary cross-entropy word-level loss, as shown below.

[0065] (6)

[0066] Table 1 shows that when the hypothesis word matches the reference word, word w j The Levenshtein word-level edit distance outputs the "correct" (cor) label d(w j ) is equal to one, and when the hypothesis word does not match the reference word, the word w j The Levenshten word-level edit distance outputs the "insert" (ins) or "replace" (sub) tag when d(w g ) is equal to zero. It is worth noting that since the speech recognizer 200 and the CEM 300 output at the sub-word level (e.g., per WP output), Table 1 also shows that the CEM 300 applies the word-end mask loss m to focus only on the final WP that constitutes the word and ignores the WP loss associated with the earlier WPs that constitute the same word.

[0067] Figure 4 A schematic diagram 400 of an example confidence-based routine for selecting an appropriate speech recognizer to transcribe an utterance 12 is shown. In the example shown, a first speech recognizer 200 operates as a default speech recognizer for generating a transcription 204 by processing incoming audio data 202 corresponding to an utterance 12 spoken by a user 10. The first speech recognizer 200 may correspond to a local speech recognizer executing on a user device 110 associated with the user 10. The first speech recognizer 200 also implements a NAM bias model 240 and a CEM 300 to incorporate external context information 242 in determining an utterance-level confidence score 350 for a speech recognition result 232 corresponding to the transcription 204 output by the first speech recognizer 200.

[0068] In some implementations, a confidence-based routine determines whether an utterance-level confidence score 350 of an utterance 12 transcribed by the first speech recognizer 200 satisfies a confidence threshold. In the example shown, an utterance-level confidence score 350 that is greater than the confidence threshold satisfies the confidence threshold, while an utterance-level confidence score 350 that is less than or equal to the confidence threshold fails to satisfy the confidence threshold. When the utterance-level confidence score 350 satisfies (e.g., is greater than) the confidence threshold (e.g., decision block 450 is "yes"), the transcription 204 generated by the first speech recognizer 200 is accepted to achieve on-device gains in quality, latency, and reliability. Here, the accepted transcription 204 can be displayed or continue to be displayed on the user device 110 and / or passed to a downstream natural language understanding (NLU) module to interpret the transcription 204 and perform related actions / operations if necessary.

[0069] When the utterance-level confidence score 350 fails to meet (e.g., is less than) the confidence threshold (e.g., decision block 450 is "no"), the confidence-based routine rejects the transcription 204 generated by the first speech recognizer 200 and passes the audio data 202 to the second speech recognizer 402 for processing to re-transcribe the utterance 12. The transcription 204 generated by the second speech recognizer 402 can be passed back to the user device 110 and / or a downstream NLU module for interpretation. In an example where the first speech recognizer 200 is local and executes on the device 110 and the second speech recognizer 402 is server-side and executes on a remote server 410, the confidence-based routine causes the user device 110 to send the audio data 202 to the remote server 410 via a network (not shown) so that the second speech recognizer 402 executing thereon can transcribe the utterance 12. The second speech recognizer 402 can utilize a large language model trained on large-scale language model training data, so that the second speech recognizer 402 is more suitable for recognizing proper nouns or less common words that are not present in the training data used to train the first speech recognizer 200 .

[0070] In some examples, while the first speech recognizer 200 is generally more accurate (e.g., achieves a lower word error rate) for recognizing short-form utterances than the second speech recognizer 402 that implements a larger language model and lexicon, the first speech recognizer 200 may ultimately be less accurate at recognizing long-tail utterances than the second speech recognizer 402. Therefore, the confidence-based routine may send all utterances having a confidence score 350 less than a confidence threshold to the second speech recognizer 402 to generate the transcription 204, and transcribe most utterances on the device 110 using the first speech recognizer 200 to obtain quality, latency, and reliability.

[0071] Although the second speech recognizer 402 is shown as being server-side, the second speech recognizer 402 may also be executed on the device. For example, the second speech recognizer 402 may be associated with a more computationally intensive speech recognizer that may generate more accurate speech recognition results for certain utterances than the first speech recognizer 200, but at the expense of reduced latency and increased power consumption. Thus, when the utterance-level confidence score associated with the recognition result generated by the first speech recognizer 200 is less than a confidence threshold, the confidence-based routine may utilize the second speech recognizer 402 to transcribe the utterance 12.

[0072] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0073] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0074] Figure 5 is a flow chart of an example operational arrangement of a method 500 for estimating word-level confidences of words recognized by a speech recognizer biased by a neural associative memory (NAM) bias model 240. Data processing hardware 112 ( Figure 1 ) can execute the stored in the memory hardware 114 ( Figure 1 ) to perform an example operational arrangement of method 500. At operation 502, method 500 includes receiving a sequence of acoustic frames 202 representing an utterance 12 spoken by user 10. At operation 504, method 500 includes receiving bias context information 242.

[0075] At operation 506, method 500 includes performing speech recognition on a sequence of acoustic frames 202 using an automatic speech recognition (ASR) model 200 in the following manner: generating, by an audio encoder 205 of the ASR model 200, a sequence of high-order feature representations 252, each high-order feature representation being associated with a corresponding acoustic frame in the sequence of acoustic frames 202; modifying, by a NAM bias model 240, the sequence of high-order feature representations 252 using bias context information 242 to generate a corresponding sequence of bias context vectors 262; and generating, by a decoder 230 of the ASR model 200, based on the bias context vectors 262, a sequence Y of hypothesized sub-word units corresponding to a final speech recognition result 232 of the utterance 12 L , the sequence Y of the hypothesized subword units L Here, each hypothesized sub-word unit is output from the decoder 230 at a corresponding output step.

[0076] Based on the bias context vector 262, the method 500 includes generating a corresponding confidence output score 302 for each corresponding sub-word unit in the hypothesized sub-word unit sequence using a confidence estimation model (CEM) 300 at operation 508. In some examples, the sub-word unit includes a word fragment. In other examples, the sub-word unit includes a grapheme or a phoneme.

[0077] Figure 6 600 that can be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0078] The computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to the memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and the storage device 630. Each of the components 610, 620, 630, 640, 650, and 660 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 610 can process instructions for execution within the computing device 600, including instructions stored in the memory 620 or on the storage device 630, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 680 coupled to the high-speed interface 640). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. In addition, multiple computing devices 600 can be connected, each of which provides part of the necessary operations (for example, as a server library, a blade server group, or a multi-processor system).

[0079] The memory 620 stores information non-temporarily within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 620 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0080] The storage device 630 is capable of providing mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configurations). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0081] The high-speed controller 640 manages bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 640 is coupled to a memory 620, a display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.

[0082] As shown, computing device 600 may be implemented in a variety of different forms. For example, it may be implemented as a standard server 600a or multiple times as a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0083] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0084] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0085] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, a processor suitable for executing a computer program includes both a general-purpose microprocessor and a special-purpose microprocessor, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CDROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0086] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and possibly a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0087] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. An automatic speech recognition (ASR) system, comprising: An ASR model (200), the ASR model comprising: an audio encoder (205) configured to encode a sequence of audio frames (202) representing a spoken utterance into a sequence of high-level feature representations (252); and a decoder (230) configured to receive the sequence of high-level feature representations (252) and output a final speech recognition result (232); a neural associative memory (NAM) bias model (240) configured to receive bias context information (242) and to modify the sequence of high-order feature representations (252) based on the bias context information (242) to generate a bias context vector (262) as an output to encourage the decoder (230) to recognize a bias phrase indicated by the bias context information (242); and A confidence estimation model (CEM (300)) configured to calculate a confidence (302) of the final speech recognition result (232) output by the decoder (230), wherein the CEM (300) is connected to the bias context vector (262) generated by the NAM bias model (240).

2. The ASR system of claim 1, wherein the bias phrases include rare words and out-of-vocabulary (OOV) terms that are not used to train the audio encoder (205) and the decoder (230).

3. The ASR system of claim 1, wherein the NAM bias model (240) and the CEM (300) are trained separately.

4. An ASR system as claimed in claim 3, wherein the CEM (300) is trained for training bias context vectors (262), the training bias context vectors corresponding to high-order feature representations (252) encoded by the audio encoder (205) from input audio frames (202), the input audio frames having been modified by the NAM bias model (240) based on bias context information (242) extracted from training utterances for input to the NAM bias model (240).

5. The ASR system of claim 1, wherein the NAM bias model (240) is initially trained based on the ASR model (200), and the CEM (300) is trained on training data based on the initially trained NAM neural bias model.

6. The ASR system of claim 5, wherein the CEM (300) randomly samples context information (242) from the training data.

7. The ASR system of claim 1, wherein the audio encoder (205) comprises a cascaded encoder architecture having a causal encoder (210) and a non-causal encoder (250).

8. The ASR system of claim 1, wherein the audio encoder (205) comprises a stack of multi-head attention layers.

9. The ASR system of claim 8, wherein the stack of multi-head attention layers comprises a stack of conformer layers or transformer layers.

10. The ASR system of claim 1, wherein the CEM (300) comprises a multi-head attention block with self-attention (310) and acoustic cross-attention (320).

11. A computer-implemented method (500) which, when executed on data processing hardware (112), causes the data processing hardware (112) to perform operations comprising: receiving a sequence of acoustic frames (202) representing utterances spoken by a user; Receiving bias context information (242); Performing speech recognition on the sequence of acoustic frames (202) using an automatic speech recognition (ASR) model (200) by: generating, by an audio encoder (205) of the ASR model (200), a sequence of high-order feature representations (252), each high-order feature representation being associated with a corresponding acoustic frame (202) in the sequence of acoustic frames (202); modifying, by a neural associative memory (NAM) bias model (240), the sequence of high-order feature representations (252) using the bias context information (242) to generate a corresponding sequence of bias context vectors (262); and Based on the bias context vector (262), a decoder (230) of the ASR model (200) generates a sequence of hypothetical subword units corresponding to a final speech recognition result (232) of the utterance, the sequence of hypothetical subword units forming one or more words of the utterance, each hypothetical subword unit being output from the decoder (230) at a corresponding output step; as well as Based on the bias context vector (262), a confidence estimation model (CEM (300)) is used to generate a corresponding confidence output score (302) for each corresponding sub-word unit in the sequence of hypothesized sub-word units.

12. The computer-implemented method (500) of claim 11, wherein: The bias context vector (262) encourages the decoder (230) to recognize any bias phrase indicated by the bias context information (242); and The bias phrases include rare words and out-of-vocabulary (OOV) terms that are not used to train the audio encoder (205) and the decoder (230).

13. The computer-implemented method (500) of claim 11, wherein the NAM bias model (240) and the CEM (300) are trained separately.

14. A computer-implemented method (500) as described in claim 13, wherein the CEM (300) is trained for training bias context vectors (262), the training bias context vectors corresponding to high-order feature representations (252) encoded by the audio encoder (205) from input audio frames, the input audio frames having been modified by the NAM bias model (240) based on bias context information (242) extracted from training utterances for input to the NAM bias model (240).

15. The computer-implemented method (500) of claim 11, wherein the NAM bias model (240) is initially trained based on the ASR model (200), and the CEM (300) is trained on training data based on the initially trained NAM neural bias model.

16. The computer-implemented method (500) of claim 15, wherein the CEM (300) randomly samples context information (242) from the training data.

17. The computer-implemented method (500) of claim 11, wherein the audio encoder (205) comprises a cascaded encoder architecture having a causal encoder (210) and a non-causal encoder (250).

18. The computer-implemented method (500) of claim 11, wherein the audio encoder (205) comprises a stack of multi-head attention layers.

19. The computer implemented method (500) of claim 11, wherein the sub-word units comprise word fragments.

20. The computer-implemented method (500) of claim 11, wherein the operations further comprise: For each word of the one or more words formed by the sequence of hypothetical sub-word units, determining a corresponding word-level confidence score (302) for the word, the corresponding word-level confidence score (302) being equal to a corresponding confidence output score (302) generated for a last sub-word unit in the word; as well as An utterance-level confidence score (350) for the speech recognition result (232) is determined by aggregating the respective word-level confidence scores (302) determined for the one or more words of the utterance.

Images