Proper Noun Recognition in End-to-End Speech Recognition

A two-pass speech recognition system with RNN-T and LAS decoders and MWER loss improves proper noun recognition, addressing real-time accuracy issues in mobile environments by enhancing model training for rare words and maintaining low latency.

JP7590520B2Active Publication Date: 2024-11-26GOOGLE LLC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023148260
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-01-28
Filing Date
2023-09-13
Publication Date
2024-11-26
Estimated Expiration
2041-01-15

AI Technical Summary

Technical Problem

Current speech recognition systems face challenges in accurately recognizing proper nouns and other rare or uncommon words in real-time, especially in mobile environments, due to the limitations of existing models like RNN-T and LAS, which lack effective methods to handle phonetic similarities and require external resources for training.

Method used

A two-pass speech recognition architecture combining RNN-T and LAS decoders with a shared encoder, utilizing a minimum word error rate (MWER) loss criterion to emphasize proper noun recognition, incorporating entity tagging and phonetic fuzzing to improve accuracy.

Benefits of technology

The proposed system achieves a 2-7% relative WER reduction in proper noun recognition and maintains low latency, making it suitable for real-time applications on mobile devices without relying on external resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590520000008
    Figure 0007590520000008
  • Figure 0007590520000009
    Figure 0007590520000009
  • Figure 0007590520000010
    Figure 0007590520000010
Patent Text Reader

Abstract

To provide a method for training a speech recognition model with a minimum word error rate loss function.SOLUTION: The method includes: receiving a training example (302) including a proper noun; generating a plurality of hypotheses (222) corresponding to the training example, each hypothesis of the plurality of hypotheses representing the proper noun and including a corresponding probability that indicates a likelihood that the hypothesis represents the proper noun; determining that the corresponding probability associated with one of the plurality of hypotheses satisfies a penalty criteria, the penalty criteria indicating that the corresponding probability satisfies a probability threshold, and that the associated hypothesis incorrectly represents the proper noun; and applying a penalty (332) to the minimum word error rate loss function.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This disclosure relates to proper noun recognition in end-to-end speech recognition. [Background technology]

[0002] Modern automated speech recognition (ASR) systems aim to achieve high quality (i.e., low word error rate (WER)) speech recognition. The focus is on providing low latency (e.g., a short delay between when a user speaks and when a transcription appears) as well as high latency (e.g., a short delay between when a user speaks and when a transcription appears). Furthermore, when using ASR systems today, the ASR system is required to decode speech in a streaming manner that corresponds to real time or even faster than real time. For example, if the ASR system is installed on a mobile phone that directly interacts with a user, an application on the mobile phone that uses the ASR system may require the speech recognition to be streamed so that it appears on the screen as soon as the word is spoken. Here, mobile phone users may also have a low tolerance for delay. Due to this low tolerance, speech recognition aims to operate on mobile devices in a way that minimizes the impact of delays and inaccuracies that can negatively affect the user experience. Summary of the Invention

[0003] One aspect of the disclosure provides a computer-implemented method that, when executed on the data processing hardware, causes the data processing hardware to perform operations, including: training a speech recognition model using a minimum word error rate loss function, receiving training examples including a proper noun, generating a plurality of hypotheses corresponding to the training examples, each hypothesis of the plurality of hypotheses representing a proper noun and including a corresponding probability indicating a likelihood that the hypothesis represents the proper noun, determining that the corresponding probability associated with one of the plurality of hypotheses satisfies a penalty criterion, and training the speech recognition model by applying a penalty to the minimum word error rate loss function, the penalty criterion indicating that the corresponding probability meets a probability threshold and that the associated hypothesis misrepresents the proper noun.

[0004] Embodiments of the present disclosure may include one or more of any of the following features. In some embodiments, the speech recognition model includes a two-pass architecture including a first-pass network including a recurrent neural network transducer (RNN-T) decoder and a second-pass network including a listen-attend-spell (LAS) decoder. In these embodiments, the speech recognition model may further include a shared encoder that encodes the acoustic frames for each of the first-pass network and the second-pass network. The training using the minimum word error rate loss function in these embodiments may occur in the LAS encoder. The operations may further include training the RNN-T decoder and training the LAS decoder with parameters of the trained RNN-T decoder fixed prior to training the LAS decoder using the minimum word error rate loss function.

[0005] In some examples, the corresponding probability meets the probability threshold if the corresponding probability is greater than the corresponding probabilities associated with other hypotheses. The operations may further include assigning a probability to each hypothesis of the multiple hypotheses. In some embodiments, the operations may further include receiving an incorrect hypothesis and assigning a respective probability to the incorrect hypothesis, and the penalty criteria further includes an indication that the hypothesis includes the generated incorrect hypothesis. In these examples, the incorrect hypothesis may include a phonetic similarity to a proper noun, and / or the operations may further include substituting the incorrect hypothesis for the generated hypothesis of the multiple hypotheses.

[0006] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations, the operations including training a speech recognition model using a minimum word error rate loss function, receiving training examples including a proper noun, generating a plurality of hypotheses corresponding to the training examples, each hypothesis of the plurality of hypotheses representing a proper noun and including a corresponding probability indicating a likelihood that the hypothesis represents the proper noun, determining that the corresponding probability associated with one of the plurality of hypotheses satisfies a penalty criterion, and training the speech recognition model by applying a penalty to the minimum word error rate loss function, the penalty criterion indicating that the corresponding probability satisfies a probability threshold and that the associated hypothesis misrepresents the proper noun.

[0007] This aspect may include one or more of any of the following features. In some embodiments, the system further includes a first pass network comprising a recurrent neural network transducer (RNN-T) decoder and a second pass network comprising a listen-attend-spell (LAS) decoder, and the speech recognition model comprises the first pass network and the second pass network. In these embodiments, the system may also include a shared encoder configured to encode the acoustic frames for each of the first pass network and the second pass network. The training using the minimum word error rate loss function in these embodiments may occur at the LAS decoder. The operations may further include training the RNN-T decoder and training the LAS decoder with parameters of the trained RNN-T decoder fixed prior to training the LAS decoder using the minimum word error rate loss function.

[0008] In some examples, the corresponding probability satisfies the probability threshold if the corresponding probability is greater than corresponding probabilities associated with other hypotheses. The operations may further include assigning a probability to each hypothesis of the multiple hypotheses. In some embodiments, the operations may further include receiving an incorrect hypothesis and assigning a respective probability to the incorrect hypothesis, and the penalty criteria further includes an indication that the hypothesis includes the generated incorrect hypothesis. In these examples, the incorrect hypothesis may include a phonetic similarity to a proper noun, and / or the operations may further include substituting the incorrect hypothesis for the generated hypothesis of the multiple hypotheses.

[0009] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the detailed description below. Other aspects, features, and advantages will become apparent from the detailed description and drawings, and from the claims. [Brief description of the drawings]

[0010] [Figure 1A]1 is a schematic diagram of an example speech environment using a two-pass speech recognition architecture with joint acoustic and text models; [Figure 1B] 1 is a schematic diagram of an example speech environment using a two-pass speech recognition architecture with joint acoustic and text models; [Diagram 2] FIG. 1 is a schematic diagram of an exemplary two-pass speech recognition architecture for speech recognition. [Figure 3A] 3 is a schematic diagram of an exemplary training procedure for training the two-pass speech recognition architecture of FIG. 2. [Figure 3B] 3 is a schematic diagram of an exemplary training procedure for training the two-pass speech recognition architecture of FIG. 2. [Figure 3C] 3 is a schematic diagram of an exemplary training procedure for training the two-pass speech recognition architecture of FIG. 2. [Figure 4] 3 is a flowchart of an example arrangement of operations for a method of training the two-pass speech recognition architecture of FIG. 2. [Diagram 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Like reference symbols in the various drawings indicate like elements. Speech recognition continues to evolve to meet the demands of unconstrainedness and agility in the mobile environment. New speech recognition architectures or improvements on existing architectures continue to be developed to improve the quality of automatic speech recognition systems (ASR). For example, speech recognition initially employed multiple models, with each model having a dedicated purpose. For example, ASR systems included an acoustic model (AM), a pronunciation model (PM), and a language model (LM). The acoustic model mapped segments of speech (i.e., frames of speech) to phonemes. The pronunciation model strung these phonemes together to form words, and the language model was used to represent the likelihood of a given phrase (i.e., the probability of a sequence of words). Although these individual models worked in tandem, each model was trained separately and often manually designed on different datasets.

[0012] Although separate model approaches have allowed for considerable improvements in the accuracy of speech recognition systems, especially when the training corpus (i.e., the collection of training data) for a given model corresponds to the effectiveness of the model, the need to train separate models individually brings its own complexity, leading to the adoption of architectures with integrated models. These integrated models attempt to directly map speech waveforms (i.e., input sequences) to output sentences (i.e., output sequences) using a single neural network. This results in a sequence-to-sequence approach in which, given any sequence of speech features, a sequence of words (or graphemes) is generated. Examples of sequence-to-sequence models include the "attention-based" model and the "listen-attend-spell" (LAS) model. The LAS model converts spoken utterances into text using a listener, an attendant, and a speller component. Here, the listener is a recurrent neural network (RNN) encoder that receives speech input (e.g., a time-frequency representation of speech input) and maps the speech input to a higher-level feature representation. The attender attends to the higher-level features to learn alignments between input features and predicted subword units (e.g., graphemes or word pieces). The speller is an attention-based RNN decoder that generates character sequences from the input by generating probability distributions over a set of hypothesis words. The unified structure allows all components of the model to be trained jointly as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is composed entirely of neural networks. The complete neural network works without any external and / or manually designed components (e.g., finite-state transducers, lexicons, or text normalization modules).Furthermore, when training E2E models, these models typically do not require bootstrapping from decision trees or time tuning from another system.

[0013] While early E2E models were accurate and provided training improvements over models trained separately, these E2E models, such as the LAS model, worked by reviewing the entire input sequence before generating output text, and therefore could not stream output as the input was received. Without streaming capabilities, LAS models cannot perform real-time voice transcription. This deficiency can cause problems when used with voice applications that are delay-sensitive and / or require real-time voice transcription. This makes the LAS model, by itself, a less than ideal model for mobile technologies (e.g., mobile phones) that often rely on real-time applications (e.g., real-time communication applications).

[0014] Furthermore, a speech recognition system having an acoustic model, a pronunciation model, and a language model, or models that are composed together, may rely on a decoder that needs to search a relatively large search graph associated with these models. A large search graph makes it unfavorable to host this type of speech recognition system entirely on-device. Here, when a speech recognition system is hosted "on-device", the device that receives the speech input uses its processor(s) to perform the functions of the speech recognition system. For example, when a speech recognition system is hosted entirely on-device, the device's processor does not need to cooperate with computing resources outside the device to perform the functions of the speech recognition system. A device that performs speech recognition that is not entirely on-device relies on remote computing (e.g., a remote computing system or cloud computing), and thus an online connection, to perform at least some functions of the speech recognition system. For example, the speech recognition system uses a network connection with a server-based model to perform decoding through a large search graph.

[0015] Unfortunately, the reliance on remote connections makes speech recognition systems vulnerable to latency issues and / or unreliability inherent in communication networks. To improve the usefulness of speech recognition by avoiding these issues, speech recognition systems have again evolved into a form of sequence-to-sequence model known as recurrent neural network transducer (RNN-T). Unlike other sequence-to-sequence models that do not employ attention mechanisms and must process entire sequences (e.g., speech waveforms) to generate an output (e.g., a sentence), RNN-T processes input samples continuously and streams output symbols, a feature that makes it particularly attractive for real-time communication. For example, speech recognition using RNN-T may output letters one by one as they are spoken. Here, RNN-T predicts the next symbol using a feedback loop that feeds back the symbols predicted by the model to itself. Because RNN-T decoding involves a beam search through a single neural network rather than a large decoder graph, RNN-T can scale to a fraction of the size of server-based speech recognition models. The reduced size allows RNN-T to be fully on-device and operate offline (i.e., without a network connection), thus avoiding network reliability issues.

[0016] In addition to operating with low latency, speech recognition systems are required to accurately recognize speech. For models that perform speech recognition, the word error rate (WER) is often used as a metric to define the accuracy of the model. WER is a measure of how many words have been changed compared to the number of words actually spoken. Generally, these word changes refer to substitutions (i.e., when words are replaced), insertions (i.e., when words are added), and / or deletions (i.e., when words are omitted). For example, a speaker may say "car," but the ASR system may transcribe the word "car" as "bar." This is an example of substitution due to phonetic similarity. When measuring the capabilities of an ASR system compared to other ASR systems, WER can indicate a measure of improvement or quality assurance capabilities compared to another system or a baseline.

[0017] Although the RNN-T model has shown promise as a strong candidate model for on-device speech recognition, the RNN-T model alone lags behind large-scale state-of-the-art conventional models (e.g., server-based models with separate AM, PM, and LM) in terms of quality (e.g., speech recognition accuracy). However, the non-streaming E2E, LAS model has speech recognition quality comparable to large-scale state-of-the-art conventional models. To leverage the quality of the non-streaming E2E LAS model, a two-pass speech recognition system (e.g., as shown in Figure 2) was developed that includes a first-pass component of an RNN-T network followed by a second-pass component of an LAS network. With this design, the two-pass model benefits from the streaming nature of the low-latency RNN-T model and improves the accuracy of the RNN-T model through the second pass incorporating the LAS network. Although the LAS network increases the latency compared to the RNN-T model alone, the latency increase is reasonably small and meets the latency constraints for on-device operation. In terms of accuracy, the two-pass model achieves a 17-22% WER reduction when compared to RNN-T alone, and has a similar WER when compared to large-scale conventional models.

[0018] Even with a two-pass model, with a first pass of an RNN-T network and a second pass of an LAS network, there are trade-offs, especially for rare or uncommon words. These types of words are sometimes called tail utterances, and are inherently difficult for speech systems to transcribe due to ambiguity, rarity in training, or specialized linguistics. Examples of tail utterances include accented speech, cross-lingual speech, numbers, and proper names. For example, proper names present a challenge in streaming ASR using a two-pass model. This is because a particular name may appear infrequently or not at all during training, yet have a pronunciation similar to a more common word. Until now, traditional models can optimize the pronunciation model (PM) to improve tail performance by injecting knowledge about the pronunciation of the proper name. Unfortunately, two-pass architectures lack an explicit pronunciation model (PM) that can be trained specifically on the pronunciation of proper nouns, and a language model (LM) that can be trained on a large corpus with extensive exposure to proper nouns. Without a PM as a specific place to inject appropriate known knowledge in a streaming two-pass system, it becomes more difficult to model specific requirements such as the pronunciation of proper nouns. Some models attempt to ameliorate the problem with uncommon / rare words by incorporating additional training data or models, but these techniques increase model size, training time, and / or inference cost.

[0019] To increase the effectiveness of the two-pass model for proper names and / or other tail utterances, the two-pass architecture uses a customized minimum word error rate (MWER) loss criterion. This loss criterion is specifically intended to emphasize the recognition of proper names. By using a loss criterion to improve the recognition of proper names, the speech recognition system does not require new data at training or an external model at inference. Here, two different methods of loss criteria can be used with respect to proper name recognition. The first method involves an entity tagging system that identifies proper names in the ground truth transcription and increases the loss of hypotheses that miss proper names during training. The second method is to inject additional hypotheses into the MWER beam, where the additional hypotheses correspond to proper names replaced with phonetically similar alternatives. For example, an additional hypothesis of "Hallmark" is added as a phonetically similar alternative to "Walmart". In the second approach, the process of training makes the model aware of possible mistakes and potential alternatives. On various proper noun test sets, these custom loss criterion methods can reduce the WER by 2-7% relative to the traditional two-pass architecture without a custom loss criterion.

[0020] 1A and 1B are an example of a speech environment 100. In the speech environment 100, a user's 10 way of interacting with a computing device, such as a user device 110, may be through voice input. The user device 110 (also commonly referred to as device 110) is configured to capture sounds (e.g., streaming voice data) from one or more users 10 in the speech-enabled environment 100. Here, the streaming voice data 12 may refer to speech spoken by the user 10 that serves as an audible query, a command to the device 110, or an audible conversation captured by the device 110. The speech-enabled system of the device 110 may process the query or command by responding to the query and / or by causing the command to be executed.

[0021] The user device 110 may correspond to any computing device associated with the user 10 and capable of receiving the voice data 12. Some examples of the user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, Internet of Things (IoT) devices, smart speakers, etc. The user device 110 includes data processing hardware 112 and memory hardware 114 in communication with the data processing hardware 112, which stores instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations. The user device 110 further includes an audio subsystem 116 having voice capture devices (e.g., microphones) 116, 116a for capturing and converting spoken utterances 12 into electrical signals within the speech response system 100, and speech output devices (e.g., speakers) 116, 116b for communicating audible audio signals (e.g., as output audio data from the device 110). In the illustrated example, the user device 110 implements a single voice capture device 116a, but the user device 110 may implement an array of voice capture devices 116a without departing from the scope of this disclosure, such that one or more capture devices 116a in the array are not physically present on the user device 110 but may be in communication with the audio subsystem 116. The user device 110 (e.g., using hardware 112, 114) is further configured to perform speech recognition processing on the streaming audio data 12 using a speech recognizer 200. In some examples, the audio subsystem 116 of the user device 110, which includes an audio capture device 116a, is configured to receive audio data 12 (e.g., spoken utterances) and convert the audio data 12 into a digital format compatible with the speech recognizer 200. The digital format may correspond to acoustic frames (e.g., parameterized acoustic frames), such as mel frames.For example, a parameterized acoustic frame corresponds to a log-mel filterbank energy.

[0022] In some examples, such as FIG. 1A, a user 10 interacts with a program or application 118 on a user device 110 that uses a speech recognizer 200. For example, FIG. 1A shows a user 10 communicating with an automated assistant application. In this example, the user 10 asks the automated assistant, "What time is the concert tonight?" The user 10 asks, "What time is the concert tonight?" This question from the user 10 is a spoken utterance 12 that is captured by the voice capture device 116a and processed by the voice subsystem 116 of the user device 110. In this example, the speech recognizer 200 of the user device 110 receives the speech input 202 (e.g., as acoustic frames) of "What time is the concert tonight?" and transcribes the speech input 202 into a transcription 204 (e.g., a textual representation of "What time is the concert tonight?"). The automated assistant of the application 118 may now use natural language processing to answer the question posed by the user 10. Natural language processing generally refers to the process of interpreting written language (e.g., the transcript 204) and determining whether the written language prompts some action. In this example, the automated assistant uses natural language processing to recognize that the question from the user 10 is about the user's schedule, and more specifically, the concert on the user's schedule. By recognizing these details through natural language processing, the automated assistant responds to the user's query with an answer such as "Doors open at 8:30 pm for the concert tonight." In some configurations, the natural language processing may occur on a remote system that communicates with the data processing hardware 112 of the user device 110.

[0023] FIG. 1B is another example of speech recognition using a speech recognizer 200. In this example, a user 10 associated with a user device 110 is communicating with a friend named Jane Doe using a communication application 118. Here, the user 10 named Ted communicates with Jane by having the speech recognizer 200 transcribe his or her speech inputs. The voice capture device 116 captures these speech inputs and conveys them in a digital format (e.g., acoustic frames) to the speech recognizer 200. The speech recognizer 200 transcribes these acoustic frames into text that is sent to Jane via the communication application 118. Because this type of application 118 communicates via text, the transcription 204 from the speech recognizer 200 may be sent to Jane without further processing (e.g., natural language processing).

[0024] In some examples, such as FIG. 2, the speech recognizer 200 is configured with a two-pass speech recognition architecture (or simply a “two-pass architecture”). In general, the two-pass architecture of the speech recognizer 200 includes at least one encoder 210, an RNN-T decoder 220, and an LAS decoder 230. In two-pass decoding, the second pass 208 (e.g., shown as the LAS decoder 230) may improve the initial output from the first pass 206 (e.g., shown as the RNN-T decoder 220) using techniques such as lattice rescoring or n-best re-ranking. In other words, the RNN-T decoder 220 generates streaming predictions (e.g., a set of N-best hypotheses), and the LAS decoder 230 finalizes the predictions (e.g., identifies the one best rescored hypothesis). Specifically, the LAS decoder 230 receives the hypothesis y R In general, the LAS decoder 230 rescores the hypothesis y RAlthough described as functioning in a rescoring mode in which the LAS decoder 230 rescores the utterance, the LAS decoder 230 may also operate in a different mode, such as a beam search mode, depending on design or other factors (e.g., the length of the utterance).

[0025] At least one encoder 210 is configured to receive as audio input 202 audio frames corresponding to streaming audio data 12. The audio frames may be pre-processed by the audio subsystem 116 into parameterized audio frames (e.g., mel-frames and / or spectral frames). In some embodiments, the parameterized audio frames correspond to log-mel filter bank energies having log-mel features. For example, the parameterized input audio frames output by the audio subsystem 116 and input to the encoder 210 may be a set of parameterized input audio frames x=(x1,...,x T ), where:

[0026]

number

[0027] The structure of the encoder 210 can be implemented in different ways, but in some embodiments, the encoder 210 uses a long-short term memory (LSTM). The encoder 210 is a neural network with a 640-dimensional time-reduction factor N=2. For example, the encoder 210 includes eight LSTM layers, where each layer includes 2,048 hidden units followed by a 640-dimensional projection layer. In some examples, a time-reduction layer with a reduction factor N=2 is inserted after the second LSTM layer of the encoder 210.

[0028] In some configurations, the encoder 210 is a shared encoder network. In other words, instead of each path network 206, 208 having its own separate encoder, each path 206, 208 shares a single encoder 210. By sharing an encoder, the ASR speech recognizer 200 using a two-pass architecture can reduce its model size and / or its computational cost. Here, the reduction in model size can help enable the speech recognizer 200 to perform well entirely on-device.

[0029] 2 also includes an additional encoder, such as an acoustic encoder 240, to adapt the output 212 of the encoder 210 to be suitable for the second pass 208 of the LAS decoder 230. The acoustic encoder 240 is configured to further encode the output 212 into an encoded output 252. In some embodiments, the acoustic encoder 240 is an LSTM encoder (e.g., a two-layer LSTM encoder) that further encodes the output 212 from the encoder 210. By including an additional encoder, the encoder 210 may still be retained as a shared encoder between the passes 206, 208.

[0030] Through a first pass 206, an encoder 210 receives each acoustic frame of the speech input 202 and generates an output 212 (e.g., denoted as an encoding of the acoustic frame, e). An RNN-T decoder 220 receives the output 212 for each frame and generates a hypothesis y R The RNN-T decoder 220 generates an output 222, denoted as , in a streaming manner at each time step. In other words, the RNN-T decoder 220 may consume the embeddings or outputs 212 per frame to generate the word piece outputs 222 as hypotheses. In some examples, the RNN-T decoder 220 generates the N-best hypotheses 222 by performing a beam search based on the received encoded acoustic frames 212. With respect to the structure of the RNN-T decoder 220, the RNN-T decoder 220 may include a prediction network and a combination network. Here, the prediction network may have two LSTM layers with 2,048 hidden units and 640-dimensional projections (per layer), as well as an embedding layer with 128 units. The output 212 of the encoder 210 and the prediction network may be fed into a combination network including a softmax prediction layer. In some examples, the combination network of the RNN-T decoder 220 includes 640 hidden units followed by a softmax layer to predict 4,096 mixed-case word pieces.

[0031] In the two-pass model of FIG. 2, during the second pass 208, the LAS decoder 230 receives the output 212 from the encoder 210 for each frame and generates the hypothesis y L When the LAS decoder 230 operates in beam search mode, the LAS decoder 230 generates the output 232 from only the output 212 and ignores the output 222 of the RNN-T decoder 220. When the LAS decoder 230 operates in rescoring mode, the LAS decoder 230 generates the output 232 from the top K hypotheses 222, y, from the RNN-T decoder 220 (e.g., corresponding to the N best hypotheses generated by the RNN-T decoder 220). RThen, the LAS decoder 230 operates in a supervised forced mode for each sequence while attending to the output 212 to compute a score. For example, the score is a combination of the log-probability of the sequence and an attention coverage penalty. The LAS decoder 230 selects the sequence with the highest score as the output 232. In other words, the LAS decoder 230 selects a single hypothesis y with the maximum likelihood from the N-best list of hypotheses 222 from the RNN-T decoder 220. R Here, in the rescoring mode, the LAS decoder 230 may include multi-headed attention (e.g., with four heads) to attention the output 212. Furthermore, the LAS decoder 230 may be a two-layer LAS decoder 230 with a softmax layer for prediction. For example, each layer of the LAS decoder 230 has 2,048 hidden units followed by a 640-dimensional projection. The softmax layer may include 4,096 dimensions to predict the same mixed-case word pieces from the softmax layer of the RNN-T decoder 220.

[0032] Neural networks are typically trained by backpropagation, which defines a loss function (e.g., a cross-entropy loss function). For example, the loss function is defined as the difference between the actual output of the network and the desired output of the network. To train a model using a cross-entropy (CE) loss function, the model is trained to optimize the CE loss function by maximizing the log-likelihood of the training data. With reference to FIGS. 3A-3C, a training procedure 300 may train each component of the speech recognizer 200 with a corresponding set of training data 302, 302a-d. The training procedure 300 may be executed on a system 500 including data processing hardware 510 (FIG. 5) and memory hardware 520 (FIG. 5) in communication with the data processing hardware, which stores instructions that, when executed on the data processing hardware 510, cause the processing hardware 510 to perform operations. For example, a training procedure 300 for training the two-pass model architecture of the speech recognizer 200 of FIG. 2 may occur in three stages 310, 320, 330. During the first stage 310, the training procedure 300 trains the encoder 210 and the RNN-T decoder 220 (e.g., using a CE loss function). In some examples, the training procedure 300 performs a training on P(y R = y|x). During the second stage 320, the training procedure 300 trains the LAS decoder 230 without updating parameters of the encoder 210 or the RNN-T decoder 220. In some embodiments, the training procedure 300 trains the LAS decoder 230 using a cross-entropy teaching enforcing loss. For example, the training procedure 300 trains the LAS decoder 230 using a cross-entropy teaching enforcing loss. L= y|x). During the third stage 330, the training procedure 300 further trains the LAS decoder 230 with a minimum WER (MWER) loss to optimize the expected word error rate by using the n-best hypotheses. For example, the WER objective function models the loss as a weighted average of the word errors in the hypotheses 222 of the N-best beams. During this third stage 330, the LAS decoder 230 may be fine-tuned according to the WER objective function expressed by the following equation:

[0033]

number

[0034]

number

[0035]

number

[0036] Referring to FIG. 3B, in some embodiments, during the third training stage 330 or fine-tuning stage, the training procedure 300 performs training using the MWER loss, but with a modified loss function MWER AUG Here, we use the modified loss function MWER AUG is a form of loss augmentation for proper names. In this training approach, the loss is configured to emphasize the performance of proper names in training. In some examples, the loss emphasizes the performance of proper names by increasing the penalty 332 applied to the model (e.g., LAS decoder 230) when the model assigns a high probability to a hypothesis y that correctly misses the proper name. To illustrate, FIG. 3B shows that during a third stage 330 of the training procedure 300, the LAS decoder 230 selects a set of possible hypotheses y that predict the input 302d. L , y L1-3 Here, the input 302d contains a proper noun Pn, but the LAS decoder 230 selects the most probable hypothesis y L The first hypothesis is that y L1 In this example, the LAS decoder 230 identifies the hypothesis y L The most probable hypothesis is y L Therefore, the modified loss function MWER AUGapplies the penalty 332. In some configurations, the training procedure 300 determines that the model (e.g., the LAS decoder 230) assigned a probability to the hypothesis y that meets the penalty criterion. The penalty criterion may include that the model assigned a probability to an incorrect hypothesis for the proper noun that meets a probability threshold (e.g., exceeds a value assigned to the probability threshold), where the probability threshold may be a preconfigured value indicating an acceptable level or value for an incorrect hypothesis. In these examples, if the training procedure 300 determines that the model (e.g., the LAS decoder 230) assigned a probability to the hypothesis y that meets the penalty criterion, the training procedure 300 applies the penalty 332 to the modified loss function. In some examples, the modified loss function for the loss extension of the proper noun is expressed by the following equation:

[0037]

number

[0038] FIG. 3C shows another example of a training procedure 300 that applies fuzz training to optimize the speech recognizer 200's ability to distinguish proper names. In this approach, fuzz training aims to teach the speech recognizer 200 how to distinguish between proper names and phonetically similar incorrect alternatives. In other words, in fuzz training, a model such as the speech recognizer 200 allows the model to gain knowledge of possible mistakes and alternative spellings. During training, if the model (e.g., the LAS decoder 230) assigns a high probability to a proper name mistake, the training procedure 300 imposes a penalty 332 on the model. By imposing the penalty 332, the training is intended to reduce the likelihood of similar mistakes in the future.

[0039] To train the speech recognizer 200 (e.g., the LAS decoder 230 of the speech recognizer 200) on these potential errors, fuzz training may perform beam modification. In general, the beam search includes a beam size or beam width parameter B that specifies the number of optimal potential solutions (e.g., hypotheses or candidates) to evaluate. The fuzz training can exploit the beam size B by either replacing a hypothesis y from the beam search or by expanding the number of hypotheses y from the beam search. To illustrate, FIG. 3C shows a beam search with five hypotheses y L , y L1-5 A beam size of 5, corresponding to 5 hypotheses y L , y L1-53 illustrates a beam search with a beam size of 3 expanded to y∈B. In this example, if the beam size is 5, the fuzzing system 350 may replace two of the hypotheses with incorrect proper noun alternatives 352, 352a-b. Similarly, if the beam size is 3, the fuzzing system 350 may generate an additional hypothesis y that includes incorrect proper noun alternatives 352, 352a-b. In some embodiments, the fuzzing system 350 generates proper noun alternatives 352 using a technique called phonetic fuzzing, which generates alternatives 352 that are phonetically similar to the proper nouns Pn included in the training data 302. Phonetic fuzzing may enable the fuzzing system 350 to generate new words or homophones that may not be highlighted or included in the traditional corpus of training data 302. RNN-T and the ground truth y * For the hypothesis corresponding to, the fuzz process can be expressed as

[0040]

number

[0041]

number

[0042] In some configurations, the training procedure 300 determines a fixed number of alternative words 352 (e.g., 25 alternative words 352) for each proper noun Pn included in the training dataset 302 prior to fuzz training. Here, the number of alternative words 352 generated prior to fuzz training can be configured to ensure diversity of the alternative words 352 while minimizing computational costs. If the training procedure 300 generates a fixed number of alternative words 352 prior to fuzz training, during fuzz training, the training procedure 300 may select already generated random alternative words 352 as needed.

[0043] 3C, during a third stage 330, the training procedure 300 trains the LAS decoder 230 using fuzz training, where the LAS decoder 230 receives training data 302, 302d containing a proper noun Pn and generates five hypotheses y L , y L1-5 (e.g., beam width B=5). The LAS decoder 230 also generates each hypothesis y L, and assign probabilities (e.g., shown as 0.2, 0.2, 0.1, 0.4, 0.1) that indicate the likelihood that the LAS decoder 230 thinks a particular hypothesis correctly identifies the input (e.g., training data 302). In this example, the fuzzing system 350 assigns a set of potential hypotheses y L , "Belmont" and "Boomundt". As shown in this example, the LAS decoder 230 assigns the highest probability (e.g., shown as 0.4) to the incorrect alternative "Belmont" 352a. Because the LAS decoder 230 has assigned the highest probability to the incorrect alternative 352, the training procedure 300 selects the fuzz training loss function L Fuzz y, where penalties such as penalty 332 provide feedback during training to adjust the weights or parameters of the neural network. In general, penalties function to manipulate the weights applied to a particular input to approximate or indicate an intended output rather than an undesired or inaccurate output. In other words, penalty 332 functions to reduce the likelihood that the LAS decoder 230 will indicate in the future that the incorrect alternative word 352 is likely to be the best hypothesis y.

[0044] FIG. 4 is a flow chart of an exemplary operation configuration of a method 400 for training a speech recognition model (e.g., speech recognizer 200). The method 400 trains the speech recognition model using a minimum word error rate (MWER) loss function by operations 402-408. In operation 402, the method 400 receives a training example 302 including a proper noun Pn. In operation 404, the method 400 generates a plurality of hypotheses y corresponding to the training example 302, where each hypothesis y of the plurality of hypotheses represents a proper noun Pn, and each hypothesis is assigned a probability indicating the likelihood for the respective hypothesis y. In operation 406, the method 400 determines that the probability associated with the hypothesis y satisfies a penalty criterion. The penalty criterion indicates that (i) the probability meets a probability threshold and (ii) the hypothesis misrepresents the proper noun. In operation 408, the method 400 applies a penalty 332 to the minimum word error rate loss function.

[0045] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems (e.g., speech recognizer 200) and methods (e.g., method 400) described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components, their connections and relationships, and their functions illustrated herein are merely exemplary and do not limit the embodiments of the invention described and / or claimed herein.

[0046] Computing device 500 includes a processor 510 (e.g., data processing hardware), a memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 that connects to memory 520 and a high-speed expansion port 540, and a low-speed interface / controller 560 that connects to a low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 may be interconnected using various buses and mounted on a common motherboard or in other suitable manner. Processor 510 processes instructions for execution within computing device 500, including instructions stored in memory 520 or storage device 530, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 connected to high-speed interface 540. In other embodiments, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and multiple types of memories. Also, multiple computing devices 500 may be connected together, with each device providing a portion of the required processing (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0047] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 520 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0048] The storage device 530 can provide mass storage for the computing device 500. In some embodiments, the storage device 530 is a computer-readable medium. In various different embodiments, the storage device 530 can be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional embodiments, the computer program product is tangibly embodied in an information medium. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information medium is a computer-readable medium or a machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0049] The high-speed controller 540 manages the bandwidth-intensive operations of the computing device 500, and the low-speed controller 560 manages the lower bandwidth-intensive operations. This distribution of roles is merely exemplary. In some embodiments, the high-speed controller 540 is connected to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and the high-speed expansion port 550 that accepts various expansion cards (not shown). In some embodiments, the low-speed controller 560 is connected to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which includes various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be connected to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device, such as a switch or router, for example, via a network adapter.

[0050] The computing device 500 can be implemented in a number of different forms, as shown in the drawing, such as a standard server 500a, or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0051] Various embodiments of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, connected to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0052] These computer programs (also known as programs, software, software applications, or codes) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or devices (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0053] The processes and logic flows described herein can be performed by one or more programmable processors executing one or more computer programs that perform functions by processing input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any type of digital computer. In general, a processor receives instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer is operatively coupled to receive and / or transfer data from one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0054] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device, such as, for example, a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to a user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices may also be used to provide interaction with a user, with feedback provided to the user, which may be, for example, any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, the computer may interact with the user by sending and receiving documents to and from a device used by the user (e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser).

[0055] Although several embodiments have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. 1. A computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations, said operations comprising: receiving training examples including ground truth transcriptions, where the ground truth transcriptions include sequences of words including proper nouns; Training a speech recognition model using a fuzz training loss function, generating an original beam of hypotheses corresponding to the training examples, where each hypothesis in the original beam of hypotheses includes a distinct sequence of words and a corresponding probability indicating the likelihood that the hypothesis correctly identifies the ground truth transcription; generating fuzz hypotheses corresponding to the training examples, where the fuzz hypotheses include distinct sequences of words that include alternatives to incorrect proper nouns and corresponding probabilities indicating the likelihood that the fuzz hypotheses will correctly identify the ground truth transcription; determining that the corresponding probability associated with the fuzz hypothesis is greater than the corresponding probability associated with each hypothesis in the original beam of hypotheses; applying a penalty to the fuzz training loss function based on determining that the corresponding probability associated with the fuzz hypothesis is greater than the corresponding probability associated with each hypothesis in the original beam of hypotheses. training the speech recognition model by

2. 2. The computer-implemented method of claim 1, wherein alternatives for the erroneous proper nouns in the distinct sequences of words of the fuzz hypotheses include phonetic similarities to the proper nouns in the sequences of words of the ground truth transcription.

3. 2. The computer-implemented method of claim 1, wherein alternatives for the incorrect proper nouns in the distinct sequences of words of the fuzz hypotheses include alternative spellings to spellings of the proper nouns in the sequences of words of the ground truth transcription.

4. The operation includes: Prior to training the speech recognition model using the fuzz training loss function, identifying the proper nouns in the sequence of words of the ground truth transcription; generating a number of distinct alternative words for the identified proper nouns; 2. The computer-implemented method of claim 1, wherein generating the fuzz hypotheses during training of the speech recognition model comprises randomly selecting one of the distinct alternatives for the identified proper noun as an alternative for the incorrect proper noun in a distinct sequence of words of the fuzz hypothesis.

5. Generating the fuzz hypotheses includes: identifying occurrences of the proper noun in a distinct sequence of the words of one of the hypotheses from the original beam of hypotheses; 2. The computer-implemented method of claim 1, comprising: replacing occurrences of the proper noun in the respective sequences of words of the one of the hypotheses from the original beam of hypotheses with an alternative word of the incorrect proper noun.

6. The computer-implemented method of claim 5 , wherein the operations further comprise combining an original beam of the hypothesis with the fuzz hypothesis.

7. 6. The computer-implemented method of claim 5, wherein the operations further comprise substituting the fuzz hypothesis for the one of the hypotheses from a search of an original beam that includes the distinct sequence of words that includes an occurrence of the proper noun.

8. The speech recognition model includes a two-pass architecture, the two-pass architecture comprising: a first-pass network including a recurrent neural network transducer (RNN-T) decoder; and a second-pass network including a Listen-Attend-Spell (LAS) decoder.

9. 9. The computer-implemented method of claim 8, wherein the speech recognition model further comprises a shared encoder, the shared encoder encoding acoustic frames for each of the first-pass network and the second-pass network.

10. The operation includes: training said RNN-T decoder; 10. The computer-implemented method of claim 8, further comprising: training the LAS decoder with parameters of the trained RNN-T decoder fixed prior to training the LAS decoder with a fuzz training loss function.

11. 1. A system comprising: Data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: receiving training examples including ground truth transcriptions, where the ground truth transcriptions include sequences of words including proper nouns; Training a speech recognition model using a fuzz training loss function, generating an original beam of hypotheses corresponding to the training examples, where each hypothesis in the original beam of hypotheses includes a distinct sequence of words and a corresponding probability indicating the likelihood that the hypothesis correctly identifies the ground truth transcription; generating fuzz hypotheses corresponding to the training examples, where the fuzz hypotheses include distinct sequences of words that include alternatives to incorrect proper nouns and corresponding probabilities indicating the likelihood that the fuzz hypotheses will correctly identify the ground truth transcription; determining that the corresponding probability associated with the fuzz hypothesis is greater than the corresponding probability associated with each hypothesis in the original beam of hypotheses; applying a penalty to the fuzz training loss function based on determining that the corresponding probability associated with the fuzz hypothesis is greater than the corresponding probability associated with each hypothesis in the original beam of hypotheses. training the speech recognition model by

12. 12. The system of claim 11, wherein alternatives for the erroneous proper nouns in the distinct sequences of words of the fuzz hypotheses include phonetic similarities to the proper nouns in the sequences of words of the ground truth transcription.

13. 12. The system of claim 11, wherein alternatives for the erroneous proper nouns in the respective sequences of words of the fuzz hypotheses comprise alternative spellings to spellings of the proper nouns in the sequences of words of the ground truth transcription.

14. The operation includes: Prior to training the speech recognition model using the fuzz training loss function, identifying the proper nouns in the sequence of words of the ground truth transcription; generating a number of distinct alternative words for the identified proper nouns; 12. The system of claim 11, wherein generating the fuzz hypotheses during training of the speech recognition model comprises randomly selecting one of the distinct alternatives for the identified proper noun as an alternative for the erroneous proper noun in a distinct sequence of words of the fuzz hypothesis.

15. Generating the fuzz hypotheses includes: identifying occurrences of the proper noun in a distinct sequence of the words of one of the hypotheses from the original beam of hypotheses; 12. The system of claim 11, further comprising: replacing occurrences of the proper noun in the respective sequences of words of the one of the hypotheses from the original beam of hypotheses with an alternative word of the incorrect proper noun.

16. The system of claim 15 , wherein the operations further comprise combining an original beam of the hypothesis with the fuzz hypothesis.

17. 16. The system of claim 15, wherein the operations further comprise substituting the fuzz hypothesis for the one of the hypotheses from a search of an original beam that includes the distinct sequence of words that includes an occurrence of the proper noun.

18. The speech recognition model includes a two-pass architecture, the two-pass architecture comprising: a first-pass network including a recurrent neural network transducer (RNN-T) decoder; and a second path network including a Listen-Attend-Spell (LAS) decoder.

19. 20. The system of claim 18, wherein the speech recognition model further comprises a shared encoder, the shared encoder encoding acoustic frames for each of the first-pass network and the second-pass network.

20. The operation includes: training said RNN-T decoder; 20. The system of claim 18, further comprising: training the LAS decoder with parameters of the trained RNN-T decoder fixed prior to training the LAS decoder using a fuzz training loss function.

Citation Information

Patent Citations

  • Voice recognition device and risk scoring device

    JP2017527846A

  • Output program, information processing device, and output control method

    JP2019159118A

  • System and Method for End-to-End speech recognition

    US20180330718A1

  • Language model construction support device, method and program

    WO2012165529A1