Large-scale language model data selection for rare word speech recognition

The integration of a data selection pipeline-trained external language model with a cascaded encoder-decoder architecture in ASR models addresses the challenge of rare word recognition, enhancing accuracy and flexibility in streaming and non-streaming modes.

JP2024541131AActive Publication Date: 2024-11-07GOOGLE LLC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024519718
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-09-30
Filing Date
2021-12-13
Publication Date
2024-11-07
Estimated Expiration
2041-12-13

AI Technical Summary

Technical Problem

Existing end-to-end automatic speech recognition (ASR) models struggle with high failure rates when recognizing rare words that were not seen during training, particularly in applications requiring streaming recognition.

Method used

A computer-implemented method involving data selection pipelines to train an external language model on a subset of rare word training text samples, integrating it with an ASR model using cascaded encoders and decoders to improve rare word recognition, allowing the model to operate in both streaming and non-streaming modes.

Benefits of technology

Enhances the ASR model's ability to accurately recognize rare words and long-tail proper nouns by leveraging the external language model, reducing latency and improving transcription accuracy in various speech recognition tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541131000001_ABST
    Figure 2024541131000001_ABST
Patent Text Reader

Abstract

A method (500) for training a language model (206) for rare word speech recognition includes obtaining a set of training text samples (422) and obtaining a set of training utterances (452) used to train the speech recognition model (200). Each training utterance in the plurality of training utterances includes audio data (454) corresponding to the utterance and a corresponding transcription (456) of the utterance. The method also includes applying rare word filtering to the set of training text samples to identify a subset of rare word training text samples (432) that includes words that do not appear in the transcriptions from the set of training utterances or that appear less than a threshold number of times in the transcriptions from the set of training utterances. The method further includes training an external language model on the transcriptions from the set of training utterances and the identified subset of rare word training text samples.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This disclosure relates to large scale language model data selection for rare word speech recognition. [Background technology]

[0002] Automatic speech recognition (ASR) systems have evolved from multiple models, each with a dedicated purpose, to unified models where a single neural network is used to directly map audio waveforms (i.e., input sequence) to output sentences (i.e., output sequence). This unification has resulted in a sequence-to-sequence approach that generates a sequence of words (or graphemes) given a sequence of audio features. With the unified structure, all components of the model can be trained jointly as a single end-to-end (E2E) neural network. Here, E2E models refer to models whose entire architecture is composed of neural networks. The complete neural network works without external components and / or manually designed components (e.g., finite state transducers, lexicons, or text normalization modules). In addition, when training E2E models, these models generally do not require bootstrapping from decision trees or time alignment from separate systems. These E2E automatic speech recognition (ASR) systems have made tremendous progress and have outperformed traditional ASR systems in several common benchmarks, including word error rate (WER). The architecture of an E2E ASR model depends heavily on the application. For example, some applications that involve user interaction, such as voice search or on-device dictation, require the model to perform recognition in a streaming manner. Other applications, such as offline video captioning, do not require the model to be streamed and can take advantage of future context to improve performance. In addition, existing E2E models experience high failure rates in recognizing rare words that were not seen during training. Rare word recognition is improved by training an external language model on a large training dataset. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the disclosure provides a computer-implemented method of training a language model for rare word speech recognition. The computer-implemented method, when executed on data processing hardware, causes the data processing hardware to perform operations including obtaining a set of training text samples and obtaining a set of training utterances used to train an automatic speech recognition (ASR) model. Each training utterance in the plurality of training utterances includes audio data corresponding to the utterance and a corresponding transcription of the utterance. The operations also include applying rare word filtering to the set of training text samples to identify a subset of rare word training text samples that includes words that do not appear in the transcriptions from the set of training utterances or that appear less than a threshold number of times in the transcriptions from the set of training utterances. The operations further include training an external language model on the transcriptions from the set of training utterances and the identified subset of rare word training text samples.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, obtaining the set of training text samples includes receiving a corpus of training text samples, performing a resampling function on the corpus of training text samples to identify high-frequency text samples occurring within the corpus of training text samples, and obtaining the set of training text samples by removing the identified high-frequency text samples from the corpus of training text samples. In some examples, the resampling function includes one of a simple power resampling function, a forced power resampling function, or a soft logarithmic resampling function.

[0005] In some implementations, the operations further include applying contrastive filtering to the set of training text samples to identify a subset of target domain training text samples that match a target domain associated with the set of training utterances, where training an external language model on the identified subset of transcription and rare word training text samples from the set of training utterances further includes training the external language model on the identified subset of target domain training text samples that match the target domain. In some examples, the external language model includes an external neural language model. In these examples, the external neural language model may include a stack of conformer layers or transformer layers.

[0006] In some implementations, the operations further include integrating the trained external language model with the trained ASR model. The trained external language model is configured to rescore a probability distribution over possible speech recognition hypotheses predicted by the trained ASR model. In these implementations, the ASR model includes a first encoder, a second encoder, and a decoder. The first encoder is configured to receive as an input a sequence of acoustic frames and generate a first high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames at each of a plurality of output steps. The second encoder is configured to receive as an input the first high-order feature representation generated by the first encoder at each of a plurality of output steps and generate a second high-order feature representation for the corresponding first high-order feature frame at each of a plurality of output steps. The decoder is configured to receive as an input the second high-order feature representation generated by the second encoder at each of a plurality of output steps and generate a first probability distribution over possible speech recognition hypotheses at each of a plurality of time steps.

[0007] In these implementations, the decoder is further configured to receive as input the first high-order feature representation generated by the first encoder at each of the multiple output steps, and generate a second probability distribution over the possible speech recognition hypotheses at each of the multiple time steps. In addition, the decoder may include a prediction network and a combination network. When the ASR model is operating in a streaming mode, the prediction network is configured to receive as input the average embedding generated by the prediction network at each of the multiple output steps and the first high-order feature representation generated by the first encoder at each of the multiple output steps, and generate a second probability distribution over the possible speech recognition hypotheses at each of the multiple output steps. Alternatively, when the ASR model is operating in a non-streaming mode, the prediction network is configured to receive as input the average embedding generated by the prediction network at each of the multiple output steps and the second high-order feature representation generated by the second encoder at each of the multiple output steps, and generate a first probability distribution over the possible speech recognition hypotheses.

[0008] Additionally or alternatively, the first encoder may include a causal encoder including an initial stack of conformer layers, where the second encoder may include a non-causal encoder including a final stack of conformer layers superimposed on the initial stack of conformer layers. The first and second encoders of the ASR model may be trained using hybrid autoregressive transducer factorization to facilitate integration of an external language model trained on text-only data including transcriptions from a set of training utterances and an identified subset of rare language training text samples.

[0009] Another aspect of the disclosure provides a system for training a language model for rare language speech recognition. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including obtaining a set of training text samples and obtaining a set of training utterances used to train an automatic speech recognition (ASR) model. Each training utterance in the plurality of training utterances includes audio data corresponding to the utterance and a corresponding transcription of the utterance. The operations also include applying rare word filtering to the set of training text samples to identify a subset of rare word training text samples that includes words that do not appear in the transcriptions from the set of training utterances or that appear less than a threshold number of times in the transcriptions from the set of training utterances. The operations further include training an external language model on the transcriptions from the set of training utterances and the identified subset of rare word training text samples.

[0010] This aspect may include one or more of the following optional features: In some implementations, obtaining the set of training text samples includes receiving a corpus of training text samples, performing a resampling function on the corpus of training text samples to identify high-frequency text samples occurring within the corpus of training text samples, and obtaining the set of training text samples by removing the identified high-frequency text samples from the corpus of training text samples. In some examples, the resampling function includes one of a simple power resampling function, a forced power resampling function, or a soft logarithmic resampling function.

[0011] In some implementations, the operations further include applying contrastive filtering to the set of training text samples to identify a subset of target domain training text samples that match a target domain associated with the set of training utterances, where training the external language model on the identified subset of transcription and rare word training text samples from the set of training utterances further includes training the external language model on the identified subset of target domain training text samples that match the target domain. In some examples, the external language model includes an external neural language model. In these examples, the external neural language model may include a stack of conformer layers or transformer layers.

[0012] In some implementations, the operations further include integrating the trained external language model with the trained ASR model. The trained external language model is configured to rescore a probability distribution over possible speech recognition hypotheses predicted by the trained ASR model. In these implementations, the ASR model includes a first encoder, a second encoder, and a decoder. The first encoder is configured to receive as an input a sequence of acoustic frames and generate, at each of a plurality of output steps, a first high-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The second encoder is configured to receive as an input the first high-order feature representation generated by the first encoder at each of the plurality of output steps and generate, at each of the plurality of output steps, a second high-order feature representation for the corresponding first high-order feature frame. The decoder is configured to receive as an input the second high-order feature representation generated by the second encoder at each of the plurality of output steps and generate, at each of a plurality of time steps, a first probability distribution over possible speech recognition hypotheses.

[0013] In these implementations, the decoder is further configured to receive as input the first high-order feature representation generated by the first encoder at each of the multiple output steps, and generate a second probability distribution over the possible speech recognition hypotheses at each of the multiple time steps. In addition, the decoder may include a prediction network and a combination network. When the ASR model is operating in a streaming mode, the prediction network is configured to receive as input the average embedding generated by the prediction network at each of the multiple output steps and the first high-order feature representation generated by the first encoder at each of the multiple output steps, and generate a second probability distribution over the possible speech recognition hypotheses at each of the multiple output steps. Alternatively, when the ASR model is operating in a non-streaming mode, the prediction network is configured to receive as input the average embedding generated by the prediction network at each of the multiple output steps and the second high-order feature representation generated by the second encoder at each of the multiple output steps, and generate a first probability distribution over the possible speech recognition hypotheses.

[0014] Additionally or alternatively, the first encoder may include a causal encoder including an initial stack of conformer layers, where the second encoder may include a non-causal encoder including a final stack of conformer layers superimposed on the initial stack of conformer layers. The first and second encoders of the ASR model may be trained using hybrid autoregressive transducer factorization to facilitate integration of an external language model trained on text-only data including transcriptions from a set of training utterances and an identified subset of rare language training text samples.

[0015] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0016] [Figure 1A] FIG. 1 is a schematic diagram of an exemplary speech environment using a speech recognition model and an external language model architecture for automatic speech recognition. [Figure 1B] FIG. 1 is a schematic diagram of an exemplary speech environment using a speech recognition model and an external language model architecture for automatic speech recognition. [Diagram 2] FIG. 2 is a schematic diagram of the speech recognition model and language model architecture of FIG. [Diagram 3] FIG. 3 is a schematic diagram of an exemplary combined and reduced prediction layer of the prediction network of the speech recognition model of FIG. 2. [Figure 4] FIG. 1 is a schematic diagram of a data selection pipeline for training a language model. [Diagram 5] 2 is a schematic diagram of an exemplary arrangement of operations of a method for training a language model. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Like reference numbers in the various drawings indicate like elements.

[0018] End-to-end (E2E) automatic speech recognition (ASR) models are traditionally structured to operate in either streaming or non-streaming modes. Traditionally, E2E ASR models include an encoder and a decoder as main components. Applications involving end-user interaction, such as voice search or on-device dictation, may require the model to perform recognition in a streaming manner, where words are expected to be output as spoken with as low latency as possible. This precludes the use of models that use future context to improve accuracy, such as bidirectional LSTM. By contract, applications such as offline video captioning do not require streaming recognition and may make full use of any available future context to improve performance. Furthermore, traditional E2E ASR models are trained on a small fraction of audio-text pairs compared to the over 100 billion text utterances on which traditional models are trained, and therefore perform poorly on long-tail proper nouns and rare words.

[0019] Implementations herein are directed to a single E2E ASR model combined with an on-device neural language model trained on selected data to improve the recognition quality of rare words of the ASR model. More specifically, implementations herein are directed to a data selection pipeline for selecting a sufficient subset of training data suitable for training a language model to improve the recognition quality of rare words and long-tail proper nouns. The ASR model may use a cascaded encoder including a streaming encoder and a non-streaming encoder, and a single decoder that learns to decode using the output of the streaming encoder or the non-streaming encoder to enable the ASR model to operate in streaming or non-streaming modes. In addition to the ASR model, the architecture can also be applied to other modes such as machine translation that implement both streaming and non-streaming modes.

[0020] 1A and 1B are examples of audio environments 100, 100a-b. In the audio environment 100, the manner in which a user 104 interacts with a computing device, such as a user device 10, may be through voice input. The user device 10 (also commonly referred to as device 10) is configured to capture sounds (e.g., streaming audio data) from one or more users 104 in the audio environment 100. Here, streaming audio data may refer to verbal utterances 106 by the user 104 that function as audible queries, commands to the device 10, or audible communications captured by the device 10. The voice-enabled system of the device 10 may address the queries or commands by responding to the queries and / or causing the commands to be executed / fulfilled by one or more downstream applications.

[0021] The user device 10 may correspond to any computing device that is associated with a user 104 and capable of receiving audio data. Some examples of the user device 10 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, Internet of Things (IoT) devices, in-vehicle infotainment systems, smart displays, smart speakers, etc. The user device 10 includes data processing hardware 12 and memory hardware 14 that communicates with the data processing hardware 12 and stores instructions that, when executed by the data processing hardware 12, cause the data processing hardware 12 to perform one or more operations. The user device 10 further includes an audio system 16 having audio capture devices (e.g., microphones) 16, 16a for capturing and converting oral speech 106 in the audio environment 100 into electrical signals, and audio output devices (e.g., speakers) 16, 16b for conveying audible audio signals (e.g., as output audio data from the device 10). Although the user device 10 implements a single audio capture device 16a in the illustrated example, the user device 10 may implement an array of audio capture devices 16a without departing from the scope of this disclosure, whereby one or more capture devices 16a in the array may not be physically present on the user device 10 but are in communication with the audio system 16.

[0022] In the speech environment 100, an automatic speech recognition (ASR) system 109 implementing an ASR model 200 (also referred to as model 200) integrated with an external language model (LM) 206 resides on a user device 10 of a user 104 and / or on a remote computing device 60 (e.g., one or more remote servers of a distributed system executing within a cloud computing environment) that communicates with the user device 10 via a network 40. The remote computing device 60 may include remote resources such as remote data processing hardware 62 (e.g., a remote server or CPU) and / or remote memory hardware 64 (e.g., a remote database or other storage hardware). The user device 10 and / or the remote computing device 60 also include an audio subsystem 108 configured to receive utterances 106 spoken by the user 104 and captured by the audio capture device 16a, and convert the utterances 106 into a corresponding digital format related to input acoustic frames 110 that can be processed by the ASR system 109. 1A, a user 104 speaks a respective utterance 106, and an audio subsystem 108 converts the utterance 106 into corresponding audio data (e.g., acoustic frames) 110 for input to an ASR system 109. A model 200 then receives the audio data 110 corresponding to the utterance 106 as input and generates / predicts a corresponding transcription 120 (also referred to as a recognition result / hypothesis 120) of the utterance 106 as output.

[0023] The model 200 also includes a decoder 204 (FIG. 2) (also referred to as a shared decoder 204) that is shared between its encoders, allowing the model 200 to be a single model that can operate in streaming and non-streaming modes (as opposed to, for example, two separate models, each dedicated to either streaming or non-streaming modes). For example, as shown in FIG. 1A, a digital assistant application 50 running on a user device 10 may require speech recognition to be streamed so that words, word fragments, and / or individual characters appear on the screen as soon as they are spoken. In addition, a user 104 of the user device 10 is likely to have low tolerance for latency when issuing queries that the digital assistant application 50 executes. In those scenarios where the application requires minimal latency, the model 200 operates in a streaming mode in which the model 200 can provide streaming transcription capabilities in real time as the user 104 speaks the utterance 106. On the other hand, if the user 104 has a higher tolerance for speech recognition latency and / or the utterance 106 to be recognized relates to long-form speech (i.e., referring to speech consisting of complete paragraphs or multiple sentences), the same model 200 may operate in a non-streaming mode and leverage a predictive network to provide an accurate transcription 120, but with increased latency.

[0024] In addition, the user 104 may require that the ASR system 109 of the user device 10 be able to accurately identify rare words or long-tail proper names, which may be accomplished by using the LM 206 in conjunction with the model 200 to help bias the output of the model 200 when detecting rare words or proper names. As described in more detail below with reference to FIG. 4, the LM 206 may be trained with data sets obtained through different data selection strategies to reduce the amount of text-only training data required to train the LM 206 to accurately bias the output of the model 200 to detect rare words or proper names. Thus, the ASR system 109 may implement a single ASR model including cascaded encoders 210, 220 for multiple different speech recognition tasks to provide both streaming and non-streaming transcription capabilities without leveraging a separately trained ASR model for each task, while also using the LM 206 to increase the accuracy of the transcription 120 when the utterance 106 includes rare words or long-tail proper names.

[0025] In some implementations, the model 200 first performs streaming encoding on the audio data 110 and then performs non-streaming encoding on the output of the streaming encoder. For example, in the illustrated example, the model 200 performs streaming speech recognition on the audio data 110 using a first encoder (i.e., a low-delay encoder) to generate partial speech recognition results 120, 120a, and performs non-streaming speech recognition on the encoded audio data 110 using a second encoder (i.e., a high-delay encoder) to generate final speech recognition results 120, 120b. In particular, the first encoder generates the partial speech recognition result 120a, while the second encoder waits for the output of the first encoder to generate the final speech recognition result 120b. Thus, the final speech recognition result 120b for the input utterance 106 may lag behind the partial speech recognition result 120a for the input utterance by a duration.

[0026] The user device 10 and / or the remote computing device 60 also execute a user interface generator 107 configured to present a representation of the transcript 120 of the utterance 106 to the user 104 of the user device 10. As described in more detail below, the user interface generator 107 may display partial speech recognition results 120a in a streaming manner during time 1 and then display final speech recognition results 120b during time 2. In some configurations, the transcript 120 output from the ASR system 109 is processed by, for example, a natural language understanding (NLU) module executing on the user device 10 or the remote computing device 60, for example, to execute a user command / query specified by the utterance 106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device 10 or the remote computing device 60) may convert the transcript 120 into synthetic speech for audible output by the user device 10 and / or another device.

[0027] In the example of FIG. 1A, a user 104 in a speech environment 100a interacts with a program or application 50 (e.g., a digital assistant application 50a) of a user device 10 that uses an ASR system 109. For example, FIG. 1A shows a user 104 communicating with the digital assistant application 50a and the digital assistant application 50a displaying a digital assistant interface 18 on the screen of the user device 10 to show a conversation between the user device 10 and the digital assistant application 50a. In this example, the user 104 asks the digital assistant application 50a, "What year was Serendipity released?" This question from the user 104 is an oral utterance 106 that is captured by an audio capture device 16a and processed by an audio system 16 of the user device 10. In this example, the audio system 16 receives the oral utterance 106 and converts it into acoustic frames 110 for input to the ASR system 109.

[0028] Continuing with the example, as the model 200 receives an acoustic frame 110 corresponding to an utterance 106 as the user 104 speaks, it encodes the acoustic frame 110 using a first encoder 210 (i.e., FIG. 2) and then decodes the encoded representation of the acoustic frame 110 into a partial speech recognition result 120a using a decoder 204 (FIG. 2). During time 1, the user interface generator 107 presents, via the digital assistant interface 18, a representation of the partial speech recognition result 120a of the utterance 106 to the user 104 of the user device 10 in a streaming manner, such that words, word fragments, and / or individual characters appear on the screen as soon as they are spoken.

[0029] After all (or a certain amount) of the acoustic frames 110 corresponding to the utterance 106 are received and the first encoder 210 has encoded these acoustic frames 110, the second encoder 220 (i.e., FIG. 2A) encodes the encoding output from the first encoder 210 to generate an encoding for the set of acoustic frames 110 corresponding to the utterance 106 already encoded by the first encoder 210. The decoder 204 then decodes the acoustic frames 110 encoded by the second encoder 220 and processes the decoded acoustic frames 110 using the LM 206, which rescores the decoded acoustic frames to generate the final speech recognition results 120b. For example, if the first encoder 210 has encoded all of the acoustic frames 110 corresponding to the utterance 106 (e.g., when the acoustic frames 110 are received), the second encoder 220 encodes all of the acoustic frames 110 encoded by the first encoder 210. In this regard, by coding across multiple encoded acoustic frames 110, the second encoder 210 can provide greater context awareness (e.g., by receiving a representation of all of the acoustic frames 110 relating to the utterance 106) in a non-streaming manner that can potentially adjust or correct aspects of the utterance 106 that were missed or misinterpreted by the streaming nature of the first encoder 210.

[0030] In some examples, an indication such as an endpoint identifying that the user 104 has finished speaking the utterance 106 serves to trigger the second encoder 220 of the model 200 to encode all acoustic frames 110. In other examples, the second encoder 220 encodes the acoustic frames 110 in parallel with the first encoder 210, which identifies an endpoint of the end of the utterance 106, thereby triggering the second encoder 220 to emit the final speech recognition result 120b. The endpoint identified by the first encoder 210 may simultaneously trigger a microphone closing event. During time 2, the user interface generator 107 presents a representation of the final speech recognition result 120b of the utterance 106 to the user 104 of the user device 10 via the digital assistant interface 18. In some implementations, the user interface generator 107 replaces (or modifies) the representation of the partial speech recognition result 120a with the final speech recognition result 120b. In this example, the utterance 106 of the user 104 includes the rare word "Serendipity" on which the model 200 was not trained. Thus, the partial speech recognition result 120a output by the model 200 and displayed on the screen at time 1 erroneously predicts that the utterance 106 of the user 104 is "Serendipity". The final speech recognition result 120b output by the model 200 and displayed on the screen at time 2 with increased latency improves the speech recognition quality in terms of accuracy by identifying that the user 104 said "Serendipity". However, because the user interface generator 107 displays the partial speech recognition result as the user is speaking the utterance 106, the longer latency associated with generating and ultimately displaying the final speech recognition result 120b is less noticeable to the user 104.

[0031] In some implementations, the model 200 utilizes a pre-fetching technique that reduces latency by fetching speech recognition results before the final speech recognition result 120b is available, where if the partial speech recognition result 120a matches the final speech recognition result 120b, the response fetched for the partial speech recognition result 120a can be issued immediately to save execution latency that typically occurs after the final speech recognition result 120b is completed.

[0032] In the example shown in FIG. 1A, the digital assistant application 50a may use natural language processing to respond to a question posed by the user 104. Natural language processing generally refers to the process of interpreting written language (e.g., partial speech recognition results 120a and / or final speech recognition results 120b) and determining whether the written language prompts some action. In this example, the digital assistant application 50a uses natural language processing to recognize that the question from the user device 10 is about the user's environment, more specifically, the song playing near the user. By recognizing these details using natural language processing, the automated assistant returns a response 19 to the user's query, which states, "Serendipity was released in 2001." In some configurations, the natural language processing is performed on a remote computing device 60 that communicates with the data processing hardware 12 of the user device 10.

[0033] FIG. 1B is another example of speech recognition by the ASR system 109 in a voice environment 100b. As shown in the example, a user 104 interacts with a voice application 50, 50b that displays a voicemail application interface 18, 18b on the screen of the user device 10 to transcribe a voicemail left for the user 104 by Jane Doe. In this example, latency is not critical, but transcription accuracy is when dealing with long-tail proper nouns or rare words. The model 200 and LM 206 of the ASR system 109 can take advantage of the full context of the audio by waiting until all of the acoustic frames 110 corresponding to the voicemail have been generated. This voicemail scenario also illustrates how the model 200 can process long-form speech, since voicemails are often multiple sentences or several paragraphs. The ability to process long-form speech is particularly advantageous over other ASR models, such as two-pass models with LAS decoders, because these two-pass models often suffer from long-form problems (e.g., higher word deletion rates in long-form speech) when applied to long-form conditions. For example, by using an RNN-T decoder as the decoder 204 in combination with a cascaded encoder 202 (e.g., the first encoder 210 and the second encoder 220), the model 200 works on both long-form and short-form speech without the drawbacks of the long form.

[0034] 1B, as discussed with respect to FIG. 1A, the model 200 uses the first encoder 210 to encode the acoustic frames 110 as they are received. After the model 200 receives all of the acoustic frames 110 and encodes them with the first encoder 210, the model 200 provides the first encoder output as input to the second encoder 220. The second encoder 220 encodes the first encoder output before the decoder 204 generates the embeddings, and the LM 206 rescores the output of the decoder 204 to generate the final speech recognition result 120b. During time 3, the user interface generator 107 presents a representation of the final speech recognition result 120b via the digital assistant interface 18b without first displaying the partial speech recognition result 120a. For example, the final speech recognition result 120b is a transcription of a long-form voicemail from Jane Doe stating, "Do you want to watch Serendipity tonight? Give me a call back when you get this."

[0035] FIG. 2 includes an exemplary model 200 that can operate in various combinations of streaming and non-streaming modes. Specifically, the model 200 includes a cascaded encoder 202, a decoder 204, and an external LM 206. The cascaded encoder 202 refers to a model structure in which the encoding path includes two encoders 210, 220 that cascade together such that the output of the encoder 210 feeds the input of the other encoder 220 before decoding. Here, the encoders 210, 220 can be cascaded regardless of the underlying architecture of each encoder. In some examples, the encoders 210, 220 include a stack of 512-dimensional conformer layers. To strictly restrict the use of models that do not have future inputs, a causal convolutional layer and a left context attention layer may be used for each conformer layer. In the self-attention layer, a multi-head (e.g., 8-head) attention mechanism may be used. The cascaded encoders 210, 220 may include 17 conformer layers. Here, the causal encoder 210 may include 15 conformer layers, while the non-causal encoder 210 may include 2 conformer layers that incorporate additional right context (e.g., 5.04 seconds). Optionally, transformer layers may be used instead of conformer layers.

[0036] In other implementations, one encoder is configured with an LSTM structure, and the other encoder is configured using a bidirectional LSTM layer or a conformer layer (e.g., a conformer-transducer). In other words, the encoders 210, 220 may have different or similar architectures. For example, the cascade encoder 202 may be roughly similar to an acoustic model (AM) in a conventional ASR system and may include a recurrent network of stacked long short-term memory (LSTM) layers. Here, the first encoder 210 is a streaming encoder that includes a unidirectional long short-term memory (LSTM) layer, while the second encoder 220 is a non-streaming encoder that includes a bidirectional LSTM layer or a conformer layer. In a cascade encoder 202 where both encoders 210, 220 include LSTM layers, the second encoder 220, which receives the output of the first encoder 210, may utilize the LSTM layers of the first encoder 210 such that the second encoder 220 includes fewer LSTM layers than the first encoder 210 (and fewer LSTM layers than a full non-streaming model). By having fewer LSTM layers, the cascade encoder 202 may reduce the number of more computationally expensive bidirectional layers, making the model 200 more streamlined than simply combining a traditional streaming model with a traditional non-streaming model. In some implementations, to limit the amount of future context the cascade encoder model 200 sees, the second encoder 220 uses some number of conformer layers (e.g., two layers) with a certain amount of right context (e.g., 5 seconds of right context), while the first encoder 210 continues to use LSTM layers. For these implementations, each conformer layer in the second encoder 220 may have 640 units to match the LSTM layer, adding approximately 10 million additional parameters.

[0037] Still referring to FIG. 2, the first encoder 210 receives a sequence of d-dimensional feature vectors (e.g., the acoustic frame 110 shown in FIGS. 1A and 1B) (x1, x2, ..., x r), where:

[0038]

number

[0039] At each time step, we generate a first high-level feature representation, e s Similarly, the second encoder 220 is cascaded to the first encoder 210 and outputs the first high-level feature e s The trained model receives as input e and outputs a second high-level representation, which is a Both the first encoder 210 and the second encoder 220 are directly connected to and shared by the decoder 204. Thus, the decoder 204 receives the first high-level feature representation e s and the second high-level feature representation e a It receives both as input.

[0040] The decoder 204 may include a recurrent neural network-transducer (RNN-T) having a connection layer 230 and a prediction network 300. The decoder 204 combines the first and second high-level feature representations e output by the cascade encoder 202 to generate a decoder output. s , e a , as well as the previous prediction y r-1The decoder output is then passed to an external LM 206, which rescores / improves the initial output from the decoder 204 using techniques such as lattice rescoring or n-best reranking. In other words, the decoder 204 generates a prediction, and the external LM 206 finalizes the prediction by improving the recognition accuracy for rare words or long-tail proper nouns. When the model 200 operates in streaming mode, the combining layer 230 combines the output of the embedding lookup 300 and the first high-level feature representation e from the first encoder 210 to generate a prediction. s Only and receive.

[0041] The decoder output is the sequence of units preceding the N previous non-blank symbols 301 {y i-1 ,...,y i-N} and input x, the current subword unit y i Probability distribution P(y i |y i-1 ,...,y0,x). Although not shown, the model 200 may include a softmax layer that receives the output of the decoder 204. In some implementations, the softmax layer is separate from the decoder 204 and receives the output y r The output of the softmax layer is then used in the beam search process to select the orthographic elements. In some implementations, the softmax layer processes the output y r represents the output of the softmax layer.

[0042] In some examples, the prediction network 300 has two 2048-dimensional LSTM layers, each followed by a 640-dimensional projection layer, such that the LSTM-based embedding lookup 300 may have approximately 23.4 million parameters. When the prediction network 300 includes an LSTM layer, to contribute to the technique for reducing the size of the prediction network 300 without sacrificing the accuracy / performance of the model 200, the prediction network 300 may be configured to use a limited history sequence of non-blank symbols y , which is limited to the N previous non-blank symbols 301 output by the final softmax layer. ui-n ,...,y ui-1 For example, FIG. 3 illustrates a stateless prediction network that receives a sequence of non-blank symbols y , limited to the N previous non-blank symbols 301 a-n output by the last softmax layer. ui-n ,...,y ui-1 1 illustrates a stateless prediction network 300 of model 200 receiving as input N. In some examples, N is equal to 2. In other examples, N is equal to 5, but this disclosure is non-limiting and N may be equal to any integer. A sequence of non-blank symbols 301a-n represents an initial speech recognition result 120a (FIG. 1). In some implementations, the prediction network 300 includes a multi-head attention mechanism 302 that shares a shared embedding matrix 304 across each head 302A-H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads may be used by the multi-head attention mechanism 302. Notably, the multi-head attention mechanism significantly improves performance while minimizing the increase in model size. As described in more detail below, each head 302A-H contains its own row of position vector 308, and rather than incurring an increase in model size by concatenating outputs 318A-H from all heads, the outputs 318A-H are instead averaged by a head averaging module 322.

[0043] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses a shared embedding matrix 304 to estimate the sequence of non-empty symbols y received as input at corresponding time steps from the multiple time steps. ui-n ,...,y ui-1 For each non-whitespace symbol 301 in

[0044]

number

[0045] In particular, the shared embedding matrix 304 is shared across all heads of the multi-head attention mechanism 302, so that all of the other heads 302B-H generate the same corresponding embeddings 306 for each non-blank symbol. Head 302A also generates its own position vector PV Aa-An 308, 308Aa~An (e.g.,

[0046]

number

[0047] ) to a sequence of non-whitespace symbols y ui-n ,...,y ui-1 Each position vector PV 308 assigned to each non-blank symbol indicates a position in the history of the sequence of non-blank symbols (e.g., the N previous non-blank symbols output by the last softmax layer). For example, the first position vector PV Aa is assigned to the most recent position in the history, while the last position vector PV An is assigned to the last position in the history of the N previous non-blank symbols output by the last softmax layer. In particular, each of the embeddings 306 may include the same dimensionality (i.e., dimension size) as each of the position vectors PV 308.

[0048] A sequence of non-blank symbols 301a-n ui-n ,...,y ui-1 The corresponding embeddings generated by the shared embedding matrix 304 for each non-blank symbol 301 in the set of symbols are the same for all of the heads 302A-H of the multi-head attention mechanism 302, but each head 302A-H defines a different set / row of position vectors 308. For example, the first head 302A defines a position vector PV Aa-An 308Aa-An, and the second head 302B defines a position vector PV Ba-Bn 308Ba-Bn, ..., the Hth head 302H defines the position vector PV Ha-Hn Define another distinct line for 308Ha-Hn.

[0049] For each non-blank symbol in the received sequence of non-blank symbols 302a-n, the first head 302A also weights the corresponding embedding 306, via the weight layer 310, in proportion to the similarity between the corresponding embedding and its associated respective position vector PV 308. In some examples, the similarity omc;ides is a cosine similarity (e.g., cosine distance). In the illustrated example, the weight layer 310 outputs a sequence of weighted embeddings 312, 312Aa-An, each of the weighted embeddings 312, 312Aa-An associated with a corresponding embedding 306 that is weighted in proportion to the respective position vector PV 308 assigned to it. In other words, the weighted embedding 312 output by the weight layer 310 for each embedding 306 may correspond to a dot product between the embedding 306 and its associated respective position vector PV 308. The weighted embeddings 312 may be interpreted as relating the embeddings in proportion to how similar they are to the positions associated with their respective position vectors PV 308. To increase computational speed, the prediction network 300 includes non-recurrent layers, so that the sequence of weighted embeddings 312Aa-An is not concatenated, but instead is provided as output from the first head 302A with

[0050]

number

[0051] The weighted embeddings 312Aa-An are averaged by a weighted average module 316 to generate a weighted average 318A of the weighted embeddings 312Aa-An represented by: In Equation 1, h represents the index of the head 302, n represents the position in the context, and e represents the embedding dimension. In addition, in Equation 1, H, N, and d e contains the size of the corresponding dimension. The position vector PV 308 does not need to be trainable and may contain random values. In particular, even if the weighted embedding 312 is averaged, the position vector PV 308 can potentially preserve position history information, alleviating the need for recurrent connections at each layer of the prediction network 300.

[0052] The operations described above with respect to the first head 302A are similarly performed by each of the other heads 302B-H of the multi-head attention mechanism 302. Due to the different set of position vectors PV 308 defined by each head 302, the weight layer 310 outputs a sequence of weighted embeddings 312Ba-Bn, 312Ha-Hn in each of the other heads 302B-H that differs from the sequence of weighted embeddings 312Aa-Aa in the first head 302A. The weighted average module 316 then generates a respective weighted average 318B-H of the corresponding weighted embeddings 312 of the sequence of non-blank symbols as output from each of the other corresponding heads 302B-H.

[0053] In the illustrated example, the prediction network 300 includes a head averaging module 322 that averages the weighted averages 318A-H output from corresponding heads 302A-H. A projection layer 326 with SWISH may receive as an input an output 324 from the head averaging module 322 corresponding to the average of the weighted averages 318A-H and generate as an output a projection output 328. A final layer normalization 330 may be used to normalize a single embedding vector Pu at a corresponding time step from multiple time steps. i The projection output 328 may be normalized to provide a single embedding vector P 350 at each of multiple time steps following the initial time step. i Generates only 350.

[0054] In some configurations, the prediction network 300 does not implement a multi-head attention mechanism 302 and performs only the operations described above with respect to the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is calculated based on a single embedding vector Pu i It is simply passed through the projection layer 326 and layer normalization 330 to provide 350 .

[0055] In other configurations, the prediction network 300 may include a conformer layer or a transformer layer instead of an LSTM layer. In another example, the prediction network 300 includes a V2 embedding lookup table instead of an LSTM layer, a transformer layer, or a conformer layer. At each time step, the V2 embedding lookup table receives as input the two previous predictions (e.g., one-hot vectors) output by the combining layer 230, calculates a respective embedding d1, d2 for each of the two previous predictions, and provides a concatenated output [d1, d2] to the combining layer 230. In comparison, the V2 embedding lookup table may have only about 2 million parameters, whereas the LSTM-based predictive network may include about 23.4 million parameters. Finally, the combining layer 230 may also be a single-layer neural network with 640 hidden units. The softmax layer may be composed of unified word chunks or graphemes generated using all unique word chunks or graphemes in the multiple training datasets.

[0056] The decoder 204 is configured to generate a probability distribution over possible speech recognition hypotheses at each output step. In other words, the combination layer 230 generates a probability distribution over possible speech recognition hypotheses at each output step (e.g., time step). Here, a "possible speech recognition hypothesis" corresponds to a set of output labels / symbols (also called "phonetic units"), each of which represents a grapheme (e.g., a symbol / character) or word fragment in a specified natural language. For example, if the natural language is English, the set of output labels may include 27 symbols, e.g., one label for each of the 26 letters in the English alphabet, and one label specifying a space. Thus, the combination layer 230 may output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels. This set of values ​​may be a vector (e.g., a one-hot vector) and may indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not limited thereto. For example, the set of output labels may include word chunks and / or whole words in addition to or instead of graphemes. The output labels may also be other types of speech units such as phonemes or subphonemes. The output distribution of the combination layer 230 may include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output of the combination layer 230 may include 100 different probability values, one for each output label. The probability distributions may then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, word chunks, and / or words) in a beam search process (e.g., by a softmax layer) to determine the transcription 120.

[0057] In some implementations, the LM206 includes a unidirectional conformer that looks back a predetermined number of tokens (e.g., seven tokens) for each output word fragment model prediction. The conformer LM206 may have a stack of layers (e.g., 12 layers), each layer including 768 model dimensions, 2048 feedforward layer dimensions, and 6 head attention. In these implementations, the conformer LM206 is trained to predict 4096 word fragments.

[0058] Integrating an ASR model with an external LM typically requires shallow fusion. However, overconfidence in the cascaded encoder 202 and decoder 204 can make weighting difficult and often lead to a high rate of word deletion. Therefore, a hybrid autoregressive transducer (HAT) model can be utilized to factorize the internal loss language model score pILM(y) of the model 200, such that the effectiveness score of the model 200 can be expressed as follows: log p(x|y)≒log p(y|x)-log plm(y) (2)

[0059] Thus, HAT factorization allows integration of the model 200 with an external LM 206 without requiring a coverage penalty, as follows: y * =arg max y [λ1log p(y|x)-λ2log pilm(y)+log plm(y)] (3)

[0060] where λ1 and λ2 denote the weights assigned to the external LM 206 and the internal language model, respectively. By using HAT factorization during the training process 300, the LM 206 is better integrated with the cascaded encoder 202 and decoder 204.

[0061] 2, in some implementations, the model 200 operates in both streaming and non-streaming modes in parallel. When operating simultaneously in both streaming and non-streaming modes, the model 200 first provides the first high-level representation e s The model 200 also performs non-streaming speech recognition on the encoded audio data 110, with a second encoder 220 generating a second higher-level representation e. The decoder 204 then generates partial speech recognition results 120, 120a. The model 200 also performs non-streaming speech recognition on the encoded audio data 110, with a second encoder 220 generating a second higher-level representation e. a , the first high-level representation e received from the first encoder 210 is s The decoder 204 then generates a speech recognition result, which is then rescored by the LM 206 to generate the final speech recognition result 120, 120b. As indicated by time, the first encoder 210 generates the partial speech recognition result 120a, while the second encoder 220 waits for the output of the first encoder 210. Finally, the LM 206 may bias the output from the decoder 204 to generate the final speech recognition result 120b. Thus, the final speech recognition result 120b for the input utterance 106 may lag behind the partial speech recognition result 120a for the input utterance. As previously mentioned, the first encoder 210 may identify an end point of the utterance 106 that triggers a microphone closing event to trigger the final speech recognition result 120b to be emitted.

[0062] In some implementations, to further reduce the size of the decoder 204, i.e., the prediction network 300 and the combination layer 230, parameter tying between the prediction network 300 and the combination layer 230 is applied. In particular, the vocabulary size |V| and the embedding dimension d e For , the shared embedding matrix 304 in the prediction network 300 is

[0063]

number

[0064] On the other hand, the last hidden layer is the dimensional size d h The feedforward projection weights from the hidden layer to the output logic are

[0065]

number

[0066] , which has an extra blank token in the vocabulary. Therefore, the feedforward layer corresponding to the last layer of the connection layer 230 has the weight matrix [d h , |V|]. The embedding dimension d e The size of the last hidden layer of the connection layer 230 is the dimension d h By having the prediction network 300 to connect to the embedding dimension d, the feedforward projection weights of the connection layer 230 and the shared embedding matrix 304 of the prediction network 300 can share their weights for all non-blank symbols via a simple transpose transformation. Because the two matrices share all their values, the values ​​only need to be stored once in memory instead of storing two separate matrices. e The size of the hidden layer is d h By setting the embedding dimension d e We reduce the number of parameters equal to the product of |V| and the vocabulary size |V|. This weight concatenation corresponds to a regularization technique.

[0067] FIG. 4 illustrates an example of a data selection pipeline 400 for training an external LM 206 of an ASR system 109. Typically, a large training data set, such as a corpus of training text samples 412, 412a-n, is used to train a language model to run in a computing environment (e.g., the cloud) that is not hindered by processing, memory / storage, and power constraints. However, in some configurations, the ASR system 109 resides on a user device 10 of a user 104, thereby limiting the number of parameters in the LM 206 and, consequently, the number of training samples in the training set used to train the LM 206. To address this, the data selection pipeline 400 processes the corpus of training text samples 412 to reduce the amount of training data (i.e., training text samples) required to train the LM 206 to accurately recognize rare words, thereby allowing the ASR system 109, including the external LM 206, to run on a device with limited processing and / or memory / storage resources. In other words, the data selection pipeline 400 filters the training text samples from the corpus to identify a subset of training samples sufficient to improve rare word recognition. The corpus of training text samples 412 may include 213 billion sentences with a size of about 12 terabytes, with 7.2 billion of the samples being distinct. The pipeline 400 may reduce the number of samples 412 in the corpus for training the LM206 to only about 4 billion sentences, which is 53 times smaller than the original corpus and does not degrade the overall performance of the LM206 in terms of word error rate (WER).

[0068] 4, the data selection pipeline 400 uses a resampling filter 420, a rare word filter 430, and a contrast filter 440 to reduce the number of training text utterances in a corpus of training text samples 412 for training the LM 206. The data selection pipeline 400 obtains a corpus of training data samples 412, 412a-n stored in a training text data store 410 and a plurality of training utterances 452, 452a-n stored in a training utterance data store 450. The training utterances 452 stored in the training utterance data store 450 are used to train the ASR model 200, with each training utterance 452 including audio data 454 corresponding to the utterance and a corresponding transcription 456 of the utterance. Although FIG. 4 shows the pipeline 400 using each of the resampling filter 420, the rare word filter 430, and the contrast filter 440 to select training text data, the pipeline may use only one or two of the resampling filter 420, the rare word filter 430, and the contrast filter 440 to select training text data.

[0069] The resampling filter 420 receives the corpus of training text samples 412 stored in the training text data store 410 and performs a resampling function to identify rare words (e.g., words that occur less frequently) in the corpus by identifying and removing high frequency training text samples from the corpus to output a set of low frequency training text samples 422 (also referred to as a "set of training text samples") corresponding to samples from the corpus of training text samples 412 that contain rare words. In the illustrated example, the resampling filter 420 measures frequency at the sentence level rather than the word level for simplicity. However, the resampling filter 420 may measure the rarity of a sentence from its own collection of words without departing from the scope of this disclosure. As used herein, a word or sentence is rarer if it has a lower frequency (fewer occurrences) in the corpus compared to other words or sentences. The term "tailedness" may be used to describe the relative amount of rare words that occur in the corpus of training text samples 412. The frequency distribution across the corpus of 412 training text samples is linear on a log-log plot, distinct_count(f)≒Af(-∝) (4) where f denotes frequency and A denotes the number of distinct training text samples 412 (i.e., with a frequency f of 1). By changing the power α, the distribution changes. For example, a larger α results in a distribution with a higher frequency of rare words. An example where α approaches infinity indicates that there are no overlapping training texts 412 in the multiple training text samples 412. However, the multiple training text samples 412 stored in the training text data store 410 include α between 1.1 and 2.5. Furthermore, training text samples 412 occurring at excessive frequency rates (e.g., "home" in the map domain) deviate from the linear distribution of frequency distribution.

[0070] To filter high frequency training text from the corpus of training text samples 412, thereby increasing the number of rare words in the set of low frequency training text samples 422, the resampling filter 420 may perform a resampling function including one of a simple power resampling function, a forced power resampling function, or a soft logarithmic resampling function. Simple power resampling may include adjusting the rarity of the frequency distribution distinct_count(f) by applying a parameter β. In this case, the simple power frequency distribution is expressed as Af -αβ In another implementation, constrained power resampling is used to manage over-frequent training text samples in the corpus of training text samples 412 by forcing each training text 412 to fit a line fit. For example, a line fit in the map domain may be expressed as 10 6 In these examples, for each training text sample with a distinct_count of 1, its resampled frequency f1 is 10 regardless of its original frequency f0. 6 In this example, the high original frequency f0 (e.g., 10 8 ) training text samples with 6 The forced power resampling operation is expressed as follows:

[0071]

number

[0072] Alternatively, the resampling filter 420 may perform a soft logarithmic resampling function that matches the original frequency distribution distinct_count(f) of the corpus of training text samples 412 and removes training texts that exceed a threshold from the corpus.

[0073]

number

[0074] where f c denotes the threshold frequency.

[0075] Once the resampling filter 420 has removed the high frequency training text from the corpus of training text samples 412 to output a set of training text samples 422 containing rare words, the set of training text samples 422 is provided as input to the rare word filter 430 and the control filter 440. Removal of the high frequency training text samples from the corpus is desirable, particularly because the high frequency text samples provide a distribution bias that may prevent the LM 206 from learning the long tail of a corpus that contains many rare words. The rare word filter 430 identifies a subset of the rare word training text samples 432 that includes words that do not occur in the transcriptions 456 from the set of training utterances 452 or that occur less than a threshold number of times in the transcriptions 456 from the set of training utterances 452. Similarly, the control filter 440 identifies a subset of the target domain training text samples 442 in the set training text samples 422 that match the target domain associated with the training utterances 452 used to train the ASR model 200. The training utterances 452 may be referred to as ASR training utterances 452, each of which includes ASR audio data 454 paired with a corresponding ASR transcript 456. The data selection pipeline 400 then combines the ASR transcripts 456, rare word training text samples 432, and target domain training text samples 442 into mini-batches for use by the language model trainer 480 to train the LM 206. The mini-batches may be combined according to a sampling ratio (e.g., 20% / 40% / 40% for the ASR transcripts 456 / rare word training text samples 432 / target domain training text samples 442).

[0076] The rare word filter 430 uses a frequency threshold f to identify training text samples for inclusion in the subset of rare word training text samples 432. t(e.g., 15) to directly filter transcriptions 456 from the set of ASR training utterances 452 that contain words that appear in the set of training text samples 422. The rare word filter 420 also identifies any training text samples 422 that do not appear in the transcriptions 456 for inclusion in the subset of rare word training text samples 432. The contrast filter 440 applies contrastive selection / filtering to the set of low frequency training text samples 422 output by the resampling filter 420 to identify a subset of target domain training text samples 442 that match the target domain associated with the set of training utterances 452 used to train the ASR model 200. The corpus of training text samples 412 may include text samples collected from a domain different from the domain for which the ASR model 200 is trained to recognize speech. For example, the text samples may be collected from typed search queries that contain more website names, while the target domain of the ASR model 200 corresponds to voice searches that contain more voice commands. This contrasting choice is made by selecting, for each training text sample in the set of low-frequency training text samples 422, score(x)=L target (x)-L background (x) (7) The score of the training text sample 452 is calculated by: where L denotes the log perplexity of the training text sample 422, target denotes the target LM 206, and background denotes the background language model trained on a fully deduplicated set of training data. A contrastive selection is then adjusted on the transcription 456 of the training utterance 452 to generate the target LM 206. If the training text sample is closer to the transcription 456 of the training utterance 452 used to train the ASR model 200, the training text sample will have a lower score. The contrast filter 440 may then discard the training text samples 422 above the threshold to identify a subset of target domain training texts 442 from the set of low-frequency training text samples 422 below the threshold. As used herein, the target domain associated with the training utterance may include utterances associated with assistant queries, voice search queries, navigation queries, or any other domain. In particular, the ASR model 200 of FIGS. 1-3 is trained on training utterances 452, each of which includes audio data 454 of a corresponding utterance and a corresponding transcription 456 of the utterance that serves as a ground truth label for the audio data 454.

[0077] 5 includes a flowchart of an exemplary arrangement of operations for a method 500 of training a language model 206 for rare word speech recognition. The method 500 includes, at operation 502, obtaining a set of training text samples 412. At operation 504, the method 500 also includes obtaining a set of training utterances 452 used to train the ASR model 200, where each training utterance 452 in the plurality of training utterances 452 includes audio data 454 corresponding to the utterance and a corresponding transcription 456 of the utterance.

[0078] At operation 506, the method 500 includes applying rare word filtering to the set of training text samples 412 to identify a subset of rare word training text samples 432. The subset of rare word training text samples 432 includes words that do not occur in the transcriptions 456 from the set of training utterances 452 or that occur less than a threshold number of times in the transcriptions 456 from the set of training utterances 452. The method 500 further includes, at operation 508, training the external language model 206 on the transcriptions 456 from the set of training utterances 452 and the identified subset of rare word training text samples 432.

[0079] 6 is a schematic diagram of an exemplary computing device 600 that may be used to implement the systems and methods described in this document. The computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the implementations described and / or claimed in this document.

[0080] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or mounted in other manners as needed. Processor 610 (also referred to as "data processing hardware 610," which may include data processing hardware 12 of user device 10 or data processing hardware 62 of remote computing device 60) can process instructions for execution within computing device 600, including instructions stored in memory 620 or on storage device 630 for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 coupled to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, as appropriate. Also, multiple computing devices 600 may be connected, with each device providing some of the required operations (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system).

[0081] Memory 620 (also referred to as "memory hardware 620", which may include memory hardware 14 of user computing device 10 or memory hardware 64 of remote computing device 60) stores information non-transiently within computing device 600. Memory 620 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (program state information) for use by computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disks or tapes.

[0082] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as the methods described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0083] The high-speed controller 640 manages bandwidth-intensive operations for the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. Such role assignments are merely exemplary. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., via a graphics processor or accelerator), and the high-speed expansion port 650, which may accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and the low-speed expansion port 690. The low-speed expansion port 690, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as, for example, a switch or a router via a network adapter.

[0084] Computing device 600 may be implemented in a number of different forms, as shown in the figure. For example, computing device 600 may be implemented as a standard server 600a, or multiple times within a group of servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0085] Various implementations of the systems and techniques described herein can be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0086] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0087] These computer programs (also known as programs, software, software applications, or codes) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0088] The processes and logic flows described herein can be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be executed by special purpose logic circuits, for example, FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. In general, a processor receives instructions and data from a read-only memory, or a random access memory, or both. The essential elements of a computer are a processor for executing instructions, and one or more memories for storing instructions and data. In general, a computer also includes one or more mass storage devices, for example, magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transmit data to, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0089] To provide for user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices can be used to provide for user interaction as well, e.g., feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to the web browser in response to a request received from the web browser on the user's client device.

[0090] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0091] 10 User Device, Device, User Computing Device 12 Data Processing Hardware 14 Memory Hardware 16 Audio System 16, 16a Audio Capture Device, Capture Device 16, 16b Audio output device 18 Digital Assistant Interface 18, 18b Voicemail Application Interface 18b Digital Assistant Interface 19 responses 40 Network 50 Digital assistant applications, programs or applications 50a Digital Assistant Applications 50, 50b Voice Applications 60 Remote Computing Devices 62 Remote Data Processing Hardware, Data Processing Hardware 64 Remote Memory Hardware, Memory Hardware 100, 100a~b Audio environment 104 users 106 Oral utterances, utterances, input utterances 107 User Interface Generator 108 Audio Subsystem 109 Automatic Speech Recognition (ASR) System, ASR System 110 Input audio frame, audio data, audio frame 120 Transcription 120 Recognition results / hypotheses, transcription 120, 120a Partial speech recognition result, initial speech recognition result 120, 120b Final voice recognition results 200 ASR Models, Models, Cascade Encoder Models 202 Cascade Encoder 204 decoder, shared decoder 206 External Language Model (LM), LM, External LM, Conformer LM 210 Cascade Encoder, First Encoder, Causal Encoder, Non-causal Encoder 220 Cascade Encoder, Second Encoder 230 Connecting layer 300 Prediction Networks, Embedding Lookup, Stateless Prediction Networks 301 non-whitespace symbol 301a~n non-blank symbols 302 Multi-head Attention Mechanism 302A~302H Head 304 Shared Embedding Matrix 306, 306a~n Embedded 308, 308Aa~An Position vector, position vector PV 310 Weight Layer 312, 312Aa~An Weighted embedding 316 Weighted Average Module 318A~H output 322 Head Average Module 324 Output 326 Projection layer 328 Projection Output 330 Final layer normalization 350 Embedding Vector Pu i 400 Data Selection Pipeline, Pipeline 410 Training Text Data Store 412, 412a~n Training text samples, training texts 420 resampling filter, rare word filter 422 low frequency training text samples, training text samples 430 Rare Word Filter 432 Rare Words Training Text Samples 440 Control Filter 442 target domain training text samples 450 training utterances data store 452, 452a~n Training utterances, ASR training utterances 454 Audio Data, ASR Audio Data 456 Transcription, ASR Training Utterances 480 Language Model Trainer 600 computing devices 600a Server 600b Laptop Computer 600c Rack Server System 610 Processors, Components, and Data Processing Hardware 620 Memory, components, memory hardware, non-transient memory 630 Storage devices, components 640 High Speed ​​Interface / Controller, Components, High Speed ​​Controller 650 High Speed ​​Expansion Port, Components 660 Low Speed ​​Interface / Controller, Components, Low Speed ​​Controller 670 Slow Bus 680 Display 690 Low Speed ​​Expansion Port

Claims

1. 1. A computer-implemented method (500) for training an external language model (206) to recognize rare words in speech, the method (500), when executed on data processing hardware (610), causing the data processing hardware (610) to: obtaining a set of training text samples (422); obtaining a set of training utterances (452) to be used to train an automatic speech recognition (ASR) model (200), each training utterance (452) in the plurality of training utterances (452) including audio data (454) corresponding to the utterance and a corresponding transcription (456) of the utterance; applying rare word filtering to the set of training text samples (422) to identify a subset of rare word training text samples (432) that includes words that do not appear in the transcripts (456) from the set of training utterances (452) or that appear less than a threshold number of times in the transcripts (456) from the set of training utterances (452); training the external language model (206) on the transcriptions (456) from the set of training utterances (452) and the identified subset of the rare word training text samples (432); performing an action including A computer-implemented method (500).

2. The step of obtaining a set of training text samples (422) comprises: receiving a corpus of training text samples (412); performing a resampling function on the corpus of training text samples (412) to identify high frequency text samples occurring within the corpus of training text samples (412); obtaining the set of training text samples (422) by removing the identified high frequency text samples from the corpus of training text samples (412); 2. The computer-implemented method of claim 1 (500), comprising:

3. 3. The computer-implemented method (500) of claim 1 or 2, wherein the resampling function comprises one of a simple power resampling function, a constrained power resampling function, or a soft logarithmic resampling function.

4. The operation, applying contrastive filtering to the set of training text samples (442) to identify a subset of target domain training text samples (442) that match a target domain associated with the set of training utterances (452); training the external language model (206) on the identified subset of the transcriptions (456) from the set of training utterances (452) and the rare word training text samples (432) further comprises training the external language model (206) on an identified subset of the target domain training text samples (442) that match the target domain; A computer-implemented method (500) according to any one of claims 1 to 3.

5. 5. The computer-implemented method of claim 1, wherein the external language model comprises an external neural language model.

6. 6. The computer-implemented method of claim 5, wherein the external neural language model comprises a stack of conformer or transformer layers.

7. 7. The computer-implemented method of claim 1, wherein the operations further comprise integrating the trained language model with the ASR model, wherein the trained external language model is configured to rescore a probability distribution over possible speech recognition hypotheses predicted by the trained ASR model.

8. The ASR model (200), receiving as input a sequence of acoustic frames (110); In each of a plurality of output steps, the method is configured to generate a first high-level feature representation for a corresponding acoustic frame (110) in the sequence of acoustic frames (110). A first encoder (210); receiving as input the first high-level feature representation generated by the first encoder (210) in each of the plurality of output steps; In each of the plurality of output steps, a second high-level feature representation for a corresponding first high-level feature frame is generated. A second encoder (220); receiving as an input the second high-level feature representation generated by the second encoder (220) in each of the plurality of output steps; configured to generate a first probability distribution (120a) over possible speech recognition hypotheses at each of the plurality of time steps. Decoder (204) and 8. The computer-implemented method of claim 7 (500), comprising:

9. The decoder (204) receiving as an input the first high-level feature representation generated by the first encoder (210) in each of the plurality of output steps; generating a second probability distribution (120b) over possible speech recognition hypotheses at each of the plurality of time steps; The computer-implemented method (500) of claim 7 or 8, further configured to:

10. The decoder (204) At each of the plurality of time steps, It receives as input the sequence of N previous non-blank symbols output by the final softmax layer, generating a respective embedding (306) for each non-blank symbol in the sequence of N previous non-blank symbols; Generate an average embedding (312) by averaging each of the embeddings (306). A prediction network (300) configured as follows: the mean embedding (312) generated by the predictive network (300) at each of the plurality of output steps; the first high-level feature representation generated by the first encoder (210) in each of the plurality of output steps when the ASR model (200) is operating in a streaming mode; or When the ASR model (200) is operating in a non-streaming mode, the second high-level feature representation generated by the second encoder (220) in each of the plurality of output steps is and one of In each of the plurality of output steps, the second probability distribution (120b) over possible speech recognition hypotheses when the ASR model (200) is operating in the streaming mode; or When the ASR model (200) is operating in a non-streaming mode, the first probability distribution (120a) over possible speech recognition hypotheses. Generate one of the A coupling network (230) configured as follows:

10. The computer-implemented method (500) of any one of claims 7 to 9, comprising:

11. the first encoder (210) comprises a causal encoder including an initial stack of conformer layers; the second encoder (220) comprises a non-causal encoder including a final stack of conformer layers superimposed on the initial stack of conformer layers; A computer-implemented method (500) according to any one of claims 8 to 10.

12. 12. The computer-implemented method of claim 8, wherein the first encoder (210) and the second encoder (220) of the ASR model (200) are trained using hybrid autoregressive transducer factorization to facilitate the integration of the external language model (206) trained on text-only data including the transcriptions (456) from the set of training utterances (452) and the identified subset of the rare word training text samples (432).

13. Data processing hardware (610); When executed on the data processing hardware (610), the data processing hardware (610) obtaining a set of training text samples (422); obtaining a set of training utterances (452) for use in training an automatic speech recognition (ASR) model (200), each training utterance (452) in the plurality of training utterances (452) including audio data (454) corresponding to the utterance and a corresponding transcription (456) of the utterance; applying rare word filtering to the set of training text samples (422) to identify a subset of rare word training text samples (432) that includes words that do not appear in the transcripts (456) from the set of training utterances (452) or that appear less than a threshold number of times in the transcripts (456) from the set of training utterances (452); training the external language model (206) on the transcriptions (456) from the set of training utterances (452) and the identified subset of the rare word training text samples (432); A memory hardware (620) for storing instructions for performing operations including A system comprising:

14. Obtaining the set of training text samples (422) receiving a corpus of training text samples (412); performing a resampling function on the corpus of training text samples (412) to identify high frequency text samples occurring within the corpus of training text samples (412); obtaining the set of training text samples (422) by removing the identified high frequency text samples from the corpus of training text samples (412); The system of claim 13, comprising:

15. 15. The system of claim 13 or 14, wherein the resampling function comprises one of a simple power resampling function, a constrained power resampling function, or a soft logarithmic resampling function.

16. The operation, applying contrastive filtering to the set of training text samples (442) to identify a subset of target domain training text samples (442) that match a target domain associated with the set of training utterances (452); training the external language model (206) on the identified subset of the transcriptions (456) from the set of training utterances (452) and the rare word training text samples (432) further comprises training the external language model (206) on the identified subset of the target domain training text samples (442) that match the target domain.

16. A system according to any one of claims 13 to 15.

17. The system of any one of claims 13 to 16, wherein the external language model (206) comprises an external neural language model.

18. 20. The system of claim 17, wherein the external neural language model comprises a stack of conformer or transformer layers.

19. 19. The system of claim 13, wherein the operations further include integrating the trained language model with the trained ASR model, wherein the trained external language model is configured to rescore a probability distribution over possible speech recognition hypotheses predicted by the trained ASR model.

20. The ASR model (200), receiving as input a sequence of acoustic frames (110); In each of a plurality of output steps, the method is configured to generate a first high-level feature representation for a corresponding acoustic frame (110) in the sequence of acoustic frames (110). A first encoder (210); receiving as an input the first high-level feature representation generated by the first encoder (210) in each of the plurality of output steps; In each of the plurality of output steps, a second high-level feature representation for a corresponding first high-level feature frame is generated. A second encoder (220); receiving as an input the second high-level feature representation generated by the second encoder (220) in each of the plurality of output steps; configured to generate a first probability distribution (120a) over possible speech recognition hypotheses at each of the plurality of time steps. Decoder (204) and 20. The system of claim 19, comprising:

21. The decoder (204) receiving as an input the first high-level feature representation generated by the first encoder (210) in each of the plurality of output steps; generating a second probability distribution (120b) over possible speech recognition hypotheses at each of the plurality of time steps.

21. The system of claim 19 or 20, further configured to:

22. The decoder (204) At each of the plurality of time steps, It receives as input the sequence of N previous non-blank symbols output by the final softmax layer, generating a respective embedding (306) for each non-blank symbol in the sequence of N previous non-blank symbols; Generate an average embedding (312) by averaging each of the embeddings (306). A prediction network (300) configured as follows: a mean embedding (312) generated by the predictive network (300) at each of the plurality of output steps; and the first high-level feature representation generated by the first encoder (210) in each of the plurality of output steps when the ASR model (200) is operating in a streaming mode; or When the ASR model (200) is operating in a non-streaming mode, a second high-level feature representation generated by the second encoder (220) in each of the plurality of output steps. Receive one of In each of the plurality of output steps, the second probability distribution (120b) over possible speech recognition hypotheses when the ASR model (200) is operating in a streaming mode; or When the ASR model (200) is operating in a non-streaming mode, the first probability distribution (120a) over possible speech recognition hypotheses. Generate one of the The coupling network is configured as follows:

22. The system of claim 19, comprising:

23. the first encoder (210) comprises a causal encoder comprising an initial stack of conformer layers; the second encoder (220) comprises a non-causal encoder comprising a final stack of conformer layers superimposed on the initial stack of conformer layers; 23. A system according to any one of claims 20 to 22.

24. 24. The system of claim 20, wherein the first encoder (210) and the second encoder (220) of the ASR model (200) are trained using hybrid autoregressive transducer factorization to facilitate integration of the external language model (206) trained on text-only data including the transcriptions (456) from the set of training utterances (452) and the identified subset of the rare language training text samples (432).

Citation Information

Patent Citations

  • Japanese speech recognition model learning device and program

    JP2018066800A

  • Input support method, input support system, and program

    JP2021043530A