Word timing output using an end-to-end model

Abstract

To provide a two-pass end-to-end voice recognition system and a computer implementation method without compromising latency and quality.SOLUTION: A method includes: receiving a training example which includes audio data representing a spoken utterance and a ground truth transcription; inserting a placeholder symbol before respective words during utterance; identifying a ground truth alignment for a beginning and an end of the word: and generating a first constrained alignment for a beginning word piece and a second constrained alignment for an ending word piece. The first constrained alignment is aligned with the ground truth alignment for the beginning of the respective words and the second constrained alignment is aligned with the ground truth alignment for the ending of the respective words. The method also includes constraining an attention head of a second pass decoder by applying the first and second constrained alignments.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] This disclosure relates to two-pass end-to-end speech recognition. [Background technology]

[0002] Modern automatic speech recognition (ASR) systems focus not only on high quality (e.g., low word error rate (WER)) but also on low latency (e.g., short delay between user utterance and transcription appearance). Furthermore, when using ASR systems today, there is a demand for them to decode utterances in a real-time-compatible streaming manner, or even faster than real-time. For example, when an ASR system is deployed on a mobile phone with direct user interaction, applications on the mobile phone using the ASR system may require speech recognition to be streaming so that words appear on the screen as soon as they are spoken. Here, mobile phone users may have low tolerance for latency. Because of this low tolerance, speech recognition strives to run on mobile devices in a way that minimizes the impact of latency and inaccuracies that could negatively affect the user experience. [Overview of the project] [Means for solving the problem]

[0003] One aspect of the present disclosure provides a computer implementation method that, when executed on data processing hardware, causes data processing hardware to perform an operation, the operation including receiving a training example for a second path decoder of a two-pass neural network model. The training example includes audio data representing oral utterances of one or more words and a corresponding ground truth transcription of the oral utterances. For each word in the oral utterance, the operation also includes inserting a placeholder symbol before each word, identifying the respective ground truth alignments for the beginning and end of each word, determining the beginning word piece and the end word piece of each word, and generating a first constrained alignment for the beginning word piece and a second constrained alignment for the end word piece of each word. A first constrained alignment is aligned with the ground truth alignment for the beginning of each word, and a second constrained alignment is aligned with the ground truth alignment for the end of each word. The operation also involves constraining the attention head of the second path decoder of the two-pass neural network model by applying the training example, which includes all of the first and second constrained alignments for each word in the training example.

[0004] Implementations of the present disclosure may include one or more of the following features. In some implementations, the operation further includes identifying expected attention probabilities for portions of the training example while training a second path decoder in the training example, determining that a constrained attention head generates an attention probability for at least one portion of the training example that fails to match the expected attention probability, and applying a training penalty to the constrained attention head. In these implementations, the attention probability for at least one portion of the training example may occur at a time corresponding to either a first or second constrained alignment. On the other hand, the attention probability for at least one portion of the training example may, in some cases, occur at a time that does not correspond to either a first or second constrained alignment.

[0005] The start and end word pieces may include the same word piece for each word, but the second path decoder may include multiple attention heads. In some examples, constraining the attention heads includes constraining the attention probabilities derived from the attention heads of the second path decoder. Each constrained alignment may include a timing buffer for each ground truth alignment, where the timing buffer constrains each of the first and second constrained alignments to a time interval including a first time period before each ground truth alignment and a second time period after each ground truth alignment.

[0006] In some examples, the operation further includes determining that the constrained attention head generates non-zero attention probabilities outside the boundaries corresponding to the first and second constrained alignments while training the second path decoder in the training example, and applying a training penalty to the constrained attention head. Additional or alternative, the operation may further include minimizing the attention loss for the constrained attention head and minimizing the cross-entropy loss for the second path decoder while training the second path decoder in the training example. While running the two-pass neural network using the second path decoder trained in the training example, in some additional examples, the operation further includes receiving audio data of the utterance, determining the time corresponding to the maximum probability in the constrained attention head of the second path decoder, and generating a word start time or word end time for the determined time corresponding to the maximum probability in the constrained attention head of the second path decoder.

[0007] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware communicating with the data processing hardware. The memory hardware, when executed by the data processing hardware, stores instructions causing the data processing hardware to perform an operation, the operation including receiving a training example for a second path decoder of a two-pass neural network model. The training example includes audio data representing oral utterances of one or more words and corresponding ground truth transcriptions of the oral utterances. For each word in the oral utterance, the operation also includes inserting a placeholder symbol before each word, identifying the respective ground truth alignments for the beginning and end of each word, determining the beginning word piece and the end word piece of each word, and generating a first constrained alignment for the beginning word piece and a second constrained alignment for the end word piece of each word. A first constrained alignment is aligned with the ground truth alignment for the beginning of each word, and a second constrained alignment is aligned with the ground truth alignment for the end of each word. The operation also involves constraining the attention head of the second path decoder of the two-pass neural network model by applying the training example, which includes all of the first and second constrained alignments for each word in the training example.

[0008] This embodiment may include one or more of the following features. In some implementations, the operation further includes identifying expected attention probabilities for portions of the training example while training a second path decoder in the training example, determining that a constrained attention head generates an attention probability for at least one of the portions of the training example that fails to match the expected attention probability, and applying a training penalty to the constrained attention head. In these implementations, the attention probability for at least one of the portions of the training example may occur at a time corresponding to either a first or second constrained alignment. On the other hand, the attention probability for at least one of the portions of the training example may, in some cases, occur at a time that does not correspond to either a first or second constrained alignment.

[0009] The start and end word pieces may include the same word piece for each word, but the second path decoder may include multiple attention heads. In some examples, constraining the attention heads includes constraining the attention probabilities derived from the attention heads of the second path decoder. Each constrained alignment may include a timing buffer for each ground truth alignment, where the timing buffer constrains each of the first and second constrained alignments to a time interval including a first time period before each ground truth alignment and a second time period after each ground truth alignment.

[0010] In some examples, the operation further includes determining that the constrained attention head generates non-zero attention probabilities outside the boundaries corresponding to the first and second constrained alignments while training the second path decoder in the training example, and applying a training penalty to the constrained attention head. Additional or alternative, the operation may further include minimizing the attention loss for the constrained attention head and minimizing the cross-entropy loss for the second path decoder while training the second path decoder in the training example. While running the two-pass neural network using the second path decoder trained in the training example, in some additional examples, the operation further includes receiving audio data of the utterance, determining the time corresponding to the maximum probability in the constrained attention head of the second path decoder, and generating a word start time or word end time for the determined time corresponding to the maximum probability in the constrained attention head of the second path decoder.

[0011] Details of one or more implementations of this disclosure are described in the accompanying drawings and the following description. Other embodiments, features, and advantages will become apparent from the description and drawings, as well as from the claims. [Brief explanation of the drawing]

[0012] [Figure 1A] This is a schematic diagram of an exemplary speech environment using a two-pass architecture with a shared acoustic and text model. [Figure 1B] This is a schematic diagram of an exemplary speech environment using a two-pass architecture with a shared acoustic and text model. [Figure 1C] This is a schematic diagram of an exemplary speech environment using a two-pass architecture with a shared acoustic and text model. [Figure 2] This is a schematic diagram of an exemplary two-pass architecture for speech recognition. [Figure 3A] This is a schematic diagram of an exemplary training process for the two-pass architecture shown in Figure 2 for speech recognition. [Figure 3B] Figure 3A is a schematic diagram of exemplary constraints in a training example for the training process. [Figure 4] This is an illustrative flowchart of the operational configuration for implementing the two-pass architecture shown in Figure 2, which incorporates constrained attention. [Figure 5] This is a schematic diagram of an exemplary computing device that may be used to implement the systems and methods described herein. [Modes for carrying out the invention]

[0013] Similar reference numerals in various drawings indicate the same elements.

[0014] Speech recognition continues to evolve to meet the demands of freedom and agility in mobile environments. New speech recognition architectures, or improvements to existing architectures, are continuously being developed to enhance the quality of automatic speech recognition systems (ASRs). For example, speech recognition initially employed multiple models, each with a specialized purpose. For instance, an ASR system might have included an acoustic model (AM), a pronunciation model (PM), and a language model (LM). The acoustic model mapped audio segments (i.e., audio frames) to phonemes. The pronunciation model connected these phonemes to form words, and the language model was used to represent the likelihood of a given phrase (i.e., the probability of a sequence of words). However, while these individual models worked together, each model was trained independently and often manually designed on different datasets.

[0015] The separate model approach has made it possible for speech recognition systems to be fairly accurate, especially when the training corpus (i.e., the body of training data) for a given model meets the requirements for model effectiveness. However, the need to train separate models independently introduces its own complexity, leading to architectures with integrated models. These integrated models attempt to use a single neural network to directly map audio waveforms (i.e., input sequences) to output sentences (i.e., output sequences). This gives rise to sequence-to-sequence methods, which, given a sequence of audio features, generate a sequence of words (or graphemes). Examples of sequence-to-sequence models include "attention-based" models and "listen-attend-spell" (LAS) models. LAS models transcribe speech utterances into characters using listener components, attender components, and speller components. Here, the listener is a recurrent neural network (RNN) encoder that receives audio input (e.g., a time-frequency representation of the speech input) and maps the audio input to a higher-level feature representation. The attendant pays attention to higher-level features to learn the alignment between input features and predicted subword units (e.g., graphemes or word pieces) or other units of speech (e.g., phonemes, sounds, or senomes). The speller is an attention-based RNN decoder that generates character sequences from the input by creating a probability distribution across a hypothetical set of words. Using an integrated structure, all components of the model can be trained jointly as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is built entirely of neural networks. A complete neural network functions without external and / or manually designed components (e.g., finite-state transducers, dictionaries, or text normalization modules).Additionally, when training E2E models, these models generally do not require bootstrapping from a decision tree or time alignment from a separate system.

[0016] Early end-to-end (E2E) models proved accurate and demonstrated superior training improvements compared to individually trained models. However, these E2E models, such as LAS models, worked by considering the entire input sequence before generating output text, and therefore did not allow for streaming of the output as the input was received. Lacking streaming capabilities, LAS models cannot perform real-time voice transcription. This flaw can cause problems when deploying LAS models for voice applications that are latency-sensitive and / or require real-time voice transcription. This makes LAS models alone unsuitable for mobile technologies (e.g., mobile phones) that often rely on real-time applications (e.g., real-time communication applications).

[0017] Additionally, an audio recognition system having an acoustic model, a pronunciation model, and a language model, or such models configured together, may rely on a decoder that must search a relatively large search graph associated with these models. In the case of a large search graph, it is not beneficial to host this type of audio recognition system entirely on-device. Here, when an audio recognition system is hosted "on-device", the device that receives the audio input uses its processor to execute the functions of the audio recognition system. For example, when an audio recognition system is entirely hosted on-device, the device's processor does not need to cooperate with any off-device computing resources to execute the functions of the audio recognition system. Instead of being entirely on-device, a device that performs audio recognition relies on remote computing (e.g., of a remote computing system or cloud computing) and thus online connectivity to execute at least some of the functions of the audio recognition system. For example, an audio recognition system uses a network connection to a server-based model to perform decoding using a large search graph.

[0018] Unfortunately, by relying on remote connections, speech recognition systems are susceptible to the latency issues and / or inherent unreliability of communication networks. To improve the usefulness of speech recognition by circumventing these problems, speech recognition systems have evolved again into a sequence-to-sequence model known as the recurrent neural network transducer (RNN-T). Unlike other sequence-to-sequence models that do not employ an attention mechanism and generally require processing an entire sequence (e.g., an audio waveform) to produce an output (e.g., a sentence), RNN-T processes input samples sequentially and streams output symbols, a feature particularly attractive for real-time communication. For example, speech recognition using an RNN-T may output characters one by one as spoken. Here, the RNN-T uses a feedback loop, feeding back the symbols predicted by the model to itself to predict the next symbol. Because decoding an RNN-T involves beam search through a single neural network rather than a large decoder graph, RNN-Ts can scale to a fraction of the size of server-based speech recognition models. The size reduction allows RNN-T to be deployed entirely on-device and run offline (i.e., without network connectivity), thus avoiding reliability issues related to communication networks.

[0019] In addition to operating with low latency, speech recognition systems must also be accurate in their speech recognition. For models that perform speech recognition, a metric that can define the accuracy of the model is often the word error rate (WER). WER refers to a measure of how many words are changed compared to the number of words actually spoken. Generally, these word changes refer to substitutions (i.e., when words are replaced), insertions (i.e., when words are added), and / or deletions (i.e., when words are omitted). For example, a speaker says "car," but the ASR system transcribes the word "car" as "bar." This is an example of substitution for phonetic similarity. When measuring the capabilities of an ASR system compared to other ASR systems, WER may indicate some measure of improvement or quality capability over another system or some baseline.

[0020] The RNN-T model is promising as a strong candidate model for on-device speech recognition. However, the RNN-T model alone still lags behind large-scale state-of-the-art traditional models (e.g., server-based models with separate AM, PM, and LM) in terms of quality (e.g., speech recognition accuracy). However, the non-streaming E2E, LAS model has speech recognition quality comparable to large-scale state-of-the-art traditional models. To utilize the quality of the non-streaming E2E LAS model, a two-pass speech recognition system (e.g., shown in Figure 2A) including a first-pass component of the RNN-T network and a second-pass component of the subsequent LAS network has been developed. In this design, the two-pass model benefits from the streaming nature of the RNN-T model, which has low latency, and at the same time enhances the accuracy of the RNN-T model through the second pass incorporating the LAS network. The LAS network increases latency compared to only the RNN-T model, but the increase in latency is moderately small and conforms to the latency constraints for on-device operation. In terms of accuracy, the two-pass model achieves a 17 - 22% WER reduction compared to the RNN-T alone and has a similar WER compared to large-scale traditional models.

[0021] Unfortunately, this two-pass model, which has an RNN-T network as the first pass and an LAS network as the second pass, has several shortcomings. For example, this type of two-pass model is generally unable to convey timing for words (e.g., start or end time for each word) because it is not trained with alignment information like conventional models. Without alignment information, two-pass models often delay their output predictions, making it difficult to determine word timing. In contrast, conventional models are trained with alignment information, such as phoneme alignment or word alignment, which enables conventional models to generate accurate word timing. This creates a trade-off for the user of the speech recognition system. On the one hand, two-pass models have the advantage of being on-device, providing privacy and minimal latency, but they lack the ability to emit word timing. On the other hand, large conventional models can generate accurate word timing, but they are too large to be implemented on-device, forcing users to use remote-based, non-streaming speech recognition systems, which may have increased latency (compared to, for example, two-pass models).

[0022] To generate word timings without compromising latency or quality, a two-pass model can be adapted to leverage its own architecture, along with additional constraints. In other words, a two-pass model cannot incorporate elements of large, conventional models due to size constraints that must be adapted for on-device use, nor can it increase its overall latency by using post-processing modules after generating the final hypothesis. Fortunately, while training the LAS network in the second pass, the attention probabilities for the LAS network learn the alignment between the audio corresponding to the training examples and the predicted subword units (e.g., graphemes, word pieces, etc.) for the training examples. By constraining the attention probabilities of the LAS network based on word-level alignment, a two-pass model can generate start and end times for each word. With these word timings, users can use the two-pass model on-device with a variety of applications, such as voice assistants, dictation applications, or video transcription.

[0023] Figures 1A and 1C show examples of a voice environment 100. In the voice environment 100, the way a user 10 interacts with a computing device such as a user device 110 may be through voice input. The user device 110 (also commonly referred to as device 110) is configured to capture sound (e.g., streaming audio data) from one or more users 10 within the voice-enabled environment 100. Here, the streaming audio data 202 may refer to oral utterances 12 by the user 10 that function as audible queries (e.g., Figure 1C), commands to device 110, or audible communications captured by device 110 (e.g., Figure 1B). The voice-enabled system of device 110 may respond to queries or commands by responding to queries and / or executing commands.

[0024] Here, the user device 110 captures audio data 202 of the spoken utterance 12 by the user 10. The user device 110 may correspond to any computing device associated with the user 10 and capable of receiving the audio data 202. Some examples of the user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smartwatches), smart appliances, Internet of Things (IoT) devices, smart speakers, etc. The user device 110 includes data processing hardware 112 and memory hardware 114 that communicates with the data processing hardware 112 and stores instructions, which, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more actions. The user device 110 further includes an audio subsystem 116 having audio capture devices (e.g., microphones) 116, 116a for capturing oral utterances 12 within the voice-enabled system 100 and converting them into electrical signals, and audio output devices (e.g., speakers) 116, 116b for communicating audible audio signals (e.g., as output audio data from device 110). In the illustrated example, the user device 110 implements a single audio capture device 116a, but the user device 110 may implement an array of audio capture devices 116a without departing from the scope of the disclosure, so that one or more capture devices 116a in the array may not be physically present on the user device 110 but may be communicating with the audio subsystem 116.

[0025] The user device 110 (for example, using hardware 112, 114) is further configured to perform speech recognition processing on streaming audio data 202 using a speech recognition device 200. In some examples, the audio subsystem 116 of the user device 110, including an audio capture device 116a, is configured to receive the audio data 202 (for example, oral utterances 12) and convert the audio data 202 into a digital format suitable for the speech recognition device 200. The digital format may correspond to acoustic frames (for example, parameterized acoustic frames), such as mel frames. For example, parameterized acoustic frames correspond to logarithmic mel filter bank energies.

[0026] In some implementations, such as Figure 1A, user 10 interacts with a program or application 118 on a user device 110 that uses a speech recognition system 200. For example, Figure 1A shows user 10 communicating with a transcription application 118 that can transcribe utterances 12 spoken by user 10. In this example, user 10's oral utterance 12 is "What time is the concert tonight?". This question from user 10 is an oral utterance 12 that is captured by an audio capture device 116a and processed by the audio subsystem 116 of user device 110. In this example, the speech recognition system 200 of user device 110 receives the audio input 202 of "what time is the concert tonight?" (for example, as an acoustic frame) and transcribes the audio input 202 into a transcription 204 (for example, a textual representation of "what time is the concert tonight?"). Here, the transcription application 118 labels each word in the transcription 204 with its corresponding start and end times, based on the word timings 206 generated by the speech recognition system 200. For example, using these start and end times, user 10 can edit the transcription 204, or the audio corresponding to the transcription 204. In some examples, the transcription application 118 corresponds to a video transcription application, which is configured to edit and / or process audio / video data on user device 110 based on the start and end times that the speech recognition system 200 associates with words in the transcription 204.

[0027] Figure 1B shows another example of speech recognition using a speech recognition device 200. In this example, a user 10 associated with a user device 110 is communicating with a friend named Jane Doe using a communication application 118. Here, user 10, named Ted, communicates with Jane by having the speech recognition device 200 transcribe his voice input. An audio capture device 116 captures these voice inputs and communicates them to the speech recognition device 200 in digital format (e.g., acoustic frames). The speech recognition device 200 transcribes these acoustic frames into text that is sent to Jane via the communication application 118. Since this type of application 118 communicates via text, the transcription 204 from the speech recognition device 200 can be sent to Jane without further processing (e.g., natural language processing). Here, the communication application 118 may use the speech recognition device 200 to associate time with one or more parts of the conversation. Depending on the communication application 118, these times may be quite specific, corresponding to word timings 206 for each word of the conversation processed by the speech recognition unit 200 (e.g., start and end times for each word), or more generally, they may correspond to the time associated with each speaker's portion of the conversation (e.g., as shown in Figure 1B).

[0028] Figure 1C shows a conversation very similar to Figure 1B, but with a voice assistant application 118. In this example, user 10 asks the automated assistant, "What time is the concert tonight?" This question from user 10 is an oral utterance 12 that is captured by the audio capture device 116a and processed by the audio subsystem 116 of user device 110. In this example, the speech recognition 200 of user device 110 receives the audio input 202 of "what time is the concert tonight?" (for example, as an acoustic frame) and transcribes the audio input 202 into a transcription 204 (for example, a textual representation of "what time is the concert tonight?"). Here, the automated assistant of application 118 may use natural language processing to respond to the question posed by user 10. Natural language processing generally refers to the process of interpreting written language (e.g., transcription 204) and determining whether the written language prompts any action. In this example, the automated assistant uses natural language processing to recognize that a question from user 10 relates to the user's schedule, and more specifically, to a concert on the user's schedule. By recognizing these details using natural language processing, the automated assistant returns a response to the user's query, where the response states, "Tonight's concert opens at 8:30 p.m." In some configurations, natural language processing is performed on a remote system communicating with data processing hardware 112 of the user device 110. Similar to Figure 1B, the speech recognition 200 may emit timestamps (e.g., word timings 206) that the voice assistant application 118 can use to provide additional details about the conversation between user 10 and the automated assistant. For example, Figure 1C shows the voice assistant application 118 labeling the user's question using the time the user's question occurred.

[0029] In some examples, such as Figure 2, the speech recognition system 200 is configured in a two-pass architecture. Generally, the two-pass architecture of the speech recognition system 200 includes at least one encoder 210, an RNN-T decoder 220, and an LAS decoder 230. In two-pass decoding, a second pass 208, 208b (for example, shown as the LAS decoder 230) can improve upon the initial output from the first pass 208, 208a (for example, shown as the RNN-T decoder 220) using techniques such as lattice rescoring or n-best re-ranking. In other words, the RNN-T decoder 220 produces a streaming prediction, and the LAS decoder 230 confirms the prediction. Specifically, the LAS decoder 230 receives the streaming hypothesis y from the RNN-T decoder 220. R The LAS decoder 230 rescores the streamed hypothesis y from the RNN-T decoder 220. R While it is generally described as functioning in a rescoring mode that rescores, the LAS decoder 230 can also operate in different modes, such as beam search mode, depending on the design or other factors (e.g., utterance length).

[0030] At least one encoder 210 is configured to receive acoustic frames corresponding to streaming oral utterances 12 as an audio input 202. The acoustic frames may be preprocessed by the audio subsystem 116 to become parameterized acoustic frames (e.g., Mel frames and / or spectral frames). In some implementations, the parameterized acoustic frames correspond to log-Mel filter bank energies with log-Mel features. For example, the parameterized input acoustic frame output by the audio subsystem 116 and input to encoder 210 is x=(x1,...,x T ) can be expressed as, however,

[0031]

number

[0032] is the log-Mel filter bank energy, where T is the number of frames at x, and d is the number of log-Mel features. In some examples, each parameterized acoustic frame contains a 128-dimensional log-Mel feature computed within a short shift window (e.g., 32 milliseconds, and shifted every 10 milliseconds). Each feature may have previous frames (e.g., three previous frames) stacked to form a higher-dimensional vector (e.g., a 512-dimensional vector using three previous frames). The features forming the vector can then be downsampled (e.g., to a 30 millisecond frame rate). Based on the audio input 202, the encoder 210 is configured to produce an encoding e. For example, the encoder 210 produces an encoded acoustic frame (e.g., an encoded Mel frame or acoustic embedding).

[0033] The structure of encoder 210 can be implemented in different ways, but in some implementations, encoder 210 is a long short-term memory (LSTM) neural network. For example, encoder 210 includes eight LSTM layers, where each layer may have 2,048 hidden units and a subsequent 640-dimensional projection layer. In some examples, a time-shortening layer is inserted after the second LSTM layer of encoder 210 with a shortening factor N=2 (for example, to ensure that encoded features occur at a specific frame rate).

[0034] In some configurations, the encoder 210 is a shared encoder network. In other words, each path 208 shares a single encoder 210, rather than each path network 208 having its own separate encoder. By sharing an encoder, the ASR speech recognition system 200 using a two-path architecture can reduce its model size and / or computational cost. Here, the reduction in model size can help enable the speech recognition system 200 to function well entirely on-device.

[0035] In some examples, the speech recognition system 200 in Figure 2 also includes an additional encoder, such as an acoustic encoder 250, to adapt the output 212 of the encoder 210 to suit the second pass 208b of the LAS decoder 230. The acoustic encoder 250 is configured to further encode the output 212 to become the encoded output 252. In some implementations, the acoustic encoder 250 is an LSTM encoder (e.g., a two-layer LSTM encoder) that further encodes the output 212 from the encoder 210. By including the additional encoder, the encoder 210 can still be maintained as a shared encoder between passes 208b.

[0036] During the first pass 208a, the encoder 210 receives each audio frame of the audio input 202 and generates an output 212 (shown, for example, as the encoding e of the audio frame). The RNN-T decoder 220 receives the output 212 for each frame and, in a streaming manner at each time step, hypothesis y RThis produces output 222, which is shown as follows. In some implementations, the RNN-T decoder 220 includes a prediction network and a co-network. Here, the prediction network may have two LSTM layers with 2,048 hidden units and 640-dimensional projections per layer, as well as an embedding layer of 128 units. The outputs 212 of the encoder 210 and the prediction network can be fed into the co-network, which includes a softmax prediction layer. In some examples, the co-network of the RNN-T decoder 220 includes a subsequent softmax layer that predicts 640 hidden units and 4,096 mixed-case word pieces.

[0037] In the two-pass model in Figure 2, during the second pass 208b, the LAS decoder 230 receives the output 212 from the encoder 210 for each frame, and hypothesizes y LThis generates output 232, which is shown as follows. When the LAS decoder 230 operates in beam search mode, it produces output 232 from output 212 only, ignoring output 222 of the RNN-T decoder 220. When the LAS decoder 230 operates in rescoring mode, it obtains the top K hypotheses from the RNN-T decoder 220, and then, paying attention to output 212, it runs on each sequence in teacher-forced mode to calculate a score. For example, the score is a combination of the log probability of the sequence and an attention coverage penalty. The LAS decoder 230 selects the sequence with the highest score to produce output 232. Here, in rescoring mode, the LAS decoder 230 may include multi-head attention (e.g., with four heads) to pay attention to output 212. Furthermore, the LAS decoder 230 may be a two-layer LAS decoder 230 with a softmax layer for prediction. For example, each layer of the LAS decoder 230 has 2,048 hidden units and a subsequent 640-dimensional projection. The softmax layer may contain 4,096 dimensions to predict the same mixed-case word piece from the softmax layer of the RNN-T decoder 220.

[0038] Generally, the two-pass model in Figure 2, without any additional constraints, has difficulty detecting word timings 206. This difficulty exists, at least in part, because the two-pass model tokenizes or splits a word into one or more word pieces. Here, for example, when a single word piece corresponds to an entire word, the start and end times for the entire word coincide with the start and end times for the single word piece. However, when a word consists of multiple word pieces, the start time for a word may correspond to one word piece, but the end time for a word may correspond to a different word piece. Unfortunately, thus, conventional two-pass models may attempt to identify when a word starts and when a word ends based on the word pieces. To overcome these problems, the two-pass model can be trained with specific constraints on the alignment of word pieces with respect to the start and end times for a given word in the training examples.

[0039] A conventional training process for the two-pass model in Figure 2 can be carried out in two stages. During the first stage, the encoder 210 and the RNN-T decoder 220

[0040]

number

[0041] It is trained to maximize. In the second stage, the encoder 210 is fixed and the LAS decoder 230 is,

[0042]

number

[0043] It is trained to maximize. When the two-pass model includes an additional encoder 250, the additional encoder 250 is trained in the second stage.

[0044]

number

[0045] The encoder 210 is fixed, but the model is trained to maximize the coefficient of

[0046] In some examples, such as Figures 3A and 3B, the training process 300 trains the two-pass model architecture 200 in several training examples 302, each including audio data representing oral utterances and corresponding ground truth transcriptions of the oral utterances. For each word of the corresponding oral utterance, the corresponding training example 302 also includes a ground truth start time 312 for the word, a ground truth end time 314 for the word, and constraints 304 indicating where each word piece in the word should occur as emitted from the LAS decoder 230. The training process 300 may be run on system 500 (Figure 5) to train the speech recognition system 200. The speech recognition system 200 to be trained may be deployed to run on the user device 110 shown in Figures 1A-1C. In some cases, the speech recognition system 200 to be trained may run on system 500 or another system communicating with the user device 110. The training process 300 instructs the two-pass model to generate (or insert) placeholder symbols before each word 310 to indicate the start of each word 310 in the utterance, and / or placeholder symbols after the last word 310 of the oral utterance 12, using the constraints 304 of the training example 302. In some configurations, the placeholder symbols are word boundaries before each word 310. <wb>Word piece 320 (shown, for example, as word pieces 320, 320a, 320d, 320g), and / or utterance boundary word piece 320 after the last word 310 of utterance 12. Training example 302 with placeholder symbols corresponds to word 310, but the two-pass model learns to include the placeholder symbols as word piece 320 during its generation of the transcription 204. Boundary word piece 320 (e.g., word boundary) during inference (i.e., using the two-pass model) <wb>For a two-pass model trained to generate (and / or speech boundaries), the boundary word piece 320 allows the speech recognition system 200 to have further detail to determine word timing 206.

[0047] To erode word timing 206 from a two-pass model using word pieces 320, the two-pass model is configured to focus on specific word pieces corresponding to the beginning or end of each word 310. More specifically, the training process 300 desires to constrain the first word piece 320 corresponding to the beginning of each word 310 to occur as close as possible to the start 312 of the alignment for each word 310, and the last word piece 320 corresponding to the end of each word 310 to occur as close as possible to the end 314 of the alignment for each word 310. Here, constraint 304 constrains all other word pieces 320 that make up word 310 to occur anywhere within the range of the ground truth start time 312 and ground truth end time 314 of word 310.

[0048] Referring to Figure 3B, during the training process 300, the LAS decoder 230 is trained using a training example 302, which includes a training example constraint 304. As described above, the training example constraint 304 is configured to constrain the first word piece 320 corresponding to the beginning of each word 310 to occur as close as possible to the beginning of the alignment for each word 310, and the last word piece 320 corresponding to the end of each word 310 to occur as close as possible to the end of the alignment for each word 310. For illustrative purposes, Figure 3B shows a simple training example 302, "the cat sat," with three words 310, 310a-c. Here, each word 310 in training example 302 has a known ground truth alignment with a ground truth alignment start time 312 and a ground truth alignment end time 314. In Figure 3B, the first word 310a "the" has a first Grand Truth start time 312, 312a and a first Grand Truth end time 314, 314a. The second word 310b "cat" has a second Grand Truth start time 312, 312b and a second Grand Truth end time 314, 314b. The third word 310c "sat" has a third Grand Truth start time 312, 312c and a third Grand Truth end time 314, 314c.

[0049] Based on the ground truth alignments 312 and 314 for each word 310, training example 302 includes training example constraints 304 that constrain each word piece 320 corresponding to word 310 to be consistent with the ground truth alignments 312 and 314. Here, the first word 310a "the" consists of three word pieces 320, 320a-c, i.e., boundary word piece 320 (for example, <wb>The first word piece 320a is represented as ), the second word piece 320b "_th", and the third word piece 320c "e". The second word 320b "cat" consists of three word pieces 320, 320d~f, i.e., boundary word piece 320 (for example, <wb>The third word 320c "sat" is made up of three word pieces 320, 320g~i, i.e., boundary word piece 320 (for example, <wb>It includes a seventh word piece 320g (shown as ), an eighth word piece 320h "sat", and a ninth word piece 320i which is utterance boundary 320 (shown as, for example).

[0050] The training process 300 is configured to determine which word piece 320 of each word 310 corresponds to the beginning of each word 310 (i.e., the starting word piece), and which word piece 320 of each word 310 corresponds to the end of each word 310 (i.e., the ending word piece). For example, in the example in Figure 3B, the training process 300 determines that the first word piece 320a is the starting word piece for the first word 310a, the fourth word piece 320d is the starting word piece for the second word 310b, and the seventh word piece 320g is the starting word piece for the third word 310c. Similarly, the training process 300 determines that the third word piece 320c is the ending word piece for the first word 310a, the sixth word piece 320f is the ending word piece for the second word 310b, and the ninth word piece 320i is the ending word piece for the third word 310c. In some examples, the starting word piece 320 and the ending word piece 320 are the same word piece 320 because a particular word 310 contains only one word piece 320.

[0051] Once the training process 300 determines the start and end word pieces for each word 310 in the training example 302, the training process 300 is configured to generate constrained alignments 330 for each of the start and end word pieces 320. In other words, the training process 300 generates alignment constraints aimed at establishing when a particular word piece 320 should occur in a time index, based on the timing of the ground truth alignments 312, 314. In some implementations, the constrained alignment 330 for a word piece 320 spans a time interval from the word piece start time 322 to the word piece end time 324. When a word piece 320 is the start word piece 320 for word 310, the start word piece 320 has a constrained alignment 330 that is aligned with the ground truth alignment start time 312. For example, the constrained alignment 330 for the starting word piece 320 spans a time interval centered on the ground truth alignment start time 312. On the other hand, when word piece 320 is the ending word piece 320 for word 310, the ending word piece 320 has a constrained alignment 330 that is aligned with the ground truth alignment end time 314. For example, the constrained alignment 330 for the ending word piece 320 spans a time interval centered on the ground truth alignment end time 314. When word piece 320 does not correspond to either the starting word piece 320 or the ending word piece 320, the word piece 320 may have a constrained alignment 330 that corresponds to a time interval from the ground truth alignment start time 312 to the ground truth alignment end time 314. In other words, the training example constraint 304 indicates that a word piece 320 that does not correspond to either the starting word piece 320 or the ending word piece 320 may occur at any point in time between when the ground truth occurred for the word 310 that corresponds to word piece 320.

[0052] In some configurations, the training process 300 includes an adjustable constrained alignment 330. In other words, the word piece start time 322 and / or word piece end time 324 can be adjusted to define different time intervals for the ground truth alignments 312, 314. Here, the time interval may be called a timing buffer, and the timing buffer will include a first time period before the ground truth alignments 312, 314 and a second time period after the ground truth alignments 312, 314. In other words, the first time period of the timing buffer is equal to the length of time between the word piece start time 322 and the ground truth alignments 312, 314, and the second time period of the timing buffer is equal to the length of time between the word piece end time 324 and the ground truth alignments 312, 314. By adjusting the timing buffer, the training example constraint 304 can optimize the WER for a two-pass model while attempting to minimize latency. For example, experiments using timing buffers showed that a timing buffer of approximately 180 milliseconds was more optimal in terms of WER and latency than timing buffers of 60 milliseconds or 300 milliseconds.

[0053] In some examples, the training process 300 applies a constrained alignment 330 (e.g., constrained alignments 330, 330a-i) to the attention mechanism associated with the LAS decoder 230. In other words, the training process 300 trains the LAS decoder 230 (e.g., the attention heads of the LAS decoder 230) using one or more training examples 302, which include training example constraints 304. In some implementations, the LAS decoder 230 includes multiple attention heads, but the training process 300 constrains one or fewer of the attention heads of the LAS decoder 230, allowing one or more attention heads to operate unconstrained. Here, during the training process 300, the constrained attention heads generate attention probabilities for each training example 302. When the attention probability generated by the attention head corresponds to a constrained alignment 330, the training process 300 is configured to compare the attention probability with the expected attention probability for training example 302 in the constrained alignment 330. In some configurations, the training example constraint 304 indicates the expected attention probability for each word piece 320 in the constrained alignment 330. For example, the expected probability for the constrained alignment 330 between the word piece start time 322 and the word piece end time 324 is set to a high or non-zero value (e.g., a value of 1) to indicate that the alignment of word piece 320 occurs at an acceptable time (i.e., within the constrained alignment 330). In some examples, training example 302 includes an expected attention probability set to a low or zero value to indicate that the alignment of word piece 320 occurs at an unacceptable alignment time (e.g., outside the constrained alignment 330). During the training process 300, if the attention probability fails to match or satisfy the expected attention probability, the training process 300 is configured to apply a training penalty to the constrained attention head.In some examples, the training process 300 applies a training penalty so that the training penalty minimizes the attention loss for the LAS decoder 230 during training. In some examples, the attention loss is represented by the following formula,

[0054] [Number]

[0055] where β is a hyperparameter that controls the weight of the attention loss L attention ; u corresponds to a word piece unit such that u ∈ U; t corresponds to time such that t ∈ T; c(u, t) corresponds to the training example constraint 304 for each word piece unit u over time t; and a(u, t) corresponds to the attention of the constrained attention head for each word piece unit u over time t. Additionally or alternatively, the training process 300 may apply a training penalty to minimize the overall loss for the LAS decoder 230 during training, provided that the overall loss is represented by the following formula. L overall = L attention + L LAS (2)

[0056] By applying a training penalty, the training process 300 instructs the constrained attention head of the LAS decoder 230 to have the maximum attention probability for each word piece 320 at the corresponding time when the word piece 320 occurs in a timely manner, across multiple training examples 302. For example, if the training process 300 trains a two-pass model, during decoding, the LAS decoder 230 operates a beam search that emits a word piece unit u at each step in the beam search. Here, the speech recognition 200 determines the word piece timing for each word piece unit u by discovering the time index that produces the maximum constrained attention head probability for this particular word piece unit u. From the word piece timing, the actual word timing 206 can be derived. For example, the word piece timing for boundary word piece 320 corresponds to the start and end of word 310. Here, the starting word piece 320 for word 310 (e.g., word boundary) <wb>Each word piece 320 will have a timing corresponding to the start time of each word 310, and each ending word piece 320 of each word 310 (shown, for example, as the speech boundary word piece 320 in Figure 3B) will have a timing corresponding to the end time of each word 310. In other words, the speech recognition system 200 may determine that the actual word timing 206 (for example, the start time of word 310 and the end time of word 310) is equal to the word piece timing for the start word piece 320 and the ending word piece 320. Based on this determination, the speech recognition system 200 is configured to generate word timing 206 for the words output by the speech recognition system 200 (for example, as shown in Figures 1A to 1C).

[0057] Figure 4 is a flowchart of an exemplary arrangement of operations for method 400, which implements a speech recognition system 200 with constrained attention. In operation 402, method 400 receives a training example 302 for a two-pass neural network model LAS decoder 230. In operation 404, method 400 performs operations 404, 404a-d for each word 310 of the training example 302. In operation 404a, method 400 inserts a placeholder symbol before each word 310. In operation 404b, method 400 identifies the respective ground truth alignments 312, 314 for the beginning and end of each word 310. In operation 404c, method 400 determines the beginning word piece 320 and the ending word piece 320 of each word 310. In operation 404d, method 400 generates a first constrained alignment 330 for the beginning word piece 320 of each word 310 and a second constrained alignment 330 for the ending word piece 320 of each word 310, where the first constrained alignment 330 is aligned with the ground truth alignments 312, 314 for the beginning of each word 310 (e.g., ground truth alignment start time 312), and the second constrained alignment 330 is aligned with the ground truth alignments 312, 314 for the end of each word 310 (e.g., ground truth alignment end time 314). In operation 406, method 400 constrains the attention head of the LAS decoder 230 of the two-pass neural network model by applying training example 302, which includes all of the first and second constrained alignments 330 for each word 310 of training example 302.

[0058] Figure 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems (e.g., speech recognition device 200) and methods (e.g., training process 300 and / or method 400) described herein. The computing device 500 represents various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the implementations of the invention described and / or claimed herein.

[0059] The computing device 500 includes a processor 510 (e.g., data processing hardware), memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 connected to memory 520 and a high-speed expansion port 550, and a low-speed bus 570 and a low-speed interface / controller 560 connected to storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and may be mounted on a common motherboard or in other ways as appropriate. The processor 510 can process instructions for execution within the computing device 500, including instructions stored in memory 520 or on storage device 530 for displaying graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 coupled to the high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, with multiple memories and multiple types of memory. Furthermore, multiple computing devices 500 may be connected, each providing a necessary part of the operation (for example, as a server bank, a group of blade servers, or a multiprocessor system).

[0060] Memory 520 stores information non-temporarily within the computing device 500. Memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-volatile memory 520 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) temporarily or permanently for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and disks or tapes.

[0061] The storage device 530 can provide a large-capacity storage device for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 may be an array of devices, including floppy disk devices, hard disk devices, optical disk devices, or tape devices, flash memory or other similar solid memory devices, or devices in a storage area network or other configuration. In additional implementations, the computer program product is tangibly embodied in the information carrier. When executed, the computer program product includes instructions that perform one or more actions, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 520, the storage device 530, or memory on the processor 510.

[0062] The high-speed controller 540 manages bandwidth-intensive operations for the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. Such allocation of functions is illustrative only. In some implementations, the high-speed controller 540 is coupled to memory 520, the display 580 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 550, which may accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet) and may be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, or to a networking device such as a switch or router, for example, through a network adapter.

[0063] The computing device 500 can be implemented in several different forms, as shown in the figure. For example, the computing device 500 can be implemented as a standard server 500a or multiple times in a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0064] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general-purpose, and is coupled to receive and transmit data and instructions from a storage system, at least one input device, and at least one output device.

[0065] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for programmable processors and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-temporary computer-readable medium, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0066] The processes and logic flows described herein may be executed by one or more programmable processors that execute one or more computer programs to perform a function by acting on input data and producing an output. Processes and logic flows may also be executed by dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Suitable processors for executing computer programs include, for example, both general-purpose and dedicated microprocessors, and any one or more processors in any type of digital computer. Generally, a processor will receive instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or will be operablely coupled to them to receive data from or transfer data to or from them. However, a computer is not required to have such devices. Computer-readable media suitable for storing computer program instructions and data include, for example, all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be complemented by or incorporated into dedicated logic circuits.

[0067] To provide user interaction, one or more aspects of the present disclosure may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to provide user interaction, for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic input, voice input, or haptic input. In addition, the computer may interact with the user by sending documents to and receiving documents from devices used by the user, for example, by sending web pages to a web browser in response to a request received from a web browser on the user's client device.

[0068] Several implementations have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other implementations are within the scope of the following claims. [Explanation of symbols]

[0069] 10 users 12. Oral speech, speech, streaming oral speech 100 Voice environment, voice-enabled environment, voice-enabled system 110 User devices, devices 112 Data processing hardware, hardware 114 Memory Hardware, Hardware 116 Audio capture devices, audio output devices, audio subsystems 116a Audio capture device, capture device 116b Audio output device 118 Programs or applications, transcription applications, communication applications, applications, voice assistant applications 200 Speech Recognition Systems, ASR Speech Recognition Systems, 2-Pass Model Architecture 202 Streaming audio data, audio data, audio input 204 Transcription 206 words timing 208 First path, second path, path network, path 208a First pass 208, first pass 208b Second path 208, second path 210 encoders Outputs 212, 222, and 232. 220 RNN-T Decoder 230 LAS decoder, 2-layer LAS decoder 250 acoustic encoders, additional encoders 252 Encoded Output 300 Training Processes 302 Training Examples 304 Constraints, Training Example Constraints 310 words 310a Word, first word 310b Word, second word 310c Word, third word 312 Ground Truth Start Time, Alignment Start Time, Ground Truth Alignment Start Time, Ground Truth Alignment 312a First Grand Truth Start Time 312 312b Second Grand Truth Start Time 312 312c Third Grand Truth Start Time 312 314 Grand Truth completion time, alignment completion time, Grand Truth alignment completion time, Grand Truth alignment 314a End time of the first Grand Truth 314 314b Second Grand Truth End Time 314 314c Third Grand Truth End Time 314 320 word boundaries <wb>Word piece, word piece, utterance boundary word piece, boundary word piece, first word piece, last word piece, utterance boundary, start word piece, end word piece 320a Word piece, first word piece 320b Word piece, second word piece 320c Word Piece, Third Word Piece 320d Word Piece, 4th Word Piece 320e Word Piece, 5th Word Piece 320f Word Piece, 6th Word Piece 320g Word Pieces, 7th Word Piece 320h Word Pieces, 8th Word Piece 320i Word Pieces, 9th Word Piece 322 word pieces start time 324 word pieces completion time 330 Constrained alignment, adjustable constrained alignment, first constrained alignment, second constrained alignment 330a~i Constrained Alignment 500 systems, computing devices 500a Standard Server, Server 500b Laptop Computer 500c Rack Server System 510 Processors, Components 520 Memory, Components, Non-volatile memory 530 Storage devices, components 540 High-Speed ​​Interfaces / Controllers, Components, High-Speed ​​Interfaces, High-Speed ​​Controllers 550 high-speed expansion ports, components 560 Low-speed interfaces / controllers, components, low-speed controllers 570 Slow Bus 580 displays 590 Low-Speed ​​Expansion Ports< / wb> < / wb> < / wb> < / wb> < / wb> < / wb> < / wb>

Claims

1. A computer implementation method that is executed on data processing hardware, wherein the data processing hardware includes Receiving a training example for a second path decoder of a two-pass neural network speech recognition model, wherein the training example comprises audio data representing oral utterances of multiple words and corresponding ground truth transcriptions of the oral utterances. For each word in the aforementioned oral utterance, Identifying the respective ground truth alignment for the beginning and end of each of the aforementioned words, Determine the starting word piece and the ending word piece for each of the aforementioned words, The alignment of the starting word piece of each word is constrained to a time interval centered on the respective ground truth alignment for the start of each word, The alignment of the ending word piece of each word is constrained to a time interval centered on the respective ground truth alignment for the ending of each word, The second path decoder is trained by applying the training example, which includes the constrained alignment of the start and end word pieces for each word, in order to teach the two-pass neural network speech recognition model how to determine the actual word timing for the transcribed words. A computer implementation method that causes an operation including the following to be performed.

2. The computer implementation method according to claim 1, wherein the start word piece and the end word piece comprise the same word piece for each of the words.

3. The computer implementation method according to claim 1, wherein the second path decoder comprises a plurality of attention heads.

4. The aforementioned operation, In the training example described above, while training the second path decoder, To identify the expected attention probability for the portion of the aforementioned training example, The constrained attention head of the second path decoder determines to generate an attention probability for at least one of the portions of the training example that fails to match the expected attention probability, and Applying a training penalty to the aforementioned restricted attention head The computer implementation method according to claim 1, further comprising:

5. The computer implementation method according to claim 4, wherein the attention probability for at least one of the portions of the training example occurs at a time corresponding to either the constrained alignment of the start word piece or the constrained alignment of the end word piece.

6. The computer implementation method according to claim 4, wherein the attention probability for at least one of the portions of the training example occurs at a time that does not correspond to the constrained alignment of the start word piece or the constrained alignment of the end word piece.

7. The aforementioned operation, In the training example described above, while training the second path decoder, The computer implementation method according to claim 1, further comprising minimizing attention loss for the constrained attention head of the second path decoder.

8. The aforementioned operation, In the training example described above, while training the second path decoder, To minimize the cross-entropy loss for the second path decoder. The computer implementation method according to claim 1, further comprising:

9. The aforementioned operation, During the execution of the aforementioned two-pass neural network speech recognition model, Receiving audio data of speech, Determining the time corresponding to the maximum probability in the constrained attention head of the second path decoder, and To generate a word start time for the determined time corresponding to the maximum probability in the constrained attention head of the second path decoder. The computer implementation method according to claim 1, further comprising:

10. The aforementioned operation, During the execution of the aforementioned two-pass neural network speech recognition model, Receiving audio data of speech, Determining the time corresponding to the maximum probability in the constrained attention head of the second path decoder, and To generate a word end time for the determined time corresponding to the maximum probability in the constrained attention head of the second path decoder. The computer implementation method according to claim 1, further comprising:

11. It is a system, Data processing hardware and The memory hardware that is communicating with the aforementioned data processing hardware The memory hardware, when executed on the data processing hardware, stores instructions that cause the data processing hardware to perform an operation, and the operation is Receiving a training example for a second path decoder of a two-pass neural network speech recognition model, wherein the training example comprises audio data representing oral utterances of multiple words and corresponding ground truth transcriptions of the oral utterances. For each word in the aforementioned oral utterance, Identifying the respective ground truth alignment for the beginning and end of each of the aforementioned words, Determine the starting word piece and the ending word piece for each of the aforementioned words, The alignment of the starting word piece of each word is constrained to a time interval centered on the respective ground truth alignment for the start of each word, The alignment of the ending word piece of each word is constrained to a time interval centered on the respective ground truth alignment for the ending of each word, The second path decoder is trained by applying the training example, which includes the constrained alignment of the start and end word pieces for each word, in order to teach the two-pass neural network speech recognition model how to determine the actual word timing for the transcribed words. A system that includes this.

12. The system according to claim 11, wherein the starting word piece and the ending word piece each comprise the same word piece for each of the words.

13. The system according to claim 11, wherein the second path decoder comprises a plurality of attention heads.

14. The aforementioned operation, In the training example described above, while training the second path decoder, To identify the expected attention probability for the portion of the aforementioned training example, The constrained attention head of the second path decoder determines to generate an attention probability for at least one of the portions of the training example that fails to match the expected attention probability, and Applying a training penalty to the aforementioned restricted attention head The system according to claim 11, further comprising:

15. The system according to claim 14, wherein the attention probability for at least one of the portions of the training example occurs at a time corresponding to either the constrained alignment of the start word piece or the constrained alignment of the end word piece.

16. The system according to claim 14, wherein the attention probability for at least one of the portions of the training example occurs at a time that does not correspond to the constrained alignment of the starting word piece or the constrained alignment of the ending word piece.

17. The aforementioned operation, In the training example described above, while training the second path decoder, The system according to claim 11, further comprising minimizing attention loss for the constrained attention head of the second path decoder.

18. The aforementioned operation, In the training example described above, while training the second path decoder, To minimize the cross-entropy loss for the second path decoder. The system according to claim 11, further comprising:

19. The aforementioned operation, During the execution of the aforementioned two-pass neural network speech recognition model, Receiving audio data of speech, Determining the time corresponding to the maximum probability in the constrained attention head of the second path decoder, and To generate a word start time for the determined time corresponding to the maximum probability in the constrained attention head of the second path decoder. The system according to claim 11, further comprising:

20. The aforementioned operation, During the execution of the aforementioned two-pass neural network speech recognition model, Receiving audio data of speech, Determining the time corresponding to the maximum probability in the constrained attention head of the second path decoder, and To generate a word end time for the determined time corresponding to the maximum probability in the constrained attention head of the second path decoder. The system according to claim 11, further comprising:

Images