Unified end-to-end speech recognition and endpoint pointing using switched connections

Abstract

The single E2E multitask model (200) includes a speech recognition model (210) and an end-pointer model (220). The speech recognition model includes an audio encoder (240) configured to encode a sequence of audio frames (144) into a corresponding high-level feature representation (245), and a decoder (250) configured to generate a probability distribution (253) over possible speech recognition hypotheses for the sequence of audio frames based on the high-level feature representation. The end-pointer model is configured to operate between a VAD mode and an EOQ detection mode. During the VAD mode, the end-pointer model receives input audio frames and, for each input audio frame, determines whether the input audio frame contains speech. During the EOQ detection mode, the end-pointer model receives latent representations of the sequence of audio frames output from the audio encoder and, for each latent representation, determines whether the latent representation contains final silence.

Description

[Technical Field]

[0001] The present disclosure relates to unified end-to-end speech recognition and endpointing using switched connections. [Background technology]

[0002] Automatic speech recognition (ASR) systems are becoming an increasingly popular technology. Modern ASR systems focus on providing not only high quality (e.g., low word error rate) but also low latency (e.g., a short delay between when a user speaks and when a transcription or response appears). For example, users of devices that implement ASR systems often expect the ASR system to decode speech in a streaming format that responds in real time or even faster than real time. Summary of the Invention

[0003] One aspect of the present disclosure provides a single end-to-end multi-task model for speech recognition and end pointing. The multi-task model includes a speech recognition model and an end-pointer model. The speech recognition model includes an audio encoder configured to encode a sequence of audio frames into a corresponding first high-level feature representation, the audio encoder including multiple multi-head attention layers. The speech recognition model also includes a decoder configured to generate a probability distribution over possible speech recognition hypotheses for the sequence of audio frames based on the first high-level feature representation. The end-pointer model is configured to operate between a voice activity detection (VAD) mode and an end-of-query (EOQ) detection mode. During the VAD mode, the end-pointer model is configured to receive input audio frames and, for each input audio frame, determine whether the input audio frame contains speech. During the EOQ detection mode, the end-pointer model is configured to share an initial stack of multi-head attention layers from the multiple multi-head attention layers with the audio encoder, receive latent representations of the sequence of audio frames output from a final layer of the initial stack of multi-head attention layers, and determine, for each latent representation, whether the latent representation contains a final silence.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some examples, the speech recognition model and the end-pointer model are jointly trained using multi-task learning on a set of training speech utterances, each training speech utterance in the set of training speech utterances including audio data characterizing the training speech utterance paired with a corresponding transcription of the training speech utterance, and a sequence of reference end-pointing labels, each including one of a reference speech label, a reference initial silence label, a reference middle silence label, or a reference final silence label. Here, the speech recognition model and the end-pointer model may be jointly trained on a set of training speech utterances, and the training is performed by: determining a speech recognition loss based on speech recognition results predicted for audio data by the speech recognition model and a corresponding transcription of the training speech utterance; training the speech recognition model based on the speech recognition loss; determining an end-pointer loss based on a sequence of reference end-pointing labels and a corresponding sequence of predicted end-pointing labels output by the end-pointer model; and training the end-pointer model based on the end-pointer loss. In some implementations, for each training speech utterance, the switch connection of the multi-task model randomly selects an end-pointer model to receive as input either a latent representation output from a final layer of an initial stack of multi-head attention layers for audio data characterizing the training speech utterance or the audio data characterizing the training speech utterance.

[0005] In some implementations, the end-pointer model determines that the input audio frame contains speech during VAD mode, and the end-pointer model switches operation from VAD mode to EOQ detection mode. In some examples, the end-pointer model determines that the latent expression contains final silence during EOQ detection mode, and the end-pointer model switches operation from EOQ detection mode to VAD mode.

[0006] In some examples, the decoder includes a prediction network and a collaborative network. The prediction network is configured to receive as input the sequence of non-blank symbols output by the final softmax layer and generate as output a dense representation. The collaborative network is configured to receive as input the dense representation generated by the prediction network at each of the multiple output steps and the first high-level feature representation generated by the audio encoder at each of the multiple output steps and generate a probability distribution over possible speech recognition hypotheses at each of the multiple output steps. The prediction network may include an LSTM-based prediction network or a V2 embedded lookup table.

[0007] In some implementations, the multiple multi-head attention layers include a conformer layer or a transformer layer. In some examples, the speech recognition model also includes a non-causal encoder configured to receive as input a first high-order feature representation encoded by the audio encoder and generate as output a second high-order feature representation corresponding to the first high-order feature representation, and the decoder is configured to generate a probability distribution over possible speech recognition hypotheses for the sequence of audio frames based on the second high-order feature representation. In some implementations, the end-pointer model includes a stack of one or more LSTM layers, followed by a fully connected layer with a softmax function configured to predict a probability distribution over possible end-pointing labels for speech, initial silence, middle silence, and final silence.

[0008] Another aspect of the present disclosure provides a computer-implemented method executed by data processing hardware that causes the data processing hardware to perform operations, the operations including: receiving a sequence of audio frames characterizing speech; processing the sequence of audio frames by an audio encoder of a single end-to-end multi-task model to generate a corresponding first high-level feature representation, the audio encoder including multiple multi-head attention layers; and generating, by a decoder of the multi-task model, a probability distribution over possible speech recognition hypotheses for the sequence of audio frames based on the first high-level feature representation. The operations also include: determining, for each corresponding audio frame in the sequence of audio frames, whether the corresponding audio frame includes speech during a voice activity detection (VAD) mode, using an end pointer model of the multi-task model that shares an initial stack of multi-head attention layers with the audio encoder; and determining, for each corresponding latent representation of a plurality of latent representations for the sequence of audio frames output from a final layer of the initial stack of multi-head attention layers, whether the corresponding latent representation includes final silence during an end-of-query (EOQ) detection mode.

[0009] Implementations of the present disclosure may include one or more of the following optional features: In some examples, the operations further include using multi-task learning to jointly train the audio encoder, decoder, and end-pointer model on a set of training speech utterances, each training speech utterance in the set of training speech utterances including audio data characterizing the training speech utterance paired with a corresponding transcription of the training speech utterance, and a sequence of reference end-pointing labels, each including one of a reference speech label, a reference initial silence label, a reference middle silence label, or a reference final silence label. Jointly training the audio encoder, decoder, and end-pointer model on the set of training speech utterances may include: determining a speech recognition loss based on a transcription of the training speech utterance and corresponding speech recognition results predicted for the audio data by the audio encoder and decoder; training at least one of the audio encoder or decoder based on the speech recognition loss; determining an end-pointer loss based on a sequence of reference end-pointing labels and the corresponding sequence of predicted end-pointing labels output by the end-pointer model; and training the end-pointer model based on the end-pointer loss. In some examples, the operations further include, for each training speech utterance, randomly selecting an end-pointer model using a switch connection of the multi-task model to receive as input one of a latent representation output from a final layer of an initial stack of multi-head attention layers for the audio data characterizing the training speech utterance or the audio data characterizing the training speech utterance.

[0010] In some implementations, the operations further include switching operation of the end pointer model from the VAD mode to the EOQ detection mode based on determining that the corresponding audio frame includes speech during the VAD mode. In some examples, the operations further include switching operation of the end pointer model from the EOQ detection mode to the VAD mode based on determining that the corresponding latent expression includes final silence during the EOQ detection mode.

[0011] In some examples, the decoder includes a prediction network and a collaborative network, and the operations further include generating, at each of a plurality of output steps, a corresponding dense representation by the prediction network based on the sequence of non-blank symbols output by the final softmax layer, and generating, by the collaborative network, a corresponding probability distribution over possible speech recognition hypotheses based on the corresponding dense representation generated by the prediction network at the corresponding output step. In some implementations, the prediction network includes an LSTM-based prediction network or a V2 embedded lookup table.

[0012] In some implementations, the multiple multi-head attention layers include conformer layers or transformer layers. In some examples, the operations also include using a non-causal encoder of the multi-task model to generate second high-order feature representations corresponding to each of the first high-order feature representations generated by the audio encoder, and generating probability distributions over possible speech recognition hypotheses for the sequence of audio frames based on the second high-order feature representations. In some implementations, the end-pointer model includes a stack of LSTM layers followed by a fully connected layer with a softmax function configured to predict probability distributions over possible end-pointing labels for speech, initial silence, middle silence, and final silence.

[0013] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram of an exemplary audio environment using an end-to-end (E2E) multitasking model for audio transcription and end-pointing. [Figure 2] FIG. 1 is a schematic diagram of an exemplary E2E multitasking model architecture with a speech recognition model, an end pointer model, and a switch connection. [Figure 3] 1 is a flowchart of an exemplary sequence of operations for a computer-implemented method for unified E2E speech recognition and end-pointing using a switch connection. [Figure 4] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0015] Like reference symbols in the various drawings indicate like elements.

[0016] Automatic speech recognition (ASR) systems are becoming an increasingly popular technology. ASR systems focus on providing not only high quality (e.g., low word error rate) but also low latency (e.g., a short delay between when a user speaks and when a transcription is displayed). For example, users of devices that implement ASR systems often expect the ASR system to decode speech in a streaming format that responds in real time or even faster than real time. Traditional speech recognition models rely on separate, distinct, and separately trained endpoint models for endpointing. Endpointing involves voice activity detection (VAD) and end-of-query (EOQ) detection. VAD classifies each input audio frame according to whether it contains speech or silence. VAD classification can be used for "frame filtering," in which non-speech frames are discarded. This means that they are not input to or processed by the speech recognition model, thereby avoiding unnecessary computations by the speech recognition model, which is especially important for battery-powered user devices. EOQ detection classifies each input audio frame according to whether it predicts whether the ongoing speech has ended or whether it contains periods of intermediate silence. For continuous query tasks (e.g., voice dictation), high-quality EOQ detection is important to pause the speech recognition model during periods of intermediate silence. This is especially important for battery-powered user devices, because continuous query tasks can last for arbitrarily long periods of time. For short query tasks, such as digital assistants or interactive voice response applications, EOQ detection predicts when the user will speak so that the speech recognition model can complete or finish transcribing the query and generate a response in a timely manner. For short query tasks, high-quality EOQ detection is important to reduce speech recognition latency, because a response to a query typically is not generated until the speech recognition model has finished transcribing. For speech recognition systems, user-perceived latency (UPL) is a critical factor in user satisfaction.Therefore, there is a need for improved VAD and EOQ detection.

[0017] Embodiments herein are directed to an end-to-end (E2E) multi-task model and method for performing speech recognition, end pointing, VAD, and EOQ detection. The E2E multi-task model integrates a speech recognition model with an end pointer model into a single model trained to perform multiple tasks. Here, the speech recognition model may be an E2E speech recognition model that integrates an acoustic model and a language model. By integrating the end pointer model with the speech recognition model into a single multi-task model, the end pointer model may generate improved EOQ detection predictions by basing EOQ detection on latent representations generated by the audio encoder of the speech recognition model rather than on raw audio frames. Here, the end pointer model shares one or more layers with the audio encoder of the speech recognition model, for example, using hard parameter sharing. In particular, the speech recognition model and the end pointer model may be jointly trained. By integrating and jointly training the speech recognition model and the end pointer model, the performance of VAD and EOQ detection may be improved. This is because joint training forces the speech recognition model and the end pointer model to learn representations that generalize well across related tasks. Furthermore, by integrating the speech recognition model and the end-pointer model into a single unified multi-task model, the infrastructure burden of building and deploying a speech recognition system is reduced, as only a single unified model is required for training, deployment, and maintenance.

[0018] In some implementations, even a single layer of the audio encoder of a speech recognition model can be significantly more complex than the end-pointer model. Therefore, to reduce the complexity of VAD before speech recognition begins, in implementations of the E2E multitasking model disclosed herein, the end-pointer model includes a switch connection that enables it to operate in two modes: VAD mode and EOQ detection mode. In VAD mode, the switch connection provides input audio frames to the end-pointer model, which performs VAD based on the audio frames. Therefore, in VAD mode, which occurs before speech recognition begins, the shared layer of the audio encoder does not need to be activated. In VAD mode, when the end-pointer model detects speech, audio frames are provided to the speech recognition model, the speech recognition model (including the audio encoder) is activated, and the end-pointer model switches operation from VAD mode to EOQ detection mode. In EOQ detection mode, the switch connection provides the latent representation output from the final layer of the shared layer to the end-pointer model, which performs EOQ detection based on the latent representation. Therefore, in the EOQ detection mode, the end pointer model may utilize or leverage latent representations already generated by the audio encoder for speech recognition purposes. Because the EOQ detection mode is active only during speech recognition, during which the audio encoder is active for speech recognition purposes, EOQ detection performance may be improved by basing it on the latent representations generated by the audio encoder without increasing computational complexity. In the EOQ detection mode, when the end pointer model detects a final silence that ends an utterance, the switch connection feeds the audio frame to the speech recognition model, the speech recognition model (including the audio encoder) becomes active, and the end pointer model switches operation from the EOQ detection mode to the VAD mode.

[0019] The disclosed E2E multi-task model has been shown to reduce the average EOQ detection latency by over 30 percent and the 90th percentile EOQ detection latency by over 20 percent for short query tasks without regressing the word error rate (WER). Furthermore, for continuous query tasks, the disclosed E2E multi-task model has been shown to improve the WER by integrating the speech recognition task and the endpoint pointing task.

[0020] 1 is a schematic diagram of an example audio environment 100 and system 101. In audio environment 100, a user's interaction with a computing device, such as a user device 110, may be through voice input. User device 110 (also commonly referred to as device 110) is configured to capture sound (e.g., streaming audio data) from one or more users 102 within audio environment 100. Here, streaming audio data may refer to voice utterances 104 by users 102 that serve as audible queries, commands to user device 110, or audible communications captured by user device 110 (e.g., dictation for transcription). A voice-enabled system of user device 110 may process the queries or commands by responding to the queries and / or having the commands performed / accomplished by one or more downstream applications.

[0021] The system 101 includes a client device 110, a remote computing system 120, and a network 130. The user device 110 may be any computing device capable of communicating with the remote computing system 120 via the network 130. The user device 110 may correspond to any computing device associated with the user 102 and capable of receiving audio data. Some examples of the user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, Internet of Things (IoT) devices, digital assistant devices, vehicle infotainment systems, smart displays, smart speakers, etc. The user device 110 includes data processing hardware 111 and memory hardware 112 in communication with the data processing hardware 111, which stores instructions that, when executed by the data processing hardware 111, cause the data processing hardware 111 to perform one or more operations. User device 110 further includes an audio system 142 with one or more audio capture devices 113, 113a-n (e.g., microphones) for capturing and converting audio utterances 104 into electrical signals within audio environment 100, and one or more audio output devices 114, 114a-n (e.g., speakers) for communicating audible audio signals (e.g., as output audio data from user device 110). In the example shown, user device 110 implements a single audio capture device 113, but user device 110 may implement an array of audio capture devices 113 without departing from the scope of this disclosure, whereby one or more audio capture devices 113 of the array may not be physically present at user device 110 but may be in communication with user device 110.Similarly, while in the example shown, user device 110 implements a single audio output device 114, user device 110 may implement an array of audio output devices 114 without departing from the scope of this disclosure, whereby one or more audio output devices 114 of the array may be in communication with user device 110, even though they may not be physically present at user device 110.

[0022] The remote computing system 120 may be a distributed system (e.g., a cloud computing environment) with scalable, elastic resources. The resources include computing resources 121 (e.g., data processing hardware) and / or storage resources 122 (e.g., memory hardware). Additionally or alternatively, the remote computing system 120 may be a centralized system. The network 130 may be wired, wireless, or a combination thereof, and may include private and / or public networks, such as the Internet.

[0023] In the exemplary system 101, an automatic speech recognition (ASR) system 140 implementing a single E2E multitasking model 200 resides on a user device 110 of a user 102 and / or on a remote computing system 120 that communicates with the user device 110 via a network 130. The user device 110 and / or the remote computing device 120 also include an audio subsystem 142 configured to receive utterances 104 spoken by the user 102 and captured by audio capture device(s) 113 and convert the utterances 104 into a corresponding digital format associated with input audio frames 144 that can be processed by the ASR system 140. In the illustrated example, the user 102 speaks each utterance 104, and the audio subsystem 142 converts the utterance 104 into a corresponding audio frame 144 (e.g., audio data) for input to the ASR system 140. The E2E multi-task model 200 then receives, as input, audio frames 144 corresponding to the utterance 104 and generates / predicts, as output, a corresponding transcription 146 (e.g., a recognition result / hypothesis) of the utterance 104. In the illustrated example, the E2E multi-task model 200 may perform streaming speech recognition to generate initial speech recognition results 146, 146a, and a rescorer (not shown for clarity of illustration) may update (i.e., rescore) the initial speech recognition results 146a to generate final speech recognition results 146, 146b.

[0024] The user device 110 and / or the remote computing device 120 also execute a user interface generator 148 configured to present a representation of the transcription 146 of the utterance 104 to the user 102 of the user device 110. As described in more detail below, the user interface generator 148 may display initial speech recognition results 146a in a streaming format during Time 1, and then display final speech recognition results 146b during Time 2. In some configurations, the transcription 146 output from the ASR system 140 is processed by a natural language processing / understanding (NLP / NLU) module executing, for example, on the user device 110 or the remote computing device 120, to execute the user command / query specified by the utterance 104. NLP / NLU generally refers to the process of interpreting written language (e.g., the speech recognition results 146) and determining whether the written language prompts some action. Additionally or alternatively, a text-to-speech system (not shown) (e.g., running on any combination of user device 110 or remote computing device 120) may convert transcription 146 into synthesized speech for audible output by user device 110 and / or other devices.

[0025] In the illustrated example, user 102 may interact with a program or application 115 (e.g., digital assistant application 115) of user device 110 that uses ASR system 140. For example, FIG. 1 shows user 102 communicating with digital assistant application 115 and digital assistant application 115 displaying digital assistant interface 116 on screen 117 of user device 110 to depict a conversation between user 102 and digital assistant application 115. In this example, user 102 asks digital assistant application 115, "What time is the concert tonight?" This question from user 102 is a speech utterance 104, which is captured by audio capture device 113 and processed by audio system 142 of user device 110. In this example, audio system 142 receives speech utterance 104 and converts it into audio frames 144 for input to ASR system 140. Continuing with the example, a single E2E multitasking model 200 that integrates the speech recognition model 210 and end pointer model 220 (see FIG. 2 ) of the ASR system 140 receives audio frames 144 corresponding to utterances 104 as the user 102 speaks, encodes the audio frames 144, and then decodes the encoded audio frames 144 into speech recognition results 146.

[0026] In the example shown in FIG. 1, digital assistant application 115 can use NLP / NLU to respond to a question posed by user 102. In this example, digital assistant application 115 uses NLP / NLU to recognize that the question from user 102 is about the user's schedule, more specifically, a concert on the user's schedule. By recognizing these details with NLP / NLU, digital assistant application 115 returns a response 118 to user utterance 104, which states, "The venue doors open at 7:30 PM, and the concert starts at 9 PM." In some configurations, NLP / NLU occurs at a remote computing system 120 that communicates with user device 110.

[0027] 2 is a schematic diagram of an exemplary E2E multi-task model 200 for performing speech recognition, end pointing, VAD, and EOQ detection. As shown, E2E multi-task model 200 includes a speech recognition model 210, an end pointer model 220, and a switch connection 222, integrated into a single multi-task model. Notably, speech recognition model 210, end pointer model 220, and switch connection 222 of E2E multi-task model 200 may be jointly trained, deployed, and maintained. As described herein, user device 110 executes E2E multi-task model 200. However, it is understood that remote computing system 120 may also perform one or more portions, or all, of E2E multi-task model 200 in addition to or instead of user device 110.

[0028] In the illustrated example, the speech recognition model 210 includes a streaming cascaded conformal transducer (Conf-T) architecture that includes an audio encoder 240 and a decoder 250. Here, the audio encoder 240 includes a cascading causal encoder architecture having a first encoder 242 and a second encoder 244. The cascading audio encoder 240 refers to a model structure in which the encoding path includes two encoders 242, 244 that are cascaded such that the output of the first encoder 242 feeds the input of the second encoder 244 before decoding.

[0029] The first encoder 242 encodes a sequence of d-dimensional feature vectors (e.g., audio frame 144 (FIG. 1)) x=(x1, x2, ..., x T ) where x t ∈R d and encodes a sequence of audio frames 144 into a corresponding latent representation 243 as the output of the final layer of the first encoder 242. A second encoder 244 is cascaded to the first encoder 242 and is trained to receive the latent representation 243 as input and encode the latent representation 243 into a corresponding first high-level feature representation 245 as the output of the final layer of the second encoder 244. This high-level feature representation 245 is

number

[0030] In some examples, the speech recognition model 210 also includes a non-causal encoder 260 configured to receive the first high-order feature representation 245 as input and to generate a second high-order feature representation 262 corresponding to the first high-order feature representation 245 as output.

[0031] In some implementations, the cascaded audio encoder 240 includes a stack of multiple (e.g., seven) multi-head (e.g., eight-head) attention layers 247, 247a-n (e.g., conformer or transformer layers), where (i) the first encoder 242 includes an initial stack of layers 247a-b (e.g., two) from the stack of layers 247 with an attention dimension of 512, and (ii) the second encoder 244 includes a time-reduction stack layer that downsamples its input by a factor of two, followed by another multi-head attention layer 247c from the stack of multiple multi-head attention layers 247, a projection layer, and the remaining multi-head attention layers 247d-n from the stack of multiple multi-head attention layers 247. Here, a causal convolutional layer and a left contextual attention layer may be used at each layer to strictly constrain the audio encoder 240 from using future inputs. The first encoder 242 may be referred to as a causal encoder, and the second encoder 244 may be referred to as a non-causal encoder.

[0032] The end pointer model 220 is configured to operate between a VAD mode and an EOQ detection mode. While the end pointer model 220 is operating in the VAD mode, the switch connection 222 provides input audio frames 144 to the end pointer model 220, and the end pointer model 220 performs VAD based on the audio frames 144. When the end pointer model 220 is operating in the VAD mode (which occurs before speech recognition begins), the speech recognition model 210 (including the shared first encoder 242) is not activated or does not need to be activated (i.e., the audio frames 144 do not need to be sent to or processed by the speech recognition model 210). This is because the end pointer model 220 performs VAD based on the audio frames 144. In the VAD mode, the end pointer model 220 outputs an endpoint label 224 for each audio frame 144, indicating whether the audio frame 144 contains speech. During VAD mode, the end-pointer model 220 selects each endpoint label 224 to be either initial silence (i.e., silence before speech begins) or speech. Here, the end-pointer model E2E multitasking model 200 may determine whether an audio frame 144 contains speech by comparing the predicted probability that speech is present with a predetermined probability threshold.

[0033] When end pointer model 220 determines that one or more audio frames 144 contain speech and outputs one or more endpoint labels 224 for the speech, E2E multitasking model 200 (i) activates speech recognition model 210 so that speech recognition model 210 begins performing speech recognition on the sequence of audio frames 144, (ii) configures switch connection 222 to provide end pointer model 220 with latent representations 243 of the sequence of audio frames 144 generated by the shared portion of audio encoder 240 (i.e., first encoder 242), and (iii) switches the operation of end pointer model 220 from VAD mode to EOQ detection mode. In EOQ detection mode, end pointer model 220 determines, for each latent representation 243, whether latent representation 243 includes a final silence representing the occurrence of an EOQ event or whether it includes an intermediate silence, and outputs a corresponding endpoint label 224 for the final silence or intermediate silence. Here, the end pointer model 220 selects each endpoint label 224 as speech, intermediate silence (e.g., silence during an utterance), or final silence (e.g., after an utterance ends). Here, the end pointer model E2E multitasking model 200 may determine whether an audio frame 144 contains speech by comparing the predicted probability of speech presence with a predetermined probability threshold. In particular, the predetermined probability threshold for the EOQ detection mode may be different from the predetermined probability threshold for the VAD mode.

[0034] While the end-pointer model 220 is operating in the EOQ detection mode, the switch connection 222 provides the latent representation 243 output from the final layer 247b (i.e., the first encoder 242) of the shared layers 247a-b to the end-pointer model 220, and the end-pointer model 220 performs EOQ detection based on the latent representation 243. Thus, in the EOQ detection mode, the end-pointer model 220 can improve EOQ detection performance without increasing computational complexity by utilizing or leveraging the latent representation 243 already generated by the audio encoder 240 for speech recognition purposes. That is, because the EOQ detection mode is active only during speech recognition, during which the audio encoder 240 is active for speech recognition purposes, the EOQ detection performance can be improved by basing it on the latent representation 243 already generated by the audio encoder 240 without increasing computational complexity.

[0035] In the illustrated example, while operating in EOQ detection mode, the end-pointer model 220 shares one or more layers 247a-b with the audio encoder 240 of the speech recognition model 210. Here, the end-pointer model 220 shares a first encoder 242 with the audio encoder 240, where the first encoder 242 represents an initial stack of multi-head attention layers 247a-b (e.g., conformer layers or transformer layers) of a stack 246 of multiple multi-head attention layers 247a-n forming the audio encoder 240, and the latent representation 243 is output by the final layer 247b of the initial stack of layers 247a-b of the first encoder 242. In some implementations, the end-pointer model 220 and the audio encoder 240 share layers using hard parameter sharing. In particular, the speech recognition model 210 and the end-pointer model 220 may be jointly trained. Integrating and jointly training the speech recognition model 210 and the end-pointer model 220 improves the performance of VAD and EOQ detection because joint training forces the speech recognition model 210 and the end-pointer model 220 to learn representations that generalize well across related tasks. If the end-pointer model 220 determines that one or more latent representations 243 contain a final silence while operating in EOQ detection mode and outputs an endpoint label 224 for the final silence, the E2E multitasking model 200 (i) switches the operation of the end-pointer model 220 from the EOQ detection mode to the VAD mode, (ii) configures the switch connection 222 to provide the input audio frame 144 to the end-pointer model 220, and (iii) disables the speech recognition model 210.

[0036] In some implementations, when end-pointer model 220 determines that one or more latent representations 243 include intermediate silence and outputs endpoint label 224 of the intermediate silence while operating in EOQ detection mode, E2E multitasking model 200 (i) temporarily switches operation of end-pointer model 220 from EOQ detection mode to VAD mode, (ii) configures switch connection 222 to provide input audio frame 144 to end-pointer model 220, and (iii) temporarily disables speech recognition model 210. When speech continues (e.g., when end-pointer model 220 operating in VAD mode detects speech), E2E multitasking model 200 returns end-pointer model 220 to EOQ detection mode and resumes speech recognition by speech recognition model 210. In this way, speech recognition model 210 does not need to operate during intermediate silence. In some implementations, the end-pointer model 220 includes a stack of LSTM layers followed by a fully connected layer with a softmax function configured to predict probability distributions over possible end-pointing labels for speech, initial silence, middle silence, and final silence.

[0037] In the shown example, decoder 250 includes an RNN-T architecture having a collaboration network 252, a prediction network 254, and a softmax layer 256. Decoder 250 uses collaboration network 252 to combine first high-level feature representation 245 and / or second high-level feature representation 262 with a dense or hidden representation 255 output from prediction network 254 for a previous prediction output 257 by softmax layer 256 to generate a prediction output 257. In the shown example, decoder 250 includes softmax layer 256. Alternatively, softmax layer 256 may be implemented separately.

[0038] In the example shown, the prediction network 254 calculates the sequence of non-blank symbols 257 (i.e., predicted output) y 0 ,...,y ui-1 is expressed as a dense or hidden representation P uiIn some implementations, the dense representation P ui 255 contains a single embedding vector. In particular, the sequence of past non-blank symbols 257 received by the prediction network 254 captures the linguistic dependencies between the non-blank symbols 257 predicted during the previous time step, assisting the collaborative network 252 in predicting the probability of the next output symbol or blank symbol during the current time step. To contribute to a technique for reducing the size of the prediction network 254 without sacrificing the accuracy / performance of the decoder 250, the prediction network 254 uses a limited historical sequence of non-blank symbols 257, y ui-n ,…,y ui-1 , which is limited to the N previous non-blank symbols 257 output by the final softmax layer 256.

[0039] In the illustrated example, the joint network 252 combines a first high-level feature representation 245 generated by the audio encoder 240 and / or a second high-level feature representation 262 generated by the non-causal encoder 260 with the dense representation P generated by the prediction network 254. ui 255. The collaboration network 252 generates a probability distribution for the next output symbol.

number

[0040] In the example shown, the final softmax layer 256 calculates the probability distribution Z i 253 and selects the output label / symbol with the highest probability to generate the transcription 146. The final softmax layer 256 is i Any technique may be used to select the output label / symbol with the highest probability in 253. In this way, the decoder 250 does not make a conditional independence assumption, but rather, for each symbol y uThe 257 predictions are based not only on the acoustics but also on the 257 sequences of labels that have been output so far. ui-n ,…,y ui-1 The decoder 250 allows the speech recognition model 210 to be used in a streaming fashion by assuming that the output symbols 257 are independent of future acoustic frames 144.

[0041] In some implementations, the prediction network 254 includes a V2 embedding lookup table that includes an embedding prediction network. At each time step, the V2 embedding lookup table may receive as input the two previous predictions (e.g., one-hot vectors) output by the collaborative network 252, calculate respective embeddings d1, d2 for each of the two previous predictions, and provide a combined output [d1, d2] to the collaborative network 252. Alternatively, the prediction network 254 may include one or more conformer or transformer layers. Alternatively, the prediction network 254 may be a long short-term memory (LSTM)-based prediction network that includes one or more LSTM layers, each followed by a projection layer and an embedding layer. In some implementations, the collaborative network 252 includes one or more neural network layers, each with multiple hidden units, and the softmax layer 256 is composed of a unified word piece or graphem set generated using all unique word pieces or graphem sets in multiple training datasets.

[0042] In particular, the speech recognition model 210 and the end-pointer model 220 may be jointly trained on a set of training speech utterances using multi-task learning, where each training speech utterance in the set of training speech utterances includes audio data characterizing the training speech utterance paired with a corresponding transcription of the training speech utterance and a sequence of reference end-pointing labels, each of which includes one of a reference speech label, a reference initial silence label, a reference middle silence label, or a reference final silence label. In some implementations, the speech recognition model 210 calculates a speech recognition loss (SLA) based on the speech recognition results predicted for the audio data by the speech recognition model 210 and the corresponding transcription of the training speech utterance.

number

number

number

number

number

number

number

number

[0043] 3 is a flowchart of an exemplary sequence of operations for a computer-implemented method 300 for unified E2E speech recognition and end-pointing using a switch connection. The operations may be performed by data processing hardware 410 (FIG. 4) by executing instructions stored in memory hardware 420 in communication with data processing hardware 420. Data processing hardware 410 may include data processing hardware 111 (FIG. 1) of user device 110 and / or data processing hardware 121 (FIG. 1) of remote system 120. Memory hardware 420 may include memory hardware 112 (FIG. 1) of user device 110 and / or memory hardware 122 of remote system 120.

[0044] At operation 302, the method 300 includes receiving a sequence of audio frames 144 characterizing speech 104. At operation 304, the method 300 also includes processing the sequence of audio frames 144 by an audio encoder 240 of a single E2E multi-tasking model 200 to generate a corresponding first high-level feature representation 245, the audio encoder 240 including multiple multi-head attention layers, and at operation 305, generating, by a decoder 250 of the E2E multi-tasking model 200, a probability distribution 253 over possible speech recognition hypotheses for the sequence of audio frames 144 based on the first high-level feature representation 245.

[0045] At operation 308, the method 300 includes using the end pointer model 220 of the E2E multitasking model 200, which shares the initial stack 248 of multi-head attention layers 247a-b from the stack 246 of the multiple multi-head attention layers 247 with the audio encoder 240, to determine, for each corresponding audio frame 144 in the sequence of audio frames 144, whether the corresponding audio frame 144 includes speech during the VAD mode. At operation 310, the method 300 includes using the end pointer model 220 during the EOQ detection mode, to determine, for each corresponding latent representation 243 of the multiple latent representations 243, for the sequence of audio frames 144 output from the first layer 247b of the initial stack 248 of the multi-head attention layers 247a-b, whether the corresponding latent representation 243 includes final silence.

[0046] 4 is a schematic diagram of an example computing device 400 that may be used to implement the systems and methods described in this document. Computing device 400 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the scope of the invention(s) described and / or claimed in this document.

[0047] Computing device 400 includes a processor 410 (i.e., data processing hardware) that can be used to implement data processing hardware 111 and / or 121, a memory 420 (i.e., memory hardware) that can be used to implement memory hardware 112 and / or 122, a storage device 430 (i.e., memory hardware) that can be used to implement memory hardware 112 and / or 122, a high-speed interface / controller 440 that connects to memory 420 and a high-speed expansion port 450, and a low-speed interface / controller 460 that connects to a low-speed bus 470 and storage device 430. Each of components 410, 420, 430, 440, 450, and 460 are interconnected using various buses and may be mounted on a common motherboard or otherwise mounted as desired. Processor 410 processes instructions for execution within computing device 400, including instructions stored in memory 420 or on storage device 430, and can display graphical information in a graphical user interface (GUI) on an external input / output device, such as a display 480 coupled to high-speed interface 440. Alternate embodiments may use multiple processors and / or multiple buses, along with multiple memories and several types of memory, as needed. Also, multiple computing devices 400 may be connected, each providing a portion of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0048] The memory 420 stores information non-transiently within the computing device 400. The memory 420 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 420 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0049] Storage device 430 is capable of providing mass storage for computing device 400. In some implementations, storage device 430 is a computer-readable medium. In various different implementations, storage device 430 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory, or another similar solid-state memory device, or an array of devices including devices in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 420, storage device 430, or memory on processor 410.

[0050] High-speed controller 440 manages bandwidth-intensive operations of computing device 400, while low-speed controller 460 manages lower-bandwidth-intensive operations. Such role assignments are merely exemplary. In some implementations, high-speed controller 440 is coupled to memory 420, display 480 (e.g., via a graphics processor or accelerator), and high-speed expansion port 450, which may accept various expansion cards (not shown). In some implementations, low-speed controller 460 is coupled to storage device 430 and low-speed expansion port 490. Low-speed expansion port 490, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled, for example, via a network adapter, to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or to a networking device, such as a switch or router.

[0051] The computing device 400, as shown in the figure, may be implemented in a variety of forms, for example, it may be implemented as a standard server 400a, or multiple times in a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

[0052] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or readable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0053] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0054] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural programming language, an object-oriented programming language, and / or an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0055] The processes and logic flows described herein are performed by one or more programmable processors (also called data processing hardware) executing one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory, or both. Essential elements of a computer include a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or is operably coupled to a mass storage device to receive data from, transfer data to, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0056] To provide for user interaction, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Similarly, other types of devices can be used to provide for user interaction. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user. For example, a computer may interact by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0057] Unless expressly stated to the contrary, the phrase "at least one of A, B, or C" is intended to refer to any combination or subset of A, B, C, such as (1) at least one A alone, (2) at least one B alone, (3) at least one C alone, (4) at least one A with at least one B, (5) at least one A with at least one C, (6) at least one B with at least C, and (7) at least one A with at least one B and at least one C. Furthermore, unless expressly stated to the contrary, the phrase "at least one of A, B, and C" is intended to refer to any combination or subset of A, B, C, such as (1) at least one A alone, (2) at least one B alone, (3) at least one C alone, (4) at least one A with at least one B, (5) at least one A with at least one C, (6) at least one B with at least one C, and (7) at least one A with at least one B and at least one C. Furthermore, unless expressly stated to the contrary, "A or B" is intended to refer to any combination of A and B, such as (1) A alone, (2) B alone, and (3) A and B.

[0058] Although several embodiments have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, alternative embodiments are within the scope of the following claims.

Claims

1. A system comprising data processing hardware configured to run an automatic speech recognition (ASR) system to implement a single end-to-end multitasking model (200) for speech recognition and endpointing; the ASR system configured to receive a sequence of audio frames; The multitasking model (200) A speech recognition model (210) and an end pointer model (220), The speech recognition model (210) an audio encoder (240) configured to encode a sequence of audio frames (144) into a corresponding first high-level feature representation (245), the audio encoder (240) including a plurality of multi-head attention layers (247); a decoder (250) configured to generate, based on the first high-level feature representation (245), a probability distribution (253) over possible speech recognition hypotheses for the sequence of audio frames (144); the end-pointer model (220) is configured to operate between a voice activity detection (VAD) mode and an end-of-query (EOQ) detection mode; During the VAD mode, the end pointer model (220) is configured to receive input audio frames (144) and, for each input audio frame (144), determine whether the input audio frame (144) contains speech; During the EOQ detection mode, the end pointer model (220) shares with the audio encoder (240) an initial stack of multi-head attention layers (247) from the plurality of multi-head attention layers (247), and the end pointer model (220) is configured to receive latent representations (243) of the sequence of audio frames (144) output from a final layer (247) of the initial stack of multi-head attention layers (247), and to determine, for each latent representation (243), whether the latent representation (243) includes a final silence.

2. The speech recognition model (210) and the end-pointer model (220) are jointly trained on a set of training speech utterances using multi-task learning, and each training speech utterance in the set of training speech utterances is audio data characterizing the training speech utterances paired with corresponding transcriptions of the training speech utterances; a sequence of reference endpoint labels, each of the reference endpoint labels comprising one of a reference speech label, a reference initial silence label, a reference middle silence label, or a reference final silence label; and The system of claim 1 , comprising:

3. The speech recognition model (210) and the end pointer model (220) are jointly trained on the set of training speech utterances, the training comprising: determining a speech recognition loss based on speech recognition results predicted for the audio data by the speech recognition model (210) and the corresponding transcription of the training speech utterance; training the speech recognition model (210) based on the speech recognition loss; determining an end pointer loss based on the sequence of reference end pointing labels and the corresponding sequence of predicted end pointing labels output by the end pointer model (220); training the end-pointer model (220) based on the end-pointer loss; The system of claim 2 , wherein the system is performed by:

4. For each training speech utterance, the switch connection (222) of the multitasking model (200) randomly selects the end-pointer model (220) and takes as input: a latent representation (243) output from the final layer (247) of the initial stack of the multi-head attention layer (247) of the audio data characterizing the training speech utterances; or the audio data characterizing the training speech utterances; 3. The system of claim 2, wherein the system receives one of:

5. 2. The system of claim 1, wherein when the end pointer model determines that the input audio frame contains speech during the VAD mode, the end pointer model switches operation from the VAD mode to the EOQ detection mode.

6. 2. The system of claim 1, wherein when the end pointer model (220) determines that the latent expression (243) contains a final silence during the EOQ detection mode, the end pointer model (220) switches operation from the EOQ detection mode to the VAD mode.

7. The decoder (250) A prediction network (254), receiving as input the sequence of non-blank symbols (257) output by the final softmax layer (256); a prediction network (254) configured to generate as output a dense representation (255); A collaborative network (252), receiving as input the dense representation (255) produced by the prediction network (254) at each of a plurality of output steps and the first high-level feature representation (245) produced by the audio encoder (240) at each of the plurality of output steps; a collaborative network (252) configured to generate, at each of the plurality of output steps, a probability distribution (253) over possible speech recognition hypotheses; The system of claim 1 , comprising:

8. The prediction network (254) Long short-term memory (LSTM) based prediction networks, or V2 embedded lookup table, The system of claim 7, comprising:

9. The system of claim 1 , wherein the plurality of multi-head attention layers (247) comprises a conformer layer or a transformer layer.

10. The speech recognition model (210) further comprises a non-causal encoder (260), the non-causal encoder comprising: receiving as input the first high-level feature representation (245) encoded by the audio encoder (240); and generating as output a second higher-level feature representation (262) corresponding to the first higher-level feature representation (245); 2. The system of claim 1, wherein the decoder is configured to generate the probability distribution over possible speech recognition hypotheses for the sequence of audio frames based on the second higher-level feature representation.

11. 11. The system of claim 1, wherein the end-pointer model (220) includes a stack of one or more long short-term memory (LSTM) layers followed by a fully connected layer with a softmax function configured to predict probability distributions over possible end-pointing labels of speech, initial silence, middle silence, and final silence.

12. A computer-implemented method (300) executed on data processing hardware (410) for causing the data processing hardware (410) to perform an operation, the operation comprising: receiving a sequence of audio frames (144) characterizing an utterance; processing the sequence of audio frames (144) by an audio encoder (240) of a single end-to-end multi-task model (200) to generate a corresponding first high-level feature representation (245), the audio encoder (240) comprising a plurality of multi-head attention layers (247); generating, by a decoder (250) of the multitasking model (200), a probability distribution (253) over possible speech recognition hypotheses for the sequence of audio frames (144) based on the first high-level feature representation (245); using an end pointer model (220) of the multitasking model (200) that shares an initial stack of a multi-head attention layer (247) from the plurality of multi-head attention layers (247) with the audio encoder (240); during a voice activity detection (VAD) mode, for each corresponding audio frame (144) in the sequence of audio frames, determining whether the corresponding audio frame (144) contains speech; During an end-of-query (EOQ) detection mode, for the sequence of audio frames (144) output from a final layer (247) of the initial stack of the multi-head attention layer (247), for each corresponding latent representation (243) of a plurality of latent representations (243), determine whether the corresponding latent representation (243) includes a final silence; A computer-implemented method (300) comprising:

13. The audio encoder (240), the decoder (250), and the end pointer model (220) are jointly trained on a set of training speech utterances using multi-task learning, and each training speech utterance in the set of training speech utterances is audio data characterizing the training speech utterances paired with corresponding transcriptions of the training speech utterances; a sequence of reference endpoint labels, each of the reference endpoint labels comprising one of a reference speech label, a reference initial silence label, a reference middle silence label, or a reference final silence label; and 13. The computer-implemented method of claim 12, comprising:

14. The audio encoder (240), the decoder (250), and the end pointer model (220) are jointly trained on the set of training speech utterances, the training comprising: determining a speech recognition loss based on the transcription of the training speech utterance and corresponding speech recognition results predicted for the audio data by the audio encoder (240) and the decoder (250); training at least one of the audio encoder (240) or the decoder (250) based on the speech recognition loss; determining an end pointer loss based on the sequence of reference end pointing labels and the corresponding sequence of predicted end pointing labels output by the end pointer model (220); training the end-pointer model (220) based on the end-pointer loss; 14. The computer-implemented method of claim 13, comprising:

15. For each training speech utterance, the multitasking model switches connections to randomly select the end-pointer model (220) and take as input: a latent representation (243) output from the final layer (247) of the initial stack of the multi-head attention layer (247) of the audio data characterizing the training speech utterances; or the audio data characterizing the training speech utterances; 14. The computer-implemented method of claim 13, further comprising receiving one of:

16. 13. The computer-implemented method of claim 12, wherein the operations further include switching the end pointer model from the VAD mode to the EOQ detection mode based on determining that the corresponding audio frame contains speech during the VAD mode.

17. 13. The computer-implemented method of claim 12, wherein the operations further include switching the end pointer model (220) from the EOQ detection mode to the VAD mode based on determining that the corresponding latent expression (243) includes a final silence during the EOQ detection mode.

18. The decoder (250) includes a prediction network (254) and a collaboration network (252); The operation further includes, in each of the plurality of output steps: generating a corresponding dense representation (255) by said prediction network (254) based on the sequence of non-blank symbols (257) output by the final softmax layer (256); generating, by the collaborative network (252), corresponding probability distributions (253) over possible speech recognition hypotheses based on the corresponding dense representations (255) generated by the predictive network (254) in the corresponding output step; 13. The computer-implemented method of claim 12, comprising:

19. The prediction network (254) Long short-term memory (LSTM) based prediction networks, or V2 embedded lookup table, 20. The computer-implemented method of claim 18, comprising:

20. 13. The computer-implemented method of claim 12, wherein the plurality of multi-head attention layers (247) comprises a conformer layer or a transformer layer.

21. The operation may further include: generating second high-level feature representations (262) corresponding to each of the first high-level feature representations (245) generated by the audio encoder (240) using a non-causal encoder (260) of the multi-task model (200); generating the probability distribution (253) over possible speech recognition hypotheses for the sequence of audio frames (144) based on the second higher-level feature representation (262); 13. The computer-implemented method of claim 12, comprising:

22. 22. The computer-implemented method of any one of claims 12 to 21, wherein the end-pointer model (220) includes a stack of long short-term memory (LSTM) layers followed by a fully connected layer with a softmax function configured to predict probability distributions over possible end-pointing labels of speech, initial silence, middle silence, and final silence.

Images