Unified end-to-end speech recognition and endpoint using switching connections

Abstract

The single E2E multitasking model (200) includes a speech recognition model (210) and an endpoint machine model (220). The speech recognition model includes an audio encoder (240) configured to encode a sequence of audio frames (144) into a corresponding high-order feature representation (245); and a decoder (250) configured to generate a probability distribution (253) of possible speech recognition hypotheses for the sequence of audio frames based on the higher order feature representation. The endpoint model is configured to operate between a VAD mode and an EOQ detection mode. During the VAD mode, the endpoint model receives input audio frames and determines, for each input audio frame, whether the input audio frame includes speech. During the EOQ detection mode, the endpoint machine model receives potential representations for the sequence of audio frames output from the audio encoder, and determines, for each of the potential representations, whether the potential representation includes a final silence.

Description

Technical Field

[0001] The present disclosure relates to unified end-to-end speech recognition and endpointization using a handoff connection. Background Art

[0002] Automatic speech recognition (ASR) systems are an increasingly used technology. Modern ASR systems focus not only on providing high quality (e.g., low word error rate), but also on providing low latency (e.g., a short delay between a user speaking and a transcription or response appearing) speech recognition of spoken utterances. For example, when using a device implementing an ASR system, it is generally desirable for the ASR system to decode the utterances in a streaming manner corresponding to real time or even faster than real time. Summary of the invention

[0003] One aspect of the present disclosure provides a single end-to-end multi-task model for performing speech recognition and endpointization. The multi-task model includes a speech recognition model and an endpointizer model. The speech recognition model includes an audio encoder configured to encode an audio frame sequence into a corresponding first high-order feature representation, the audio encoder including multiple multi-head attention layers. The speech recognition model also includes a decoder configured to generate a probability distribution of possible speech recognition hypotheses for the audio frame sequence based on the first high-order feature representation. The endpointizer model is configured to operate between a voice activity detection (VAD) mode and an end-of-query (EOQ) detection mode. During the VAD mode, the endpointizer model is configured to receive input audio frames and determine whether the input audio frame includes speech for each input audio frame. During the EOQ detection mode, the endpointizer model shares an initial stack of multi-head attention layers from multiple multi-head attention layers with the audio encoder, and is configured to receive a potential representation for the audio frame sequence output from the final layer of the initial stack of multi-head attention layers and determine whether the potential representation includes final silence for each potential representation in the potential representation.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some examples, a speech recognition model and an endpointizer model are jointly trained on a set of training speech utterances using multi-task learning, wherein each training speech utterance in the set of training speech utterances includes audio data representing the training speech utterance paired with a corresponding transcription of the training speech utterance, and a reference endpointization label sequence, each reference endpointization label including one of a reference speech label, a reference initial silence label, a reference intermediate silence label, or a reference final silence label. Here, the speech recognition model and the endpointizer model may be jointly trained on the set of training speech utterances by: determining a speech recognition loss based on the speech recognition results predicted by the speech recognition model for the audio data and the corresponding transcription of the training speech utterance; training the speech recognition model based on the speech recognition loss; determining an endpointizer loss based on a reference endpointization label sequence and a corresponding predicted endpointization label sequence output by the endpointizer model; and training the endpointizer model based on the endpointizer loss. In some implementations, for each training speech utterance, a switching connection of the multi-task model randomly selects an endpointizer model to receive as input one of: a latent representation output from a final layer of an initial stack of multi-head attention layers for audio data representing the training speech utterance; or audio data representing the training speech utterance.

[0005] In some implementations, the endpointizer model determines during VAD mode that the input audio frame includes speech, and the endpointizer model switches operation from VAD mode to EOQ detection mode. In some examples, the endpointizer model determines during EOQ detection mode that the potential representation includes final silence, and the endpointizer model switches operation from EOQ detection mode to VAD mode.

[0006] In some examples, the decoder includes a prediction network and a joint network. The prediction is configured to receive a sequence of non-blank symbols output by a final Softmax layer as input and generate a dense representation as output. The joint network is configured to: receive a dense representation generated by the prediction network at each output step in a plurality of output steps and a first high-order feature representation generated by the audio encoder at each output step in the plurality of output steps as input; and generate a probability distribution of possible speech recognition hypotheses at each output step in the plurality of output steps. The prediction network may include an LSTM-based prediction network or a V2 embedding lookup table.

[0007] In some implementations, the multiple multi-head attention layers include a conformer layer or a transformer layer. In some examples, the speech recognition model also includes a non-causal encoder configured to receive a first high-order feature representation encoded by an audio encoder as input and generate a corresponding second high-order feature representation as output for the first high-order feature representation, and the decoder is configured to generate a probability distribution of possible speech recognition hypotheses for the audio frame sequence based on the second high-order feature representation. In some implementations, the endpointizer model includes a stack of one or more LSTM layers followed by a fully connected layer having a Softmax function configured to predict a probability distribution of possible endpointized labels for speech, initial silence, intermediate silence, and final silence.

[0008] Another aspect of the present disclosure provides a computer implementation executed on data processing hardware, the computer implementation causing the data processing hardware to perform operations, the operations including: receiving a sequence of audio frames representing speech; processing the sequence of audio frames by an audio encoder of a single end-to-end multi-task model to generate a corresponding first high-order feature representation, the audio encoder including multiple multi-headed attention layers; and generating a probability distribution of possible speech recognition hypotheses for the sequence of audio frames based on the first high-order feature representation by a decoder of the multi-task model. The operations also include: using an endpointizer model of the multi-task model, the endpointizer model shares an initial stack of multi-headed attention layers from multiple multi-headed attention layers with the audio encoder, during a voice activity detection (VAD) mode, determining for each corresponding audio frame in the sequence of audio frames whether the corresponding audio frame includes speech, and during an end-of-query (EOQ) detection mode, determining for each corresponding potential representation of multiple potential representations for the sequence of audio frames output from a final layer of the initial stack of multi-headed attention layers whether the corresponding potential representation includes final silence.

[0009] Implementations of the present disclosure may include one or more of the following optional features. In some examples, the operations also include using multi-task learning to jointly train an audio encoder, a decoder, and an endpointizer model on a training speech utterance set, each training speech utterance in the training speech utterance set including audio data representing the training speech utterance paired with a corresponding transcription of the training speech utterance, and a reference endpointization label sequence, each reference endpointization label including one of a reference speech label, a reference initial silence label, a reference intermediate silence label, or a reference final silence label. Jointly training an audio encoder, a decoder, and an endpointizer model on a training speech utterance set may include: determining a speech recognition loss based on the transcription of the training speech utterance and the corresponding speech recognition results predicted by the audio encoder and the decoder for the audio data; training at least one of the audio encoder or the decoder based on the speech recognition loss; determining an endpointizer loss based on a reference endpointization label sequence and a corresponding predicted endpointization label sequence output by the endpointizer model; and training the endpointizer model based on the endpointizer loss. In some examples, the operations also include: for each training speech utterance, randomly selecting an endpointizer model using a switched connection of the multi-task model to receive as input one of: a latent representation output from a final layer of an initial stack of multi-head attention layers for audio data representing the training speech utterance; or audio data representing the speech utterance.

[0010] In some implementations, the operations further include: based on determining during the VAD mode that the corresponding audio frame includes speech, switching the operation of the endpointizer model from the VAD mode to the EOQ detection mode. In some examples, the operations further include: based on determining during the EOQ detection mode that the corresponding potential representation includes final silence, switching the operation of the endpointizer model from the EOQ detection mode to the VAD mode.

[0011] In some examples, the decoder includes a prediction network and a joint network, and the operations further include: generating, by the prediction network at each output step in a plurality of output steps, a corresponding dense representation based on a sequence of non-blank symbols output by a final Softmax layer; and generating, by the joint network, a corresponding probability distribution of possible speech recognition hypotheses based on the corresponding dense representation generated by the prediction network at the corresponding output step. In some implementations, the prediction network includes an LSTM-based prediction network or a V2 embedding lookup table.

[0012] In some implementations, the multiple multi-head attention layers include a conformer layer or a transformer layer. In some examples, the operations also include: generating a corresponding second high-order feature representation for each first high-order feature representation generated by the audio encoder using a non-causal encoder of the multi-task model; and generating a probability distribution of possible speech recognition hypotheses for the sequence of audio frames based on the second high-order feature representation. In some implementations, the endpointizer model includes a stack of LSTM layers followed by a fully connected layer having a Softmax function configured to predict a probability distribution of possible endpointization labels for speech, initial silence, intermediate silence, and final silence.

[0013] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram of an example speech environment using an end-to-end (E2E) multi-task model to transcribe speech and endpointize.

[0015] Figure 2 is a schematic diagram of an example E2E multi-task model architecture with a speech recognition model, an endpointizer model, and a switching connection.

[0016] Figure 3 is a flow diagram of an example operational arrangement of a computer-implemented method for performing unified E2E speech recognition and endpointization using a handoff connection.

[0017] Figure 4 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0018] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0019] Automatic speech recognition (ASR) systems are an increasingly used technology. Modern ASR systems focus not only on providing high quality (e.g., low word error rate), but also on providing low latency (e.g., short delay between user speaking and transcription or response appearance) speech recognition of spoken utterances. For example, when using a device that implements an ASR system, it is generally expected that the ASR system decodes the utterance in a streaming manner corresponding to real-time or even faster than real-time. Conventional speech recognition models rely on separate, different and separately trained endpoint models to perform endpointization. Endpointization includes voice activity detection (VAD) and query end (EOQ) detection. VAD classifies each input audio frame according to whether it contains speech or silence. VAD classification can be used for "frame filtering", thereby discarding non-speech frames. That is, it is not input to the speech recognition model or is not processed by the speech recognition model, thereby avoiding unnecessary calculations of the speech recognition model, which is particularly important for battery-powered user devices. EOQ detection classifies each input audio frame according to whether the ongoing utterance is predicted to have ended or whether it contains an intermediate silence period. For continuous query tasks (e.g., voice dictation), high-quality EOQ detection is essential for pausing the speech recognition model during intermediate silent periods. This is especially important for battery-powered user devices because continuous query tasks may last for arbitrarily long periods of time. For short query tasks such as for digital assistants or interactive voice response applications, EOQ detection predicts when the user has finished speaking so that the speech recognition model can complete or terminate the transcription of the query and generate a response in a timely manner. For short query tasks, high-quality EOQ detection is essential for reducing speech recognition latency because responses to queries are typically not generated until the speech recognition model terminates transcription. For speech recognition systems, user perceived latency (UPL) is a very important factor in user satisfaction. Therefore, there is a need to improve VAD and EOQ detection.

[0020] The implementation of this article is directed to an end-to-end (E2E) multi-task model and method for performing speech recognition, endpointization, VAD, and EOQ detection. The E2E multi-task model integrates a speech recognition model with an endpointizer model into a single model trained to perform multiple tasks. Here, the speech recognition model can be an E2E speech recognition model that integrates an acoustic model and a language model. By integrating the endpointizer model with the speech recognition model into a single multi-task model, the endpointizer model can generate improved EOQ detection predictions by basing EOQ detection on a potential representation generated by an audio encoder of the speech recognition model rather than on raw audio frames. Here, the endpointizer model shares one or more layers with the audio encoder of the speech recognition model, for example, using hard parameter sharing. It is worth noting that the speech recognition model and the endpointizer model can be jointly trained. By integrating and jointly training the speech recognition model and the endpointizer model, VAD and EOQ detection performance can be improved because the joint training forces the speech recognition model and the endpointizer model to learn representations that generalize well across related tasks. Furthermore, by integrating the speech recognition model and the endpointizer model into a single integrated multi-task model, the infrastructure burden of building and deploying a speech recognition system is reduced since only a single integrated model needs to be trained, deployed, and maintained.

[0021] In some implementations, even a single layer of the audio encoder of the speech recognition model may be much more complex than the endpoint model. Therefore, in order to reduce the complexity of VAD before initiating speech recognition, the E2E multi-task model implementation disclosed herein includes a switching connection that allows the endpoint model to operate in two modes (VAD mode and EOQ detection mode). In VAD mode, the switching connection provides input audio frames to the endpoint model, and the endpoint model performs VAD based on the audio frames. Therefore, in the VAD mode that occurs before starting speech recognition, there is no need to activate the shared layer of the audio encoder. In VAD mode, when the endpoint model detects speech, the audio frame will then be fed to the speech recognition model, the speech recognition model (including the audio encoder) will be activated, and the endpoint model will switch operation from VAD mode to EOQ detection mode. In EOQ detection mode, the switching connection provides the potential representation output from the final layer in the shared layer to the endpoint model, and the endpoint model performs EOQ detection based on the potential representation. Thus, in EOQ detection mode, the endpointizer model can take advantage of or leverage the latent representation that the audio encoder has already generated for speech recognition purposes. Since EOQ detection mode is only active during speech recognition, during which the audio encoder is active for speech recognition purposes, EOQ detection performance can be improved without increasing computational complexity by being based on the latent representation generated by the audio encoder. In EOQ detection mode, when the endpointizer model detects the final silence that ends an utterance, a switch connection will feed the audio frame to the speech recognition model, the speech recognition model (including the audio encoder) will be activated, and the endpointizer model will switch operation from EOQ detection mode to VAD mode.

[0022] For short query tasks, the disclosed E2E multi-task model has been shown to reduce the average EOQ detection latency by more than 30% and the 90th percentile EOQ detection latency by more than 20%, without regression in word error rate (WER). In addition, for continuous query tasks, the disclosed E2E multi-task model has been shown to improve WER by integrating speech recognition and endpoint tasks.

[0023] Figure 11 is a schematic diagram of an example of a speech environment 100 and a system 101. In the speech environment 100, the way a user interacts with a computing device (such as a user device 110) may be through voice input. The user device 110 (also generally referred to as the device 110) is configured to capture sound (e.g., streamed audio data) from one or more users 102 within the speech environment 100. Here, the streamed audio data may refer to the spoken utterance 104 of the user 102, which serves as an audible query, a command to the user device 110, or an audible communication captured by the user device 110 (e.g., dictation for transcription). The speech-enabled system of the user device 110 may process the query or command by answering the query and / or causing one or more downstream applications to execute / fulfill the command.

[0024] System 101 includes a user device 110, a remote computing system 120, and a network 130. User device 110 may be any computing device capable of communicating with remote computing system 120 via network 130. User device 110 may correspond to any computing device associated with user 102 and capable of receiving audio data. Some examples of user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablet computers, laptop computers, etc.), computers, wearable devices (e.g., smart watches), smart home appliances, Internet of Things (IoT) devices, digital assistants, in-vehicle infotainment systems, smart displays, smart speakers, etc. User device 110 includes data processing hardware 111 and memory hardware 112 that communicates with data processing hardware 111 and stores instructions that, when executed by data processing hardware 111, cause data processing hardware 111 to perform one or more operations. The user device 110 also includes an audio system 142 having one or more audio capture devices 113, 113a to 113n (e.g., microphones) for capturing spoken utterances 104 within the speech environment 100 and converting the spoken utterances into electrical signals, and one or more audio output devices 114, 114a to 114n (e.g., speakers) for transmitting audible audio signals (e.g., as output audio data from the user device 110). Although in the illustrated example, the user device 110 implements a single audio capture device 113, the user device 110 may implement an array of audio capture devices 113 without departing from the scope of the present disclosure, wherein one or more of the audio capture devices 113 in the array may not physically reside on the user device 110 but may communicate with the user device 110. Similarly, although in the illustrated example, the user device 110 implements a single audio output device 114, the user device 110 may implement an array of audio output devices 114 without departing from the scope of the present disclosure, wherein one or more audio output capture devices 114 in the array may not physically reside on the user device 110 but may communicate with the user device 110.

[0025] Remote computing system 120 may be a distributed system with scalable elastic resources (e.g., a cloud computing environment). These resources include computing resources 121 (e.g., data processing hardware) and / or storage resources 122 (e.g., memory hardware). Additionally or alternatively, remote computing system 120 may be a centralized system. Network 130 may be wired, wireless, or a combination thereof, and may include private networks and / or public networks, such as the Internet.

[0026] In the example system 101, an automatic speech recognition (ASR) system 140 implementing a single E2E multi-task model 200 resides on a user device 110 of a user 102 and / or resides on a remote computing system 120 that communicates with the user device 110 via a network 130. The user device 110 and / or the remote computing system 120 also includes an audio subsystem 142 that is configured to receive an utterance 104 spoken by the user 102 and captured by an audio capture device 113, and convert the utterance 104 into a corresponding digital format associated with an input audio frame 144 that can be processed by the ASR system 140. In the example shown, the user 102 speaks a respective utterance 104, and the audio subsystem 142 converts the utterance 104 into a corresponding audio frame 144 (e.g., audio data) for input to the ASR system 140. Thereafter, the E2E multi-task model 200 receives as input an audio frame 144 corresponding to the utterance 104 and generates / predicts as output a corresponding transcription 146 (e.g., a recognition result / hypothesis) of the utterance 104. In the example shown, the E2E multi-task model 200 may perform streaming speech recognition to produce initial speech recognition results 146, 146a, and a re-scorer (not shown for clarity of illustration) may update (i.e., re-score) the initial speech recognition results 146a to produce final speech recognition results 146, 146b.

[0027] The user device 110 and / or the remote computing system 120 also executes a user interface generator 148 that is configured to present a representation of a transcription 146 of the utterance 104 to the user 102 of the user device 110. As described in more detail below, the user interface generator 148 can display an initial speech recognition result 146a in a streaming manner during time 1 and then display a final speech recognition result 146b during time 2. In some configurations, the transcription 146 output from the ASR system 140 is processed by a natural language processing / understanding (NLP / NLU) module, such as executed on the user device 110 or the remote computing system 120, to perform the user command / query specified by the utterance 104. NLP / NLU generally refers to the process of interpreting written language (e.g., speech recognition results 146) and determining whether the written language prompts any action. Additionally or alternatively, a text-to-speech system (not shown for clarity of illustration) (e.g., executing on any combination of user device 110 or remote computing system 120) may convert transcription 146 into synthesized speech for audible output by user device 110 and / or another device.

[0028] In the example shown, user 102 may interact with a program or application 115 (e.g., a digital assistant application) of user device 110 using ASR system 140. For example, Figure 1A user 102 is depicted communicating with a digital assistant application 115 and the digital assistant application 115 displays a digital assistant interface 116 on a screen 117 of a user device 110 to depict a conversation between the user 102 and the digital assistant application 115. In this example, the user 102 asks the digital assistant application 115 "What time is the concert tonight?" This question from the user 102 is a spoken utterance 104 captured by an audio capture device 113 and processed by an audio system 142 of the user device 110. In this example, the audio system 142 receives the spoken utterance 104 and converts the spoken utterance into audio frames 144 for input to the ASR system 140. Continuing with the example, the ASR system 140 includes a speech recognition model 210 and an endpointizer model 220 (see Figure 2 )The integrated single E2E multi-task model 200 encodes the audio frame 144 when receiving the audio frame 144 corresponding to the utterance 104 when the user 102 speaks, and then decodes the encoded audio frame 144 into the speech recognition result 146.

[0029] exist Figure 1 In the example shown in , the digital assistant application 115 can use NLP / NLU to respond to the question posed by the user 102. In this example, the digital assistant application 115 uses NLP / NLU to recognize that the question from the user 102 is related to the user's schedule, and more specifically, to the concerts on the user's schedule. By using NLP / NLU to recognize these details, the digital assistant application 115 returns a response 118 to the user's utterance 104, where the response 118 states "Venue doors open at7:30PM and concert starts at 9pm". In some configurations, the NLP / NLU occurs on a remote computing system 120 that communicates with the user device 110.

[0030] Figure 2Schematic diagram of an example E2E multi-task model 200 for performing speech recognition, endpointization, VAD and EOQ detection. As shown, the E2E multi-task model 200 includes a speech recognition model 210, an endpointizer model 220 and a switching connection 222, and the speech recognition model 210, the endpointizer model 220 and the switching connection 222 are integrated into a single multi-task model. It is worth noting that the speech recognition model 210, the endpointizer model 220 and the switching connection 222 of the E2E multi-task model 200 can be jointly trained, deployed and maintained. As described herein, the user device 110 executes the E2E multi-task model 200. However, it is understood that in addition to the user device 110 or instead of the user device 110, the remote computing system 120 can also execute one or more parts or all of the E2E multi-task model 200.

[0031] In the example shown, the speech recognition model 210 includes a streaming cascaded conformer-transducer (Conf-T) architecture that includes an audio encoder 240 and a decoder 250. Here, the audio encoder 240 includes a cascaded, causal encoder architecture having a first encoder 242 and a second encoder 244. The cascaded audio encoder 240 refers to a model structure in which the encoding path includes two encoders 242, 244 that are cascaded such that the output of the first encoder 242 feeds the input of the second encoder 244 before decoding.

[0032] The first encoder 242 receives or obtains a d-dimensional feature vector sequence (eg, an audio frame 144 ( Figure 1 ))x=(x1,x2,…,x T ),in The audio frame 144 sequence is encoded into a corresponding latent representation 243 as the output of the final layer of the first encoder 242. The second encoder 244 is cascaded with the first encoder 242 and is trained to receive the latent representation 243 as input and encode the latent representation 243 into a corresponding first high-order feature representation 245 as the output of the final layer of the second encoder 244. The first high-order feature representation 245 is represented as Here, each audio frame 144 includes a 128-dimensional log-mel feature vector, which is calculated every 10 milliseconds for a 32 millisecond window and stacked with three previous feature vectors to produce a 512-dimensional audio frame 144.

[0033] In some examples, the speech recognition model 210 also includes a non-causal encoder 260 that is configured to receive the first high-order feature representation 245 as input and generate a corresponding second high-order feature representation 262 for the first high-order feature representation 245 as output.

[0034] In some implementations, the cascaded audio encoder 240 includes a stack of multiple (e.g., seven) multi-headed (e.g., eight-headed) attention layers 247, 247a to 247n (e.g., conformer layers or transformer layers), where (i) the first encoder 242 includes an initial stack of layers 247a to 247b (e.g., two) from a stack of multiple layers 247 with an attention dimension of 512, and (ii) the second encoder 244 includes a time reduction stack layer that downsamples its input by a factor of two, followed by another multi-headed attention layer 247c from the stack of multiple multi-headed attention layers 247, a projection layer, and the remaining multi-headed attention layers 247d to 247n from the stack of multiple multi-headed attention layers 247. Here, causal convolutions and left-context attention layers can be used for each layer to strictly restrict the audio encoder 240 from using future inputs. The first encoder 242 can be referred to as a causal encoder, and the second encoder 244 can be referred to as a non-causal encoder.

[0035] The endpointizer model 220 is configured to operate between a VAD mode and an EOQ detection mode. When the endpointizer model 220 operates in the VAD mode, the switching connection 222 provides the input audio frame 144 to the endpointizer model 220, and the endpointizer model 220 performs VAD based on the audio frame 144. When the endpointizer model 220 operates in the VAD mode that occurs before speech recognition begins, the speech recognition model 210 (including the shared first encoder 242) is not activated or does not need to be activated (i.e., the audio frame 144 does not need to be sent to the speech recognition model 210 or does not need to be processed by the speech recognition model 210) because the endpointizer model 220 is performing VAD based on the audio frame 144. In the VAD mode, the endpointizer model 220 outputs an endpoint tag 224 for each audio frame 144, which indicates whether the audio frame 144 includes speech. During VAD mode, the endpointizer model 220 selects each endpoint label 224 as initial silence (i.e., silence before speech starts) or speech. Here, the endpointizer model E2E multi-task model 200 can determine whether the audio frame 144 includes speech by comparing the speech presence prediction probability with a predetermined probability threshold.

[0036] When the endpointizer model 220 determines that one or more audio frames 144 include speech and outputs one or more endpoint labels 224 of the speech, the E2E multi-task model 200: (i) activates the speech recognition model 210 so that the speech recognition model 210 begins to perform speech recognition on the sequence of audio frames 144; (ii) configures the switching connection 222 to provide the endpointizer model 220 with the potential representation 243 of the sequence of audio frames 144 generated by the shared portion of the audio encoder 240 (i.e., the first encoder 242); and (iii) switches the operation of the endpointizer model 220 from the VAD mode to the EOQ detection mode. In the EOQ detection mode, the endpointizer model 220 determines for each potential representation 243 whether the potential representation 243 includes a final silence indicating that an EOQ event has occurred or includes an intermediate silence, and outputs a corresponding endpoint label 224 of the final silence or the intermediate silence. Here, the endpointizer model 220 selects each endpoint label 224 as speech, intermediate silence (e.g., silence in the middle of an utterance), or final silence (e.g., after an utterance ends). Here, the endpointizer model E2E multi-task model 200 can determine whether the audio frame 144 includes speech by comparing the predicted probability of speech presence with a predetermined probability threshold. It is worth noting that the predetermined probability threshold for the EOQ detection mode can be different from the predetermined probability threshold for the VAD mode.

[0037] When the endpointizer model 220 operates in the EOQ detection mode, the switching connection 222 provides the endpointizer model 220 with the latent representation 243 output from the final layer 247b in the shared layers 247a to 247b (i.e., the first encoder 242), and the endpointizer model 220 performs EOQ detection based on the latent representation 243. Therefore, in the EOQ detection mode, the endpointizer model 220 can utilize or make use of the latent representation 243 that the audio encoder 240 has already generated for speech recognition purposes to improve the EOQ detection performance without increasing computational complexity. That is, since the EOQ detection mode is active only during speech recognition, during which the audio encoder 240 is active for speech recognition purposes, the EOQ detection performance can be improved by being based on the latent representation 243 that has already been generated by the audio encoder 240 without increasing computational complexity.

[0038] In the example shown, when operating in EOQ detection mode, the endpointizer model 220 shares one or more layers 247a to 247b with the audio encoder 240 of the speech recognition model 210. Here, the endpointizer model 220 shares a first encoder 242 with the audio encoder 240, the first encoder 242 represents an initial stack of multi-headed attention layers 247a to 247b (e.g., conformer or transformer layers) forming a stack 246 of multiple multi-headed attention layers 247a to 247n of the audio encoder 240, and the potential representation 243 is output by the final layer 247b of the initial stack of layers 247a to 247b of the first encoder 242. In some implementations, the endpointizer model 220 and the audio encoder 240 share layers using hard parameter sharing. It is noteworthy that the speech recognition model 210 and the endpointizer model 220 can be jointly trained. By integrating and jointly training the speech recognition model 210 and the endpointizer model 220, VAD and EOQ detection performance are improved because the joint training forces the speech recognition model 210 and the endpointizer model 220 to learn representations that generalize well across related tasks. When the endpointizer model 220 determines that one or more potential representations 243 include final silence and outputs an endpoint label 224 of final silence while operating in EOQ detection mode, the E2E multi-task model 200: (i) switches the operation of the endpointizer model 220 from the EOQ detection mode to the VAD mode; (ii) configures the switching connection 222 to provide the input audio frame 144 to the endpointizer model 220; and (iii) disables the speech recognition model 210.

[0039] In some implementations, when the endpointizer model 220 determines that one or more potential representations 243 include intermediate silence and outputs an endpoint label 224 for the intermediate silence while operating in the EOQ detection mode, the E2E multi-task model 200: (i) temporarily switches the operation of the endpointizer model 220 from the EOQ detection mode to the VAD mode; (ii) configures the switching connection 222 to provide the input audio frame 144 to the endpointizer model 220; and (iii) temporarily disables the speech recognition model 210. When speech continues (e.g., when the endpointizer model 220 operating in the VAD mode detects speech), the E2E multi-task model 200 restores the endpointizer model 220 back to the EOQ detection mode and resumes speech recognition through the speech recognition model 210. In this way, the speech recognition model 210 does not need to operate during the intermediate silence. In some implementations, the endpointizer model 220 includes a stack of LSTM layers followed by a fully connected layer with a Softmax function configured to predict a probability distribution of possible endpointization labels for speech, initial silence, intermediate silence, and final silence.

[0040] In the example shown, the decoder 250 includes an RNN-T architecture having a joint network 252, a prediction network 254, and a Softmax layer 256. The decoder 250 combines the first high-order feature representation 245 and / or the second high-order feature representation 262 with a dense or hidden representation 255 output from the prediction network 254 for a previous prediction output 257 of the Softmax layer 256 using the joint network 252 to produce a prediction output 257. In the example shown, the decoder 250 includes the Softmax layer 256. Alternatively, the Softmax layer 256 can be implemented separately.

[0041] In the example shown, the prediction network 254 takes the sequence (y0, ..., y1) of non-blank symbols 257 (ie, the predicted output) outputted so far by the final softmax layer 256. ui-1 ) is processed into dense or hidden representation 255. In some implementations, the dense representation 255 including

[0042] single embedding vector. Notably, the sequence of past non-blank symbols 257 received at the prediction network 254 captures the linguistic correlation between the non-blank symbols 257 currently predicted during the previous time step to assist the joint network 252 in predicting the probability of the next output symbol or blank symbol during the current time step. In order to facilitate the technique for reducing the size of the prediction network 254 without sacrificing the accuracy / performance of the decoder 250, the prediction network 254 can receive a limited historical sequence y of non-blank symbols 257 ui-n ,...,y ui-1 , the finite history sequence is limited to the N previous non-blank symbols 257 output by the final Softmax layer 256.

[0043] In the example shown, the joint network 252 combines the first high-order feature representation 245 produced by the audio encoder 240 and / or the second high-order feature representation 262 produced by the non-causal encoder 260 with the dense representation produced by the prediction network 254. 255. The joint network 252 predicts the probability distribution of the next output symbol 253. In other words, the joint network 252 generates a probability distribution 253 of possible speech recognition hypotheses at each time step. Here, the "possible speech recognition hypotheses" correspond to a set of output labels, each output label representing a symbol / character in a specified natural language. For example, when the natural language is English, the output label set may include twenty-seven (27) symbols, for example, one label for each of the 26 letters in the English alphabet, and one label specifying a space. Therefore, the joint network 252 can output a set of values ​​indicating the likelihood of each of the predetermined output label sets occurring. The value set can be a vector and can indicate a probability distribution of the output label set. In some cases, the output labels are graphemes (e.g., individual characters, and possibly punctuation and other symbols), but the output label set is not limited to this. For example, in addition to or in place of graphemes, the output label set may also include word fragments and / or entire words. The output distribution of the joint network 252 may include a posterior probability value in each of the different output labels. Therefore, if there are 100 different output labels representing different graphemes or other symbols, the output Z of the joint network 252 is i 253 may include 100 different probability values, one for each output label. The probability distribution may then be used (e.g., by a Softmax layer 256) to select candidate orthographic elements (e.g., graphemes, fragments, and / or words) and assign scores thereto in a beam search process for use in determining a transcription 146.

[0044] In the example shown, the final Softmax layer 256 receives the probability distribution Z i 253 and selects the output label / symbol with the highest probability to produce the transcription 146. The final Softmax layer 256 can use any technique to select the output label / symbol with the highest probability in the distribution Z i 253 outputs the label / symbol with the highest probability. In this way, decoder 250 does not make conditional independence assumptions, but rather for each symbol y u The prediction of 257 is conditioned not only on the acoustics, but also on the label 257 sequence y output so far ui-n ,…,y ui-1 The decoder 250 does assume that the output symbols 257 are independent of the future acoustic frames 144, which allows the speech recognition model 210 to be employed in a streaming manner.

[0045] In some implementations, the prediction network 254 includes a V2 embedding lookup table that includes an embedding prediction network. At each time step, the V2 embedding lookup table can receive the previous two predictions (e.g., one-hot vectors) output by the joint network 252 as input, calculate the corresponding embedding d1, d2 for each of the previous two predictions, and provide a cascade output [d1, d2] to the joint network 252. Alternatively, the prediction network 254 may include one or more conformer layers or transformer layers. Alternatively, the prediction network 254 may be an LSTM-based prediction network including one or more long short-term memory (LSTM) layers, each LSTM layer is followed by a projection layer and an embedding layer. In some implementations, the joint network 252 includes one or more neural network layers, each neural network layer has a plurality of hidden units, and the Softmax layer 256 is composed of a unified word fragment or grapheme set, which is generated using all unique word fragments or graphemes in multiple training data sets.

[0046] It is worth noting that the speech recognition model 210 and the endpointizer model 220 can be jointly trained on a training speech utterance set using multi-task learning. Here, each training speech utterance in the training speech utterance set includes audio data representing the training speech utterance paired with a corresponding transcription of the training speech utterance, and a sequence of reference endpointization labels, each reference endpointization label including one of a reference speech label, a reference initial silence label, a reference intermediate silence label, or a reference final silence label. In some implementations, the speech recognition model 210 is trained on an ASR task using the training speech utterance set by determining a speech recognition loss based on the speech recognition results predicted by the speech recognition model 210 for the audio data and the corresponding transcription of the training speech utterance And based on speech recognition loss Here, the endpointizer model 220 is trained on the endpointization task using the training speech utterance set by: determining the endpointization loss based on the reference endpointization label sequence and the corresponding predicted endpointization label sequence output by the endpointizer model 220 And based on the endpointizer loss To train the endpointizer model 220. In other implementations, based on the same weighted combination loss To train the speech recognition model 210 and the endpointizer model 220, the weighted combination loss is based on the speech recognition loss and endpoint loss Determined by, the weighted combination loss can be expressed as

[0047]

[0048] Where λ∈[0,1] is a hyperparameter that defines the relative weights given to the speech recognition and endpointization tasks. In some examples, for each training speech utterance, the switching connection 222 randomly selects the endpointizer model 220 to receive as input one of the following: the latent representation 243 output from the final layer of the initial stack of multi-head attention layers (i.e., the final layer of the first encoder 242) for the audio data representing the training speech utterance; or the audio data representing the training speech utterance.

[0049] Figure 3 4 is a flow chart of an exemplary operational arrangement of a computer-implemented method 300 for performing unified E2E speech recognition and endpointization using a handoff connection. The method 300 may be implemented on the data processing hardware 410 ( Figure 4 ) to perform these operations. The data processing hardware 410 may include the data processing hardware 111 ( Figure 1 ) and / or the data processing hardware 121 of the remote computing system 120 ( Figure 1 ). The memory hardware 420 may include the data memory hardware 112 ( Figure 1 ) and / or memory hardware 122 of a remote computing system 120.

[0050] At operation 302, the method 300 includes receiving a sequence of audio frames 144 representing an utterance 104. The method 300 includes: at operation 304, processing the sequence of audio frames 144 by an audio encoder 240 of a single E2E multi-task model 200 to generate a corresponding first high-order feature representation 245, the audio encoder 240 including a plurality of multi-head attention layers; and at operation 305, generating a probability distribution 253 of possible speech recognition hypotheses for the sequence of audio frames 144 based on the first high-order feature representation 245 by a decoder 250 of the E2E multi-task model 200; and

[0051] At operation 308, the method 300 includes determining, during the VAD mode, for each corresponding audio frame 144 in the sequence of audio frames 144 using the endpointizer model 220 of the E2E multi-task model 200, which shares with the audio encoder 240 an initial stack 248 of multi-headed attention layers 247a-247b from a stack 246 of multiple multi-headed attention layers 247. At operation 310, the method 300 includes determining, during the EOQ detection mode, for each corresponding potential representation 243 of a plurality of potential representations 243 for the sequence of audio frames 144 output from a final layer 247b of the initial stack 248 of multi-headed attention layers 247a-247b, whether the corresponding potential representation 243 includes a final silence.

[0052] Figure 4 4 is a schematic diagram of an example computing device 400 that can be used to implement the systems and methods described in this document. Computing device 400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0053] The computing device 400 includes a processor 410 (i.e., data processing hardware) that can be used to implement the data processing hardware 111 and / or 121, a memory 420 (i.e., memory hardware) that can be used to implement the memory hardware 112 and / or 122, a storage device 430 (i.e., memory hardware) that can be used to implement the memory hardware 112 and / or 122, a high-speed interface / controller 440 connected to the memory 420 and the high-speed expansion port 450, and a low-speed interface / controller 460 connected to the low-speed bus 470 and the storage device 430. Each of the components 410, 420, 430, 440, 450, and 460 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 410 can process instructions for execution within the computing device 400, including instructions stored in the memory 420 or on the storage device 430, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 480 coupled to the high-speed interface 440). In other implementations, multiple processors and / or multiple buses and multiple memories and types of memories may be used as appropriate. In addition, multiple computing devices 400 may be connected, each providing a portion of the necessary operations (e.g., as a server group, a group of blade servers, or a multi-processor system).

[0054] Memory 420 stores information non-temporarily within computing device 400. Memory 420 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 420 may be a physical device for temporarily or permanently storing programs (e.g., sequences of instructions) or data (e.g., program state information) for use by computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0055] Storage device 430 is capable of providing mass storage for computing device 400. In some implementations, storage device 430 is a computer-readable medium. In various implementations, storage device 430 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as memory 420, storage device 430, or memory on processor 410.

[0056] The high-speed controller 440 manages bandwidth-intensive operations of the computing device 400, while the low-speed controller 460 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 440 is coupled to the memory 420, the display 480 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 450 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 460 is coupled to the storage device 430 and the low-speed expansion port 490. The low-speed expansion port 490, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.

[0057] As shown, computing device 400 can be implemented in a variety of different forms. For example, the computing device can be implemented as a standard server 400a or multiple implementations in a group of such servers 400a, implemented as a laptop computer 400b, or implemented as part of a rack server system 400c.

[0058] Various implementations of the systems and techniques described herein may be implemented in digital electronic and / or optical circuitry, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that may be executed and / or interpreted on a programmable system that includes at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0059] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0060] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0061] The processes and logic flows described in this specification may be performed by one or more programmable processors (also referred to as data processing hardware) that execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows may also be performed by a dedicated logic circuit system (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or may be operably coupled to receive data from one or more mass storage devices or to transfer data to one or more mass storage devices or both. However, a computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0062] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and possibly a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0063] Unless expressly stated to the contrary, the phrase “at least one of A, B, or C” is intended to refer to any combination or subset of A, B, and C, such as: (1) only at least one A; (2) only at least one B; (3) only at least one C; (4) at least one A and at least one B; (5) at least one A and at least one C; (6) at least one B and at least one C; and (7) at least one A and at least one B and at least one C. Furthermore, unless expressly stated to the contrary, the phrase “at least one of A, B, and C” is intended to refer to any combination or subset of A, B, and C, such as: (1) only at least one A; (2) only at least one B; (3) only at least one C; (4) at least one A and at least one B; (5) at least one A and at least one C; (6) at least one B and at least one C; and (7) at least one A and at least one B and at least one C. Furthermore, unless expressly stated to the contrary, “A or B” is intended to refer to any combination of A and B, such as: (1) only A; (2) only B; and (3) A and B.

[0064] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A single end-to-end multi-task model (200) for performing speech recognition and end-point localization, characterized in that The multi-task model (200) comprises: A speech recognition model (210), the speech recognition model comprising: an audio encoder (240) configured to encode a sequence of audio frames (144) into a corresponding first high-order feature representation (245), the audio encoder (240) comprising a plurality of multi-head attention layers (247); and a decoder (250) configured to generate a probability distribution (253) of possible speech recognition hypotheses for the sequence of audio frames (144) based on the first high-order feature representation (245); and An endpointizer model (220) configured to operate between a voice activity detection (VAD) mode and an end-of-query (EOQ) detection mode, wherein: During the VAD mode, the endpointizer model (220) is configured to receive input audio frames (144) and determine, for each input audio frame (144), whether the input audio frame (144) includes speech; and During the EOQ detection mode, the endpointizer model (220) shares an initial stack of multi-headed attention layers (247) from the plurality of multi-headed attention layers (247) with the audio encoder (240), and is configured to receive a latent representation (243) for the sequence of audio frames (144) output from a final layer (247) of the initial stack of multi-headed attention layers (247), and to determine for each of the latent representations (243) whether the latent representation (243) includes a final silence.

2. The multi-task model (200) according to claim 1, characterized in that: The speech recognition model (210) and the endpointizer model (220) are jointly trained on a training speech utterance set using multi-task learning, each training speech utterance in the training speech utterance set comprising: audio data representing the training speech utterances paired with corresponding transcriptions of the training speech utterances; and A reference endpointization label sequence, each reference endpointization label comprises one of a reference speech label, a reference initial silence label, a reference intermediate silence label, or a reference final silence label.

3. The multi-task model (200) according to claim 2, characterized in that: The speech recognition model (210) and the endpointizer model (220) are jointly trained on the training speech utterance set by: determining a speech recognition loss based on a speech recognition result predicted by the speech recognition model (210) for the audio data and the corresponding transcription of the training speech utterance; Training the speech recognition model based on the speech recognition loss (210); determining an endpointizer loss based on the reference endpointized tag sequence and a corresponding predicted endpointized tag sequence output by the endpointizer model (220); as well as The endpointizer model is trained based on the endpointizer loss (220).

4. The multi-task model (200) according to any one of claims 2 to 3, characterized in that: For each training speech utterance, the switching connection (222) of the multi-task model (200) randomly selects the endpointizer model (220) to receive as input one of the following: a latent representation (243) output from the final layer (247) of the initial stack of multi-headed attention layers (247) for the audio data representing the training speech utterance; or The audio data representing the training speech utterances.

5. The multi-task model (200) according to any one of claims 1 to 4, characterized in that: When the endpointizer model (220) determines during the VAD mode that the input audio frame (144) includes speech, the endpointizer model (220) switches operation from the VAD mode to the EOQ detection mode.

6. The multi-task model (200) according to any one of claims 1 to 5, characterized in that: When the endpointizer model (220) determines during the EOQ detection mode that the potential representation (243) includes terminal silence, the endpointizer model (220) switches operation from the EOQ detection mode to the VAD mode.

7. The multi-task model (200) according to any one of claims 1 to 6, characterized in that: The decoder (250) comprises: A prediction network (254), the prediction network being configured to: receiving as input a sequence of non-whitespace symbols (257) output by the final Softmax layer (256); and generating a dense representation (255) as output; and A joint network (252), the joint network being configured to: receiving as input the dense representation (255) generated by the prediction network (254) at each of a plurality of output steps and the first high-order feature representation (245) generated by the audio encoder (240) at each of the plurality of output steps; and The probability distribution of possible speech recognition hypotheses is generated at each output step in the plurality of output steps (253).

8. The multi-task model (200) according to claim 7, characterized in that: The prediction network (254) comprises: Long Short-Term Memory (LSTM) based prediction networks; or V2 embeds the lookup table.

9. The multi-task model (200) according to any one of claims 1 to 8, characterized in that: The multiple multi-head attention layers (247) include conformer layers or transformer layers.

10. The multi-task model (200) according to any one of claims 1 to 9, characterized in that: The speech recognition model (210) further comprises a non-causal encoder (260), wherein the non-causal encoder is configured to: receiving as input the first high-level feature representation (245) encoded by the audio encoder (240); and generating a corresponding second high-order feature representation (262) as output for the first high-order feature representation (245); and The decoder (250) is configured to generate the probability distribution (253) of possible speech recognition hypotheses for the sequence of audio frames (144) based on the second high-order feature representation (262).

11. The multi-task model (200) according to any one of claims 1 to 10, characterized in that: The endpointizer model (220) includes a stack of one or more long short-term memory (LSTM) layers followed by a fully connected layer having a Softmax function configured to predict a probability distribution of possible endpointization labels for speech, initial silence, intermediate silence, and final silence.

12. A computer-implemented method (300) executed on data processing hardware (410), the computer-implemented method causing the data processing hardware (410) to perform an operation, characterized in that The operations include: receiving a sequence of audio frames (144) representing speech; The audio frame (144) sequence is processed by an audio encoder (240) of a single end-to-end multi-task model (200) to generate a corresponding first high-order feature representation (245), wherein the audio encoder (240) includes a plurality of multi-head attention layers (247); Generating, by a decoder (250) of the multi-task model (200), a probability distribution (253) of possible speech recognition hypotheses for the sequence of audio frames (144) based on the first high-order feature representation (245); and Using an endpointizer model (220) of the multi-task model (200), the endpointizer model shares an initial stack of multi-headed attention layers (247) from the plurality of multi-headed attention layers (247) with the audio encoder (240): During a voice activity detection (VAD) mode, determining for each corresponding audio frame (144) in the sequence of audio frames whether the corresponding audio frame (144) includes speech; and During an end-of-query (EOQ) detection mode, for each corresponding potential representation (243) of a plurality of potential representations (243) for the sequence of audio frames (144) output from a final layer (247) of an initial stack of multi-head attention layers (247), it is determined whether the corresponding potential representation (243) includes a final silence.

13. The computer-implemented method of claim 12, wherein: The audio encoder (240), the decoder (250), and the endpointizer model (220) are jointly trained using multi-task learning on a training speech utterance set, each training speech utterance in the training speech utterance set comprising: audio data representing the training speech utterances paired with corresponding transcriptions of the training speech utterances; and A reference endpointization label sequence, each reference endpointization label comprises one of a reference speech label, a reference initial silence label, a reference intermediate silence label, or a reference final silence label.

14. The computer-implemented method of claim 13, wherein: The audio encoder (240), the decoder (250) and the endpointizer model (220) are jointly trained on the training speech utterance set including: determining a speech recognition loss based on the transcription of the training speech utterance and corresponding speech recognition results predicted by the audio encoder (240) and the decoder (250) for the audio data; training at least one of the audio encoder (240) or the decoder (250) based on the speech recognition loss; determining an endpointizer loss based on the reference endpointized tag sequence and a corresponding predicted endpointized tag sequence output by the endpointizer model (220); and The endpointizer model is trained based on the endpointizer loss (220).

15. The computer-implemented method according to any one of claims 13 to 14, characterized in that: For each training speech utterance, the switching connection of the multi-task model randomly selects the endpointizer model (220) to receive as input one of the following: a latent representation (243) output from the final layer (247) of the initial stack of multi-headed attention layers (247) for the audio data representing the training speech utterance; or The audio data representing the speech utterance.

16. The computer-implemented method according to any one of claims 12 to 15, characterized in that: The operations also include switching operation of the endpointizer model (220) from the VAD mode to the EOQ detection mode based on determining during the VAD mode that the corresponding audio frame (144) includes speech.

17. The computer-implemented method according to any one of claims 12 to 16, characterized in that: The operations also include switching the operation of the endpointizer model (220) from the EOQ detection mode to the VAD mode based on determining during the EOQ detection mode that the corresponding potential representation (243) includes final silence.

18. The computer-implemented method according to any one of claims 12 to 17, characterized in that: The decoder (250) includes a prediction network (254) and a joint network (252); and The operations further include, at each output step in the plurality of output steps: Generate a corresponding dense representation (255) by the prediction network (254) based on the sequence of non-whitespace symbols (257) output by the final Softmax layer (256); and A corresponding probability distribution (253) of possible speech recognition hypotheses is generated by the joint network (252) based on the corresponding dense representation (255) generated by the prediction network (254) at the corresponding output step.

19. The computer-implemented method of claim 18, wherein: The prediction network (254) comprises: Long Short-Term Memory (LSTM) based prediction networks; or V2 embeds the lookup table.

20. The computer-implemented method of any one of claims 12 to 19, wherein: The multiple multi-head attention layers (247) include conformer layers or transformer layers.

21. The computer-implemented method of any one of claims 12 to 20, wherein: The operations also include: generating, using the non-causal encoder (260) of the multi-task model (200), for each first high-order feature representation (245) generated by the audio encoder (240), a corresponding second high-order feature representation (262); and The probability distribution (253) of possible speech recognition hypotheses is generated for the sequence of audio frames (144) based on the second high-order feature representation (262).

22. The computer-implemented method of any one of claims 12 to 21, wherein: The endpointizer model (220) includes a stack of long short-term memory (LSTM) layers followed by a fully connected layer having a Softmax function configured to predict a probability distribution of possible endpointization labels for speech, initial silence, intermediate silence, and final silence.

Images