Intended Query Detection Using E2E Modeling for Continuous Conversation

Abstract

The method (500) includes receiving audio data (110) corresponding to a spoken utterance (106) as an input to a speech recognition model (200). The method also includes performing speech recognition on the audio data by encoding the audio data corresponding to the spoken utterance into a corresponding audio encoding (224) using an audio encoder (220) at each of a plurality of time steps and decoding the corresponding audio encoding into a probability distribution (242) over possible output labels using a speech recognition joint network (240). At each of the plurality of time steps, the method also includes determining an intended query decision (212) indicating whether the spoken utterance includes a query directed to the digital assistant (105) using an intended query (IQ) joint network (230) configured to receive a label history representation (350) associated with a sequence of non-blank symbols (252) output by a final softmax layer (250).

Description

[Technical field]

[0001] The present disclosure relates to intended query detection using end-to-end (E2E) modeling for sustained conversations. [Background technology]

[0002] In a voice-enabled environment, a user need only speak a query or command out loud, and the digital assistant will address and respond to the query and / or execute the command. A voice-enabled environment (e.g., at home, at work, at school, etc.) may be implemented with a network of connected microphone devices located in various rooms or environmental areas. Through such a network of microphones, a user has the ability to verbally query a digital assistant from essentially anywhere in the environment, without the need to have a computer or other device in front of or nearby. For example, while cooking in the kitchen, a user may activate a digital assistant using a hotword such as "Ok, computer, set the timer for 20 minutes," and in response, the digital assistant will confirm that the timer is set (in the form of a synthesized voice output) and will alert the user (e.g., in the form of an alarm or other audible alert from an audio speaker) when the timer reaches 20 minutes. In many cases, a user may issue a subsequent query to the digital assistant. However, the user's need to repeat a hotword to address the digital assistant places a cognitive burden on the user and interrupts the flow of a continued conversation. Summary of the Invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on the data processing hardware, causes the data processing hardware to perform operations including receiving audio data corresponding to a spoken utterance as an input to a speech recognition model. The operations also include performing speech recognition on the audio data by using the speech recognition model to encode the audio data corresponding to the spoken utterance into a corresponding audio encoding using an audio encoder at each of a plurality of time steps, and decoding the corresponding audio encoding encoded by the audio encoder at the corresponding time step into a probability distribution over possible output labels for the spoken utterance at the corresponding time step using a speech recognition joint network. At each of the plurality of time steps, the operations also include determining an intended query decision indicating whether the spoken utterance includes a query directed to a digital assistant interface using an intended query (IQ) joint network configured to receive a label history representation associated with the sequence of non-blank symbols output by the final softmax layer.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the speech recognition model includes an audio encoder, a speech recognition joint network, and a prediction network. The prediction network is configured to receive the sequence of non-blank symbols output by the final softmax layer and generate a label history representation at each of a plurality of time steps. In these implementations, the speech recognition model is trained during a first training stage by optimizing the audio encoder, the speech recognition joint network, and the prediction network using a regular label sequence of the word pieces, and the IQ joint network is initialized by the joint network during a second training stage by freezing the audio encoder and the prediction network and fine-tuning the IQ joint network with an extended label sequence of both the word pieces and the IQ tokens to teach the IQ joint network how to predict a distribution of IQ tokens indicative of whether an input utterance includes a query targeted to a digital assistant interface. In these implementations, generating a label history representation for a corresponding sequence of non-blank symbols includes, for each non-blank symbol in the sequence of non-blank symbols received as input at each of the multiple time steps, generating, by the prediction network, an embedding of the corresponding non-blank symbol using the shared embedding matrix, assigning, by the prediction network, a respective position vector to the corresponding non-blank symbol, and weighting, by the prediction network, the embedding proportional to a similarity between the embedding and the respective position vector, where the operation also includes generating, as output from the prediction network, a single embedding vector based on a weighted average of the weighted embeddings at the corresponding time step, the single embedding vector including the label history representation, where the prediction network may include a multi-head attention mechanism that shares the shared embedding matrix across each head of the multi-head attention mechanism.

[0005] In some examples, audio data corresponding to the spoken utterance is received during a current dialog session between the user and the digital assistant interface. In some implementations, the output label includes a word piece, a word, a phoneme, or a grapheme. In some examples, the audio encoder includes a causal encoder including one of a plurality of one-way long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. In some implementations, the speech recognition model is trained using a hybrid autoregressive transducer factorization. In some examples, the operations further include providing a response to the received spoken utterance when the intended query determination indicates that the spoken utterance includes a query directed to the digital assistant interface. Alternatively, the operations further include discarding the received spoken utterance when the intended query determination indicates that the spoken utterance does not include a query directed to the digital assistant interface.

[0006] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including receiving audio data corresponding to the spoken utterance as an input to a speech recognition model. The operations also include performing speech recognition on the audio data by using the speech recognition model to encode the audio data corresponding to the spoken utterance into a corresponding audio encoding using an audio encoder at each of a plurality of time steps, and decoding the corresponding audio encoding encoded by the audio encoder at the corresponding time step into a probability distribution over possible output labels for the spoken utterance at the corresponding time step using a speech recognition joint network. At each of the plurality of time steps, the operations also include determining an intended query decision indicating whether the spoken utterance includes a query directed to a digital assistant interface using an intended query (IQ) joint network configured to receive a label history representation associated with the sequence of non-blank symbols output by the final softmax layer.

[0007] This aspect may include one or more of the following optional features. In some implementations, the speech recognition model includes an audio encoder, a speech recognition joint network, and a prediction network. The prediction network is configured to receive the sequence of non-blank symbols output by the final softmax layer and generate a label history representation at each of a plurality of time steps. In these implementations, the speech recognition model is trained during a first training stage by optimizing the audio encoder, the speech recognition joint network, and the prediction network using a regular label sequence of the word pieces, and the IQ joint network is initialized by the joint network during a second training stage by freezing the audio encoder and the prediction network and fine-tuning the IQ joint network with an extended label sequence of both the word pieces and the IQ tokens to teach the IQ joint network how to predict a distribution of IQ tokens indicating whether an input utterance includes a query targeted to a digital assistant interface. In these implementations, generating a label history representation for a corresponding sequence of non-blank symbols includes, for each non-blank symbol in the sequence of non-blank symbols received as input at each of the multiple time steps, generating, by the prediction network, an embedding of the corresponding non-blank symbol using the shared embedding matrix, assigning, by the prediction network, a respective position vector to the corresponding non-blank symbol, and weighting, by the prediction network, the embedding proportional to a similarity between the embedding and the respective position vector, where the operation also includes generating, as output from the prediction network, a single embedding vector based on a weighted average of the weighted embeddings at the corresponding time step, the single embedding vector including the label history representation, where the prediction network may include a multi-head attention mechanism that shares the shared embedding matrix across each head of the multi-head attention mechanism.

[0008] In some examples, audio data corresponding to the spoken utterance is received during a current dialog session between the user and the digital assistant interface. In some implementations, the output label includes a word piece, a word, a phoneme, or a grapheme. In some examples, the audio encoder includes a causal encoder including one of a plurality of one-way long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. In some implementations, the speech recognition model is trained using a hybrid autoregressive transducer factorization. In some examples, the operations further include providing a response to the received spoken utterance when the intended query determination indicates that the spoken utterance includes a query directed to the digital assistant interface. Alternatively, the operations further include discarding the received spoken utterance when the intended query determination indicates that the spoken utterance does not include a query directed to the digital assistant interface.

[0009] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0010] [Figure 1A] FIG. 1 is a schematic diagram of an example system including an automatic speech recognition (ASR) system including an intended query detector for transcribing spoken utterances. [Figure 1B] FIG. 1 is a schematic diagram of an example system including an automatic speech recognition (ASR) system including an intended query detector for transcribing spoken utterances. [Diagram 2] FIG. 1 is a schematic diagram of an exemplary ASR system integrating an intended query detector. [Diagram 3] FIG. 3 is a schematic diagram of an exemplary linked, reduced prediction network of the ASR system of FIG. 2. [Figure 4A] 1 shows an example long transcribed training utterance. [Figure 4B] 4B shows exemplary annotated transcribed training utterances for the long transcribed training utterances of FIG. 4A. [Diagram 5] 1 is a flowchart of an exemplary arrangement of operations of a computer-implemented method for executing an intended query detection model in an ASR system. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0011] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The way in which a user interacts with an assistant-enabled device is designed to be primarily, but not exclusively, through voice input. As a result, the assistant-enabled device needs to have some way of identifying when any given utterance in the surrounding environment is directed at the device as opposed to being directed at an individual in the environment or originating from a non-human source (e.g., a television or music player). One way to achieve this is to use hot words that are reserved by agreement between users in the environment as a predetermined word or words to be spoken to get the attention of the device. In an exemplary environment, the hot word used to get the attention of the assistant is the words "OK, computer". As a result, whenever the words "OK, computer" are spoken, they are picked up by the microphone and communicated to a hot word detector, which performs speech understanding techniques to determine if a hot word has been spoken, and if so, waits for the next command or query. Thus, utterances directed to an assistant-enabled device take the general form [hot word][query], where the "hot word" in this example is "OK computer" and the "query" can be any question, command, declaration, or other request that can be voice-recognized, analyzed, and acted upon by the system, either alone or in conjunction with a server over a network.

[0013] If the user continues the conversation with an assistant-enabled device, such as a mobile phone or smart speaker, the user's interaction with the phone or speaker may be awkward. The user may say, "Ok, computer, play the assignments playlist." The phone or speaker may start playing the first song in the playlist. The user may want to go to the next song and may say, "Ok, computer, next." To go to yet another song, the user may say, "Ok, computer, next" again. To alleviate the need to keep repeating the hotword before speaking a command, the assistant-enabled device may be configured to recognize / detect that the utterance "next" is directed / targeted at the assistant-enabled device to go to the next song. In this example, instead of saying "Ok, computer, next," the user may simply say "next" to invoke the assistant-enabled device to trigger it to take the respective action. In contrast, the user may say to another user in the environment, "What do you think of that movie?" Here, the assistant-enabled device may recognize / detect that the query is not intended for the assistant-enabled device and therefore will not respond to the query.

[0014] Embodiments herein are directed to integrating an automatic speech recognition (ASR) system with an intended query detection model configured and trained to detect intended and unintended queries that occur naturally in spoken utterances of natural conversational interactions. An exemplary intended query detection model is built on or integrated with an end-to-end (E2E) ASR model, such as a recurrent neural network-transducer (RNN-T) model. In an exemplary method, an ASR system receives audio data corresponding to the spoken utterance as an input to the ASR model. The system performs speech recognition on the audio data using the ASR model by encoding, at each of a plurality of time steps, the audio data corresponding to the spoken utterance into a corresponding audio encoding using an audio encoder, and decoding, at the corresponding time step, the corresponding audio encoding encoded by the audio encoder into a probability distribution over the output label of the spoken utterance at the corresponding time step using a speech recognition joint network. At each of a plurality of time steps, an intended query (IQ) joint network configured to receive the label history representation associated with the sequence of non-whitespace symbols output by the final softmax layer determines an intended query decision indicating whether the spoken utterance includes a query directed to the digital assistant.

[0015] 1A and 1B illustrate example systems 100a, 100b including an example voice environment. In a voice environment, a user 104 interacting with a computing device, such as a user device 10, may be through voice input. The user device 10 (also commonly referred to as device 10) is configured to capture sounds (e.g., streaming audio data) from one or more users 102 in the voice environment. Here, the streaming audio data may refer to utterances 106 spoken by the users 102 that function as audible queries, commands to the device 10, or audible communications captured by the device 10. The voice-enabled system of the device 10 may act on the queries or commands by answering the queries and / or causing the commands to be performed / fulfilled by one or more downstream applications.

[0016] The systems 100a, 100b include a user device 10 running a digital assistant 105 (also commonly referred to as a digital assistant interface) with which a user 102 of the device 10 can interact during a dialogue session between the user 102 and the digital assistant 105 by issuing queries that include commands to perform actions. In the illustrated example, the user device 10 corresponds to a portable smart speaker with which the user 102 can interact. However, the user device 10 can include other computing devices such as, but not limited to, a smartphone, a tablet, a smart display, a desktop / laptop, a smart watch, a smart glasses / headset, a smart appliance, headphones, or a vehicle infotainment device. The user device 10 includes data processing hardware 12 and memory hardware 14 that stores instructions that, when executed on the data processing hardware 12, cause the data processing hardware 12 to perform operations. In some examples, the user device 10 is configured to communicate with a remote system 130 via a network 120. The remote system 130 may include remote resources such as remote data processing hardware 132 (e.g., a remote server or CPU) and / or remote memory hardware 134 (e.g., a remote database or other storage hardware). The user device 10 includes an array of one or more microphones 16 configured to capture sounds, such as voice, directed at the user device 10. The user device 10 may also include or communicate with an audio output device (e.g., a speaker) 18 that may output audio, such as music and / or synthesized voice, from the digital assistant 105. Additionally, the user device 10 may include or communicate with one or more cameras configured to capture images of the environment and output image data.

[0017] In a speech environment, an automatic speech recognition (ASR) system 118 implementing a recurrent neural network-transducer (RNN-T) model 200 including an intended query (IQ) joint network 230 resides in a user device 10 of a user 102 and / or in a remote system 130 in communication with the user device 10 via a network 120. The user device 10 and / or the remote system 130 also includes an audio subsystem 108 configured to receive speech 106 produced by the user 102 and captured by the microphone 16 and convert the utterances 106 into a corresponding digital form associated with input acoustic frames 110 processable by the ASR system 118. In the illustrated example, the user speaks each utterance 106 and the audio subsystem 108 converts the utterances 106 into corresponding audio data (e.g., acoustic frames) 110 for input to the ASR system 118. The RNN-T model 200 then receives as input acoustic frames 110 corresponding to the utterance 106 and generates / predicts as output a corresponding transcription 120 of the utterance 106. In the illustrated example, the RNN-T model 200 may perform streaming speech recognition to create partial transcriptions (e.g., streaming speech recognition results) 120 as the user speaks.

[0018] In the example shown in FIG. 1A, a user 102 interacts with a program or application (e.g., a digital assistant application 105) of the user device 10 that uses the ASR system 118. In this example, the user 102 speaks a first utterance 106 ("Ok, computer, what's the weather forecast today?") during time 1. The microphone 16 of the AED 104 receives the first utterance 106, and the audio subsystem 108 converts the first utterance 106 into corresponding audio data 110 that corresponds to the first utterance 106. Initial processing of the audio data 110 may include filtering the audio data 110 and converting the audio data 110 from an analog signal to a digital signal. As the audio subsystem 108 processes the audio data 110, the device 10 may store the audio data 110 in a buffer in the memory hardware 14 for further processing. With the audio data 110 in the buffer, the device 110 may use a hot word detector (not shown) to detect whether the audio data 110 contains hot words. The hot word detector is configured to identify hot words contained in the audio data 110 without performing speech recognition on the audio data 110.

[0019] In some implementations, the hot word detector is configured to identify hot words in an early portion of the first utterance 106. In this example, the hot word detector may determine that the first utterance 106, "Ok computer, what's the weather like tomorrow?", includes the hot word 112, "ok computer", if the hot word detector detects acoustic features of the input audio that are characteristic of the hot word 112. The acoustic features may be Mel-frequency Cepstral Coefficients (MFCCs), which are a representation of the short-term power spectrum of the first query 106, or may be Mel-scale filter bank energies of the first utterance 106. For example, the hot word detector may detect that the first utterance 106, "Ok computer, what's the weather like tomorrow?", includes the hot word 112, "ok computer", based on generating MFCCs from the audio data 110 and classifying the MFCCs as including MFCCs similar to MFCCs that are characteristic of the hot word "OK computer" as stored in the hot word model of the hot word detector. As another example, the hot word detector may detect that the first utterance 106, “Ok computer, what's the weather forecast for tomorrow?”, contains the hot word 112, “ok computer,” based on generating mel-scale filter bank energy from the audio data 110 and classifying the mel-scale filter bank energy as containing mel-scale filter bank energy similar to mel-scale filter bank energy that is characteristic of the hot word “ok computer” as stored in the hot word model of the hot word detector.

[0020] When the hot word detector determines that the audio data 110 corresponding to the first utterance 106 includes the hot word 112, the device 10 may trigger a wake-up process to begin speech recognition on the audio data 110 corresponding to the first utterance 106. Here, because the device 10 detects that the audio data 110 includes the hot word 112, the device 10 determines that the first utterance 106 is intended for the device 10 (i.e., a digital assistant). The device 10 may utilize remote resources (e.g., a remote system 130) to perform various functions related to speech processing and / or synthesis playback communication. In some implementations, the ASR system 118 is located on the remote system 130 in addition to or instead of the device 10. When the hot word detector triggers the device 10 to wake up in response to detecting the hot word 112 in the first utterance 106, the user device 10 may transmit the acoustic frames 110 corresponding to the first utterance 106 to the remote system 130 via the network 120. Here, the device 10 may transmit a portion of the audio data 110 including the hot word 112 for the remote system 130 to verify the presence of the hot word 112. Alternatively, the device 10 may transmit only a portion of the audio data 110 corresponding to a portion of the first utterance 106 after the hot word 112 to the remote system 130, which executes the ASR system 118 to perform speech recognition and returns a transcription 120 of the audio data 110 to the device 10. The device 10 may further include an NLU module (not shown) that performs semantic interpretation on the transcription 120 of the first utterance 106 to identify commands directed to the device 10. In particular, the NLU module identifies words in the first utterance 106 identified by the ASR system 118 and performs semantic interpretation to identify any voice commands in the first utterance 106.The NLU module of the device 10 (and / or the remote system 130) may identify the words "What's the weather forecast for tomorrow?" as a command specifying an action for the digital assistant 105 to take (i.e., check the weather forecast for the user's location).

[0021] As shown, the user device 10 generates a synthesized speech 123a ("It's 80 degrees and cloudy with possible thunderstorms") in response to the first utterance 106 during time 2. While the user device 10 audibly outputs the synthesized speech 123a, or shortly thereafter, the user 102 speaks a second utterance 146 ("How about Saturday?") during time 3. Notably, the device 10 is operating in a continuous conversational mode such that the ASR model 200 continues to accept and perform speech recognition processing on any subsequent speech if the user 10 speaks a subsequent query in a natural manner without requiring the user 10 to repeatedly speak the hot word to invoke the ASR model 200. Continuing with the example, while receiving the acoustic frames 110 corresponding to the second utterance 146 as the user 102 speaks, the RNN-T model 200 performs speech recognition on the acoustic frames 110 to produce a speech recognition result 120 corresponding to the second utterance 106 (i.e., "How's Saturday going?"). In the illustrated example, when the RNN-T model 200 performs streaming speech recognition on the acoustic frames 100, the IQ joint network 230 detects that the acoustic frames 110 corresponding to the second utterance 146 indicate an intended query, rather than an unintended query (i.e., not directed at the user device 10), to which the user 102 expects the user device 10 to respond. Based on the IQ joint network 230 detecting that the second utterance includes a query directed to the digital assistant 105, at time 4, the digital assistant 105 generates and audibly outputs synthesized speech 123b ("Saturday will be sunny with temperatures in the mid to upper 60s") in response to the second utterance 146.

[0022] Continuing with the example with reference to FIG. 1B , at time 5, during and / or shortly after the audible output of the synthetic speech 123b, while the ASR model 200 is still accepting subsequent speech, the user 102 begins speaking a third utterance 148 ("Let's plan a hike on Saturday instead"), and the RNN-T model 200 performs speech recognition on the acoustic data 110 corresponding to the third utterance 148 to generate the partial speech recognition result 120. While the user 102 is still speaking, or after the user 102 has finished speaking the third utterance 148, the IQ joint network 230 jointly detect that the acoustic frames 110 corresponding to the third utterance 148 indicate an unintended query, rather than an intended query, while the ASR model 200 is producing the partial speech recognition result 120. For example, as shown, the user 102 may speak the third utterance 148 to another user 103 in the environment. The third utterance 148 may be generated by the RNN-T model 200. The third utterance 148 may include audio from background audio, such as audio emanating from a media device. Here, because the IQ joint network 230 determines that the third utterance 148 is not intended for the device 10, the device 10 (i.e., the RNN-T model 200) may stop performing speech recognition on any remaining acoustic data 110, discard any partial speech recognition results 120, and / or suppress any action by the digital assistant 105. That is, continuing the example, the IP joint network 230 may determine that the third utterance 148 is not intended for the device 10 without requiring the ASR model 200 to complete performing speech recognition on the entire third utterance 148, as a hard microphone end event may occur before the third utterance 148 is completed.

[0023] FIG. 2 is a schematic diagram of an exemplary RNN-T model 200 integrating an IQ joint network 230 to detect an intended query in a spoken utterance, showing the speaker directing the digital assistant 105 to perform an action / command associated with the spoken utterance. In particular, FIG. 1A shows the user 102 speaking a first utterance 106 that includes the hot word 112 "ok, computer," but the RNN-T model 200 integrating an IQ joint network 230 may detect an intended query in the spoken utterance where the hot word 112 is not present. Examples of unintended queries include, but are not limited to, conversations between one or more users in an environment, synthetic voices generated by different devices in the environment, and users speaking out loud (e.g., talking to themselves). These utterances may trigger a conventional ASR system to process the query, where the digital assistant may interrupt the user in response to the utterance, which may increase the user's frustration.

[0024] As shown, the RNN-T model 200 includes an encoder network 220, a predictor / decoder network 300, an ASR joint network 240 (i.e., wordpiece joint network 240), and a final softmax output layer 250. The encoder network 220 (e.g., an audio encoder), which is roughly similar to an acoustic model (AM) in a conventional ASR system, receives a sequence of feature vectors x=(x1, x2, ..., x t ) 222 is received,

number

[0025] In the illustrated example, the prediction / decoder network 300, similar to a language model (LM), converts the sequence of non-blank symbols 252 output so far, y 0 , ..., y 0 , into a label history representation 350 (also commonly referred to as a dense or language representation) that represents a probability distribution of whether the current time step corresponds to a possible output label of the audio data 110 or not, by a softmax layer 250. u-1 It includes an LSTM-based prediction network that processes

[0026] FIG. 3 shows a sequence of non-empty symbols y , which is limited to the N previous non-empty symbols 252 a-n output by a final softmax layer 250. u-n ,…,y u-1 1 illustrates a prediction network 300 for an RNN-T model 200 receiving N. In some examples, N is equal to 2. In other examples, N is equal to 5, although this disclosure is non-limiting and N may be equal to any integer. A sequence of non-blank symbols 252a-n may represent a partial speech recognition result 120a, 120b (FIG. 1). In some implementations, the prediction network 300 includes a multi-head attention mechanism 302 that shares a shared embedding matrix 304 across each head 302A-H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads may be used in the multi-head attention mechanism 302. Notably, the multi-head attention mechanism 302 significantly improves performance while minimizing the increase in model size. As explained in more detail below, each head 302A-H contains its own row of position vectors 308, and rather than increasing the model size by concatenating the outputs 318A-H from all heads, the outputs 318A-H are instead averaged by a head averaging module 322.

[0027] Referring to the first head 302A of the multi-head attention mechanism 302, the head 302A uses a shared embedding matrix 304 to estimate the sequence of non-empty symbols y received as input at corresponding time steps from the multiple time steps.ui-n ,…,y ui-1 For each non-whitespace symbol 301 in

number

number

[0028] A sequence of non-blank symbols 252a-n, y u-n ,…,y u-1 The corresponding embedding generated by the shared embedding matrix 304 for each non-blank symbol 301 in is the same for all of the heads 302A-H of the multi-head attention mechanism 302, but each head 302A-H defines a different set / row of position vectors 308. For example, the first head 302A defines a position vector PV Aa~An The second head 302B defines the row 308Aa to 308An of the position vector PV Ba~Bn308 of Ba~Bn , ..., define different rows, and the Hth head 302H is a position vector PV Ha~Hn 308 of Ha~Hn Define another different line of

[0029] For each non-blank symbol in the received sequence of non-blank symbols 252a-n, the first head 302A, via the weight layer 310, weights the corresponding embedding 306 in proportion to the similarity between the corresponding embedding and the respective position vector PV 308 assigned thereto. In some examples, the similarity may include cosine similarity (e.g., cosine distance). In the illustrated example, the weight layer 310 outputs a sequence of weighted embeddings 312, 312Aa-An, each associated with the corresponding embedding 306 weighted in proportion to the respective position vector PV 308 assigned thereto. Stated differently, the weighted embedding 312 output by the weight layer 310 for each embedding 306 may correspond to a dot product between the embedding 306 and the respective position vector PV 308. The weighted embedding 312 may be interpreted as weighting over the embedding in proportion to how similar the embedding is to the positioning associated with the respective position vector PV 308. To increase computational speed, the prediction network 300 includes a non-recurrent layer, and therefore the sequence of weighted embeddings 312Aa-An are not concatenated but instead averaged by a weighted average module 316 to produce a weighted average 318A of the weighted embeddings 312Aa-An as output from the first head 302A, expressed as follows:

number

[0030] The operations described above with respect to the first head 302A are similarly performed for each of the other heads 302B-H of the multi-head attention mechanism 302. Due to the different set of positioned vectors PV 308 defined by each head 302, the weight layer 310 outputs a sequence of weighted embeddings 312Ba-Bn, 312Ha-Hn for each of the other heads 302B-H that is different from the sequence of weighted embeddings 312Aa-Aa for the first head 302A. The weighted average module 316 then generates a respective weighted average 318B-H of the corresponding weighted embeddings 312 of the sequence of non-blank symbols as output from each of the other corresponding heads 302B-H.

[0031] In the illustrated example, the prediction network 300 includes a head averaging module 322 that averages weighted averages 318A-H output from corresponding heads 302A-H. A projection layer 326 with SWISH may receive as an input an output 324 from the head averaging module 322 corresponding to the average of the weighted averages 318A-H and generate as an output a projection output 328. A final layer normalization 330 normalizes the embedding vector P from multiple time steps to a single embedding vector P at a corresponding time step. u The prediction network 300 may normalize the projection output 328 to provide a label history representation 350 (e.g., a label history representation). At each of multiple time steps following the initial time step, the prediction network 300 computes a single embedding vector P u Generates only 350.

[0032] In some configurations, the prediction network 300 does not implement a multi-head attention mechanism 302 and only performs the operations described above for the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is simply passed through the projection layer 326 and layer normalization 330 to produce a single embedding vector P u 350 is offered.

[0033] Referring back to FIG. 2, the ASR joint network 240 extracts a single embedding vector P u 350, and the audio encoding 224 from the encoder 220. The word-piece joint network 240 calculates a probability distribution over the possible speech recognition hypotheses at the corresponding time step.

number

number

[0034] The softmax layer 250 calculates the next output symbol y predicted by the RNN-T model 200 at the corresponding output step.u As 252, the distribution y * Any technique can be used to select the output label / symbol with the highest probability in 242. In this way, the RNN-T model 200 does not make any conditional independence assumptions. Instead, the RNN-T model 200 predicts each symbol conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model 200 assumes that the output symbols are independent of the future audio data 110 which allows the RNN-T model 200 to be used in a streaming fashion. In some examples, the softmax layer 250 is composed of a unified word piece or grapheme set that is generated using all unique word pieces or graphemes of multiple training datasets.

[0035] The IQ joint network 230 uses the high-level feature representation 224 output by the encoder network 220 and the single embedding vector P u 350, generates a probability distribution for which the corresponding time step corresponds to an intended query or an unintended query. In other words, the IQ joint network 230 generates a probability distribution for which the corresponding time step corresponds to an intended query or an unintended query. <intended>, for unintended queries <unintended>The method may output a set of values ​​indicating the likelihood of occurrence of each of a given set of output labels including: The set of values ​​may be a vector and may indicate a probability distribution over the set of output labels. The probability distribution providing the conditional probability of intended queries and the conditional probability of unintended queries may be determined using the following mathematical formula:

number

number

number

number

[0036] 1A, 1B, and 2, to ensure that the RNN-T 200 produces accurate speech recognition results, the training system 130 (FIGS. 1A and 1B) trains the RNN-T 200 (including the IQ joint network 230) using a hybrid autoregressive transducer (HAT) factorization in a two-stage training process to generate a set of possible subword units and two intended query tokens (e.g., <intended>and <unintended>) in some examples. In some examples, the possible subword units include word pieces (e.g., 4,096 possible word pieces). In other examples, the RNN-T 200 is trained to predict graphemes, phonemes, whole words, or other types of subword units. During the first stage, the training system 130 trains the encoder network 220, the prediction network 300, and the ASR joint network 240 on a speech recognition task to perform speech recognition. Specifically, the training system 130 performs the first training stage by optimizing the encoder network 220, the ASR joint network 240, and the prediction network 300 using a regular label sequence of word pieces. During the second stage, the training system 130 keeps the parameters of the encoder network 220, the prediction network 300, and the ASR joint network 240 fixed, while initializing and fine-tuning (e.g., training) the IQ joint network 230 to learn how to detect whether an input utterance contains an intended query or not. That is, the training system 130 trains the IQ joint network 230 to learn how to predict intended and unintended positions in an utterance. The training system 130 may also indicate whether a transcribed utterance contains a query intended for a digital assistant. <intended>and <unintended>To predict the distribution of tokens, we use the extended labels (e.g., <intended>and <unintended>) to train the IQ joint network 230. For example, the training data generator 140 may generate a set of semantic commands for the digital assistant, followed by a set of training utterances, each of which may include a set of training utterances, such as ... <intended>Tokens can be inserted after word pieces that do not contain semantic commands and / or when a silent pause is determined. <unintended>Tokens can be inserted. Thus, the training system 130 uses regular label sequences of word pieces in the first stage, while the training data generator 140 generates extended label sequences of word pieces in the second stage to train the IQ joint network 230.

[0037] For short-form utterances that contain a single voice query, the training data generator 140 appends <eos>Add tokens, e.g., for silences determined using forced alignment. <pause>Although short forms of speech can be used to model the end of speech events and normal short pauses, they may not cover the wider range of possible disfluencies. <eos>If a token is added to the end of each utterance, the disfluency detection joint network 210 may be unable to detect any additional speech end events after determining the first speech end event in an utterance, which may cause disfluency detection problems for longer utterances or utterances with multiple disfluencies. <eos>It can learn to stop emitting tokens.

[0038] Referring back to FIG. 2, the feature vector x222 input to the encoder network 220 may include 80-dimensional log-mel filter bank features formed by stacking three 25 millisecond (ms) acoustic frames with a 10 ms shift, and may be downsampled to a 30 ms frame rate. In some examples, the encoder network 220 includes twelve 512-dimensional conformer layers. The conformer layers include a stack of causal convolutions with a kernel size of 15, and a left context attention layer with eight head self-attentions. Although the described encoder network 220 includes a stack of multi-head attention layers / blocks with self-attention including conformer layers / blocks (e.g., 12 conformer blocks), the disclosure is not so limited. For example, the encoder network 220 may include a stack of transformer layers / or any other type of multi-head attention layer / block. The encoder network 220 may include a series of multi-head self-attention layers, depthwise convolutional layers, and feedforward layers. Alternatively, the encoder network 220 may include multiple long short-term memory (LSTM) layers instead of multi-head attention layers / blocks.

[0039] The prediction network 300 may include an LTSM-based network with an embedding dimension of 320. The dimension D of the probability distribution representation 242 output by the ASR joint network 240 ジョイント may be set to 640. In some examples, the ASR joint network 240 includes hidden units. Additionally or alternatively, the wordpiece joint network 240 does not include a fully connected (FC) layer. The IQ joint network 230 may include an FC layer with 640 hidden units. Alternatively, the prediction network 300 may include a stack of transformer or conformer blocks (or other types of multi-head attention blocks). The prediction network 300 may also be an embedding lookup table (e.g., a V2 embedding lookup table) that improves latency by outputting a lookup sparse embedding instead of generating a hidden representation. In some implementations, the prediction network 300 is a stateless prediction network.

[0040] The ASR joint network 240 and the prediction network 300 may collectively form an RNN-T decoder of the RNN-T model 200. In some implementations, to further reduce the size of the RNN-T decoder, i.e., the prediction network 300 and the ASR joint network 240, parameter tying between the prediction network 300 and the ASR joint network 240 is applied. Specifically, the vocabulary size |V| and the embedding dimension d e , the shared embedding matrix 304 in the prediction network 300 is

number

number

[0041] FIG. 4A illustrates an example transcribed training utterance 400. FIG. 4B illustrates an example semantically annotated transcribed training utterance 420 generated by an NLU model for the transcribed training utterance 400 of FIG. 4A. FIG. 4C illustrates an example annotated transcribed training utterance 450 generated by the training data generator 140 for the transcribed training utterance 400 of FIG. 4A. As shown in FIG. 4A, for each utterance, the training data generator 140 determines silence portions 402, 402a-n, for example, using forced alignment. In FIG. 4B, the NLU model analyzes the transcribed training utterance 400 to determine which of the silence portions 402 constitute slots 422, 422a-n that represent semantic commands (e.g., change, wake up, timer, at 8 o'clock), and labels that identify each semantic command as a slot 422. In FIG. 4C, the training data generator 140 generates a respective silent portion 402 (including each slot 422) during which a semantic command is completed (e.g., silent pause 402c). <intended>Tokens (e.g., token 452a) are labeled with a silent pause that does not represent the completion of a common semantic command (e.g., silent pauses 402a, 402b, 402d, and 402e), respectively. <unintended>Labeling the tokens (e.g., tokens 454a, 454b, 454c, 454d) and the final silent pause (e.g., silent pause 402f) <intended>A token (eg, token 452b) is labeled.

[0042] 5 is a flowchart of an exemplary operation arrangement of a computer-implemented method 500 for detecting an intended query and performing speech recognition. At operation 502, the method 500 includes receiving audio data 110, 222 corresponding to the spoken utterance 106 as input to a speech recognition model 200. The method 500 also includes performing speech recognition on the audio data 110, 222 at each of a plurality of time steps using the speech recognition model 200 via operations 504 and 506. At operation 504, the method 500 includes encoding the audio data 110, 222 corresponding to the spoken utterance 106 into a corresponding audio encoding 224 using an audio encoder 220.

[0043] At operation 506, the method 500 also includes using the speech recognition joint network 240 to decode the corresponding audio encoding 224 encoded by the audio encoder 220 at the corresponding time step into a probability distribution 242 over the possible output labels of the spoken utterance 106 at the corresponding time step. At each of the multiple time steps, the method 500 also includes, at operation 508, using an intended query (IQ) joint network 230 configured to receive a label history representation 350 associated with the sequence of non-blank symbols 252 output by the final softmax layer 250 to determine an intended query decision 212 indicating whether the spoken utterance 106 includes a query directed to the digital assistant interface 105.

[0044] 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described herein. Computing device 600 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0045] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 that connects to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 that connects to a low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may be mounted on a common motherboard or may exist in other manners as desired. Processor 610 (e.g., data processing hardware 12, 132 of FIGS. 1A and 1B) may process instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630 to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 680 coupled to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and multiple types of memories. Also, multiple computing devices 600 may be connected, each performing some of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0046] The memory 620 (i.e., memory hardware 14, 134 in FIG. 1A and FIG. 1B) stores information non-transiently within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0047] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 620, the storage device 630, or a memory on the processor 610.

[0048] The high-speed controller 640 manages more bandwidth-intensive operations of the computing device 600, and the low-speed controller 660 manages less bandwidth-intensive operations. Such an allocation of roles is merely an example. In some implementations, the high-speed controller 640 is coupled to a memory 620, a display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and may be coupled, via a network adapter or the like, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0049] The computing device 600 can be implemented in many different forms, as shown in the figure: for example, it can be implemented as a standard server 600a, or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0050] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0051] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0052] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0053] The processes and logic flows described herein may be performed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to operate on input data and generate output to perform functions. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose processors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for performing instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all types of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0054] To provide for interaction with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to a user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with a user, for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, spoken language, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0055] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.< / intended> < / unintended> < / intended> < / eos> < / eos> < / pause> < / eos> < / unintended> < / intended> < / unintended> < / intended> < / unintended> < / intended> < / unintended> < / intended> < / unintended> < / unintended> < / intended> < / unintended> < / intended> < / unintended> < / intended>

Claims

1. A computer-implemented method (500), when executed on data processing hardware (610), causing the data processing hardware (610) to: receiving audio data (110, 222) corresponding to a spoken utterance (106, 146, 148) as input to a speech recognition model (200); Using the speech recognition model (200), at each of a plurality of time steps: encoding the audio data (110, 222) corresponding to the spoken utterance (106, 146, 148) into a corresponding audio encoding (224) using an audio encoder (220); performing speech recognition on the audio data (110, 222) by decoding, using a speech recognition joint network (240), the corresponding audio encoding (224) encoded by the audio encoder (220) at the corresponding time step into a probability distribution (242) over possible output labels for the spoken utterance (106, 146, 148) at the corresponding time step; and determining, at each of the plurality of time steps, an intended query (IQ) decision (212) indicating whether the spoken utterance (106, 146, 148) includes a query directed to (105) using an intended query (IQ) joint network (230) configured to receive a label history representation (350) associated with a sequence of non-empty symbols (252) output by a final softmax layer (250).

2. the speech recognition model (200) includes the audio encoder (220), the speech recognition joint network (240), and a prediction network (230), the prediction network (230) configured to receive the sequence of non-blank symbols (252) output by the final softmax layer (250) and generate the label history representation (350) at each of the plurality of time steps; The speech recognition model (200) is trained during a first training phase by optimizing the audio encoder (220), the speech recognition joint network (240), and the prediction network (230) using a common label sequence of word pieces; 2. The method of claim 1, wherein the IQ joint network is initialized with the speech recognition joint network by freezing the audio encoder and the prediction network and fine-tuning the IQ joint network with an extended label sequence of both word pieces and IQ tokens during a second training phase to teach the IQ joint network to learn how to predict a distribution of IQ tokens that indicate whether an input utterance contains a query directed to the digital assistant interface.

3. generating said label history representation (350) for said sequence of corresponding non-blank symbols (252), For each non-blank symbol (252) in the sequence of non-blank symbols (252) received as input at each of the plurality of time steps, generating, by the prediction network (230), embeddings (306) of the corresponding non-blank symbols (252) using the shared embedding matrix (304); assigning, by said prediction network (230), each position vector (308) to said corresponding non-blank symbol (252); weighting, by the predictive network (230), the embeddings (306) in proportion to the similarity between the embeddings (306) and the respective position vectors (308); generating, as output from the prediction network (230), a single embedding vector (350) for the corresponding time step, the single embedding vector (350) being based on a weighted average of the weighted embeddings (318), the single embedding vector (350) including the label history representation (350); The method of claim 2 , comprising:

4. 4. The method of claim 3, wherein the prediction network includes a multi-head attention mechanism that shares the shared embedding matrix across each head of the multi-head attention mechanism.

5. The method (500) according to any one of claims 1 to 4, wherein the audio data (110, 222) corresponding to a spoken utterance (106, 146, 148) is received during a current dialog session between a user (102) and the digital assistant interface (105).

6. The method (500) of any of claims 1 to 5, wherein the output labels include wordpieces, words, phonemes, or graphemes.

7. The audio encoder (220): Multiple one-way long short-term memory (LSTM) layers, Multiple conformer layers, or The method (500) of any of claims 1 to 6, further comprising a causal encoder comprising one of a plurality of transformer layers.

8. The method (500) of any of claims 1 to 7, wherein the speech recognition model (200) is trained using hybrid autoregressive transducer factorization.

9. The method (500) of any one of claims 1 to 8, wherein the operations further include providing a response to the received spoken utterance (106, 146, 148) when the intended query determination (212) indicates that the spoken utterance (106, 146, 148) includes a query directed to the digital assistant interface (105).

10. The method (500) of any one of claims 1 to 8, wherein the operation further includes discarding the received spoken utterance (106, 146, 148) when the intended query determination (212) indicates that the spoken utterance (106, 146, 148) does not include a query directed to the digital assistant interface (105).

11. Data processing hardware (610); and memory hardware (620) in communication with the data processing hardware (610), the memory hardware (620), when executed on the data processing hardware (610), causing the data processing hardware (610) to: receiving audio data (110, 222) corresponding to a spoken utterance (106, 146, 148) as input to a speech recognition model (200); Using the speech recognition model (200), at each of a plurality of time steps: encoding the audio data (110, 222) corresponding to the spoken utterance (106, 146, 148) into a corresponding audio encoding (224) using an audio encoder (220); decoding the corresponding audio encoding (224) encoded by the audio encoder (220) at the corresponding time step into a probability distribution (242) over possible output labels for the spoken utterance (106, 146, 148) at the corresponding time step using a speech recognition joint network (240); performing speech recognition on the audio data (110, 222) by At each of the plurality of time steps, using an intended query (IQ) joint network configured to receive a label history representation (350) associated with a sequence of non-blank symbols (252) output by a final softmax layer (250), determine an intended query decision (212) indicating whether the spoken utterance (106, 146, 148) includes a query directed to a digital assistant interface (105); A system (100) storing instructions causing the system to perform operations including:

12. the speech recognition model (200) includes the audio encoder (220), the speech recognition joint network (240), and a prediction network (230), the prediction network (230) configured to receive the sequence of non-blank symbols (252) output by the final softmax layer (250) and generate the label history representation (350) at each of the plurality of time steps; The speech recognition model (200) is trained during a first training phase by optimizing the audio encoder (220), the speech recognition joint network (240), and the prediction network (230) using a common label sequence of word pieces; 12. The system (100) of claim 11, wherein the IQ joint network (230) is initialized with the speech recognition joint network (240) by freezing the audio encoder (220) and the prediction network (230) and fine-tuning the IQ joint network (230) with an extended label sequence of both word pieces and IQ tokens (452, 454) during a second training phase to teach the IQ joint network (230) to learn how to predict a distribution of IQ tokens that indicate whether an input utterance contains a query directed to the digital assistant interface (105).

13. generating said label history representation (350) for said sequence of corresponding non-blank symbols (252), For each non-blank symbol (252) in the sequence of non-blank symbols (252) received as input at each of the plurality of time steps, generating, by the prediction network (230), embeddings (306) of the corresponding non-blank symbols (252) using the shared embedding matrix (304); assigning, by said prediction network (230), each position vector (308) to said corresponding non-blank symbol (252); weighting, by the predictive network (230), the embeddings (306) in proportion to the similarity between the embeddings (306) and the respective position vectors (308); generating, as output from the prediction network (230), a single embedding vector (350) for the corresponding time step, the single embedding vector (350) being based on a weighted average of the weighted embeddings (312), the single embedding vector (350) including the label history representation (350); The system (100) of claim 12, comprising:

14. 14. The system of claim 13, wherein the prediction network includes a multi-head attention mechanism that shares the shared embedding matrix across each head of the multi-head attention mechanism.

15. The system (100) according to any one of claims 11 to 14, wherein the audio data (110, 222) corresponding to a spoken utterance (106, 146, 148) is received during a current dialog session between a user (102) and the digital assistant interface (105).

16. The system (100) of any of claims 11 to 15, wherein the output labels include wordpieces, words, phonemes, or graphemes.

17. The audio encoder (220): Multiple one-way long short-term memory (LSTM) layers, Multiple conformer layers, or The system (100) of any of claims 11 to 16, further comprising a causal encoder comprising one of a plurality of transformer layers.

18. The system (100) of any of claims 11 to 17, wherein the speech recognition model (200) is trained using hybrid autoregressive transducer factorization.

19. The system (100) of any one of claims 11 to 18, wherein the operation further includes providing a response to the received spoken utterance (106, 146, 148) when the intended query determination (212) indicates that the spoken utterance (106, 146, 148) includes a query directed to the digital assistant interface (105).

20. The system (100) of any one of claims 11 to 18, wherein the operations further include discarding the received spoken utterance (106, 146, 148) when the intended query determination (212) indicates that the spoken utterance (106, 146, 148) does not include a query directed to the digital assistant interface (105).

Images