Intended Query Detection Using E2E Modeling for Continuous Conversations

Abstract

The method (500) includes receiving audio data (110) corresponding to a spoken utterance (106) as an input to a speech recognition model (200). The method also includes performing speech recognition on the audio data by encoding the audio data corresponding to the spoken utterance into a corresponding audio encoding (224) using an audio encoder (220) at each of a plurality of time steps and decoding the corresponding audio encoding into a probability distribution (242) over possible output labels using a speech recognition joint network (240). At each of the plurality of time steps, the method also includes determining an intended query decision (212) indicating whether the spoken utterance includes a query directed to the digital assistant (105) using an intended query (IQ) joint network (230) configured to receive a label history representation (350) associated with a sequence of non-blank symbols (252) output by a final softmax layer (250).

Description

[Technical Field]

[0001] The present disclosure relates to intended query detection using end-to-end (E2E) modeling for sustained conversations. [Background technology]

[0002] In a voice-enabled environment, a user simply speaks a query or command out loud, and the digital assistant addresses and responds to the query and / or executes the command. Voice-enabled environments (e.g., home, work, school, etc.) can be implemented using a network of connected microphone devices located in various rooms or environmental areas. Through such a network of microphones, a user has the ability to verbally query a digital assistant from essentially anywhere in the environment, without having to have a computer or other device in front of or nearby. For example, while cooking in the kitchen, a user may activate a digital assistant using a hotword such as "OK, computer, set the timer for 20 minutes." In response, the digital assistant confirms that the timer is set (in the form of a synthesized voice output) and alerts the user (e.g., in the form of an alarm or other audible alert from an audio speaker) when the timer reaches 20 minutes. Often, a user may issue subsequent queries to the digital assistant. However, having to repeat a hotword to address the digital assistant places a cognitive burden on the user and interrupts the flow of a continuous conversation. Summary of the Invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on the data processing hardware, causes the data processing hardware to perform operations including receiving audio data corresponding to a spoken utterance as input to a speech recognition model. The operations also include performing speech recognition on the audio data by, at each of a plurality of time steps, using an audio encoder to encode the audio data corresponding to the spoken utterance into a corresponding audio encoding, and using a speech recognition joint network to decode the corresponding audio encoding encoded by the audio encoder at the corresponding time step into a probability distribution over possible output labels for the spoken utterance at the corresponding time step. At each of the plurality of time steps, the operations also include determining an intended query decision indicating whether the spoken utterance includes a query directed to a digital assistant interface, using an intended query (IQ) joint network configured to receive a label history representation associated with the sequence of non-blank symbols output by the final softmax layer.

[0004] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the speech recognition model includes an audio encoder, a speech recognition joint network, and a prediction network. The prediction network is configured to receive the sequence of non-blank symbols output by the final softmax layer and generate a label history representation at each of a plurality of time steps. In these embodiments, the speech recognition model is trained during a first training stage by optimizing the audio encoder, the speech recognition joint network, and the prediction network using a regular label sequence of word pieces, and the IQ joint network is initialized by the joint network during a second training stage by freezing the audio encoder and the prediction network and fine-tuning the IQ joint network with an extended label sequence of both word pieces and IQ tokens to teach the IQ joint network how to predict a distribution of IQ tokens that indicate whether an input utterance includes a query directed to a digital assistant interface. In these embodiments, generating a label history representation for a corresponding sequence of non-blank symbols includes, for each non-blank symbol in the sequence of non-blank symbols received as input at each of a plurality of time steps, generating, by a prediction network, an embedding of the corresponding non-blank symbol using a shared embedding matrix; assigning, by the prediction network, a respective position vector to the corresponding non-blank symbol; and weighting, by the prediction network, the embedding proportional to a similarity between the embedding and the respective position vector. Here, the operation also includes generating, as output from the prediction network, a single embedding vector based on a weighted average of the weighted embeddings at the corresponding time step, the single embedding vector including the label history representation. Here, the prediction network may include a multi-head attention mechanism that shares the shared embedding matrix across each head of the multi-head attention mechanism.

[0005] In some examples, audio data corresponding to the spoken utterance is received during a current dialog session between the user and the digital assistant interface. In some implementations, the output label includes a wordpiece, a word, a phoneme, or a grapheme. In some examples, the audio encoder includes a causal encoder including one of a plurality of one-way long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. In some implementations, the speech recognition model is trained using hybrid autoregressive transducer factorization. In some examples, the operations further include providing a response to the received spoken utterance when the intended query determination indicates that the spoken utterance includes a query directed to the digital assistant interface. Alternatively, the operations further include discarding the received spoken utterance when the intended query determination indicates that the spoken utterance does not include a query directed to the digital assistant interface.

[0006] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including receiving audio data corresponding to a spoken utterance as input to a speech recognition model. The operations also include performing speech recognition on the audio data using the speech recognition model by, at each of a plurality of time steps, encoding the audio data corresponding to the spoken utterance into a corresponding audio encoding using an audio encoder and decoding, at the corresponding time step, the corresponding audio encoding encoded by the audio encoder into a probability distribution over possible output labels for the spoken utterance at the corresponding time step using a speech recognition joint network. At each of the plurality of time steps, the operations also include determining an intended query decision indicating whether the spoken utterance includes a query directed to a digital assistant interface using an intended query (IQ) joint network configured to receive a label history representation associated with the sequence of non-blank symbols output by the final softmax layer.

[0007] This aspect may include one or more of the following optional features. In some implementations, the speech recognition model includes an audio encoder, a speech recognition joint network, and a prediction network. The prediction network is configured to receive the sequence of non-blank symbols output by the final softmax layer and generate a label history representation at each of a plurality of time steps. In these implementations, the speech recognition model is trained during a first training stage by optimizing the audio encoder, the speech recognition joint network, and the prediction network using a regular label sequence of word pieces, and the IQ joint network is initialized by the joint network during a second training stage by freezing the audio encoder and the prediction network and fine-tuning the IQ joint network with an extended label sequence of both word pieces and IQ tokens to teach the IQ joint network how to predict a distribution of IQ tokens that indicate whether an input utterance includes a query directed to a digital assistant interface. In these embodiments, generating a label history representation for a corresponding sequence of non-blank symbols includes, for each non-blank symbol in the sequence of non-blank symbols received as input at each of a plurality of time steps, generating, by a prediction network, an embedding of the corresponding non-blank symbol using a shared embedding matrix; assigning, by the prediction network, a respective position vector to the corresponding non-blank symbol; and weighting, by the prediction network, the embedding proportional to a similarity between the embedding and the respective position vector. Here, the operation also includes generating, as output from the prediction network, a single embedding vector based on a weighted average of the weighted embeddings at the corresponding time step, the single embedding vector including the label history representation. Here, the prediction network may include a multi-head attention mechanism that shares the shared embedding matrix across each head of the multi-head attention mechanism.

[0008] In some examples, audio data corresponding to the spoken utterance is received during a current dialog session between the user and the digital assistant interface. In some implementations, the output label includes a wordpiece, a word, a phoneme, or a grapheme. In some examples, the audio encoder includes a causal encoder including one of a plurality of one-way long short-term memory (LSTM) layers, a plurality of conformer layers, or a plurality of transformer layers. In some implementations, the speech recognition model is trained using hybrid autoregressive transducer factorization. In some examples, the operations further include providing a response to the received spoken utterance when the intended query determination indicates that the spoken utterance includes a query directed to the digital assistant interface. Alternatively, the operations further include discarding the received spoken utterance when the intended query determination indicates that the spoken utterance does not include a query directed to the digital assistant interface.

[0009] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0010] [Figure 1A] FIG. 1 is a schematic diagram of an example system including an automatic speech recognition (ASR) system including an intended query detector for transcribing spoken utterances. [Figure 1B] FIG. 1 is a schematic diagram of an example system including an automatic speech recognition (ASR) system including an intended query detector for transcribing spoken utterances. [Figure 2] FIG. 1 is a schematic diagram of an exemplary ASR system integrating an intended query detector. [Figure 3] FIG. 3 is a schematic diagram of an exemplary linked and reduced prediction network of the ASR system of FIG. 2. [Figure 4A] 1 shows an example long transcribed training utterance. [Figure 4B] 4B shows exemplary annotated transcribed training utterances for the long transcribed training utterances of FIG. 4A. [Figure 5] 1 is a flowchart of an exemplary arrangement of operations of a computer-implemented method for executing an intended query detection model in an ASR system. [Figure 6] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0011] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0012] A user's interaction with an Assistant-enabled device is designed to be primarily, but not exclusively, through voice input. As a result, the Assistant-enabled device needs some way to identify when any given utterance in the surrounding environment is directed at the device as opposed to being directed at an individual in the environment or originating from a non-human source (e.g., a television or music player). One way to accomplish this is through the use of hot words, reserved by agreement among users in the environment as a predetermined word or words to be spoken to attract the device's attention. In the exemplary environment, the hot word used to attract the Assistant's attention is the words "OK, computer." As a result, whenever the words "OK, computer" are spoken, they are picked up by the microphone and transmitted to a hot word detector, which performs speech understanding techniques to determine if the hot word was spoken and, if so, waits for the next command or query. Thus, utterances directed to an Assistant-enabled device take the general form [hot word][query], where the "hot word" in this example is "OK computer" and the "query" can be any question, command, declaration, or other request that can be voice-recognized, analyzed, and acted upon by the system, either alone or in conjunction with a server over a network.

[0013] When a user continues a conversation with an Assistant-enabled device, such as a mobile phone or smart speaker, the user's interaction with the phone or speaker can be awkward. The user may say, "OK, computer, play the assignments playlist." The phone or speaker may begin playing the first song in the playlist. The user may want to skip to the next song and say, "OK, computer, next." To skip to another song, the user may say, "OK, computer, next" again. To alleviate the need to continually repeat a hotword before speaking a command, the Assistant-enabled device may be configured to recognize / detect that the utterance "next" is directed / targeted at the Assistant-enabled device to skip to the next song. In this example, instead of saying "OK, computer, next," the user can simply say "next" to invoke the Assistant-enabled device to trigger the respective action. In contrast, a user may say to another user in the environment, "What do you think of that movie?" Here, the Assistant-enabled device may recognize / detect that the query is not intended for the Assistant-enabled device and therefore does not respond to the query.

[0014] Embodiments herein are directed to integrating an automatic speech recognition (ASR) system with an intended query detection model configured and trained to detect intended and unintended queries that naturally occur in spoken utterances of natural conversational interactions. The exemplary intended query detection model is built on or integrated with an end-to-end (E2E) ASR model, such as a recurrent neural network-transducer (RNN-T) model. In an exemplary method, the ASR system receives audio data corresponding to the spoken utterance as input to the ASR model. The system uses the ASR model to perform speech recognition on the audio data by, at each of a plurality of time steps, encoding the audio data corresponding to the spoken utterance into a corresponding audio encoding using an audio encoder, and decoding, at the corresponding time step, the corresponding audio encoding encoded by the audio encoder into a probability distribution over output labels of the spoken utterance at the corresponding time step using a speech recognition joint network. At each of the multiple time steps, an intended query (IQ) joint network configured to receive the label history representation associated with the sequence of non-blank symbols output by the final softmax layer determines an intended query decision indicating whether the spoken utterance includes a query directed to the digital assistant.

[0015] 1A and 1B illustrate example systems 100a, 100b that include example audio environments. In an audio environment, a user 104 interacts with a computing device, such as a user device 10, via voice input. The user device 10 (also commonly referred to as device 10) is configured to capture sounds (e.g., streaming audio data) from one or more users 102 within the audio environment. Here, the streaming audio data may refer to utterances 106 spoken by the users 102, which serve as audible queries, commands to the device 10, or audible communications captured by the device 10. The voice-enabled system of the device 10 may address the queries or commands by answering the queries and / or causing the commands to be performed / fulfilled by one or more downstream applications.

[0016] Systems 100a, 100b include a user device 10 running a digital assistant 105 (also commonly referred to as a digital assistant interface) with which a user 102 of the device 10 can interact during a dialog session between the user 102 and the digital assistant 105 by issuing queries that include commands to perform actions. In the illustrated example, the user device 10 corresponds to a portable smart speaker with which the user 102 can interact. However, the user device 10 can include other computing devices, such as, but not limited to, a smartphone, tablet, smart display, desktop / laptop, smartwatch, smart glasses / headset, smart appliance, headphones, or vehicle infotainment device. The user device 10 includes data processing hardware 12 and memory hardware 14 that stores instructions that, when executed on the data processing hardware 12, cause the data processing hardware 12 to perform operations. In some examples, the user device 10 is configured to communicate with a remote system 130 over a network 120. The remote system 130 may include remote resources such as remote data processing hardware 132 (e.g., a remote server or CPU) and / or remote memory hardware 134 (e.g., a remote database or other storage hardware). The user device 10 includes an array of one or more microphones 16 configured to capture sounds, such as voice, directed at the user device 10. The user device 10 may also include or communicate with an audio output device (e.g., a speaker) 18 that may output audio, such as music and / or synthesized voice, from the digital assistant 105. Additionally, the user device 10 may include or communicate with one or more cameras configured to capture images of the environment and output image data.

[0017] In a speech environment, an automatic speech recognition (ASR) system 118 implementing a recurrent neural network-transducer (RNN-T) model 200 including an intended-query (IQ) joint network 230 resides within the user device 10 of the user 102 and / or resides within a remote system 130 in communication with the user device 10 via a network 120. The user device 10 and / or the remote system 130 also includes an audio subsystem 108 configured to receive utterances 106 produced by the user 102 and captured by the microphone 16 and convert the utterances 106 into a corresponding digital format associated with input acoustic frames 110 that can be processed by the ASR system 118. In the illustrated example, the user speaks each utterance 106, and the audio subsystem 108 converts the utterances 106 into corresponding audio data (e.g., acoustic frames) 110 for input to the ASR system 118. The RNN-T model 200 then receives as input acoustic frames 110 corresponding to the utterance 106 and generates / predicts as output corresponding transcriptions 120 of the utterance 106. In the illustrated example, the RNN-T model 200 may perform streaming speech recognition to create partial transcriptions (e.g., streaming speech recognition results) 120 as the user speaks.

[0018] In the example shown in FIG. 1A , a user 102 interacts with a program or application (e.g., a digital assistant application 105) of a user device 10 that uses an ASR system 118. In this example, the user 102 speaks a first utterance 106 (“Ok, computer, what's the weather forecast for today?”) during time 1. The microphone 16 of the AED 104 receives the first utterance 106, and the audio subsystem 108 converts the first utterance 106 into corresponding audio data 110 that corresponds to the first utterance 106. The initial processing of the audio data 110 may include filtering the audio data 110 and converting the audio data 110 from an analog signal to a digital signal. As the audio subsystem 108 processes the audio data 110, the device 10 may store the audio data 110 in a buffer in the memory hardware 14 for further processing. While the audio data 110 is in the buffer, the device 110 may use a hotword detector (not shown) to detect whether the audio data 110 contains hotwords. The hotword detector is configured to identify hotwords contained in the audio data 110 without performing voice recognition on the audio data 110.

[0019] In some implementations, the hot word detector is configured to identify hot words in an early portion of the first utterance 106. In this example, the hot word detector may determine that the first utterance 106, "Ok computer, what's the weather like tomorrow?", includes the hot word 112, "ok computer," if the hot word detector detects acoustic features of the input audio that are characteristic of the hot word 112. The acoustic features may be Mel-Frequency Cepstral Coefficients (MFCCs), which are a representation of the short-term power spectrum of the first utterance 106, or may be Mel-scale filter bank energy of the first utterance 106. For example, the hot word detector may detect that the first utterance 106, "Ok computer, what's the weather like tomorrow?", includes the hot word 112, "ok computer," based on generating MFCCs from the audio data 110 and classifying the MFCCs as including MFCCs similar to MFCCs that are characteristic of the hot word "OK computer" as stored in the hot word model of the hot word detector. As another example, the hot word detector may detect that the first utterance 106, "Ok, computer, what's the weather forecast for tomorrow?", contains the hot word 112, "ok, computer," based on generating mel-scale filter bank energy from the audio data 110 and classifying the mel-scale filter bank energy as containing mel-scale filter bank energy similar to mel-scale filter bank energy that is characteristic of the hot word "ok computer" as stored in the hot word model of the hot word detector.

[0020] When the hotword detector determines that the audio data 110 corresponding to the first utterance 106 includes the hotword 112, the device 10 may trigger a wake-up process to begin speech recognition on the audio data 110 corresponding to the first utterance 106. Here, because the device 10 detects that the audio data 110 includes the hotword 112, the device 10 determines that the first utterance 106 is intended for the device 10 (i.e., a digital assistant). The device 10 may utilize remote resources (e.g., a remote system 130) to perform various functions related to speech processing and / or synthesis playback communication. In some implementations, the ASR system 118 is located on the remote system 130 in addition to or instead of the device 10. When the hotword detector triggers the device 10 to wake up in response to detecting the hotword 112 in the first utterance 106, the user device 10 may transmit the acoustic frames 110 corresponding to the first utterance 106 to the remote system 130 via the network 120. Here, the device 10 may transmit a portion of the audio data 110 including the hotword 112 to the remote system 130 to confirm the presence of the hotword 112. Alternatively, the device 10 may transmit only a portion of the audio data 110 corresponding to a portion of the first utterance 106 after the hotword 112 to the remote system 130, and the remote system 130 executes the ASR system 118 to perform speech recognition and returns a transcription 120 of the audio data 110 to the device 10. The device 10 may further include an NLU module (not shown) that performs semantic interpretation on the transcription 120 of the first utterance 106 to identify commands directed to the device 10. Specifically, the NLU module identifies words in the first utterance 106 identified by the ASR system 118 and performs semantic interpretation to identify any voice commands in the first utterance 106.The NLU module of device 10 (and / or remote system 130) may identify the words "What's the weather forecast for tomorrow?" as a command specifying an action for digital assistant 105 to take (i.e., check the weather forecast for the user's location).

[0021] As shown, user device 10 generates synthesized speech 123a ("It's 80 degrees and cloudy with a chance of thunderstorms") in response to first utterance 106 during time 2. While user device 10 audibly outputs synthesized speech 123a, or shortly thereafter, user 102 speaks second utterance 146 ("How about Saturday?") during time 3. Notably, device 10 is operating in a continuous conversation mode such that ASR model 200 continues to accept and perform speech recognition processing on any subsequent speech if user 10 speaks subsequent queries in a natural manner without requiring user 10 to repeatedly speak the hotword to invoke ASR model 200. Continuing with the example, while receiving the acoustic frames 110 corresponding to the second utterance 146 as the user 102 speaks, the RNN-T model 200 performs speech recognition on the acoustic frames 110 to produce a speech recognition result 120 (i.e., "How's Saturday?") corresponding to the second utterance 106. In the illustrated example, when the RNN-T model 200 performs streaming speech recognition on the acoustic frames 110, the IQ joint network 230 detects that the acoustic frames 110 corresponding to the second utterance 146 indicate an intended query, rather than an unintended query (i.e., not directed at the user device 10), to which the user 102 expects the user device 10 to respond. Based on IQ joint network 230 detecting that the second utterance includes a query directed to digital assistant 105, at time 4, digital assistant 105 generates and audibly outputs synthesized speech 123b ("Saturday will be sunny with temperatures in the mid to upper 60s") in response to second utterance 146.

[0022] Continuing the example with reference to FIG. 1B , at time 5, during and / or shortly after the audible output of synthesized speech 123b, while ASR model 200 is still accepting subsequent speech, user 102 begins speaking third utterance 148 (“Let's plan a hike on Saturday instead”), and RNN-T model 200 performs speech recognition on acoustic data 110 corresponding to third utterance 148 to generate partial speech recognition result 120. While user 102 is still speaking, or after user 102 has finished speaking third utterance 148, IQ joint network 230 jointly detects, while ASR model 200 is producing partial speech recognition result 120, that acoustic frames 110 corresponding to third utterance 148 indicate an unintended query rather than an intended query. For example, as shown, user 102 may speak third utterance 148 to other users 103 in the environment. The third utterance 148 may include speech from background audio, such as speech emanating from a media device. Here, because the IQ joint network 230 determines that the third utterance 148 is not intended for the device 10, the device 10 (i.e., the RNN-T model 200) may stop performing speech recognition on any remaining acoustic data 110, discard any partial speech recognition results 120, and / or suppress any action by the digital assistant 105. That is, continuing the example, the IQ joint network 230 may determine that the third utterance 148 is not intended for the device 10 without requiring the ASR model 200 to complete performing speech recognition on the entire third utterance 148, as a hard microphone end event may occur before the third utterance 148 is completed.

[0023] FIG. 2 is a schematic diagram of an example RNN-T model 200 integrating an IQ joint network 230 to detect intended queries in spoken utterances, illustrating the speaker directing the digital assistant 105 to perform an action / command associated with the spoken utterance. Notably, FIG. 1A illustrates a user 102 uttering a first utterance 106 that includes the hot word 112, "ok, computer," but an RNN-T model 200 integrating an IQ joint network 230 can detect intended queries in spoken utterances where the hot word 112 is absent. Examples of unintended queries include, but are not limited to, conversations between one or more users in an environment, synthesized speech generated by different devices in an environment, and users speaking out loud (e.g., talking to themselves). These utterances can trigger traditional ASR systems to process the query, which can cause the digital assistant to interrupt the user in response to the utterance, increasing user frustration.

[0024] As shown, the RNN-T model 200 includes an encoder network 220, a predictor / decoder network 300, an ASR joint network 240 (i.e., wordpiece joint network 240), and a final softmax output layer 250. The encoder network 220 (e.g., an audio encoder), which is roughly analogous to an acoustic model (AM) in a conventional ASR system, receives a sequence of feature vectors x=(x1, x2, ..., x t )222 is received,

number

[0025] In the illustrated example, the prediction / decoder network 300, similar to a language model (LM), converts the sequence of non-blank symbols 252 output so far, y 0 , ..., y 0 , into a label history representation 350 (also commonly referred to as a dense or language representation) that represents a probability distribution of whether the current time step corresponds to a possible output label of the audio data 110. u-1 It includes an LSTM-based prediction network that processes

[0026] FIG. 3 shows a sequence of non-blank symbols y y , which is limited to N previous non-blank symbols 252 a-n output by a final softmax layer 250 as input. u-n ,…,y u-1 1 illustrates a prediction network 300 for an RNN-T model 200 receiving N. In some examples, N is equal to 2. In other examples, N is equal to 5, although this disclosure is non-limiting and N may be equal to any integer. A sequence of non-blank symbols 252a-n may represent a partial speech recognition result 120a, 120b (FIG. 1). In some implementations, the prediction network 300 includes a multi-head attention mechanism 302 that shares a shared embedding matrix 304 across each head 302A-H of the multi-head attention mechanism. In one example, the multi-head attention mechanism 302 includes four heads. However, any number of heads may be used in the multi-head attention mechanism 302. Notably, the multi-head attention mechanism 302 significantly improves performance while minimizing an increase in model size. As explained in more detail below, each head 302A-H contains its own row of position vectors 308, and rather than increasing the model size by concatenating the outputs 318A-H from all heads, the outputs 318A-H are instead averaged by a head averaging module 322.

[0027] Referring to the first head 302A of the multi-head attention mechanism 302, head 302A uses a shared embedding matrix 304 to generate a sequence of non-empty symbols y received as input at corresponding time steps from multiple time steps.ui-n ,…,y ui-1 For each non-blank symbol 301 in

number

number

[0028] A sequence of non-blank symbols 252a-n, y u-n ,…,y u-1 The corresponding embedding generated by the shared embedding matrix 304 for each non-blank symbol 301 in is the same for all of the heads 302A-H of the multi-head attention mechanism 302, but each head 302A-H defines a different set / row of position vectors 308. For example, the first head 302A defines a position vector PV Aa~An The second head 302B defines the position vector PV Ba~Bn308 of Ba~Bn , ..., define different rows, and the Hth head 302H is the position vector PV Ha~Hn 308 of Ha~Hn Define another different line of

[0029] For each non-blank symbol in the received sequence of non-blank symbols 252a-n, the first head 302A, via the weight layer 310, weights the corresponding embedding 306 in proportion to the similarity between the corresponding embedding and its assigned respective position vector PV 308. In some examples, the similarity may include cosine similarity (e.g., cosine distance). In the illustrated example, the weight layer 310 outputs a sequence of weighted embeddings 312, 312Aa-An, each associated with a corresponding embedding 306 weighted in proportion to its assigned respective position vector PV 308. Stated differently, the weighted embedding 312 output by the weight layer 310 for each embedding 306 may correspond to a dot product between the embedding 306 and its assigned respective position vector PV 308. The weighted embedding 312 may be interpreted as weighting over the embedding in proportion to how similar the embedding is to the positioning associated with its assigned respective position vector PV 308. To increase computational speed, the prediction network 300 includes a non-recurrent layer, and therefore the sequence of weighted embeddings 312Aa-An is not concatenated but instead averaged by a weighted average module 316 to produce a weighted average 318A of the weighted embeddings 312Aa-An as output from the first head 302A, expressed as follows:

number

[0030] The operations described above with respect to the first head 302A are similarly performed for each of the other heads 302B-H of the multi-head attention mechanism 302. Due to the different sets of positioning vectors PV 308 defined by each head 302, the weight layer 310 outputs a sequence of weighted embeddings 312Ba-Bn, 312Ha-Hn for each of the other heads 302B-H that differs from the sequence of weighted embeddings 312Aa-Aa for the first head 302A. The weighted average module 316 then generates a respective weighted average 318B-H of the corresponding weighted embeddings 312 of the sequence of non-blank symbols as output from each of the other corresponding heads 302B-H.

[0031] In the illustrated example, the prediction network 300 includes a head averaging module 322 that averages weighted averages 318A-H output from corresponding heads 302A-H. A projection layer 326 with SWISH may receive as input an output 324 from the head averaging module 322 corresponding to the average of the weighted averages 318A-H and generate as output a projection output 328. A final layer, normalization 330, normalizes the embedding vector P from multiple time steps to a single embedding vector P at a corresponding time step. u The prediction network 300 may normalize the projection outputs 328 to provide a label history representation 350 (e.g., a label history representation). At each of multiple time steps following the initial time step, the prediction network 300 generates a single embedding vector P u Generates only 350.

[0032] In some configurations, the prediction network 300 does not implement a multi-head attention mechanism 302, and only performs the operations described above for the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is simply passed through the projection layer 326 and layer normalization 330 to produce a single embedding vector P u 350 will be offered.

[0033] Referring back to FIG. 2, the ASR joint network 240 extracts a single embedding vector P from the prediction network 300. u 350, and the audio encoding 224 from the encoder 220. The wordpiece joint network 240 calculates a probability distribution over the possible speech recognition hypotheses at the corresponding time step.

number

number

[0034] The softmax layer 250 calculates the next output symbol y predicted by the RNN-T model 200 at the corresponding output step.u As 252, the distribution y * Any technique can be used to select the output label / symbol with the highest probability within 242. In this way, the RNN-T model 200 does not make any conditional independence assumptions. Instead, the RNN-T model 200 predicts each symbol conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model 200 assumes that the output symbols are independent of future audio data 110, which allows the RNN-T model 200 to be used in a streaming manner. In some examples, the softmax layer 250 is composed of a unified word piece or grapheme set generated using all unique word pieces or graphemes from multiple training datasets.

[0035] The IQ joint network 230 uses the high-dimensional feature representation 224 output by the encoder network 220 and the single embedding vector P u 350, generates a probability distribution for which the corresponding time step corresponds to an intended query or an unintended query. In other words, the IQ joint network 230 generates a probability distribution for which the corresponding time step corresponds to an intended query or an unintended query. <intended>, for unintended queries <unintended>The method may output a set of values ​​indicating the likelihood of occurrence of each of a predetermined set of output labels, including: The set of values ​​may be a vector and may indicate a probability distribution over the set of output labels. The probability distribution providing the conditional probability of an intended query and the conditional probability of an unintended query may be determined using the following mathematical formula:

number

number

number

number

[0036] 1A, 1B, and 2, to ensure that the RNN-T 200 produces accurate speech recognition results, the training system 130 (FIGS. 1A and 1B) trains the RNN-T 200 (including the IQ joint network 230) using hybrid autoregressive transducer (HAT) factorization in a two-stage training process to generate possible subword units and two intended query tokens (e.g., <intended>and <unintended>) in some examples. In some examples, the possible subword units include word pieces (e.g., 4,096 possible word pieces). In other examples, the RNN-T 200 is trained to predict graphemes, phonemes, whole words, or other types of subword units. During the first stage, the training system 130 trains the encoder network 220, the prediction network 300, and the ASR joint network 240 for a speech recognition task to perform speech recognition. Specifically, the training system 130 performs the first training stage by optimizing the encoder network 220, the ASR joint network 240, and the prediction network 300 using a common label sequence of word pieces. During the second stage, the training system 130 keeps the parameters of the encoder network 220, the prediction network 300, and the ASR joint network 240 fixed, while initializing and fine-tuning (e.g., training) the IQ joint network 230 to learn how to detect whether an input utterance contains the intended query. That is, the training system 130 trains the IQ joint network 230 to learn how to predict intended and unintended positions in an utterance. The training system 130 indicates whether a transcribed utterance contains a query intended for the digital assistant. <intended>and <unintended>To predict the distribution of tokens, we use the extended labels (e.g., <intended>and <unintended>), for example, by the training data generator 140 (FIG. 1). For example, the training data generator 140 may generate a set of semantic commands for the digital assistant, followed by <intended>Tokens can be inserted after word pieces that do not contain semantic commands and / or when a silent pause is determined. <unintended>Tokens can be inserted. Thus, the training system 130 uses regular label sequences of word pieces in the first stage, while the training data generator 140 generates extended label sequences of word pieces in the second stage to train the IQ joint network 230.

[0037] For short-form utterances containing a single spoken query, the training data generator 140 appends <eos>Add tokens, e.g., for silences determined using forced alignment. <pause>Short forms of utterances can be used to model the end of speech events and regular short pauses, but they may not cover the wider range of possible disfluencies. <eos>If a token is added to the end of each utterance, the disfluency detection joint network 210 may be unable to detect any additional speech end events after determining the first speech end event in an utterance, which may cause disfluency detection problems for longer utterances or utterances with multiple disfluencies. <eos>It can learn to stop emitting tokens.

[0038] Referring back to FIG. 2, the feature vector x222 input to the encoder network 220 may include 80-dimensional log-melt filter bank features formed by stacking three 25-millisecond (ms) acoustic frames with a 10-ms shift and downsampled to a 30-ms frame rate. In some examples, the encoder network 220 includes twelve 512-dimensional conformer layers. The conformer layers include a stack of causal convolutions with a kernel size of 15 and a left context attention layer with eight head self-attentions. While the illustrated encoder network 220 includes a stack of multi-head attention layers / blocks with self-attention including conformer layers / blocks (e.g., 12 conformer blocks), the disclosure is not so limited. For example, the encoder network 220 may include a stack of transformer layers or any other type of multi-head attention layer / block. The encoder network 220 may include a series of multi-head self-attention layers, depthwise convolutional layers, and feedforward layers. Alternatively, the encoder network 220 may include multiple long short-term memory (LSTM) layers instead of multi-head attention layers / blocks.

[0039] The prediction network 300 may include an LTSM-based network with an embedding dimension of 320. The dimension D of the probability distribution representation 242 output by the ASR joint network 240 ジョイント may be set to 640. In some examples, the ASR joint network 240 includes hidden units. Additionally or alternatively, the wordpiece joint network 240 does not include a fully connected (FC) layer. The IQ joint network 230 may include an FC layer with 640 hidden units. Alternatively, the prediction network 300 may include a stack of transformer or conformer blocks (or other types of multi-head attention blocks). The prediction network 300 may also include an embedding lookup table (e.g., a V2 embedding lookup table) that improves latency by outputting a lookup sparse embedding instead of generating a hidden representation. In some implementations, the prediction network 300 is a stateless prediction network.

[0040] The ASR joint network 240 and the prediction network 300 may collectively form the RNN-T decoder of the RNN-T model 200. In some implementations, parameter tying between the prediction network 300 and the ASR joint network 240 is applied to further reduce the size of the RNN-T decoder, i.e., the prediction network 300 and the ASR joint network 240. Specifically, the vocabulary size |V| and the embedding dimension d e , the shared embedding matrix 304 in the prediction network 300 is

number

number

[0041] FIG. 4A illustrates an example transcribed training utterance 400. FIG. 4B illustrates an example semantically annotated transcribed training utterance 420 generated by an NLU model for the transcribed training utterance 400 of FIG. 4A. FIG. 4C depicts an example annotated transcribed training utterance 450 generated by the training data generator 140 for the transcribed training utterance 400 of FIG. 4A. As shown in FIG. 4A, for each utterance, the training data generator 140 determines silence portions 402, 402a-n, for example, using forced alignment. In FIG. 4B, the NLU model analyzes the transcribed training utterance 400 to determine which of the silence portions 402 constitute slots 422, 422a-n that represent semantic commands (e.g., change, wake up, timer, at 8 o'clock), and labels identifying each semantic command as a slot 422. In FIG. 4C, the training data generator 140 generates a respective silent pause 402 for each silent portion 402 (including each slot 422) where a semantic command is completed (e.g., silent pause 402c). <intended>Tokens (e.g., token 452a) are labeled with a symbol for each silent pause (e.g., silent pauses 402a, 402b, 402d, and 402e) that do not represent the completion of a common semantic command. <unintended>Labeling tokens (e.g., tokens 454a, 454b, 454c, 454d) and final silent pauses (e.g., silent pause 402f) <intended>Label the token (e.g., token 452b).

[0042] 5 is a flowchart of an exemplary arrangement of operations of a computer-implemented method 500 for detecting an intended query and performing speech recognition. At operation 502, the method 500 includes receiving audio data 110, 222 corresponding to a spoken utterance 106 as input to a speech recognition model 200. The method 500 also includes using the speech recognition model 200 to perform speech recognition on the audio data 110, 222 at each of a plurality of time steps via operations 504 and 506. At operation 504, the method 500 includes encoding the audio data 110, 222 corresponding to the spoken utterance 106 into a corresponding audio encoding 224 using an audio encoder 220.

[0043] At operation 506, the method 500 also includes using the speech recognition joint network 240 to decode the corresponding audio encoding 224 encoded by the audio encoder 220 at the corresponding time step into a probability distribution 242 over the possible output labels of the spoken utterance 106 at the corresponding time step. At each of the multiple time steps, the method 500 also includes, at operation 508, using the intended query (IQ) joint network 230 configured to receive the label history representation 350 associated with the sequence of non-blank symbols 252 output by the final softmax layer 250 to determine an intended query decision 212 indicating whether the spoken utterance 106 includes a query directed to the digital assistant interface 105.

[0044] 6 is a schematic diagram of an exemplary computing device 600 that can be used to implement the systems and methods described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the scope of the invention as described and / or claimed in this document.

[0045] Computing device 600 includes processor 610, memory 620, storage device 630, high-speed interface / controller 640 connecting to memory 620 and high-speed expansion port 650, and low-speed interface / controller 660 connecting to low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 are interconnected using various buses and may reside on a common motherboard or otherwise exist as desired. Processor 610 (e.g., data processing hardware 12, 132 of FIGS. 1A and 1B) can process instructions for execution within computing device 600, including instructions stored in memory 620 or storage device 630 to display graphical information for a graphical user interface (GUI) on an external input / output device, such as display 680 coupled to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and types of memory. Also, multiple computing devices 600 may be connected, each performing some of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0046] The memory 620 (i.e., memory hardware 14, 134 in FIGS. 1A and 1B) stores information non-temporarily within the computing device 600. The memory 620 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and disk or tape.

[0047] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be a device array including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 620, the storage device 630, or memory on the processor 610.

[0048] The high-speed controller 640 manages more bandwidth-intensive operations of the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. This allocation of roles is merely exemplary. In some implementations, the high-speed controller 640 is coupled to memory 620, a display 680 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 650 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to a storage device 630 and a low-speed expansion port 690. The low-speed expansion port 690 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and may be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, or network devices such as a switch or router via a network adapter or the like.

[0049] The computing device 600, as shown, can be implemented in many different forms. For example, it can be implemented as a standard server 600a, or multiple times in a group of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

[0050] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general-purpose, and may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device, coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0051] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0052] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0053] The processes and logic flows described herein may be performed by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose processors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operably coupled to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0054] To provide for user interaction, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, for displaying information to a user and, optionally, a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, verbal, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0055] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.< / intended> < / unintended> < / intended> < / eos> < / eos> < / pause> < / eos> < / unintended> < / intended> < / unintended> < / intended> < / unintended> < / intended> < / unintended> < / intended> < / unintended> < / unintended> < / intended> < / unintended> < / intended> < / unintended> < / intended>

Claims

1. A computer-implemented method (500), when executed on data processing hardware (610), causing the data processing hardware (610) to: receiving audio data (110, 222) corresponding to spoken speech (106, 146, 148) as input to a speech recognition model (200); receiving, using a prediction network (300), a sequence of previous non-blank symbols that is restricted to the previous non-blank symbols output by a final softmax layer (250); Using the speech recognition model (200), at each of a plurality of time steps: Using an audio encoder (220), encoding and outputting the audio data (110, 222) corresponding to the spoken utterance (106, 146, 148) into a corresponding audio encoding (224); Using a speech recognition joint network (240), receiving the audio encoding (224) from the audio encoder (220); receiving a single embedding vector (350) from the prediction network (300) that includes a label history representation (350); performing speech recognition on the audio data (110, 222) by decoding the corresponding audio encoding (224) encoded by the audio encoder (220) at the corresponding time step into a probability distribution (242) over possible output labels for the spoken utterance (106, 146, 148) at the corresponding time step, the probability distribution (242) indicating a probability distribution of an outcome of the speech recognition; At each of the plurality of time steps, an intended query (IQ) joint network (230) configured to receive, as input, the probability distribution (242) and a label history representation (350) associated with a sequence of non-blank symbols (252) output by a final softmax layer (250) is used to: The method (500) performs an operation including: determining an intended query determination (212) indicating whether the spoken utterance (106, 146, 148) includes a query directed to the digital assistant interface (105) based on a probability distribution corresponding to an intended query or an unintended query generated based on the audio encoding (224) output by the audio encoder (220) and the single embedding vector (350) output from the prediction network (300).

2. the speech recognition model (200) includes the audio encoder (220), the speech recognition joint network (240), and a prediction network (230), the prediction network (230) configured to receive the sequence of non-blank symbols (252) output by the final softmax layer (250) and generate the label history representation (350) at each of the plurality of time steps; the speech recognition model (200) is trained during a first training stage by optimizing the audio encoder (220), the speech recognition joint network (240), and the prediction network (230) using a common label sequence of word pieces; 10. The method of claim 1, wherein the IQ joint network is initialized with the speech recognition joint network by freezing the audio encoder and the prediction network and fine-tuning the IQ joint network with an extended label sequence of both word pieces and IQ tokens during a second training phase to teach the IQ joint network to learn how to predict a distribution of IQ tokens that indicate whether an input utterance contains a query directed to the digital assistant interface.

3. generating the label history representation (350) for the sequence of corresponding non-blank symbols (252), For each non-blank symbol (252) in the sequence of non-blank symbols (252) received as input at each of the plurality of time steps: generating, by the prediction network (230), embeddings (306) of the corresponding non-blank symbols (252) using the shared embedding matrix (304); assigning each position vector (308) to the corresponding non-blank symbol (252) by the prediction network (230); weighting the embeddings (306) by the prediction network (230) in proportion to the similarity between the embeddings (306) and the respective position vectors (308); generating a single embedding vector (350) for the corresponding time step as output from the prediction network (230), the single embedding vector (350) being based on a weighted average of the weighted embeddings (318), the single embedding vector (350) including the label history representation (350); The method (500) of claim 2, comprising:

4. 4. The method of claim 3, wherein the prediction network includes a multi-head attention mechanism that shares the shared embedding matrix across each head of the multi-head attention mechanism.

5. The method of claim 1, wherein the audio data corresponding to the spoken utterance is received during a current dialog session between the user and the digital assistant interface.

6. The method (500) of any of claims 1 to 5, wherein the output labels comprise wordpieces, words, phonemes, or graphemes.

7. The audio encoder (220) Multiple one-way long short-term memory (LSTM) layers, Multiple conformer layers, or 10. The method of claim 1, further comprising a causal encoder comprising one of a plurality of transformer layers.

8. The method of claim 1 , wherein the speech recognition model is trained using hybrid autoregressive transducer factorization.

9. 10. The method of claim 1, wherein the operations further include providing a response to the received spoken utterance when the intended query determination indicates that the spoken utterance includes a query directed to the digital assistant interface.

10. The method (500) of claim 1, wherein the operation further includes discarding the received spoken utterance (106, 146, 148) when the intended query determination (212) indicates that the spoken utterance (106, 146, 148) does not include a query directed to the digital assistant interface (105).

11. data processing hardware (610); and memory hardware (620) in communication with the data processing hardware (610), wherein the memory hardware (620), when executed on the data processing hardware (610), causes the data processing hardware (610) to: receiving audio data (110, 222) corresponding to spoken speech (106, 146, 148) as input to a speech recognition model (200); receiving, using a prediction network (300), a sequence of previous non-blank symbols that is restricted to the previous non-blank symbols output by a final softmax layer (250); Using the speech recognition model (200), at each of a plurality of time steps: encoding the audio data (110, 222) corresponding to the spoken utterance (106, 146, 148) into a corresponding audio encoding (224) using an audio encoder (220) and outputting the audio data (110, 222) corresponding to the spoken utterance (106, 146, 148); Using a speech recognition joint network (240), receiving the audio encoding (224) from the audio encoder (220); receiving a single embedding vector (350) from the prediction network (300) that includes a label history representation (350); decoding the corresponding audio encoding (224) encoded by the audio encoder (220) at the corresponding time step into a probability distribution (242) over possible output labels for the spoken utterance (106, 146, 148) at the corresponding time step; performing speech recognition on the audio data (110, 222) by the following method, wherein the probability distribution (242) indicates a probability distribution of a result of the speech recognition; At each of the plurality of time steps, using an intended query (IQ) joint network configured to receive, as input, the probability distribution (242) and a label history representation (350) associated with a sequence of non-blank symbols (252) output by a final softmax layer (250), Determining an intended query determination (212) indicating whether the spoken utterance (106, 146, 148) includes a query directed to the digital assistant interface (105) based on a probability distribution corresponding to an intended query or an unintended query generated based on the audio encoding (224) output by the audio encoder (220) and the single embedding vector (350) output from the prediction network (300); A system (100) storing instructions causing the system to perform operations including:

12. the speech recognition model (200) includes the audio encoder (220), the speech recognition joint network (240), and a prediction network (230), the prediction network (230) configured to receive the sequence of non-blank symbols (252) output by the final softmax layer (250) and to generate the label history representation (350) at each of the plurality of time steps; the speech recognition model (200) is trained during a first training stage by optimizing the audio encoder (220), the speech recognition joint network (240), and the prediction network (230) using a common label sequence of word pieces; 12. The system (100) of claim 11, wherein the IQ joint network (230) is initialized with the speech recognition joint network (240) during a second training phase by freezing the audio encoder (220) and the prediction network (230) and fine-tuning the IQ joint network (230) with an extended label sequence of both word pieces and IQ tokens (452, 454) to teach the IQ joint network (230) to learn how to predict a distribution of IQ tokens that indicate whether an input utterance contains a query directed to the digital assistant interface (105).

13. generating the label history representation (350) for the sequence of corresponding non-blank symbols (252), For each non-blank symbol (252) in the sequence of non-blank symbols (252) received as input at each of the plurality of time steps: generating, by the prediction network (230), embeddings (306) of the corresponding non-blank symbols (252) using the shared embedding matrix (304); assigning each position vector (308) to the corresponding non-blank symbol (252) by the prediction network (230); weighting the embeddings (306) by the prediction network (230) in proportion to the similarity between the embeddings (306) and the respective position vectors (308); generating a single embedding vector (350) for the corresponding time step as output from the prediction network (230), the single embedding vector (350) being based on a weighted average of the weighted embeddings (312), the single embedding vector (350) including the label history representation (350); The system (100) of claim 12, comprising:

14. 14. The system of claim 13, wherein the prediction network includes a multi-head attention mechanism that shares the shared embedding matrix across each head of the multi-head attention mechanism.

15. The system (100) of claim 11, wherein the audio data (110, 222) corresponding to the spoken utterance (106, 146, 148) is received during a current dialog session between the user (102) and the digital assistant interface (105).

16. The system (100) of claim 11, wherein the output label comprises a wordpiece, a word, a phoneme, or a grapheme.

17. The audio encoder (220) Multiple one-way long short-term memory (LSTM) layers, Multiple conformer layers, or The system (100) of claim 11, comprising a causal encoder comprising one of a plurality of transformer layers.

18. The system (100) of claim 11, wherein the speech recognition model (200) is trained using hybrid autoregressive transducer factorization.

19. 12. The system of claim 11, wherein the operation further includes providing a response to the received spoken utterance when the intended query determination indicates that the spoken utterance includes a query directed to the digital assistant interface.

20. 12. The system of claim 11, wherein the operations further include discarding the received spoken utterance when the intended query determination indicates that the spoken utterance does not include a query directed to the digital assistant interface.

Images