Historical-based ASR error correction

Abstract

A method (400) includes receiving subsequent audio data (127) captured by an assistant-supporting device (102), the subsequent audio data (127) corresponding to a subsequent query (129) spoken to a digital assistant after a user submits a previous query (127) to the digital assistant (109). The method also includes processing the subsequent audio data using the speech recognizer (210) to generate a plurality of candidate hypotheses (225). For each corresponding candidate hypothesis among the plurality of candidate hypotheses, the method further includes determining a corresponding similarity metric between a previous query and the corresponding candidate hypothesis (245), and determining a transcription of a subsequent query spoken by the user based on the similarity metrics determined for the plurality of candidate hypotheses.

Description

Technical Field

[0001] The present disclosure relates to history-based automatic speech recognition (ASR) error correction. Background Art

[0002] A voice-enabled environment (e.g., a residence, workplace, school, car, etc.) allows a user to speak aloud queries or commands to a computer-based system, which responds and answers the queries and / or performs functions based on the commands. A voice-enabled environment can be implemented using a network of connected microphone devices distributed in various rooms or areas of the environment. The device can operate in a sleep state and initiate a wake-up process to perform voice recognition on the utterance to the system in response to detecting a hot word spoken by the user preceding the utterance. The user can speak a follow-on query or command after receiving an answer responded to by the computer-based system. Requiring the user to speak a hot word for each subsequent follow-on query or command is both annoying and unnatural for the user. Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations, the operations comprising receiving subsequent audio data captured by an assistant-enabled device, the subsequent audio data corresponding to a subsequent query spoken by a user of the assistant-enabled device to a digital assistant after the user submitted a previous query to the digital assistant. The operations also include processing the subsequent audio data using a speech recognizer to generate a plurality of candidate hypotheses, each candidate hypothesis corresponding to a candidate transcription of the subsequent query and represented by a corresponding series of hypothesis terms. For each corresponding candidate hypothesis among the plurality of candidate hypotheses, the operations also include determining a corresponding similarity measure between the previous query and the corresponding candidate hypothesis, and determining a transcription of the subsequent query spoken by the user based on the similarity measures determined for the plurality of candidate hypotheses. The similarity measure indicates a similarity between a topic associated with the corresponding candidate hypothesis and a topic associated with the previous query.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation also includes determining a previous query-based salient term (QBST) vector associated with a previous query submitted by the user, and for each corresponding candidate hypothesis, determining a corresponding candidate QBST vector associated with the corresponding candidate hypothesis. Here, determining a corresponding similarity measure between the previous query and the corresponding candidate hypothesis includes: determining a corresponding similarity measure between the previous QBST vector and the corresponding candidate QBST vector based on the previous QBST vector and the corresponding candidate QBST vector. In these implementations, the previous QBST vector may indicate a corresponding set of salient terms associated with the previous query, and each corresponding candidate QBST vector may indicate a corresponding set of salient terms associated with the corresponding candidate hypothesis. In some examples, the corresponding similarity measure indicates a topic drift between the previous QBST vector and the corresponding candidate QBST vector. In an additional example, the corresponding similarity measure indicates a cosine score between the previous QBST vector and the corresponding candidate QBST vector.

[0005] In some examples, the operations also include: receiving initial audio data captured by the assistant-enabled device while the assistant-enabled device is in a sleep state, wherein the initial audio data includes hot words and previous queries submitted by a user to the digital assistant, such that the hot words, when detected by the assistant-enabled device, cause the assistant-enabled device to wake up from a sleep state and trigger a speech recognizer to perform speech recognition on at least a portion of the initial audio data including the previous query. In these examples, after the speech recognizer performs speech recognition on at least a portion of the initial audio data, the operations also include instructing the assistant-enabled device to operate in a subsequent query mode. Here, receiving subsequent audio data by the assistant-enabled device includes receiving subsequent audio data while the assistant-enabled device is operating in a subsequent query mode, wherein no hot words are present in the subsequent audio data.

[0006] In some implementations, the operations further include: performing query interpretation on the transcription to identify an operation specified by a subsequent query, instructing the digital assistant to perform the operation specified by the subsequent query, and receiving a subsequent response from the digital assistant, the subsequent response indicating performance of the operation specified by the subsequent query. In these implementations, the operations may also include presenting the subsequent response for output from an assistant-enabled device.

[0007] For each corresponding candidate hypothesis among the plurality of candidate hypotheses generated by the speech recognizer, the operation may also include obtaining a corresponding likelihood score that the speech recognizer has assigned to the corresponding candidate hypothesis, and ranking the plurality of candidate hypotheses based on the corresponding likelihood scores assigned by the speech recognizer to the plurality of candidate hypotheses and the corresponding similarity scores determined for each corresponding candidate hypothesis among the plurality of candidate hypotheses. Here, determination of a transcription of a subsequent query may be based on the ranking of the plurality of candidate hypotheses.

[0008] It is noteworthy that the data processing hardware executes the speech recognizer and can reside on either the assistant-enabled device or a remote server. The remote server can communicate with the assistant-enabled device via a network.

[0009] In some examples, the speech recognizer includes an end-to-end speech recognition model. In other examples, the speech recognizer includes an acoustic model and a language model. In addition, the assistant-enabled device can communicate with one or more microphones configured to capture subsequent audio data and initial audio data corresponding to a previous query. The assistant-enabled device can optionally include a battery-powered device.

[0010] Another aspect of the present disclosure provides a system that includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations that include receiving subsequent audio data captured by an assistant-enabled device, the subsequent audio data corresponding to a subsequent query spoken by a user of the assistant-enabled device to a digital assistant after the user submitted a previous query to the digital assistant. The operations also include processing the subsequent audio data using a speech recognizer to generate a plurality of candidate hypotheses, each candidate hypothesis corresponding to a candidate transcription of the subsequent query and represented by a corresponding series of hypothesis terms. For each corresponding candidate hypothesis among the plurality of candidate hypotheses, the operations also include determining a corresponding similarity measure between the previous query and the corresponding candidate hypothesis, and determining a transcription of the subsequent query spoken by the user based on the similarity measures determined for the plurality of candidate hypotheses. The similarity measure indicates a similarity between a topic associated with the corresponding candidate hypothesis and a topic associated with the previous query.

[0011] This aspect may include one or more of the following optional features. In some implementations, the operation also includes determining a previous query-based salient term (QBST) vector associated with a previous query submitted by the user, and for each corresponding candidate hypothesis, determining a corresponding candidate QBST vector associated with the corresponding candidate hypothesis. Here, determining a corresponding similarity measure between the previous query and the corresponding candidate hypothesis includes: determining a corresponding similarity measure between the previous QBST vector and the corresponding candidate QBST vector based on the previous QBST vector and the corresponding candidate QBST vector. In these implementations, the previous QBST vector may indicate a corresponding set of salient terms associated with the previous query, and each corresponding candidate QBST vector may indicate a corresponding set of salient terms associated with the corresponding candidate hypothesis. In some examples, the corresponding similarity measure indicates a topic drift between the previous QBST vector and the corresponding candidate QBST vector. In an additional example, the corresponding similarity measure indicates a cosine score between the previous QBST vector and the corresponding candidate QBST vector.

[0012] In some examples, the operations also include: receiving initial audio data captured by the assistant-enabled device while the assistant-enabled device is in a sleep state, wherein the initial audio data includes hot words and previous queries submitted by a user to the digital assistant, such that the hot words, when detected by the assistant-enabled device, cause the assistant-enabled device to wake up from a sleep state and trigger a speech recognizer to perform speech recognition on at least a portion of the initial audio data including the previous query. In these examples, after the speech recognizer performs speech recognition on at least a portion of the initial audio data, the operations also include instructing the assistant-enabled device to operate in a subsequent query mode. Here, receiving subsequent audio data by the assistant-enabled device includes receiving subsequent audio data while the assistant-enabled device is operating in a subsequent query mode, wherein no hot words are present in the subsequent audio data.

[0013] In some implementations, the operations further include: performing query interpretation on the transcription to identify an operation specified by a subsequent query, instructing the digital assistant to perform the operation specified by the subsequent query, and receiving a subsequent response from the digital assistant, the subsequent response indicating performance of the operation specified by the subsequent query. In these implementations, the operations may also include presenting the subsequent response for output from an assistant-enabled device.

[0014] For each corresponding candidate hypothesis among the plurality of candidate hypotheses generated by the speech recognizer, the operation may also include obtaining a corresponding likelihood score that the speech recognizer has assigned to the corresponding candidate hypothesis, and ranking the plurality of candidate hypotheses based on the corresponding likelihood scores assigned by the speech recognizer to the plurality of candidate hypotheses and the corresponding similarity scores determined for each corresponding candidate hypothesis among the plurality of candidate hypotheses. Here, determination of a transcription of a subsequent query may be based on the ranking of the plurality of candidate hypotheses.

[0015] It is noteworthy that the data processing hardware executes the speech recognizer and can reside on either the assistant-enabled device or a remote server. The remote server can communicate with the assistant-enabled device via a network.

[0016] In some examples, the speech recognizer includes an end-to-end speech recognition model. In other examples, the speech recognizer includes an acoustic model and a language model. In addition, the assistant-enabled device can communicate with one or more microphones configured to capture subsequent audio data and initial audio data corresponding to a previous query. The assistant-enabled device can optionally include a battery-powered device.

[0017] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1A is a schematic diagram of a user speaking an initial query preceded by a spoken hot word to invoke an assistant-enabled device to trigger an automatic speech recognition (ASR) system to process the initial query.

[0019] Figure 1B The user receives the Figure 1A Schematic diagram of a subsequent query without a preceding hot word being spoken after an answer to an initial query and the ASR system processing the subsequent query.

[0020] Figure 2 yes Figure 1A and Figure 1B Schematic diagram of an ASR system; the ASR system is based on Figure 1A The similarity of the initial query issued by the user in the speech recognition hypothesis grid is Figure 1A The subsequent query selects the best candidate hypothesis.

[0021] Figure 3A and Figure 3B is a diagram showing an illustration of an example of a word lattice.

[0022] Figure 4 is a flow diagram of an example arrangement of operations of a method for history-based ASR error correction for subsequent queries.

[0023] Figure 5 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0024] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0025] As automatic speech recognition (ASR) systems continue to provide more and more accurate transcriptions of what users say, ASR systems are becoming more and more popular in client devices. In addition, in some cases, ASR systems generate inaccurate transcriptions that misidentify what the user actually said. This usually occurs when words are acoustically similar or when the user says a unique word that the ASR system does not know. For example, "I say" and "Ice age" sound very similar, and the ASR system may have difficulty discerning which phrase the user intended to convey. In some examples, if a client device incorrectly transcribes "I say" when the user actually intended to convey "Ice age", the user can correct the transcription by using the client device (e.g., entering the correct transcription via the keyboard of the client device).

[0026] Voice-based interfaces, such as digital assistants, are becoming increasingly common on a variety of devices, including, but not limited to, mobile phones and smart speakers / displays that include microphones for capturing voice. A common way to initiate voice interaction with an assistant-enabled device (AED) is to speak a fixed phrase, such as a hot word, which, when detected by the AED in streaming audio, triggers the AED to initiate a wake-up process to begin recording and processing subsequent speech to determine the query spoken by the user. Therefore, the hot word is an important component in the overall digital assistant interface stack because it allows the user to wake up their AED from a low-power state to a high-power state so that the AED can continue to perform more expensive processing, such as full ASR or server-based ASR.

[0027] Typically, a user will make a follow-up query after receiving a response to an initial query. For example, a user may speak an initial query to the AED, "what is the temperature in Detroit? ", and the AED returns a response to the user, "76-degrees Fahrenheit." Thereafter, the user may ask the AED a follow-up query, "Is it Sunny? " Here, the continuous conversation mode or follow-up query mode keeps the AED in a high power state for a predefined amount of time after providing a response by keeping the microphone on and running ASR (on the device and / or server side) to process any speech to determine whether any of the subsequent speech is intended to be directed toward the AED. After the predefined amount of time expires, the AED returns to a low power state.

[0028] However, in order to not degrade the user experience, processing incoming audio when the AED is in a high-power state to determine whether the subsequent speech is intended to be directed toward the AED is accompanied by challenges. For example, the subsequently captured speech can be processed by the ASR system and treated as a subsequent query, even if the subsequent speech is not intended to be directed toward the AED (whether it is spoken by the user, another user, or background speech from a nearby device such as a TV or music player). In such cases, when the captured speech that is not intended for the AED is processed as a follow-up, although the speech is not intended for the AED, the AED may return an answer or perform an action that the user has never requested. In addition, the answer returned or the action performed in response to the unintended subsequent query is likely to be irrelevant even to the initial query spoken by the user, which is preceded by the user's explicit hot words that were initially spoken to invoke the ASR. On the other hand, the initial query spoken by the user can be used to improve the accuracy and latency of transcribing the subsequent query spoken by the user based on the intuition that the subsequent query should be similar to the initial query before the subsequent query in subject matter.

[0029] Therefore, the implementation of the present invention relates to a method of utilizing an initial query to an AED to bias the selection of a speech recognition candidate among multiple speech recognition candidates for subsequent queries based on similarity to the initial query. Specifically, a user submits an initial query including a hot word and a query to an AED having an always-on digital signal processor (DSP) operating in a hot word detection mode. When the DSP operating in the hot word detection mode detects a hot word in the initial query, the DSP initiates a wake-up process on a second processor to perform more expensive processing (such as full ASR) to provide a response to the initial query. The second processor may include an application processor (AP) or other type of system-on-chip (SoC) processor.

[0030] In response to receiving a response to the initial query, the AED instructs the second processor to remain awake to operate in a subsequent query mode (also referred to as a 'continuous conversation mode') to permit the ASR system to process any subsequent queries that the user may speak without the user having to repeat the hotword. When in the continuous conversation mode, the ASR system (executing on the second processor and / or on a server in communication with the second processor) receives subsequent audio data corresponding to the subsequent query spoken by the user and processes the audio data to generate a plurality of candidate hypotheses, each candidate hypothesis corresponding to a candidate transcription of the subsequent query and represented by a corresponding series of hypothetical terms (e.g., words). Here, the ASR system generates a corresponding likelihood score for each candidate hypothesis, the likelihood score indicating the probability that the candidate hypothesis is a correct transcription of the subsequent query spoken by the user. The implementations herein are specifically directed to an ASR system that determines a similarity measure between the initial query and each candidate hypothesis among the plurality of candidate hypotheses (or among a list of the N best (n-best) candidate hypotheses among the plurality of candidate hypotheses with the highest likelihood score), and thereafter uses these similarity measures to bias the selection of candidate hypotheses associated with topics that are most similar to the topic of the initial query. As described in more detail below, the similarity measure determined between the initial query and each candidate hypothesis may be based on respective query-based salient term (QBST) vectors generated for the initial query and each of the candidate hypotheses.

[0031] The user can explicitly opt-in to enable the functionality of the subsequent query mode, and can opt-out to disable the functionality of the subsequent query mode. When the subsequent query mode is disabled, the second processor will return to a sleep state after processing the initial query, and will only wake up to process subsequent speech when called by the always-on DSP in response to detecting a hot word. The implementation described herein for using a similarity metric based on a QBST vector is also applicable to the case where the user speaks a subsequent query when the subsequent query mode is disabled or speaks a subsequent query after the subsequent query mode fails and the second processor returns to a sleep state. Therefore, the subsequent query can correspond to any query submitted after the initial query, so that the user may have to say a predetermined hot word or perform some other action (e.g., a gesture, eye gaze, lifting the user device, etc.) to wake up and call the second processor to process the subsequent query.

[0032] refer to Figure 1A and Figure 1B In some implementations, the example system 100 includes an AED (i.e., user device) 102 that is associated with one or more users 10 and optionally communicates with a remote system 110 via a network 104. The AED 102 may correspond to a computing device such as a mobile phone, a computer (laptop or desktop), a tablet, a smart speaker / display, a smart appliance, a smart headset, a wearable device, a vehicle infotainment system, etc., and is equipped with data processing hardware 103 and memory hardware 105. The AED 102 includes or communicates with one or more microphones 106 for capturing streaming audio 118 in the environment of the AED 102, which may include utterances 119, 129 spoken by the respective user 10. The remote system 110 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) with scalable / elastic computing resources 112 (e.g., data processing hardware) and / or storage resources 114 (e.g., memory hardware).

[0033] The data processing hardware 103 of the AED 102 includes a first processor 150 and a second processor 160. The first processor 150 may include an always-on DSP 150 (also referred to as DSP 150) configured to detect the presence of a hot word 116 in a streaming audio 118 without performing semantic analysis or speech recognition processing on the streaming audio 118. The DSP 150 may receive the streaming audio 118, which includes acoustic features extracted by an acoustic feature extractor from an utterance 119 spoken by the user 10. The DSP (i.e., first processor) 150, and more generally the AED 102, may be in a hot word detection mode ( Figure 1A) 152 and subsequent query mode (ie, subsequent mode) ( Figure 1B ) 154. In some examples, due to storage / memory / processing limitations, the DSP 150 operates in only one mode at a time, either the hot word detection mode 152 or the subsequent query mode 154. That is, in these examples, either the hot word detection mode 152 is enabled and the subsequent mode 154 is disabled, or the hot word detection mode 152 is disabled and the subsequent mode 154 is enabled. However, in other examples, the operation of the DSP 150 in the hot word detection mode 152 remains enabled while the DSP 150 simultaneously operates in the subsequent query mode 154. It will be apparent that when the AED 102 is in a low power state and the second processor 160 is asleep, the always-on processor 150 can operate in the hot word detection mode 152. When the always-on processor 150 is in the subsequent query mode 154, the second processor 160 can be awake / active to perform speech recognition on the subsequent audio data 127 representing the subsequent query. Alternatively, the second processor 160 may be in sleep mode while the always-on processor 150 operates in a subsequent query mode 152, such that the always-on processor 152 may perform voice activity detection (VAD) on the subsequent audio data 127, and only when the VAD determines that the subsequent audio data 127 includes human voice, trigger the second processor 160 to wake up from the sleep state to perform speech recognition on the subsequent audio data 127.

[0034] Reference now Figure 1AIn some implementations, the always-on DSP 150 operates in a hotword detection mode 152 while the subsequent query mode 154 is disabled and the second processor 160 is asleep. During operation in the hotword detection mode 152, the DSP 150 is configured to detect the presence of the hotword 116 “Ok Google” in the streaming audio 118 to initiate a wakeup process on the second processor 160 for processing the hotword 116 and / or the initial query 117 following the hotword 116 in the streaming audio 118. In the example shown, the utterance 119 includes the hotword 116 “Ok Google” followed by the initial query 117 “What is the temperature in Detroit?” The AED 102 can extract acoustic features from the streaming audio 118 and store the extracted acoustic features in a buffer of the memory hardware 105 for use in detecting whether the streaming audio 118 includes the presence of the hotword 116. The DSP 150 may execute a hotword detection model configured to generate a probability score indicating the presence of a hotword 116 in the acoustic features of the streaming audio 118 captured by the AED 102, and detect the hotword 116 in the streaming audio 118 when the probability score satisfies a hotword detection threshold. The DSP 150 may include multiple hotword detection models, each trained to detect a different hotword associated with a particular term / phrase. These hotwords may be predefined hotwords and / or custom hotwords assigned by the user 10. In some implementations, the hotword detection model includes a trained neural network-based model received from the remote system 110 via the network 104.

[0035] In the example shown, when the DSP 150 detects acoustic features that are characteristic of the hot word 116 in the streaming audio 118, the DSP 150 may determine that the utterance 119 “Ok Google, what is the temperature in Detroit?” includes the hot word 116 “Ok Google.” For example, the DSP 150 may detect that the utterance 119 “Ok Google, what is the temperature in Detroit?” includes the hot word 116 “Ok Google” based on generating MFCCs from the audio data and classifying that the MFCCs include MFCCs that are similar to MFCCs that are characteristic of the hot word “Ok Google” stored in the hot word detection model. As another example, the DSP 150 may detect that the utterance 119 “Ok Google, What is the temperature in Detroit?” includes the hot word 116 “Ok Google” based on generating mel-scale filterbank energies from the audio data and classifying that the mel-scale filterbank energies include mel-scale filterbank energies that are similar to mel-scale filterbank energies that are characteristic of the hot word “OK Google” stored in a hot word detection model.

[0036] In response to detecting the presence of a hot word 116 in the streaming audio 118 corresponding to an utterance 119 spoken by the user 10, the DSP 150 provides audio data 120 representing a hot word event to initiate a wake-up process on the second processor 160 for confirming the presence of the hot word 116. The audio data (interchangeably referred to as a hot word event) 120 includes a first portion 121 representing the hot word 116 and a second portion 122 representing the initial query 117. Here, the second processor 160 may execute a more robust hot word detection model to confirm whether the audio data 120 includes the hot word 116. Additionally or alternatively, the second processor 160 may perform speech recognition on the audio data 120 via an ASR model 210 of an automatic speech recognition (ASR) system 200 to confirm whether the audio data 120 includes the hot word.

[0037] When the second processor 160 confirms that the audio data 120 includes the hot word 116, the second processor 160 can execute the ASR model 210 to process the audio data 120 to generate a speech recognition result 215, and execute the natural language understanding (NLU) module 220 of the ASR system 200 to perform semantic interpretation on the speech recognition result 215 to determine that the audio data 120 includes the initial query 117 for the digital assistant 109 to perform an operation. In this example, the ASR model 210 can process the audio data 120 to generate a speech recognition result 215 for "What is the temperature in Detroit?", and the NLU module 220 can identify "What is the temperature in Detroit?" as the initial query 117 for the digital assistant 109 to perform an operation of obtaining a response 192 indicating "It is 76-degrees in Detroit (the temperature in Detroit is 76 degrees)" (i.e., an answer to the initial query 117). The digital assistant 109 can provide the response 192 for output from the AED 102. For example, the digital assistant 109 may audibly output the response 192 from the AED 102 as synthesized speech and / or display a textual representation of the response 192 on a screen of the AED 102 .

[0038] In some implementations, the ASR system 200 including the ASR model 210 and the NLU module 220 is located on the remote system 110 in addition to or in lieu of the AED 102. When the DSP 150 triggers the second processor 160 to wake up in response to detecting the hot word 116 in the utterance 119, the second processor can transmit the audio data 120 corresponding to the utterance 119 to the remote system 110 via the network 104. The AED 102 can transmit the first portion 121 of the audio data 120 including the hot word 116 for the remote system 110 to perform speech recognition via the ASR model 210 to confirm the presence of the hot word 116. Alternatively, the AED 102 can transmit only the second portion 122 of the audio data 120 corresponding to the initial query 117 spoken after the hot word 116 in the utterance 119 to the remote system 110. The remote system 110 executes the ASR model 210 to generate a speech recognition result 215 of the audio data 120. The remote system 110 may also execute the NLU module 220 to perform semantic interpretation on the speech recognition result 215 to identify the initial query 117 for the digital assistant 109 to perform operations. Alternatively, the remote system 110 may transmit the speech recognition result 215 to the AED 102, and the AED 102 may execute the NLU module 220 to identify the initial query 117.

[0039] The digital assistant 109 may be located on the remote system 110 and / or the AED 102. The digital assistant 109 is configured to perform the operations specified by the initial query 117 from the second processor 160. In some examples, the digital assistant 109 accesses a search engine to obtain a response 192 associated with the initial query 117. In other examples, the digital assistant 109 accesses the memory hardware 105 of the AED 102 and / or the memory hardware 114 of the remote system to obtain the response 192 associated with the initial query 117. Alternatively, the digital assistant 109 performs the operations associated with the initial query 117 (i.e., “call mom”).

[0040] In some implementations, the user 10 has a subsequent query based on the response 192 to the initial query 117. That is, after receiving the response 192 "it is 76-degrees in Detroit", the user 10 may have a subsequent query asking for additional weather conditions in Detroit. The current technology allows the user to provide a subsequent query by causing the second processor 160 to remain awake for a predetermined amount of time to process all audio data after the initial query 117 before returning to a sleep state, without the user saying the hotword again. That is, the ASR system 200 executed on the second processor 160 and / or the remote system 100 can continuously perform speech recognition and / or semantic interpretation on all subsequent audio data captured by the user device 102 during the subsequent query model 154 to determine whether the user 10 has submitted a subsequent query for the digital assistant 109 to perform an operation. As described below with reference Figure 1B and Figure 2 As described in more detail, the ASR system 200 may utilize the initial query 117 (and the corresponding transcription 215) to determine whether the initial query 117 is related to a plurality of candidate hypotheses 225 ( Figure 1B and Figure 2 ) that are determined by the ASR model 210 in response to processing subsequent audio data 127 captured by the user device 102 during the subsequent query mode 154 ( Figure 1B and Figure 2 The candidate hypothesis 225 associated with the highest similarity measure may indicate a candidate hypothesis associated with a topic that is most similar to the topic associated with the initial query, thereby enabling the ASR system 200 to select the candidate hypothesis 225 as the final transcription 175 for the subsequent query 129.

[0041] Reference now Figure 1BIn response to receiving a response 192 to an initial query 117 submitted by user 10, digital assistant 109 instructs DSP 150 (and optional second processor 160) to operate in subsequent query mode 154. Here, instructing DSP 150 to operate in subsequent query mode 154 may cause hotword detection mode 152 to be disabled. While operating in subsequent mode 154, first processor 150 and / or second processor 160 receive subsequent audio data 127 corresponding to a subsequent query 129 spoken by user 10 and captured by AED 102. In the example shown, in response to AED 102 providing user 10 with response 192 “It is 76-degrees in Detroit,” user 10 speaks subsequent query 129 “Is it Sunny?”. Notably, user 10 merely speaks subsequent query 129 without providing a response 192 such as “It is 76-degrees in Detroit.” Figure 1A In some other examples, however, the subsequent query 129 includes the hot word 116.

[0042] In some examples, during operation in the subsequent query mode 154, the DSP 152 or the second processor 160 performs voice activity detection (VAD) that is trained to determine whether there is voice activity in the subsequent audio data 127. That is, the VAD model determines whether the subsequent audio data 127 includes voice activity (such as human speech) or includes non-voice activity audio (i.e., stereo, speaker, background noise, etc.). The VAD model can be trained to output a voice activity score that indicates the likelihood that the subsequent audio data 127 includes voice activity. Here, when the voice activity score meets the voice activity threshold, the first processor 150 or the second processor 160 can determine that the subsequent audio data 127 includes voice activity. In some examples, the VAD model outputs a binary voice activity indication, where "1" represents "yes" and "0" represents "no", indicating that the subsequent audio data 127 includes ("yes") or does not include ("no") voice activity. The VAD model can be trained to distinguish between human speech and synthesized speech / synthesized speech. Detection of voice activity may cause the first processor 150 and / or the second processor 160 to invoke the ASR system 200 to perform speech recognition and semantic analysis on the subsequent audio data 127 to determine whether the subsequent audio data 127 includes a subsequent query 129 specifying a subsequent operation to be performed by the digital assistant 109.

[0043] Thus, the ASR model 210 can process the subsequent audio data 127 to generate a plurality of candidate hypotheses 225 for the subsequent audio data 127. The ASR system 200 can also execute the NLU module 220 to perform semantic interpretation on one or more of the candidate hypotheses 225 to identify a subsequent query 129 to be submitted to the digital assistant 109. In the example shown, the ASR system 200 determines that the subsequent audio data 127 corresponds to the subsequent query 129 for “Is it sunny?”, and the assistant 109 retrieves a subsequent response 193 for the subsequent query 129. Here, the second processor 160 provides the subsequent response 193 from the assistant 109 as output from the AED 102 in the form of synthesized speech and / or text to indicate “It is currently overcast with clouds clearing in the afternoon”.

[0044] Alternatively, the ASR system 200 may ultimately determine that while processing the subsequent audio data 127, the subsequent audio data 127 is not directed to the AED 102, and therefore, there is no subsequent query 129 for the digital assistant 109 to execute. In this case, the second processor 160 may provide an indication of the absence of the subsequent query in the subsequent audio data 127 and return to the sleep state or remain in the awake state for a predetermined amount of time. In some examples, the second processor 160 prompts the user 10 to repeat the subsequent query 129 in response to the indication of the absence of the subsequent query 129 in the subsequent audio data 127.

[0045] As mentioned above Figure 1A As described, the ASR model 210 and the NLU module 220 may be located on the remote system 110 in addition to or in lieu of the AED 102. Here, the AED 102 may transmit the subsequent audio data 127 to the remote system 110 via the network 104, whereby the remote system 110 may execute the ASR model 210 to generate candidate hypotheses 225 for the audio data 120. The remote system 110 may also execute the NLU module 220 to perform semantic interpretation on one or more of the candidate hypotheses 225 to identify the subsequent query 129 for the digital assistant 109 to perform operations. Alternatively, the remote system 110 may transmit one or more of the candidate hypotheses 225 to the AED 102, and the AED 102 may execute the NLU module 220 to identify the subsequent query 129.

[0046] Figure 2An example ASR system 200 is shown that biases selection of a plurality of candidate hypotheses 225, 225a-225n output by an ASR model 210 for subsequent audio data 127 based on topical similarity to one or more previous queries 117. While the examples herein refer to the previous query 117 as an initial query in a conversation with a digital assistant 109, topical similarities associated with any query in one or more queries preceding the current subsequent query 127 may be utilized to select a candidate hypothesis 225 as a transcription of the subsequent query 127. The ASR system 200 includes an ASR model 210, a query-based salient terms (QBST) module 230, a similarity scorer 240, and a re-ranker 250. Figure 2 Operations (A) through (E) are shown to illustrate data flow. As described herein, either the AED 102 or the remote system 110 performs operations (A) through (E). In some examples, the AED 102 performs all operations (A) through (E) entirely on the device, while in other examples, the remote system 110 performs all operations (A) through (E). In some additional examples, the AED 102 performs a first portion of the operations, while the remote system performs a second portion of the operations.

[0047] In the illustrated example, during operation (A), the ASR system 200 receives subsequent audio data 127 captured by the microphone 106 of the AED 120, and the ASR model 210 processes the subsequent audio data 127 to determine / generate a plurality of candidate hypotheses 225 for a subsequent query 129. Here, each candidate hypothesis 225 corresponds to a candidate transcription of the subsequent query 129 and is represented by a corresponding series of hypothesis terms. For example, the data processing hardware 105 of the AED 102 may execute the ASR model 210 to generate a word lattice 300 based on the subsequent audio data 127, the word lattice indicating a plurality of candidate hypothesis transcriptions 225 that are possible for the subsequent query 129. The ASR model 210 may evaluate potential paths through the word lattice 300 to determine the plurality of candidate hypotheses 225.

[0048] Figure 3A Shows Figure 2 1. An example of a word lattice 300, 300a that may be provided by the ASR model 210 of FIG. The word lattice 300a represents a plurality of possible word combinations that may form different candidate hypotheses 225 for a subsequent query 129 when the AED 102 is operating in the continuous conversation model (ie, the subsequent query mode).

[0049] The word grid 300a includes one or more nodes 202a to 202g corresponding to possible boundaries between words. The word grid 300a includes multiple edges 204a to 204l of possible words in the candidate hypothesis generated by the word grid 300a. In addition, each of the edges 304a to 304l can have one or more weights or probabilities that the edge is the correct edge from the corresponding node. The weight is determined by the ASR model 210 and can be based on, for example, the confidence of the match between the subsequent speech data 127 and the word of the edge and the degree of grammatical and / or lexical fit of the word with other words in the word grid 300a.

[0050] For example, initially, the most likely path through word lattice 300a (e.g., most likely candidate hypothesis 225) may include edges 204c, 204e, 204i, 204k with text “we're coming about 11:30”. The second best path through word lattice 300a (e.g., second best candidate hypothesis 225) may include edges 204d, 204h, 204j, 204l with text “deer hunting scouts 7:30”.

[0051] Each pair of nodes may have one or more paths corresponding to alternative words in each candidate hypothesis 225. For example, the initial most likely path between the pair of nodes starting at node 202a and ending at node 202c is edge 204c "we're". This path has alternative paths including edges 204a, 204b "we are" and edge 204d "deer".

[0052] Figure 3B yes Figure 21. An example of a hierarchical word lattice 300, 300b that may be provided by the ASR model 210 of FIG. The word lattice 300b includes nodes 252a-252l that represent words that constitute various candidate hypotheses 225 for a subsequent query 129 when the AED 102 operates in the continuous conversation model (ie, the subsequent query mode). The edges between nodes 252a to 252l show that possible candidate hypotheses 225 include: (1) nodes 252c, 252e, 252i, 252k "we're coming about 11:30"; (2) nodes 252a, 252b, 252e, 252i, 252k "we are coming about 11:30"; (3) nodes 252a, 252b, 252f, 252g, 252i, 252k "we are come at about 11:30"; (4) nodes 252d, 252f, 252g, 252i, 252k "deer come at about 11:30"; (5) nodes 252d, 252h, 252j, 252k "deer hunting scouts 11:30"; and (6) nodes 252d, 252h, 252j, 252l "deer hunting scouts 7:30".

[0053] Similarly, the edge between nodes 252a to 252l can have an associated weight or probability based on the confidence of speech recognition (e.g., candidate hypothesis) and the grammatical / lexical analysis of the resulting text. In this example, "we're coming about11:30" can be the best hypothesis at present, and "deer hunting scouts 7:30" can be the next best hypothesis. One or more partitions 254a to 254d can be made in the word grid 300b, and the one or more partitions 254a to 254d group words and their alternatives together. For example, partition 254a includes the word "we're" and the alternatives "we are" and "deer". Partition 252b includes the word "coming" and the alternatives "come at" and "hunting". Partition 254c includes the word "about" and the alternative "scouts", and partition 254d includes the word "11:30" and the alternative "7:30".

[0054] Return to reference Figure 2, the ASR model 210 can generate multiple candidate hypotheses 225 from the word grid 300. That is, the ASR model 210 generates a likelihood score 155 for each of the candidate hypotheses 225 of the word grid 300. Each likelihood score 155 indicates the probability that the candidate hypothesis 225 is correct (e.g., matches the utterance of the subsequent query 129 spoken by the user 10). Each likelihood score 155 can include a combination of an acoustic modeling score and / or a priori likelihood score. In some implementations, the ASR model 210 includes an end-to-end (E2E) speech recognition model configured to receive subsequent speech data 127 and generate the word grid 300. Specifically, the E2E speech recognition model processes the subsequent audio data 127 to generate a corresponding likelihood score 155 for each of the multiple candidate hypotheses 225 from the word grid 300. In these implementations, the ASR model 210 can utilize an integrated or external language model to generate the likelihood scores 155. In some examples, the ASR model 210 includes separate acoustic models, language models, and / or pronunciation models. The ASR model 210 may share acoustic models and language models with additional hypothesis scorers (e.g., acoustic models and language models), or have separate acoustic models and language models. As used herein, the ASR model 210 may be interchangeably referred to as a 'speech recognizer' that includes a single E2E neural network or a combination of separate acoustic, language, and / or pronunciation models.

[0055] During operation (B), the ASR system 200 identifies a set of top-ranked candidate hypotheses 225 from the plurality of candidate hypotheses 225 in the word lattice 300. For example, using the likelihood scores 155 from the ASR model 210, the ASR system 200 selects n candidate hypotheses 225 with the highest likelihood scores 155, where n is an integer. In some cases, the ASR system 200 selects candidate hypotheses 225 with likelihood scores 155 that satisfy a likelihood score threshold. Alternatively, the ASR model 210 can use the likelihood scores 155 to rank the set of top-ranked candidate hypotheses 225.

[0056] continue Figure 1A and Figure 1B, the ASR model 210 generates candidate hypotheses 225 for a subsequent query “Is it unny?” spoken by the user 10. In this example, the top two candidate transcriptions (e.g., the two most likely to be correct) are selected as the set of top-ranked candidate hypotheses 225. For simplicity, only the top two candidate hypotheses 225 are depicted in step B, and thus, the set of top-ranked candidate hypotheses 225 may include a group of two or more top-ranked candidate hypotheses 225 from the word grid 300. The top-ranked candidate hypotheses 225 include a first candidate hypothesis 225 “is it funny” having a likelihood score 155 of 0.55 and a second candidate hypothesis 135 “is its unny” having a likelihood score 155 of 0.45. Here, a higher likelihood score 155 indicates a greater confidence that the candidate hypothesis 135 is correct. Notably, the first candidate hypothesis (“is it funny”) 225 having the highest likelihood score 155 does not include the utterance of the subsequent query 129 actually spoken by the user 10. Thus, if the ASR system 200 selects the first candidate hypothesis 135 as the final transcription 275, then the NLU 220 ( Figure 1A and Figure 1B ) will perform semantic interpretation on the transcription 275 to identify "is it funny" as speech that is not intended for the AED 102 (because there is no context as to what / who the term "it" corresponds to), or incorrectly identify it as a follow-up query 129 to be submitted to the digital assistant 109. In the former case, the digital assistant 109 can ignore the follow-up query, or prompt the user to submit the follow-up query again. In the latter case, in which the incorrect transcription "is it funny" is identified as a follow-up query 129 submitted to the digital assistant 109, the digital assistant 109 will retrieve a follow-up response 193 that is not related to the follow-up query 129 actually spoken by the user 10. In either case, the user experience is severely degraded because the digital assistant 109 cannot return a follow-up response 193 that answers the follow-up query 129 actually spoken by the user 10.

[0057] The ASR system 200 may perform operations (C) and (D) as additional processing steps that utilize the initial / previous query 117 to bias the selection of a candidate hypothesis 225 from the plurality of candidate hypotheses 225 based on subject matter similarity to the initial / previous query 117. At operation (C), the QBST module 230 receives a set of top-ranked candidate hypotheses 225, 225a through 225n and determines a corresponding set of candidate QBST vectors 235, 235a through 235n, each candidate QBST vector being associated with a corresponding one of the set of top-ranked candidate hypotheses 225. The QBST module 230 similarly receives a previous query 117 (e.g., the initial query 117 “What is the temperature in Detroit”) and determines a corresponding previous QBST vector 217. If the current subsequent query being processed corresponds to a second or later subsequent query during an ongoing ongoing conversation between the user 10 and the digital assistant 109, the QBST 230 may optionally receive multiple previous queries 117 from the ongoing conversation and determine a corresponding previous QBST vector 217 for each of the multiple previous queries 117. For simplicity, the present disclosure will only describe the QBST 230 that receives only a single previous query 117.

[0058] Each QBST vector 217, 225 may indicate a set of significant terms with corresponding weights. For example, the previous QBST vector 217 corresponding to the previous query 117 ("What is the temperature in Detroit?") may indicate a corresponding set of significant terms (with high weights), which include, but are not limited to, "temperature", "weather", "forecast", "warm", "cool", etc. On the other hand, the first candidate QBST vector 235a corresponding to the top-ranked but incorrect first candidate hypothesis 225 "is it funny" may indicate a corresponding set of significant terms including "funny", "joke", "comedy", etc., while the second candidate QBST vector 235b corresponding to the correct second candidate hypothesis "is it sunny" may indicate a corresponding set of significant terms including "sun", "clear sky", "weather", "forecast", etc., which share more significant terms with the previous query 117.

[0059] At operation (D), the similarity scorer 240 receives the previous QBST vector 217 and the candidate QBST vector 235 and determines, for each of the candidate hypotheses 225, 225a to 225n, a corresponding similarity metric 245, 245a to 245n between the corresponding previous QBST vector 217 and the corresponding candidate QBST vector 235 output from the QBST module 230 for the corresponding candidate hypothesis 225. In other words, the similarity scorer 240 compares the previous QBST vector 217 with each of the candidate QBST vectors 235 to determine similarity metrics 245, each of which indicates a topic drift between the previous QBST vector 217 and the corresponding candidate QBST vector 235. Each similarity metric 245 may include a similarity score between the previous QBST vector 217 and the corresponding candidate QBST vector 235. In some examples, each similarity metric 245 includes a cosine score / distance between the previous QBST vector 217 and the corresponding candidate QBST vector 235 to account for the term weights in the pair of QBST vectors 217, 235. For example, continuing with the above example, a similarity metric 245 including cosine scores between the previous QBST vector 217 and each of the first and second candidate QBST vectors 235a, 235b may be represented as follows: Similarity (what is the temperature in Detroit, is it funny) = 0.001 Similarity (what is the temperature in Detroit, is it sunny) = 0.793

[0060] Thus, the ASR system 200 can use the similarity measure 245 (e.g., cosine score) output by the similarity scorer 240 for each of the top-ranked candidate hypotheses 225 identified during operation (B) to select a final transcription 275 for indicating the subsequent query 129 that will ultimately be processed by the digital assistant 109. For example, in the above example, although the second candidate hypothesis 225b "Is it sunny" has a lower likelihood score 155 than the top-ranked second candidate hypothesis 225a "Is it funny", it is associated with a higher similarity score 245 and, therefore, is more similar to the overall topic of the previous query 117 in the conversation with the digital assistant 109. That is, through the intuition that the current subsequent query is on the same overall topic as the previous query, the QBST module 230 and the similarity scorer 240 enable the ASR system 200 to select one of the candidate hypotheses 225 that is most similar to the previous query 117 as the correct final transcription 275 for the subsequent query 129 actually spoken by the user 10.

[0061] The similarity scorer 240 can be adapted to consider multiple different similarities between the previous query 117 and the plurality of candidate hypotheses 225. In one similarity consideration, as described above, the similarity scorer 240 determines a similarity measure 245 between the previous query 117 and each of the N alternative candidate hypotheses 225 so that the candidate hypotheses can be sorted / ranked based on their similarity to the previous query. In an additional similarity consideration, the similarity scorer 240 determines a similarity measure 245 between the candidate hypotheses 225 so that some candidate hypotheses 225 that are not meaningfully different from the previous query 117 can be pruned. For example, if the top candidate hypothesis 225 is "it sunny" and one of the other alternative candidate hypotheses 225 is "is it sunny", then that alternative can be pruned and removed from consideration because it is close enough to the top candidate hypothesis 225 that it will result in the retrieval of the same subsequent response 193. In this case, two of the candidate hypotheses 225 may be selected for processing and fulfillment by the NLU model 220 to ultimately determine which candidate hypothesis resulted in the retrieval of the best subsequent response 193. In another additional similarity consideration, the similarity scorer 240 determines how much closer the alternative candidate hypothesis 225 is to the previous query 118 in similarity compared to the original (e.g., highest ranked) candidate hypothesis 225: Similarity(previous query, alternative candidate hypothesis) - Similarity(previous query, top ranked candidate hypothesis) Here, the equation estimates an improvement that allows the ASR system 200 to attempt to select an alternative candidate hypothesis 225 that is meaningfully different from the highest ranked candidate hypothesis but not significantly different from the previous query 118. For example, considering the previous query "temperature in New York (New York's temperature)" and candidate hypotheses for subsequent queries including "isthere sun in Newark (Newark has the sun)" and "is there fun in New York (New York good fun)", the sharing of the salient terms "sun", "weather", "temperature" can potentially lead to the selection of the candidate hypothesis "is there sun in Newark". However, because the alternative replaces "New York" with "Newark", which is unlikely to be the correct transcription, any improvement in topic similarity that is considered can be weighted accordingly, so that the less likely alternative candidate hypothesis is not selected even if it is closer to the same topic as the initial query 117.

[0062] During operation (E), the re-ranker 250 receives the similarity metric 245 and the similarity score / metric 225 of the candidate hypothesis 225 from the similarity scorer 240, and receives the likelihood score 155 of the candidate hypothesis 225 from the ASR model 210. The re-ranker 250 is configured to output a re-ranking result 255 including a ranking of the plurality of candidate hypotheses 225 based on the biased likelihood score 156. In the example shown, the re-ranking result 255 includes the candidate hypothesis 225 "Is it sunny" having a biased likelihood score 156 of 0.8 as the most likely correct transcription 275. The NLU model 220 can perform semantic analysis on the correct transcription 275 to identify an appropriate follow-up query 129 for the digital assistant 109 to fulfill.

[0063] Additionally or alternatively, the QBST module 230 may determine a QBST vector for the response 192 to the initial query 117 submitted by the user, and determine a corresponding similarity measure between each candidate QBST vector 235 and the QBST vector determined for the response 192 to the initial query 117. Thus, selection of candidate hypotheses for the final transcription 275 of the subsequent query 129 may be biased based on the subject matter similarity between the response 192 to the initial query 117 and each of the candidate hypotheses.

[0064] Figure 4 4 is a flow chart of an example arrangement of operations of a method 400 for transcribing subsequent audio data during operation of an assistant-enabled device 102 in a continuous conversation mode. The method 400 may include performing a Figure 5) is a computer-implemented method executed on a computer. The data processing hardware 510 may include the data processing hardware 103 of the assistant-enabled device 102 of Figure 1 or the data processing hardware 112 of the remote system 110. At operation 402, the method 400 includes receiving subsequent audio data 127 captured by the AED 102. Here, the subsequent audio data 127 may correspond to a subsequent query 129 spoken by the user 10 of the AED 102 to the digital assistant 109 after the user 10 of the AED 102 submitted a previous query 117 to the digital assistant 109. After the user 10 submitted the previous query 117, the AED 102 may capture the subsequent audio data 127 while operating in a continuous conversation mode. During operation in the continuous conversation mode, the AED 102 is awake and instructs the speech recognizer 210 to perform speech recognition on the audio data captured by the AED 102 without the user explicitly speaking a predetermined hot word.

[0065] At operation 404, the method 400 includes processing the subsequent audio data 127 using the speech recognizer 210 to generate a plurality of candidate hypotheses 225. Each candidate hypothesis 225 corresponds to a candidate transcription of the subsequent query 129 and is represented by a corresponding series of hypothesis terms.

[0066] At operation 406, for each corresponding candidate hypothesis 225 among the plurality of candidate hypotheses, the method 400 further includes determining a corresponding similarity measure 245 between the previous query 117 and the corresponding candidate hypothesis 225. Here, the similarity measure indicates a similarity between a topic associated with the corresponding candidate hypothesis and a topic associated with the previous query. In some examples, the similarity measure includes a cosine distance between a QBST vector 217 associated with the previous query 117 and a corresponding candidate QBST vector 235 associated with the corresponding candidate hypothesis 225. At operation 408, the method 400 includes determining a transcription 225 of a subsequent query 129 spoken by the user based on the similarity measures 245 determined for the plurality of candidate hypotheses 225.

[0067] A software application (i.e., software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0068] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0069] Figure 5 5 is a schematic diagram of an example computing device 500 that can be used to implement the systems and methods described in this document. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0070] The computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor (i.e., data processing hardware) 510 can process instructions for execution within the computing device 500, including instructions stored in the memory 520 or on the storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 580 coupled to the high-speed interface 540). The data processing hardware 510 may include the data processing hardware 103 of the assistant-supporting device 102 of FIG. 1 or the data processing hardware 112 of the remote system 110. In other implementations, multiple processors and / or multiple buses may be used as appropriate, as well as multiple memories and types of memories. Also, multiple computing devices 500 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, blade server group, or multi-processor system).

[0071] Memory 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 520 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0072] Storage device 530 is capable of providing mass storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various implementations, storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as memory 520, storage device 530, or a memory on processor 510.

[0073] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or a router, for example, through a network adapter.

[0074] The computing device 500 may be implemented in a variety of different forms, as shown. For example, the computing device may be implemented as a standard server 500a or multiple times in a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0075] Various implementations of the systems and techniques described herein may be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that may be executed and / or interpreted on a programmable system that includes at least one programmable processor that may be either special purpose or general purpose and that may be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0076] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0077] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, a processor suitable for executing a computer program includes both a general-purpose microprocessor and a special-purpose microprocessor, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from one or more mass storage devices or to transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0078] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0079] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (400) executed on data processing hardware (510) causing the data processing hardware (510) to perform operations comprising: receiving subsequent audio data (127) captured by an assistant-enabled device (102), the subsequent audio data (127) corresponding to a subsequent query (119) spoken by a user of the assistant-enabled device (102) to the digital assistant (109) after the user submitted a previous query (117) to the digital assistant (109); processing the subsequent audio data (127) using a speech recognizer (210) to generate a plurality of candidate hypotheses (225), each candidate hypothesis (225) corresponding to a candidate transcription of the subsequent query (119) and represented by a corresponding series of hypothesis terms; For each corresponding candidate hypothesis (225) among the plurality of candidate hypotheses (225), determining a corresponding similarity measure (245) between the previous query (117) and the corresponding candidate hypothesis (225), the similarity measure (245) indicating a similarity between a topic associated with the corresponding candidate hypothesis (225) and a topic associated with the previous query (117); as well as A transcription of the subsequent query (119) spoken by the user is determined based on the similarity measure (245) determined for the plurality of candidate hypotheses (225).

2. The computer-implemented method (400) of claim 1, wherein the operations further comprise: determining a previous query-based salient term (QBST) vector (217) associated with the previous query (117) submitted by the user; as well as For each corresponding candidate hypothesis (225), determining a corresponding candidate QBST vector (235) associated with the corresponding candidate hypothesis (225), Wherein, determining the corresponding similarity measure (245) between the previous query (117) and the corresponding candidate hypothesis (225) includes: determining the corresponding similarity measure (245) between the previous QBST vector (217) and the corresponding candidate QBST vector (235) based on the previous QBST vector (217) and the corresponding candidate QBST vector (235).

3. The computer-implemented method (400) of claim 2, wherein: The previous QBST vector (217) indicates a corresponding set of salient terms associated with the previous query (117); and Each corresponding candidate QBST vector (235) indicates a respective set of salient terms associated with the corresponding candidate hypothesis (225).

4. The computer-implemented method (400) of claim 2 or 3, wherein the corresponding similarity metric (245) indicates a topic drift between the previous QBST vector (217) and the corresponding candidate QBST vector (235).

5. The computer-implemented method (400) of claim 2 or 3, wherein the corresponding similarity metric (245) comprises a cosine score between the previous QBST vector (217) and the corresponding candidate QBST vector (235).

6. The computer-implemented method (400) of any one of claims 1 to 5, wherein the operations further comprise: receiving initial audio data (127) captured by the assistant-enabled device (102) while the assistant-enabled device (102) is in a sleep state, the initial audio data (127) including a hot word (116) submitted by the user to the digital assistant (109) and the previous query (117), the hot word (116) when detected by the assistant-enabled device (102) causing the assistant-enabled device (102) to wake up from the sleep state and triggering the speech recognizer (210) to perform speech recognition on at least a portion of the initial audio data (127) including the previous query (117); and instructing the assistant-enabled device (102) to operate in a subsequent query mode after the speech recognizer (210) performs speech recognition on at least the portion of the initial audio data (127), Wherein, receiving the subsequent audio data (127) by the assistant-supporting device (102) includes receiving the subsequent audio data (127) during operation of the assistant-supporting device (102) in the subsequent query mode, wherein the hot word (116) is not present in the subsequent audio data (127).

7. The computer-implemented method (400) of any one of claims 1 to 6, wherein the operations further comprise: performing query interpretation on the transcription to identify an operation specified by the subsequent query (119); instructing the digital assistant (109) to perform the operation specified by the subsequent query (119); as well as A subsequent response (193) is received from the digital assistant (109), the subsequent response (193) indicating performance of the operation specified by the subsequent query (119).

8. The computer-implemented method (400) of claim 7, wherein the operations further comprise presenting the subsequent response (193) for output from the assistant-supporting device (102).

9. The computer-implemented method (400) of any one of claims 1 to 8, wherein the operations further comprise: For each corresponding candidate hypothesis (225) among the plurality of candidate hypotheses (225) generated by the speech recognizer (210), obtaining a corresponding likelihood score (155) that the speech recognizer (210) has assigned to the corresponding candidate hypothesis (225); and ranking the plurality of candidate hypotheses (225) based on the corresponding likelihood scores (155) assigned to the plurality of candidate hypotheses (225) by the speech recognizer (210) and the corresponding similarity scores (245) determined for each corresponding candidate hypothesis (225) among the plurality of candidate hypotheses (225), Wherein the determination of the transcription of the subsequent query (119) is based on the ranking of the plurality of candidate hypotheses (225).

10. The computer-implemented method (400) of any one of claims 1 to 9, wherein the data processing hardware (510) resides on the assistant-supporting device (102) and executes the speech recognizer (210).

11. A computer-implemented method (400) according to any one of claims 1 to 9, wherein the data processing hardware (510) resides on a remote server (110) and executes the speech recognizer (210), and the remote server (110) communicates with the assistant-supporting device (102) via a network (104).

12. The computer-implemented method (400) of any one of claims 1 to 11, wherein the speech recognizer (210) comprises an end-to-end speech recognition model.

13. The computer-implemented method (400) of any one of claims 1 to 11, wherein the speech recognizer (210) comprises an acoustic model and a language model.

14. A computer-implemented method (400) according to any one of claims 1 to 13, wherein the assistant-enabled device (102) communicates with one or more microphones (106), and the one or more microphones are configured to capture the subsequent audio data (127) and initial audio data (118) corresponding to the previous query (117).

15. The computer-implemented method (400) of any one of claims 1 to 14, wherein the assistant-enabled device (102) comprises a battery-powered device.

16. A system (100), comprising: Data processing hardware (510); as well as Memory hardware (520) in communication with the data processing hardware (510) and storing instructions that, when executed on the data processing hardware (510), cause the data processing hardware (510) to perform operations comprising: receiving subsequent audio data (127) captured by an assistant-enabled device (102), the subsequent audio data (127) corresponding to a subsequent query (119) spoken by a user of the assistant-enabled device (102) to the digital assistant (109) after the user submitted a previous query (117) to the digital assistant (109); processing the subsequent audio data (127) using a speech recognizer (210) to generate a plurality of candidate hypotheses (225), each candidate hypothesis (225) corresponding to a candidate transcription of the subsequent query (119) and represented by a corresponding series of hypothesis terms; For each corresponding candidate hypothesis (225) among the plurality of candidate hypotheses (225), determining a corresponding similarity measure (245) between the previous query (117) and the corresponding candidate hypothesis (225), the similarity measure (245) indicating a similarity between a topic associated with the corresponding candidate hypothesis (225) and a topic associated with the previous query (117); and A transcription of the subsequent query (119) spoken by the user is determined based on the similarity measure (245) determined for the plurality of candidate hypotheses (225).

17. The system (100) of claim 16, wherein the operations further comprise: determining a previous query-based salient term (QBST) vector (217) associated with the previous query (117) submitted by the user; as well as For each corresponding candidate hypothesis (225), determining a corresponding candidate QBST vector (235) associated with the corresponding candidate hypothesis (225), Wherein, determining the corresponding similarity measure (245) between the previous query (117) and the corresponding candidate hypothesis (225) includes: determining the corresponding similarity measure (245) between the previous QBST vector (217) and the corresponding candidate QBST vector (235) based on the previous QBST vector (217) and the corresponding candidate QBST vector (235).

18. The system (100) of claim 17, wherein: The previous QBST vector (217) indicates a corresponding set of salient terms associated with the previous query (117); and Each corresponding candidate QBST vector (235) indicates a respective set of salient terms associated with the corresponding candidate hypothesis (225).

19. The system (100) of claim 17 or 18, wherein the corresponding similarity metric (245) indicates a topic drift between the previous QBST vector (217) and the corresponding candidate QBST vector (235).

20. The system (100) of claim 17 or 18, wherein the corresponding similarity metric (245) comprises a cosine score between the previous QBST vector (217) and the corresponding candidate QBST vector (235).

21. The system (100) according to any one of claims 16 to 20, wherein the operations further comprise: receiving initial audio data (127) captured by the assistant-enabled device while the assistant-enabled device is in a sleep state, the initial audio data (127) including a hot word (116) submitted by the user to the digital assistant (109) and the previous query (117), the hot word (116) when detected by the assistant-enabled device causing the assistant-enabled device to wake up from the sleep state and triggering the speech recognizer (210) to perform speech recognition on at least a portion of the initial audio data (127) including the previous query (117); and instructing the assistant-enabled device (102) to operate in a subsequent query mode after the speech recognizer (210) performs speech recognition on at least the portion of the initial audio data (127), Wherein, receiving the subsequent audio data (127) by the assistant-supporting device includes receiving the subsequent audio data (127) during operation of the assistant-supporting device in the subsequent query mode, wherein the hot word (116) is not present in the subsequent audio data (127).

22. The system (100) according to any one of claims 16 to 21, wherein the operations further comprise: performing query interpretation on the transcription to identify an operation specified by the subsequent query; instructing the digital assistant (109) to perform the operation specified by the subsequent query; as well as A subsequent response is received from the digital assistant (109), the subsequent response indicating performance of the operation specified by the subsequent query.

23. The system (100) of claim 22, wherein the operations further comprise presenting the subsequent response for output from the assistant-supporting device (102).

24. The system (100) according to any one of claims 16 to 23, wherein the operations further comprise: For each corresponding candidate hypothesis (225) among the plurality of candidate hypotheses (225) generated by the speech recognizer (210), obtaining a corresponding likelihood score (155) that the speech recognizer (210) has assigned to the corresponding candidate hypothesis (225); and ranking the plurality of candidate hypotheses (225) based on the corresponding likelihood scores (155) assigned to the plurality of candidate hypotheses (225) by the speech recognizer (210) and the corresponding similarity scores (245) determined for each corresponding candidate hypothesis (225) among the plurality of candidate hypotheses (225), Wherein the determination of the transcription of the subsequent query is based on the ranking of the plurality of candidate hypotheses (225).

25. The system (100) of any one of claims 16 to 24, wherein the data processing hardware (510) resides on a device supporting the assistant and executes the speech recognizer (210).

26. A system (100) according to any one of claims 16 to 24, wherein the data processing hardware (510) resides on a remote server (110) and executes the speech recognizer (210), and the remote server (110) communicates with the assistant-supporting device (102) via a network (104).

27. The system (100) according to any one of claims 16 to 26, wherein the speech recognizer (210) comprises an end-to-end speech recognition model.

28. The system (100) according to any one of claims 16 to 26, wherein the speech recognizer (210) comprises an acoustic model and a language model.

29. A system (100) according to any one of claims 16 to 28, wherein the assistant-enabled device communicates with one or more microphones (106), and the one or more microphones are configured to capture the subsequent audio data (127) and the initial audio data (128) corresponding to the previous query (117).

30. The system (100) of any one of claims 16 to 28, wherein the assistant-enabled device comprises a battery-powered device.

Images